Diff of the two buildlogs: -- --- b1/build.log 2023-08-02 15:11:42.761877636 +0000 +++ b2/build.log 2023-08-02 15:13:16.638758019 +0000 @@ -1,6 +1,6 @@ I: pbuilder: network access will be disabled during build -I: Current time: Tue Sep 3 09:33:24 -12 2024 -I: pbuilder-time-stamp: 1725399204 +I: Current time: Thu Aug 3 05:11:44 +14 2023 +I: pbuilder-time-stamp: 1690989104 I: Building the build Environment I: extracting base tarball [/var/cache/pbuilder/bullseye-reproducible-base.tgz] I: copying local configuration @@ -17,8 +17,8 @@ I: copying [./golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz] I: Extracting source gpgv: unknown type of key resource 'trustedkeys.kbx' -gpgv: keyblock resource '/tmp/dpkg-verify-sig.bXYMr7Rk/trustedkeys.kbx': General error -gpgv: Signature made Sat Jan 18 19:24:28 2020 -12 +gpgv: keyblock resource '/tmp/dpkg-verify-sig.zybKIOlR/trustedkeys.kbx': General error +gpgv: Signature made Sun Jan 19 21:24:28 2020 +14 gpgv: using RSA key 50BC7CF939D20C272A6B065652B6BBD953968D1B gpgv: Can't check signature: No public key dpkg-source: warning: failed to verify signature on ./golang-github-hashicorp-raft_1.1.2-1.dsc @@ -33,135 +33,166 @@ dpkg-source: info: applying t-skip--TestRaft_SnapshotRestore_PeerChange.patch I: using fakeroot in build. I: Installing the build-deps -I: user script /srv/workspace/pbuilder/1777596/tmp/hooks/D02_print_environment starting +I: user script /srv/workspace/pbuilder/2531752/tmp/hooks/D01_modify_environment starting +debug: Running on ionos11-amd64. +I: Changing host+domainname to test build reproducibility +I: Adding a custom variable just for the fun of it... +I: Changing /bin/sh to bash +'/bin/sh' -> '/bin/bash' +lrwxrwxrwx 1 root root 9 Aug 3 05:11 /bin/sh -> /bin/bash +I: Setting pbuilder2's login shell to /bin/bash +I: Setting pbuilder2's GECOS to second user,second room,second work-phone,second home-phone,second other +I: user script /srv/workspace/pbuilder/2531752/tmp/hooks/D01_modify_environment finished +I: user script /srv/workspace/pbuilder/2531752/tmp/hooks/D02_print_environment starting I: set - BUILDDIR='/build' - BUILDUSERGECOS='first user,first room,first work-phone,first home-phone,first other' - BUILDUSERNAME='pbuilder1' - BUILD_ARCH='amd64' - DEBIAN_FRONTEND='noninteractive' - DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all,-fixfilepath parallel=16 ' - DISTRIBUTION='bullseye' - HOME='/root' - HOST_ARCH='amd64' + BASH=/bin/sh + BASHOPTS=checkwinsize:cmdhist:complete_fullquote:extquote:force_fignore:globasciiranges:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath + BASH_ALIASES=() + BASH_ARGC=() + BASH_ARGV=() + BASH_CMDS=() + BASH_LINENO=([0]="12" [1]="0") + BASH_SOURCE=([0]="/tmp/hooks/D02_print_environment" [1]="/tmp/hooks/D02_print_environment") + BASH_VERSINFO=([0]="5" [1]="1" [2]="4" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu") + BASH_VERSION='5.1.4(1)-release' + BUILDDIR=/build + BUILDUSERGECOS='second user,second room,second work-phone,second home-phone,second other' + BUILDUSERNAME=pbuilder2 + BUILD_ARCH=amd64 + DEBIAN_FRONTEND=noninteractive + DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all,-fixfilepath parallel=15 ' + DIRSTACK=() + DISTRIBUTION=bullseye + EUID=0 + FUNCNAME=([0]="Echo" [1]="main") + GROUPS=() + HOME=/root + HOSTNAME=i-capture-the-hostname + HOSTTYPE=x86_64 + HOST_ARCH=amd64 IFS=' ' - INVOCATION_ID='9558fb43ab4d4e9e9949400debb94c87' - LANG='C' - LANGUAGE='en_US:en' - LC_ALL='C' - MAIL='/var/mail/root' - OPTIND='1' - PATH='/usr/sbin:/usr/bin:/sbin:/bin:/usr/games' - PBCURRENTCOMMANDLINEOPERATION='build' - PBUILDER_OPERATION='build' - PBUILDER_PKGDATADIR='/usr/share/pbuilder' - PBUILDER_PKGLIBDIR='/usr/lib/pbuilder' - PBUILDER_SYSCONFDIR='/etc' - PPID='1777596' - PS1='# ' - PS2='> ' + INVOCATION_ID=86b54b7b40ac49509f413555e39cf35d + LANG=C + LANGUAGE=et_EE:et + LC_ALL=C + MACHTYPE=x86_64-pc-linux-gnu + MAIL=/var/mail/root + OPTERR=1 + OPTIND=1 + OSTYPE=linux-gnu + PATH=/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path + PBCURRENTCOMMANDLINEOPERATION=build + PBUILDER_OPERATION=build + PBUILDER_PKGDATADIR=/usr/share/pbuilder + PBUILDER_PKGLIBDIR=/usr/lib/pbuilder + PBUILDER_SYSCONFDIR=/etc + PIPESTATUS=([0]="0") + POSIXLY_CORRECT=y + PPID=2531752 PS4='+ ' - PWD='/' - SHELL='/bin/bash' - SHLVL='2' - SUDO_COMMAND='/usr/bin/timeout -k 18.1h 18h /usr/bin/ionice -c 3 /usr/bin/nice /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.zfXk6ywP/pbuilderrc_DlJx --distribution bullseye --hookdir /etc/pbuilder/first-build-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/bullseye-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.zfXk6ywP/b1 --logfile b1/build.log golang-github-hashicorp-raft_1.1.2-1.dsc' - SUDO_GID='110' - SUDO_UID='105' - SUDO_USER='jenkins' - TERM='unknown' - TZ='/usr/share/zoneinfo/Etc/GMT+12' - USER='root' - _='/usr/bin/systemd-run' - http_proxy='http://85.184.249.68:3128' + PWD=/ + SHELL=/bin/bash + SHELLOPTS=braceexpand:errexit:hashall:interactive-comments:posix + SHLVL=3 + SUDO_COMMAND='/usr/bin/timeout -k 24.1h 24h /usr/bin/ionice -c 3 /usr/bin/nice -n 11 /usr/bin/unshare --uts -- /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.zfXk6ywP/pbuilderrc_RcYU --distribution bullseye --hookdir /etc/pbuilder/rebuild-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/bullseye-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.zfXk6ywP/b2 --logfile b2/build.log golang-github-hashicorp-raft_1.1.2-1.dsc' + SUDO_GID=111 + SUDO_UID=106 + SUDO_USER=jenkins + TERM=unknown + TZ=/usr/share/zoneinfo/Etc/GMT-14 + UID=0 + USER=root + _='I: set' + http_proxy=http://78.137.99.97:3128 I: uname -a - Linux ionos5-amd64 6.1.0-10-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.38-2 (2023-07-27) x86_64 GNU/Linux + Linux i-capture-the-hostname 6.1.0-10-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.38-2 (2023-07-27) x86_64 GNU/Linux I: ls -l /bin total 5476 - -rwxr-xr-x 1 root root 1234376 Mar 27 2022 bash - -rwxr-xr-x 3 root root 38984 Jul 20 2020 bunzip2 - -rwxr-xr-x 3 root root 38984 Jul 20 2020 bzcat - lrwxrwxrwx 1 root root 6 Jul 20 2020 bzcmp -> bzdiff - -rwxr-xr-x 1 root root 2225 Jul 20 2020 bzdiff - lrwxrwxrwx 1 root root 6 Jul 20 2020 bzegrep -> bzgrep - -rwxr-xr-x 1 root root 4877 Sep 4 2019 bzexe - lrwxrwxrwx 1 root root 6 Jul 20 2020 bzfgrep -> bzgrep - -rwxr-xr-x 1 root root 3775 Jul 20 2020 bzgrep - -rwxr-xr-x 3 root root 38984 Jul 20 2020 bzip2 - -rwxr-xr-x 1 root root 18424 Jul 20 2020 bzip2recover - lrwxrwxrwx 1 root root 6 Jul 20 2020 bzless -> bzmore - -rwxr-xr-x 1 root root 1297 Jul 20 2020 bzmore - -rwxr-xr-x 1 root root 43936 Sep 23 2020 cat - -rwxr-xr-x 1 root root 72672 Sep 23 2020 chgrp - -rwxr-xr-x 1 root root 64448 Sep 23 2020 chmod - -rwxr-xr-x 1 root root 72672 Sep 23 2020 chown - -rwxr-xr-x 1 root root 151168 Sep 23 2020 cp - -rwxr-xr-x 1 root root 125560 Dec 10 2020 dash - -rwxr-xr-x 1 root root 113664 Sep 23 2020 date - -rwxr-xr-x 1 root root 80968 Sep 23 2020 dd - -rwxr-xr-x 1 root root 93936 Sep 23 2020 df - -rwxr-xr-x 1 root root 147176 Sep 23 2020 dir - -rwxr-xr-x 1 root root 84440 Jan 20 2022 dmesg - lrwxrwxrwx 1 root root 8 Nov 6 2019 dnsdomainname -> hostname - lrwxrwxrwx 1 root root 8 Nov 6 2019 domainname -> hostname - -rwxr-xr-x 1 root root 39712 Sep 23 2020 echo - -rwxr-xr-x 1 root root 28 Jan 24 2023 egrep - -rwxr-xr-x 1 root root 39680 Sep 23 2020 false - -rwxr-xr-x 1 root root 28 Jan 24 2023 fgrep - -rwxr-xr-x 1 root root 69032 Jan 20 2022 findmnt - -rwsr-xr-x 1 root root 34896 Feb 26 2021 fusermount - -rwxr-xr-x 1 root root 203072 Jan 24 2023 grep - -rwxr-xr-x 2 root root 2346 Apr 9 2022 gunzip - -rwxr-xr-x 1 root root 6447 Apr 9 2022 gzexe - -rwxr-xr-x 1 root root 98048 Apr 9 2022 gzip - -rwxr-xr-x 1 root root 22600 Nov 6 2019 hostname - -rwxr-xr-x 1 root root 72840 Sep 23 2020 ln - -rwxr-xr-x 1 root root 56952 Feb 7 2020 login - -rwxr-xr-x 1 root root 147176 Sep 23 2020 ls - -rwxr-xr-x 1 root root 149736 Jan 20 2022 lsblk - -rwxr-xr-x 1 root root 85184 Sep 23 2020 mkdir - -rwxr-xr-x 1 root root 76896 Sep 23 2020 mknod - -rwxr-xr-x 1 root root 48064 Sep 23 2020 mktemp - -rwxr-xr-x 1 root root 59632 Jan 20 2022 more - -rwsr-xr-x 1 root root 55528 Jan 20 2022 mount - -rwxr-xr-x 1 root root 18664 Jan 20 2022 mountpoint - -rwxr-xr-x 1 root root 147080 Sep 23 2020 mv - lrwxrwxrwx 1 root root 8 Nov 6 2019 nisdomainname -> hostname - lrwxrwxrwx 1 root root 14 Dec 16 2021 pidof -> /sbin/killall5 - -rwxr-xr-x 1 root root 43872 Sep 23 2020 pwd - lrwxrwxrwx 1 root root 4 Mar 27 2022 rbash -> bash - -rwxr-xr-x 1 root root 52032 Sep 23 2020 readlink - -rwxr-xr-x 1 root root 72704 Sep 23 2020 rm - -rwxr-xr-x 1 root root 52032 Sep 23 2020 rmdir - -rwxr-xr-x 1 root root 27472 Sep 27 2020 run-parts - -rwxr-xr-x 1 root root 122224 Dec 22 2018 sed - lrwxrwxrwx 1 root root 4 Aug 8 03:47 sh -> dash - -rwxr-xr-x 1 root root 43808 Sep 23 2020 sleep - -rwxr-xr-x 1 root root 84928 Sep 23 2020 stty - -rwsr-xr-x 1 root root 71912 Jan 20 2022 su - -rwxr-xr-x 1 root root 39744 Sep 23 2020 sync - -rwxr-xr-x 1 root root 531928 Feb 16 2021 tar - -rwxr-xr-x 1 root root 14456 Sep 27 2020 tempfile - -rwxr-xr-x 1 root root 101408 Sep 23 2020 touch - -rwxr-xr-x 1 root root 39680 Sep 23 2020 true - -rwxr-xr-x 1 root root 14328 Feb 26 2021 ulockmgr_server - -rwsr-xr-x 1 root root 35040 Jan 20 2022 umount - -rwxr-xr-x 1 root root 39744 Sep 23 2020 uname - -rwxr-xr-x 2 root root 2346 Apr 9 2022 uncompress - -rwxr-xr-x 1 root root 147176 Sep 23 2020 vdir - -rwxr-xr-x 1 root root 63744 Jan 20 2022 wdctl - lrwxrwxrwx 1 root root 8 Nov 6 2019 ypdomainname -> hostname - -rwxr-xr-x 1 root root 1984 Apr 9 2022 zcat - -rwxr-xr-x 1 root root 1678 Apr 9 2022 zcmp - -rwxr-xr-x 1 root root 5898 Apr 9 2022 zdiff - -rwxr-xr-x 1 root root 29 Apr 9 2022 zegrep - -rwxr-xr-x 1 root root 29 Apr 9 2022 zfgrep - -rwxr-xr-x 1 root root 2081 Apr 9 2022 zforce - -rwxr-xr-x 1 root root 8049 Apr 9 2022 zgrep - -rwxr-xr-x 1 root root 2206 Apr 9 2022 zless - -rwxr-xr-x 1 root root 1842 Apr 9 2022 zmore - -rwxr-xr-x 1 root root 4577 Apr 9 2022 znew -I: user script /srv/workspace/pbuilder/1777596/tmp/hooks/D02_print_environment finished + -rwxr-xr-x 1 root root 1234376 Mar 28 2022 bash + -rwxr-xr-x 3 root root 38984 Jul 21 2020 bunzip2 + -rwxr-xr-x 3 root root 38984 Jul 21 2020 bzcat + lrwxrwxrwx 1 root root 6 Jul 21 2020 bzcmp -> bzdiff + -rwxr-xr-x 1 root root 2225 Jul 21 2020 bzdiff + lrwxrwxrwx 1 root root 6 Jul 21 2020 bzegrep -> bzgrep + -rwxr-xr-x 1 root root 4877 Sep 5 2019 bzexe + lrwxrwxrwx 1 root root 6 Jul 21 2020 bzfgrep -> bzgrep + -rwxr-xr-x 1 root root 3775 Jul 21 2020 bzgrep + -rwxr-xr-x 3 root root 38984 Jul 21 2020 bzip2 + -rwxr-xr-x 1 root root 18424 Jul 21 2020 bzip2recover + lrwxrwxrwx 1 root root 6 Jul 21 2020 bzless -> bzmore + -rwxr-xr-x 1 root root 1297 Jul 21 2020 bzmore + -rwxr-xr-x 1 root root 43936 Sep 24 2020 cat + -rwxr-xr-x 1 root root 72672 Sep 24 2020 chgrp + -rwxr-xr-x 1 root root 64448 Sep 24 2020 chmod + -rwxr-xr-x 1 root root 72672 Sep 24 2020 chown + -rwxr-xr-x 1 root root 151168 Sep 24 2020 cp + -rwxr-xr-x 1 root root 125560 Dec 11 2020 dash + -rwxr-xr-x 1 root root 113664 Sep 24 2020 date + -rwxr-xr-x 1 root root 80968 Sep 24 2020 dd + -rwxr-xr-x 1 root root 93936 Sep 24 2020 df + -rwxr-xr-x 1 root root 147176 Sep 24 2020 dir + -rwxr-xr-x 1 root root 84440 Jan 21 2022 dmesg + lrwxrwxrwx 1 root root 8 Nov 8 2019 dnsdomainname -> hostname + lrwxrwxrwx 1 root root 8 Nov 8 2019 domainname -> hostname + -rwxr-xr-x 1 root root 39712 Sep 24 2020 echo + -rwxr-xr-x 1 root root 28 Jan 25 2023 egrep + -rwxr-xr-x 1 root root 39680 Sep 24 2020 false + -rwxr-xr-x 1 root root 28 Jan 25 2023 fgrep + -rwxr-xr-x 1 root root 69032 Jan 21 2022 findmnt + -rwsr-xr-x 1 root root 34896 Feb 27 2021 fusermount + -rwxr-xr-x 1 root root 203072 Jan 25 2023 grep + -rwxr-xr-x 2 root root 2346 Apr 10 2022 gunzip + -rwxr-xr-x 1 root root 6447 Apr 10 2022 gzexe + -rwxr-xr-x 1 root root 98048 Apr 10 2022 gzip + -rwxr-xr-x 1 root root 22600 Nov 8 2019 hostname + -rwxr-xr-x 1 root root 72840 Sep 24 2020 ln + -rwxr-xr-x 1 root root 56952 Feb 8 2020 login + -rwxr-xr-x 1 root root 147176 Sep 24 2020 ls + -rwxr-xr-x 1 root root 149736 Jan 21 2022 lsblk + -rwxr-xr-x 1 root root 85184 Sep 24 2020 mkdir + -rwxr-xr-x 1 root root 76896 Sep 24 2020 mknod + -rwxr-xr-x 1 root root 48064 Sep 24 2020 mktemp + -rwxr-xr-x 1 root root 59632 Jan 21 2022 more + -rwsr-xr-x 1 root root 55528 Jan 21 2022 mount + -rwxr-xr-x 1 root root 18664 Jan 21 2022 mountpoint + -rwxr-xr-x 1 root root 147080 Sep 24 2020 mv + lrwxrwxrwx 1 root root 8 Nov 8 2019 nisdomainname -> hostname + lrwxrwxrwx 1 root root 14 Dec 17 2021 pidof -> /sbin/killall5 + -rwxr-xr-x 1 root root 43872 Sep 24 2020 pwd + lrwxrwxrwx 1 root root 4 Mar 28 2022 rbash -> bash + -rwxr-xr-x 1 root root 52032 Sep 24 2020 readlink + -rwxr-xr-x 1 root root 72704 Sep 24 2020 rm + -rwxr-xr-x 1 root root 52032 Sep 24 2020 rmdir + -rwxr-xr-x 1 root root 27472 Sep 28 2020 run-parts + -rwxr-xr-x 1 root root 122224 Dec 23 2018 sed + lrwxrwxrwx 1 root root 9 Aug 3 05:11 sh -> /bin/bash + -rwxr-xr-x 1 root root 43808 Sep 24 2020 sleep + -rwxr-xr-x 1 root root 84928 Sep 24 2020 stty + -rwsr-xr-x 1 root root 71912 Jan 21 2022 su + -rwxr-xr-x 1 root root 39744 Sep 24 2020 sync + -rwxr-xr-x 1 root root 531928 Feb 17 2021 tar + -rwxr-xr-x 1 root root 14456 Sep 28 2020 tempfile + -rwxr-xr-x 1 root root 101408 Sep 24 2020 touch + -rwxr-xr-x 1 root root 39680 Sep 24 2020 true + -rwxr-xr-x 1 root root 14328 Feb 27 2021 ulockmgr_server + -rwsr-xr-x 1 root root 35040 Jan 21 2022 umount + -rwxr-xr-x 1 root root 39744 Sep 24 2020 uname + -rwxr-xr-x 2 root root 2346 Apr 10 2022 uncompress + -rwxr-xr-x 1 root root 147176 Sep 24 2020 vdir + -rwxr-xr-x 1 root root 63744 Jan 21 2022 wdctl + lrwxrwxrwx 1 root root 8 Nov 8 2019 ypdomainname -> hostname + -rwxr-xr-x 1 root root 1984 Apr 10 2022 zcat + -rwxr-xr-x 1 root root 1678 Apr 10 2022 zcmp + -rwxr-xr-x 1 root root 5898 Apr 10 2022 zdiff + -rwxr-xr-x 1 root root 29 Apr 10 2022 zegrep + -rwxr-xr-x 1 root root 29 Apr 10 2022 zfgrep + -rwxr-xr-x 1 root root 2081 Apr 10 2022 zforce + -rwxr-xr-x 1 root root 8049 Apr 10 2022 zgrep + -rwxr-xr-x 1 root root 2206 Apr 10 2022 zless + -rwxr-xr-x 1 root root 1842 Apr 10 2022 zmore + -rwxr-xr-x 1 root root 4577 Apr 10 2022 znew +I: user script /srv/workspace/pbuilder/2531752/tmp/hooks/D02_print_environment finished -> Attempting to satisfy build-dependencies -> Creating pbuilder-satisfydepends-dummy package Package: pbuilder-satisfydepends-dummy @@ -320,7 +351,7 @@ Get: 111 http://deb.debian.org/debian bullseye/main amd64 golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-1 [32.1 kB] Get: 112 http://deb.debian.org/debian bullseye/main amd64 golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB] Get: 113 http://deb.debian.org/debian bullseye/main amd64 golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB] -Fetched 96.4 MB in 1s (86.9 MB/s) +Fetched 96.4 MB in 2s (42.1 MB/s) debconf: delaying package configuration, since apt-utils is not installed Selecting previously unselected package bsdextrautils. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 19707 files and directories currently installed.) @@ -793,7 +824,11 @@ fakeroot is already the newest version (1.25.3-1.1). 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. I: Building the package -I: Running cd /build/golang-github-hashicorp-raft-1.1.2/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-genchanges -S > ../golang-github-hashicorp-raft_1.1.2-1_source.changes +I: user script /srv/workspace/pbuilder/2531752/tmp/hooks/A99_set_merged_usr starting +Not re-configuring usrmerge for bullseye +I: user script /srv/workspace/pbuilder/2531752/tmp/hooks/A99_set_merged_usr finished +hostname: Name or service not known +I: Running cd /build/golang-github-hashicorp-raft-1.1.2/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-genchanges -S > ../golang-github-hashicorp-raft_1.1.2-1_source.changes dpkg-buildpackage: info: source package golang-github-hashicorp-raft dpkg-buildpackage: info: source version 1.1.2-1 dpkg-buildpackage: info: source distribution unstable @@ -811,26 +846,26 @@ dh_autoreconf -O--buildsystem=golang dh_auto_configure -O--buildsystem=golang dh_auto_build -O--buildsystem=golang - cd obj-x86_64-linux-gnu && go install -trimpath -v -p 16 github.com/hashicorp/raft github.com/hashicorp/raft/bench -internal/unsafeheader -runtime/internal/sys + cd obj-x86_64-linux-gnu && go install -trimpath -v -p 15 github.com/hashicorp/raft github.com/hashicorp/raft/bench crypto/subtle internal/race -crypto/internal/subtle +internal/unsafeheader +encoding container/list -internal/cpu math/bits -runtime/internal/atomic +crypto/internal/subtle +runtime/internal/sys +internal/nettrace +sync/atomic vendor/golang.org/x/crypto/cryptobyte/asn1 -vendor/golang.org/x/crypto/internal/subtle -encoding -unicode/utf16 unicode -sync/atomic +runtime/internal/atomic unicode/utf8 -internal/nettrace -runtime/cgo +vendor/golang.org/x/crypto/internal/subtle +internal/cpu golang.org/x/sys/internal/unsafeheader +unicode/utf16 +runtime/cgo runtime/internal/math internal/bytealg internal/testlog @@ -838,33 +873,33 @@ runtime internal/reflectlite sync -internal/singleflight math/rand +internal/singleflight errors sort -io internal/oserror github.com/hashicorp/golang-lru/simplelru +io strconv vendor/golang.org/x/net/dns/dnsmessage syscall +hash bytes strings -hash crypto/internal/randutil +hash/crc32 crypto/hmac hash/crc64 -hash/crc32 vendor/golang.org/x/crypto/hkdf -crypto/rc4 crypto +crypto/rc4 reflect vendor/golang.org/x/text/transform bufio -path github.com/hashicorp/go-immutable-radix -html +path regexp/syntax +html internal/syscall/execenv internal/syscall/unix time @@ -876,77 +911,77 @@ os encoding/base64 crypto/cipher -crypto/md5 crypto/sha512 -crypto/sha1 -vendor/golang.org/x/crypto/poly1305 +crypto/md5 crypto/ed25519/internal/edwards25519 +vendor/golang.org/x/crypto/poly1305 crypto/sha256 +crypto/sha1 golang.org/x/sys/unix encoding/pem fmt -os/signal +path/filepath runtime/debug +os/signal net -path/filepath crypto/des -vendor/golang.org/x/crypto/chacha20 crypto/aes +vendor/golang.org/x/crypto/chacha20 io/ioutil vendor/golang.org/x/sys/cpu vendor/golang.org/x/crypto/chacha20poly1305 log +encoding/json compress/flate -encoding/hex -vendor/golang.org/x/net/http2/hpack +mime net/url -mime/quotedprintable -encoding/json vendor/golang.org/x/crypto/curve25519 -net/http/internal +encoding/hex +mime/quotedprintable math/big -mime -encoding/gob +vendor/golang.org/x/net/http2/hpack +net/http/internal vendor/golang.org/x/text/unicode/norm -go/token vendor/golang.org/x/text/unicode/bidi +encoding/gob +go/token text/template/parse flag runtime/trace -compress/gzip -testing github.com/mattn/go-isatty -vendor/golang.org/x/text/secure/bidirule github.com/mattn/go-colorable -vendor/golang.org/x/net/idna github.com/fatih/color -text/template +compress/gzip +testing +vendor/golang.org/x/text/secure/bidirule github.com/hashicorp/go-hclog +text/template +vendor/golang.org/x/net/idna crypto/rand +crypto/dsa encoding/asn1 crypto/elliptic -crypto/dsa -crypto/ed25519 crypto/rsa +crypto/ed25519 crypto/x509/pkix vendor/golang.org/x/crypto/cryptobyte html/template crypto/ecdsa -crypto/x509 -net/textproto vendor/golang.org/x/net/http/httpproxy +net/textproto +crypto/x509 vendor/golang.org/x/net/http/httpguts mime/multipart crypto/tls net/http/httptrace net/http -net/rpc github.com/armon/go-metrics +net/rpc github.com/hashicorp/go-msgpack/codec github.com/hashicorp/raft github.com/hashicorp/raft/bench dh_auto_test -O--buildsystem=golang - cd obj-x86_64-linux-gnu && go test -vet=off -v -p 16 github.com/hashicorp/raft github.com/hashicorp/raft/bench + cd obj-x86_64-linux-gnu && go test -vet=off -v -p 15 github.com/hashicorp/raft github.com/hashicorp/raft/bench === RUN TestCommitment_setVoters --- PASS: TestCommitment_setVoters (0.00s) === RUN TestCommitment_match_max @@ -988,32 +1023,32 @@ === RUN TestFileSnapshotSinkImpl --- PASS: TestFileSnapshotSinkImpl (0.00s) === RUN TestFileSS_CreateSnapshotMissingParentDir - testing.go:153: 2024-09-03T21:34:18.957Z [INFO] creating new snapshot: path=/tmp/raft516028444/raft714094795/snapshots/3-10-1725399258957.tmp + testing.go:153: 2023-08-02T15:12:47.909Z [INFO] creating new snapshot: path=/tmp/raft672931563/raft341413710/snapshots/3-10-1690989167909.tmp --- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.00s) === RUN TestFileSS_CreateSnapshot - testing.go:153: 2024-09-03T21:34:18.958Z [INFO] creating new snapshot: path=/tmp/raft748350894/snapshots/3-10-1725399258958.tmp - testing.go:153: 2024-09-03T21:34:18.958Z [WARN] found temporary snapshot: name=3-10-1725399258958.tmp + testing.go:153: 2023-08-02T15:12:47.910Z [INFO] creating new snapshot: path=/tmp/raft985338965/snapshots/3-10-1690989167910.tmp + testing.go:153: 2023-08-02T15:12:47.910Z [WARN] found temporary snapshot: name=3-10-1690989167910.tmp --- PASS: TestFileSS_CreateSnapshot (0.00s) === RUN TestFileSS_CancelSnapshot - testing.go:153: 2024-09-03T21:34:18.959Z [INFO] creating new snapshot: path=/tmp/raft730217781/snapshots/3-10-1725399258959.tmp + testing.go:153: 2023-08-02T15:12:47.911Z [INFO] creating new snapshot: path=/tmp/raft193297328/snapshots/3-10-1690989167911.tmp --- PASS: TestFileSS_CancelSnapshot (0.00s) === RUN TestFileSS_Retention - testing.go:153: 2024-09-03T21:34:18.959Z [INFO] creating new snapshot: path=/tmp/raft819524368/snapshots/3-10-1725399258959.tmp - testing.go:153: 2024-09-03T21:34:18.959Z [INFO] creating new snapshot: path=/tmp/raft819524368/snapshots/3-11-1725399258959.tmp - testing.go:153: 2024-09-03T21:34:18.959Z [INFO] creating new snapshot: path=/tmp/raft819524368/snapshots/3-12-1725399258959.tmp - testing.go:153: 2024-09-03T21:34:18.959Z [INFO] reaping snapshot: path=/tmp/raft819524368/snapshots/3-10-1725399258959 - testing.go:153: 2024-09-03T21:34:18.960Z [INFO] creating new snapshot: path=/tmp/raft819524368/snapshots/3-13-1725399258960.tmp - testing.go:153: 2024-09-03T21:34:18.960Z [INFO] reaping snapshot: path=/tmp/raft819524368/snapshots/3-11-1725399258959 - testing.go:153: 2024-09-03T21:34:18.960Z [INFO] creating new snapshot: path=/tmp/raft819524368/snapshots/3-14-1725399258960.tmp - testing.go:153: 2024-09-03T21:34:18.960Z [INFO] reaping snapshot: path=/tmp/raft819524368/snapshots/3-12-1725399258959 + testing.go:153: 2023-08-02T15:12:47.911Z [INFO] creating new snapshot: path=/tmp/raft572282703/snapshots/3-10-1690989167911.tmp + testing.go:153: 2023-08-02T15:12:47.911Z [INFO] creating new snapshot: path=/tmp/raft572282703/snapshots/3-11-1690989167911.tmp + testing.go:153: 2023-08-02T15:12:47.912Z [INFO] creating new snapshot: path=/tmp/raft572282703/snapshots/3-12-1690989167912.tmp + testing.go:153: 2023-08-02T15:12:47.912Z [INFO] reaping snapshot: path=/tmp/raft572282703/snapshots/3-10-1690989167911 + testing.go:153: 2023-08-02T15:12:47.912Z [INFO] creating new snapshot: path=/tmp/raft572282703/snapshots/3-13-1690989167912.tmp + testing.go:153: 2023-08-02T15:12:47.912Z [INFO] reaping snapshot: path=/tmp/raft572282703/snapshots/3-11-1690989167911 + testing.go:153: 2023-08-02T15:12:47.912Z [INFO] creating new snapshot: path=/tmp/raft572282703/snapshots/3-14-1690989167912.tmp + testing.go:153: 2023-08-02T15:12:47.913Z [INFO] reaping snapshot: path=/tmp/raft572282703/snapshots/3-12-1690989167912 --- PASS: TestFileSS_Retention (0.00s) === RUN TestFileSS_BadPerm --- PASS: TestFileSS_BadPerm (0.00s) === RUN TestFileSS_MissingParentDir --- PASS: TestFileSS_MissingParentDir (0.00s) === RUN TestFileSS_Ordering - testing.go:153: 2024-09-03T21:34:18.961Z [INFO] creating new snapshot: path=/tmp/raft320057811/snapshots/5-130350-1725399258961.tmp - testing.go:153: 2024-09-03T21:34:18.961Z [INFO] creating new snapshot: path=/tmp/raft320057811/snapshots/36-204917-1725399258961.tmp + testing.go:153: 2023-08-02T15:12:47.914Z [INFO] creating new snapshot: path=/tmp/raft600400566/snapshots/5-130350-1690989167914.tmp + testing.go:153: 2023-08-02T15:12:47.915Z [INFO] creating new snapshot: path=/tmp/raft600400566/snapshots/36-204917-1690989167915.tmp --- PASS: TestFileSS_Ordering (0.00s) === RUN TestDeferFutureSuccess --- PASS: TestDeferFutureSuccess (0.00s) @@ -1061,27 +1096,27 @@ === RUN TestNetworkTransport_PooledConn --- PASS: TestNetworkTransport_PooledConn (0.00s) === RUN TestNetworkTransport_ListenBackoff - net_transport_test.go:718: countingWriter: 2024-09-03T21:34:18.983Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2023-08-02T15:12:47.939Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2024-09-03T21:34:18.990Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2023-08-02T15:12:47.945Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2024-09-03T21:34:19.001Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2023-08-02T15:12:47.955Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2024-09-03T21:34:19.021Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2023-08-02T15:12:47.976Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2024-09-03T21:34:19.061Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2023-08-02T15:12:48.016Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2024-09-03T21:34:19.141Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2023-08-02T15:12:48.097Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2024-09-03T21:34:19.302Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2023-08-02T15:12:48.257Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2024-09-03T21:34:19.622Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2023-08-02T15:12:48.577Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2024-09-03T21:34:20.262Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2023-08-02T15:12:49.218Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2024-09-03T21:34:21.263Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2023-08-02T15:12:50.219Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2024-09-03T21:34:22.263Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2023-08-02T15:12:51.219Z [ERROR] test: failed to accept connection: error="intentional error in test" --- PASS: TestNetworkTransport_ListenBackoff (4.00s) === RUN TestPeersJSON_BadConfiguration @@ -1091,1103 +1126,1105 @@ === RUN TestPeersJSON_ReadConfigJSON --- PASS: TestPeersJSON_ReadConfigJSON (0.00s) === RUN TestRaft_StartStop -testLoggerAdapter verbose: server-bb081b85-c285-3357-c812-5431211d8853: 2024-09-03T21:34:22.985Z [INFO] server-bb081b85-c285-3357-c812-5431211d8853: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bb081b85-c285-3357-c812-5431211d8853 Address:bb081b85-c285-3357-c812-5431211d8853}]" - testing.go:149: server-bb081b85-c285-3357-c812-5431211d8853: 2024-09-03T21:34:22.985Z [INFO] server-bb081b85-c285-3357-c812-5431211d8853: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bb081b85-c285-3357-c812-5431211d8853 Address:bb081b85-c285-3357-c812-5431211d8853}]" +testLoggerAdapter verbose: server-69f27bdd-0b4f-2b12-601b-dc92a8ad8e42: 2023-08-02T15:12:51.948Z [INFO] server-69f27bdd-0b4f-2b12-601b-dc92a8ad8e42: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-69f27bdd-0b4f-2b12-601b-dc92a8ad8e42 Address:69f27bdd-0b4f-2b12-601b-dc92a8ad8e42}]" + testing.go:149: server-69f27bdd-0b4f-2b12-601b-dc92a8ad8e42: 2023-08-02T15:12:51.948Z [INFO] server-69f27bdd-0b4f-2b12-601b-dc92a8ad8e42: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-69f27bdd-0b4f-2b12-601b-dc92a8ad8e42 Address:69f27bdd-0b4f-2b12-601b-dc92a8ad8e42}]" +testLoggerAdapter verbose: server-69f27bdd-0b4f-2b12-601b-dc92a8ad8e42: 2023-08-02T15:12:51.948Z [INFO] server-69f27bdd-0b4f-2b12-601b-dc92a8ad8e42: entering follower state: follower="Node at 69f27bdd-0b4f-2b12-601b-dc92a8ad8e42 [Shutdown]" leader= + testing.go:149: server-69f27bdd-0b4f-2b12-601b-dc92a8ad8e42: 2023-08-02T15:12:51.948Z [INFO] server-69f27bdd-0b4f-2b12-601b-dc92a8ad8e42: entering follower state: follower="Node at 69f27bdd-0b4f-2b12-601b-dc92a8ad8e42 [Shutdown]" leader= --- PASS: TestRaft_StartStop (0.00s) === RUN TestRaft_AfterShutdown -testLoggerAdapter verbose: server-77de41b6-7539-a4d0-0671-eb1279c23537: 2024-09-03T21:34:22.985Z [INFO] server-77de41b6-7539-a4d0-0671-eb1279c23537: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77de41b6-7539-a4d0-0671-eb1279c23537 Address:77de41b6-7539-a4d0-0671-eb1279c23537}]" - testing.go:149: server-77de41b6-7539-a4d0-0671-eb1279c23537: 2024-09-03T21:34:22.985Z [INFO] server-77de41b6-7539-a4d0-0671-eb1279c23537: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77de41b6-7539-a4d0-0671-eb1279c23537 Address:77de41b6-7539-a4d0-0671-eb1279c23537}]" +testLoggerAdapter verbose: server-0ffb1b59-0940-86ce-355e-ea09372129af: 2023-08-02T15:12:51.950Z [INFO] server-0ffb1b59-0940-86ce-355e-ea09372129af: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0ffb1b59-0940-86ce-355e-ea09372129af Address:0ffb1b59-0940-86ce-355e-ea09372129af}]" + testing.go:149: server-0ffb1b59-0940-86ce-355e-ea09372129af: 2023-08-02T15:12:51.950Z [INFO] server-0ffb1b59-0940-86ce-355e-ea09372129af: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0ffb1b59-0940-86ce-355e-ea09372129af Address:0ffb1b59-0940-86ce-355e-ea09372129af}]" --- PASS: TestRaft_AfterShutdown (0.00s) === RUN TestRaft_LiveBootstrap -testLoggerAdapter verbose: server-a0db6add-839d-a858-9aab-7ee8fe5c7250: 2024-09-03T21:34:22.985Z [INFO] server-a0db6add-839d-a858-9aab-7ee8fe5c7250: initial configuration: index=0 servers=[] - testing.go:149: server-a0db6add-839d-a858-9aab-7ee8fe5c7250: 2024-09-03T21:34:22.985Z [INFO] server-a0db6add-839d-a858-9aab-7ee8fe5c7250: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: server-2abc2d08-68d3-fc8c-4e13-16e16d2d4cc2: 2024-09-03T21:34:22.986Z [INFO] server-2abc2d08-68d3-fc8c-4e13-16e16d2d4cc2: initial configuration: index=0 servers=[] - testing.go:149: server-2abc2d08-68d3-fc8c-4e13-16e16d2d4cc2: 2024-09-03T21:34:22.986Z [INFO] server-2abc2d08-68d3-fc8c-4e13-16e16d2d4cc2: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: server-da8162d5-a0b7-0586-160e-f4b36e106164: 2024-09-03T21:34:22.986Z [INFO] server-da8162d5-a0b7-0586-160e-f4b36e106164: initial configuration: index=0 servers=[] - testing.go:149: server-da8162d5-a0b7-0586-160e-f4b36e106164: 2024-09-03T21:34:22.986Z [INFO] server-da8162d5-a0b7-0586-160e-f4b36e106164: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: server-a0db6add-839d-a858-9aab-7ee8fe5c7250: 2024-09-03T21:34:22.986Z [INFO] server-a0db6add-839d-a858-9aab-7ee8fe5c7250: entering follower state: follower="Node at a0db6add-839d-a858-9aab-7ee8fe5c7250 [Follower]" leader= - testing.go:149: server-a0db6add-839d-a858-9aab-7ee8fe5c7250: 2024-09-03T21:34:22.986Z [INFO] server-a0db6add-839d-a858-9aab-7ee8fe5c7250: entering follower state: follower="Node at a0db6add-839d-a858-9aab-7ee8fe5c7250 [Follower]" leader= -testLoggerAdapter verbose: server-2abc2d08-68d3-fc8c-4e13-16e16d2d4cc2: 2024-09-03T21:34:22.986Z [INFO] server-2abc2d08-68d3-fc8c-4e13-16e16d2d4cc2: entering follower state: follower="Node at 2abc2d08-68d3-fc8c-4e13-16e16d2d4cc2 [Follower]" leader= - testing.go:149: server-2abc2d08-68d3-fc8c-4e13-16e16d2d4cc2: 2024-09-03T21:34:22.986Z [INFO] server-2abc2d08-68d3-fc8c-4e13-16e16d2d4cc2: entering follower state: follower="Node at 2abc2d08-68d3-fc8c-4e13-16e16d2d4cc2 [Follower]" leader= -testLoggerAdapter verbose: server-da8162d5-a0b7-0586-160e-f4b36e106164: 2024-09-03T21:34:22.986Z [INFO] server-da8162d5-a0b7-0586-160e-f4b36e106164: entering follower state: follower="Node at da8162d5-a0b7-0586-160e-f4b36e106164 [Follower]" leader= - testing.go:149: server-da8162d5-a0b7-0586-160e-f4b36e106164: 2024-09-03T21:34:22.986Z [INFO] server-da8162d5-a0b7-0586-160e-f4b36e106164: entering follower state: follower="Node at da8162d5-a0b7-0586-160e-f4b36e106164 [Follower]" leader= -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:22.986Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-09-03T21:34:22.986Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-a0db6add-839d-a858-9aab-7ee8fe5c7250: 2024-09-03T21:34:23.052Z [WARN] server-a0db6add-839d-a858-9aab-7ee8fe5c7250: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-a0db6add-839d-a858-9aab-7ee8fe5c7250: 2024-09-03T21:34:23.052Z [WARN] server-a0db6add-839d-a858-9aab-7ee8fe5c7250: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-a0db6add-839d-a858-9aab-7ee8fe5c7250: 2024-09-03T21:34:23.052Z [INFO] server-a0db6add-839d-a858-9aab-7ee8fe5c7250: entering candidate state: node="Node at a0db6add-839d-a858-9aab-7ee8fe5c7250 [Candidate]" term=2 - testing.go:149: server-a0db6add-839d-a858-9aab-7ee8fe5c7250: 2024-09-03T21:34:23.052Z [INFO] server-a0db6add-839d-a858-9aab-7ee8fe5c7250: entering candidate state: node="Node at a0db6add-839d-a858-9aab-7ee8fe5c7250 [Candidate]" term=2 -testLoggerAdapter verbose: server-a0db6add-839d-a858-9aab-7ee8fe5c7250: 2024-09-03T21:34:23.053Z [INFO] server-a0db6add-839d-a858-9aab-7ee8fe5c7250: election won: tally=2 - testing.go:149: server-a0db6add-839d-a858-9aab-7ee8fe5c7250: 2024-09-03T21:34:23.053Z [INFO] server-a0db6add-839d-a858-9aab-7ee8fe5c7250: election won: tally=2 -testLoggerAdapter verbose: server-a0db6add-839d-a858-9aab-7ee8fe5c7250: 2024-09-03T21:34:23.053Z [INFO] server-a0db6add-839d-a858-9aab-7ee8fe5c7250: entering leader state: leader="Node at a0db6add-839d-a858-9aab-7ee8fe5c7250 [Leader]" - testing.go:149: server-a0db6add-839d-a858-9aab-7ee8fe5c7250: 2024-09-03T21:34:23.053Z [INFO] server-a0db6add-839d-a858-9aab-7ee8fe5c7250: entering leader state: leader="Node at a0db6add-839d-a858-9aab-7ee8fe5c7250 [Leader]" -testLoggerAdapter verbose: server-a0db6add-839d-a858-9aab-7ee8fe5c7250: 2024-09-03T21:34:23.053Z [INFO] server-a0db6add-839d-a858-9aab-7ee8fe5c7250: added peer, starting replication: peer=server-2abc2d08-68d3-fc8c-4e13-16e16d2d4cc2 - testing.go:149: server-a0db6add-839d-a858-9aab-7ee8fe5c7250: 2024-09-03T21:34:23.053Z [INFO] server-a0db6add-839d-a858-9aab-7ee8fe5c7250: added peer, starting replication: peer=server-2abc2d08-68d3-fc8c-4e13-16e16d2d4cc2 -testLoggerAdapter verbose: server-a0db6add-839d-a858-9aab-7ee8fe5c7250: 2024-09-03T21:34:23.053Z [INFO] server-a0db6add-839d-a858-9aab-7ee8fe5c7250: added peer, starting replication: peer=server-da8162d5-a0b7-0586-160e-f4b36e106164 - testing.go:149: server-a0db6add-839d-a858-9aab-7ee8fe5c7250: 2024-09-03T21:34:23.053Z [INFO] server-a0db6add-839d-a858-9aab-7ee8fe5c7250: added peer, starting replication: peer=server-da8162d5-a0b7-0586-160e-f4b36e106164 -testLoggerAdapter verbose: server-da8162d5-a0b7-0586-160e-f4b36e106164: 2024-09-03T21:34:23.053Z [WARN] server-da8162d5-a0b7-0586-160e-f4b36e106164: failed to get previous log: previous-index=1 last-index=0 error="log not found" - testing.go:149: server-da8162d5-a0b7-0586-160e-f4b36e106164: 2024-09-03T21:34:23.053Z [WARN] server-da8162d5-a0b7-0586-160e-f4b36e106164: failed to get previous log: previous-index=1 last-index=0 error="log not found" -testLoggerAdapter verbose: server-a0db6add-839d-a858-9aab-7ee8fe5c7250: 2024-09-03T21:34:23.053Z [WARN] server-a0db6add-839d-a858-9aab-7ee8fe5c7250: appendEntries rejected, sending older logs: peer="{Voter server-da8162d5-a0b7-0586-160e-f4b36e106164 da8162d5-a0b7-0586-160e-f4b36e106164}" next=1 - testing.go:149: server-a0db6add-839d-a858-9aab-7ee8fe5c7250: 2024-09-03T21:34:23.053Z [WARN] server-a0db6add-839d-a858-9aab-7ee8fe5c7250: appendEntries rejected, sending older logs: peer="{Voter server-da8162d5-a0b7-0586-160e-f4b36e106164 da8162d5-a0b7-0586-160e-f4b36e106164}" next=1 -testLoggerAdapter verbose: server-a0db6add-839d-a858-9aab-7ee8fe5c7250: 2024-09-03T21:34:23.053Z [INFO] server-a0db6add-839d-a858-9aab-7ee8fe5c7250: pipelining replication: peer="{Voter server-da8162d5-a0b7-0586-160e-f4b36e106164 da8162d5-a0b7-0586-160e-f4b36e106164}" - testing.go:149: server-a0db6add-839d-a858-9aab-7ee8fe5c7250: 2024-09-03T21:34:23.053Z [INFO] server-a0db6add-839d-a858-9aab-7ee8fe5c7250: pipelining replication: peer="{Voter server-da8162d5-a0b7-0586-160e-f4b36e106164 da8162d5-a0b7-0586-160e-f4b36e106164}" -testLoggerAdapter verbose: server-2abc2d08-68d3-fc8c-4e13-16e16d2d4cc2: 2024-09-03T21:34:23.053Z [WARN] server-2abc2d08-68d3-fc8c-4e13-16e16d2d4cc2: failed to get previous log: previous-index=1 last-index=0 error="log not found" - testing.go:149: server-2abc2d08-68d3-fc8c-4e13-16e16d2d4cc2: 2024-09-03T21:34:23.053Z [WARN] server-2abc2d08-68d3-fc8c-4e13-16e16d2d4cc2: failed to get previous log: previous-index=1 last-index=0 error="log not found" -testLoggerAdapter verbose: server-a0db6add-839d-a858-9aab-7ee8fe5c7250: 2024-09-03T21:34:23.053Z [WARN] server-a0db6add-839d-a858-9aab-7ee8fe5c7250: appendEntries rejected, sending older logs: peer="{Voter server-2abc2d08-68d3-fc8c-4e13-16e16d2d4cc2 2abc2d08-68d3-fc8c-4e13-16e16d2d4cc2}" next=1 - testing.go:149: server-a0db6add-839d-a858-9aab-7ee8fe5c7250: 2024-09-03T21:34:23.053Z [WARN] server-a0db6add-839d-a858-9aab-7ee8fe5c7250: appendEntries rejected, sending older logs: peer="{Voter server-2abc2d08-68d3-fc8c-4e13-16e16d2d4cc2 2abc2d08-68d3-fc8c-4e13-16e16d2d4cc2}" next=1 -testLoggerAdapter verbose: server-a0db6add-839d-a858-9aab-7ee8fe5c7250: 2024-09-03T21:34:23.053Z [INFO] server-a0db6add-839d-a858-9aab-7ee8fe5c7250: pipelining replication: peer="{Voter server-2abc2d08-68d3-fc8c-4e13-16e16d2d4cc2 2abc2d08-68d3-fc8c-4e13-16e16d2d4cc2}" - testing.go:149: server-a0db6add-839d-a858-9aab-7ee8fe5c7250: 2024-09-03T21:34:23.053Z [INFO] server-a0db6add-839d-a858-9aab-7ee8fe5c7250: pipelining replication: peer="{Voter server-2abc2d08-68d3-fc8c-4e13-16e16d2d4cc2 2abc2d08-68d3-fc8c-4e13-16e16d2d4cc2}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:23.158Z [INFO] stable state for Follower reached at 2024-09-03 21:34:23.053456297 +0000 UTC m=+4.100113434 (2 nodes), 67.0873ms from start of poll, 67.586239ms from cluster start. Timeout at 2024-09-03 21:34:23.158586124 +0000 UTC m=+4.205243255, 105.129821ms after stability - testing.go:149: cluster: 2024-09-03T21:34:23.158Z [INFO] stable state for Follower reached at 2024-09-03 21:34:23.053456297 +0000 UTC m=+4.100113434 (2 nodes), 67.0873ms from start of poll, 67.586239ms from cluster start. Timeout at 2024-09-03 21:34:23.158586124 +0000 UTC m=+4.205243255, 105.129821ms after stability -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:23.158Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:23.158Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:23.263Z [INFO] stable state for Leader reached at 2024-09-03 21:34:23.158770426 +0000 UTC m=+4.205427564 (1 nodes), 1.035µs from start of poll, 172.900369ms from cluster start. Timeout at 2024-09-03 21:34:23.263791612 +0000 UTC m=+4.310448743, 105.021179ms after stability - testing.go:149: cluster: 2024-09-03T21:34:23.263Z [INFO] stable state for Leader reached at 2024-09-03 21:34:23.158770426 +0000 UTC m=+4.205427564 (1 nodes), 1.035µs from start of poll, 172.900369ms from cluster start. Timeout at 2024-09-03 21:34:23.263791612 +0000 UTC m=+4.310448743, 105.021179ms after stability -testLoggerAdapter verbose: server-a0db6add-839d-a858-9aab-7ee8fe5c7250: 2024-09-03T21:34:23.274Z [INFO] server-a0db6add-839d-a858-9aab-7ee8fe5c7250: aborting pipeline replication: peer="{Voter server-da8162d5-a0b7-0586-160e-f4b36e106164 da8162d5-a0b7-0586-160e-f4b36e106164}" - testing.go:149: server-a0db6add-839d-a858-9aab-7ee8fe5c7250: 2024-09-03T21:34:23.274Z [INFO] server-a0db6add-839d-a858-9aab-7ee8fe5c7250: aborting pipeline replication: peer="{Voter server-da8162d5-a0b7-0586-160e-f4b36e106164 da8162d5-a0b7-0586-160e-f4b36e106164}" -testLoggerAdapter verbose: server-a0db6add-839d-a858-9aab-7ee8fe5c7250: 2024-09-03T21:34:23.274Z [INFO] server-a0db6add-839d-a858-9aab-7ee8fe5c7250: aborting pipeline replication: peer="{Voter server-2abc2d08-68d3-fc8c-4e13-16e16d2d4cc2 2abc2d08-68d3-fc8c-4e13-16e16d2d4cc2}" - testing.go:149: server-a0db6add-839d-a858-9aab-7ee8fe5c7250: 2024-09-03T21:34:23.274Z [INFO] server-a0db6add-839d-a858-9aab-7ee8fe5c7250: aborting pipeline replication: peer="{Voter server-2abc2d08-68d3-fc8c-4e13-16e16d2d4cc2 2abc2d08-68d3-fc8c-4e13-16e16d2d4cc2}" ---- PASS: TestRaft_LiveBootstrap (0.29s) +testLoggerAdapter verbose: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:51.950Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: initial configuration: index=0 servers=[] + testing.go:149: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:51.950Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: server-5717c896-6e56-63cc-1905-f457616df16e: 2023-08-02T15:12:51.950Z [INFO] server-5717c896-6e56-63cc-1905-f457616df16e: initial configuration: index=0 servers=[] + testing.go:149: server-5717c896-6e56-63cc-1905-f457616df16e: 2023-08-02T15:12:51.950Z [INFO] server-5717c896-6e56-63cc-1905-f457616df16e: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: server-66fa6e33-5c99-450c-62f9-714c39011593: 2023-08-02T15:12:51.951Z [INFO] server-66fa6e33-5c99-450c-62f9-714c39011593: initial configuration: index=0 servers=[] + testing.go:149: server-66fa6e33-5c99-450c-62f9-714c39011593: 2023-08-02T15:12:51.951Z [INFO] server-66fa6e33-5c99-450c-62f9-714c39011593: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:51.951Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: entering follower state: follower="Node at a53a533d-11bd-a760-3863-f25981d83c5b [Follower]" leader= + testing.go:149: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:51.951Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: entering follower state: follower="Node at a53a533d-11bd-a760-3863-f25981d83c5b [Follower]" leader= +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:51.951Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-08-02T15:12:51.951Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-5717c896-6e56-63cc-1905-f457616df16e: 2023-08-02T15:12:51.951Z [INFO] server-5717c896-6e56-63cc-1905-f457616df16e: entering follower state: follower="Node at 5717c896-6e56-63cc-1905-f457616df16e [Follower]" leader= + testing.go:149: server-5717c896-6e56-63cc-1905-f457616df16e: 2023-08-02T15:12:51.951Z [INFO] server-5717c896-6e56-63cc-1905-f457616df16e: entering follower state: follower="Node at 5717c896-6e56-63cc-1905-f457616df16e [Follower]" leader= +testLoggerAdapter verbose: server-66fa6e33-5c99-450c-62f9-714c39011593: 2023-08-02T15:12:51.951Z [INFO] server-66fa6e33-5c99-450c-62f9-714c39011593: entering follower state: follower="Node at 66fa6e33-5c99-450c-62f9-714c39011593 [Follower]" leader= + testing.go:149: server-66fa6e33-5c99-450c-62f9-714c39011593: 2023-08-02T15:12:51.951Z [INFO] server-66fa6e33-5c99-450c-62f9-714c39011593: entering follower state: follower="Node at 66fa6e33-5c99-450c-62f9-714c39011593 [Follower]" leader= +testLoggerAdapter verbose: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.024Z [WARN] server-a53a533d-11bd-a760-3863-f25981d83c5b: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.024Z [WARN] server-a53a533d-11bd-a760-3863-f25981d83c5b: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.024Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: entering candidate state: node="Node at a53a533d-11bd-a760-3863-f25981d83c5b [Candidate]" term=2 + testing.go:149: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.024Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: entering candidate state: node="Node at a53a533d-11bd-a760-3863-f25981d83c5b [Candidate]" term=2 +testLoggerAdapter verbose: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.025Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: election won: tally=2 + testing.go:149: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.025Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: election won: tally=2 +testLoggerAdapter verbose: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.025Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: entering leader state: leader="Node at a53a533d-11bd-a760-3863-f25981d83c5b [Leader]" + testing.go:149: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.025Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: entering leader state: leader="Node at a53a533d-11bd-a760-3863-f25981d83c5b [Leader]" +testLoggerAdapter verbose: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.025Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: added peer, starting replication: peer=server-5717c896-6e56-63cc-1905-f457616df16e + testing.go:149: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.025Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: added peer, starting replication: peer=server-5717c896-6e56-63cc-1905-f457616df16e +testLoggerAdapter verbose: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.025Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: added peer, starting replication: peer=server-66fa6e33-5c99-450c-62f9-714c39011593 + testing.go:149: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.025Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: added peer, starting replication: peer=server-66fa6e33-5c99-450c-62f9-714c39011593 +testLoggerAdapter verbose: server-66fa6e33-5c99-450c-62f9-714c39011593: 2023-08-02T15:12:52.025Z [WARN] server-66fa6e33-5c99-450c-62f9-714c39011593: failed to get previous log: previous-index=1 last-index=0 error="log not found" +testLoggerAdapter verbose: server-5717c896-6e56-63cc-1905-f457616df16e: 2023-08-02T15:12:52.025Z [WARN] server-5717c896-6e56-63cc-1905-f457616df16e: failed to get previous log: previous-index=1 last-index=0 error="log not found" + testing.go:149: server-66fa6e33-5c99-450c-62f9-714c39011593: 2023-08-02T15:12:52.025Z [WARN] server-66fa6e33-5c99-450c-62f9-714c39011593: failed to get previous log: previous-index=1 last-index=0 error="log not found" + testing.go:149: server-5717c896-6e56-63cc-1905-f457616df16e: 2023-08-02T15:12:52.025Z [WARN] server-5717c896-6e56-63cc-1905-f457616df16e: failed to get previous log: previous-index=1 last-index=0 error="log not found" +testLoggerAdapter verbose: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.025Z [WARN] server-a53a533d-11bd-a760-3863-f25981d83c5b: appendEntries rejected, sending older logs: peer="{Voter server-66fa6e33-5c99-450c-62f9-714c39011593 66fa6e33-5c99-450c-62f9-714c39011593}" next=1 + testing.go:149: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.025Z [WARN] server-a53a533d-11bd-a760-3863-f25981d83c5b: appendEntries rejected, sending older logs: peer="{Voter server-66fa6e33-5c99-450c-62f9-714c39011593 66fa6e33-5c99-450c-62f9-714c39011593}" next=1 +testLoggerAdapter verbose: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.025Z [WARN] server-a53a533d-11bd-a760-3863-f25981d83c5b: appendEntries rejected, sending older logs: peer="{Voter server-5717c896-6e56-63cc-1905-f457616df16e 5717c896-6e56-63cc-1905-f457616df16e}" next=1 + testing.go:149: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.025Z [WARN] server-a53a533d-11bd-a760-3863-f25981d83c5b: appendEntries rejected, sending older logs: peer="{Voter server-5717c896-6e56-63cc-1905-f457616df16e 5717c896-6e56-63cc-1905-f457616df16e}" next=1 +testLoggerAdapter verbose: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.025Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: pipelining replication: peer="{Voter server-66fa6e33-5c99-450c-62f9-714c39011593 66fa6e33-5c99-450c-62f9-714c39011593}" + testing.go:149: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.025Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: pipelining replication: peer="{Voter server-66fa6e33-5c99-450c-62f9-714c39011593 66fa6e33-5c99-450c-62f9-714c39011593}" +testLoggerAdapter verbose: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.025Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: pipelining replication: peer="{Voter server-5717c896-6e56-63cc-1905-f457616df16e 5717c896-6e56-63cc-1905-f457616df16e}" + testing.go:149: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.025Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: pipelining replication: peer="{Voter server-5717c896-6e56-63cc-1905-f457616df16e 5717c896-6e56-63cc-1905-f457616df16e}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:52.130Z [INFO] stable state for Follower reached at 2023-08-02 15:12:52.025283967 +0000 UTC m=+4.121220312 (2 nodes), 74.057405ms from start of poll, 74.475155ms from cluster start. Timeout at 2023-08-02 15:12:52.130663787 +0000 UTC m=+4.226600141, 105.379829ms after stability + testing.go:149: cluster: 2023-08-02T15:12:52.130Z [INFO] stable state for Follower reached at 2023-08-02 15:12:52.025283967 +0000 UTC m=+4.121220312 (2 nodes), 74.057405ms from start of poll, 74.475155ms from cluster start. Timeout at 2023-08-02 15:12:52.130663787 +0000 UTC m=+4.226600141, 105.379829ms after stability +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:52.130Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:12:52.130Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:52.236Z [INFO] stable state for Leader reached at 2023-08-02 15:12:52.130885617 +0000 UTC m=+4.226821942 (1 nodes), 4.619µs from start of poll, 180.076785ms from cluster start. Timeout at 2023-08-02 15:12:52.236173857 +0000 UTC m=+4.332110194, 105.288252ms after stability + testing.go:149: cluster: 2023-08-02T15:12:52.236Z [INFO] stable state for Leader reached at 2023-08-02 15:12:52.130885617 +0000 UTC m=+4.226821942 (1 nodes), 4.619µs from start of poll, 180.076785ms from cluster start. Timeout at 2023-08-02 15:12:52.236173857 +0000 UTC m=+4.332110194, 105.288252ms after stability +testLoggerAdapter verbose: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.246Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: aborting pipeline replication: peer="{Voter server-66fa6e33-5c99-450c-62f9-714c39011593 66fa6e33-5c99-450c-62f9-714c39011593}" + testing.go:149: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.246Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: aborting pipeline replication: peer="{Voter server-66fa6e33-5c99-450c-62f9-714c39011593 66fa6e33-5c99-450c-62f9-714c39011593}" +testLoggerAdapter verbose: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.246Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: aborting pipeline replication: peer="{Voter server-5717c896-6e56-63cc-1905-f457616df16e 5717c896-6e56-63cc-1905-f457616df16e}" + testing.go:149: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.246Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: aborting pipeline replication: peer="{Voter server-5717c896-6e56-63cc-1905-f457616df16e 5717c896-6e56-63cc-1905-f457616df16e}" +--- PASS: TestRaft_LiveBootstrap (0.30s) === RUN TestRaft_RecoverCluster_NoState -testLoggerAdapter verbose: server-d2a6df42-8d9e-82e5-075c-4278c4eecade: 2024-09-03T21:34:23.274Z [INFO] server-d2a6df42-8d9e-82e5-075c-4278c4eecade: initial configuration: index=0 servers=[] - testing.go:149: server-d2a6df42-8d9e-82e5-075c-4278c4eecade: 2024-09-03T21:34:23.274Z [INFO] server-d2a6df42-8d9e-82e5-075c-4278c4eecade: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: server-d6e77d3f-4bb6-c31b-3140-27b729478fcf: 2023-08-02T15:12:52.247Z [INFO] server-d6e77d3f-4bb6-c31b-3140-27b729478fcf: initial configuration: index=0 servers=[] + testing.go:149: server-d6e77d3f-4bb6-c31b-3140-27b729478fcf: 2023-08-02T15:12:52.247Z [INFO] server-d6e77d3f-4bb6-c31b-3140-27b729478fcf: initial configuration: index=0 servers=[] --- PASS: TestRaft_RecoverCluster_NoState (0.00s) === RUN TestRaft_RecoverCluster raft_test.go:119: Disabled unreliable test --- SKIP: TestRaft_RecoverCluster (0.00s) === RUN TestRaft_HasExistingState -testLoggerAdapter verbose: server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: 2024-09-03T21:34:23.275Z [INFO] server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f Address:89e2d8e2-8870-9e7a-a923-95dd58b83c9f} {Suffrage:Voter ID:server-f58fe523-3984-8e71-7bab-6a2161b7ec3a Address:f58fe523-3984-8e71-7bab-6a2161b7ec3a}]" - testing.go:149: server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: 2024-09-03T21:34:23.275Z [INFO] server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f Address:89e2d8e2-8870-9e7a-a923-95dd58b83c9f} {Suffrage:Voter ID:server-f58fe523-3984-8e71-7bab-6a2161b7ec3a Address:f58fe523-3984-8e71-7bab-6a2161b7ec3a}]" -testLoggerAdapter verbose: server-f58fe523-3984-8e71-7bab-6a2161b7ec3a: 2024-09-03T21:34:23.275Z [INFO] server-f58fe523-3984-8e71-7bab-6a2161b7ec3a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f Address:89e2d8e2-8870-9e7a-a923-95dd58b83c9f} {Suffrage:Voter ID:server-f58fe523-3984-8e71-7bab-6a2161b7ec3a Address:f58fe523-3984-8e71-7bab-6a2161b7ec3a}]" - testing.go:149: server-f58fe523-3984-8e71-7bab-6a2161b7ec3a: 2024-09-03T21:34:23.275Z [INFO] server-f58fe523-3984-8e71-7bab-6a2161b7ec3a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f Address:89e2d8e2-8870-9e7a-a923-95dd58b83c9f} {Suffrage:Voter ID:server-f58fe523-3984-8e71-7bab-6a2161b7ec3a Address:f58fe523-3984-8e71-7bab-6a2161b7ec3a}]" -testLoggerAdapter verbose: server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: 2024-09-03T21:34:23.275Z [INFO] server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: entering follower state: follower="Node at 89e2d8e2-8870-9e7a-a923-95dd58b83c9f [Follower]" leader= -testLoggerAdapter verbose: server-abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5: 2024-09-03T21:34:23.275Z [INFO] server-abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5: initial configuration: index=0 servers=[] - testing.go:149: server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: 2024-09-03T21:34:23.275Z [INFO] server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: entering follower state: follower="Node at 89e2d8e2-8870-9e7a-a923-95dd58b83c9f [Follower]" leader= -testLoggerAdapter verbose: server-f58fe523-3984-8e71-7bab-6a2161b7ec3a: 2024-09-03T21:34:23.275Z [INFO] server-f58fe523-3984-8e71-7bab-6a2161b7ec3a: entering follower state: follower="Node at f58fe523-3984-8e71-7bab-6a2161b7ec3a [Follower]" leader= - testing.go:149: server-abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5: 2024-09-03T21:34:23.275Z [INFO] server-abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: server-abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5: 2024-09-03T21:34:23.275Z [INFO] server-abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5: entering follower state: follower="Node at abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5 [Follower]" leader= - testing.go:149: server-abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5: 2024-09-03T21:34:23.275Z [INFO] server-abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5: entering follower state: follower="Node at abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5 [Follower]" leader= -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:23.275Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:23.275Z [INFO] starting stability test: raft-state=Leader - testing.go:149: server-f58fe523-3984-8e71-7bab-6a2161b7ec3a: 2024-09-03T21:34:23.275Z [INFO] server-f58fe523-3984-8e71-7bab-6a2161b7ec3a: entering follower state: follower="Node at f58fe523-3984-8e71-7bab-6a2161b7ec3a [Follower]" leader= -testLoggerAdapter verbose: server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: 2024-09-03T21:34:23.325Z [WARN] server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: 2024-09-03T21:34:23.325Z [WARN] server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: 2024-09-03T21:34:23.326Z [INFO] server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: entering candidate state: node="Node at 89e2d8e2-8870-9e7a-a923-95dd58b83c9f [Candidate]" term=2 - testing.go:149: server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: 2024-09-03T21:34:23.326Z [INFO] server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: entering candidate state: node="Node at 89e2d8e2-8870-9e7a-a923-95dd58b83c9f [Candidate]" term=2 -testLoggerAdapter verbose: server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: 2024-09-03T21:34:23.326Z [INFO] server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: election won: tally=2 - testing.go:149: server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: 2024-09-03T21:34:23.326Z [INFO] server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: election won: tally=2 -testLoggerAdapter verbose: server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: 2024-09-03T21:34:23.326Z [INFO] server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: entering leader state: leader="Node at 89e2d8e2-8870-9e7a-a923-95dd58b83c9f [Leader]" - testing.go:149: server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: 2024-09-03T21:34:23.326Z [INFO] server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: entering leader state: leader="Node at 89e2d8e2-8870-9e7a-a923-95dd58b83c9f [Leader]" -testLoggerAdapter verbose: server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: 2024-09-03T21:34:23.326Z [INFO] server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: added peer, starting replication: peer=server-f58fe523-3984-8e71-7bab-6a2161b7ec3a - testing.go:149: server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: 2024-09-03T21:34:23.326Z [INFO] server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: added peer, starting replication: peer=server-f58fe523-3984-8e71-7bab-6a2161b7ec3a -testLoggerAdapter verbose: server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: 2024-09-03T21:34:23.326Z [INFO] server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: pipelining replication: peer="{Voter server-f58fe523-3984-8e71-7bab-6a2161b7ec3a f58fe523-3984-8e71-7bab-6a2161b7ec3a}" - testing.go:149: server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: 2024-09-03T21:34:23.326Z [INFO] server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: pipelining replication: peer="{Voter server-f58fe523-3984-8e71-7bab-6a2161b7ec3a f58fe523-3984-8e71-7bab-6a2161b7ec3a}" -testLoggerAdapter verbose: server-abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5: 2024-09-03T21:34:23.329Z [WARN] server-abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5: no known peers, aborting election - testing.go:149: server-abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5: 2024-09-03T21:34:23.329Z [WARN] server-abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5: no known peers, aborting election -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:23.431Z [INFO] stable state for Leader reached at 2024-09-03 21:34:23.326381215 +0000 UTC m=+4.373038352 (1 nodes), 50.656922ms from start of poll, 51.142689ms from cluster start. Timeout at 2024-09-03 21:34:23.43147848 +0000 UTC m=+4.478135611, 105.097259ms after stability - testing.go:149: cluster: 2024-09-03T21:34:23.431Z [INFO] stable state for Leader reached at 2024-09-03 21:34:23.326381215 +0000 UTC m=+4.373038352 (1 nodes), 50.656922ms from start of poll, 51.142689ms from cluster start. Timeout at 2024-09-03 21:34:23.43147848 +0000 UTC m=+4.478135611, 105.097259ms after stability -testLoggerAdapter verbose: server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: 2024-09-03T21:34:23.431Z [INFO] server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: updating configuration: command=AddStaging server-id=server-abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5 server-addr=abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5 servers="[{Suffrage:Voter ID:server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f Address:89e2d8e2-8870-9e7a-a923-95dd58b83c9f} {Suffrage:Voter ID:server-f58fe523-3984-8e71-7bab-6a2161b7ec3a Address:f58fe523-3984-8e71-7bab-6a2161b7ec3a} {Suffrage:Voter ID:server-abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5 Address:abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5}]" - testing.go:149: server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: 2024-09-03T21:34:23.431Z [INFO] server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: updating configuration: command=AddStaging server-id=server-abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5 server-addr=abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5 servers="[{Suffrage:Voter ID:server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f Address:89e2d8e2-8870-9e7a-a923-95dd58b83c9f} {Suffrage:Voter ID:server-f58fe523-3984-8e71-7bab-6a2161b7ec3a Address:f58fe523-3984-8e71-7bab-6a2161b7ec3a} {Suffrage:Voter ID:server-abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5 Address:abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5}]" -testLoggerAdapter verbose: server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: 2024-09-03T21:34:23.431Z [INFO] server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: added peer, starting replication: peer=server-abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5 - testing.go:149: server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: 2024-09-03T21:34:23.431Z [INFO] server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: added peer, starting replication: peer=server-abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5 -testLoggerAdapter verbose: server-abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5: 2024-09-03T21:34:23.431Z [WARN] server-abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5: failed to get previous log: previous-index=3 last-index=0 error="log not found" - testing.go:149: server-abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5: 2024-09-03T21:34:23.431Z [WARN] server-abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5: failed to get previous log: previous-index=3 last-index=0 error="log not found" -testLoggerAdapter verbose: server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: 2024-09-03T21:34:23.431Z [WARN] server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: appendEntries rejected, sending older logs: peer="{Voter server-abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5 abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5}" next=1 - testing.go:149: server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: 2024-09-03T21:34:23.431Z [WARN] server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: appendEntries rejected, sending older logs: peer="{Voter server-abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5 abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5}" next=1 -testLoggerAdapter verbose: server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: 2024-09-03T21:34:23.431Z [INFO] server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: pipelining replication: peer="{Voter server-abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5 abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5}" - testing.go:149: server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: 2024-09-03T21:34:23.431Z [INFO] server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: pipelining replication: peer="{Voter server-abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5 abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:23.432Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:23.432Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:23.537Z [INFO] stable state for Leader reached at 2024-09-03 21:34:23.432052584 +0000 UTC m=+4.478709715 (1 nodes), 780ns from start of poll, 156.814052ms from cluster start. Timeout at 2024-09-03 21:34:23.537101051 +0000 UTC m=+4.583758189, 105.048474ms after stability - testing.go:149: cluster: 2024-09-03T21:34:23.537Z [INFO] stable state for Leader reached at 2024-09-03 21:34:23.432052584 +0000 UTC m=+4.478709715 (1 nodes), 780ns from start of poll, 156.814052ms from cluster start. Timeout at 2024-09-03 21:34:23.537101051 +0000 UTC m=+4.583758189, 105.048474ms after stability -testLoggerAdapter verbose: server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: 2024-09-03T21:34:23.537Z [INFO] server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: aborting pipeline replication: peer="{Voter server-abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5 abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5}" - testing.go:149: server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: 2024-09-03T21:34:23.537Z [INFO] server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: aborting pipeline replication: peer="{Voter server-abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5 abcd0dd4-4f13-d622-d5f0-d6d5c3ed44a5}" -testLoggerAdapter verbose: server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: 2024-09-03T21:34:23.537Z [INFO] server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: aborting pipeline replication: peer="{Voter server-f58fe523-3984-8e71-7bab-6a2161b7ec3a f58fe523-3984-8e71-7bab-6a2161b7ec3a}" - testing.go:149: server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: 2024-09-03T21:34:23.537Z [INFO] server-89e2d8e2-8870-9e7a-a923-95dd58b83c9f: aborting pipeline replication: peer="{Voter server-f58fe523-3984-8e71-7bab-6a2161b7ec3a f58fe523-3984-8e71-7bab-6a2161b7ec3a}" ---- PASS: TestRaft_HasExistingState (0.26s) +testLoggerAdapter verbose: server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9: 2023-08-02T15:12:52.248Z [INFO] server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9 Address:14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9} {Suffrage:Voter ID:server-2327d8ee-561b-c943-ccda-380026d867c5 Address:2327d8ee-561b-c943-ccda-380026d867c5}]" + testing.go:149: server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9: 2023-08-02T15:12:52.248Z [INFO] server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9 Address:14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9} {Suffrage:Voter ID:server-2327d8ee-561b-c943-ccda-380026d867c5 Address:2327d8ee-561b-c943-ccda-380026d867c5}]" +testLoggerAdapter verbose: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.248Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9 Address:14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9} {Suffrage:Voter ID:server-2327d8ee-561b-c943-ccda-380026d867c5 Address:2327d8ee-561b-c943-ccda-380026d867c5}]" + testing.go:149: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.248Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9 Address:14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9} {Suffrage:Voter ID:server-2327d8ee-561b-c943-ccda-380026d867c5 Address:2327d8ee-561b-c943-ccda-380026d867c5}]" +testLoggerAdapter verbose: server-5017f136-cbac-923b-fb86-7e0d01c79fcf: 2023-08-02T15:12:52.248Z [INFO] server-5017f136-cbac-923b-fb86-7e0d01c79fcf: initial configuration: index=0 servers=[] + testing.go:149: server-5017f136-cbac-923b-fb86-7e0d01c79fcf: 2023-08-02T15:12:52.248Z [INFO] server-5017f136-cbac-923b-fb86-7e0d01c79fcf: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:52.248Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:12:52.248Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9: 2023-08-02T15:12:52.248Z [INFO] server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9: entering follower state: follower="Node at 14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9 [Follower]" leader= + testing.go:149: server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9: 2023-08-02T15:12:52.248Z [INFO] server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9: entering follower state: follower="Node at 14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9 [Follower]" leader= +testLoggerAdapter verbose: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.248Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: entering follower state: follower="Node at 2327d8ee-561b-c943-ccda-380026d867c5 [Follower]" leader= + testing.go:149: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.248Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: entering follower state: follower="Node at 2327d8ee-561b-c943-ccda-380026d867c5 [Follower]" leader= +testLoggerAdapter verbose: server-5017f136-cbac-923b-fb86-7e0d01c79fcf: 2023-08-02T15:12:52.248Z [INFO] server-5017f136-cbac-923b-fb86-7e0d01c79fcf: entering follower state: follower="Node at 5017f136-cbac-923b-fb86-7e0d01c79fcf [Follower]" leader= + testing.go:149: server-5017f136-cbac-923b-fb86-7e0d01c79fcf: 2023-08-02T15:12:52.248Z [INFO] server-5017f136-cbac-923b-fb86-7e0d01c79fcf: entering follower state: follower="Node at 5017f136-cbac-923b-fb86-7e0d01c79fcf [Follower]" leader= +testLoggerAdapter verbose: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.318Z [WARN] server-2327d8ee-561b-c943-ccda-380026d867c5: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.318Z [WARN] server-2327d8ee-561b-c943-ccda-380026d867c5: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.318Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: entering candidate state: node="Node at 2327d8ee-561b-c943-ccda-380026d867c5 [Candidate]" term=2 + testing.go:149: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.318Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: entering candidate state: node="Node at 2327d8ee-561b-c943-ccda-380026d867c5 [Candidate]" term=2 +testLoggerAdapter verbose: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.318Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: election won: tally=2 + testing.go:149: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.318Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: election won: tally=2 +testLoggerAdapter verbose: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.318Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: entering leader state: leader="Node at 2327d8ee-561b-c943-ccda-380026d867c5 [Leader]" + testing.go:149: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.318Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: entering leader state: leader="Node at 2327d8ee-561b-c943-ccda-380026d867c5 [Leader]" +testLoggerAdapter verbose: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.318Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: added peer, starting replication: peer=server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9 + testing.go:149: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.318Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: added peer, starting replication: peer=server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9 +testLoggerAdapter verbose: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.319Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: pipelining replication: peer="{Voter server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9 14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9}" + testing.go:149: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.319Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: pipelining replication: peer="{Voter server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9 14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9}" +testLoggerAdapter verbose: server-5017f136-cbac-923b-fb86-7e0d01c79fcf: 2023-08-02T15:12:52.346Z [WARN] server-5017f136-cbac-923b-fb86-7e0d01c79fcf: no known peers, aborting election + testing.go:149: server-5017f136-cbac-923b-fb86-7e0d01c79fcf: 2023-08-02T15:12:52.346Z [WARN] server-5017f136-cbac-923b-fb86-7e0d01c79fcf: no known peers, aborting election +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:52.424Z [INFO] stable state for Leader reached at 2023-08-02 15:12:52.319127734 +0000 UTC m=+4.415064061 (1 nodes), 70.401473ms from start of poll, 70.93425ms from cluster start. Timeout at 2023-08-02 15:12:52.424341428 +0000 UTC m=+4.520277769, 105.213708ms after stability + testing.go:149: cluster: 2023-08-02T15:12:52.424Z [INFO] stable state for Leader reached at 2023-08-02 15:12:52.319127734 +0000 UTC m=+4.415064061 (1 nodes), 70.401473ms from start of poll, 70.93425ms from cluster start. Timeout at 2023-08-02 15:12:52.424341428 +0000 UTC m=+4.520277769, 105.213708ms after stability +testLoggerAdapter verbose: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.424Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: updating configuration: command=AddStaging server-id=server-5017f136-cbac-923b-fb86-7e0d01c79fcf server-addr=5017f136-cbac-923b-fb86-7e0d01c79fcf servers="[{Suffrage:Voter ID:server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9 Address:14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9} {Suffrage:Voter ID:server-2327d8ee-561b-c943-ccda-380026d867c5 Address:2327d8ee-561b-c943-ccda-380026d867c5} {Suffrage:Voter ID:server-5017f136-cbac-923b-fb86-7e0d01c79fcf Address:5017f136-cbac-923b-fb86-7e0d01c79fcf}]" + testing.go:149: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.424Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: updating configuration: command=AddStaging server-id=server-5017f136-cbac-923b-fb86-7e0d01c79fcf server-addr=5017f136-cbac-923b-fb86-7e0d01c79fcf servers="[{Suffrage:Voter ID:server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9 Address:14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9} {Suffrage:Voter ID:server-2327d8ee-561b-c943-ccda-380026d867c5 Address:2327d8ee-561b-c943-ccda-380026d867c5} {Suffrage:Voter ID:server-5017f136-cbac-923b-fb86-7e0d01c79fcf Address:5017f136-cbac-923b-fb86-7e0d01c79fcf}]" +testLoggerAdapter verbose: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.424Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: added peer, starting replication: peer=server-5017f136-cbac-923b-fb86-7e0d01c79fcf + testing.go:149: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.424Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: added peer, starting replication: peer=server-5017f136-cbac-923b-fb86-7e0d01c79fcf +testLoggerAdapter verbose: server-5017f136-cbac-923b-fb86-7e0d01c79fcf: 2023-08-02T15:12:52.424Z [WARN] server-5017f136-cbac-923b-fb86-7e0d01c79fcf: failed to get previous log: previous-index=3 last-index=0 error="log not found" + testing.go:149: server-5017f136-cbac-923b-fb86-7e0d01c79fcf: 2023-08-02T15:12:52.424Z [WARN] server-5017f136-cbac-923b-fb86-7e0d01c79fcf: failed to get previous log: previous-index=3 last-index=0 error="log not found" +testLoggerAdapter verbose: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.424Z [WARN] server-2327d8ee-561b-c943-ccda-380026d867c5: appendEntries rejected, sending older logs: peer="{Voter server-5017f136-cbac-923b-fb86-7e0d01c79fcf 5017f136-cbac-923b-fb86-7e0d01c79fcf}" next=1 + testing.go:149: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.424Z [WARN] server-2327d8ee-561b-c943-ccda-380026d867c5: appendEntries rejected, sending older logs: peer="{Voter server-5017f136-cbac-923b-fb86-7e0d01c79fcf 5017f136-cbac-923b-fb86-7e0d01c79fcf}" next=1 +testLoggerAdapter verbose: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.424Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: pipelining replication: peer="{Voter server-5017f136-cbac-923b-fb86-7e0d01c79fcf 5017f136-cbac-923b-fb86-7e0d01c79fcf}" + testing.go:149: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.424Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: pipelining replication: peer="{Voter server-5017f136-cbac-923b-fb86-7e0d01c79fcf 5017f136-cbac-923b-fb86-7e0d01c79fcf}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:52.425Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:12:52.425Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:52.530Z [INFO] stable state for Leader reached at 2023-08-02 15:12:52.425102363 +0000 UTC m=+4.521038688 (1 nodes), 1.015µs from start of poll, 176.908877ms from cluster start. Timeout at 2023-08-02 15:12:52.530191388 +0000 UTC m=+4.626127726, 105.089038ms after stability + testing.go:149: cluster: 2023-08-02T15:12:52.530Z [INFO] stable state for Leader reached at 2023-08-02 15:12:52.425102363 +0000 UTC m=+4.521038688 (1 nodes), 1.015µs from start of poll, 176.908877ms from cluster start. Timeout at 2023-08-02 15:12:52.530191388 +0000 UTC m=+4.626127726, 105.089038ms after stability +testLoggerAdapter verbose: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.530Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: aborting pipeline replication: peer="{Voter server-5017f136-cbac-923b-fb86-7e0d01c79fcf 5017f136-cbac-923b-fb86-7e0d01c79fcf}" + testing.go:149: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.530Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: aborting pipeline replication: peer="{Voter server-5017f136-cbac-923b-fb86-7e0d01c79fcf 5017f136-cbac-923b-fb86-7e0d01c79fcf}" +testLoggerAdapter verbose: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.530Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: aborting pipeline replication: peer="{Voter server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9 14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9}" + testing.go:149: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.530Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: aborting pipeline replication: peer="{Voter server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9 14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9}" +--- PASS: TestRaft_HasExistingState (0.28s) === RUN TestRaft_SingleNode -testLoggerAdapter verbose: server-29992533-24ce-c24e-5111-0ac56233ad47: 2024-09-03T21:34:23.538Z [INFO] server-29992533-24ce-c24e-5111-0ac56233ad47: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29992533-24ce-c24e-5111-0ac56233ad47 Address:29992533-24ce-c24e-5111-0ac56233ad47}]" - testing.go:149: server-29992533-24ce-c24e-5111-0ac56233ad47: 2024-09-03T21:34:23.538Z [INFO] server-29992533-24ce-c24e-5111-0ac56233ad47: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29992533-24ce-c24e-5111-0ac56233ad47 Address:29992533-24ce-c24e-5111-0ac56233ad47}]" -testLoggerAdapter verbose: server-29992533-24ce-c24e-5111-0ac56233ad47: 2024-09-03T21:34:23.538Z [INFO] server-29992533-24ce-c24e-5111-0ac56233ad47: entering follower state: follower="Node at 29992533-24ce-c24e-5111-0ac56233ad47 [Follower]" leader= - testing.go:149: server-29992533-24ce-c24e-5111-0ac56233ad47: 2024-09-03T21:34:23.538Z [INFO] server-29992533-24ce-c24e-5111-0ac56233ad47: entering follower state: follower="Node at 29992533-24ce-c24e-5111-0ac56233ad47 [Follower]" leader= -testLoggerAdapter verbose: server-29992533-24ce-c24e-5111-0ac56233ad47: 2024-09-03T21:34:23.590Z [WARN] server-29992533-24ce-c24e-5111-0ac56233ad47: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-29992533-24ce-c24e-5111-0ac56233ad47: 2024-09-03T21:34:23.590Z [WARN] server-29992533-24ce-c24e-5111-0ac56233ad47: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-29992533-24ce-c24e-5111-0ac56233ad47: 2024-09-03T21:34:23.591Z [INFO] server-29992533-24ce-c24e-5111-0ac56233ad47: entering candidate state: node="Node at 29992533-24ce-c24e-5111-0ac56233ad47 [Candidate]" term=2 - testing.go:149: server-29992533-24ce-c24e-5111-0ac56233ad47: 2024-09-03T21:34:23.591Z [INFO] server-29992533-24ce-c24e-5111-0ac56233ad47: entering candidate state: node="Node at 29992533-24ce-c24e-5111-0ac56233ad47 [Candidate]" term=2 -testLoggerAdapter verbose: server-29992533-24ce-c24e-5111-0ac56233ad47: 2024-09-03T21:34:23.591Z [INFO] server-29992533-24ce-c24e-5111-0ac56233ad47: election won: tally=1 - testing.go:149: server-29992533-24ce-c24e-5111-0ac56233ad47: 2024-09-03T21:34:23.591Z [INFO] server-29992533-24ce-c24e-5111-0ac56233ad47: election won: tally=1 -testLoggerAdapter verbose: server-29992533-24ce-c24e-5111-0ac56233ad47: 2024-09-03T21:34:23.591Z [INFO] server-29992533-24ce-c24e-5111-0ac56233ad47: entering leader state: leader="Node at 29992533-24ce-c24e-5111-0ac56233ad47 [Leader]" - testing.go:149: server-29992533-24ce-c24e-5111-0ac56233ad47: 2024-09-03T21:34:23.591Z [INFO] server-29992533-24ce-c24e-5111-0ac56233ad47: entering leader state: leader="Node at 29992533-24ce-c24e-5111-0ac56233ad47 [Leader]" ---- PASS: TestRaft_SingleNode (0.05s) +testLoggerAdapter verbose: server-e1e975b8-7824-3bc1-f543-1a4998307c2f: 2023-08-02T15:12:52.531Z [INFO] server-e1e975b8-7824-3bc1-f543-1a4998307c2f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e1e975b8-7824-3bc1-f543-1a4998307c2f Address:e1e975b8-7824-3bc1-f543-1a4998307c2f}]" + testing.go:149: server-e1e975b8-7824-3bc1-f543-1a4998307c2f: 2023-08-02T15:12:52.531Z [INFO] server-e1e975b8-7824-3bc1-f543-1a4998307c2f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e1e975b8-7824-3bc1-f543-1a4998307c2f Address:e1e975b8-7824-3bc1-f543-1a4998307c2f}]" +testLoggerAdapter verbose: server-e1e975b8-7824-3bc1-f543-1a4998307c2f: 2023-08-02T15:12:52.531Z [INFO] server-e1e975b8-7824-3bc1-f543-1a4998307c2f: entering follower state: follower="Node at e1e975b8-7824-3bc1-f543-1a4998307c2f [Follower]" leader= + testing.go:149: server-e1e975b8-7824-3bc1-f543-1a4998307c2f: 2023-08-02T15:12:52.531Z [INFO] server-e1e975b8-7824-3bc1-f543-1a4998307c2f: entering follower state: follower="Node at e1e975b8-7824-3bc1-f543-1a4998307c2f [Follower]" leader= +testLoggerAdapter verbose: server-e1e975b8-7824-3bc1-f543-1a4998307c2f: 2023-08-02T15:12:52.612Z [WARN] server-e1e975b8-7824-3bc1-f543-1a4998307c2f: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-e1e975b8-7824-3bc1-f543-1a4998307c2f: 2023-08-02T15:12:52.612Z [WARN] server-e1e975b8-7824-3bc1-f543-1a4998307c2f: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-e1e975b8-7824-3bc1-f543-1a4998307c2f: 2023-08-02T15:12:52.612Z [INFO] server-e1e975b8-7824-3bc1-f543-1a4998307c2f: entering candidate state: node="Node at e1e975b8-7824-3bc1-f543-1a4998307c2f [Candidate]" term=2 + testing.go:149: server-e1e975b8-7824-3bc1-f543-1a4998307c2f: 2023-08-02T15:12:52.612Z [INFO] server-e1e975b8-7824-3bc1-f543-1a4998307c2f: entering candidate state: node="Node at e1e975b8-7824-3bc1-f543-1a4998307c2f [Candidate]" term=2 +testLoggerAdapter verbose: server-e1e975b8-7824-3bc1-f543-1a4998307c2f: 2023-08-02T15:12:52.612Z [INFO] server-e1e975b8-7824-3bc1-f543-1a4998307c2f: election won: tally=1 + testing.go:149: server-e1e975b8-7824-3bc1-f543-1a4998307c2f: 2023-08-02T15:12:52.612Z [INFO] server-e1e975b8-7824-3bc1-f543-1a4998307c2f: election won: tally=1 +testLoggerAdapter verbose: server-e1e975b8-7824-3bc1-f543-1a4998307c2f: 2023-08-02T15:12:52.612Z [INFO] server-e1e975b8-7824-3bc1-f543-1a4998307c2f: entering leader state: leader="Node at e1e975b8-7824-3bc1-f543-1a4998307c2f [Leader]" + testing.go:149: server-e1e975b8-7824-3bc1-f543-1a4998307c2f: 2023-08-02T15:12:52.612Z [INFO] server-e1e975b8-7824-3bc1-f543-1a4998307c2f: entering leader state: leader="Node at e1e975b8-7824-3bc1-f543-1a4998307c2f [Leader]" +--- PASS: TestRaft_SingleNode (0.08s) === RUN TestRaft_TripleNode -testLoggerAdapter verbose: server-5ef67b22-7abd-9476-02f8-9c38c80f36ed: 2024-09-03T21:34:23.591Z [INFO] server-5ef67b22-7abd-9476-02f8-9c38c80f36ed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5ef67b22-7abd-9476-02f8-9c38c80f36ed Address:5ef67b22-7abd-9476-02f8-9c38c80f36ed} {Suffrage:Voter ID:server-3dd0c908-be4f-1354-7f9b-02c35156e270 Address:3dd0c908-be4f-1354-7f9b-02c35156e270} {Suffrage:Voter ID:server-51837acb-1cf5-7c1f-d908-a82204cc8c1f Address:51837acb-1cf5-7c1f-d908-a82204cc8c1f}]" - testing.go:149: server-5ef67b22-7abd-9476-02f8-9c38c80f36ed: 2024-09-03T21:34:23.591Z [INFO] server-5ef67b22-7abd-9476-02f8-9c38c80f36ed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5ef67b22-7abd-9476-02f8-9c38c80f36ed Address:5ef67b22-7abd-9476-02f8-9c38c80f36ed} {Suffrage:Voter ID:server-3dd0c908-be4f-1354-7f9b-02c35156e270 Address:3dd0c908-be4f-1354-7f9b-02c35156e270} {Suffrage:Voter ID:server-51837acb-1cf5-7c1f-d908-a82204cc8c1f Address:51837acb-1cf5-7c1f-d908-a82204cc8c1f}]" -testLoggerAdapter verbose: server-3dd0c908-be4f-1354-7f9b-02c35156e270: 2024-09-03T21:34:23.591Z [INFO] server-3dd0c908-be4f-1354-7f9b-02c35156e270: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5ef67b22-7abd-9476-02f8-9c38c80f36ed Address:5ef67b22-7abd-9476-02f8-9c38c80f36ed} {Suffrage:Voter ID:server-3dd0c908-be4f-1354-7f9b-02c35156e270 Address:3dd0c908-be4f-1354-7f9b-02c35156e270} {Suffrage:Voter ID:server-51837acb-1cf5-7c1f-d908-a82204cc8c1f Address:51837acb-1cf5-7c1f-d908-a82204cc8c1f}]" - testing.go:149: server-3dd0c908-be4f-1354-7f9b-02c35156e270: 2024-09-03T21:34:23.591Z [INFO] server-3dd0c908-be4f-1354-7f9b-02c35156e270: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5ef67b22-7abd-9476-02f8-9c38c80f36ed Address:5ef67b22-7abd-9476-02f8-9c38c80f36ed} {Suffrage:Voter ID:server-3dd0c908-be4f-1354-7f9b-02c35156e270 Address:3dd0c908-be4f-1354-7f9b-02c35156e270} {Suffrage:Voter ID:server-51837acb-1cf5-7c1f-d908-a82204cc8c1f Address:51837acb-1cf5-7c1f-d908-a82204cc8c1f}]" -testLoggerAdapter verbose: server-51837acb-1cf5-7c1f-d908-a82204cc8c1f: 2024-09-03T21:34:23.592Z [INFO] server-51837acb-1cf5-7c1f-d908-a82204cc8c1f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5ef67b22-7abd-9476-02f8-9c38c80f36ed Address:5ef67b22-7abd-9476-02f8-9c38c80f36ed} {Suffrage:Voter ID:server-3dd0c908-be4f-1354-7f9b-02c35156e270 Address:3dd0c908-be4f-1354-7f9b-02c35156e270} {Suffrage:Voter ID:server-51837acb-1cf5-7c1f-d908-a82204cc8c1f Address:51837acb-1cf5-7c1f-d908-a82204cc8c1f}]" - testing.go:149: server-51837acb-1cf5-7c1f-d908-a82204cc8c1f: 2024-09-03T21:34:23.592Z [INFO] server-51837acb-1cf5-7c1f-d908-a82204cc8c1f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5ef67b22-7abd-9476-02f8-9c38c80f36ed Address:5ef67b22-7abd-9476-02f8-9c38c80f36ed} {Suffrage:Voter ID:server-3dd0c908-be4f-1354-7f9b-02c35156e270 Address:3dd0c908-be4f-1354-7f9b-02c35156e270} {Suffrage:Voter ID:server-51837acb-1cf5-7c1f-d908-a82204cc8c1f Address:51837acb-1cf5-7c1f-d908-a82204cc8c1f}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:23.592Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-09-03T21:34:23.592Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-5ef67b22-7abd-9476-02f8-9c38c80f36ed: 2024-09-03T21:34:23.592Z [INFO] server-5ef67b22-7abd-9476-02f8-9c38c80f36ed: entering follower state: follower="Node at 5ef67b22-7abd-9476-02f8-9c38c80f36ed [Follower]" leader= - testing.go:149: server-5ef67b22-7abd-9476-02f8-9c38c80f36ed: 2024-09-03T21:34:23.592Z [INFO] server-5ef67b22-7abd-9476-02f8-9c38c80f36ed: entering follower state: follower="Node at 5ef67b22-7abd-9476-02f8-9c38c80f36ed [Follower]" leader= -testLoggerAdapter verbose: server-3dd0c908-be4f-1354-7f9b-02c35156e270: 2024-09-03T21:34:23.592Z [INFO] server-3dd0c908-be4f-1354-7f9b-02c35156e270: entering follower state: follower="Node at 3dd0c908-be4f-1354-7f9b-02c35156e270 [Follower]" leader= - testing.go:149: server-3dd0c908-be4f-1354-7f9b-02c35156e270: 2024-09-03T21:34:23.592Z [INFO] server-3dd0c908-be4f-1354-7f9b-02c35156e270: entering follower state: follower="Node at 3dd0c908-be4f-1354-7f9b-02c35156e270 [Follower]" leader= -testLoggerAdapter verbose: server-51837acb-1cf5-7c1f-d908-a82204cc8c1f: 2024-09-03T21:34:23.592Z [INFO] server-51837acb-1cf5-7c1f-d908-a82204cc8c1f: entering follower state: follower="Node at 51837acb-1cf5-7c1f-d908-a82204cc8c1f [Follower]" leader= - testing.go:149: server-51837acb-1cf5-7c1f-d908-a82204cc8c1f: 2024-09-03T21:34:23.592Z [INFO] server-51837acb-1cf5-7c1f-d908-a82204cc8c1f: entering follower state: follower="Node at 51837acb-1cf5-7c1f-d908-a82204cc8c1f [Follower]" leader= -testLoggerAdapter verbose: server-3dd0c908-be4f-1354-7f9b-02c35156e270: 2024-09-03T21:34:23.657Z [WARN] server-3dd0c908-be4f-1354-7f9b-02c35156e270: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-3dd0c908-be4f-1354-7f9b-02c35156e270: 2024-09-03T21:34:23.657Z [WARN] server-3dd0c908-be4f-1354-7f9b-02c35156e270: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-3dd0c908-be4f-1354-7f9b-02c35156e270: 2024-09-03T21:34:23.658Z [INFO] server-3dd0c908-be4f-1354-7f9b-02c35156e270: entering candidate state: node="Node at 3dd0c908-be4f-1354-7f9b-02c35156e270 [Candidate]" term=2 - testing.go:149: server-3dd0c908-be4f-1354-7f9b-02c35156e270: 2024-09-03T21:34:23.658Z [INFO] server-3dd0c908-be4f-1354-7f9b-02c35156e270: entering candidate state: node="Node at 3dd0c908-be4f-1354-7f9b-02c35156e270 [Candidate]" term=2 -testLoggerAdapter verbose: server-3dd0c908-be4f-1354-7f9b-02c35156e270: 2024-09-03T21:34:23.658Z [INFO] server-3dd0c908-be4f-1354-7f9b-02c35156e270: election won: tally=2 - testing.go:149: server-3dd0c908-be4f-1354-7f9b-02c35156e270: 2024-09-03T21:34:23.658Z [INFO] server-3dd0c908-be4f-1354-7f9b-02c35156e270: election won: tally=2 -testLoggerAdapter verbose: server-3dd0c908-be4f-1354-7f9b-02c35156e270: 2024-09-03T21:34:23.658Z [INFO] server-3dd0c908-be4f-1354-7f9b-02c35156e270: entering leader state: leader="Node at 3dd0c908-be4f-1354-7f9b-02c35156e270 [Leader]" - testing.go:149: server-3dd0c908-be4f-1354-7f9b-02c35156e270: 2024-09-03T21:34:23.658Z [INFO] server-3dd0c908-be4f-1354-7f9b-02c35156e270: entering leader state: leader="Node at 3dd0c908-be4f-1354-7f9b-02c35156e270 [Leader]" -testLoggerAdapter verbose: server-3dd0c908-be4f-1354-7f9b-02c35156e270: 2024-09-03T21:34:23.658Z [INFO] server-3dd0c908-be4f-1354-7f9b-02c35156e270: added peer, starting replication: peer=server-5ef67b22-7abd-9476-02f8-9c38c80f36ed - testing.go:149: server-3dd0c908-be4f-1354-7f9b-02c35156e270: 2024-09-03T21:34:23.658Z [INFO] server-3dd0c908-be4f-1354-7f9b-02c35156e270: added peer, starting replication: peer=server-5ef67b22-7abd-9476-02f8-9c38c80f36ed -testLoggerAdapter verbose: server-3dd0c908-be4f-1354-7f9b-02c35156e270: 2024-09-03T21:34:23.658Z [INFO] server-3dd0c908-be4f-1354-7f9b-02c35156e270: added peer, starting replication: peer=server-51837acb-1cf5-7c1f-d908-a82204cc8c1f - testing.go:149: server-3dd0c908-be4f-1354-7f9b-02c35156e270: 2024-09-03T21:34:23.658Z [INFO] server-3dd0c908-be4f-1354-7f9b-02c35156e270: added peer, starting replication: peer=server-51837acb-1cf5-7c1f-d908-a82204cc8c1f -testLoggerAdapter verbose: server-3dd0c908-be4f-1354-7f9b-02c35156e270: 2024-09-03T21:34:23.658Z [INFO] server-3dd0c908-be4f-1354-7f9b-02c35156e270: pipelining replication: peer="{Voter server-51837acb-1cf5-7c1f-d908-a82204cc8c1f 51837acb-1cf5-7c1f-d908-a82204cc8c1f}" - testing.go:149: server-3dd0c908-be4f-1354-7f9b-02c35156e270: 2024-09-03T21:34:23.658Z [INFO] server-3dd0c908-be4f-1354-7f9b-02c35156e270: pipelining replication: peer="{Voter server-51837acb-1cf5-7c1f-d908-a82204cc8c1f 51837acb-1cf5-7c1f-d908-a82204cc8c1f}" -testLoggerAdapter verbose: server-3dd0c908-be4f-1354-7f9b-02c35156e270: 2024-09-03T21:34:23.658Z [INFO] server-3dd0c908-be4f-1354-7f9b-02c35156e270: pipelining replication: peer="{Voter server-5ef67b22-7abd-9476-02f8-9c38c80f36ed 5ef67b22-7abd-9476-02f8-9c38c80f36ed}" - testing.go:149: server-3dd0c908-be4f-1354-7f9b-02c35156e270: 2024-09-03T21:34:23.658Z [INFO] server-3dd0c908-be4f-1354-7f9b-02c35156e270: pipelining replication: peer="{Voter server-5ef67b22-7abd-9476-02f8-9c38c80f36ed 5ef67b22-7abd-9476-02f8-9c38c80f36ed}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:23.764Z [INFO] stable state for Follower reached at 2024-09-03 21:34:23.658436998 +0000 UTC m=+4.705094128 (2 nodes), 66.341497ms from start of poll, 66.731522ms from cluster start. Timeout at 2024-09-03 21:34:23.764031955 +0000 UTC m=+4.810689116, 105.594988ms after stability - testing.go:149: cluster: 2024-09-03T21:34:23.764Z [INFO] stable state for Follower reached at 2024-09-03 21:34:23.658436998 +0000 UTC m=+4.705094128 (2 nodes), 66.341497ms from start of poll, 66.731522ms from cluster start. Timeout at 2024-09-03 21:34:23.764031955 +0000 UTC m=+4.810689116, 105.594988ms after stability -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:23.764Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:23.764Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:23.869Z [INFO] stable state for Leader reached at 2024-09-03 21:34:23.764200967 +0000 UTC m=+4.810858113 (1 nodes), 1.261µs from start of poll, 172.495507ms from cluster start. Timeout at 2024-09-03 21:34:23.869278008 +0000 UTC m=+4.915935138, 105.077025ms after stability - testing.go:149: cluster: 2024-09-03T21:34:23.869Z [INFO] stable state for Leader reached at 2024-09-03 21:34:23.764200967 +0000 UTC m=+4.810858113 (1 nodes), 1.261µs from start of poll, 172.495507ms from cluster start. Timeout at 2024-09-03 21:34:23.869278008 +0000 UTC m=+4.915935138, 105.077025ms after stability -testLoggerAdapter verbose: server-3dd0c908-be4f-1354-7f9b-02c35156e270: 2024-09-03T21:34:23.880Z [INFO] server-3dd0c908-be4f-1354-7f9b-02c35156e270: aborting pipeline replication: peer="{Voter server-51837acb-1cf5-7c1f-d908-a82204cc8c1f 51837acb-1cf5-7c1f-d908-a82204cc8c1f}" - testing.go:149: server-3dd0c908-be4f-1354-7f9b-02c35156e270: 2024-09-03T21:34:23.880Z [INFO] server-3dd0c908-be4f-1354-7f9b-02c35156e270: aborting pipeline replication: peer="{Voter server-51837acb-1cf5-7c1f-d908-a82204cc8c1f 51837acb-1cf5-7c1f-d908-a82204cc8c1f}" -testLoggerAdapter verbose: server-3dd0c908-be4f-1354-7f9b-02c35156e270: 2024-09-03T21:34:23.880Z [INFO] server-3dd0c908-be4f-1354-7f9b-02c35156e270: aborting pipeline replication: peer="{Voter server-5ef67b22-7abd-9476-02f8-9c38c80f36ed 5ef67b22-7abd-9476-02f8-9c38c80f36ed}" - testing.go:149: server-3dd0c908-be4f-1354-7f9b-02c35156e270: 2024-09-03T21:34:23.880Z [INFO] server-3dd0c908-be4f-1354-7f9b-02c35156e270: aborting pipeline replication: peer="{Voter server-5ef67b22-7abd-9476-02f8-9c38c80f36ed 5ef67b22-7abd-9476-02f8-9c38c80f36ed}" -testLoggerAdapter verbose: server-3dd0c908-be4f-1354-7f9b-02c35156e270: 2024-09-03T21:34:23.930Z [ERROR] server-3dd0c908-be4f-1354-7f9b-02c35156e270: failed to heartbeat to: peer=5ef67b22-7abd-9476-02f8-9c38c80f36ed error="command timed out" - testing.go:149: server-3dd0c908-be4f-1354-7f9b-02c35156e270: 2024-09-03T21:34:23.930Z [ERROR] server-3dd0c908-be4f-1354-7f9b-02c35156e270: failed to heartbeat to: peer=5ef67b22-7abd-9476-02f8-9c38c80f36ed error="command timed out" ---- PASS: TestRaft_TripleNode (0.34s) +testLoggerAdapter verbose: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.613Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62 Address:6c3170fc-6ae7-6efe-0d04-2f64bc585a62} {Suffrage:Voter ID:server-2106ef7a-f83b-291a-1cf3-215ef1d30f61 Address:2106ef7a-f83b-291a-1cf3-215ef1d30f61} {Suffrage:Voter ID:server-2d685416-8549-aec5-cd16-ee788af57344 Address:2d685416-8549-aec5-cd16-ee788af57344}]" + testing.go:149: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.613Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62 Address:6c3170fc-6ae7-6efe-0d04-2f64bc585a62} {Suffrage:Voter ID:server-2106ef7a-f83b-291a-1cf3-215ef1d30f61 Address:2106ef7a-f83b-291a-1cf3-215ef1d30f61} {Suffrage:Voter ID:server-2d685416-8549-aec5-cd16-ee788af57344 Address:2d685416-8549-aec5-cd16-ee788af57344}]" +testLoggerAdapter verbose: server-2106ef7a-f83b-291a-1cf3-215ef1d30f61: 2023-08-02T15:12:52.613Z [INFO] server-2106ef7a-f83b-291a-1cf3-215ef1d30f61: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62 Address:6c3170fc-6ae7-6efe-0d04-2f64bc585a62} {Suffrage:Voter ID:server-2106ef7a-f83b-291a-1cf3-215ef1d30f61 Address:2106ef7a-f83b-291a-1cf3-215ef1d30f61} {Suffrage:Voter ID:server-2d685416-8549-aec5-cd16-ee788af57344 Address:2d685416-8549-aec5-cd16-ee788af57344}]" +testLoggerAdapter verbose: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.613Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: entering follower state: follower="Node at 6c3170fc-6ae7-6efe-0d04-2f64bc585a62 [Follower]" leader= + testing.go:149: server-2106ef7a-f83b-291a-1cf3-215ef1d30f61: 2023-08-02T15:12:52.613Z [INFO] server-2106ef7a-f83b-291a-1cf3-215ef1d30f61: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62 Address:6c3170fc-6ae7-6efe-0d04-2f64bc585a62} {Suffrage:Voter ID:server-2106ef7a-f83b-291a-1cf3-215ef1d30f61 Address:2106ef7a-f83b-291a-1cf3-215ef1d30f61} {Suffrage:Voter ID:server-2d685416-8549-aec5-cd16-ee788af57344 Address:2d685416-8549-aec5-cd16-ee788af57344}]" + testing.go:149: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.613Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: entering follower state: follower="Node at 6c3170fc-6ae7-6efe-0d04-2f64bc585a62 [Follower]" leader= +testLoggerAdapter verbose: server-2106ef7a-f83b-291a-1cf3-215ef1d30f61: 2023-08-02T15:12:52.613Z [INFO] server-2106ef7a-f83b-291a-1cf3-215ef1d30f61: entering follower state: follower="Node at 2106ef7a-f83b-291a-1cf3-215ef1d30f61 [Follower]" leader= + testing.go:149: server-2106ef7a-f83b-291a-1cf3-215ef1d30f61: 2023-08-02T15:12:52.613Z [INFO] server-2106ef7a-f83b-291a-1cf3-215ef1d30f61: entering follower state: follower="Node at 2106ef7a-f83b-291a-1cf3-215ef1d30f61 [Follower]" leader= +testLoggerAdapter verbose: server-2d685416-8549-aec5-cd16-ee788af57344: 2023-08-02T15:12:52.613Z [INFO] server-2d685416-8549-aec5-cd16-ee788af57344: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62 Address:6c3170fc-6ae7-6efe-0d04-2f64bc585a62} {Suffrage:Voter ID:server-2106ef7a-f83b-291a-1cf3-215ef1d30f61 Address:2106ef7a-f83b-291a-1cf3-215ef1d30f61} {Suffrage:Voter ID:server-2d685416-8549-aec5-cd16-ee788af57344 Address:2d685416-8549-aec5-cd16-ee788af57344}]" + testing.go:149: server-2d685416-8549-aec5-cd16-ee788af57344: 2023-08-02T15:12:52.613Z [INFO] server-2d685416-8549-aec5-cd16-ee788af57344: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62 Address:6c3170fc-6ae7-6efe-0d04-2f64bc585a62} {Suffrage:Voter ID:server-2106ef7a-f83b-291a-1cf3-215ef1d30f61 Address:2106ef7a-f83b-291a-1cf3-215ef1d30f61} {Suffrage:Voter ID:server-2d685416-8549-aec5-cd16-ee788af57344 Address:2d685416-8549-aec5-cd16-ee788af57344}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:52.613Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-08-02T15:12:52.613Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-2d685416-8549-aec5-cd16-ee788af57344: 2023-08-02T15:12:52.613Z [INFO] server-2d685416-8549-aec5-cd16-ee788af57344: entering follower state: follower="Node at 2d685416-8549-aec5-cd16-ee788af57344 [Follower]" leader= + testing.go:149: server-2d685416-8549-aec5-cd16-ee788af57344: 2023-08-02T15:12:52.613Z [INFO] server-2d685416-8549-aec5-cd16-ee788af57344: entering follower state: follower="Node at 2d685416-8549-aec5-cd16-ee788af57344 [Follower]" leader= +testLoggerAdapter verbose: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.675Z [WARN] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.675Z [WARN] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.676Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: entering candidate state: node="Node at 6c3170fc-6ae7-6efe-0d04-2f64bc585a62 [Candidate]" term=2 + testing.go:149: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.676Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: entering candidate state: node="Node at 6c3170fc-6ae7-6efe-0d04-2f64bc585a62 [Candidate]" term=2 +testLoggerAdapter verbose: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.676Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: election won: tally=2 + testing.go:149: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.676Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: election won: tally=2 +testLoggerAdapter verbose: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.676Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: entering leader state: leader="Node at 6c3170fc-6ae7-6efe-0d04-2f64bc585a62 [Leader]" + testing.go:149: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.676Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: entering leader state: leader="Node at 6c3170fc-6ae7-6efe-0d04-2f64bc585a62 [Leader]" +testLoggerAdapter verbose: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.676Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: added peer, starting replication: peer=server-2106ef7a-f83b-291a-1cf3-215ef1d30f61 + testing.go:149: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.676Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: added peer, starting replication: peer=server-2106ef7a-f83b-291a-1cf3-215ef1d30f61 +testLoggerAdapter verbose: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.676Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: added peer, starting replication: peer=server-2d685416-8549-aec5-cd16-ee788af57344 + testing.go:149: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.676Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: added peer, starting replication: peer=server-2d685416-8549-aec5-cd16-ee788af57344 +testLoggerAdapter verbose: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.676Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: pipelining replication: peer="{Voter server-2d685416-8549-aec5-cd16-ee788af57344 2d685416-8549-aec5-cd16-ee788af57344}" + testing.go:149: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.676Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: pipelining replication: peer="{Voter server-2d685416-8549-aec5-cd16-ee788af57344 2d685416-8549-aec5-cd16-ee788af57344}" +testLoggerAdapter verbose: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.676Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: pipelining replication: peer="{Voter server-2106ef7a-f83b-291a-1cf3-215ef1d30f61 2106ef7a-f83b-291a-1cf3-215ef1d30f61}" + testing.go:149: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.676Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: pipelining replication: peer="{Voter server-2106ef7a-f83b-291a-1cf3-215ef1d30f61 2106ef7a-f83b-291a-1cf3-215ef1d30f61}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:52.781Z [INFO] stable state for Follower reached at 2023-08-02 15:12:52.676499237 +0000 UTC m=+4.772435572 (2 nodes), 62.613624ms from start of poll, 63.151441ms from cluster start. Timeout at 2023-08-02 15:12:52.781804772 +0000 UTC m=+4.877741108, 105.305536ms after stability + testing.go:149: cluster: 2023-08-02T15:12:52.781Z [INFO] stable state for Follower reached at 2023-08-02 15:12:52.676499237 +0000 UTC m=+4.772435572 (2 nodes), 62.613624ms from start of poll, 63.151441ms from cluster start. Timeout at 2023-08-02 15:12:52.781804772 +0000 UTC m=+4.877741108, 105.305536ms after stability +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:52.781Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:12:52.781Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:52.887Z [INFO] stable state for Leader reached at 2023-08-02 15:12:52.781997923 +0000 UTC m=+4.877934250 (1 nodes), 1.687µs from start of poll, 168.650119ms from cluster start. Timeout at 2023-08-02 15:12:52.887111162 +0000 UTC m=+4.983047500, 105.11325ms after stability + testing.go:149: cluster: 2023-08-02T15:12:52.887Z [INFO] stable state for Leader reached at 2023-08-02 15:12:52.781997923 +0000 UTC m=+4.877934250 (1 nodes), 1.687µs from start of poll, 168.650119ms from cluster start. Timeout at 2023-08-02 15:12:52.887111162 +0000 UTC m=+4.983047500, 105.11325ms after stability +testLoggerAdapter verbose: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.897Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: aborting pipeline replication: peer="{Voter server-2d685416-8549-aec5-cd16-ee788af57344 2d685416-8549-aec5-cd16-ee788af57344}" + testing.go:149: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.897Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: aborting pipeline replication: peer="{Voter server-2d685416-8549-aec5-cd16-ee788af57344 2d685416-8549-aec5-cd16-ee788af57344}" +testLoggerAdapter verbose: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.897Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: aborting pipeline replication: peer="{Voter server-2106ef7a-f83b-291a-1cf3-215ef1d30f61 2106ef7a-f83b-291a-1cf3-215ef1d30f61}" + testing.go:149: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.897Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: aborting pipeline replication: peer="{Voter server-2106ef7a-f83b-291a-1cf3-215ef1d30f61 2106ef7a-f83b-291a-1cf3-215ef1d30f61}" +--- PASS: TestRaft_TripleNode (0.29s) === RUN TestRaft_LeaderFail -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:23.931Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5a8c3d35-3841-5006-44e3-26a6b04c7381 Address:5a8c3d35-3841-5006-44e3-26a6b04c7381} {Suffrage:Voter ID:server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500 Address:1ebcdbe2-8138-6c90-62c3-5df3dd9af500} {Suffrage:Voter ID:server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c Address:3ce7168e-57db-a8db-f9f5-f1bc45652c3c}]" - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:23.931Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5a8c3d35-3841-5006-44e3-26a6b04c7381 Address:5a8c3d35-3841-5006-44e3-26a6b04c7381} {Suffrage:Voter ID:server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500 Address:1ebcdbe2-8138-6c90-62c3-5df3dd9af500} {Suffrage:Voter ID:server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c Address:3ce7168e-57db-a8db-f9f5-f1bc45652c3c}]" -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:23.931Z [INFO] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5a8c3d35-3841-5006-44e3-26a6b04c7381 Address:5a8c3d35-3841-5006-44e3-26a6b04c7381} {Suffrage:Voter ID:server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500 Address:1ebcdbe2-8138-6c90-62c3-5df3dd9af500} {Suffrage:Voter ID:server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c Address:3ce7168e-57db-a8db-f9f5-f1bc45652c3c}]" - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:23.931Z [INFO] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5a8c3d35-3841-5006-44e3-26a6b04c7381 Address:5a8c3d35-3841-5006-44e3-26a6b04c7381} {Suffrage:Voter ID:server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500 Address:1ebcdbe2-8138-6c90-62c3-5df3dd9af500} {Suffrage:Voter ID:server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c Address:3ce7168e-57db-a8db-f9f5-f1bc45652c3c}]" -testLoggerAdapter verbose: server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c: 2024-09-03T21:34:23.932Z [INFO] server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5a8c3d35-3841-5006-44e3-26a6b04c7381 Address:5a8c3d35-3841-5006-44e3-26a6b04c7381} {Suffrage:Voter ID:server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500 Address:1ebcdbe2-8138-6c90-62c3-5df3dd9af500} {Suffrage:Voter ID:server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c Address:3ce7168e-57db-a8db-f9f5-f1bc45652c3c}]" - testing.go:149: server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c: 2024-09-03T21:34:23.932Z [INFO] server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5a8c3d35-3841-5006-44e3-26a6b04c7381 Address:5a8c3d35-3841-5006-44e3-26a6b04c7381} {Suffrage:Voter ID:server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500 Address:1ebcdbe2-8138-6c90-62c3-5df3dd9af500} {Suffrage:Voter ID:server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c Address:3ce7168e-57db-a8db-f9f5-f1bc45652c3c}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:23.932Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-09-03T21:34:23.932Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:23.932Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: entering follower state: follower="Node at 5a8c3d35-3841-5006-44e3-26a6b04c7381 [Follower]" leader= - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:23.932Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: entering follower state: follower="Node at 5a8c3d35-3841-5006-44e3-26a6b04c7381 [Follower]" leader= -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:23.932Z [INFO] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: entering follower state: follower="Node at 1ebcdbe2-8138-6c90-62c3-5df3dd9af500 [Follower]" leader= - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:23.932Z [INFO] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: entering follower state: follower="Node at 1ebcdbe2-8138-6c90-62c3-5df3dd9af500 [Follower]" leader= -testLoggerAdapter verbose: server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c: 2024-09-03T21:34:23.932Z [INFO] server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c: entering follower state: follower="Node at 3ce7168e-57db-a8db-f9f5-f1bc45652c3c [Follower]" leader= - testing.go:149: server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c: 2024-09-03T21:34:23.932Z [INFO] server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c: entering follower state: follower="Node at 3ce7168e-57db-a8db-f9f5-f1bc45652c3c [Follower]" leader= -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:23.989Z [WARN] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:23.989Z [WARN] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:23.989Z [INFO] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: entering candidate state: node="Node at 1ebcdbe2-8138-6c90-62c3-5df3dd9af500 [Candidate]" term=2 - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:23.989Z [INFO] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: entering candidate state: node="Node at 1ebcdbe2-8138-6c90-62c3-5df3dd9af500 [Candidate]" term=2 -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:23.989Z [INFO] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: election won: tally=2 - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:23.989Z [INFO] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: election won: tally=2 -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:23.989Z [INFO] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: entering leader state: leader="Node at 1ebcdbe2-8138-6c90-62c3-5df3dd9af500 [Leader]" - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:23.989Z [INFO] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: entering leader state: leader="Node at 1ebcdbe2-8138-6c90-62c3-5df3dd9af500 [Leader]" -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:23.989Z [INFO] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: added peer, starting replication: peer=server-5a8c3d35-3841-5006-44e3-26a6b04c7381 - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:23.989Z [INFO] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: added peer, starting replication: peer=server-5a8c3d35-3841-5006-44e3-26a6b04c7381 -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:23.989Z [INFO] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: added peer, starting replication: peer=server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:23.989Z [INFO] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: added peer, starting replication: peer=server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:23.989Z [INFO] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: pipelining replication: peer="{Voter server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c 3ce7168e-57db-a8db-f9f5-f1bc45652c3c}" - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:23.989Z [INFO] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: pipelining replication: peer="{Voter server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c 3ce7168e-57db-a8db-f9f5-f1bc45652c3c}" -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:23.990Z [INFO] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: pipelining replication: peer="{Voter server-5a8c3d35-3841-5006-44e3-26a6b04c7381 5a8c3d35-3841-5006-44e3-26a6b04c7381}" - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:23.990Z [INFO] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: pipelining replication: peer="{Voter server-5a8c3d35-3841-5006-44e3-26a6b04c7381 5a8c3d35-3841-5006-44e3-26a6b04c7381}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:24.095Z [INFO] stable state for Follower reached at 2024-09-03 21:34:23.990185384 +0000 UTC m=+5.036842533 (2 nodes), 58.11897ms from start of poll, 58.635185ms from cluster start. Timeout at 2024-09-03 21:34:24.095221236 +0000 UTC m=+5.141878367, 105.035834ms after stability - testing.go:149: cluster: 2024-09-03T21:34:24.095Z [INFO] stable state for Follower reached at 2024-09-03 21:34:23.990185384 +0000 UTC m=+5.036842533 (2 nodes), 58.11897ms from start of poll, 58.635185ms from cluster start. Timeout at 2024-09-03 21:34:24.095221236 +0000 UTC m=+5.141878367, 105.035834ms after stability -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:24.095Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:24.095Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:24.200Z [INFO] stable state for Leader reached at 2024-09-03 21:34:24.095385945 +0000 UTC m=+5.142043076 (1 nodes), 1.119µs from start of poll, 163.835728ms from cluster start. Timeout at 2024-09-03 21:34:24.200858944 +0000 UTC m=+5.247516085, 105.473009ms after stability - testing.go:149: cluster: 2024-09-03T21:34:24.200Z [INFO] stable state for Leader reached at 2024-09-03 21:34:24.095385945 +0000 UTC m=+5.142043076 (1 nodes), 1.119µs from start of poll, 163.835728ms from cluster start. Timeout at 2024-09-03 21:34:24.200858944 +0000 UTC m=+5.247516085, 105.473009ms after stability - raft_test.go:336: [INFO] Disconnecting Node at 1ebcdbe2-8138-6c90-62c3-5df3dd9af500 [Leader] -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.214Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to heartbeat to: peer=3ce7168e-57db-a8db-f9f5-f1bc45652c3c error="failed to connect to peer: 3ce7168e-57db-a8db-f9f5-f1bc45652c3c" - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.214Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to heartbeat to: peer=3ce7168e-57db-a8db-f9f5-f1bc45652c3c error="failed to connect to peer: 3ce7168e-57db-a8db-f9f5-f1bc45652c3c" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:24.217Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:24.217Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.218Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to pipeline appendEntries: peer="{Voter server-5a8c3d35-3841-5006-44e3-26a6b04c7381 5a8c3d35-3841-5006-44e3-26a6b04c7381}" error="append pipeline closed" - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.218Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to pipeline appendEntries: peer="{Voter server-5a8c3d35-3841-5006-44e3-26a6b04c7381 5a8c3d35-3841-5006-44e3-26a6b04c7381}" error="append pipeline closed" -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.218Z [INFO] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: aborting pipeline replication: peer="{Voter server-5a8c3d35-3841-5006-44e3-26a6b04c7381 5a8c3d35-3841-5006-44e3-26a6b04c7381}" - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.218Z [INFO] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: aborting pipeline replication: peer="{Voter server-5a8c3d35-3841-5006-44e3-26a6b04c7381 5a8c3d35-3841-5006-44e3-26a6b04c7381}" -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.219Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to heartbeat to: peer=5a8c3d35-3841-5006-44e3-26a6b04c7381 error="failed to connect to peer: 5a8c3d35-3841-5006-44e3-26a6b04c7381" - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.219Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to heartbeat to: peer=5a8c3d35-3841-5006-44e3-26a6b04c7381 error="failed to connect to peer: 5a8c3d35-3841-5006-44e3-26a6b04c7381" -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.220Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to pipeline appendEntries: peer="{Voter server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c 3ce7168e-57db-a8db-f9f5-f1bc45652c3c}" error="append pipeline closed" - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.220Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to pipeline appendEntries: peer="{Voter server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c 3ce7168e-57db-a8db-f9f5-f1bc45652c3c}" error="append pipeline closed" -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.220Z [INFO] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: aborting pipeline replication: peer="{Voter server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c 3ce7168e-57db-a8db-f9f5-f1bc45652c3c}" - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.220Z [INFO] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: aborting pipeline replication: peer="{Voter server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c 3ce7168e-57db-a8db-f9f5-f1bc45652c3c}" -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.225Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to appendEntries to: peer="{Voter server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c 3ce7168e-57db-a8db-f9f5-f1bc45652c3c}" error="failed to connect to peer: 3ce7168e-57db-a8db-f9f5-f1bc45652c3c" - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.225Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to appendEntries to: peer="{Voter server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c 3ce7168e-57db-a8db-f9f5-f1bc45652c3c}" error="failed to connect to peer: 3ce7168e-57db-a8db-f9f5-f1bc45652c3c" -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.227Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to appendEntries to: peer="{Voter server-5a8c3d35-3841-5006-44e3-26a6b04c7381 5a8c3d35-3841-5006-44e3-26a6b04c7381}" error="failed to connect to peer: 5a8c3d35-3841-5006-44e3-26a6b04c7381" - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.227Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to appendEntries to: peer="{Voter server-5a8c3d35-3841-5006-44e3-26a6b04c7381 5a8c3d35-3841-5006-44e3-26a6b04c7381}" error="failed to connect to peer: 5a8c3d35-3841-5006-44e3-26a6b04c7381" -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.235Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to heartbeat to: peer=3ce7168e-57db-a8db-f9f5-f1bc45652c3c error="failed to connect to peer: 3ce7168e-57db-a8db-f9f5-f1bc45652c3c" - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.235Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to heartbeat to: peer=3ce7168e-57db-a8db-f9f5-f1bc45652c3c error="failed to connect to peer: 3ce7168e-57db-a8db-f9f5-f1bc45652c3c" -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.237Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to heartbeat to: peer=5a8c3d35-3841-5006-44e3-26a6b04c7381 error="failed to connect to peer: 5a8c3d35-3841-5006-44e3-26a6b04c7381" - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.237Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to heartbeat to: peer=5a8c3d35-3841-5006-44e3-26a6b04c7381 error="failed to connect to peer: 5a8c3d35-3841-5006-44e3-26a6b04c7381" -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.243Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to appendEntries to: peer="{Voter server-5a8c3d35-3841-5006-44e3-26a6b04c7381 5a8c3d35-3841-5006-44e3-26a6b04c7381}" error="failed to connect to peer: 5a8c3d35-3841-5006-44e3-26a6b04c7381" - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.243Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to appendEntries to: peer="{Voter server-5a8c3d35-3841-5006-44e3-26a6b04c7381 5a8c3d35-3841-5006-44e3-26a6b04c7381}" error="failed to connect to peer: 5a8c3d35-3841-5006-44e3-26a6b04c7381" -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.244Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to appendEntries to: peer="{Voter server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c 3ce7168e-57db-a8db-f9f5-f1bc45652c3c}" error="failed to connect to peer: 3ce7168e-57db-a8db-f9f5-f1bc45652c3c" - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.244Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to appendEntries to: peer="{Voter server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c 3ce7168e-57db-a8db-f9f5-f1bc45652c3c}" error="failed to connect to peer: 3ce7168e-57db-a8db-f9f5-f1bc45652c3c" -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.254Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to heartbeat to: peer=5a8c3d35-3841-5006-44e3-26a6b04c7381 error="failed to connect to peer: 5a8c3d35-3841-5006-44e3-26a6b04c7381" - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.254Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to heartbeat to: peer=5a8c3d35-3841-5006-44e3-26a6b04c7381 error="failed to connect to peer: 5a8c3d35-3841-5006-44e3-26a6b04c7381" -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.255Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to heartbeat to: peer=3ce7168e-57db-a8db-f9f5-f1bc45652c3c error="failed to connect to peer: 3ce7168e-57db-a8db-f9f5-f1bc45652c3c" - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.255Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to heartbeat to: peer=3ce7168e-57db-a8db-f9f5-f1bc45652c3c error="failed to connect to peer: 3ce7168e-57db-a8db-f9f5-f1bc45652c3c" -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.259Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to appendEntries to: peer="{Voter server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c 3ce7168e-57db-a8db-f9f5-f1bc45652c3c}" error="failed to connect to peer: 3ce7168e-57db-a8db-f9f5-f1bc45652c3c" - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.259Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to appendEntries to: peer="{Voter server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c 3ce7168e-57db-a8db-f9f5-f1bc45652c3c}" error="failed to connect to peer: 3ce7168e-57db-a8db-f9f5-f1bc45652c3c" -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.260Z [WARN] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to contact: server-id=server-5a8c3d35-3841-5006-44e3-26a6b04c7381 time=50.047523ms - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.260Z [WARN] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to contact: server-id=server-5a8c3d35-3841-5006-44e3-26a6b04c7381 time=50.047523ms -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.262Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to appendEntries to: peer="{Voter server-5a8c3d35-3841-5006-44e3-26a6b04c7381 5a8c3d35-3841-5006-44e3-26a6b04c7381}" error="failed to connect to peer: 5a8c3d35-3841-5006-44e3-26a6b04c7381" - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.262Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to appendEntries to: peer="{Voter server-5a8c3d35-3841-5006-44e3-26a6b04c7381 5a8c3d35-3841-5006-44e3-26a6b04c7381}" error="failed to connect to peer: 5a8c3d35-3841-5006-44e3-26a6b04c7381" -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.270Z [WARN] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to contact: server-id=server-5a8c3d35-3841-5006-44e3-26a6b04c7381 time=60.274234ms - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.270Z [WARN] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to contact: server-id=server-5a8c3d35-3841-5006-44e3-26a6b04c7381 time=60.274234ms -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.270Z [WARN] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to contact: server-id=server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c time=59.520312ms - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.270Z [WARN] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to contact: server-id=server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c time=59.520312ms -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.270Z [WARN] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to contact quorum of nodes, stepping down - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.270Z [WARN] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to contact quorum of nodes, stepping down -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.270Z [INFO] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: entering follower state: follower="Node at 1ebcdbe2-8138-6c90-62c3-5df3dd9af500 [Follower]" leader= - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.270Z [INFO] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: entering follower state: follower="Node at 1ebcdbe2-8138-6c90-62c3-5df3dd9af500 [Follower]" leader= -testLoggerAdapter verbose: server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c: 2024-09-03T21:34:24.271Z [WARN] server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c: heartbeat timeout reached, starting election: last-leader=1ebcdbe2-8138-6c90-62c3-5df3dd9af500 - testing.go:149: server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c: 2024-09-03T21:34:24.271Z [WARN] server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c: heartbeat timeout reached, starting election: last-leader=1ebcdbe2-8138-6c90-62c3-5df3dd9af500 -testLoggerAdapter verbose: server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c: 2024-09-03T21:34:24.271Z [INFO] server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c: entering candidate state: node="Node at 3ce7168e-57db-a8db-f9f5-f1bc45652c3c [Candidate]" term=3 - testing.go:149: server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c: 2024-09-03T21:34:24.271Z [INFO] server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c: entering candidate state: node="Node at 3ce7168e-57db-a8db-f9f5-f1bc45652c3c [Candidate]" term=3 -testLoggerAdapter verbose: server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c: 2024-09-03T21:34:24.271Z [ERROR] server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c: failed to make requestVote RPC: target="{Voter server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500 1ebcdbe2-8138-6c90-62c3-5df3dd9af500}" error="failed to connect to peer: 1ebcdbe2-8138-6c90-62c3-5df3dd9af500" - testing.go:149: server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c: 2024-09-03T21:34:24.271Z [ERROR] server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c: failed to make requestVote RPC: target="{Voter server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500 1ebcdbe2-8138-6c90-62c3-5df3dd9af500}" error="failed to connect to peer: 1ebcdbe2-8138-6c90-62c3-5df3dd9af500" -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.271Z [WARN] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: rejecting vote request since we have a leader: from=3ce7168e-57db-a8db-f9f5-f1bc45652c3c leader=1ebcdbe2-8138-6c90-62c3-5df3dd9af500 - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.271Z [WARN] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: rejecting vote request since we have a leader: from=3ce7168e-57db-a8db-f9f5-f1bc45652c3c leader=1ebcdbe2-8138-6c90-62c3-5df3dd9af500 -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.275Z [WARN] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: heartbeat timeout reached, starting election: last-leader=1ebcdbe2-8138-6c90-62c3-5df3dd9af500 - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.275Z [WARN] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: heartbeat timeout reached, starting election: last-leader=1ebcdbe2-8138-6c90-62c3-5df3dd9af500 -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.275Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: entering candidate state: node="Node at 5a8c3d35-3841-5006-44e3-26a6b04c7381 [Candidate]" term=3 - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.275Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: entering candidate state: node="Node at 5a8c3d35-3841-5006-44e3-26a6b04c7381 [Candidate]" term=3 -testLoggerAdapter verbose: server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c: 2024-09-03T21:34:24.275Z [INFO] server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c: duplicate requestVote for same term: term=3 - testing.go:149: server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c: 2024-09-03T21:34:24.275Z [INFO] server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c: duplicate requestVote for same term: term=3 -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.275Z [ERROR] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: failed to make requestVote RPC: target="{Voter server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500 1ebcdbe2-8138-6c90-62c3-5df3dd9af500}" error="failed to connect to peer: 1ebcdbe2-8138-6c90-62c3-5df3dd9af500" - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.275Z [ERROR] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: failed to make requestVote RPC: target="{Voter server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500 1ebcdbe2-8138-6c90-62c3-5df3dd9af500}" error="failed to connect to peer: 1ebcdbe2-8138-6c90-62c3-5df3dd9af500" -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.282Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to heartbeat to: peer=3ce7168e-57db-a8db-f9f5-f1bc45652c3c error="failed to connect to peer: 3ce7168e-57db-a8db-f9f5-f1bc45652c3c" - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.282Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to heartbeat to: peer=3ce7168e-57db-a8db-f9f5-f1bc45652c3c error="failed to connect to peer: 3ce7168e-57db-a8db-f9f5-f1bc45652c3c" -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.283Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to heartbeat to: peer=5a8c3d35-3841-5006-44e3-26a6b04c7381 error="failed to connect to peer: 5a8c3d35-3841-5006-44e3-26a6b04c7381" - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.283Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to heartbeat to: peer=5a8c3d35-3841-5006-44e3-26a6b04c7381 error="failed to connect to peer: 5a8c3d35-3841-5006-44e3-26a6b04c7381" -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.287Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to appendEntries to: peer="{Voter server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c 3ce7168e-57db-a8db-f9f5-f1bc45652c3c}" error="failed to connect to peer: 3ce7168e-57db-a8db-f9f5-f1bc45652c3c" - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.287Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to appendEntries to: peer="{Voter server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c 3ce7168e-57db-a8db-f9f5-f1bc45652c3c}" error="failed to connect to peer: 3ce7168e-57db-a8db-f9f5-f1bc45652c3c" -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.291Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to appendEntries to: peer="{Voter server-5a8c3d35-3841-5006-44e3-26a6b04c7381 5a8c3d35-3841-5006-44e3-26a6b04c7381}" error="failed to connect to peer: 5a8c3d35-3841-5006-44e3-26a6b04c7381" - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.291Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to appendEntries to: peer="{Voter server-5a8c3d35-3841-5006-44e3-26a6b04c7381 5a8c3d35-3841-5006-44e3-26a6b04c7381}" error="failed to connect to peer: 5a8c3d35-3841-5006-44e3-26a6b04c7381" -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.325Z [WARN] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.325Z [WARN] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.326Z [INFO] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: entering candidate state: node="Node at 1ebcdbe2-8138-6c90-62c3-5df3dd9af500 [Candidate]" term=3 - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.326Z [INFO] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: entering candidate state: node="Node at 1ebcdbe2-8138-6c90-62c3-5df3dd9af500 [Candidate]" term=3 -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.326Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to make requestVote RPC: target="{Voter server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c 3ce7168e-57db-a8db-f9f5-f1bc45652c3c}" error="failed to connect to peer: 3ce7168e-57db-a8db-f9f5-f1bc45652c3c" - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.326Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to make requestVote RPC: target="{Voter server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c 3ce7168e-57db-a8db-f9f5-f1bc45652c3c}" error="failed to connect to peer: 3ce7168e-57db-a8db-f9f5-f1bc45652c3c" -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.326Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to make requestVote RPC: target="{Voter server-5a8c3d35-3841-5006-44e3-26a6b04c7381 5a8c3d35-3841-5006-44e3-26a6b04c7381}" error="failed to connect to peer: 5a8c3d35-3841-5006-44e3-26a6b04c7381" - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.326Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to make requestVote RPC: target="{Voter server-5a8c3d35-3841-5006-44e3-26a6b04c7381 5a8c3d35-3841-5006-44e3-26a6b04c7381}" error="failed to connect to peer: 5a8c3d35-3841-5006-44e3-26a6b04c7381" -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.352Z [WARN] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: Election timeout reached, restarting election - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.352Z [WARN] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: Election timeout reached, restarting election -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.352Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: entering candidate state: node="Node at 5a8c3d35-3841-5006-44e3-26a6b04c7381 [Candidate]" term=4 - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.352Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: entering candidate state: node="Node at 5a8c3d35-3841-5006-44e3-26a6b04c7381 [Candidate]" term=4 -testLoggerAdapter verbose: server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c: 2024-09-03T21:34:24.352Z [INFO] server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c: entering follower state: follower="Node at 3ce7168e-57db-a8db-f9f5-f1bc45652c3c [Follower]" leader= - testing.go:149: server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c: 2024-09-03T21:34:24.352Z [INFO] server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c: entering follower state: follower="Node at 3ce7168e-57db-a8db-f9f5-f1bc45652c3c [Follower]" leader= -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.352Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: election won: tally=2 - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.352Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: election won: tally=2 -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.352Z [ERROR] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: failed to make requestVote RPC: target="{Voter server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500 1ebcdbe2-8138-6c90-62c3-5df3dd9af500}" error="failed to connect to peer: 1ebcdbe2-8138-6c90-62c3-5df3dd9af500" - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.352Z [ERROR] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: failed to make requestVote RPC: target="{Voter server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500 1ebcdbe2-8138-6c90-62c3-5df3dd9af500}" error="failed to connect to peer: 1ebcdbe2-8138-6c90-62c3-5df3dd9af500" -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.352Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: entering leader state: leader="Node at 5a8c3d35-3841-5006-44e3-26a6b04c7381 [Leader]" - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.352Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: entering leader state: leader="Node at 5a8c3d35-3841-5006-44e3-26a6b04c7381 [Leader]" -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.352Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: added peer, starting replication: peer=server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500 - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.352Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: added peer, starting replication: peer=server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500 -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.352Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: added peer, starting replication: peer=server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.352Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: added peer, starting replication: peer=server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.352Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: pipelining replication: peer="{Voter server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c 3ce7168e-57db-a8db-f9f5-f1bc45652c3c}" - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.352Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: pipelining replication: peer="{Voter server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c 3ce7168e-57db-a8db-f9f5-f1bc45652c3c}" -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.352Z [ERROR] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: failed to appendEntries to: peer="{Voter server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500 1ebcdbe2-8138-6c90-62c3-5df3dd9af500}" error="failed to connect to peer: 1ebcdbe2-8138-6c90-62c3-5df3dd9af500" - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.352Z [ERROR] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: failed to appendEntries to: peer="{Voter server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500 1ebcdbe2-8138-6c90-62c3-5df3dd9af500}" error="failed to connect to peer: 1ebcdbe2-8138-6c90-62c3-5df3dd9af500" -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.358Z [ERROR] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: failed to heartbeat to: peer=1ebcdbe2-8138-6c90-62c3-5df3dd9af500 error="failed to connect to peer: 1ebcdbe2-8138-6c90-62c3-5df3dd9af500" - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.358Z [ERROR] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: failed to heartbeat to: peer=1ebcdbe2-8138-6c90-62c3-5df3dd9af500 error="failed to connect to peer: 1ebcdbe2-8138-6c90-62c3-5df3dd9af500" -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.372Z [ERROR] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: failed to appendEntries to: peer="{Voter server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500 1ebcdbe2-8138-6c90-62c3-5df3dd9af500}" error="failed to connect to peer: 1ebcdbe2-8138-6c90-62c3-5df3dd9af500" - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.372Z [ERROR] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: failed to appendEntries to: peer="{Voter server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500 1ebcdbe2-8138-6c90-62c3-5df3dd9af500}" error="failed to connect to peer: 1ebcdbe2-8138-6c90-62c3-5df3dd9af500" -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.374Z [ERROR] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: failed to heartbeat to: peer=1ebcdbe2-8138-6c90-62c3-5df3dd9af500 error="failed to connect to peer: 1ebcdbe2-8138-6c90-62c3-5df3dd9af500" - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.374Z [ERROR] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: failed to heartbeat to: peer=1ebcdbe2-8138-6c90-62c3-5df3dd9af500 error="failed to connect to peer: 1ebcdbe2-8138-6c90-62c3-5df3dd9af500" -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.389Z [ERROR] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: failed to appendEntries to: peer="{Voter server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500 1ebcdbe2-8138-6c90-62c3-5df3dd9af500}" error="failed to connect to peer: 1ebcdbe2-8138-6c90-62c3-5df3dd9af500" - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.389Z [ERROR] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: failed to appendEntries to: peer="{Voter server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500 1ebcdbe2-8138-6c90-62c3-5df3dd9af500}" error="failed to connect to peer: 1ebcdbe2-8138-6c90-62c3-5df3dd9af500" -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.393Z [ERROR] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: failed to heartbeat to: peer=1ebcdbe2-8138-6c90-62c3-5df3dd9af500 error="failed to connect to peer: 1ebcdbe2-8138-6c90-62c3-5df3dd9af500" - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.393Z [ERROR] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: failed to heartbeat to: peer=1ebcdbe2-8138-6c90-62c3-5df3dd9af500 error="failed to connect to peer: 1ebcdbe2-8138-6c90-62c3-5df3dd9af500" -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.402Z [WARN] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: failed to contact: server-id=server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500 time=50.367081ms - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.402Z [WARN] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: failed to contact: server-id=server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500 time=50.367081ms -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.407Z [WARN] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: Election timeout reached, restarting election - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.407Z [WARN] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: Election timeout reached, restarting election -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.407Z [INFO] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: entering candidate state: node="Node at 1ebcdbe2-8138-6c90-62c3-5df3dd9af500 [Candidate]" term=4 - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.407Z [INFO] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: entering candidate state: node="Node at 1ebcdbe2-8138-6c90-62c3-5df3dd9af500 [Candidate]" term=4 -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.408Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to make requestVote RPC: target="{Voter server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c 3ce7168e-57db-a8db-f9f5-f1bc45652c3c}" error="failed to connect to peer: 3ce7168e-57db-a8db-f9f5-f1bc45652c3c" - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.408Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to make requestVote RPC: target="{Voter server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c 3ce7168e-57db-a8db-f9f5-f1bc45652c3c}" error="failed to connect to peer: 3ce7168e-57db-a8db-f9f5-f1bc45652c3c" -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.408Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to make requestVote RPC: target="{Voter server-5a8c3d35-3841-5006-44e3-26a6b04c7381 5a8c3d35-3841-5006-44e3-26a6b04c7381}" error="failed to connect to peer: 5a8c3d35-3841-5006-44e3-26a6b04c7381" - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.408Z [ERROR] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to make requestVote RPC: target="{Voter server-5a8c3d35-3841-5006-44e3-26a6b04c7381 5a8c3d35-3841-5006-44e3-26a6b04c7381}" error="failed to connect to peer: 5a8c3d35-3841-5006-44e3-26a6b04c7381" -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.415Z [ERROR] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: failed to appendEntries to: peer="{Voter server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500 1ebcdbe2-8138-6c90-62c3-5df3dd9af500}" error="failed to connect to peer: 1ebcdbe2-8138-6c90-62c3-5df3dd9af500" - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.415Z [ERROR] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: failed to appendEntries to: peer="{Voter server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500 1ebcdbe2-8138-6c90-62c3-5df3dd9af500}" error="failed to connect to peer: 1ebcdbe2-8138-6c90-62c3-5df3dd9af500" -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.423Z [ERROR] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: failed to heartbeat to: peer=1ebcdbe2-8138-6c90-62c3-5df3dd9af500 error="failed to connect to peer: 1ebcdbe2-8138-6c90-62c3-5df3dd9af500" - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.423Z [ERROR] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: failed to heartbeat to: peer=1ebcdbe2-8138-6c90-62c3-5df3dd9af500 error="failed to connect to peer: 1ebcdbe2-8138-6c90-62c3-5df3dd9af500" -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.450Z [WARN] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: failed to contact: server-id=server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500 time=98.298434ms - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.450Z [WARN] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: failed to contact: server-id=server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500 time=98.298434ms -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:24.457Z [INFO] stable state for Leader reached at 2024-09-03 21:34:24.352297714 +0000 UTC m=+5.398954856 (1 nodes), 134.847221ms from start of poll, 420.747508ms from cluster start. Timeout at 2024-09-03 21:34:24.457398535 +0000 UTC m=+5.504055677, 105.100821ms after stability - testing.go:149: cluster: 2024-09-03T21:34:24.457Z [INFO] stable state for Leader reached at 2024-09-03 21:34:24.352297714 +0000 UTC m=+5.398954856 (1 nodes), 134.847221ms from start of poll, 420.747508ms from cluster start. Timeout at 2024-09-03 21:34:24.457398535 +0000 UTC m=+5.504055677, 105.100821ms after stability - raft_test.go:371: [INFO] Reconnecting Node at 1ebcdbe2-8138-6c90-62c3-5df3dd9af500 [Candidate] -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.463Z [WARN] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: Election timeout reached, restarting election - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.463Z [WARN] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: Election timeout reached, restarting election -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.463Z [INFO] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: entering candidate state: node="Node at 1ebcdbe2-8138-6c90-62c3-5df3dd9af500 [Candidate]" term=5 - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.463Z [INFO] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: entering candidate state: node="Node at 1ebcdbe2-8138-6c90-62c3-5df3dd9af500 [Candidate]" term=5 -testLoggerAdapter verbose: server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c: 2024-09-03T21:34:24.463Z [WARN] server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c: rejecting vote request since we have a leader: from=1ebcdbe2-8138-6c90-62c3-5df3dd9af500 leader=5a8c3d35-3841-5006-44e3-26a6b04c7381 - testing.go:149: server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c: 2024-09-03T21:34:24.463Z [WARN] server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c: rejecting vote request since we have a leader: from=1ebcdbe2-8138-6c90-62c3-5df3dd9af500 leader=5a8c3d35-3841-5006-44e3-26a6b04c7381 -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.463Z [WARN] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: rejecting vote request since we have a leader: from=1ebcdbe2-8138-6c90-62c3-5df3dd9af500 leader=5a8c3d35-3841-5006-44e3-26a6b04c7381 - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.463Z [WARN] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: rejecting vote request since we have a leader: from=1ebcdbe2-8138-6c90-62c3-5df3dd9af500 leader=5a8c3d35-3841-5006-44e3-26a6b04c7381 -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.465Z [ERROR] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: peer has newer term, stopping replication: peer="{Voter server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500 1ebcdbe2-8138-6c90-62c3-5df3dd9af500}" - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.465Z [ERROR] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: peer has newer term, stopping replication: peer="{Voter server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500 1ebcdbe2-8138-6c90-62c3-5df3dd9af500}" -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.465Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: entering follower state: follower="Node at 5a8c3d35-3841-5006-44e3-26a6b04c7381 [Follower]" leader= - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.465Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: entering follower state: follower="Node at 5a8c3d35-3841-5006-44e3-26a6b04c7381 [Follower]" leader= -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.465Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: aborting pipeline replication: peer="{Voter server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c 3ce7168e-57db-a8db-f9f5-f1bc45652c3c}" - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.465Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: aborting pipeline replication: peer="{Voter server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c 3ce7168e-57db-a8db-f9f5-f1bc45652c3c}" -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.537Z [WARN] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: Election timeout reached, restarting election - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.537Z [WARN] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: Election timeout reached, restarting election -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.537Z [INFO] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: entering candidate state: node="Node at 1ebcdbe2-8138-6c90-62c3-5df3dd9af500 [Candidate]" term=6 - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.537Z [INFO] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: entering candidate state: node="Node at 1ebcdbe2-8138-6c90-62c3-5df3dd9af500 [Candidate]" term=6 -testLoggerAdapter verbose: server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c: 2024-09-03T21:34:24.538Z [WARN] server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c: rejecting vote request since we have a leader: from=1ebcdbe2-8138-6c90-62c3-5df3dd9af500 leader=5a8c3d35-3841-5006-44e3-26a6b04c7381 -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.538Z [WARN] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: rejecting vote request since our last term is greater: candidate=1ebcdbe2-8138-6c90-62c3-5df3dd9af500 last-term=4 last-candidate-term=2 - testing.go:149: server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c: 2024-09-03T21:34:24.538Z [WARN] server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c: rejecting vote request since we have a leader: from=1ebcdbe2-8138-6c90-62c3-5df3dd9af500 leader=5a8c3d35-3841-5006-44e3-26a6b04c7381 - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.538Z [WARN] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: rejecting vote request since our last term is greater: candidate=1ebcdbe2-8138-6c90-62c3-5df3dd9af500 last-term=4 last-candidate-term=2 -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.547Z [WARN] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.547Z [WARN] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.547Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: entering candidate state: node="Node at 5a8c3d35-3841-5006-44e3-26a6b04c7381 [Candidate]" term=7 - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.547Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: entering candidate state: node="Node at 5a8c3d35-3841-5006-44e3-26a6b04c7381 [Candidate]" term=7 -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.547Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: election won: tally=2 - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.547Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: election won: tally=2 -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.547Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: entering leader state: leader="Node at 5a8c3d35-3841-5006-44e3-26a6b04c7381 [Leader]" - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.547Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: entering leader state: leader="Node at 5a8c3d35-3841-5006-44e3-26a6b04c7381 [Leader]" -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.547Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: added peer, starting replication: peer=server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500 - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.547Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: added peer, starting replication: peer=server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500 -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.547Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: added peer, starting replication: peer=server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.547Z [INFO] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: entering follower state: follower="Node at 1ebcdbe2-8138-6c90-62c3-5df3dd9af500 [Follower]" leader= - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.547Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: added peer, starting replication: peer=server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.547Z [INFO] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: entering follower state: follower="Node at 1ebcdbe2-8138-6c90-62c3-5df3dd9af500 [Follower]" leader= -testLoggerAdapter verbose: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.547Z [WARN] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to get previous log: previous-index=5 last-index=3 error="log not found" - testing.go:149: server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: 2024-09-03T21:34:24.547Z [WARN] server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500: failed to get previous log: previous-index=5 last-index=3 error="log not found" -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.547Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: pipelining replication: peer="{Voter server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c 3ce7168e-57db-a8db-f9f5-f1bc45652c3c}" - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.547Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: pipelining replication: peer="{Voter server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c 3ce7168e-57db-a8db-f9f5-f1bc45652c3c}" -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.547Z [WARN] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: appendEntries rejected, sending older logs: peer="{Voter server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500 1ebcdbe2-8138-6c90-62c3-5df3dd9af500}" next=4 - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.547Z [WARN] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: appendEntries rejected, sending older logs: peer="{Voter server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500 1ebcdbe2-8138-6c90-62c3-5df3dd9af500}" next=4 -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.547Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: pipelining replication: peer="{Voter server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500 1ebcdbe2-8138-6c90-62c3-5df3dd9af500}" - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.547Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: pipelining replication: peer="{Voter server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500 1ebcdbe2-8138-6c90-62c3-5df3dd9af500}" -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.552Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: aborting pipeline replication: peer="{Voter server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c 3ce7168e-57db-a8db-f9f5-f1bc45652c3c}" - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.552Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: aborting pipeline replication: peer="{Voter server-3ce7168e-57db-a8db-f9f5-f1bc45652c3c 3ce7168e-57db-a8db-f9f5-f1bc45652c3c}" -testLoggerAdapter verbose: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.553Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: aborting pipeline replication: peer="{Voter server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500 1ebcdbe2-8138-6c90-62c3-5df3dd9af500}" - testing.go:149: server-5a8c3d35-3841-5006-44e3-26a6b04c7381: 2024-09-03T21:34:24.553Z [INFO] server-5a8c3d35-3841-5006-44e3-26a6b04c7381: aborting pipeline replication: peer="{Voter server-1ebcdbe2-8138-6c90-62c3-5df3dd9af500 1ebcdbe2-8138-6c90-62c3-5df3dd9af500}" ---- PASS: TestRaft_LeaderFail (0.62s) +testLoggerAdapter verbose: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:52.899Z [INFO] server-c2229225-6527-5735-0db9-d0ba6638f6ab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2229225-6527-5735-0db9-d0ba6638f6ab Address:c2229225-6527-5735-0db9-d0ba6638f6ab} {Suffrage:Voter ID:server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 Address:b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47} {Suffrage:Voter ID:server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d Address:90e7a8fc-6dd5-aad7-a258-1318a6f7815d}]" + testing.go:149: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:52.899Z [INFO] server-c2229225-6527-5735-0db9-d0ba6638f6ab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2229225-6527-5735-0db9-d0ba6638f6ab Address:c2229225-6527-5735-0db9-d0ba6638f6ab} {Suffrage:Voter ID:server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 Address:b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47} {Suffrage:Voter ID:server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d Address:90e7a8fc-6dd5-aad7-a258-1318a6f7815d}]" +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.899Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2229225-6527-5735-0db9-d0ba6638f6ab Address:c2229225-6527-5735-0db9-d0ba6638f6ab} {Suffrage:Voter ID:server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 Address:b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47} {Suffrage:Voter ID:server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d Address:90e7a8fc-6dd5-aad7-a258-1318a6f7815d}]" + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.899Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2229225-6527-5735-0db9-d0ba6638f6ab Address:c2229225-6527-5735-0db9-d0ba6638f6ab} {Suffrage:Voter ID:server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 Address:b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47} {Suffrage:Voter ID:server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d Address:90e7a8fc-6dd5-aad7-a258-1318a6f7815d}]" +testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:52.899Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2229225-6527-5735-0db9-d0ba6638f6ab Address:c2229225-6527-5735-0db9-d0ba6638f6ab} {Suffrage:Voter ID:server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 Address:b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47} {Suffrage:Voter ID:server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d Address:90e7a8fc-6dd5-aad7-a258-1318a6f7815d}]" + testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:52.899Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2229225-6527-5735-0db9-d0ba6638f6ab Address:c2229225-6527-5735-0db9-d0ba6638f6ab} {Suffrage:Voter ID:server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 Address:b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47} {Suffrage:Voter ID:server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d Address:90e7a8fc-6dd5-aad7-a258-1318a6f7815d}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:52.899Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-08-02T15:12:52.899Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:52.899Z [INFO] server-c2229225-6527-5735-0db9-d0ba6638f6ab: entering follower state: follower="Node at c2229225-6527-5735-0db9-d0ba6638f6ab [Follower]" leader= + testing.go:149: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:52.899Z [INFO] server-c2229225-6527-5735-0db9-d0ba6638f6ab: entering follower state: follower="Node at c2229225-6527-5735-0db9-d0ba6638f6ab [Follower]" leader= +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.899Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: entering follower state: follower="Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Follower]" leader= + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.899Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: entering follower state: follower="Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Follower]" leader= +testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:52.899Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: entering follower state: follower="Node at 90e7a8fc-6dd5-aad7-a258-1318a6f7815d [Follower]" leader= + testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:52.899Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: entering follower state: follower="Node at 90e7a8fc-6dd5-aad7-a258-1318a6f7815d [Follower]" leader= +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.985Z [WARN] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.985Z [WARN] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.985Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: entering candidate state: node="Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Candidate]" term=2 + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.985Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: entering candidate state: node="Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Candidate]" term=2 +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.986Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: election won: tally=2 + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.986Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: election won: tally=2 +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.986Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: entering leader state: leader="Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Leader]" + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.986Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: entering leader state: leader="Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Leader]" +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.986Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: added peer, starting replication: peer=server-c2229225-6527-5735-0db9-d0ba6638f6ab + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.986Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: added peer, starting replication: peer=server-c2229225-6527-5735-0db9-d0ba6638f6ab +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.986Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: added peer, starting replication: peer=server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.986Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: added peer, starting replication: peer=server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.986Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: pipelining replication: peer="{Voter server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d 90e7a8fc-6dd5-aad7-a258-1318a6f7815d}" + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.986Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: pipelining replication: peer="{Voter server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d 90e7a8fc-6dd5-aad7-a258-1318a6f7815d}" +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.986Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: pipelining replication: peer="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.986Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: pipelining replication: peer="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:53.091Z [INFO] stable state for Follower reached at 2023-08-02 15:12:52.986144386 +0000 UTC m=+5.082080720 (2 nodes), 86.687489ms from start of poll, 87.245575ms from cluster start. Timeout at 2023-08-02 15:12:53.091261464 +0000 UTC m=+5.187197809, 105.117089ms after stability + testing.go:149: cluster: 2023-08-02T15:12:53.091Z [INFO] stable state for Follower reached at 2023-08-02 15:12:52.986144386 +0000 UTC m=+5.082080720 (2 nodes), 86.687489ms from start of poll, 87.245575ms from cluster start. Timeout at 2023-08-02 15:12:53.091261464 +0000 UTC m=+5.187197809, 105.117089ms after stability +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:53.091Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:12:53.091Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:53.196Z [INFO] stable state for Leader reached at 2023-08-02 15:12:53.091428116 +0000 UTC m=+5.187364444 (1 nodes), 1.345µs from start of poll, 192.529299ms from cluster start. Timeout at 2023-08-02 15:12:53.196520949 +0000 UTC m=+5.292457284, 105.09284ms after stability + testing.go:149: cluster: 2023-08-02T15:12:53.196Z [INFO] stable state for Leader reached at 2023-08-02 15:12:53.091428116 +0000 UTC m=+5.187364444 (1 nodes), 1.345µs from start of poll, 192.529299ms from cluster start. Timeout at 2023-08-02 15:12:53.196520949 +0000 UTC m=+5.292457284, 105.09284ms after stability + raft_test.go:336: [INFO] Disconnecting Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Leader] +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.209Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to pipeline appendEntries: peer="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" error="append pipeline closed" + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.209Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to pipeline appendEntries: peer="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" error="append pipeline closed" +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.209Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: aborting pipeline replication: peer="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.209Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: aborting pipeline replication: peer="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.210Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to heartbeat to: peer=c2229225-6527-5735-0db9-d0ba6638f6ab error="failed to connect to peer: c2229225-6527-5735-0db9-d0ba6638f6ab" + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.210Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to heartbeat to: peer=c2229225-6527-5735-0db9-d0ba6638f6ab error="failed to connect to peer: c2229225-6527-5735-0db9-d0ba6638f6ab" +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.211Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to heartbeat to: peer=90e7a8fc-6dd5-aad7-a258-1318a6f7815d error="failed to connect to peer: 90e7a8fc-6dd5-aad7-a258-1318a6f7815d" + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.211Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to heartbeat to: peer=90e7a8fc-6dd5-aad7-a258-1318a6f7815d error="failed to connect to peer: 90e7a8fc-6dd5-aad7-a258-1318a6f7815d" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:53.212Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:12:53.212Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.212Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to pipeline appendEntries: peer="{Voter server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d 90e7a8fc-6dd5-aad7-a258-1318a6f7815d}" error="append pipeline closed" + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.212Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to pipeline appendEntries: peer="{Voter server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d 90e7a8fc-6dd5-aad7-a258-1318a6f7815d}" error="append pipeline closed" +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.212Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: aborting pipeline replication: peer="{Voter server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d 90e7a8fc-6dd5-aad7-a258-1318a6f7815d}" + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.212Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: aborting pipeline replication: peer="{Voter server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d 90e7a8fc-6dd5-aad7-a258-1318a6f7815d}" +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.218Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to appendEntries to: peer="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" error="failed to connect to peer: c2229225-6527-5735-0db9-d0ba6638f6ab" + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.218Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to appendEntries to: peer="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" error="failed to connect to peer: c2229225-6527-5735-0db9-d0ba6638f6ab" +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.221Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to appendEntries to: peer="{Voter server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d 90e7a8fc-6dd5-aad7-a258-1318a6f7815d}" error="failed to connect to peer: 90e7a8fc-6dd5-aad7-a258-1318a6f7815d" + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.221Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to appendEntries to: peer="{Voter server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d 90e7a8fc-6dd5-aad7-a258-1318a6f7815d}" error="failed to connect to peer: 90e7a8fc-6dd5-aad7-a258-1318a6f7815d" +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.228Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to heartbeat to: peer=90e7a8fc-6dd5-aad7-a258-1318a6f7815d error="failed to connect to peer: 90e7a8fc-6dd5-aad7-a258-1318a6f7815d" + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.228Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to heartbeat to: peer=90e7a8fc-6dd5-aad7-a258-1318a6f7815d error="failed to connect to peer: 90e7a8fc-6dd5-aad7-a258-1318a6f7815d" +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.229Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to heartbeat to: peer=c2229225-6527-5735-0db9-d0ba6638f6ab error="failed to connect to peer: c2229225-6527-5735-0db9-d0ba6638f6ab" + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.229Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to heartbeat to: peer=c2229225-6527-5735-0db9-d0ba6638f6ab error="failed to connect to peer: c2229225-6527-5735-0db9-d0ba6638f6ab" +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.236Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to appendEntries to: peer="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" error="failed to connect to peer: c2229225-6527-5735-0db9-d0ba6638f6ab" + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.236Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to appendEntries to: peer="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" error="failed to connect to peer: c2229225-6527-5735-0db9-d0ba6638f6ab" +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.239Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to appendEntries to: peer="{Voter server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d 90e7a8fc-6dd5-aad7-a258-1318a6f7815d}" error="failed to connect to peer: 90e7a8fc-6dd5-aad7-a258-1318a6f7815d" + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.239Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to appendEntries to: peer="{Voter server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d 90e7a8fc-6dd5-aad7-a258-1318a6f7815d}" error="failed to connect to peer: 90e7a8fc-6dd5-aad7-a258-1318a6f7815d" +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.245Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to heartbeat to: peer=90e7a8fc-6dd5-aad7-a258-1318a6f7815d error="failed to connect to peer: 90e7a8fc-6dd5-aad7-a258-1318a6f7815d" + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.245Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to heartbeat to: peer=90e7a8fc-6dd5-aad7-a258-1318a6f7815d error="failed to connect to peer: 90e7a8fc-6dd5-aad7-a258-1318a6f7815d" +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.249Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to heartbeat to: peer=c2229225-6527-5735-0db9-d0ba6638f6ab error="failed to connect to peer: c2229225-6527-5735-0db9-d0ba6638f6ab" + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.249Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to heartbeat to: peer=c2229225-6527-5735-0db9-d0ba6638f6ab error="failed to connect to peer: c2229225-6527-5735-0db9-d0ba6638f6ab" +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.255Z [WARN] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to contact: server-id=server-c2229225-6527-5735-0db9-d0ba6638f6ab time=51.598509ms + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.255Z [WARN] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to contact: server-id=server-c2229225-6527-5735-0db9-d0ba6638f6ab time=51.598509ms +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.256Z [WARN] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to contact: server-id=server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d time=51.07772ms + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.256Z [WARN] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to contact: server-id=server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d time=51.07772ms +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.256Z [WARN] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to contact quorum of nodes, stepping down + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.256Z [WARN] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to contact quorum of nodes, stepping down +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.256Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: entering follower state: follower="Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Follower]" leader= + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.256Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: entering follower state: follower="Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Follower]" leader= +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.255Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to appendEntries to: peer="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" error="failed to connect to peer: c2229225-6527-5735-0db9-d0ba6638f6ab" + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.255Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to appendEntries to: peer="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" error="failed to connect to peer: c2229225-6527-5735-0db9-d0ba6638f6ab" +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.257Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to appendEntries to: peer="{Voter server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d 90e7a8fc-6dd5-aad7-a258-1318a6f7815d}" error="failed to connect to peer: 90e7a8fc-6dd5-aad7-a258-1318a6f7815d" + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.257Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to appendEntries to: peer="{Voter server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d 90e7a8fc-6dd5-aad7-a258-1318a6f7815d}" error="failed to connect to peer: 90e7a8fc-6dd5-aad7-a258-1318a6f7815d" +testLoggerAdapter verbose: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.307Z [WARN] server-c2229225-6527-5735-0db9-d0ba6638f6ab: heartbeat timeout reached, starting election: last-leader=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 + testing.go:149: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.307Z [WARN] server-c2229225-6527-5735-0db9-d0ba6638f6ab: heartbeat timeout reached, starting election: last-leader=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 +testLoggerAdapter verbose: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.308Z [INFO] server-c2229225-6527-5735-0db9-d0ba6638f6ab: entering candidate state: node="Node at c2229225-6527-5735-0db9-d0ba6638f6ab [Candidate]" term=3 + testing.go:149: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.308Z [INFO] server-c2229225-6527-5735-0db9-d0ba6638f6ab: entering candidate state: node="Node at c2229225-6527-5735-0db9-d0ba6638f6ab [Candidate]" term=3 +testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.308Z [WARN] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: rejecting vote request since we have a leader: from=c2229225-6527-5735-0db9-d0ba6638f6ab leader=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 + testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.308Z [WARN] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: rejecting vote request since we have a leader: from=c2229225-6527-5735-0db9-d0ba6638f6ab leader=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 +testLoggerAdapter verbose: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.308Z [ERROR] server-c2229225-6527-5735-0db9-d0ba6638f6ab: failed to make requestVote RPC: target="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" + testing.go:149: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.308Z [ERROR] server-c2229225-6527-5735-0db9-d0ba6638f6ab: failed to make requestVote RPC: target="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" +testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.310Z [WARN] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: heartbeat timeout reached, starting election: last-leader=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 + testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.310Z [WARN] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: heartbeat timeout reached, starting election: last-leader=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 +testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.310Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: entering candidate state: node="Node at 90e7a8fc-6dd5-aad7-a258-1318a6f7815d [Candidate]" term=3 + testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.310Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: entering candidate state: node="Node at 90e7a8fc-6dd5-aad7-a258-1318a6f7815d [Candidate]" term=3 +testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.310Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to make requestVote RPC: target="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" + testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.310Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to make requestVote RPC: target="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" +testLoggerAdapter verbose: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.310Z [INFO] server-c2229225-6527-5735-0db9-d0ba6638f6ab: duplicate requestVote for same term: term=3 + testing.go:149: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.310Z [INFO] server-c2229225-6527-5735-0db9-d0ba6638f6ab: duplicate requestVote for same term: term=3 +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.317Z [WARN] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.317Z [WARN] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.317Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: entering candidate state: node="Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Candidate]" term=3 + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.317Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: entering candidate state: node="Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Candidate]" term=3 +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.317Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to make requestVote RPC: target="{Voter server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d 90e7a8fc-6dd5-aad7-a258-1318a6f7815d}" error="failed to connect to peer: 90e7a8fc-6dd5-aad7-a258-1318a6f7815d" + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.317Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to make requestVote RPC: target="{Voter server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d 90e7a8fc-6dd5-aad7-a258-1318a6f7815d}" error="failed to connect to peer: 90e7a8fc-6dd5-aad7-a258-1318a6f7815d" +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.317Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to make requestVote RPC: target="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" error="failed to connect to peer: c2229225-6527-5735-0db9-d0ba6638f6ab" + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.317Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to make requestVote RPC: target="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" error="failed to connect to peer: c2229225-6527-5735-0db9-d0ba6638f6ab" +testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.362Z [WARN] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: Election timeout reached, restarting election + testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.362Z [WARN] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: Election timeout reached, restarting election +testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.362Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: entering candidate state: node="Node at 90e7a8fc-6dd5-aad7-a258-1318a6f7815d [Candidate]" term=4 + testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.362Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: entering candidate state: node="Node at 90e7a8fc-6dd5-aad7-a258-1318a6f7815d [Candidate]" term=4 +testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.362Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to make requestVote RPC: target="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" + testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.362Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to make requestVote RPC: target="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" +testLoggerAdapter verbose: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.362Z [INFO] server-c2229225-6527-5735-0db9-d0ba6638f6ab: entering follower state: follower="Node at c2229225-6527-5735-0db9-d0ba6638f6ab [Follower]" leader= + testing.go:149: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.362Z [INFO] server-c2229225-6527-5735-0db9-d0ba6638f6ab: entering follower state: follower="Node at c2229225-6527-5735-0db9-d0ba6638f6ab [Follower]" leader= +testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.362Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: election won: tally=2 + testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.362Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: election won: tally=2 +testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.362Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: entering leader state: leader="Node at 90e7a8fc-6dd5-aad7-a258-1318a6f7815d [Leader]" + testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.362Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: entering leader state: leader="Node at 90e7a8fc-6dd5-aad7-a258-1318a6f7815d [Leader]" +testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.362Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: added peer, starting replication: peer=server-c2229225-6527-5735-0db9-d0ba6638f6ab + testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.362Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: added peer, starting replication: peer=server-c2229225-6527-5735-0db9-d0ba6638f6ab +testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.363Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: added peer, starting replication: peer=server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 + testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.363Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: added peer, starting replication: peer=server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 +testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.363Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to appendEntries to: peer="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" + testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.363Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to appendEntries to: peer="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" +testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.363Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: pipelining replication: peer="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" + testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.363Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: pipelining replication: peer="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" +testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.369Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to heartbeat to: peer=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" + testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.369Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to heartbeat to: peer=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" +testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.383Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to appendEntries to: peer="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" + testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.383Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to appendEntries to: peer="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" +testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.389Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to heartbeat to: peer=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" + testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.389Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to heartbeat to: peer=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.402Z [WARN] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: Election timeout reached, restarting election + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.402Z [WARN] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: Election timeout reached, restarting election +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.403Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: entering candidate state: node="Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Candidate]" term=4 + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.403Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: entering candidate state: node="Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Candidate]" term=4 +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.403Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to make requestVote RPC: target="{Voter server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d 90e7a8fc-6dd5-aad7-a258-1318a6f7815d}" error="failed to connect to peer: 90e7a8fc-6dd5-aad7-a258-1318a6f7815d" + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.403Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to make requestVote RPC: target="{Voter server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d 90e7a8fc-6dd5-aad7-a258-1318a6f7815d}" error="failed to connect to peer: 90e7a8fc-6dd5-aad7-a258-1318a6f7815d" +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.403Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to make requestVote RPC: target="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" error="failed to connect to peer: c2229225-6527-5735-0db9-d0ba6638f6ab" + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.403Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to make requestVote RPC: target="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" error="failed to connect to peer: c2229225-6527-5735-0db9-d0ba6638f6ab" +testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.404Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to appendEntries to: peer="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" + testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.404Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to appendEntries to: peer="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" +testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.409Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to heartbeat to: peer=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" + testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.409Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to heartbeat to: peer=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" +testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.414Z [WARN] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to contact: server-id=server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 time=50.992637ms + testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.414Z [WARN] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to contact: server-id=server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 time=50.992637ms +testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.433Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to appendEntries to: peer="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" + testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.433Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to appendEntries to: peer="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" +testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.436Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to heartbeat to: peer=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" + testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.436Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to heartbeat to: peer=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" +testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.460Z [WARN] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to contact: server-id=server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 time=97.797067ms + testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.460Z [WARN] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to contact: server-id=server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 time=97.797067ms +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:53.468Z [INFO] stable state for Leader reached at 2023-08-02 15:12:53.362960679 +0000 UTC m=+5.458897005 (1 nodes), 150.682524ms from start of poll, 464.06186ms from cluster start. Timeout at 2023-08-02 15:12:53.468104224 +0000 UTC m=+5.564040554, 105.143549ms after stability + testing.go:149: cluster: 2023-08-02T15:12:53.468Z [INFO] stable state for Leader reached at 2023-08-02 15:12:53.362960679 +0000 UTC m=+5.458897005 (1 nodes), 150.682524ms from start of poll, 464.06186ms from cluster start. Timeout at 2023-08-02 15:12:53.468104224 +0000 UTC m=+5.564040554, 105.143549ms after stability + raft_test.go:371: [INFO] Reconnecting Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Candidate] +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.474Z [WARN] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: Election timeout reached, restarting election + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.474Z [WARN] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: Election timeout reached, restarting election +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.474Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: entering candidate state: node="Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Candidate]" term=5 + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.474Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: entering candidate state: node="Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Candidate]" term=5 +testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.475Z [WARN] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: rejecting vote request since we have a leader: from=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 leader=90e7a8fc-6dd5-aad7-a258-1318a6f7815d + testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.475Z [WARN] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: rejecting vote request since we have a leader: from=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 leader=90e7a8fc-6dd5-aad7-a258-1318a6f7815d +testLoggerAdapter verbose: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.475Z [WARN] server-c2229225-6527-5735-0db9-d0ba6638f6ab: rejecting vote request since we have a leader: from=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 leader=90e7a8fc-6dd5-aad7-a258-1318a6f7815d + testing.go:149: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.475Z [WARN] server-c2229225-6527-5735-0db9-d0ba6638f6ab: rejecting vote request since we have a leader: from=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 leader=90e7a8fc-6dd5-aad7-a258-1318a6f7815d +testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.483Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: peer has newer term, stopping replication: peer="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" + testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.483Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: peer has newer term, stopping replication: peer="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" +testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.484Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: entering follower state: follower="Node at 90e7a8fc-6dd5-aad7-a258-1318a6f7815d [Follower]" leader= + testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.484Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: entering follower state: follower="Node at 90e7a8fc-6dd5-aad7-a258-1318a6f7815d [Follower]" leader= +testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.484Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: aborting pipeline replication: peer="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" + testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.484Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: aborting pipeline replication: peer="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.526Z [WARN] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: Election timeout reached, restarting election + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.526Z [WARN] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: Election timeout reached, restarting election +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.527Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: entering candidate state: node="Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Candidate]" term=6 + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.527Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: entering candidate state: node="Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Candidate]" term=6 +testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.527Z [WARN] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: rejecting vote request since our last term is greater: candidate=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 last-term=4 last-candidate-term=2 + testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.527Z [WARN] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: rejecting vote request since our last term is greater: candidate=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 last-term=4 last-candidate-term=2 +testLoggerAdapter verbose: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.527Z [WARN] server-c2229225-6527-5735-0db9-d0ba6638f6ab: rejecting vote request since we have a leader: from=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 leader=90e7a8fc-6dd5-aad7-a258-1318a6f7815d + testing.go:149: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.527Z [WARN] server-c2229225-6527-5735-0db9-d0ba6638f6ab: rejecting vote request since we have a leader: from=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 leader=90e7a8fc-6dd5-aad7-a258-1318a6f7815d +testLoggerAdapter verbose: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.528Z [WARN] server-c2229225-6527-5735-0db9-d0ba6638f6ab: heartbeat timeout reached, starting election: last-leader=90e7a8fc-6dd5-aad7-a258-1318a6f7815d + testing.go:149: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.528Z [WARN] server-c2229225-6527-5735-0db9-d0ba6638f6ab: heartbeat timeout reached, starting election: last-leader=90e7a8fc-6dd5-aad7-a258-1318a6f7815d +testLoggerAdapter verbose: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.528Z [INFO] server-c2229225-6527-5735-0db9-d0ba6638f6ab: entering candidate state: node="Node at c2229225-6527-5735-0db9-d0ba6638f6ab [Candidate]" term=5 + testing.go:149: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.528Z [INFO] server-c2229225-6527-5735-0db9-d0ba6638f6ab: entering candidate state: node="Node at c2229225-6527-5735-0db9-d0ba6638f6ab [Candidate]" term=5 +testLoggerAdapter verbose: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.528Z [INFO] server-c2229225-6527-5735-0db9-d0ba6638f6ab: entering follower state: follower="Node at c2229225-6527-5735-0db9-d0ba6638f6ab [Follower]" leader= + testing.go:149: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.528Z [INFO] server-c2229225-6527-5735-0db9-d0ba6638f6ab: entering follower state: follower="Node at c2229225-6527-5735-0db9-d0ba6638f6ab [Follower]" leader= +testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.570Z [WARN] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.570Z [WARN] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.570Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: entering candidate state: node="Node at 90e7a8fc-6dd5-aad7-a258-1318a6f7815d [Candidate]" term=7 + testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.570Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: entering candidate state: node="Node at 90e7a8fc-6dd5-aad7-a258-1318a6f7815d [Candidate]" term=7 +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.570Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: entering follower state: follower="Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Follower]" leader= + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.570Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: entering follower state: follower="Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Follower]" leader= +testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.570Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: election won: tally=2 + testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.570Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: election won: tally=2 +testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.570Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: entering leader state: leader="Node at 90e7a8fc-6dd5-aad7-a258-1318a6f7815d [Leader]" + testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.570Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: entering leader state: leader="Node at 90e7a8fc-6dd5-aad7-a258-1318a6f7815d [Leader]" +testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.570Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: added peer, starting replication: peer=server-c2229225-6527-5735-0db9-d0ba6638f6ab + testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.570Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: added peer, starting replication: peer=server-c2229225-6527-5735-0db9-d0ba6638f6ab +testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.570Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: added peer, starting replication: peer=server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 + testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.570Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: added peer, starting replication: peer=server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 +testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.570Z [WARN] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to get previous log: previous-index=5 last-index=3 error="log not found" + testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.570Z [WARN] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to get previous log: previous-index=5 last-index=3 error="log not found" +testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.570Z [WARN] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: appendEntries rejected, sending older logs: peer="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" next=4 + testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.570Z [WARN] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: appendEntries rejected, sending older logs: peer="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" next=4 +testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.570Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: pipelining replication: peer="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" + testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.570Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: pipelining replication: peer="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" +testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.570Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: aborting pipeline replication: peer="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" + testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.570Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: aborting pipeline replication: peer="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" +testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.620Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to make requestVote RPC: target="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" error="command timed out" + testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.620Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to make requestVote RPC: target="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" error="command timed out" +--- PASS: TestRaft_LeaderFail (0.72s) === RUN TestRaft_BehindFollower -testLoggerAdapter verbose: server-6919d5b4-99da-c295-853b-28d9f25c2575: 2024-09-03T21:34:24.553Z [INFO] server-6919d5b4-99da-c295-853b-28d9f25c2575: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6919d5b4-99da-c295-853b-28d9f25c2575 Address:6919d5b4-99da-c295-853b-28d9f25c2575} {Suffrage:Voter ID:server-183fd635-40c0-401c-fd23-bfa19f572e84 Address:183fd635-40c0-401c-fd23-bfa19f572e84} {Suffrage:Voter ID:server-c5a4b299-1163-212c-d947-2a6bcfa106dd Address:c5a4b299-1163-212c-d947-2a6bcfa106dd}]" - testing.go:149: server-6919d5b4-99da-c295-853b-28d9f25c2575: 2024-09-03T21:34:24.553Z [INFO] server-6919d5b4-99da-c295-853b-28d9f25c2575: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6919d5b4-99da-c295-853b-28d9f25c2575 Address:6919d5b4-99da-c295-853b-28d9f25c2575} {Suffrage:Voter ID:server-183fd635-40c0-401c-fd23-bfa19f572e84 Address:183fd635-40c0-401c-fd23-bfa19f572e84} {Suffrage:Voter ID:server-c5a4b299-1163-212c-d947-2a6bcfa106dd Address:c5a4b299-1163-212c-d947-2a6bcfa106dd}]" -testLoggerAdapter verbose: server-183fd635-40c0-401c-fd23-bfa19f572e84: 2024-09-03T21:34:24.554Z [INFO] server-183fd635-40c0-401c-fd23-bfa19f572e84: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6919d5b4-99da-c295-853b-28d9f25c2575 Address:6919d5b4-99da-c295-853b-28d9f25c2575} {Suffrage:Voter ID:server-183fd635-40c0-401c-fd23-bfa19f572e84 Address:183fd635-40c0-401c-fd23-bfa19f572e84} {Suffrage:Voter ID:server-c5a4b299-1163-212c-d947-2a6bcfa106dd Address:c5a4b299-1163-212c-d947-2a6bcfa106dd}]" - testing.go:149: server-183fd635-40c0-401c-fd23-bfa19f572e84: 2024-09-03T21:34:24.554Z [INFO] server-183fd635-40c0-401c-fd23-bfa19f572e84: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6919d5b4-99da-c295-853b-28d9f25c2575 Address:6919d5b4-99da-c295-853b-28d9f25c2575} {Suffrage:Voter ID:server-183fd635-40c0-401c-fd23-bfa19f572e84 Address:183fd635-40c0-401c-fd23-bfa19f572e84} {Suffrage:Voter ID:server-c5a4b299-1163-212c-d947-2a6bcfa106dd Address:c5a4b299-1163-212c-d947-2a6bcfa106dd}]" -testLoggerAdapter verbose: server-6919d5b4-99da-c295-853b-28d9f25c2575: 2024-09-03T21:34:24.554Z [INFO] server-6919d5b4-99da-c295-853b-28d9f25c2575: entering follower state: follower="Node at 6919d5b4-99da-c295-853b-28d9f25c2575 [Follower]" leader= -testLoggerAdapter verbose: server-c5a4b299-1163-212c-d947-2a6bcfa106dd: 2024-09-03T21:34:24.554Z [INFO] server-c5a4b299-1163-212c-d947-2a6bcfa106dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6919d5b4-99da-c295-853b-28d9f25c2575 Address:6919d5b4-99da-c295-853b-28d9f25c2575} {Suffrage:Voter ID:server-183fd635-40c0-401c-fd23-bfa19f572e84 Address:183fd635-40c0-401c-fd23-bfa19f572e84} {Suffrage:Voter ID:server-c5a4b299-1163-212c-d947-2a6bcfa106dd Address:c5a4b299-1163-212c-d947-2a6bcfa106dd}]" - testing.go:149: server-6919d5b4-99da-c295-853b-28d9f25c2575: 2024-09-03T21:34:24.554Z [INFO] server-6919d5b4-99da-c295-853b-28d9f25c2575: entering follower state: follower="Node at 6919d5b4-99da-c295-853b-28d9f25c2575 [Follower]" leader= -testLoggerAdapter verbose: server-183fd635-40c0-401c-fd23-bfa19f572e84: 2024-09-03T21:34:24.554Z [INFO] server-183fd635-40c0-401c-fd23-bfa19f572e84: entering follower state: follower="Node at 183fd635-40c0-401c-fd23-bfa19f572e84 [Follower]" leader= - testing.go:149: server-183fd635-40c0-401c-fd23-bfa19f572e84: 2024-09-03T21:34:24.554Z [INFO] server-183fd635-40c0-401c-fd23-bfa19f572e84: entering follower state: follower="Node at 183fd635-40c0-401c-fd23-bfa19f572e84 [Follower]" leader= - testing.go:149: server-c5a4b299-1163-212c-d947-2a6bcfa106dd: 2024-09-03T21:34:24.554Z [INFO] server-c5a4b299-1163-212c-d947-2a6bcfa106dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6919d5b4-99da-c295-853b-28d9f25c2575 Address:6919d5b4-99da-c295-853b-28d9f25c2575} {Suffrage:Voter ID:server-183fd635-40c0-401c-fd23-bfa19f572e84 Address:183fd635-40c0-401c-fd23-bfa19f572e84} {Suffrage:Voter ID:server-c5a4b299-1163-212c-d947-2a6bcfa106dd Address:c5a4b299-1163-212c-d947-2a6bcfa106dd}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:24.554Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:24.554Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-c5a4b299-1163-212c-d947-2a6bcfa106dd: 2024-09-03T21:34:24.554Z [INFO] server-c5a4b299-1163-212c-d947-2a6bcfa106dd: entering follower state: follower="Node at c5a4b299-1163-212c-d947-2a6bcfa106dd [Follower]" leader= - testing.go:149: server-c5a4b299-1163-212c-d947-2a6bcfa106dd: 2024-09-03T21:34:24.554Z [INFO] server-c5a4b299-1163-212c-d947-2a6bcfa106dd: entering follower state: follower="Node at c5a4b299-1163-212c-d947-2a6bcfa106dd [Follower]" leader= -testLoggerAdapter verbose: server-183fd635-40c0-401c-fd23-bfa19f572e84: 2024-09-03T21:34:24.609Z [WARN] server-183fd635-40c0-401c-fd23-bfa19f572e84: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-183fd635-40c0-401c-fd23-bfa19f572e84: 2024-09-03T21:34:24.609Z [WARN] server-183fd635-40c0-401c-fd23-bfa19f572e84: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-183fd635-40c0-401c-fd23-bfa19f572e84: 2024-09-03T21:34:24.610Z [INFO] server-183fd635-40c0-401c-fd23-bfa19f572e84: entering candidate state: node="Node at 183fd635-40c0-401c-fd23-bfa19f572e84 [Candidate]" term=2 - testing.go:149: server-183fd635-40c0-401c-fd23-bfa19f572e84: 2024-09-03T21:34:24.610Z [INFO] server-183fd635-40c0-401c-fd23-bfa19f572e84: entering candidate state: node="Node at 183fd635-40c0-401c-fd23-bfa19f572e84 [Candidate]" term=2 -testLoggerAdapter verbose: server-183fd635-40c0-401c-fd23-bfa19f572e84: 2024-09-03T21:34:24.610Z [INFO] server-183fd635-40c0-401c-fd23-bfa19f572e84: election won: tally=2 - testing.go:149: server-183fd635-40c0-401c-fd23-bfa19f572e84: 2024-09-03T21:34:24.610Z [INFO] server-183fd635-40c0-401c-fd23-bfa19f572e84: election won: tally=2 -testLoggerAdapter verbose: server-183fd635-40c0-401c-fd23-bfa19f572e84: 2024-09-03T21:34:24.610Z [INFO] server-183fd635-40c0-401c-fd23-bfa19f572e84: entering leader state: leader="Node at 183fd635-40c0-401c-fd23-bfa19f572e84 [Leader]" - testing.go:149: server-183fd635-40c0-401c-fd23-bfa19f572e84: 2024-09-03T21:34:24.610Z [INFO] server-183fd635-40c0-401c-fd23-bfa19f572e84: entering leader state: leader="Node at 183fd635-40c0-401c-fd23-bfa19f572e84 [Leader]" -testLoggerAdapter verbose: server-183fd635-40c0-401c-fd23-bfa19f572e84: 2024-09-03T21:34:24.610Z [INFO] server-183fd635-40c0-401c-fd23-bfa19f572e84: added peer, starting replication: peer=server-6919d5b4-99da-c295-853b-28d9f25c2575 - testing.go:149: server-183fd635-40c0-401c-fd23-bfa19f572e84: 2024-09-03T21:34:24.610Z [INFO] server-183fd635-40c0-401c-fd23-bfa19f572e84: added peer, starting replication: peer=server-6919d5b4-99da-c295-853b-28d9f25c2575 -testLoggerAdapter verbose: server-183fd635-40c0-401c-fd23-bfa19f572e84: 2024-09-03T21:34:24.610Z [INFO] server-183fd635-40c0-401c-fd23-bfa19f572e84: added peer, starting replication: peer=server-c5a4b299-1163-212c-d947-2a6bcfa106dd - testing.go:149: server-183fd635-40c0-401c-fd23-bfa19f572e84: 2024-09-03T21:34:24.610Z [INFO] server-183fd635-40c0-401c-fd23-bfa19f572e84: added peer, starting replication: peer=server-c5a4b299-1163-212c-d947-2a6bcfa106dd -testLoggerAdapter verbose: server-183fd635-40c0-401c-fd23-bfa19f572e84: 2024-09-03T21:34:24.610Z [INFO] server-183fd635-40c0-401c-fd23-bfa19f572e84: pipelining replication: peer="{Voter server-c5a4b299-1163-212c-d947-2a6bcfa106dd c5a4b299-1163-212c-d947-2a6bcfa106dd}" - testing.go:149: server-183fd635-40c0-401c-fd23-bfa19f572e84: 2024-09-03T21:34:24.610Z [INFO] server-183fd635-40c0-401c-fd23-bfa19f572e84: pipelining replication: peer="{Voter server-c5a4b299-1163-212c-d947-2a6bcfa106dd c5a4b299-1163-212c-d947-2a6bcfa106dd}" -testLoggerAdapter verbose: server-183fd635-40c0-401c-fd23-bfa19f572e84: 2024-09-03T21:34:24.610Z [INFO] server-183fd635-40c0-401c-fd23-bfa19f572e84: pipelining replication: peer="{Voter server-6919d5b4-99da-c295-853b-28d9f25c2575 6919d5b4-99da-c295-853b-28d9f25c2575}" - testing.go:149: server-183fd635-40c0-401c-fd23-bfa19f572e84: 2024-09-03T21:34:24.610Z [INFO] server-183fd635-40c0-401c-fd23-bfa19f572e84: pipelining replication: peer="{Voter server-6919d5b4-99da-c295-853b-28d9f25c2575 6919d5b4-99da-c295-853b-28d9f25c2575}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:24.715Z [INFO] stable state for Leader reached at 2024-09-03 21:34:24.610527815 +0000 UTC m=+5.657184966 (1 nodes), 55.999815ms from start of poll, 56.692025ms from cluster start. Timeout at 2024-09-03 21:34:24.715607689 +0000 UTC m=+5.762264820, 105.079854ms after stability - testing.go:149: cluster: 2024-09-03T21:34:24.715Z [INFO] stable state for Leader reached at 2024-09-03 21:34:24.610527815 +0000 UTC m=+5.657184966 (1 nodes), 55.999815ms from start of poll, 56.692025ms from cluster start. Timeout at 2024-09-03 21:34:24.715607689 +0000 UTC m=+5.762264820, 105.079854ms after stability -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:24.715Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-09-03T21:34:24.715Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:24.820Z [INFO] stable state for Follower reached at 2024-09-03 21:34:24.715757003 +0000 UTC m=+5.762414138 (2 nodes), 1.148µs from start of poll, 161.921197ms from cluster start. Timeout at 2024-09-03 21:34:24.820844057 +0000 UTC m=+5.867501202, 105.087064ms after stability - testing.go:149: cluster: 2024-09-03T21:34:24.820Z [INFO] stable state for Follower reached at 2024-09-03 21:34:24.715757003 +0000 UTC m=+5.762414138 (2 nodes), 1.148µs from start of poll, 161.921197ms from cluster start. Timeout at 2024-09-03 21:34:24.820844057 +0000 UTC m=+5.867501202, 105.087064ms after stability -testLoggerAdapter verbose: server-183fd635-40c0-401c-fd23-bfa19f572e84: 2024-09-03T21:34:24.821Z [ERROR] server-183fd635-40c0-401c-fd23-bfa19f572e84: failed to pipeline appendEntries: peer="{Voter server-6919d5b4-99da-c295-853b-28d9f25c2575 6919d5b4-99da-c295-853b-28d9f25c2575}" error="append pipeline closed" - testing.go:149: server-183fd635-40c0-401c-fd23-bfa19f572e84: 2024-09-03T21:34:24.821Z [ERROR] server-183fd635-40c0-401c-fd23-bfa19f572e84: failed to pipeline appendEntries: peer="{Voter server-6919d5b4-99da-c295-853b-28d9f25c2575 6919d5b4-99da-c295-853b-28d9f25c2575}" error="append pipeline closed" -testLoggerAdapter verbose: server-183fd635-40c0-401c-fd23-bfa19f572e84: 2024-09-03T21:34:24.821Z [INFO] server-183fd635-40c0-401c-fd23-bfa19f572e84: aborting pipeline replication: peer="{Voter server-6919d5b4-99da-c295-853b-28d9f25c2575 6919d5b4-99da-c295-853b-28d9f25c2575}" - testing.go:149: server-183fd635-40c0-401c-fd23-bfa19f572e84: 2024-09-03T21:34:24.821Z [INFO] server-183fd635-40c0-401c-fd23-bfa19f572e84: aborting pipeline replication: peer="{Voter server-6919d5b4-99da-c295-853b-28d9f25c2575 6919d5b4-99da-c295-853b-28d9f25c2575}" -testLoggerAdapter verbose: server-183fd635-40c0-401c-fd23-bfa19f572e84: 2024-09-03T21:34:24.821Z [ERROR] server-183fd635-40c0-401c-fd23-bfa19f572e84: failed to appendEntries to: peer="{Voter server-6919d5b4-99da-c295-853b-28d9f25c2575 6919d5b4-99da-c295-853b-28d9f25c2575}" error="failed to connect to peer: 6919d5b4-99da-c295-853b-28d9f25c2575" - testing.go:149: server-183fd635-40c0-401c-fd23-bfa19f572e84: 2024-09-03T21:34:24.821Z [ERROR] server-183fd635-40c0-401c-fd23-bfa19f572e84: failed to appendEntries to: peer="{Voter server-6919d5b4-99da-c295-853b-28d9f25c2575 6919d5b4-99da-c295-853b-28d9f25c2575}" error="failed to connect to peer: 6919d5b4-99da-c295-853b-28d9f25c2575" +testLoggerAdapter verbose: server-180d44c5-13c8-1722-a927-e21b06e48e82: 2023-08-02T15:12:53.621Z [INFO] server-180d44c5-13c8-1722-a927-e21b06e48e82: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-180d44c5-13c8-1722-a927-e21b06e48e82 Address:180d44c5-13c8-1722-a927-e21b06e48e82} {Suffrage:Voter ID:server-2c5e301c-60c6-626a-ce4c-88387582e874 Address:2c5e301c-60c6-626a-ce4c-88387582e874} {Suffrage:Voter ID:server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9 Address:d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9}]" + testing.go:149: server-180d44c5-13c8-1722-a927-e21b06e48e82: 2023-08-02T15:12:53.621Z [INFO] server-180d44c5-13c8-1722-a927-e21b06e48e82: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-180d44c5-13c8-1722-a927-e21b06e48e82 Address:180d44c5-13c8-1722-a927-e21b06e48e82} {Suffrage:Voter ID:server-2c5e301c-60c6-626a-ce4c-88387582e874 Address:2c5e301c-60c6-626a-ce4c-88387582e874} {Suffrage:Voter ID:server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9 Address:d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9}]" +testLoggerAdapter verbose: server-2c5e301c-60c6-626a-ce4c-88387582e874: 2023-08-02T15:12:53.621Z [INFO] server-2c5e301c-60c6-626a-ce4c-88387582e874: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-180d44c5-13c8-1722-a927-e21b06e48e82 Address:180d44c5-13c8-1722-a927-e21b06e48e82} {Suffrage:Voter ID:server-2c5e301c-60c6-626a-ce4c-88387582e874 Address:2c5e301c-60c6-626a-ce4c-88387582e874} {Suffrage:Voter ID:server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9 Address:d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9}]" + testing.go:149: server-2c5e301c-60c6-626a-ce4c-88387582e874: 2023-08-02T15:12:53.621Z [INFO] server-2c5e301c-60c6-626a-ce4c-88387582e874: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-180d44c5-13c8-1722-a927-e21b06e48e82 Address:180d44c5-13c8-1722-a927-e21b06e48e82} {Suffrage:Voter ID:server-2c5e301c-60c6-626a-ce4c-88387582e874 Address:2c5e301c-60c6-626a-ce4c-88387582e874} {Suffrage:Voter ID:server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9 Address:d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9}]" +testLoggerAdapter verbose: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.622Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-180d44c5-13c8-1722-a927-e21b06e48e82 Address:180d44c5-13c8-1722-a927-e21b06e48e82} {Suffrage:Voter ID:server-2c5e301c-60c6-626a-ce4c-88387582e874 Address:2c5e301c-60c6-626a-ce4c-88387582e874} {Suffrage:Voter ID:server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9 Address:d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9}]" + testing.go:149: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.622Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-180d44c5-13c8-1722-a927-e21b06e48e82 Address:180d44c5-13c8-1722-a927-e21b06e48e82} {Suffrage:Voter ID:server-2c5e301c-60c6-626a-ce4c-88387582e874 Address:2c5e301c-60c6-626a-ce4c-88387582e874} {Suffrage:Voter ID:server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9 Address:d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:53.622Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:12:53.622Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-180d44c5-13c8-1722-a927-e21b06e48e82: 2023-08-02T15:12:53.622Z [INFO] server-180d44c5-13c8-1722-a927-e21b06e48e82: entering follower state: follower="Node at 180d44c5-13c8-1722-a927-e21b06e48e82 [Follower]" leader= + testing.go:149: server-180d44c5-13c8-1722-a927-e21b06e48e82: 2023-08-02T15:12:53.622Z [INFO] server-180d44c5-13c8-1722-a927-e21b06e48e82: entering follower state: follower="Node at 180d44c5-13c8-1722-a927-e21b06e48e82 [Follower]" leader= +testLoggerAdapter verbose: server-2c5e301c-60c6-626a-ce4c-88387582e874: 2023-08-02T15:12:53.622Z [INFO] server-2c5e301c-60c6-626a-ce4c-88387582e874: entering follower state: follower="Node at 2c5e301c-60c6-626a-ce4c-88387582e874 [Follower]" leader= + testing.go:149: server-2c5e301c-60c6-626a-ce4c-88387582e874: 2023-08-02T15:12:53.622Z [INFO] server-2c5e301c-60c6-626a-ce4c-88387582e874: entering follower state: follower="Node at 2c5e301c-60c6-626a-ce4c-88387582e874 [Follower]" leader= +testLoggerAdapter verbose: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.622Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: entering follower state: follower="Node at d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9 [Follower]" leader= + testing.go:149: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.622Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: entering follower state: follower="Node at d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9 [Follower]" leader= +testLoggerAdapter verbose: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.678Z [WARN] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.678Z [WARN] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.678Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: entering candidate state: node="Node at d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9 [Candidate]" term=2 + testing.go:149: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.678Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: entering candidate state: node="Node at d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9 [Candidate]" term=2 +testLoggerAdapter verbose: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.678Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: election won: tally=2 + testing.go:149: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.678Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: election won: tally=2 +testLoggerAdapter verbose: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.678Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: entering leader state: leader="Node at d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9 [Leader]" + testing.go:149: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.678Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: entering leader state: leader="Node at d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9 [Leader]" +testLoggerAdapter verbose: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.678Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: added peer, starting replication: peer=server-180d44c5-13c8-1722-a927-e21b06e48e82 + testing.go:149: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.678Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: added peer, starting replication: peer=server-180d44c5-13c8-1722-a927-e21b06e48e82 +testLoggerAdapter verbose: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.678Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: added peer, starting replication: peer=server-2c5e301c-60c6-626a-ce4c-88387582e874 + testing.go:149: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.678Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: added peer, starting replication: peer=server-2c5e301c-60c6-626a-ce4c-88387582e874 +testLoggerAdapter verbose: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.678Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: pipelining replication: peer="{Voter server-2c5e301c-60c6-626a-ce4c-88387582e874 2c5e301c-60c6-626a-ce4c-88387582e874}" + testing.go:149: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.678Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: pipelining replication: peer="{Voter server-2c5e301c-60c6-626a-ce4c-88387582e874 2c5e301c-60c6-626a-ce4c-88387582e874}" +testLoggerAdapter verbose: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.678Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: pipelining replication: peer="{Voter server-180d44c5-13c8-1722-a927-e21b06e48e82 180d44c5-13c8-1722-a927-e21b06e48e82}" + testing.go:149: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.678Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: pipelining replication: peer="{Voter server-180d44c5-13c8-1722-a927-e21b06e48e82 180d44c5-13c8-1722-a927-e21b06e48e82}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:53.784Z [INFO] stable state for Leader reached at 2023-08-02 15:12:53.678509592 +0000 UTC m=+5.774445927 (1 nodes), 56.300257ms from start of poll, 56.860054ms from cluster start. Timeout at 2023-08-02 15:12:53.784065388 +0000 UTC m=+5.880001758, 105.555831ms after stability + testing.go:149: cluster: 2023-08-02T15:12:53.784Z [INFO] stable state for Leader reached at 2023-08-02 15:12:53.678509592 +0000 UTC m=+5.774445927 (1 nodes), 56.300257ms from start of poll, 56.860054ms from cluster start. Timeout at 2023-08-02 15:12:53.784065388 +0000 UTC m=+5.880001758, 105.555831ms after stability +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:53.784Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-08-02T15:12:53.784Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:53.889Z [INFO] stable state for Follower reached at 2023-08-02 15:12:53.78451474 +0000 UTC m=+5.880451076 (2 nodes), 1.627µs from start of poll, 162.865203ms from cluster start. Timeout at 2023-08-02 15:12:53.889579491 +0000 UTC m=+5.985515846, 105.06477ms after stability + testing.go:149: cluster: 2023-08-02T15:12:53.889Z [INFO] stable state for Follower reached at 2023-08-02 15:12:53.78451474 +0000 UTC m=+5.880451076 (2 nodes), 1.627µs from start of poll, 162.865203ms from cluster start. Timeout at 2023-08-02 15:12:53.889579491 +0000 UTC m=+5.985515846, 105.06477ms after stability +testLoggerAdapter verbose: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.889Z [ERROR] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: failed to pipeline appendEntries: peer="{Voter server-180d44c5-13c8-1722-a927-e21b06e48e82 180d44c5-13c8-1722-a927-e21b06e48e82}" error="append pipeline closed" + testing.go:149: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.889Z [ERROR] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: failed to pipeline appendEntries: peer="{Voter server-180d44c5-13c8-1722-a927-e21b06e48e82 180d44c5-13c8-1722-a927-e21b06e48e82}" error="append pipeline closed" +testLoggerAdapter verbose: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.889Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: aborting pipeline replication: peer="{Voter server-180d44c5-13c8-1722-a927-e21b06e48e82 180d44c5-13c8-1722-a927-e21b06e48e82}" + testing.go:149: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.889Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: aborting pipeline replication: peer="{Voter server-180d44c5-13c8-1722-a927-e21b06e48e82 180d44c5-13c8-1722-a927-e21b06e48e82}" +testLoggerAdapter verbose: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.889Z [ERROR] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: failed to appendEntries to: peer="{Voter server-180d44c5-13c8-1722-a927-e21b06e48e82 180d44c5-13c8-1722-a927-e21b06e48e82}" error="failed to connect to peer: 180d44c5-13c8-1722-a927-e21b06e48e82" + testing.go:149: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.889Z [ERROR] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: failed to appendEntries to: peer="{Voter server-180d44c5-13c8-1722-a927-e21b06e48e82 180d44c5-13c8-1722-a927-e21b06e48e82}" error="failed to connect to peer: 180d44c5-13c8-1722-a927-e21b06e48e82" raft_test.go:420: [INFO] Finished apply without behind follower -testLoggerAdapter verbose: server-183fd635-40c0-401c-fd23-bfa19f572e84: 2024-09-03T21:34:24.831Z [INFO] server-183fd635-40c0-401c-fd23-bfa19f572e84: pipelining replication: peer="{Voter server-6919d5b4-99da-c295-853b-28d9f25c2575 6919d5b4-99da-c295-853b-28d9f25c2575}" - testing.go:149: server-183fd635-40c0-401c-fd23-bfa19f572e84: 2024-09-03T21:34:24.831Z [INFO] server-183fd635-40c0-401c-fd23-bfa19f572e84: pipelining replication: peer="{Voter server-6919d5b4-99da-c295-853b-28d9f25c2575 6919d5b4-99da-c295-853b-28d9f25c2575}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:24.838Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:24.838Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:24.943Z [INFO] stable state for Leader reached at 2024-09-03 21:34:24.838570925 +0000 UTC m=+5.885228062 (1 nodes), 1.364µs from start of poll, 284.735121ms from cluster start. Timeout at 2024-09-03 21:34:24.943661505 +0000 UTC m=+5.990318655, 105.090593ms after stability - testing.go:149: cluster: 2024-09-03T21:34:24.943Z [INFO] stable state for Leader reached at 2024-09-03 21:34:24.838570925 +0000 UTC m=+5.885228062 (1 nodes), 1.364µs from start of poll, 284.735121ms from cluster start. Timeout at 2024-09-03 21:34:24.943661505 +0000 UTC m=+5.990318655, 105.090593ms after stability -testLoggerAdapter verbose: server-183fd635-40c0-401c-fd23-bfa19f572e84: 2024-09-03T21:34:24.943Z [INFO] server-183fd635-40c0-401c-fd23-bfa19f572e84: aborting pipeline replication: peer="{Voter server-c5a4b299-1163-212c-d947-2a6bcfa106dd c5a4b299-1163-212c-d947-2a6bcfa106dd}" - testing.go:149: server-183fd635-40c0-401c-fd23-bfa19f572e84: 2024-09-03T21:34:24.943Z [INFO] server-183fd635-40c0-401c-fd23-bfa19f572e84: aborting pipeline replication: peer="{Voter server-c5a4b299-1163-212c-d947-2a6bcfa106dd c5a4b299-1163-212c-d947-2a6bcfa106dd}" -testLoggerAdapter verbose: server-183fd635-40c0-401c-fd23-bfa19f572e84: 2024-09-03T21:34:24.943Z [INFO] server-183fd635-40c0-401c-fd23-bfa19f572e84: aborting pipeline replication: peer="{Voter server-6919d5b4-99da-c295-853b-28d9f25c2575 6919d5b4-99da-c295-853b-28d9f25c2575}" - testing.go:149: server-183fd635-40c0-401c-fd23-bfa19f572e84: 2024-09-03T21:34:24.943Z [INFO] server-183fd635-40c0-401c-fd23-bfa19f572e84: aborting pipeline replication: peer="{Voter server-6919d5b4-99da-c295-853b-28d9f25c2575 6919d5b4-99da-c295-853b-28d9f25c2575}" ---- PASS: TestRaft_BehindFollower (0.39s) +testLoggerAdapter verbose: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.900Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: pipelining replication: peer="{Voter server-180d44c5-13c8-1722-a927-e21b06e48e82 180d44c5-13c8-1722-a927-e21b06e48e82}" + testing.go:149: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.900Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: pipelining replication: peer="{Voter server-180d44c5-13c8-1722-a927-e21b06e48e82 180d44c5-13c8-1722-a927-e21b06e48e82}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:53.908Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:12:53.908Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:54.014Z [INFO] stable state for Leader reached at 2023-08-02 15:12:53.908642736 +0000 UTC m=+6.004579063 (1 nodes), 980ns from start of poll, 286.99319ms from cluster start. Timeout at 2023-08-02 15:12:54.01426461 +0000 UTC m=+6.110200937, 105.621874ms after stability + testing.go:149: cluster: 2023-08-02T15:12:54.014Z [INFO] stable state for Leader reached at 2023-08-02 15:12:53.908642736 +0000 UTC m=+6.004579063 (1 nodes), 980ns from start of poll, 286.99319ms from cluster start. Timeout at 2023-08-02 15:12:54.01426461 +0000 UTC m=+6.110200937, 105.621874ms after stability +testLoggerAdapter verbose: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:54.014Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: aborting pipeline replication: peer="{Voter server-2c5e301c-60c6-626a-ce4c-88387582e874 2c5e301c-60c6-626a-ce4c-88387582e874}" + testing.go:149: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:54.014Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: aborting pipeline replication: peer="{Voter server-2c5e301c-60c6-626a-ce4c-88387582e874 2c5e301c-60c6-626a-ce4c-88387582e874}" +testLoggerAdapter verbose: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:54.014Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: aborting pipeline replication: peer="{Voter server-180d44c5-13c8-1722-a927-e21b06e48e82 180d44c5-13c8-1722-a927-e21b06e48e82}" + testing.go:149: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:54.014Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: aborting pipeline replication: peer="{Voter server-180d44c5-13c8-1722-a927-e21b06e48e82 180d44c5-13c8-1722-a927-e21b06e48e82}" +--- PASS: TestRaft_BehindFollower (0.40s) === RUN TestRaft_ApplyNonLeader -testLoggerAdapter verbose: server-187e453b-9ff5-0d06-7b05-b08980196b5d: 2024-09-03T21:34:24.944Z [INFO] server-187e453b-9ff5-0d06-7b05-b08980196b5d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-187e453b-9ff5-0d06-7b05-b08980196b5d Address:187e453b-9ff5-0d06-7b05-b08980196b5d} {Suffrage:Voter ID:server-6a8f4387-45c3-9154-d7b9-2a6ff3e48410 Address:6a8f4387-45c3-9154-d7b9-2a6ff3e48410} {Suffrage:Voter ID:server-ced7294b-966b-4485-a376-7f515453cdcb Address:ced7294b-966b-4485-a376-7f515453cdcb}]" - testing.go:149: server-187e453b-9ff5-0d06-7b05-b08980196b5d: 2024-09-03T21:34:24.944Z [INFO] server-187e453b-9ff5-0d06-7b05-b08980196b5d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-187e453b-9ff5-0d06-7b05-b08980196b5d Address:187e453b-9ff5-0d06-7b05-b08980196b5d} {Suffrage:Voter ID:server-6a8f4387-45c3-9154-d7b9-2a6ff3e48410 Address:6a8f4387-45c3-9154-d7b9-2a6ff3e48410} {Suffrage:Voter ID:server-ced7294b-966b-4485-a376-7f515453cdcb Address:ced7294b-966b-4485-a376-7f515453cdcb}]" -testLoggerAdapter verbose: server-6a8f4387-45c3-9154-d7b9-2a6ff3e48410: 2024-09-03T21:34:24.944Z [INFO] server-6a8f4387-45c3-9154-d7b9-2a6ff3e48410: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-187e453b-9ff5-0d06-7b05-b08980196b5d Address:187e453b-9ff5-0d06-7b05-b08980196b5d} {Suffrage:Voter ID:server-6a8f4387-45c3-9154-d7b9-2a6ff3e48410 Address:6a8f4387-45c3-9154-d7b9-2a6ff3e48410} {Suffrage:Voter ID:server-ced7294b-966b-4485-a376-7f515453cdcb Address:ced7294b-966b-4485-a376-7f515453cdcb}]" - testing.go:149: server-6a8f4387-45c3-9154-d7b9-2a6ff3e48410: 2024-09-03T21:34:24.944Z [INFO] server-6a8f4387-45c3-9154-d7b9-2a6ff3e48410: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-187e453b-9ff5-0d06-7b05-b08980196b5d Address:187e453b-9ff5-0d06-7b05-b08980196b5d} {Suffrage:Voter ID:server-6a8f4387-45c3-9154-d7b9-2a6ff3e48410 Address:6a8f4387-45c3-9154-d7b9-2a6ff3e48410} {Suffrage:Voter ID:server-ced7294b-966b-4485-a376-7f515453cdcb Address:ced7294b-966b-4485-a376-7f515453cdcb}]" -testLoggerAdapter verbose: server-ced7294b-966b-4485-a376-7f515453cdcb: 2024-09-03T21:34:24.944Z [INFO] server-ced7294b-966b-4485-a376-7f515453cdcb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-187e453b-9ff5-0d06-7b05-b08980196b5d Address:187e453b-9ff5-0d06-7b05-b08980196b5d} {Suffrage:Voter ID:server-6a8f4387-45c3-9154-d7b9-2a6ff3e48410 Address:6a8f4387-45c3-9154-d7b9-2a6ff3e48410} {Suffrage:Voter ID:server-ced7294b-966b-4485-a376-7f515453cdcb Address:ced7294b-966b-4485-a376-7f515453cdcb}]" - testing.go:149: server-ced7294b-966b-4485-a376-7f515453cdcb: 2024-09-03T21:34:24.944Z [INFO] server-ced7294b-966b-4485-a376-7f515453cdcb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-187e453b-9ff5-0d06-7b05-b08980196b5d Address:187e453b-9ff5-0d06-7b05-b08980196b5d} {Suffrage:Voter ID:server-6a8f4387-45c3-9154-d7b9-2a6ff3e48410 Address:6a8f4387-45c3-9154-d7b9-2a6ff3e48410} {Suffrage:Voter ID:server-ced7294b-966b-4485-a376-7f515453cdcb Address:ced7294b-966b-4485-a376-7f515453cdcb}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:24.944Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:24.944Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-187e453b-9ff5-0d06-7b05-b08980196b5d: 2024-09-03T21:34:24.945Z [INFO] server-187e453b-9ff5-0d06-7b05-b08980196b5d: entering follower state: follower="Node at 187e453b-9ff5-0d06-7b05-b08980196b5d [Follower]" leader= - testing.go:149: server-187e453b-9ff5-0d06-7b05-b08980196b5d: 2024-09-03T21:34:24.945Z [INFO] server-187e453b-9ff5-0d06-7b05-b08980196b5d: entering follower state: follower="Node at 187e453b-9ff5-0d06-7b05-b08980196b5d [Follower]" leader= -testLoggerAdapter verbose: server-6a8f4387-45c3-9154-d7b9-2a6ff3e48410: 2024-09-03T21:34:24.945Z [INFO] server-6a8f4387-45c3-9154-d7b9-2a6ff3e48410: entering follower state: follower="Node at 6a8f4387-45c3-9154-d7b9-2a6ff3e48410 [Follower]" leader= - testing.go:149: server-6a8f4387-45c3-9154-d7b9-2a6ff3e48410: 2024-09-03T21:34:24.945Z [INFO] server-6a8f4387-45c3-9154-d7b9-2a6ff3e48410: entering follower state: follower="Node at 6a8f4387-45c3-9154-d7b9-2a6ff3e48410 [Follower]" leader= -testLoggerAdapter verbose: server-ced7294b-966b-4485-a376-7f515453cdcb: 2024-09-03T21:34:24.945Z [INFO] server-ced7294b-966b-4485-a376-7f515453cdcb: entering follower state: follower="Node at ced7294b-966b-4485-a376-7f515453cdcb [Follower]" leader= - testing.go:149: server-ced7294b-966b-4485-a376-7f515453cdcb: 2024-09-03T21:34:24.945Z [INFO] server-ced7294b-966b-4485-a376-7f515453cdcb: entering follower state: follower="Node at ced7294b-966b-4485-a376-7f515453cdcb [Follower]" leader= -testLoggerAdapter verbose: server-187e453b-9ff5-0d06-7b05-b08980196b5d: 2024-09-03T21:34:25.019Z [WARN] server-187e453b-9ff5-0d06-7b05-b08980196b5d: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-187e453b-9ff5-0d06-7b05-b08980196b5d: 2024-09-03T21:34:25.019Z [WARN] server-187e453b-9ff5-0d06-7b05-b08980196b5d: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-187e453b-9ff5-0d06-7b05-b08980196b5d: 2024-09-03T21:34:25.019Z [INFO] server-187e453b-9ff5-0d06-7b05-b08980196b5d: entering candidate state: node="Node at 187e453b-9ff5-0d06-7b05-b08980196b5d [Candidate]" term=2 - testing.go:149: server-187e453b-9ff5-0d06-7b05-b08980196b5d: 2024-09-03T21:34:25.019Z [INFO] server-187e453b-9ff5-0d06-7b05-b08980196b5d: entering candidate state: node="Node at 187e453b-9ff5-0d06-7b05-b08980196b5d [Candidate]" term=2 -testLoggerAdapter verbose: server-187e453b-9ff5-0d06-7b05-b08980196b5d: 2024-09-03T21:34:25.020Z [INFO] server-187e453b-9ff5-0d06-7b05-b08980196b5d: election won: tally=2 - testing.go:149: server-187e453b-9ff5-0d06-7b05-b08980196b5d: 2024-09-03T21:34:25.020Z [INFO] server-187e453b-9ff5-0d06-7b05-b08980196b5d: election won: tally=2 -testLoggerAdapter verbose: server-187e453b-9ff5-0d06-7b05-b08980196b5d: 2024-09-03T21:34:25.020Z [INFO] server-187e453b-9ff5-0d06-7b05-b08980196b5d: entering leader state: leader="Node at 187e453b-9ff5-0d06-7b05-b08980196b5d [Leader]" - testing.go:149: server-187e453b-9ff5-0d06-7b05-b08980196b5d: 2024-09-03T21:34:25.020Z [INFO] server-187e453b-9ff5-0d06-7b05-b08980196b5d: entering leader state: leader="Node at 187e453b-9ff5-0d06-7b05-b08980196b5d [Leader]" -testLoggerAdapter verbose: server-187e453b-9ff5-0d06-7b05-b08980196b5d: 2024-09-03T21:34:25.020Z [INFO] server-187e453b-9ff5-0d06-7b05-b08980196b5d: added peer, starting replication: peer=server-6a8f4387-45c3-9154-d7b9-2a6ff3e48410 - testing.go:149: server-187e453b-9ff5-0d06-7b05-b08980196b5d: 2024-09-03T21:34:25.020Z [INFO] server-187e453b-9ff5-0d06-7b05-b08980196b5d: added peer, starting replication: peer=server-6a8f4387-45c3-9154-d7b9-2a6ff3e48410 -testLoggerAdapter verbose: server-187e453b-9ff5-0d06-7b05-b08980196b5d: 2024-09-03T21:34:25.020Z [INFO] server-187e453b-9ff5-0d06-7b05-b08980196b5d: added peer, starting replication: peer=server-ced7294b-966b-4485-a376-7f515453cdcb - testing.go:149: server-187e453b-9ff5-0d06-7b05-b08980196b5d: 2024-09-03T21:34:25.020Z [INFO] server-187e453b-9ff5-0d06-7b05-b08980196b5d: added peer, starting replication: peer=server-ced7294b-966b-4485-a376-7f515453cdcb -testLoggerAdapter verbose: server-187e453b-9ff5-0d06-7b05-b08980196b5d: 2024-09-03T21:34:25.020Z [INFO] server-187e453b-9ff5-0d06-7b05-b08980196b5d: pipelining replication: peer="{Voter server-ced7294b-966b-4485-a376-7f515453cdcb ced7294b-966b-4485-a376-7f515453cdcb}" - testing.go:149: server-187e453b-9ff5-0d06-7b05-b08980196b5d: 2024-09-03T21:34:25.020Z [INFO] server-187e453b-9ff5-0d06-7b05-b08980196b5d: pipelining replication: peer="{Voter server-ced7294b-966b-4485-a376-7f515453cdcb ced7294b-966b-4485-a376-7f515453cdcb}" -testLoggerAdapter verbose: server-187e453b-9ff5-0d06-7b05-b08980196b5d: 2024-09-03T21:34:25.020Z [INFO] server-187e453b-9ff5-0d06-7b05-b08980196b5d: pipelining replication: peer="{Voter server-6a8f4387-45c3-9154-d7b9-2a6ff3e48410 6a8f4387-45c3-9154-d7b9-2a6ff3e48410}" - testing.go:149: server-187e453b-9ff5-0d06-7b05-b08980196b5d: 2024-09-03T21:34:25.020Z [INFO] server-187e453b-9ff5-0d06-7b05-b08980196b5d: pipelining replication: peer="{Voter server-6a8f4387-45c3-9154-d7b9-2a6ff3e48410 6a8f4387-45c3-9154-d7b9-2a6ff3e48410}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:25.125Z [INFO] stable state for Leader reached at 2024-09-03 21:34:25.02050695 +0000 UTC m=+6.067164093 (1 nodes), 75.487626ms from start of poll, 75.909145ms from cluster start. Timeout at 2024-09-03 21:34:25.125602711 +0000 UTC m=+6.172259842, 105.095749ms after stability - testing.go:149: cluster: 2024-09-03T21:34:25.125Z [INFO] stable state for Leader reached at 2024-09-03 21:34:25.02050695 +0000 UTC m=+6.067164093 (1 nodes), 75.487626ms from start of poll, 75.909145ms from cluster start. Timeout at 2024-09-03 21:34:25.125602711 +0000 UTC m=+6.172259842, 105.095749ms after stability -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:25.125Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-09-03T21:34:25.125Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:25.230Z [INFO] stable state for Follower reached at 2024-09-03 21:34:25.125730844 +0000 UTC m=+6.172387995 (2 nodes), 1.103µs from start of poll, 181.133047ms from cluster start. Timeout at 2024-09-03 21:34:25.230921972 +0000 UTC m=+6.277579194, 105.191199ms after stability - testing.go:149: cluster: 2024-09-03T21:34:25.230Z [INFO] stable state for Follower reached at 2024-09-03 21:34:25.125730844 +0000 UTC m=+6.172387995 (2 nodes), 1.103µs from start of poll, 181.133047ms from cluster start. Timeout at 2024-09-03 21:34:25.230921972 +0000 UTC m=+6.277579194, 105.191199ms after stability -testLoggerAdapter verbose: server-187e453b-9ff5-0d06-7b05-b08980196b5d: 2024-09-03T21:34:25.231Z [INFO] server-187e453b-9ff5-0d06-7b05-b08980196b5d: aborting pipeline replication: peer="{Voter server-ced7294b-966b-4485-a376-7f515453cdcb ced7294b-966b-4485-a376-7f515453cdcb}" - testing.go:149: server-187e453b-9ff5-0d06-7b05-b08980196b5d: 2024-09-03T21:34:25.231Z [INFO] server-187e453b-9ff5-0d06-7b05-b08980196b5d: aborting pipeline replication: peer="{Voter server-ced7294b-966b-4485-a376-7f515453cdcb ced7294b-966b-4485-a376-7f515453cdcb}" -testLoggerAdapter verbose: server-187e453b-9ff5-0d06-7b05-b08980196b5d: 2024-09-03T21:34:25.231Z [INFO] server-187e453b-9ff5-0d06-7b05-b08980196b5d: aborting pipeline replication: peer="{Voter server-6a8f4387-45c3-9154-d7b9-2a6ff3e48410 6a8f4387-45c3-9154-d7b9-2a6ff3e48410}" - testing.go:149: server-187e453b-9ff5-0d06-7b05-b08980196b5d: 2024-09-03T21:34:25.231Z [INFO] server-187e453b-9ff5-0d06-7b05-b08980196b5d: aborting pipeline replication: peer="{Voter server-6a8f4387-45c3-9154-d7b9-2a6ff3e48410 6a8f4387-45c3-9154-d7b9-2a6ff3e48410}" ---- PASS: TestRaft_ApplyNonLeader (0.29s) +testLoggerAdapter verbose: server-a5eb522d-6f8d-744e-6cbc-1743e3136578: 2023-08-02T15:12:54.018Z [INFO] server-a5eb522d-6f8d-744e-6cbc-1743e3136578: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a5eb522d-6f8d-744e-6cbc-1743e3136578 Address:a5eb522d-6f8d-744e-6cbc-1743e3136578} {Suffrage:Voter ID:server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4 Address:24fa25a3-0bd1-da1e-da8f-2325522ee3f4} {Suffrage:Voter ID:server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1 Address:cf2e5c9d-5e3e-7ece-b910-4964c0c364e1}]" + testing.go:149: server-a5eb522d-6f8d-744e-6cbc-1743e3136578: 2023-08-02T15:12:54.018Z [INFO] server-a5eb522d-6f8d-744e-6cbc-1743e3136578: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a5eb522d-6f8d-744e-6cbc-1743e3136578 Address:a5eb522d-6f8d-744e-6cbc-1743e3136578} {Suffrage:Voter ID:server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4 Address:24fa25a3-0bd1-da1e-da8f-2325522ee3f4} {Suffrage:Voter ID:server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1 Address:cf2e5c9d-5e3e-7ece-b910-4964c0c364e1}]" +testLoggerAdapter verbose: server-a5eb522d-6f8d-744e-6cbc-1743e3136578: 2023-08-02T15:12:54.018Z [INFO] server-a5eb522d-6f8d-744e-6cbc-1743e3136578: entering follower state: follower="Node at a5eb522d-6f8d-744e-6cbc-1743e3136578 [Follower]" leader= + testing.go:149: server-a5eb522d-6f8d-744e-6cbc-1743e3136578: 2023-08-02T15:12:54.018Z [INFO] server-a5eb522d-6f8d-744e-6cbc-1743e3136578: entering follower state: follower="Node at a5eb522d-6f8d-744e-6cbc-1743e3136578 [Follower]" leader= +testLoggerAdapter verbose: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.018Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a5eb522d-6f8d-744e-6cbc-1743e3136578 Address:a5eb522d-6f8d-744e-6cbc-1743e3136578} {Suffrage:Voter ID:server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4 Address:24fa25a3-0bd1-da1e-da8f-2325522ee3f4} {Suffrage:Voter ID:server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1 Address:cf2e5c9d-5e3e-7ece-b910-4964c0c364e1}]" + testing.go:149: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.018Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a5eb522d-6f8d-744e-6cbc-1743e3136578 Address:a5eb522d-6f8d-744e-6cbc-1743e3136578} {Suffrage:Voter ID:server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4 Address:24fa25a3-0bd1-da1e-da8f-2325522ee3f4} {Suffrage:Voter ID:server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1 Address:cf2e5c9d-5e3e-7ece-b910-4964c0c364e1}]" +testLoggerAdapter verbose: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.018Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: entering follower state: follower="Node at 24fa25a3-0bd1-da1e-da8f-2325522ee3f4 [Follower]" leader= + testing.go:149: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.018Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: entering follower state: follower="Node at 24fa25a3-0bd1-da1e-da8f-2325522ee3f4 [Follower]" leader= +testLoggerAdapter verbose: server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1: 2023-08-02T15:12:54.018Z [INFO] server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a5eb522d-6f8d-744e-6cbc-1743e3136578 Address:a5eb522d-6f8d-744e-6cbc-1743e3136578} {Suffrage:Voter ID:server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4 Address:24fa25a3-0bd1-da1e-da8f-2325522ee3f4} {Suffrage:Voter ID:server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1 Address:cf2e5c9d-5e3e-7ece-b910-4964c0c364e1}]" + testing.go:149: server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1: 2023-08-02T15:12:54.018Z [INFO] server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a5eb522d-6f8d-744e-6cbc-1743e3136578 Address:a5eb522d-6f8d-744e-6cbc-1743e3136578} {Suffrage:Voter ID:server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4 Address:24fa25a3-0bd1-da1e-da8f-2325522ee3f4} {Suffrage:Voter ID:server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1 Address:cf2e5c9d-5e3e-7ece-b910-4964c0c364e1}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:54.018Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:12:54.018Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1: 2023-08-02T15:12:54.018Z [INFO] server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1: entering follower state: follower="Node at cf2e5c9d-5e3e-7ece-b910-4964c0c364e1 [Follower]" leader= + testing.go:149: server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1: 2023-08-02T15:12:54.018Z [INFO] server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1: entering follower state: follower="Node at cf2e5c9d-5e3e-7ece-b910-4964c0c364e1 [Follower]" leader= +testLoggerAdapter verbose: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.080Z [WARN] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.080Z [WARN] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.080Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: entering candidate state: node="Node at 24fa25a3-0bd1-da1e-da8f-2325522ee3f4 [Candidate]" term=2 + testing.go:149: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.080Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: entering candidate state: node="Node at 24fa25a3-0bd1-da1e-da8f-2325522ee3f4 [Candidate]" term=2 +testLoggerAdapter verbose: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.080Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: election won: tally=2 + testing.go:149: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.080Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: election won: tally=2 +testLoggerAdapter verbose: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.080Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: entering leader state: leader="Node at 24fa25a3-0bd1-da1e-da8f-2325522ee3f4 [Leader]" + testing.go:149: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.080Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: entering leader state: leader="Node at 24fa25a3-0bd1-da1e-da8f-2325522ee3f4 [Leader]" +testLoggerAdapter verbose: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.080Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: added peer, starting replication: peer=server-a5eb522d-6f8d-744e-6cbc-1743e3136578 + testing.go:149: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.080Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: added peer, starting replication: peer=server-a5eb522d-6f8d-744e-6cbc-1743e3136578 +testLoggerAdapter verbose: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.080Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: added peer, starting replication: peer=server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1 + testing.go:149: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.080Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: added peer, starting replication: peer=server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1 +testLoggerAdapter verbose: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.080Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: pipelining replication: peer="{Voter server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1 cf2e5c9d-5e3e-7ece-b910-4964c0c364e1}" + testing.go:149: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.080Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: pipelining replication: peer="{Voter server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1 cf2e5c9d-5e3e-7ece-b910-4964c0c364e1}" +testLoggerAdapter verbose: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.080Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: pipelining replication: peer="{Voter server-a5eb522d-6f8d-744e-6cbc-1743e3136578 a5eb522d-6f8d-744e-6cbc-1743e3136578}" + testing.go:149: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.080Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: pipelining replication: peer="{Voter server-a5eb522d-6f8d-744e-6cbc-1743e3136578 a5eb522d-6f8d-744e-6cbc-1743e3136578}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:54.185Z [INFO] stable state for Leader reached at 2023-08-02 15:12:54.080785978 +0000 UTC m=+6.176722325 (1 nodes), 61.9597ms from start of poll, 62.977816ms from cluster start. Timeout at 2023-08-02 15:12:54.185876214 +0000 UTC m=+6.281812539, 105.090214ms after stability + testing.go:149: cluster: 2023-08-02T15:12:54.185Z [INFO] stable state for Leader reached at 2023-08-02 15:12:54.080785978 +0000 UTC m=+6.176722325 (1 nodes), 61.9597ms from start of poll, 62.977816ms from cluster start. Timeout at 2023-08-02 15:12:54.185876214 +0000 UTC m=+6.281812539, 105.090214ms after stability +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:54.186Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-08-02T15:12:54.186Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:54.291Z [INFO] stable state for Follower reached at 2023-08-02 15:12:54.186048148 +0000 UTC m=+6.281984476 (2 nodes), 2.491µs from start of poll, 168.239967ms from cluster start. Timeout at 2023-08-02 15:12:54.291147839 +0000 UTC m=+6.387084175, 105.099699ms after stability + testing.go:149: cluster: 2023-08-02T15:12:54.291Z [INFO] stable state for Follower reached at 2023-08-02 15:12:54.186048148 +0000 UTC m=+6.281984476 (2 nodes), 2.491µs from start of poll, 168.239967ms from cluster start. Timeout at 2023-08-02 15:12:54.291147839 +0000 UTC m=+6.387084175, 105.099699ms after stability +testLoggerAdapter verbose: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.291Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: aborting pipeline replication: peer="{Voter server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1 cf2e5c9d-5e3e-7ece-b910-4964c0c364e1}" + testing.go:149: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.291Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: aborting pipeline replication: peer="{Voter server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1 cf2e5c9d-5e3e-7ece-b910-4964c0c364e1}" +testLoggerAdapter verbose: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.291Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: aborting pipeline replication: peer="{Voter server-a5eb522d-6f8d-744e-6cbc-1743e3136578 a5eb522d-6f8d-744e-6cbc-1743e3136578}" + testing.go:149: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.291Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: aborting pipeline replication: peer="{Voter server-a5eb522d-6f8d-744e-6cbc-1743e3136578 a5eb522d-6f8d-744e-6cbc-1743e3136578}" +--- PASS: TestRaft_ApplyNonLeader (0.27s) === RUN TestRaft_ApplyConcurrent -testLoggerAdapter verbose: server-e6d8107e-dc4a-128a-933b-71369f26a417: 2024-09-03T21:34:25.232Z [INFO] server-e6d8107e-dc4a-128a-933b-71369f26a417: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e6d8107e-dc4a-128a-933b-71369f26a417 Address:e6d8107e-dc4a-128a-933b-71369f26a417} {Suffrage:Voter ID:server-ad8608ac-6d76-4c33-1aa0-1e2879f4cacf Address:ad8608ac-6d76-4c33-1aa0-1e2879f4cacf} {Suffrage:Voter ID:server-cb0573de-37e2-bde4-e54f-8ff6c051c813 Address:cb0573de-37e2-bde4-e54f-8ff6c051c813}]" - testing.go:149: server-e6d8107e-dc4a-128a-933b-71369f26a417: 2024-09-03T21:34:25.232Z [INFO] server-e6d8107e-dc4a-128a-933b-71369f26a417: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e6d8107e-dc4a-128a-933b-71369f26a417 Address:e6d8107e-dc4a-128a-933b-71369f26a417} {Suffrage:Voter ID:server-ad8608ac-6d76-4c33-1aa0-1e2879f4cacf Address:ad8608ac-6d76-4c33-1aa0-1e2879f4cacf} {Suffrage:Voter ID:server-cb0573de-37e2-bde4-e54f-8ff6c051c813 Address:cb0573de-37e2-bde4-e54f-8ff6c051c813}]" -testLoggerAdapter verbose: server-ad8608ac-6d76-4c33-1aa0-1e2879f4cacf: 2024-09-03T21:34:25.232Z [INFO] server-ad8608ac-6d76-4c33-1aa0-1e2879f4cacf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e6d8107e-dc4a-128a-933b-71369f26a417 Address:e6d8107e-dc4a-128a-933b-71369f26a417} {Suffrage:Voter ID:server-ad8608ac-6d76-4c33-1aa0-1e2879f4cacf Address:ad8608ac-6d76-4c33-1aa0-1e2879f4cacf} {Suffrage:Voter ID:server-cb0573de-37e2-bde4-e54f-8ff6c051c813 Address:cb0573de-37e2-bde4-e54f-8ff6c051c813}]" - testing.go:149: server-ad8608ac-6d76-4c33-1aa0-1e2879f4cacf: 2024-09-03T21:34:25.232Z [INFO] server-ad8608ac-6d76-4c33-1aa0-1e2879f4cacf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e6d8107e-dc4a-128a-933b-71369f26a417 Address:e6d8107e-dc4a-128a-933b-71369f26a417} {Suffrage:Voter ID:server-ad8608ac-6d76-4c33-1aa0-1e2879f4cacf Address:ad8608ac-6d76-4c33-1aa0-1e2879f4cacf} {Suffrage:Voter ID:server-cb0573de-37e2-bde4-e54f-8ff6c051c813 Address:cb0573de-37e2-bde4-e54f-8ff6c051c813}]" -testLoggerAdapter verbose: server-cb0573de-37e2-bde4-e54f-8ff6c051c813: 2024-09-03T21:34:25.232Z [INFO] server-cb0573de-37e2-bde4-e54f-8ff6c051c813: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e6d8107e-dc4a-128a-933b-71369f26a417 Address:e6d8107e-dc4a-128a-933b-71369f26a417} {Suffrage:Voter ID:server-ad8608ac-6d76-4c33-1aa0-1e2879f4cacf Address:ad8608ac-6d76-4c33-1aa0-1e2879f4cacf} {Suffrage:Voter ID:server-cb0573de-37e2-bde4-e54f-8ff6c051c813 Address:cb0573de-37e2-bde4-e54f-8ff6c051c813}]" - testing.go:149: server-cb0573de-37e2-bde4-e54f-8ff6c051c813: 2024-09-03T21:34:25.232Z [INFO] server-cb0573de-37e2-bde4-e54f-8ff6c051c813: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e6d8107e-dc4a-128a-933b-71369f26a417 Address:e6d8107e-dc4a-128a-933b-71369f26a417} {Suffrage:Voter ID:server-ad8608ac-6d76-4c33-1aa0-1e2879f4cacf Address:ad8608ac-6d76-4c33-1aa0-1e2879f4cacf} {Suffrage:Voter ID:server-cb0573de-37e2-bde4-e54f-8ff6c051c813 Address:cb0573de-37e2-bde4-e54f-8ff6c051c813}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:25.232Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-ad8608ac-6d76-4c33-1aa0-1e2879f4cacf: 2024-09-03T21:34:25.232Z [INFO] server-ad8608ac-6d76-4c33-1aa0-1e2879f4cacf: entering follower state: follower="Node at ad8608ac-6d76-4c33-1aa0-1e2879f4cacf [Follower]" leader= - testing.go:149: cluster: 2024-09-03T21:34:25.232Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-cb0573de-37e2-bde4-e54f-8ff6c051c813: 2024-09-03T21:34:25.232Z [INFO] server-cb0573de-37e2-bde4-e54f-8ff6c051c813: entering follower state: follower="Node at cb0573de-37e2-bde4-e54f-8ff6c051c813 [Follower]" leader= -testLoggerAdapter verbose: server-e6d8107e-dc4a-128a-933b-71369f26a417: 2024-09-03T21:34:25.232Z [INFO] server-e6d8107e-dc4a-128a-933b-71369f26a417: entering follower state: follower="Node at e6d8107e-dc4a-128a-933b-71369f26a417 [Follower]" leader= - testing.go:149: server-cb0573de-37e2-bde4-e54f-8ff6c051c813: 2024-09-03T21:34:25.232Z [INFO] server-cb0573de-37e2-bde4-e54f-8ff6c051c813: entering follower state: follower="Node at cb0573de-37e2-bde4-e54f-8ff6c051c813 [Follower]" leader= - testing.go:149: server-e6d8107e-dc4a-128a-933b-71369f26a417: 2024-09-03T21:34:25.232Z [INFO] server-e6d8107e-dc4a-128a-933b-71369f26a417: entering follower state: follower="Node at e6d8107e-dc4a-128a-933b-71369f26a417 [Follower]" leader= - testing.go:149: server-ad8608ac-6d76-4c33-1aa0-1e2879f4cacf: 2024-09-03T21:34:25.232Z [INFO] server-ad8608ac-6d76-4c33-1aa0-1e2879f4cacf: entering follower state: follower="Node at ad8608ac-6d76-4c33-1aa0-1e2879f4cacf [Follower]" leader= -testLoggerAdapter verbose: server-e6d8107e-dc4a-128a-933b-71369f26a417: 2024-09-03T21:34:25.395Z [WARN] server-e6d8107e-dc4a-128a-933b-71369f26a417: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-e6d8107e-dc4a-128a-933b-71369f26a417: 2024-09-03T21:34:25.395Z [WARN] server-e6d8107e-dc4a-128a-933b-71369f26a417: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-e6d8107e-dc4a-128a-933b-71369f26a417: 2024-09-03T21:34:25.395Z [INFO] server-e6d8107e-dc4a-128a-933b-71369f26a417: entering candidate state: node="Node at e6d8107e-dc4a-128a-933b-71369f26a417 [Candidate]" term=2 - testing.go:149: server-e6d8107e-dc4a-128a-933b-71369f26a417: 2024-09-03T21:34:25.395Z [INFO] server-e6d8107e-dc4a-128a-933b-71369f26a417: entering candidate state: node="Node at e6d8107e-dc4a-128a-933b-71369f26a417 [Candidate]" term=2 -testLoggerAdapter verbose: server-e6d8107e-dc4a-128a-933b-71369f26a417: 2024-09-03T21:34:25.395Z [INFO] server-e6d8107e-dc4a-128a-933b-71369f26a417: election won: tally=2 - testing.go:149: server-e6d8107e-dc4a-128a-933b-71369f26a417: 2024-09-03T21:34:25.395Z [INFO] server-e6d8107e-dc4a-128a-933b-71369f26a417: election won: tally=2 -testLoggerAdapter verbose: server-e6d8107e-dc4a-128a-933b-71369f26a417: 2024-09-03T21:34:25.396Z [INFO] server-e6d8107e-dc4a-128a-933b-71369f26a417: entering leader state: leader="Node at e6d8107e-dc4a-128a-933b-71369f26a417 [Leader]" - testing.go:149: server-e6d8107e-dc4a-128a-933b-71369f26a417: 2024-09-03T21:34:25.396Z [INFO] server-e6d8107e-dc4a-128a-933b-71369f26a417: entering leader state: leader="Node at e6d8107e-dc4a-128a-933b-71369f26a417 [Leader]" -testLoggerAdapter verbose: server-e6d8107e-dc4a-128a-933b-71369f26a417: 2024-09-03T21:34:25.396Z [INFO] server-e6d8107e-dc4a-128a-933b-71369f26a417: added peer, starting replication: peer=server-ad8608ac-6d76-4c33-1aa0-1e2879f4cacf - testing.go:149: server-e6d8107e-dc4a-128a-933b-71369f26a417: 2024-09-03T21:34:25.396Z [INFO] server-e6d8107e-dc4a-128a-933b-71369f26a417: added peer, starting replication: peer=server-ad8608ac-6d76-4c33-1aa0-1e2879f4cacf -testLoggerAdapter verbose: server-e6d8107e-dc4a-128a-933b-71369f26a417: 2024-09-03T21:34:25.396Z [INFO] server-e6d8107e-dc4a-128a-933b-71369f26a417: added peer, starting replication: peer=server-cb0573de-37e2-bde4-e54f-8ff6c051c813 - testing.go:149: server-e6d8107e-dc4a-128a-933b-71369f26a417: 2024-09-03T21:34:25.396Z [INFO] server-e6d8107e-dc4a-128a-933b-71369f26a417: added peer, starting replication: peer=server-cb0573de-37e2-bde4-e54f-8ff6c051c813 -testLoggerAdapter verbose: server-e6d8107e-dc4a-128a-933b-71369f26a417: 2024-09-03T21:34:25.396Z [INFO] server-e6d8107e-dc4a-128a-933b-71369f26a417: pipelining replication: peer="{Voter server-cb0573de-37e2-bde4-e54f-8ff6c051c813 cb0573de-37e2-bde4-e54f-8ff6c051c813}" - testing.go:149: server-e6d8107e-dc4a-128a-933b-71369f26a417: 2024-09-03T21:34:25.396Z [INFO] server-e6d8107e-dc4a-128a-933b-71369f26a417: pipelining replication: peer="{Voter server-cb0573de-37e2-bde4-e54f-8ff6c051c813 cb0573de-37e2-bde4-e54f-8ff6c051c813}" -testLoggerAdapter verbose: server-ad8608ac-6d76-4c33-1aa0-1e2879f4cacf: 2024-09-03T21:34:25.396Z [WARN] server-ad8608ac-6d76-4c33-1aa0-1e2879f4cacf: rejecting vote request since our last term is greater: candidate=e6d8107e-dc4a-128a-933b-71369f26a417 last-term=2 last-candidate-term=1 - testing.go:149: server-ad8608ac-6d76-4c33-1aa0-1e2879f4cacf: 2024-09-03T21:34:25.396Z [WARN] server-ad8608ac-6d76-4c33-1aa0-1e2879f4cacf: rejecting vote request since our last term is greater: candidate=e6d8107e-dc4a-128a-933b-71369f26a417 last-term=2 last-candidate-term=1 -testLoggerAdapter verbose: server-e6d8107e-dc4a-128a-933b-71369f26a417: 2024-09-03T21:34:25.396Z [INFO] server-e6d8107e-dc4a-128a-933b-71369f26a417: pipelining replication: peer="{Voter server-ad8608ac-6d76-4c33-1aa0-1e2879f4cacf ad8608ac-6d76-4c33-1aa0-1e2879f4cacf}" - testing.go:149: server-e6d8107e-dc4a-128a-933b-71369f26a417: 2024-09-03T21:34:25.396Z [INFO] server-e6d8107e-dc4a-128a-933b-71369f26a417: pipelining replication: peer="{Voter server-ad8608ac-6d76-4c33-1aa0-1e2879f4cacf ad8608ac-6d76-4c33-1aa0-1e2879f4cacf}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:25.601Z [INFO] stable state for Leader reached at 2024-09-03 21:34:25.396384566 +0000 UTC m=+6.443041703 (1 nodes), 163.393957ms from start of poll, 164.098258ms from cluster start. Timeout at 2024-09-03 21:34:25.601389027 +0000 UTC m=+6.648046166, 205.004463ms after stability - testing.go:149: cluster: 2024-09-03T21:34:25.601Z [INFO] stable state for Leader reached at 2024-09-03 21:34:25.396384566 +0000 UTC m=+6.443041703 (1 nodes), 163.393957ms from start of poll, 164.098258ms from cluster start. Timeout at 2024-09-03 21:34:25.601389027 +0000 UTC m=+6.648046166, 205.004463ms after stability -testLoggerAdapter verbose: server-e6d8107e-dc4a-128a-933b-71369f26a417: 2024-09-03T21:34:25.614Z [INFO] server-e6d8107e-dc4a-128a-933b-71369f26a417: aborting pipeline replication: peer="{Voter server-cb0573de-37e2-bde4-e54f-8ff6c051c813 cb0573de-37e2-bde4-e54f-8ff6c051c813}" - testing.go:149: server-e6d8107e-dc4a-128a-933b-71369f26a417: 2024-09-03T21:34:25.614Z [INFO] server-e6d8107e-dc4a-128a-933b-71369f26a417: aborting pipeline replication: peer="{Voter server-cb0573de-37e2-bde4-e54f-8ff6c051c813 cb0573de-37e2-bde4-e54f-8ff6c051c813}" -testLoggerAdapter verbose: server-e6d8107e-dc4a-128a-933b-71369f26a417: 2024-09-03T21:34:25.614Z [INFO] server-e6d8107e-dc4a-128a-933b-71369f26a417: aborting pipeline replication: peer="{Voter server-ad8608ac-6d76-4c33-1aa0-1e2879f4cacf ad8608ac-6d76-4c33-1aa0-1e2879f4cacf}" - testing.go:149: server-e6d8107e-dc4a-128a-933b-71369f26a417: 2024-09-03T21:34:25.614Z [INFO] server-e6d8107e-dc4a-128a-933b-71369f26a417: aborting pipeline replication: peer="{Voter server-ad8608ac-6d76-4c33-1aa0-1e2879f4cacf ad8608ac-6d76-4c33-1aa0-1e2879f4cacf}" ---- PASS: TestRaft_ApplyConcurrent (0.38s) +testLoggerAdapter verbose: server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67: 2023-08-02T15:12:54.292Z [INFO] server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67 Address:e6f973fe-395d-dd10-ecc1-65aa2c5e7d67} {Suffrage:Voter ID:server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e Address:dc4de7ca-3447-cbf4-6782-8826c8c93c9e} {Suffrage:Voter ID:server-e3107dc3-d161-48d5-9aa6-82092b13434f Address:e3107dc3-d161-48d5-9aa6-82092b13434f}]" + testing.go:149: server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67: 2023-08-02T15:12:54.292Z [INFO] server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67 Address:e6f973fe-395d-dd10-ecc1-65aa2c5e7d67} {Suffrage:Voter ID:server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e Address:dc4de7ca-3447-cbf4-6782-8826c8c93c9e} {Suffrage:Voter ID:server-e3107dc3-d161-48d5-9aa6-82092b13434f Address:e3107dc3-d161-48d5-9aa6-82092b13434f}]" +testLoggerAdapter verbose: server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e: 2023-08-02T15:12:54.292Z [INFO] server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67 Address:e6f973fe-395d-dd10-ecc1-65aa2c5e7d67} {Suffrage:Voter ID:server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e Address:dc4de7ca-3447-cbf4-6782-8826c8c93c9e} {Suffrage:Voter ID:server-e3107dc3-d161-48d5-9aa6-82092b13434f Address:e3107dc3-d161-48d5-9aa6-82092b13434f}]" + testing.go:149: server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e: 2023-08-02T15:12:54.292Z [INFO] server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67 Address:e6f973fe-395d-dd10-ecc1-65aa2c5e7d67} {Suffrage:Voter ID:server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e Address:dc4de7ca-3447-cbf4-6782-8826c8c93c9e} {Suffrage:Voter ID:server-e3107dc3-d161-48d5-9aa6-82092b13434f Address:e3107dc3-d161-48d5-9aa6-82092b13434f}]" +testLoggerAdapter verbose: server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67: 2023-08-02T15:12:54.292Z [INFO] server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67: entering follower state: follower="Node at e6f973fe-395d-dd10-ecc1-65aa2c5e7d67 [Follower]" leader= +testLoggerAdapter verbose: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.292Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67 Address:e6f973fe-395d-dd10-ecc1-65aa2c5e7d67} {Suffrage:Voter ID:server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e Address:dc4de7ca-3447-cbf4-6782-8826c8c93c9e} {Suffrage:Voter ID:server-e3107dc3-d161-48d5-9aa6-82092b13434f Address:e3107dc3-d161-48d5-9aa6-82092b13434f}]" + testing.go:149: server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67: 2023-08-02T15:12:54.292Z [INFO] server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67: entering follower state: follower="Node at e6f973fe-395d-dd10-ecc1-65aa2c5e7d67 [Follower]" leader= + testing.go:149: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.292Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67 Address:e6f973fe-395d-dd10-ecc1-65aa2c5e7d67} {Suffrage:Voter ID:server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e Address:dc4de7ca-3447-cbf4-6782-8826c8c93c9e} {Suffrage:Voter ID:server-e3107dc3-d161-48d5-9aa6-82092b13434f Address:e3107dc3-d161-48d5-9aa6-82092b13434f}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:54.292Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:12:54.292Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.292Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: entering follower state: follower="Node at e3107dc3-d161-48d5-9aa6-82092b13434f [Follower]" leader= + testing.go:149: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.292Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: entering follower state: follower="Node at e3107dc3-d161-48d5-9aa6-82092b13434f [Follower]" leader= +testLoggerAdapter verbose: server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e: 2023-08-02T15:12:54.292Z [INFO] server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e: entering follower state: follower="Node at dc4de7ca-3447-cbf4-6782-8826c8c93c9e [Follower]" leader= + testing.go:149: server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e: 2023-08-02T15:12:54.292Z [INFO] server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e: entering follower state: follower="Node at dc4de7ca-3447-cbf4-6782-8826c8c93c9e [Follower]" leader= +testLoggerAdapter verbose: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.437Z [WARN] server-e3107dc3-d161-48d5-9aa6-82092b13434f: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.437Z [WARN] server-e3107dc3-d161-48d5-9aa6-82092b13434f: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.437Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: entering candidate state: node="Node at e3107dc3-d161-48d5-9aa6-82092b13434f [Candidate]" term=2 + testing.go:149: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.437Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: entering candidate state: node="Node at e3107dc3-d161-48d5-9aa6-82092b13434f [Candidate]" term=2 +testLoggerAdapter verbose: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.437Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: election won: tally=2 + testing.go:149: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.437Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: election won: tally=2 +testLoggerAdapter verbose: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.437Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: entering leader state: leader="Node at e3107dc3-d161-48d5-9aa6-82092b13434f [Leader]" + testing.go:149: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.437Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: entering leader state: leader="Node at e3107dc3-d161-48d5-9aa6-82092b13434f [Leader]" +testLoggerAdapter verbose: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.437Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: added peer, starting replication: peer=server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67 + testing.go:149: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.437Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: added peer, starting replication: peer=server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67 +testLoggerAdapter verbose: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.437Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: added peer, starting replication: peer=server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e + testing.go:149: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.437Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: added peer, starting replication: peer=server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e +testLoggerAdapter verbose: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.437Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: pipelining replication: peer="{Voter server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e dc4de7ca-3447-cbf4-6782-8826c8c93c9e}" + testing.go:149: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.437Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: pipelining replication: peer="{Voter server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e dc4de7ca-3447-cbf4-6782-8826c8c93c9e}" +testLoggerAdapter verbose: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.437Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: pipelining replication: peer="{Voter server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67 e6f973fe-395d-dd10-ecc1-65aa2c5e7d67}" + testing.go:149: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.437Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: pipelining replication: peer="{Voter server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67 e6f973fe-395d-dd10-ecc1-65aa2c5e7d67}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:54.642Z [INFO] stable state for Leader reached at 2023-08-02 15:12:54.437844916 +0000 UTC m=+6.533781249 (1 nodes), 144.904933ms from start of poll, 145.444755ms from cluster start. Timeout at 2023-08-02 15:12:54.642921318 +0000 UTC m=+6.738857645, 205.076396ms after stability + testing.go:149: cluster: 2023-08-02T15:12:54.642Z [INFO] stable state for Leader reached at 2023-08-02 15:12:54.437844916 +0000 UTC m=+6.533781249 (1 nodes), 144.904933ms from start of poll, 145.444755ms from cluster start. Timeout at 2023-08-02 15:12:54.642921318 +0000 UTC m=+6.738857645, 205.076396ms after stability +testLoggerAdapter verbose: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.656Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: aborting pipeline replication: peer="{Voter server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e dc4de7ca-3447-cbf4-6782-8826c8c93c9e}" + testing.go:149: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.656Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: aborting pipeline replication: peer="{Voter server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e dc4de7ca-3447-cbf4-6782-8826c8c93c9e}" +testLoggerAdapter verbose: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.656Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: aborting pipeline replication: peer="{Voter server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67 e6f973fe-395d-dd10-ecc1-65aa2c5e7d67}" + testing.go:149: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.656Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: aborting pipeline replication: peer="{Voter server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67 e6f973fe-395d-dd10-ecc1-65aa2c5e7d67}" +--- PASS: TestRaft_ApplyConcurrent (0.36s) === RUN TestRaft_ApplyConcurrent_Timeout -testLoggerAdapter verbose: server-94e5b0d8-eefe-4d10-96b2-0c8e6db627ac: 2024-09-03T21:34:25.615Z [INFO] server-94e5b0d8-eefe-4d10-96b2-0c8e6db627ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-94e5b0d8-eefe-4d10-96b2-0c8e6db627ac Address:94e5b0d8-eefe-4d10-96b2-0c8e6db627ac}]" - testing.go:149: server-94e5b0d8-eefe-4d10-96b2-0c8e6db627ac: 2024-09-03T21:34:25.615Z [INFO] server-94e5b0d8-eefe-4d10-96b2-0c8e6db627ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-94e5b0d8-eefe-4d10-96b2-0c8e6db627ac Address:94e5b0d8-eefe-4d10-96b2-0c8e6db627ac}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:25.615Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:25.615Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-94e5b0d8-eefe-4d10-96b2-0c8e6db627ac: 2024-09-03T21:34:25.615Z [INFO] server-94e5b0d8-eefe-4d10-96b2-0c8e6db627ac: entering follower state: follower="Node at 94e5b0d8-eefe-4d10-96b2-0c8e6db627ac [Follower]" leader= - testing.go:149: server-94e5b0d8-eefe-4d10-96b2-0c8e6db627ac: 2024-09-03T21:34:25.615Z [INFO] server-94e5b0d8-eefe-4d10-96b2-0c8e6db627ac: entering follower state: follower="Node at 94e5b0d8-eefe-4d10-96b2-0c8e6db627ac [Follower]" leader= -testLoggerAdapter verbose: server-94e5b0d8-eefe-4d10-96b2-0c8e6db627ac: 2024-09-03T21:34:25.720Z [WARN] server-94e5b0d8-eefe-4d10-96b2-0c8e6db627ac: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-94e5b0d8-eefe-4d10-96b2-0c8e6db627ac: 2024-09-03T21:34:25.720Z [WARN] server-94e5b0d8-eefe-4d10-96b2-0c8e6db627ac: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-94e5b0d8-eefe-4d10-96b2-0c8e6db627ac: 2024-09-03T21:34:25.720Z [INFO] server-94e5b0d8-eefe-4d10-96b2-0c8e6db627ac: entering candidate state: node="Node at 94e5b0d8-eefe-4d10-96b2-0c8e6db627ac [Candidate]" term=2 - testing.go:149: server-94e5b0d8-eefe-4d10-96b2-0c8e6db627ac: 2024-09-03T21:34:25.720Z [INFO] server-94e5b0d8-eefe-4d10-96b2-0c8e6db627ac: entering candidate state: node="Node at 94e5b0d8-eefe-4d10-96b2-0c8e6db627ac [Candidate]" term=2 -testLoggerAdapter verbose: server-94e5b0d8-eefe-4d10-96b2-0c8e6db627ac: 2024-09-03T21:34:25.721Z [INFO] server-94e5b0d8-eefe-4d10-96b2-0c8e6db627ac: election won: tally=1 - testing.go:149: server-94e5b0d8-eefe-4d10-96b2-0c8e6db627ac: 2024-09-03T21:34:25.721Z [INFO] server-94e5b0d8-eefe-4d10-96b2-0c8e6db627ac: election won: tally=1 -testLoggerAdapter verbose: server-94e5b0d8-eefe-4d10-96b2-0c8e6db627ac: 2024-09-03T21:34:25.721Z [INFO] server-94e5b0d8-eefe-4d10-96b2-0c8e6db627ac: entering leader state: leader="Node at 94e5b0d8-eefe-4d10-96b2-0c8e6db627ac [Leader]" - testing.go:149: server-94e5b0d8-eefe-4d10-96b2-0c8e6db627ac: 2024-09-03T21:34:25.721Z [INFO] server-94e5b0d8-eefe-4d10-96b2-0c8e6db627ac: entering leader state: leader="Node at 94e5b0d8-eefe-4d10-96b2-0c8e6db627ac [Leader]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:25.922Z [INFO] stable state for Leader reached at 2024-09-03 21:34:25.721168784 +0000 UTC m=+6.767825914 (1 nodes), 105.604593ms from start of poll, 105.843382ms from cluster start. Timeout at 2024-09-03 21:34:25.922255418 +0000 UTC m=+6.968912561, 201.086647ms after stability - testing.go:149: cluster: 2024-09-03T21:34:25.922Z [INFO] stable state for Leader reached at 2024-09-03 21:34:25.721168784 +0000 UTC m=+6.767825914 (1 nodes), 105.604593ms from start of poll, 105.843382ms from cluster start. Timeout at 2024-09-03 21:34:25.922255418 +0000 UTC m=+6.968912561, 201.086647ms after stability ---- PASS: TestRaft_ApplyConcurrent_Timeout (0.31s) +testLoggerAdapter verbose: server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: 2023-08-02T15:12:54.657Z [INFO] server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b Address:f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b}]" + testing.go:149: server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: 2023-08-02T15:12:54.657Z [INFO] server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b Address:f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:54.657Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:12:54.657Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: 2023-08-02T15:12:54.657Z [INFO] server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: entering follower state: follower="Node at f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b [Follower]" leader= + testing.go:149: server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: 2023-08-02T15:12:54.657Z [INFO] server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: entering follower state: follower="Node at f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b [Follower]" leader= +testLoggerAdapter verbose: server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: 2023-08-02T15:12:54.846Z [WARN] server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: 2023-08-02T15:12:54.846Z [WARN] server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: 2023-08-02T15:12:54.846Z [INFO] server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: entering candidate state: node="Node at f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b [Candidate]" term=2 + testing.go:149: server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: 2023-08-02T15:12:54.846Z [INFO] server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: entering candidate state: node="Node at f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b [Candidate]" term=2 +testLoggerAdapter verbose: server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: 2023-08-02T15:12:54.846Z [INFO] server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: election won: tally=1 + testing.go:149: server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: 2023-08-02T15:12:54.846Z [INFO] server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: election won: tally=1 +testLoggerAdapter verbose: server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: 2023-08-02T15:12:54.846Z [INFO] server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: entering leader state: leader="Node at f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b [Leader]" + testing.go:149: server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: 2023-08-02T15:12:54.846Z [INFO] server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: entering leader state: leader="Node at f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b [Leader]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:55.048Z [INFO] stable state for Leader reached at 2023-08-02 15:12:54.846906416 +0000 UTC m=+6.942842751 (1 nodes), 189.72564ms from start of poll, 189.94733ms from cluster start. Timeout at 2023-08-02 15:12:55.048355248 +0000 UTC m=+7.144291579, 201.448828ms after stability + testing.go:149: cluster: 2023-08-02T15:12:55.048Z [INFO] stable state for Leader reached at 2023-08-02 15:12:54.846906416 +0000 UTC m=+6.942842751 (1 nodes), 189.72564ms from start of poll, 189.94733ms from cluster start. Timeout at 2023-08-02 15:12:55.048355248 +0000 UTC m=+7.144291579, 201.448828ms after stability +--- PASS: TestRaft_ApplyConcurrent_Timeout (0.39s) === RUN TestRaft_JoinNode -testLoggerAdapter verbose: server-a8f2d2b3-b476-3785-aa8e-d99addc55383: 2024-09-03T21:34:25.924Z [INFO] server-a8f2d2b3-b476-3785-aa8e-d99addc55383: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8f2d2b3-b476-3785-aa8e-d99addc55383 Address:a8f2d2b3-b476-3785-aa8e-d99addc55383} {Suffrage:Voter ID:server-7df16b39-896f-645c-938c-de80be9431f5 Address:7df16b39-896f-645c-938c-de80be9431f5}]" - testing.go:149: server-a8f2d2b3-b476-3785-aa8e-d99addc55383: 2024-09-03T21:34:25.924Z [INFO] server-a8f2d2b3-b476-3785-aa8e-d99addc55383: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8f2d2b3-b476-3785-aa8e-d99addc55383 Address:a8f2d2b3-b476-3785-aa8e-d99addc55383} {Suffrage:Voter ID:server-7df16b39-896f-645c-938c-de80be9431f5 Address:7df16b39-896f-645c-938c-de80be9431f5}]" -testLoggerAdapter verbose: server-7df16b39-896f-645c-938c-de80be9431f5: 2024-09-03T21:34:25.924Z [INFO] server-7df16b39-896f-645c-938c-de80be9431f5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8f2d2b3-b476-3785-aa8e-d99addc55383 Address:a8f2d2b3-b476-3785-aa8e-d99addc55383} {Suffrage:Voter ID:server-7df16b39-896f-645c-938c-de80be9431f5 Address:7df16b39-896f-645c-938c-de80be9431f5}]" - testing.go:149: server-7df16b39-896f-645c-938c-de80be9431f5: 2024-09-03T21:34:25.924Z [INFO] server-7df16b39-896f-645c-938c-de80be9431f5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8f2d2b3-b476-3785-aa8e-d99addc55383 Address:a8f2d2b3-b476-3785-aa8e-d99addc55383} {Suffrage:Voter ID:server-7df16b39-896f-645c-938c-de80be9431f5 Address:7df16b39-896f-645c-938c-de80be9431f5}]" -testLoggerAdapter verbose: server-60d0cae4-d433-7609-00bf-49851a705df5: 2024-09-03T21:34:25.924Z [INFO] server-60d0cae4-d433-7609-00bf-49851a705df5: initial configuration: index=0 servers=[] - testing.go:149: server-60d0cae4-d433-7609-00bf-49851a705df5: 2024-09-03T21:34:25.924Z [INFO] server-60d0cae4-d433-7609-00bf-49851a705df5: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:25.924Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:25.924Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-a8f2d2b3-b476-3785-aa8e-d99addc55383: 2024-09-03T21:34:25.924Z [INFO] server-a8f2d2b3-b476-3785-aa8e-d99addc55383: entering follower state: follower="Node at a8f2d2b3-b476-3785-aa8e-d99addc55383 [Follower]" leader= - testing.go:149: server-a8f2d2b3-b476-3785-aa8e-d99addc55383: 2024-09-03T21:34:25.924Z [INFO] server-a8f2d2b3-b476-3785-aa8e-d99addc55383: entering follower state: follower="Node at a8f2d2b3-b476-3785-aa8e-d99addc55383 [Follower]" leader= -testLoggerAdapter verbose: server-7df16b39-896f-645c-938c-de80be9431f5: 2024-09-03T21:34:25.924Z [INFO] server-7df16b39-896f-645c-938c-de80be9431f5: entering follower state: follower="Node at 7df16b39-896f-645c-938c-de80be9431f5 [Follower]" leader= - testing.go:149: server-7df16b39-896f-645c-938c-de80be9431f5: 2024-09-03T21:34:25.924Z [INFO] server-7df16b39-896f-645c-938c-de80be9431f5: entering follower state: follower="Node at 7df16b39-896f-645c-938c-de80be9431f5 [Follower]" leader= -testLoggerAdapter verbose: server-60d0cae4-d433-7609-00bf-49851a705df5: 2024-09-03T21:34:25.924Z [INFO] server-60d0cae4-d433-7609-00bf-49851a705df5: entering follower state: follower="Node at 60d0cae4-d433-7609-00bf-49851a705df5 [Follower]" leader= - testing.go:149: server-60d0cae4-d433-7609-00bf-49851a705df5: 2024-09-03T21:34:25.924Z [INFO] server-60d0cae4-d433-7609-00bf-49851a705df5: entering follower state: follower="Node at 60d0cae4-d433-7609-00bf-49851a705df5 [Follower]" leader= -testLoggerAdapter verbose: server-60d0cae4-d433-7609-00bf-49851a705df5: 2024-09-03T21:34:25.997Z [WARN] server-60d0cae4-d433-7609-00bf-49851a705df5: no known peers, aborting election - testing.go:149: server-60d0cae4-d433-7609-00bf-49851a705df5: 2024-09-03T21:34:25.997Z [WARN] server-60d0cae4-d433-7609-00bf-49851a705df5: no known peers, aborting election -testLoggerAdapter verbose: server-7df16b39-896f-645c-938c-de80be9431f5: 2024-09-03T21:34:25.998Z [WARN] server-7df16b39-896f-645c-938c-de80be9431f5: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-7df16b39-896f-645c-938c-de80be9431f5: 2024-09-03T21:34:25.998Z [WARN] server-7df16b39-896f-645c-938c-de80be9431f5: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-7df16b39-896f-645c-938c-de80be9431f5: 2024-09-03T21:34:25.998Z [INFO] server-7df16b39-896f-645c-938c-de80be9431f5: entering candidate state: node="Node at 7df16b39-896f-645c-938c-de80be9431f5 [Candidate]" term=2 - testing.go:149: server-7df16b39-896f-645c-938c-de80be9431f5: 2024-09-03T21:34:25.998Z [INFO] server-7df16b39-896f-645c-938c-de80be9431f5: entering candidate state: node="Node at 7df16b39-896f-645c-938c-de80be9431f5 [Candidate]" term=2 -testLoggerAdapter verbose: server-7df16b39-896f-645c-938c-de80be9431f5: 2024-09-03T21:34:25.998Z [INFO] server-7df16b39-896f-645c-938c-de80be9431f5: election won: tally=2 - testing.go:149: server-7df16b39-896f-645c-938c-de80be9431f5: 2024-09-03T21:34:25.998Z [INFO] server-7df16b39-896f-645c-938c-de80be9431f5: election won: tally=2 -testLoggerAdapter verbose: server-7df16b39-896f-645c-938c-de80be9431f5: 2024-09-03T21:34:25.998Z [INFO] server-7df16b39-896f-645c-938c-de80be9431f5: entering leader state: leader="Node at 7df16b39-896f-645c-938c-de80be9431f5 [Leader]" - testing.go:149: server-7df16b39-896f-645c-938c-de80be9431f5: 2024-09-03T21:34:25.998Z [INFO] server-7df16b39-896f-645c-938c-de80be9431f5: entering leader state: leader="Node at 7df16b39-896f-645c-938c-de80be9431f5 [Leader]" -testLoggerAdapter verbose: server-7df16b39-896f-645c-938c-de80be9431f5: 2024-09-03T21:34:25.998Z [INFO] server-7df16b39-896f-645c-938c-de80be9431f5: added peer, starting replication: peer=server-a8f2d2b3-b476-3785-aa8e-d99addc55383 - testing.go:149: server-7df16b39-896f-645c-938c-de80be9431f5: 2024-09-03T21:34:25.998Z [INFO] server-7df16b39-896f-645c-938c-de80be9431f5: added peer, starting replication: peer=server-a8f2d2b3-b476-3785-aa8e-d99addc55383 -testLoggerAdapter verbose: server-7df16b39-896f-645c-938c-de80be9431f5: 2024-09-03T21:34:25.998Z [INFO] server-7df16b39-896f-645c-938c-de80be9431f5: pipelining replication: peer="{Voter server-a8f2d2b3-b476-3785-aa8e-d99addc55383 a8f2d2b3-b476-3785-aa8e-d99addc55383}" - testing.go:149: server-7df16b39-896f-645c-938c-de80be9431f5: 2024-09-03T21:34:25.998Z [INFO] server-7df16b39-896f-645c-938c-de80be9431f5: pipelining replication: peer="{Voter server-a8f2d2b3-b476-3785-aa8e-d99addc55383 a8f2d2b3-b476-3785-aa8e-d99addc55383}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:26.103Z [INFO] stable state for Leader reached at 2024-09-03 21:34:25.998412181 +0000 UTC m=+7.045069314 (1 nodes), 73.6547ms from start of poll, 74.158133ms from cluster start. Timeout at 2024-09-03 21:34:26.103499508 +0000 UTC m=+7.150156644, 105.08733ms after stability - testing.go:149: cluster: 2024-09-03T21:34:26.103Z [INFO] stable state for Leader reached at 2024-09-03 21:34:25.998412181 +0000 UTC m=+7.045069314 (1 nodes), 73.6547ms from start of poll, 74.158133ms from cluster start. Timeout at 2024-09-03 21:34:26.103499508 +0000 UTC m=+7.150156644, 105.08733ms after stability -testLoggerAdapter verbose: server-7df16b39-896f-645c-938c-de80be9431f5: 2024-09-03T21:34:26.103Z [INFO] server-7df16b39-896f-645c-938c-de80be9431f5: updating configuration: command=AddStaging server-id=server-60d0cae4-d433-7609-00bf-49851a705df5 server-addr=60d0cae4-d433-7609-00bf-49851a705df5 servers="[{Suffrage:Voter ID:server-a8f2d2b3-b476-3785-aa8e-d99addc55383 Address:a8f2d2b3-b476-3785-aa8e-d99addc55383} {Suffrage:Voter ID:server-7df16b39-896f-645c-938c-de80be9431f5 Address:7df16b39-896f-645c-938c-de80be9431f5} {Suffrage:Voter ID:server-60d0cae4-d433-7609-00bf-49851a705df5 Address:60d0cae4-d433-7609-00bf-49851a705df5}]" - testing.go:149: server-7df16b39-896f-645c-938c-de80be9431f5: 2024-09-03T21:34:26.103Z [INFO] server-7df16b39-896f-645c-938c-de80be9431f5: updating configuration: command=AddStaging server-id=server-60d0cae4-d433-7609-00bf-49851a705df5 server-addr=60d0cae4-d433-7609-00bf-49851a705df5 servers="[{Suffrage:Voter ID:server-a8f2d2b3-b476-3785-aa8e-d99addc55383 Address:a8f2d2b3-b476-3785-aa8e-d99addc55383} {Suffrage:Voter ID:server-7df16b39-896f-645c-938c-de80be9431f5 Address:7df16b39-896f-645c-938c-de80be9431f5} {Suffrage:Voter ID:server-60d0cae4-d433-7609-00bf-49851a705df5 Address:60d0cae4-d433-7609-00bf-49851a705df5}]" -testLoggerAdapter verbose: server-7df16b39-896f-645c-938c-de80be9431f5: 2024-09-03T21:34:26.103Z [INFO] server-7df16b39-896f-645c-938c-de80be9431f5: added peer, starting replication: peer=server-60d0cae4-d433-7609-00bf-49851a705df5 - testing.go:149: server-7df16b39-896f-645c-938c-de80be9431f5: 2024-09-03T21:34:26.103Z [INFO] server-7df16b39-896f-645c-938c-de80be9431f5: added peer, starting replication: peer=server-60d0cae4-d433-7609-00bf-49851a705df5 -testLoggerAdapter verbose: server-60d0cae4-d433-7609-00bf-49851a705df5: 2024-09-03T21:34:26.103Z [WARN] server-60d0cae4-d433-7609-00bf-49851a705df5: failed to get previous log: previous-index=3 last-index=0 error="log not found" - testing.go:149: server-60d0cae4-d433-7609-00bf-49851a705df5: 2024-09-03T21:34:26.103Z [WARN] server-60d0cae4-d433-7609-00bf-49851a705df5: failed to get previous log: previous-index=3 last-index=0 error="log not found" -testLoggerAdapter verbose: server-7df16b39-896f-645c-938c-de80be9431f5: 2024-09-03T21:34:26.103Z [WARN] server-7df16b39-896f-645c-938c-de80be9431f5: appendEntries rejected, sending older logs: peer="{Voter server-60d0cae4-d433-7609-00bf-49851a705df5 60d0cae4-d433-7609-00bf-49851a705df5}" next=1 - testing.go:149: server-7df16b39-896f-645c-938c-de80be9431f5: 2024-09-03T21:34:26.103Z [WARN] server-7df16b39-896f-645c-938c-de80be9431f5: appendEntries rejected, sending older logs: peer="{Voter server-60d0cae4-d433-7609-00bf-49851a705df5 60d0cae4-d433-7609-00bf-49851a705df5}" next=1 -testLoggerAdapter verbose: server-7df16b39-896f-645c-938c-de80be9431f5: 2024-09-03T21:34:26.104Z [INFO] server-7df16b39-896f-645c-938c-de80be9431f5: pipelining replication: peer="{Voter server-60d0cae4-d433-7609-00bf-49851a705df5 60d0cae4-d433-7609-00bf-49851a705df5}" - testing.go:149: server-7df16b39-896f-645c-938c-de80be9431f5: 2024-09-03T21:34:26.104Z [INFO] server-7df16b39-896f-645c-938c-de80be9431f5: pipelining replication: peer="{Voter server-60d0cae4-d433-7609-00bf-49851a705df5 60d0cae4-d433-7609-00bf-49851a705df5}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:26.104Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:26.104Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:26.209Z [INFO] stable state for Leader reached at 2024-09-03 21:34:26.10423303 +0000 UTC m=+7.150890166 (1 nodes), 842ns from start of poll, 179.978985ms from cluster start. Timeout at 2024-09-03 21:34:26.209307202 +0000 UTC m=+7.255964923, 105.074757ms after stability - testing.go:149: cluster: 2024-09-03T21:34:26.209Z [INFO] stable state for Leader reached at 2024-09-03 21:34:26.10423303 +0000 UTC m=+7.150890166 (1 nodes), 842ns from start of poll, 179.978985ms from cluster start. Timeout at 2024-09-03 21:34:26.209307202 +0000 UTC m=+7.255964923, 105.074757ms after stability -testLoggerAdapter verbose: server-7df16b39-896f-645c-938c-de80be9431f5: 2024-09-03T21:34:26.209Z [INFO] server-7df16b39-896f-645c-938c-de80be9431f5: aborting pipeline replication: peer="{Voter server-a8f2d2b3-b476-3785-aa8e-d99addc55383 a8f2d2b3-b476-3785-aa8e-d99addc55383}" - testing.go:149: server-7df16b39-896f-645c-938c-de80be9431f5: 2024-09-03T21:34:26.209Z [INFO] server-7df16b39-896f-645c-938c-de80be9431f5: aborting pipeline replication: peer="{Voter server-a8f2d2b3-b476-3785-aa8e-d99addc55383 a8f2d2b3-b476-3785-aa8e-d99addc55383}" -testLoggerAdapter verbose: server-7df16b39-896f-645c-938c-de80be9431f5: 2024-09-03T21:34:26.209Z [INFO] server-7df16b39-896f-645c-938c-de80be9431f5: aborting pipeline replication: peer="{Voter server-60d0cae4-d433-7609-00bf-49851a705df5 60d0cae4-d433-7609-00bf-49851a705df5}" - testing.go:149: server-7df16b39-896f-645c-938c-de80be9431f5: 2024-09-03T21:34:26.209Z [INFO] server-7df16b39-896f-645c-938c-de80be9431f5: aborting pipeline replication: peer="{Voter server-60d0cae4-d433-7609-00bf-49851a705df5 60d0cae4-d433-7609-00bf-49851a705df5}" ---- PASS: TestRaft_JoinNode (0.29s) +testLoggerAdapter verbose: server-403d52cd-d42b-d401-2e45-a7e7a20a02d3: 2023-08-02T15:12:55.051Z [INFO] server-403d52cd-d42b-d401-2e45-a7e7a20a02d3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-403d52cd-d42b-d401-2e45-a7e7a20a02d3 Address:403d52cd-d42b-d401-2e45-a7e7a20a02d3} {Suffrage:Voter ID:server-f311d765-c353-90cb-94a0-c34bfa0aa721 Address:f311d765-c353-90cb-94a0-c34bfa0aa721}]" + testing.go:149: server-403d52cd-d42b-d401-2e45-a7e7a20a02d3: 2023-08-02T15:12:55.051Z [INFO] server-403d52cd-d42b-d401-2e45-a7e7a20a02d3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-403d52cd-d42b-d401-2e45-a7e7a20a02d3 Address:403d52cd-d42b-d401-2e45-a7e7a20a02d3} {Suffrage:Voter ID:server-f311d765-c353-90cb-94a0-c34bfa0aa721 Address:f311d765-c353-90cb-94a0-c34bfa0aa721}]" +testLoggerAdapter verbose: server-403d52cd-d42b-d401-2e45-a7e7a20a02d3: 2023-08-02T15:12:55.051Z [INFO] server-403d52cd-d42b-d401-2e45-a7e7a20a02d3: entering follower state: follower="Node at 403d52cd-d42b-d401-2e45-a7e7a20a02d3 [Follower]" leader= +testLoggerAdapter verbose: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.051Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-403d52cd-d42b-d401-2e45-a7e7a20a02d3 Address:403d52cd-d42b-d401-2e45-a7e7a20a02d3} {Suffrage:Voter ID:server-f311d765-c353-90cb-94a0-c34bfa0aa721 Address:f311d765-c353-90cb-94a0-c34bfa0aa721}]" + testing.go:149: server-403d52cd-d42b-d401-2e45-a7e7a20a02d3: 2023-08-02T15:12:55.051Z [INFO] server-403d52cd-d42b-d401-2e45-a7e7a20a02d3: entering follower state: follower="Node at 403d52cd-d42b-d401-2e45-a7e7a20a02d3 [Follower]" leader= + testing.go:149: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.051Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-403d52cd-d42b-d401-2e45-a7e7a20a02d3 Address:403d52cd-d42b-d401-2e45-a7e7a20a02d3} {Suffrage:Voter ID:server-f311d765-c353-90cb-94a0-c34bfa0aa721 Address:f311d765-c353-90cb-94a0-c34bfa0aa721}]" +testLoggerAdapter verbose: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.051Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: entering follower state: follower="Node at f311d765-c353-90cb-94a0-c34bfa0aa721 [Follower]" leader= + testing.go:149: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.051Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: entering follower state: follower="Node at f311d765-c353-90cb-94a0-c34bfa0aa721 [Follower]" leader= +testLoggerAdapter verbose: server-69de86e2-46d2-260c-3434-5955a9711433: 2023-08-02T15:12:55.051Z [INFO] server-69de86e2-46d2-260c-3434-5955a9711433: initial configuration: index=0 servers=[] + testing.go:149: server-69de86e2-46d2-260c-3434-5955a9711433: 2023-08-02T15:12:55.051Z [INFO] server-69de86e2-46d2-260c-3434-5955a9711433: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:55.051Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:12:55.051Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-69de86e2-46d2-260c-3434-5955a9711433: 2023-08-02T15:12:55.051Z [INFO] server-69de86e2-46d2-260c-3434-5955a9711433: entering follower state: follower="Node at 69de86e2-46d2-260c-3434-5955a9711433 [Follower]" leader= + testing.go:149: server-69de86e2-46d2-260c-3434-5955a9711433: 2023-08-02T15:12:55.051Z [INFO] server-69de86e2-46d2-260c-3434-5955a9711433: entering follower state: follower="Node at 69de86e2-46d2-260c-3434-5955a9711433 [Follower]" leader= +testLoggerAdapter verbose: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.106Z [WARN] server-f311d765-c353-90cb-94a0-c34bfa0aa721: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.106Z [WARN] server-f311d765-c353-90cb-94a0-c34bfa0aa721: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.106Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: entering candidate state: node="Node at f311d765-c353-90cb-94a0-c34bfa0aa721 [Candidate]" term=2 + testing.go:149: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.106Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: entering candidate state: node="Node at f311d765-c353-90cb-94a0-c34bfa0aa721 [Candidate]" term=2 +testLoggerAdapter verbose: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.111Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: election won: tally=2 + testing.go:149: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.111Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: election won: tally=2 +testLoggerAdapter verbose: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.111Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: entering leader state: leader="Node at f311d765-c353-90cb-94a0-c34bfa0aa721 [Leader]" + testing.go:149: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.111Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: entering leader state: leader="Node at f311d765-c353-90cb-94a0-c34bfa0aa721 [Leader]" +testLoggerAdapter verbose: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.111Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: added peer, starting replication: peer=server-403d52cd-d42b-d401-2e45-a7e7a20a02d3 + testing.go:149: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.111Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: added peer, starting replication: peer=server-403d52cd-d42b-d401-2e45-a7e7a20a02d3 +testLoggerAdapter verbose: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.112Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: pipelining replication: peer="{Voter server-403d52cd-d42b-d401-2e45-a7e7a20a02d3 403d52cd-d42b-d401-2e45-a7e7a20a02d3}" + testing.go:149: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.112Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: pipelining replication: peer="{Voter server-403d52cd-d42b-d401-2e45-a7e7a20a02d3 403d52cd-d42b-d401-2e45-a7e7a20a02d3}" +testLoggerAdapter verbose: server-69de86e2-46d2-260c-3434-5955a9711433: 2023-08-02T15:12:55.118Z [WARN] server-69de86e2-46d2-260c-3434-5955a9711433: no known peers, aborting election + testing.go:149: server-69de86e2-46d2-260c-3434-5955a9711433: 2023-08-02T15:12:55.118Z [WARN] server-69de86e2-46d2-260c-3434-5955a9711433: no known peers, aborting election +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:55.217Z [INFO] stable state for Leader reached at 2023-08-02 15:12:55.111869237 +0000 UTC m=+7.207805596 (1 nodes), 60.159062ms from start of poll, 60.826267ms from cluster start. Timeout at 2023-08-02 15:12:55.216979364 +0000 UTC m=+7.312915735, 105.110139ms after stability + testing.go:149: cluster: 2023-08-02T15:12:55.217Z [INFO] stable state for Leader reached at 2023-08-02 15:12:55.111869237 +0000 UTC m=+7.207805596 (1 nodes), 60.159062ms from start of poll, 60.826267ms from cluster start. Timeout at 2023-08-02 15:12:55.216979364 +0000 UTC m=+7.312915735, 105.110139ms after stability +testLoggerAdapter verbose: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.217Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: updating configuration: command=AddStaging server-id=server-69de86e2-46d2-260c-3434-5955a9711433 server-addr=69de86e2-46d2-260c-3434-5955a9711433 servers="[{Suffrage:Voter ID:server-403d52cd-d42b-d401-2e45-a7e7a20a02d3 Address:403d52cd-d42b-d401-2e45-a7e7a20a02d3} {Suffrage:Voter ID:server-f311d765-c353-90cb-94a0-c34bfa0aa721 Address:f311d765-c353-90cb-94a0-c34bfa0aa721} {Suffrage:Voter ID:server-69de86e2-46d2-260c-3434-5955a9711433 Address:69de86e2-46d2-260c-3434-5955a9711433}]" + testing.go:149: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.217Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: updating configuration: command=AddStaging server-id=server-69de86e2-46d2-260c-3434-5955a9711433 server-addr=69de86e2-46d2-260c-3434-5955a9711433 servers="[{Suffrage:Voter ID:server-403d52cd-d42b-d401-2e45-a7e7a20a02d3 Address:403d52cd-d42b-d401-2e45-a7e7a20a02d3} {Suffrage:Voter ID:server-f311d765-c353-90cb-94a0-c34bfa0aa721 Address:f311d765-c353-90cb-94a0-c34bfa0aa721} {Suffrage:Voter ID:server-69de86e2-46d2-260c-3434-5955a9711433 Address:69de86e2-46d2-260c-3434-5955a9711433}]" +testLoggerAdapter verbose: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.217Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: added peer, starting replication: peer=server-69de86e2-46d2-260c-3434-5955a9711433 + testing.go:149: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.217Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: added peer, starting replication: peer=server-69de86e2-46d2-260c-3434-5955a9711433 +testLoggerAdapter verbose: server-69de86e2-46d2-260c-3434-5955a9711433: 2023-08-02T15:12:55.217Z [WARN] server-69de86e2-46d2-260c-3434-5955a9711433: failed to get previous log: previous-index=3 last-index=0 error="log not found" + testing.go:149: server-69de86e2-46d2-260c-3434-5955a9711433: 2023-08-02T15:12:55.217Z [WARN] server-69de86e2-46d2-260c-3434-5955a9711433: failed to get previous log: previous-index=3 last-index=0 error="log not found" +testLoggerAdapter verbose: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.217Z [WARN] server-f311d765-c353-90cb-94a0-c34bfa0aa721: appendEntries rejected, sending older logs: peer="{Voter server-69de86e2-46d2-260c-3434-5955a9711433 69de86e2-46d2-260c-3434-5955a9711433}" next=1 + testing.go:149: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.217Z [WARN] server-f311d765-c353-90cb-94a0-c34bfa0aa721: appendEntries rejected, sending older logs: peer="{Voter server-69de86e2-46d2-260c-3434-5955a9711433 69de86e2-46d2-260c-3434-5955a9711433}" next=1 +testLoggerAdapter verbose: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.217Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: pipelining replication: peer="{Voter server-69de86e2-46d2-260c-3434-5955a9711433 69de86e2-46d2-260c-3434-5955a9711433}" + testing.go:149: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.217Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: pipelining replication: peer="{Voter server-69de86e2-46d2-260c-3434-5955a9711433 69de86e2-46d2-260c-3434-5955a9711433}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:55.217Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:12:55.217Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:55.323Z [INFO] stable state for Leader reached at 2023-08-02 15:12:55.217828545 +0000 UTC m=+7.313764871 (1 nodes), 907ns from start of poll, 166.785542ms from cluster start. Timeout at 2023-08-02 15:12:55.323024326 +0000 UTC m=+7.418960672, 105.195801ms after stability + testing.go:149: cluster: 2023-08-02T15:12:55.323Z [INFO] stable state for Leader reached at 2023-08-02 15:12:55.217828545 +0000 UTC m=+7.313764871 (1 nodes), 907ns from start of poll, 166.785542ms from cluster start. Timeout at 2023-08-02 15:12:55.323024326 +0000 UTC m=+7.418960672, 105.195801ms after stability +testLoggerAdapter verbose: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.323Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: aborting pipeline replication: peer="{Voter server-69de86e2-46d2-260c-3434-5955a9711433 69de86e2-46d2-260c-3434-5955a9711433}" + testing.go:149: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.323Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: aborting pipeline replication: peer="{Voter server-69de86e2-46d2-260c-3434-5955a9711433 69de86e2-46d2-260c-3434-5955a9711433}" +testLoggerAdapter verbose: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.323Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: aborting pipeline replication: peer="{Voter server-403d52cd-d42b-d401-2e45-a7e7a20a02d3 403d52cd-d42b-d401-2e45-a7e7a20a02d3}" + testing.go:149: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.323Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: aborting pipeline replication: peer="{Voter server-403d52cd-d42b-d401-2e45-a7e7a20a02d3 403d52cd-d42b-d401-2e45-a7e7a20a02d3}" +--- PASS: TestRaft_JoinNode (0.27s) === RUN TestRaft_JoinNode_ConfigStore -testLoggerAdapter verbose: server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: 2024-09-03T21:34:26.210Z [INFO] server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0 Address:924a4f6f-ad95-faef-7eb9-85b0166ad3f0}]" - testing.go:149: server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: 2024-09-03T21:34:26.210Z [INFO] server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0 Address:924a4f6f-ad95-faef-7eb9-85b0166ad3f0}]" -testLoggerAdapter verbose: server-046990d2-3491-3240-a77c-831f08418693: 2024-09-03T21:34:26.210Z [INFO] server-046990d2-3491-3240-a77c-831f08418693: initial configuration: index=0 servers=[] - testing.go:149: server-046990d2-3491-3240-a77c-831f08418693: 2024-09-03T21:34:26.210Z [INFO] server-046990d2-3491-3240-a77c-831f08418693: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: server-40d658d0-5b29-e961-4194-3dff392bcd9c: 2024-09-03T21:34:26.210Z [INFO] server-40d658d0-5b29-e961-4194-3dff392bcd9c: initial configuration: index=0 servers=[] - testing.go:149: server-40d658d0-5b29-e961-4194-3dff392bcd9c: 2024-09-03T21:34:26.210Z [INFO] server-40d658d0-5b29-e961-4194-3dff392bcd9c: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:26.210Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:26.210Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: 2024-09-03T21:34:26.210Z [INFO] server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: entering follower state: follower="Node at 924a4f6f-ad95-faef-7eb9-85b0166ad3f0 [Follower]" leader= - testing.go:149: server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: 2024-09-03T21:34:26.210Z [INFO] server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: entering follower state: follower="Node at 924a4f6f-ad95-faef-7eb9-85b0166ad3f0 [Follower]" leader= -testLoggerAdapter verbose: server-046990d2-3491-3240-a77c-831f08418693: 2024-09-03T21:34:26.210Z [INFO] server-046990d2-3491-3240-a77c-831f08418693: entering follower state: follower="Node at 046990d2-3491-3240-a77c-831f08418693 [Follower]" leader= - testing.go:149: server-046990d2-3491-3240-a77c-831f08418693: 2024-09-03T21:34:26.210Z [INFO] server-046990d2-3491-3240-a77c-831f08418693: entering follower state: follower="Node at 046990d2-3491-3240-a77c-831f08418693 [Follower]" leader= -testLoggerAdapter verbose: server-40d658d0-5b29-e961-4194-3dff392bcd9c: 2024-09-03T21:34:26.210Z [INFO] server-40d658d0-5b29-e961-4194-3dff392bcd9c: entering follower state: follower="Node at 40d658d0-5b29-e961-4194-3dff392bcd9c [Follower]" leader= - testing.go:149: server-40d658d0-5b29-e961-4194-3dff392bcd9c: 2024-09-03T21:34:26.210Z [INFO] server-40d658d0-5b29-e961-4194-3dff392bcd9c: entering follower state: follower="Node at 40d658d0-5b29-e961-4194-3dff392bcd9c [Follower]" leader= -testLoggerAdapter verbose: server-046990d2-3491-3240-a77c-831f08418693: 2024-09-03T21:34:26.287Z [WARN] server-046990d2-3491-3240-a77c-831f08418693: no known peers, aborting election - testing.go:149: server-046990d2-3491-3240-a77c-831f08418693: 2024-09-03T21:34:26.287Z [WARN] server-046990d2-3491-3240-a77c-831f08418693: no known peers, aborting election -testLoggerAdapter verbose: server-40d658d0-5b29-e961-4194-3dff392bcd9c: 2024-09-03T21:34:26.298Z [WARN] server-40d658d0-5b29-e961-4194-3dff392bcd9c: no known peers, aborting election - testing.go:149: server-40d658d0-5b29-e961-4194-3dff392bcd9c: 2024-09-03T21:34:26.298Z [WARN] server-40d658d0-5b29-e961-4194-3dff392bcd9c: no known peers, aborting election -testLoggerAdapter verbose: server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: 2024-09-03T21:34:26.309Z [WARN] server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: 2024-09-03T21:34:26.309Z [WARN] server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: 2024-09-03T21:34:26.309Z [INFO] server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: entering candidate state: node="Node at 924a4f6f-ad95-faef-7eb9-85b0166ad3f0 [Candidate]" term=2 - testing.go:149: server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: 2024-09-03T21:34:26.309Z [INFO] server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: entering candidate state: node="Node at 924a4f6f-ad95-faef-7eb9-85b0166ad3f0 [Candidate]" term=2 -testLoggerAdapter verbose: server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: 2024-09-03T21:34:26.309Z [INFO] server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: election won: tally=1 - testing.go:149: server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: 2024-09-03T21:34:26.309Z [INFO] server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: election won: tally=1 -testLoggerAdapter verbose: server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: 2024-09-03T21:34:26.309Z [INFO] server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: entering leader state: leader="Node at 924a4f6f-ad95-faef-7eb9-85b0166ad3f0 [Leader]" - testing.go:149: server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: 2024-09-03T21:34:26.309Z [INFO] server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: entering leader state: leader="Node at 924a4f6f-ad95-faef-7eb9-85b0166ad3f0 [Leader]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:26.414Z [INFO] stable state for Leader reached at 2024-09-03 21:34:26.309574787 +0000 UTC m=+7.356231923 (1 nodes), 98.873899ms from start of poll, 99.422174ms from cluster start. Timeout at 2024-09-03 21:34:26.414681878 +0000 UTC m=+7.461339075, 105.107152ms after stability - testing.go:149: cluster: 2024-09-03T21:34:26.414Z [INFO] stable state for Leader reached at 2024-09-03 21:34:26.309574787 +0000 UTC m=+7.356231923 (1 nodes), 98.873899ms from start of poll, 99.422174ms from cluster start. Timeout at 2024-09-03 21:34:26.414681878 +0000 UTC m=+7.461339075, 105.107152ms after stability -testLoggerAdapter verbose: server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: 2024-09-03T21:34:26.414Z [INFO] server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: updating configuration: command=AddStaging server-id=server-046990d2-3491-3240-a77c-831f08418693 server-addr=046990d2-3491-3240-a77c-831f08418693 servers="[{Suffrage:Voter ID:server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0 Address:924a4f6f-ad95-faef-7eb9-85b0166ad3f0} {Suffrage:Voter ID:server-046990d2-3491-3240-a77c-831f08418693 Address:046990d2-3491-3240-a77c-831f08418693}]" - testing.go:149: server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: 2024-09-03T21:34:26.414Z [INFO] server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: updating configuration: command=AddStaging server-id=server-046990d2-3491-3240-a77c-831f08418693 server-addr=046990d2-3491-3240-a77c-831f08418693 servers="[{Suffrage:Voter ID:server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0 Address:924a4f6f-ad95-faef-7eb9-85b0166ad3f0} {Suffrage:Voter ID:server-046990d2-3491-3240-a77c-831f08418693 Address:046990d2-3491-3240-a77c-831f08418693}]" -testLoggerAdapter verbose: server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: 2024-09-03T21:34:26.414Z [INFO] server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: added peer, starting replication: peer=server-046990d2-3491-3240-a77c-831f08418693 - testing.go:149: server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: 2024-09-03T21:34:26.414Z [INFO] server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: added peer, starting replication: peer=server-046990d2-3491-3240-a77c-831f08418693 -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:26.415Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:26.415Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-046990d2-3491-3240-a77c-831f08418693: 2024-09-03T21:34:26.415Z [WARN] server-046990d2-3491-3240-a77c-831f08418693: failed to get previous log: previous-index=3 last-index=0 error="log not found" - testing.go:149: server-046990d2-3491-3240-a77c-831f08418693: 2024-09-03T21:34:26.415Z [WARN] server-046990d2-3491-3240-a77c-831f08418693: failed to get previous log: previous-index=3 last-index=0 error="log not found" -testLoggerAdapter verbose: server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: 2024-09-03T21:34:26.415Z [WARN] server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: appendEntries rejected, sending older logs: peer="{Voter server-046990d2-3491-3240-a77c-831f08418693 046990d2-3491-3240-a77c-831f08418693}" next=1 - testing.go:149: server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: 2024-09-03T21:34:26.415Z [WARN] server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: appendEntries rejected, sending older logs: peer="{Voter server-046990d2-3491-3240-a77c-831f08418693 046990d2-3491-3240-a77c-831f08418693}" next=1 -testLoggerAdapter verbose: server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: 2024-09-03T21:34:26.415Z [INFO] server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: pipelining replication: peer="{Voter server-046990d2-3491-3240-a77c-831f08418693 046990d2-3491-3240-a77c-831f08418693}" - testing.go:149: server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: 2024-09-03T21:34:26.415Z [INFO] server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: pipelining replication: peer="{Voter server-046990d2-3491-3240-a77c-831f08418693 046990d2-3491-3240-a77c-831f08418693}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:26.520Z [INFO] stable state for Leader reached at 2024-09-03 21:34:26.415098926 +0000 UTC m=+7.461756057 (1 nodes), 1.02µs from start of poll, 204.946308ms from cluster start. Timeout at 2024-09-03 21:34:26.52017497 +0000 UTC m=+7.566832103, 105.076046ms after stability - testing.go:149: cluster: 2024-09-03T21:34:26.520Z [INFO] stable state for Leader reached at 2024-09-03 21:34:26.415098926 +0000 UTC m=+7.461756057 (1 nodes), 1.02µs from start of poll, 204.946308ms from cluster start. Timeout at 2024-09-03 21:34:26.52017497 +0000 UTC m=+7.566832103, 105.076046ms after stability -testLoggerAdapter verbose: server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: 2024-09-03T21:34:26.520Z [INFO] server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: updating configuration: command=AddStaging server-id=server-40d658d0-5b29-e961-4194-3dff392bcd9c server-addr=40d658d0-5b29-e961-4194-3dff392bcd9c servers="[{Suffrage:Voter ID:server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0 Address:924a4f6f-ad95-faef-7eb9-85b0166ad3f0} {Suffrage:Voter ID:server-046990d2-3491-3240-a77c-831f08418693 Address:046990d2-3491-3240-a77c-831f08418693} {Suffrage:Voter ID:server-40d658d0-5b29-e961-4194-3dff392bcd9c Address:40d658d0-5b29-e961-4194-3dff392bcd9c}]" - testing.go:149: server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: 2024-09-03T21:34:26.520Z [INFO] server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: updating configuration: command=AddStaging server-id=server-40d658d0-5b29-e961-4194-3dff392bcd9c server-addr=40d658d0-5b29-e961-4194-3dff392bcd9c servers="[{Suffrage:Voter ID:server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0 Address:924a4f6f-ad95-faef-7eb9-85b0166ad3f0} {Suffrage:Voter ID:server-046990d2-3491-3240-a77c-831f08418693 Address:046990d2-3491-3240-a77c-831f08418693} {Suffrage:Voter ID:server-40d658d0-5b29-e961-4194-3dff392bcd9c Address:40d658d0-5b29-e961-4194-3dff392bcd9c}]" -testLoggerAdapter verbose: server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: 2024-09-03T21:34:26.520Z [INFO] server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: added peer, starting replication: peer=server-40d658d0-5b29-e961-4194-3dff392bcd9c - testing.go:149: server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: 2024-09-03T21:34:26.520Z [INFO] server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: added peer, starting replication: peer=server-40d658d0-5b29-e961-4194-3dff392bcd9c -testLoggerAdapter verbose: server-40d658d0-5b29-e961-4194-3dff392bcd9c: 2024-09-03T21:34:26.520Z [WARN] server-40d658d0-5b29-e961-4194-3dff392bcd9c: failed to get previous log: previous-index=4 last-index=0 error="log not found" - testing.go:149: server-40d658d0-5b29-e961-4194-3dff392bcd9c: 2024-09-03T21:34:26.520Z [WARN] server-40d658d0-5b29-e961-4194-3dff392bcd9c: failed to get previous log: previous-index=4 last-index=0 error="log not found" -testLoggerAdapter verbose: server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: 2024-09-03T21:34:26.520Z [WARN] server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: appendEntries rejected, sending older logs: peer="{Voter server-40d658d0-5b29-e961-4194-3dff392bcd9c 40d658d0-5b29-e961-4194-3dff392bcd9c}" next=1 - testing.go:149: server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: 2024-09-03T21:34:26.520Z [WARN] server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: appendEntries rejected, sending older logs: peer="{Voter server-40d658d0-5b29-e961-4194-3dff392bcd9c 40d658d0-5b29-e961-4194-3dff392bcd9c}" next=1 -testLoggerAdapter verbose: server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: 2024-09-03T21:34:26.520Z [INFO] server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: pipelining replication: peer="{Voter server-40d658d0-5b29-e961-4194-3dff392bcd9c 40d658d0-5b29-e961-4194-3dff392bcd9c}" - testing.go:149: server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: 2024-09-03T21:34:26.520Z [INFO] server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: pipelining replication: peer="{Voter server-40d658d0-5b29-e961-4194-3dff392bcd9c 40d658d0-5b29-e961-4194-3dff392bcd9c}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:26.520Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:26.520Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:26.625Z [INFO] stable state for Leader reached at 2024-09-03 21:34:26.520804996 +0000 UTC m=+7.567462127 (1 nodes), 726ns from start of poll, 310.652378ms from cluster start. Timeout at 2024-09-03 21:34:26.625898746 +0000 UTC m=+7.672555877, 105.09375ms after stability - testing.go:149: cluster: 2024-09-03T21:34:26.625Z [INFO] stable state for Leader reached at 2024-09-03 21:34:26.520804996 +0000 UTC m=+7.567462127 (1 nodes), 726ns from start of poll, 310.652378ms from cluster start. Timeout at 2024-09-03 21:34:26.625898746 +0000 UTC m=+7.672555877, 105.09375ms after stability -testLoggerAdapter verbose: server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: 2024-09-03T21:34:26.626Z [INFO] server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: aborting pipeline replication: peer="{Voter server-046990d2-3491-3240-a77c-831f08418693 046990d2-3491-3240-a77c-831f08418693}" - testing.go:149: server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: 2024-09-03T21:34:26.626Z [INFO] server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: aborting pipeline replication: peer="{Voter server-046990d2-3491-3240-a77c-831f08418693 046990d2-3491-3240-a77c-831f08418693}" -testLoggerAdapter verbose: server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: 2024-09-03T21:34:26.626Z [INFO] server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: aborting pipeline replication: peer="{Voter server-40d658d0-5b29-e961-4194-3dff392bcd9c 40d658d0-5b29-e961-4194-3dff392bcd9c}" - testing.go:149: server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: 2024-09-03T21:34:26.626Z [INFO] server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: aborting pipeline replication: peer="{Voter server-40d658d0-5b29-e961-4194-3dff392bcd9c 40d658d0-5b29-e961-4194-3dff392bcd9c}" -testLoggerAdapter verbose: server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: 2024-09-03T21:34:26.676Z [ERROR] server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: failed to heartbeat to: peer=046990d2-3491-3240-a77c-831f08418693 error="command timed out" - testing.go:149: server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: 2024-09-03T21:34:26.676Z [ERROR] server-924a4f6f-ad95-faef-7eb9-85b0166ad3f0: failed to heartbeat to: peer=046990d2-3491-3240-a77c-831f08418693 error="command timed out" ---- PASS: TestRaft_JoinNode_ConfigStore (0.47s) +testLoggerAdapter verbose: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.324Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d Address:9a6571b0-ba9b-6e3e-6646-3a21e92ff67d}]" + testing.go:149: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.324Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d Address:9a6571b0-ba9b-6e3e-6646-3a21e92ff67d}]" +testLoggerAdapter verbose: server-bf17074b-abfd-043e-0171-489e92d0ccb3: 2023-08-02T15:12:55.324Z [INFO] server-bf17074b-abfd-043e-0171-489e92d0ccb3: initial configuration: index=0 servers=[] + testing.go:149: server-bf17074b-abfd-043e-0171-489e92d0ccb3: 2023-08-02T15:12:55.324Z [INFO] server-bf17074b-abfd-043e-0171-489e92d0ccb3: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91: 2023-08-02T15:12:55.324Z [INFO] server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91: initial configuration: index=0 servers=[] + testing.go:149: server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91: 2023-08-02T15:12:55.324Z [INFO] server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:55.324Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:12:55.324Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.324Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: entering follower state: follower="Node at 9a6571b0-ba9b-6e3e-6646-3a21e92ff67d [Follower]" leader= + testing.go:149: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.324Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: entering follower state: follower="Node at 9a6571b0-ba9b-6e3e-6646-3a21e92ff67d [Follower]" leader= +testLoggerAdapter verbose: server-bf17074b-abfd-043e-0171-489e92d0ccb3: 2023-08-02T15:12:55.324Z [INFO] server-bf17074b-abfd-043e-0171-489e92d0ccb3: entering follower state: follower="Node at bf17074b-abfd-043e-0171-489e92d0ccb3 [Follower]" leader= + testing.go:149: server-bf17074b-abfd-043e-0171-489e92d0ccb3: 2023-08-02T15:12:55.324Z [INFO] server-bf17074b-abfd-043e-0171-489e92d0ccb3: entering follower state: follower="Node at bf17074b-abfd-043e-0171-489e92d0ccb3 [Follower]" leader= +testLoggerAdapter verbose: server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91: 2023-08-02T15:12:55.325Z [INFO] server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91: entering follower state: follower="Node at 157a3fd6-85d6-fb89-488c-d7d46c6dbb91 [Follower]" leader= + testing.go:149: server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91: 2023-08-02T15:12:55.325Z [INFO] server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91: entering follower state: follower="Node at 157a3fd6-85d6-fb89-488c-d7d46c6dbb91 [Follower]" leader= +testLoggerAdapter verbose: server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91: 2023-08-02T15:12:55.376Z [WARN] server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91: no known peers, aborting election + testing.go:149: server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91: 2023-08-02T15:12:55.376Z [WARN] server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91: no known peers, aborting election +testLoggerAdapter verbose: server-bf17074b-abfd-043e-0171-489e92d0ccb3: 2023-08-02T15:12:55.399Z [WARN] server-bf17074b-abfd-043e-0171-489e92d0ccb3: no known peers, aborting election + testing.go:149: server-bf17074b-abfd-043e-0171-489e92d0ccb3: 2023-08-02T15:12:55.399Z [WARN] server-bf17074b-abfd-043e-0171-489e92d0ccb3: no known peers, aborting election +testLoggerAdapter verbose: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.420Z [WARN] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.420Z [WARN] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.420Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: entering candidate state: node="Node at 9a6571b0-ba9b-6e3e-6646-3a21e92ff67d [Candidate]" term=2 + testing.go:149: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.420Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: entering candidate state: node="Node at 9a6571b0-ba9b-6e3e-6646-3a21e92ff67d [Candidate]" term=2 +testLoggerAdapter verbose: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.420Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: election won: tally=1 + testing.go:149: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.420Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: election won: tally=1 +testLoggerAdapter verbose: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.420Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: entering leader state: leader="Node at 9a6571b0-ba9b-6e3e-6646-3a21e92ff67d [Leader]" + testing.go:149: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.420Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: entering leader state: leader="Node at 9a6571b0-ba9b-6e3e-6646-3a21e92ff67d [Leader]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:55.526Z [INFO] stable state for Leader reached at 2023-08-02 15:12:55.42059634 +0000 UTC m=+7.516532684 (1 nodes), 95.657028ms from start of poll, 96.334108ms from cluster start. Timeout at 2023-08-02 15:12:55.526351416 +0000 UTC m=+7.622287751, 105.755067ms after stability + testing.go:149: cluster: 2023-08-02T15:12:55.526Z [INFO] stable state for Leader reached at 2023-08-02 15:12:55.42059634 +0000 UTC m=+7.516532684 (1 nodes), 95.657028ms from start of poll, 96.334108ms from cluster start. Timeout at 2023-08-02 15:12:55.526351416 +0000 UTC m=+7.622287751, 105.755067ms after stability +testLoggerAdapter verbose: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.526Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: updating configuration: command=AddStaging server-id=server-bf17074b-abfd-043e-0171-489e92d0ccb3 server-addr=bf17074b-abfd-043e-0171-489e92d0ccb3 servers="[{Suffrage:Voter ID:server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d Address:9a6571b0-ba9b-6e3e-6646-3a21e92ff67d} {Suffrage:Voter ID:server-bf17074b-abfd-043e-0171-489e92d0ccb3 Address:bf17074b-abfd-043e-0171-489e92d0ccb3}]" + testing.go:149: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.526Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: updating configuration: command=AddStaging server-id=server-bf17074b-abfd-043e-0171-489e92d0ccb3 server-addr=bf17074b-abfd-043e-0171-489e92d0ccb3 servers="[{Suffrage:Voter ID:server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d Address:9a6571b0-ba9b-6e3e-6646-3a21e92ff67d} {Suffrage:Voter ID:server-bf17074b-abfd-043e-0171-489e92d0ccb3 Address:bf17074b-abfd-043e-0171-489e92d0ccb3}]" +testLoggerAdapter verbose: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.526Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: added peer, starting replication: peer=server-bf17074b-abfd-043e-0171-489e92d0ccb3 + testing.go:149: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.526Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: added peer, starting replication: peer=server-bf17074b-abfd-043e-0171-489e92d0ccb3 +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:55.526Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:12:55.526Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-bf17074b-abfd-043e-0171-489e92d0ccb3: 2023-08-02T15:12:55.526Z [WARN] server-bf17074b-abfd-043e-0171-489e92d0ccb3: failed to get previous log: previous-index=3 last-index=0 error="log not found" + testing.go:149: server-bf17074b-abfd-043e-0171-489e92d0ccb3: 2023-08-02T15:12:55.526Z [WARN] server-bf17074b-abfd-043e-0171-489e92d0ccb3: failed to get previous log: previous-index=3 last-index=0 error="log not found" +testLoggerAdapter verbose: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.526Z [WARN] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: appendEntries rejected, sending older logs: peer="{Voter server-bf17074b-abfd-043e-0171-489e92d0ccb3 bf17074b-abfd-043e-0171-489e92d0ccb3}" next=1 + testing.go:149: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.526Z [WARN] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: appendEntries rejected, sending older logs: peer="{Voter server-bf17074b-abfd-043e-0171-489e92d0ccb3 bf17074b-abfd-043e-0171-489e92d0ccb3}" next=1 +testLoggerAdapter verbose: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.526Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: pipelining replication: peer="{Voter server-bf17074b-abfd-043e-0171-489e92d0ccb3 bf17074b-abfd-043e-0171-489e92d0ccb3}" + testing.go:149: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.526Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: pipelining replication: peer="{Voter server-bf17074b-abfd-043e-0171-489e92d0ccb3 bf17074b-abfd-043e-0171-489e92d0ccb3}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:55.632Z [INFO] stable state for Leader reached at 2023-08-02 15:12:55.526786437 +0000 UTC m=+7.622722763 (1 nodes), 925ns from start of poll, 202.524187ms from cluster start. Timeout at 2023-08-02 15:12:55.632153074 +0000 UTC m=+7.728089479, 105.366716ms after stability + testing.go:149: cluster: 2023-08-02T15:12:55.632Z [INFO] stable state for Leader reached at 2023-08-02 15:12:55.526786437 +0000 UTC m=+7.622722763 (1 nodes), 925ns from start of poll, 202.524187ms from cluster start. Timeout at 2023-08-02 15:12:55.632153074 +0000 UTC m=+7.728089479, 105.366716ms after stability +testLoggerAdapter verbose: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.632Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: updating configuration: command=AddStaging server-id=server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91 server-addr=157a3fd6-85d6-fb89-488c-d7d46c6dbb91 servers="[{Suffrage:Voter ID:server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d Address:9a6571b0-ba9b-6e3e-6646-3a21e92ff67d} {Suffrage:Voter ID:server-bf17074b-abfd-043e-0171-489e92d0ccb3 Address:bf17074b-abfd-043e-0171-489e92d0ccb3} {Suffrage:Voter ID:server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91 Address:157a3fd6-85d6-fb89-488c-d7d46c6dbb91}]" + testing.go:149: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.632Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: updating configuration: command=AddStaging server-id=server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91 server-addr=157a3fd6-85d6-fb89-488c-d7d46c6dbb91 servers="[{Suffrage:Voter ID:server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d Address:9a6571b0-ba9b-6e3e-6646-3a21e92ff67d} {Suffrage:Voter ID:server-bf17074b-abfd-043e-0171-489e92d0ccb3 Address:bf17074b-abfd-043e-0171-489e92d0ccb3} {Suffrage:Voter ID:server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91 Address:157a3fd6-85d6-fb89-488c-d7d46c6dbb91}]" +testLoggerAdapter verbose: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.632Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: added peer, starting replication: peer=server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91 + testing.go:149: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.632Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: added peer, starting replication: peer=server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91 +testLoggerAdapter verbose: server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91: 2023-08-02T15:12:55.632Z [WARN] server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91: failed to get previous log: previous-index=4 last-index=0 error="log not found" + testing.go:149: server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91: 2023-08-02T15:12:55.632Z [WARN] server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91: failed to get previous log: previous-index=4 last-index=0 error="log not found" +testLoggerAdapter verbose: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.632Z [WARN] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: appendEntries rejected, sending older logs: peer="{Voter server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91 157a3fd6-85d6-fb89-488c-d7d46c6dbb91}" next=1 + testing.go:149: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.632Z [WARN] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: appendEntries rejected, sending older logs: peer="{Voter server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91 157a3fd6-85d6-fb89-488c-d7d46c6dbb91}" next=1 +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:55.632Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.632Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: pipelining replication: peer="{Voter server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91 157a3fd6-85d6-fb89-488c-d7d46c6dbb91}" + testing.go:149: cluster: 2023-08-02T15:12:55.632Z [INFO] starting stability test: raft-state=Leader + testing.go:149: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.632Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: pipelining replication: peer="{Voter server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91 157a3fd6-85d6-fb89-488c-d7d46c6dbb91}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:55.738Z [INFO] stable state for Leader reached at 2023-08-02 15:12:55.632792751 +0000 UTC m=+7.728729079 (1 nodes), 4.907µs from start of poll, 308.530503ms from cluster start. Timeout at 2023-08-02 15:12:55.738092604 +0000 UTC m=+7.834028930, 105.299851ms after stability + testing.go:149: cluster: 2023-08-02T15:12:55.738Z [INFO] stable state for Leader reached at 2023-08-02 15:12:55.632792751 +0000 UTC m=+7.728729079 (1 nodes), 4.907µs from start of poll, 308.530503ms from cluster start. Timeout at 2023-08-02 15:12:55.738092604 +0000 UTC m=+7.834028930, 105.299851ms after stability +testLoggerAdapter verbose: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.738Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: aborting pipeline replication: peer="{Voter server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91 157a3fd6-85d6-fb89-488c-d7d46c6dbb91}" + testing.go:149: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.738Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: aborting pipeline replication: peer="{Voter server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91 157a3fd6-85d6-fb89-488c-d7d46c6dbb91}" +testLoggerAdapter verbose: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.738Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: aborting pipeline replication: peer="{Voter server-bf17074b-abfd-043e-0171-489e92d0ccb3 bf17074b-abfd-043e-0171-489e92d0ccb3}" + testing.go:149: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.738Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: aborting pipeline replication: peer="{Voter server-bf17074b-abfd-043e-0171-489e92d0ccb3 bf17074b-abfd-043e-0171-489e92d0ccb3}" +--- PASS: TestRaft_JoinNode_ConfigStore (0.41s) === RUN TestRaft_RemoveFollower -testLoggerAdapter verbose: server-c2c71fb1-8177-025b-924f-804d57e2b60e: 2024-09-03T21:34:26.677Z [INFO] server-c2c71fb1-8177-025b-924f-804d57e2b60e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2c71fb1-8177-025b-924f-804d57e2b60e Address:c2c71fb1-8177-025b-924f-804d57e2b60e} {Suffrage:Voter ID:server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e Address:5f69425d-95bd-60c4-0c0f-0546f57d9a3e} {Suffrage:Voter ID:server-9623dfb5-9cb2-854f-13da-2577b8a6e365 Address:9623dfb5-9cb2-854f-13da-2577b8a6e365}]" - testing.go:149: server-c2c71fb1-8177-025b-924f-804d57e2b60e: 2024-09-03T21:34:26.677Z [INFO] server-c2c71fb1-8177-025b-924f-804d57e2b60e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2c71fb1-8177-025b-924f-804d57e2b60e Address:c2c71fb1-8177-025b-924f-804d57e2b60e} {Suffrage:Voter ID:server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e Address:5f69425d-95bd-60c4-0c0f-0546f57d9a3e} {Suffrage:Voter ID:server-9623dfb5-9cb2-854f-13da-2577b8a6e365 Address:9623dfb5-9cb2-854f-13da-2577b8a6e365}]" -testLoggerAdapter verbose: server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: 2024-09-03T21:34:26.677Z [INFO] server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2c71fb1-8177-025b-924f-804d57e2b60e Address:c2c71fb1-8177-025b-924f-804d57e2b60e} {Suffrage:Voter ID:server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e Address:5f69425d-95bd-60c4-0c0f-0546f57d9a3e} {Suffrage:Voter ID:server-9623dfb5-9cb2-854f-13da-2577b8a6e365 Address:9623dfb5-9cb2-854f-13da-2577b8a6e365}]" - testing.go:149: server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: 2024-09-03T21:34:26.677Z [INFO] server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2c71fb1-8177-025b-924f-804d57e2b60e Address:c2c71fb1-8177-025b-924f-804d57e2b60e} {Suffrage:Voter ID:server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e Address:5f69425d-95bd-60c4-0c0f-0546f57d9a3e} {Suffrage:Voter ID:server-9623dfb5-9cb2-854f-13da-2577b8a6e365 Address:9623dfb5-9cb2-854f-13da-2577b8a6e365}]" -testLoggerAdapter verbose: server-9623dfb5-9cb2-854f-13da-2577b8a6e365: 2024-09-03T21:34:26.677Z [INFO] server-9623dfb5-9cb2-854f-13da-2577b8a6e365: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2c71fb1-8177-025b-924f-804d57e2b60e Address:c2c71fb1-8177-025b-924f-804d57e2b60e} {Suffrage:Voter ID:server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e Address:5f69425d-95bd-60c4-0c0f-0546f57d9a3e} {Suffrage:Voter ID:server-9623dfb5-9cb2-854f-13da-2577b8a6e365 Address:9623dfb5-9cb2-854f-13da-2577b8a6e365}]" - testing.go:149: server-9623dfb5-9cb2-854f-13da-2577b8a6e365: 2024-09-03T21:34:26.677Z [INFO] server-9623dfb5-9cb2-854f-13da-2577b8a6e365: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2c71fb1-8177-025b-924f-804d57e2b60e Address:c2c71fb1-8177-025b-924f-804d57e2b60e} {Suffrage:Voter ID:server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e Address:5f69425d-95bd-60c4-0c0f-0546f57d9a3e} {Suffrage:Voter ID:server-9623dfb5-9cb2-854f-13da-2577b8a6e365 Address:9623dfb5-9cb2-854f-13da-2577b8a6e365}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:26.677Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:26.677Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-c2c71fb1-8177-025b-924f-804d57e2b60e: 2024-09-03T21:34:26.677Z [INFO] server-c2c71fb1-8177-025b-924f-804d57e2b60e: entering follower state: follower="Node at c2c71fb1-8177-025b-924f-804d57e2b60e [Follower]" leader= - testing.go:149: server-c2c71fb1-8177-025b-924f-804d57e2b60e: 2024-09-03T21:34:26.677Z [INFO] server-c2c71fb1-8177-025b-924f-804d57e2b60e: entering follower state: follower="Node at c2c71fb1-8177-025b-924f-804d57e2b60e [Follower]" leader= -testLoggerAdapter verbose: server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: 2024-09-03T21:34:26.677Z [INFO] server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: entering follower state: follower="Node at 5f69425d-95bd-60c4-0c0f-0546f57d9a3e [Follower]" leader= - testing.go:149: server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: 2024-09-03T21:34:26.677Z [INFO] server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: entering follower state: follower="Node at 5f69425d-95bd-60c4-0c0f-0546f57d9a3e [Follower]" leader= -testLoggerAdapter verbose: server-9623dfb5-9cb2-854f-13da-2577b8a6e365: 2024-09-03T21:34:26.677Z [INFO] server-9623dfb5-9cb2-854f-13da-2577b8a6e365: entering follower state: follower="Node at 9623dfb5-9cb2-854f-13da-2577b8a6e365 [Follower]" leader= - testing.go:149: server-9623dfb5-9cb2-854f-13da-2577b8a6e365: 2024-09-03T21:34:26.677Z [INFO] server-9623dfb5-9cb2-854f-13da-2577b8a6e365: entering follower state: follower="Node at 9623dfb5-9cb2-854f-13da-2577b8a6e365 [Follower]" leader= -testLoggerAdapter verbose: server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: 2024-09-03T21:34:26.748Z [WARN] server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: 2024-09-03T21:34:26.748Z [WARN] server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: 2024-09-03T21:34:26.748Z [INFO] server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: entering candidate state: node="Node at 5f69425d-95bd-60c4-0c0f-0546f57d9a3e [Candidate]" term=2 - testing.go:149: server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: 2024-09-03T21:34:26.748Z [INFO] server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: entering candidate state: node="Node at 5f69425d-95bd-60c4-0c0f-0546f57d9a3e [Candidate]" term=2 -testLoggerAdapter verbose: server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: 2024-09-03T21:34:26.748Z [INFO] server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: election won: tally=2 - testing.go:149: server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: 2024-09-03T21:34:26.748Z [INFO] server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: election won: tally=2 -testLoggerAdapter verbose: server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: 2024-09-03T21:34:26.748Z [INFO] server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: entering leader state: leader="Node at 5f69425d-95bd-60c4-0c0f-0546f57d9a3e [Leader]" - testing.go:149: server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: 2024-09-03T21:34:26.748Z [INFO] server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: entering leader state: leader="Node at 5f69425d-95bd-60c4-0c0f-0546f57d9a3e [Leader]" -testLoggerAdapter verbose: server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: 2024-09-03T21:34:26.748Z [INFO] server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: added peer, starting replication: peer=server-c2c71fb1-8177-025b-924f-804d57e2b60e - testing.go:149: server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: 2024-09-03T21:34:26.748Z [INFO] server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: added peer, starting replication: peer=server-c2c71fb1-8177-025b-924f-804d57e2b60e -testLoggerAdapter verbose: server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: 2024-09-03T21:34:26.749Z [INFO] server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: added peer, starting replication: peer=server-9623dfb5-9cb2-854f-13da-2577b8a6e365 - testing.go:149: server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: 2024-09-03T21:34:26.749Z [INFO] server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: added peer, starting replication: peer=server-9623dfb5-9cb2-854f-13da-2577b8a6e365 -testLoggerAdapter verbose: server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: 2024-09-03T21:34:26.749Z [INFO] server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: pipelining replication: peer="{Voter server-9623dfb5-9cb2-854f-13da-2577b8a6e365 9623dfb5-9cb2-854f-13da-2577b8a6e365}" - testing.go:149: server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: 2024-09-03T21:34:26.749Z [INFO] server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: pipelining replication: peer="{Voter server-9623dfb5-9cb2-854f-13da-2577b8a6e365 9623dfb5-9cb2-854f-13da-2577b8a6e365}" -testLoggerAdapter verbose: server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: 2024-09-03T21:34:26.749Z [INFO] server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: pipelining replication: peer="{Voter server-c2c71fb1-8177-025b-924f-804d57e2b60e c2c71fb1-8177-025b-924f-804d57e2b60e}" - testing.go:149: server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: 2024-09-03T21:34:26.749Z [INFO] server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: pipelining replication: peer="{Voter server-c2c71fb1-8177-025b-924f-804d57e2b60e c2c71fb1-8177-025b-924f-804d57e2b60e}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:26.854Z [INFO] stable state for Leader reached at 2024-09-03 21:34:26.74933404 +0000 UTC m=+7.795991180 (1 nodes), 72.019571ms from start of poll, 72.417839ms from cluster start. Timeout at 2024-09-03 21:34:26.854413025 +0000 UTC m=+7.901070155, 105.078975ms after stability - testing.go:149: cluster: 2024-09-03T21:34:26.854Z [INFO] stable state for Leader reached at 2024-09-03 21:34:26.74933404 +0000 UTC m=+7.795991180 (1 nodes), 72.019571ms from start of poll, 72.417839ms from cluster start. Timeout at 2024-09-03 21:34:26.854413025 +0000 UTC m=+7.901070155, 105.078975ms after stability -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:26.859Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-09-03T21:34:26.859Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:26.964Z [INFO] stable state for Follower reached at 2024-09-03 21:34:26.859746985 +0000 UTC m=+7.906404143 (2 nodes), 1.122µs from start of poll, 182.830802ms from cluster start. Timeout at 2024-09-03 21:34:26.964840448 +0000 UTC m=+8.011497590, 105.093447ms after stability - testing.go:149: cluster: 2024-09-03T21:34:26.964Z [INFO] stable state for Follower reached at 2024-09-03 21:34:26.859746985 +0000 UTC m=+7.906404143 (2 nodes), 1.122µs from start of poll, 182.830802ms from cluster start. Timeout at 2024-09-03 21:34:26.964840448 +0000 UTC m=+8.011497590, 105.093447ms after stability -testLoggerAdapter verbose: server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: 2024-09-03T21:34:26.965Z [INFO] server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: updating configuration: command=RemoveServer server-id=server-c2c71fb1-8177-025b-924f-804d57e2b60e server-addr= servers="[{Suffrage:Voter ID:server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e Address:5f69425d-95bd-60c4-0c0f-0546f57d9a3e} {Suffrage:Voter ID:server-9623dfb5-9cb2-854f-13da-2577b8a6e365 Address:9623dfb5-9cb2-854f-13da-2577b8a6e365}]" - testing.go:149: server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: 2024-09-03T21:34:26.965Z [INFO] server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: updating configuration: command=RemoveServer server-id=server-c2c71fb1-8177-025b-924f-804d57e2b60e server-addr= servers="[{Suffrage:Voter ID:server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e Address:5f69425d-95bd-60c4-0c0f-0546f57d9a3e} {Suffrage:Voter ID:server-9623dfb5-9cb2-854f-13da-2577b8a6e365 Address:9623dfb5-9cb2-854f-13da-2577b8a6e365}]" -testLoggerAdapter verbose: server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: 2024-09-03T21:34:26.965Z [INFO] server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: removed peer, stopping replication: peer=server-c2c71fb1-8177-025b-924f-804d57e2b60e last-index=3 - testing.go:149: server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: 2024-09-03T21:34:26.965Z [INFO] server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: removed peer, stopping replication: peer=server-c2c71fb1-8177-025b-924f-804d57e2b60e last-index=3 -testLoggerAdapter verbose: server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: 2024-09-03T21:34:26.965Z [INFO] server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: aborting pipeline replication: peer="{Voter server-c2c71fb1-8177-025b-924f-804d57e2b60e c2c71fb1-8177-025b-924f-804d57e2b60e}" - testing.go:149: server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: 2024-09-03T21:34:26.965Z [INFO] server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: aborting pipeline replication: peer="{Voter server-c2c71fb1-8177-025b-924f-804d57e2b60e c2c71fb1-8177-025b-924f-804d57e2b60e}" -testLoggerAdapter verbose: server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: 2024-09-03T21:34:27.070Z [INFO] server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: aborting pipeline replication: peer="{Voter server-9623dfb5-9cb2-854f-13da-2577b8a6e365 9623dfb5-9cb2-854f-13da-2577b8a6e365}" - testing.go:149: server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: 2024-09-03T21:34:27.070Z [INFO] server-5f69425d-95bd-60c4-0c0f-0546f57d9a3e: aborting pipeline replication: peer="{Voter server-9623dfb5-9cb2-854f-13da-2577b8a6e365 9623dfb5-9cb2-854f-13da-2577b8a6e365}" +testLoggerAdapter verbose: server-7fad1ebc-7dca-2b0b-423e-052f90989cfb: 2023-08-02T15:12:55.739Z [INFO] server-7fad1ebc-7dca-2b0b-423e-052f90989cfb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7fad1ebc-7dca-2b0b-423e-052f90989cfb Address:7fad1ebc-7dca-2b0b-423e-052f90989cfb} {Suffrage:Voter ID:server-5774e529-6576-711c-31bc-25c62d49a15c Address:5774e529-6576-711c-31bc-25c62d49a15c} {Suffrage:Voter ID:server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a Address:e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a}]" + testing.go:149: server-7fad1ebc-7dca-2b0b-423e-052f90989cfb: 2023-08-02T15:12:55.739Z [INFO] server-7fad1ebc-7dca-2b0b-423e-052f90989cfb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7fad1ebc-7dca-2b0b-423e-052f90989cfb Address:7fad1ebc-7dca-2b0b-423e-052f90989cfb} {Suffrage:Voter ID:server-5774e529-6576-711c-31bc-25c62d49a15c Address:5774e529-6576-711c-31bc-25c62d49a15c} {Suffrage:Voter ID:server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a Address:e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a}]" +testLoggerAdapter verbose: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.739Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7fad1ebc-7dca-2b0b-423e-052f90989cfb Address:7fad1ebc-7dca-2b0b-423e-052f90989cfb} {Suffrage:Voter ID:server-5774e529-6576-711c-31bc-25c62d49a15c Address:5774e529-6576-711c-31bc-25c62d49a15c} {Suffrage:Voter ID:server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a Address:e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a}]" + testing.go:149: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.739Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7fad1ebc-7dca-2b0b-423e-052f90989cfb Address:7fad1ebc-7dca-2b0b-423e-052f90989cfb} {Suffrage:Voter ID:server-5774e529-6576-711c-31bc-25c62d49a15c Address:5774e529-6576-711c-31bc-25c62d49a15c} {Suffrage:Voter ID:server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a Address:e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a}]" +testLoggerAdapter verbose: server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a: 2023-08-02T15:12:55.739Z [INFO] server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7fad1ebc-7dca-2b0b-423e-052f90989cfb Address:7fad1ebc-7dca-2b0b-423e-052f90989cfb} {Suffrage:Voter ID:server-5774e529-6576-711c-31bc-25c62d49a15c Address:5774e529-6576-711c-31bc-25c62d49a15c} {Suffrage:Voter ID:server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a Address:e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a}]" + testing.go:149: server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a: 2023-08-02T15:12:55.739Z [INFO] server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7fad1ebc-7dca-2b0b-423e-052f90989cfb Address:7fad1ebc-7dca-2b0b-423e-052f90989cfb} {Suffrage:Voter ID:server-5774e529-6576-711c-31bc-25c62d49a15c Address:5774e529-6576-711c-31bc-25c62d49a15c} {Suffrage:Voter ID:server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a Address:e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:55.739Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:12:55.739Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-7fad1ebc-7dca-2b0b-423e-052f90989cfb: 2023-08-02T15:12:55.739Z [INFO] server-7fad1ebc-7dca-2b0b-423e-052f90989cfb: entering follower state: follower="Node at 7fad1ebc-7dca-2b0b-423e-052f90989cfb [Follower]" leader= + testing.go:149: server-7fad1ebc-7dca-2b0b-423e-052f90989cfb: 2023-08-02T15:12:55.739Z [INFO] server-7fad1ebc-7dca-2b0b-423e-052f90989cfb: entering follower state: follower="Node at 7fad1ebc-7dca-2b0b-423e-052f90989cfb [Follower]" leader= +testLoggerAdapter verbose: server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a: 2023-08-02T15:12:55.739Z [INFO] server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a: entering follower state: follower="Node at e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a [Follower]" leader= + testing.go:149: server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a: 2023-08-02T15:12:55.739Z [INFO] server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a: entering follower state: follower="Node at e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a [Follower]" leader= +testLoggerAdapter verbose: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.739Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: entering follower state: follower="Node at 5774e529-6576-711c-31bc-25c62d49a15c [Follower]" leader= + testing.go:149: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.739Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: entering follower state: follower="Node at 5774e529-6576-711c-31bc-25c62d49a15c [Follower]" leader= +testLoggerAdapter verbose: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.810Z [WARN] server-5774e529-6576-711c-31bc-25c62d49a15c: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.810Z [WARN] server-5774e529-6576-711c-31bc-25c62d49a15c: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.810Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: entering candidate state: node="Node at 5774e529-6576-711c-31bc-25c62d49a15c [Candidate]" term=2 + testing.go:149: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.810Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: entering candidate state: node="Node at 5774e529-6576-711c-31bc-25c62d49a15c [Candidate]" term=2 +testLoggerAdapter verbose: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.810Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: election won: tally=2 + testing.go:149: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.810Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: election won: tally=2 +testLoggerAdapter verbose: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.810Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: entering leader state: leader="Node at 5774e529-6576-711c-31bc-25c62d49a15c [Leader]" + testing.go:149: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.810Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: entering leader state: leader="Node at 5774e529-6576-711c-31bc-25c62d49a15c [Leader]" +testLoggerAdapter verbose: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.810Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: added peer, starting replication: peer=server-7fad1ebc-7dca-2b0b-423e-052f90989cfb + testing.go:149: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.810Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: added peer, starting replication: peer=server-7fad1ebc-7dca-2b0b-423e-052f90989cfb +testLoggerAdapter verbose: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.810Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: added peer, starting replication: peer=server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a + testing.go:149: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.810Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: added peer, starting replication: peer=server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a +testLoggerAdapter verbose: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.810Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: pipelining replication: peer="{Voter server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a}" + testing.go:149: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.810Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: pipelining replication: peer="{Voter server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a}" +testLoggerAdapter verbose: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.810Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: pipelining replication: peer="{Voter server-7fad1ebc-7dca-2b0b-423e-052f90989cfb 7fad1ebc-7dca-2b0b-423e-052f90989cfb}" + testing.go:149: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.810Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: pipelining replication: peer="{Voter server-7fad1ebc-7dca-2b0b-423e-052f90989cfb 7fad1ebc-7dca-2b0b-423e-052f90989cfb}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:55.915Z [INFO] stable state for Leader reached at 2023-08-02 15:12:55.810624041 +0000 UTC m=+7.906560375 (1 nodes), 70.90321ms from start of poll, 71.384558ms from cluster start. Timeout at 2023-08-02 15:12:55.91569773 +0000 UTC m=+8.011634065, 105.07369ms after stability + testing.go:149: cluster: 2023-08-02T15:12:55.915Z [INFO] stable state for Leader reached at 2023-08-02 15:12:55.810624041 +0000 UTC m=+7.906560375 (1 nodes), 70.90321ms from start of poll, 71.384558ms from cluster start. Timeout at 2023-08-02 15:12:55.91569773 +0000 UTC m=+8.011634065, 105.07369ms after stability +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:55.922Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-08-02T15:12:55.922Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:56.027Z [INFO] stable state for Follower reached at 2023-08-02 15:12:55.92270497 +0000 UTC m=+8.018641307 (2 nodes), 2.03µs from start of poll, 183.46549ms from cluster start. Timeout at 2023-08-02 15:12:56.027732181 +0000 UTC m=+8.123668508, 105.027201ms after stability + testing.go:149: cluster: 2023-08-02T15:12:56.027Z [INFO] stable state for Follower reached at 2023-08-02 15:12:55.92270497 +0000 UTC m=+8.018641307 (2 nodes), 2.03µs from start of poll, 183.46549ms from cluster start. Timeout at 2023-08-02 15:12:56.027732181 +0000 UTC m=+8.123668508, 105.027201ms after stability +testLoggerAdapter verbose: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:56.027Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: updating configuration: command=RemoveServer server-id=server-7fad1ebc-7dca-2b0b-423e-052f90989cfb server-addr= servers="[{Suffrage:Voter ID:server-5774e529-6576-711c-31bc-25c62d49a15c Address:5774e529-6576-711c-31bc-25c62d49a15c} {Suffrage:Voter ID:server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a Address:e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a}]" + testing.go:149: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:56.027Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: updating configuration: command=RemoveServer server-id=server-7fad1ebc-7dca-2b0b-423e-052f90989cfb server-addr= servers="[{Suffrage:Voter ID:server-5774e529-6576-711c-31bc-25c62d49a15c Address:5774e529-6576-711c-31bc-25c62d49a15c} {Suffrage:Voter ID:server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a Address:e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a}]" +testLoggerAdapter verbose: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:56.028Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: removed peer, stopping replication: peer=server-7fad1ebc-7dca-2b0b-423e-052f90989cfb last-index=3 + testing.go:149: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:56.028Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: removed peer, stopping replication: peer=server-7fad1ebc-7dca-2b0b-423e-052f90989cfb last-index=3 +testLoggerAdapter verbose: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:56.028Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: aborting pipeline replication: peer="{Voter server-7fad1ebc-7dca-2b0b-423e-052f90989cfb 7fad1ebc-7dca-2b0b-423e-052f90989cfb}" + testing.go:149: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:56.028Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: aborting pipeline replication: peer="{Voter server-7fad1ebc-7dca-2b0b-423e-052f90989cfb 7fad1ebc-7dca-2b0b-423e-052f90989cfb}" +testLoggerAdapter verbose: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:56.133Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: aborting pipeline replication: peer="{Voter server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a}" + testing.go:149: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:56.133Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: aborting pipeline replication: peer="{Voter server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a}" --- PASS: TestRaft_RemoveFollower (0.39s) === RUN TestRaft_RemoveLeader -testLoggerAdapter verbose: server-ea14af0e-835c-448f-77c9-701e19b26a9e: 2024-09-03T21:34:27.071Z [INFO] server-ea14af0e-835c-448f-77c9-701e19b26a9e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ea14af0e-835c-448f-77c9-701e19b26a9e Address:ea14af0e-835c-448f-77c9-701e19b26a9e} {Suffrage:Voter ID:server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8 Address:0d47dad8-69a2-2b1a-01a6-635a5e5f49d8} {Suffrage:Voter ID:server-78f9132a-2896-3d88-a901-efa749acc57a Address:78f9132a-2896-3d88-a901-efa749acc57a}]" - testing.go:149: server-ea14af0e-835c-448f-77c9-701e19b26a9e: 2024-09-03T21:34:27.071Z [INFO] server-ea14af0e-835c-448f-77c9-701e19b26a9e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ea14af0e-835c-448f-77c9-701e19b26a9e Address:ea14af0e-835c-448f-77c9-701e19b26a9e} {Suffrage:Voter ID:server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8 Address:0d47dad8-69a2-2b1a-01a6-635a5e5f49d8} {Suffrage:Voter ID:server-78f9132a-2896-3d88-a901-efa749acc57a Address:78f9132a-2896-3d88-a901-efa749acc57a}]" -testLoggerAdapter verbose: server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8: 2024-09-03T21:34:27.071Z [INFO] server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ea14af0e-835c-448f-77c9-701e19b26a9e Address:ea14af0e-835c-448f-77c9-701e19b26a9e} {Suffrage:Voter ID:server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8 Address:0d47dad8-69a2-2b1a-01a6-635a5e5f49d8} {Suffrage:Voter ID:server-78f9132a-2896-3d88-a901-efa749acc57a Address:78f9132a-2896-3d88-a901-efa749acc57a}]" - testing.go:149: server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8: 2024-09-03T21:34:27.071Z [INFO] server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ea14af0e-835c-448f-77c9-701e19b26a9e Address:ea14af0e-835c-448f-77c9-701e19b26a9e} {Suffrage:Voter ID:server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8 Address:0d47dad8-69a2-2b1a-01a6-635a5e5f49d8} {Suffrage:Voter ID:server-78f9132a-2896-3d88-a901-efa749acc57a Address:78f9132a-2896-3d88-a901-efa749acc57a}]" -testLoggerAdapter verbose: server-78f9132a-2896-3d88-a901-efa749acc57a: 2024-09-03T21:34:27.071Z [INFO] server-78f9132a-2896-3d88-a901-efa749acc57a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ea14af0e-835c-448f-77c9-701e19b26a9e Address:ea14af0e-835c-448f-77c9-701e19b26a9e} {Suffrage:Voter ID:server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8 Address:0d47dad8-69a2-2b1a-01a6-635a5e5f49d8} {Suffrage:Voter ID:server-78f9132a-2896-3d88-a901-efa749acc57a Address:78f9132a-2896-3d88-a901-efa749acc57a}]" - testing.go:149: server-78f9132a-2896-3d88-a901-efa749acc57a: 2024-09-03T21:34:27.071Z [INFO] server-78f9132a-2896-3d88-a901-efa749acc57a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ea14af0e-835c-448f-77c9-701e19b26a9e Address:ea14af0e-835c-448f-77c9-701e19b26a9e} {Suffrage:Voter ID:server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8 Address:0d47dad8-69a2-2b1a-01a6-635a5e5f49d8} {Suffrage:Voter ID:server-78f9132a-2896-3d88-a901-efa749acc57a Address:78f9132a-2896-3d88-a901-efa749acc57a}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:27.071Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:27.071Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-ea14af0e-835c-448f-77c9-701e19b26a9e: 2024-09-03T21:34:27.071Z [INFO] server-ea14af0e-835c-448f-77c9-701e19b26a9e: entering follower state: follower="Node at ea14af0e-835c-448f-77c9-701e19b26a9e [Follower]" leader= - testing.go:149: server-ea14af0e-835c-448f-77c9-701e19b26a9e: 2024-09-03T21:34:27.071Z [INFO] server-ea14af0e-835c-448f-77c9-701e19b26a9e: entering follower state: follower="Node at ea14af0e-835c-448f-77c9-701e19b26a9e [Follower]" leader= -testLoggerAdapter verbose: server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8: 2024-09-03T21:34:27.071Z [INFO] server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8: entering follower state: follower="Node at 0d47dad8-69a2-2b1a-01a6-635a5e5f49d8 [Follower]" leader= - testing.go:149: server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8: 2024-09-03T21:34:27.071Z [INFO] server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8: entering follower state: follower="Node at 0d47dad8-69a2-2b1a-01a6-635a5e5f49d8 [Follower]" leader= -testLoggerAdapter verbose: server-78f9132a-2896-3d88-a901-efa749acc57a: 2024-09-03T21:34:27.071Z [INFO] server-78f9132a-2896-3d88-a901-efa749acc57a: entering follower state: follower="Node at 78f9132a-2896-3d88-a901-efa749acc57a [Follower]" leader= - testing.go:149: server-78f9132a-2896-3d88-a901-efa749acc57a: 2024-09-03T21:34:27.071Z [INFO] server-78f9132a-2896-3d88-a901-efa749acc57a: entering follower state: follower="Node at 78f9132a-2896-3d88-a901-efa749acc57a [Follower]" leader= -testLoggerAdapter verbose: server-78f9132a-2896-3d88-a901-efa749acc57a: 2024-09-03T21:34:27.145Z [WARN] server-78f9132a-2896-3d88-a901-efa749acc57a: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-78f9132a-2896-3d88-a901-efa749acc57a: 2024-09-03T21:34:27.145Z [WARN] server-78f9132a-2896-3d88-a901-efa749acc57a: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-78f9132a-2896-3d88-a901-efa749acc57a: 2024-09-03T21:34:27.145Z [INFO] server-78f9132a-2896-3d88-a901-efa749acc57a: entering candidate state: node="Node at 78f9132a-2896-3d88-a901-efa749acc57a [Candidate]" term=2 - testing.go:149: server-78f9132a-2896-3d88-a901-efa749acc57a: 2024-09-03T21:34:27.145Z [INFO] server-78f9132a-2896-3d88-a901-efa749acc57a: entering candidate state: node="Node at 78f9132a-2896-3d88-a901-efa749acc57a [Candidate]" term=2 -testLoggerAdapter verbose: server-78f9132a-2896-3d88-a901-efa749acc57a: 2024-09-03T21:34:27.145Z [INFO] server-78f9132a-2896-3d88-a901-efa749acc57a: election won: tally=2 - testing.go:149: server-78f9132a-2896-3d88-a901-efa749acc57a: 2024-09-03T21:34:27.145Z [INFO] server-78f9132a-2896-3d88-a901-efa749acc57a: election won: tally=2 -testLoggerAdapter verbose: server-78f9132a-2896-3d88-a901-efa749acc57a: 2024-09-03T21:34:27.145Z [INFO] server-78f9132a-2896-3d88-a901-efa749acc57a: entering leader state: leader="Node at 78f9132a-2896-3d88-a901-efa749acc57a [Leader]" - testing.go:149: server-78f9132a-2896-3d88-a901-efa749acc57a: 2024-09-03T21:34:27.145Z [INFO] server-78f9132a-2896-3d88-a901-efa749acc57a: entering leader state: leader="Node at 78f9132a-2896-3d88-a901-efa749acc57a [Leader]" -testLoggerAdapter verbose: server-78f9132a-2896-3d88-a901-efa749acc57a: 2024-09-03T21:34:27.145Z [INFO] server-78f9132a-2896-3d88-a901-efa749acc57a: added peer, starting replication: peer=server-ea14af0e-835c-448f-77c9-701e19b26a9e - testing.go:149: server-78f9132a-2896-3d88-a901-efa749acc57a: 2024-09-03T21:34:27.145Z [INFO] server-78f9132a-2896-3d88-a901-efa749acc57a: added peer, starting replication: peer=server-ea14af0e-835c-448f-77c9-701e19b26a9e -testLoggerAdapter verbose: server-78f9132a-2896-3d88-a901-efa749acc57a: 2024-09-03T21:34:27.145Z [INFO] server-78f9132a-2896-3d88-a901-efa749acc57a: added peer, starting replication: peer=server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8 - testing.go:149: server-78f9132a-2896-3d88-a901-efa749acc57a: 2024-09-03T21:34:27.145Z [INFO] server-78f9132a-2896-3d88-a901-efa749acc57a: added peer, starting replication: peer=server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8 -testLoggerAdapter verbose: server-78f9132a-2896-3d88-a901-efa749acc57a: 2024-09-03T21:34:27.145Z [INFO] server-78f9132a-2896-3d88-a901-efa749acc57a: pipelining replication: peer="{Voter server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8 0d47dad8-69a2-2b1a-01a6-635a5e5f49d8}" - testing.go:149: server-78f9132a-2896-3d88-a901-efa749acc57a: 2024-09-03T21:34:27.145Z [INFO] server-78f9132a-2896-3d88-a901-efa749acc57a: pipelining replication: peer="{Voter server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8 0d47dad8-69a2-2b1a-01a6-635a5e5f49d8}" -testLoggerAdapter verbose: server-78f9132a-2896-3d88-a901-efa749acc57a: 2024-09-03T21:34:27.145Z [INFO] server-78f9132a-2896-3d88-a901-efa749acc57a: pipelining replication: peer="{Voter server-ea14af0e-835c-448f-77c9-701e19b26a9e ea14af0e-835c-448f-77c9-701e19b26a9e}" - testing.go:149: server-78f9132a-2896-3d88-a901-efa749acc57a: 2024-09-03T21:34:27.145Z [INFO] server-78f9132a-2896-3d88-a901-efa749acc57a: pipelining replication: peer="{Voter server-ea14af0e-835c-448f-77c9-701e19b26a9e ea14af0e-835c-448f-77c9-701e19b26a9e}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:27.250Z [INFO] stable state for Leader reached at 2024-09-03 21:34:27.145741908 +0000 UTC m=+8.192399051 (1 nodes), 74.135591ms from start of poll, 74.548925ms from cluster start. Timeout at 2024-09-03 21:34:27.250816032 +0000 UTC m=+8.297473164, 105.074113ms after stability - testing.go:149: cluster: 2024-09-03T21:34:27.250Z [INFO] stable state for Leader reached at 2024-09-03 21:34:27.145741908 +0000 UTC m=+8.192399051 (1 nodes), 74.135591ms from start of poll, 74.548925ms from cluster start. Timeout at 2024-09-03 21:34:27.250816032 +0000 UTC m=+8.297473164, 105.074113ms after stability -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:27.256Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-09-03T21:34:27.256Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:27.361Z [INFO] stable state for Follower reached at 2024-09-03 21:34:27.256080507 +0000 UTC m=+8.302737639 (2 nodes), 1.65µs from start of poll, 184.887513ms from cluster start. Timeout at 2024-09-03 21:34:27.361180072 +0000 UTC m=+8.407837203, 105.099564ms after stability - testing.go:149: cluster: 2024-09-03T21:34:27.361Z [INFO] stable state for Follower reached at 2024-09-03 21:34:27.256080507 +0000 UTC m=+8.302737639 (2 nodes), 1.65µs from start of poll, 184.887513ms from cluster start. Timeout at 2024-09-03 21:34:27.361180072 +0000 UTC m=+8.407837203, 105.099564ms after stability -testLoggerAdapter verbose: server-78f9132a-2896-3d88-a901-efa749acc57a: 2024-09-03T21:34:27.361Z [INFO] server-78f9132a-2896-3d88-a901-efa749acc57a: updating configuration: command=RemoveServer server-id=server-78f9132a-2896-3d88-a901-efa749acc57a server-addr= servers="[{Suffrage:Voter ID:server-ea14af0e-835c-448f-77c9-701e19b26a9e Address:ea14af0e-835c-448f-77c9-701e19b26a9e} {Suffrage:Voter ID:server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8 Address:0d47dad8-69a2-2b1a-01a6-635a5e5f49d8}]" - testing.go:149: server-78f9132a-2896-3d88-a901-efa749acc57a: 2024-09-03T21:34:27.361Z [INFO] server-78f9132a-2896-3d88-a901-efa749acc57a: updating configuration: command=RemoveServer server-id=server-78f9132a-2896-3d88-a901-efa749acc57a server-addr= servers="[{Suffrage:Voter ID:server-ea14af0e-835c-448f-77c9-701e19b26a9e Address:ea14af0e-835c-448f-77c9-701e19b26a9e} {Suffrage:Voter ID:server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8 Address:0d47dad8-69a2-2b1a-01a6-635a5e5f49d8}]" -testLoggerAdapter verbose: server-78f9132a-2896-3d88-a901-efa749acc57a: 2024-09-03T21:34:27.361Z [INFO] server-78f9132a-2896-3d88-a901-efa749acc57a: removed ourself, shutting down - testing.go:149: server-78f9132a-2896-3d88-a901-efa749acc57a: 2024-09-03T21:34:27.361Z [INFO] server-78f9132a-2896-3d88-a901-efa749acc57a: removed ourself, shutting down -testLoggerAdapter verbose: server-78f9132a-2896-3d88-a901-efa749acc57a: 2024-09-03T21:34:27.361Z [INFO] server-78f9132a-2896-3d88-a901-efa749acc57a: aborting pipeline replication: peer="{Voter server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8 0d47dad8-69a2-2b1a-01a6-635a5e5f49d8}" - testing.go:149: server-78f9132a-2896-3d88-a901-efa749acc57a: 2024-09-03T21:34:27.361Z [INFO] server-78f9132a-2896-3d88-a901-efa749acc57a: aborting pipeline replication: peer="{Voter server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8 0d47dad8-69a2-2b1a-01a6-635a5e5f49d8}" -testLoggerAdapter verbose: server-78f9132a-2896-3d88-a901-efa749acc57a: 2024-09-03T21:34:27.361Z [INFO] server-78f9132a-2896-3d88-a901-efa749acc57a: aborting pipeline replication: peer="{Voter server-ea14af0e-835c-448f-77c9-701e19b26a9e ea14af0e-835c-448f-77c9-701e19b26a9e}" - testing.go:149: server-78f9132a-2896-3d88-a901-efa749acc57a: 2024-09-03T21:34:27.361Z [INFO] server-78f9132a-2896-3d88-a901-efa749acc57a: aborting pipeline replication: peer="{Voter server-ea14af0e-835c-448f-77c9-701e19b26a9e ea14af0e-835c-448f-77c9-701e19b26a9e}" -testLoggerAdapter verbose: server-ea14af0e-835c-448f-77c9-701e19b26a9e: 2024-09-03T21:34:27.426Z [WARN] server-ea14af0e-835c-448f-77c9-701e19b26a9e: heartbeat timeout reached, starting election: last-leader=78f9132a-2896-3d88-a901-efa749acc57a - testing.go:149: server-ea14af0e-835c-448f-77c9-701e19b26a9e: 2024-09-03T21:34:27.426Z [WARN] server-ea14af0e-835c-448f-77c9-701e19b26a9e: heartbeat timeout reached, starting election: last-leader=78f9132a-2896-3d88-a901-efa749acc57a -testLoggerAdapter verbose: server-ea14af0e-835c-448f-77c9-701e19b26a9e: 2024-09-03T21:34:27.426Z [INFO] server-ea14af0e-835c-448f-77c9-701e19b26a9e: entering candidate state: node="Node at ea14af0e-835c-448f-77c9-701e19b26a9e [Candidate]" term=3 - testing.go:149: server-ea14af0e-835c-448f-77c9-701e19b26a9e: 2024-09-03T21:34:27.426Z [INFO] server-ea14af0e-835c-448f-77c9-701e19b26a9e: entering candidate state: node="Node at ea14af0e-835c-448f-77c9-701e19b26a9e [Candidate]" term=3 -testLoggerAdapter verbose: server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8: 2024-09-03T21:34:27.426Z [WARN] server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8: rejecting vote request since we have a leader: from=ea14af0e-835c-448f-77c9-701e19b26a9e leader=78f9132a-2896-3d88-a901-efa749acc57a - testing.go:149: server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8: 2024-09-03T21:34:27.426Z [WARN] server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8: rejecting vote request since we have a leader: from=ea14af0e-835c-448f-77c9-701e19b26a9e leader=78f9132a-2896-3d88-a901-efa749acc57a -testLoggerAdapter verbose: server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8: 2024-09-03T21:34:27.480Z [WARN] server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8: heartbeat timeout reached, starting election: last-leader=78f9132a-2896-3d88-a901-efa749acc57a - testing.go:149: server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8: 2024-09-03T21:34:27.480Z [WARN] server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8: heartbeat timeout reached, starting election: last-leader=78f9132a-2896-3d88-a901-efa749acc57a -testLoggerAdapter verbose: server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8: 2024-09-03T21:34:27.480Z [INFO] server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8: entering candidate state: node="Node at 0d47dad8-69a2-2b1a-01a6-635a5e5f49d8 [Candidate]" term=3 - testing.go:149: server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8: 2024-09-03T21:34:27.480Z [INFO] server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8: entering candidate state: node="Node at 0d47dad8-69a2-2b1a-01a6-635a5e5f49d8 [Candidate]" term=3 -testLoggerAdapter verbose: server-ea14af0e-835c-448f-77c9-701e19b26a9e: 2024-09-03T21:34:27.480Z [INFO] server-ea14af0e-835c-448f-77c9-701e19b26a9e: duplicate requestVote for same term: term=3 - testing.go:149: server-ea14af0e-835c-448f-77c9-701e19b26a9e: 2024-09-03T21:34:27.480Z [INFO] server-ea14af0e-835c-448f-77c9-701e19b26a9e: duplicate requestVote for same term: term=3 -testLoggerAdapter verbose: server-ea14af0e-835c-448f-77c9-701e19b26a9e: 2024-09-03T21:34:27.518Z [WARN] server-ea14af0e-835c-448f-77c9-701e19b26a9e: Election timeout reached, restarting election - testing.go:149: server-ea14af0e-835c-448f-77c9-701e19b26a9e: 2024-09-03T21:34:27.518Z [WARN] server-ea14af0e-835c-448f-77c9-701e19b26a9e: Election timeout reached, restarting election -testLoggerAdapter verbose: server-ea14af0e-835c-448f-77c9-701e19b26a9e: 2024-09-03T21:34:27.518Z [INFO] server-ea14af0e-835c-448f-77c9-701e19b26a9e: entering candidate state: node="Node at ea14af0e-835c-448f-77c9-701e19b26a9e [Candidate]" term=4 - testing.go:149: server-ea14af0e-835c-448f-77c9-701e19b26a9e: 2024-09-03T21:34:27.518Z [INFO] server-ea14af0e-835c-448f-77c9-701e19b26a9e: entering candidate state: node="Node at ea14af0e-835c-448f-77c9-701e19b26a9e [Candidate]" term=4 -testLoggerAdapter verbose: server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8: 2024-09-03T21:34:27.518Z [INFO] server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8: entering follower state: follower="Node at 0d47dad8-69a2-2b1a-01a6-635a5e5f49d8 [Follower]" leader= - testing.go:149: server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8: 2024-09-03T21:34:27.518Z [INFO] server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8: entering follower state: follower="Node at 0d47dad8-69a2-2b1a-01a6-635a5e5f49d8 [Follower]" leader= -testLoggerAdapter verbose: server-ea14af0e-835c-448f-77c9-701e19b26a9e: 2024-09-03T21:34:27.518Z [INFO] server-ea14af0e-835c-448f-77c9-701e19b26a9e: election won: tally=2 - testing.go:149: server-ea14af0e-835c-448f-77c9-701e19b26a9e: 2024-09-03T21:34:27.518Z [INFO] server-ea14af0e-835c-448f-77c9-701e19b26a9e: election won: tally=2 -testLoggerAdapter verbose: server-ea14af0e-835c-448f-77c9-701e19b26a9e: 2024-09-03T21:34:27.518Z [INFO] server-ea14af0e-835c-448f-77c9-701e19b26a9e: entering leader state: leader="Node at ea14af0e-835c-448f-77c9-701e19b26a9e [Leader]" - testing.go:149: server-ea14af0e-835c-448f-77c9-701e19b26a9e: 2024-09-03T21:34:27.518Z [INFO] server-ea14af0e-835c-448f-77c9-701e19b26a9e: entering leader state: leader="Node at ea14af0e-835c-448f-77c9-701e19b26a9e [Leader]" -testLoggerAdapter verbose: server-ea14af0e-835c-448f-77c9-701e19b26a9e: 2024-09-03T21:34:27.519Z [INFO] server-ea14af0e-835c-448f-77c9-701e19b26a9e: added peer, starting replication: peer=server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8 - testing.go:149: server-ea14af0e-835c-448f-77c9-701e19b26a9e: 2024-09-03T21:34:27.519Z [INFO] server-ea14af0e-835c-448f-77c9-701e19b26a9e: added peer, starting replication: peer=server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8 -testLoggerAdapter verbose: server-ea14af0e-835c-448f-77c9-701e19b26a9e: 2024-09-03T21:34:27.519Z [INFO] server-ea14af0e-835c-448f-77c9-701e19b26a9e: pipelining replication: peer="{Voter server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8 0d47dad8-69a2-2b1a-01a6-635a5e5f49d8}" - testing.go:149: server-ea14af0e-835c-448f-77c9-701e19b26a9e: 2024-09-03T21:34:27.519Z [INFO] server-ea14af0e-835c-448f-77c9-701e19b26a9e: pipelining replication: peer="{Voter server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8 0d47dad8-69a2-2b1a-01a6-635a5e5f49d8}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:27.571Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:27.571Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:27.677Z [INFO] stable state for Leader reached at 2024-09-03 21:34:27.571947923 +0000 UTC m=+8.618605057 (1 nodes), 38.425µs from start of poll, 500.754931ms from cluster start. Timeout at 2024-09-03 21:34:27.677032449 +0000 UTC m=+8.723689581, 105.084524ms after stability - testing.go:149: cluster: 2024-09-03T21:34:27.677Z [INFO] stable state for Leader reached at 2024-09-03 21:34:27.571947923 +0000 UTC m=+8.618605057 (1 nodes), 38.425µs from start of poll, 500.754931ms from cluster start. Timeout at 2024-09-03 21:34:27.677032449 +0000 UTC m=+8.723689581, 105.084524ms after stability -testLoggerAdapter verbose: server-ea14af0e-835c-448f-77c9-701e19b26a9e: 2024-09-03T21:34:27.677Z [INFO] server-ea14af0e-835c-448f-77c9-701e19b26a9e: aborting pipeline replication: peer="{Voter server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8 0d47dad8-69a2-2b1a-01a6-635a5e5f49d8}" - testing.go:149: server-ea14af0e-835c-448f-77c9-701e19b26a9e: 2024-09-03T21:34:27.677Z [INFO] server-ea14af0e-835c-448f-77c9-701e19b26a9e: aborting pipeline replication: peer="{Voter server-0d47dad8-69a2-2b1a-01a6-635a5e5f49d8 0d47dad8-69a2-2b1a-01a6-635a5e5f49d8}" ---- PASS: TestRaft_RemoveLeader (0.61s) +testLoggerAdapter verbose: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.134Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b Address:8a122bf6-ae8e-d64b-47cf-5e59c182a21b} {Suffrage:Voter ID:server-f090f725-515f-71ea-a7db-f4ed53d23adf Address:f090f725-515f-71ea-a7db-f4ed53d23adf} {Suffrage:Voter ID:server-08f55ce9-54f6-1660-4710-cb41b5e07326 Address:08f55ce9-54f6-1660-4710-cb41b5e07326}]" + testing.go:149: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.134Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b Address:8a122bf6-ae8e-d64b-47cf-5e59c182a21b} {Suffrage:Voter ID:server-f090f725-515f-71ea-a7db-f4ed53d23adf Address:f090f725-515f-71ea-a7db-f4ed53d23adf} {Suffrage:Voter ID:server-08f55ce9-54f6-1660-4710-cb41b5e07326 Address:08f55ce9-54f6-1660-4710-cb41b5e07326}]" +testLoggerAdapter verbose: server-f090f725-515f-71ea-a7db-f4ed53d23adf: 2023-08-02T15:12:56.134Z [INFO] server-f090f725-515f-71ea-a7db-f4ed53d23adf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b Address:8a122bf6-ae8e-d64b-47cf-5e59c182a21b} {Suffrage:Voter ID:server-f090f725-515f-71ea-a7db-f4ed53d23adf Address:f090f725-515f-71ea-a7db-f4ed53d23adf} {Suffrage:Voter ID:server-08f55ce9-54f6-1660-4710-cb41b5e07326 Address:08f55ce9-54f6-1660-4710-cb41b5e07326}]" + testing.go:149: server-f090f725-515f-71ea-a7db-f4ed53d23adf: 2023-08-02T15:12:56.134Z [INFO] server-f090f725-515f-71ea-a7db-f4ed53d23adf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b Address:8a122bf6-ae8e-d64b-47cf-5e59c182a21b} {Suffrage:Voter ID:server-f090f725-515f-71ea-a7db-f4ed53d23adf Address:f090f725-515f-71ea-a7db-f4ed53d23adf} {Suffrage:Voter ID:server-08f55ce9-54f6-1660-4710-cb41b5e07326 Address:08f55ce9-54f6-1660-4710-cb41b5e07326}]" +testLoggerAdapter verbose: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.134Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b Address:8a122bf6-ae8e-d64b-47cf-5e59c182a21b} {Suffrage:Voter ID:server-f090f725-515f-71ea-a7db-f4ed53d23adf Address:f090f725-515f-71ea-a7db-f4ed53d23adf} {Suffrage:Voter ID:server-08f55ce9-54f6-1660-4710-cb41b5e07326 Address:08f55ce9-54f6-1660-4710-cb41b5e07326}]" + testing.go:149: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.134Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b Address:8a122bf6-ae8e-d64b-47cf-5e59c182a21b} {Suffrage:Voter ID:server-f090f725-515f-71ea-a7db-f4ed53d23adf Address:f090f725-515f-71ea-a7db-f4ed53d23adf} {Suffrage:Voter ID:server-08f55ce9-54f6-1660-4710-cb41b5e07326 Address:08f55ce9-54f6-1660-4710-cb41b5e07326}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:56.134Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:12:56.134Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.134Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: entering follower state: follower="Node at 8a122bf6-ae8e-d64b-47cf-5e59c182a21b [Follower]" leader= + testing.go:149: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.134Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: entering follower state: follower="Node at 8a122bf6-ae8e-d64b-47cf-5e59c182a21b [Follower]" leader= +testLoggerAdapter verbose: server-f090f725-515f-71ea-a7db-f4ed53d23adf: 2023-08-02T15:12:56.134Z [INFO] server-f090f725-515f-71ea-a7db-f4ed53d23adf: entering follower state: follower="Node at f090f725-515f-71ea-a7db-f4ed53d23adf [Follower]" leader= + testing.go:149: server-f090f725-515f-71ea-a7db-f4ed53d23adf: 2023-08-02T15:12:56.134Z [INFO] server-f090f725-515f-71ea-a7db-f4ed53d23adf: entering follower state: follower="Node at f090f725-515f-71ea-a7db-f4ed53d23adf [Follower]" leader= +testLoggerAdapter verbose: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.134Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: entering follower state: follower="Node at 08f55ce9-54f6-1660-4710-cb41b5e07326 [Follower]" leader= + testing.go:149: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.134Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: entering follower state: follower="Node at 08f55ce9-54f6-1660-4710-cb41b5e07326 [Follower]" leader= +testLoggerAdapter verbose: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.186Z [WARN] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.186Z [WARN] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.186Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: entering candidate state: node="Node at 8a122bf6-ae8e-d64b-47cf-5e59c182a21b [Candidate]" term=2 + testing.go:149: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.186Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: entering candidate state: node="Node at 8a122bf6-ae8e-d64b-47cf-5e59c182a21b [Candidate]" term=2 +testLoggerAdapter verbose: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.186Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: election won: tally=2 + testing.go:149: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.186Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: election won: tally=2 +testLoggerAdapter verbose: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.186Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: entering leader state: leader="Node at 8a122bf6-ae8e-d64b-47cf-5e59c182a21b [Leader]" + testing.go:149: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.186Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: entering leader state: leader="Node at 8a122bf6-ae8e-d64b-47cf-5e59c182a21b [Leader]" +testLoggerAdapter verbose: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.186Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: added peer, starting replication: peer=server-f090f725-515f-71ea-a7db-f4ed53d23adf + testing.go:149: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.186Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: added peer, starting replication: peer=server-f090f725-515f-71ea-a7db-f4ed53d23adf +testLoggerAdapter verbose: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.187Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: added peer, starting replication: peer=server-08f55ce9-54f6-1660-4710-cb41b5e07326 + testing.go:149: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.187Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: added peer, starting replication: peer=server-08f55ce9-54f6-1660-4710-cb41b5e07326 +testLoggerAdapter verbose: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.187Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: pipelining replication: peer="{Voter server-08f55ce9-54f6-1660-4710-cb41b5e07326 08f55ce9-54f6-1660-4710-cb41b5e07326}" + testing.go:149: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.187Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: pipelining replication: peer="{Voter server-08f55ce9-54f6-1660-4710-cb41b5e07326 08f55ce9-54f6-1660-4710-cb41b5e07326}" +testLoggerAdapter verbose: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.187Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: pipelining replication: peer="{Voter server-f090f725-515f-71ea-a7db-f4ed53d23adf f090f725-515f-71ea-a7db-f4ed53d23adf}" + testing.go:149: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.187Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: pipelining replication: peer="{Voter server-f090f725-515f-71ea-a7db-f4ed53d23adf f090f725-515f-71ea-a7db-f4ed53d23adf}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:56.292Z [INFO] stable state for Leader reached at 2023-08-02 15:12:56.187298041 +0000 UTC m=+8.283234380 (1 nodes), 52.450177ms from start of poll, 53.053767ms from cluster start. Timeout at 2023-08-02 15:12:56.292551678 +0000 UTC m=+8.388488026, 105.253646ms after stability + testing.go:149: cluster: 2023-08-02T15:12:56.292Z [INFO] stable state for Leader reached at 2023-08-02 15:12:56.187298041 +0000 UTC m=+8.283234380 (1 nodes), 52.450177ms from start of poll, 53.053767ms from cluster start. Timeout at 2023-08-02 15:12:56.292551678 +0000 UTC m=+8.388488026, 105.253646ms after stability +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:56.298Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-08-02T15:12:56.298Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:56.404Z [INFO] stable state for Follower reached at 2023-08-02 15:12:56.2987994 +0000 UTC m=+8.394735728 (2 nodes), 1.734µs from start of poll, 164.555115ms from cluster start. Timeout at 2023-08-02 15:12:56.403998394 +0000 UTC m=+8.499934977, 105.199249ms after stability + testing.go:149: cluster: 2023-08-02T15:12:56.404Z [INFO] stable state for Follower reached at 2023-08-02 15:12:56.2987994 +0000 UTC m=+8.394735728 (2 nodes), 1.734µs from start of poll, 164.555115ms from cluster start. Timeout at 2023-08-02 15:12:56.403998394 +0000 UTC m=+8.499934977, 105.199249ms after stability +testLoggerAdapter verbose: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.404Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: updating configuration: command=RemoveServer server-id=server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b server-addr= servers="[{Suffrage:Voter ID:server-f090f725-515f-71ea-a7db-f4ed53d23adf Address:f090f725-515f-71ea-a7db-f4ed53d23adf} {Suffrage:Voter ID:server-08f55ce9-54f6-1660-4710-cb41b5e07326 Address:08f55ce9-54f6-1660-4710-cb41b5e07326}]" + testing.go:149: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.404Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: updating configuration: command=RemoveServer server-id=server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b server-addr= servers="[{Suffrage:Voter ID:server-f090f725-515f-71ea-a7db-f4ed53d23adf Address:f090f725-515f-71ea-a7db-f4ed53d23adf} {Suffrage:Voter ID:server-08f55ce9-54f6-1660-4710-cb41b5e07326 Address:08f55ce9-54f6-1660-4710-cb41b5e07326}]" +testLoggerAdapter verbose: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.404Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: removed ourself, shutting down + testing.go:149: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.404Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: removed ourself, shutting down +testLoggerAdapter verbose: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.404Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: aborting pipeline replication: peer="{Voter server-08f55ce9-54f6-1660-4710-cb41b5e07326 08f55ce9-54f6-1660-4710-cb41b5e07326}" + testing.go:149: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.404Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: aborting pipeline replication: peer="{Voter server-08f55ce9-54f6-1660-4710-cb41b5e07326 08f55ce9-54f6-1660-4710-cb41b5e07326}" +testLoggerAdapter verbose: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.404Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: aborting pipeline replication: peer="{Voter server-f090f725-515f-71ea-a7db-f4ed53d23adf f090f725-515f-71ea-a7db-f4ed53d23adf}" + testing.go:149: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.404Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: aborting pipeline replication: peer="{Voter server-f090f725-515f-71ea-a7db-f4ed53d23adf f090f725-515f-71ea-a7db-f4ed53d23adf}" +testLoggerAdapter verbose: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.484Z [WARN] server-08f55ce9-54f6-1660-4710-cb41b5e07326: heartbeat timeout reached, starting election: last-leader=8a122bf6-ae8e-d64b-47cf-5e59c182a21b + testing.go:149: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.484Z [WARN] server-08f55ce9-54f6-1660-4710-cb41b5e07326: heartbeat timeout reached, starting election: last-leader=8a122bf6-ae8e-d64b-47cf-5e59c182a21b +testLoggerAdapter verbose: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.484Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: entering candidate state: node="Node at 08f55ce9-54f6-1660-4710-cb41b5e07326 [Candidate]" term=3 + testing.go:149: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.484Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: entering candidate state: node="Node at 08f55ce9-54f6-1660-4710-cb41b5e07326 [Candidate]" term=3 +testLoggerAdapter verbose: server-f090f725-515f-71ea-a7db-f4ed53d23adf: 2023-08-02T15:12:56.484Z [WARN] server-f090f725-515f-71ea-a7db-f4ed53d23adf: rejecting vote request since we have a leader: from=08f55ce9-54f6-1660-4710-cb41b5e07326 leader=8a122bf6-ae8e-d64b-47cf-5e59c182a21b + testing.go:149: server-f090f725-515f-71ea-a7db-f4ed53d23adf: 2023-08-02T15:12:56.484Z [WARN] server-f090f725-515f-71ea-a7db-f4ed53d23adf: rejecting vote request since we have a leader: from=08f55ce9-54f6-1660-4710-cb41b5e07326 leader=8a122bf6-ae8e-d64b-47cf-5e59c182a21b +testLoggerAdapter verbose: server-f090f725-515f-71ea-a7db-f4ed53d23adf: 2023-08-02T15:12:56.484Z [WARN] server-f090f725-515f-71ea-a7db-f4ed53d23adf: heartbeat timeout reached, starting election: last-leader=8a122bf6-ae8e-d64b-47cf-5e59c182a21b + testing.go:149: server-f090f725-515f-71ea-a7db-f4ed53d23adf: 2023-08-02T15:12:56.484Z [WARN] server-f090f725-515f-71ea-a7db-f4ed53d23adf: heartbeat timeout reached, starting election: last-leader=8a122bf6-ae8e-d64b-47cf-5e59c182a21b +testLoggerAdapter verbose: server-f090f725-515f-71ea-a7db-f4ed53d23adf: 2023-08-02T15:12:56.484Z [INFO] server-f090f725-515f-71ea-a7db-f4ed53d23adf: entering candidate state: node="Node at f090f725-515f-71ea-a7db-f4ed53d23adf [Candidate]" term=3 + testing.go:149: server-f090f725-515f-71ea-a7db-f4ed53d23adf: 2023-08-02T15:12:56.484Z [INFO] server-f090f725-515f-71ea-a7db-f4ed53d23adf: entering candidate state: node="Node at f090f725-515f-71ea-a7db-f4ed53d23adf [Candidate]" term=3 +testLoggerAdapter verbose: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.484Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: duplicate requestVote for same term: term=3 + testing.go:149: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.484Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: duplicate requestVote for same term: term=3 +testLoggerAdapter verbose: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.546Z [WARN] server-08f55ce9-54f6-1660-4710-cb41b5e07326: Election timeout reached, restarting election + testing.go:149: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.546Z [WARN] server-08f55ce9-54f6-1660-4710-cb41b5e07326: Election timeout reached, restarting election +testLoggerAdapter verbose: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.546Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: entering candidate state: node="Node at 08f55ce9-54f6-1660-4710-cb41b5e07326 [Candidate]" term=4 + testing.go:149: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.546Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: entering candidate state: node="Node at 08f55ce9-54f6-1660-4710-cb41b5e07326 [Candidate]" term=4 +testLoggerAdapter verbose: server-f090f725-515f-71ea-a7db-f4ed53d23adf: 2023-08-02T15:12:56.546Z [INFO] server-f090f725-515f-71ea-a7db-f4ed53d23adf: entering follower state: follower="Node at f090f725-515f-71ea-a7db-f4ed53d23adf [Follower]" leader= + testing.go:149: server-f090f725-515f-71ea-a7db-f4ed53d23adf: 2023-08-02T15:12:56.546Z [INFO] server-f090f725-515f-71ea-a7db-f4ed53d23adf: entering follower state: follower="Node at f090f725-515f-71ea-a7db-f4ed53d23adf [Follower]" leader= +testLoggerAdapter verbose: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.546Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: election won: tally=2 + testing.go:149: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.546Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: election won: tally=2 +testLoggerAdapter verbose: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.546Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: entering leader state: leader="Node at 08f55ce9-54f6-1660-4710-cb41b5e07326 [Leader]" + testing.go:149: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.546Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: entering leader state: leader="Node at 08f55ce9-54f6-1660-4710-cb41b5e07326 [Leader]" +testLoggerAdapter verbose: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.546Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: added peer, starting replication: peer=server-f090f725-515f-71ea-a7db-f4ed53d23adf + testing.go:149: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.546Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: added peer, starting replication: peer=server-f090f725-515f-71ea-a7db-f4ed53d23adf +testLoggerAdapter verbose: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.546Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: pipelining replication: peer="{Voter server-f090f725-515f-71ea-a7db-f4ed53d23adf f090f725-515f-71ea-a7db-f4ed53d23adf}" + testing.go:149: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.546Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: pipelining replication: peer="{Voter server-f090f725-515f-71ea-a7db-f4ed53d23adf f090f725-515f-71ea-a7db-f4ed53d23adf}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:56.615Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:12:56.615Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:56.720Z [INFO] stable state for Leader reached at 2023-08-02 15:12:56.615217861 +0000 UTC m=+8.711154208 (1 nodes), 35.438µs from start of poll, 480.973595ms from cluster start. Timeout at 2023-08-02 15:12:56.720502669 +0000 UTC m=+8.816439011, 105.284803ms after stability + testing.go:149: cluster: 2023-08-02T15:12:56.720Z [INFO] stable state for Leader reached at 2023-08-02 15:12:56.615217861 +0000 UTC m=+8.711154208 (1 nodes), 35.438µs from start of poll, 480.973595ms from cluster start. Timeout at 2023-08-02 15:12:56.720502669 +0000 UTC m=+8.816439011, 105.284803ms after stability +testLoggerAdapter verbose: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.720Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: aborting pipeline replication: peer="{Voter server-f090f725-515f-71ea-a7db-f4ed53d23adf f090f725-515f-71ea-a7db-f4ed53d23adf}" + testing.go:149: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.720Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: aborting pipeline replication: peer="{Voter server-f090f725-515f-71ea-a7db-f4ed53d23adf f090f725-515f-71ea-a7db-f4ed53d23adf}" +--- PASS: TestRaft_RemoveLeader (0.59s) === RUN TestRaft_RemoveLeader_NoShutdown -testLoggerAdapter verbose: server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: 2024-09-03T21:34:27.678Z [INFO] server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab Address:42887ace-6c07-eaf5-5f4a-7a1018ed91ab} {Suffrage:Voter ID:server-1d30c5f5-0889-0328-0e8f-e252ad1048e1 Address:1d30c5f5-0889-0328-0e8f-e252ad1048e1} {Suffrage:Voter ID:server-7d989ea5-17e5-fd78-5502-d3e54835f463 Address:7d989ea5-17e5-fd78-5502-d3e54835f463}]" - testing.go:149: server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: 2024-09-03T21:34:27.678Z [INFO] server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab Address:42887ace-6c07-eaf5-5f4a-7a1018ed91ab} {Suffrage:Voter ID:server-1d30c5f5-0889-0328-0e8f-e252ad1048e1 Address:1d30c5f5-0889-0328-0e8f-e252ad1048e1} {Suffrage:Voter ID:server-7d989ea5-17e5-fd78-5502-d3e54835f463 Address:7d989ea5-17e5-fd78-5502-d3e54835f463}]" -testLoggerAdapter verbose: server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: 2024-09-03T21:34:27.678Z [INFO] server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab Address:42887ace-6c07-eaf5-5f4a-7a1018ed91ab} {Suffrage:Voter ID:server-1d30c5f5-0889-0328-0e8f-e252ad1048e1 Address:1d30c5f5-0889-0328-0e8f-e252ad1048e1} {Suffrage:Voter ID:server-7d989ea5-17e5-fd78-5502-d3e54835f463 Address:7d989ea5-17e5-fd78-5502-d3e54835f463}]" - testing.go:149: server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: 2024-09-03T21:34:27.678Z [INFO] server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab Address:42887ace-6c07-eaf5-5f4a-7a1018ed91ab} {Suffrage:Voter ID:server-1d30c5f5-0889-0328-0e8f-e252ad1048e1 Address:1d30c5f5-0889-0328-0e8f-e252ad1048e1} {Suffrage:Voter ID:server-7d989ea5-17e5-fd78-5502-d3e54835f463 Address:7d989ea5-17e5-fd78-5502-d3e54835f463}]" -testLoggerAdapter verbose: server-7d989ea5-17e5-fd78-5502-d3e54835f463: 2024-09-03T21:34:27.678Z [INFO] server-7d989ea5-17e5-fd78-5502-d3e54835f463: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab Address:42887ace-6c07-eaf5-5f4a-7a1018ed91ab} {Suffrage:Voter ID:server-1d30c5f5-0889-0328-0e8f-e252ad1048e1 Address:1d30c5f5-0889-0328-0e8f-e252ad1048e1} {Suffrage:Voter ID:server-7d989ea5-17e5-fd78-5502-d3e54835f463 Address:7d989ea5-17e5-fd78-5502-d3e54835f463}]" - testing.go:149: server-7d989ea5-17e5-fd78-5502-d3e54835f463: 2024-09-03T21:34:27.678Z [INFO] server-7d989ea5-17e5-fd78-5502-d3e54835f463: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab Address:42887ace-6c07-eaf5-5f4a-7a1018ed91ab} {Suffrage:Voter ID:server-1d30c5f5-0889-0328-0e8f-e252ad1048e1 Address:1d30c5f5-0889-0328-0e8f-e252ad1048e1} {Suffrage:Voter ID:server-7d989ea5-17e5-fd78-5502-d3e54835f463 Address:7d989ea5-17e5-fd78-5502-d3e54835f463}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:27.678Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-09-03T21:34:27.678Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: 2024-09-03T21:34:27.678Z [INFO] server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: entering follower state: follower="Node at 42887ace-6c07-eaf5-5f4a-7a1018ed91ab [Follower]" leader= - testing.go:149: server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: 2024-09-03T21:34:27.678Z [INFO] server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: entering follower state: follower="Node at 42887ace-6c07-eaf5-5f4a-7a1018ed91ab [Follower]" leader= -testLoggerAdapter verbose: server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: 2024-09-03T21:34:27.678Z [INFO] server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: entering follower state: follower="Node at 1d30c5f5-0889-0328-0e8f-e252ad1048e1 [Follower]" leader= - testing.go:149: server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: 2024-09-03T21:34:27.678Z [INFO] server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: entering follower state: follower="Node at 1d30c5f5-0889-0328-0e8f-e252ad1048e1 [Follower]" leader= -testLoggerAdapter verbose: server-7d989ea5-17e5-fd78-5502-d3e54835f463: 2024-09-03T21:34:27.678Z [INFO] server-7d989ea5-17e5-fd78-5502-d3e54835f463: entering follower state: follower="Node at 7d989ea5-17e5-fd78-5502-d3e54835f463 [Follower]" leader= - testing.go:149: server-7d989ea5-17e5-fd78-5502-d3e54835f463: 2024-09-03T21:34:27.678Z [INFO] server-7d989ea5-17e5-fd78-5502-d3e54835f463: entering follower state: follower="Node at 7d989ea5-17e5-fd78-5502-d3e54835f463 [Follower]" leader= -testLoggerAdapter verbose: server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: 2024-09-03T21:34:27.751Z [WARN] server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: 2024-09-03T21:34:27.751Z [WARN] server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: 2024-09-03T21:34:27.751Z [INFO] server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: entering candidate state: node="Node at 1d30c5f5-0889-0328-0e8f-e252ad1048e1 [Candidate]" term=2 - testing.go:149: server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: 2024-09-03T21:34:27.751Z [INFO] server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: entering candidate state: node="Node at 1d30c5f5-0889-0328-0e8f-e252ad1048e1 [Candidate]" term=2 -testLoggerAdapter verbose: server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: 2024-09-03T21:34:27.751Z [INFO] server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: election won: tally=2 - testing.go:149: server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: 2024-09-03T21:34:27.751Z [INFO] server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: election won: tally=2 -testLoggerAdapter verbose: server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: 2024-09-03T21:34:27.751Z [INFO] server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: entering leader state: leader="Node at 1d30c5f5-0889-0328-0e8f-e252ad1048e1 [Leader]" - testing.go:149: server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: 2024-09-03T21:34:27.751Z [INFO] server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: entering leader state: leader="Node at 1d30c5f5-0889-0328-0e8f-e252ad1048e1 [Leader]" -testLoggerAdapter verbose: server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: 2024-09-03T21:34:27.751Z [INFO] server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: added peer, starting replication: peer=server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab - testing.go:149: server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: 2024-09-03T21:34:27.751Z [INFO] server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: added peer, starting replication: peer=server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab -testLoggerAdapter verbose: server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: 2024-09-03T21:34:27.751Z [INFO] server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: added peer, starting replication: peer=server-7d989ea5-17e5-fd78-5502-d3e54835f463 - testing.go:149: server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: 2024-09-03T21:34:27.751Z [INFO] server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: added peer, starting replication: peer=server-7d989ea5-17e5-fd78-5502-d3e54835f463 -testLoggerAdapter verbose: server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: 2024-09-03T21:34:27.751Z [INFO] server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: pipelining replication: peer="{Voter server-7d989ea5-17e5-fd78-5502-d3e54835f463 7d989ea5-17e5-fd78-5502-d3e54835f463}" - testing.go:149: server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: 2024-09-03T21:34:27.751Z [INFO] server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: pipelining replication: peer="{Voter server-7d989ea5-17e5-fd78-5502-d3e54835f463 7d989ea5-17e5-fd78-5502-d3e54835f463}" -testLoggerAdapter verbose: server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: 2024-09-03T21:34:27.751Z [INFO] server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: pipelining replication: peer="{Voter server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab 42887ace-6c07-eaf5-5f4a-7a1018ed91ab}" - testing.go:149: server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: 2024-09-03T21:34:27.751Z [INFO] server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: pipelining replication: peer="{Voter server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab 42887ace-6c07-eaf5-5f4a-7a1018ed91ab}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:27.856Z [INFO] stable state for Follower reached at 2024-09-03 21:34:27.751866964 +0000 UTC m=+8.798524094 (2 nodes), 73.58945ms from start of poll, 74.003471ms from cluster start. Timeout at 2024-09-03 21:34:27.856892233 +0000 UTC m=+8.903549366, 105.025272ms after stability - testing.go:149: cluster: 2024-09-03T21:34:27.856Z [INFO] stable state for Follower reached at 2024-09-03 21:34:27.751866964 +0000 UTC m=+8.798524094 (2 nodes), 73.58945ms from start of poll, 74.003471ms from cluster start. Timeout at 2024-09-03 21:34:27.856892233 +0000 UTC m=+8.903549366, 105.025272ms after stability -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:27.857Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:27.857Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:27.962Z [INFO] stable state for Leader reached at 2024-09-03 21:34:27.857034034 +0000 UTC m=+8.903691175 (1 nodes), 1.148µs from start of poll, 179.170552ms from cluster start. Timeout at 2024-09-03 21:34:27.962127002 +0000 UTC m=+9.008784134, 105.092959ms after stability - testing.go:149: cluster: 2024-09-03T21:34:27.962Z [INFO] stable state for Leader reached at 2024-09-03 21:34:27.857034034 +0000 UTC m=+8.903691175 (1 nodes), 1.148µs from start of poll, 179.170552ms from cluster start. Timeout at 2024-09-03 21:34:27.962127002 +0000 UTC m=+9.008784134, 105.092959ms after stability -testLoggerAdapter verbose: server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: 2024-09-03T21:34:27.964Z [INFO] server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: updating configuration: command=RemoveServer server-id=server-1d30c5f5-0889-0328-0e8f-e252ad1048e1 server-addr= servers="[{Suffrage:Voter ID:server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab Address:42887ace-6c07-eaf5-5f4a-7a1018ed91ab} {Suffrage:Voter ID:server-7d989ea5-17e5-fd78-5502-d3e54835f463 Address:7d989ea5-17e5-fd78-5502-d3e54835f463}]" - testing.go:149: server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: 2024-09-03T21:34:27.964Z [INFO] server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: updating configuration: command=RemoveServer server-id=server-1d30c5f5-0889-0328-0e8f-e252ad1048e1 server-addr= servers="[{Suffrage:Voter ID:server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab Address:42887ace-6c07-eaf5-5f4a-7a1018ed91ab} {Suffrage:Voter ID:server-7d989ea5-17e5-fd78-5502-d3e54835f463 Address:7d989ea5-17e5-fd78-5502-d3e54835f463}]" -testLoggerAdapter verbose: server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: 2024-09-03T21:34:27.964Z [INFO] server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: removed ourself, transitioning to follower - testing.go:149: server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: 2024-09-03T21:34:27.964Z [INFO] server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: removed ourself, transitioning to follower -testLoggerAdapter verbose: server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: 2024-09-03T21:34:27.964Z [INFO] server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: entering follower state: follower="Node at 1d30c5f5-0889-0328-0e8f-e252ad1048e1 [Follower]" leader= - testing.go:149: server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: 2024-09-03T21:34:27.964Z [INFO] server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: entering follower state: follower="Node at 1d30c5f5-0889-0328-0e8f-e252ad1048e1 [Follower]" leader= -testLoggerAdapter verbose: server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: 2024-09-03T21:34:27.964Z [INFO] server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: aborting pipeline replication: peer="{Voter server-7d989ea5-17e5-fd78-5502-d3e54835f463 7d989ea5-17e5-fd78-5502-d3e54835f463}" - testing.go:149: server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: 2024-09-03T21:34:27.964Z [INFO] server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: aborting pipeline replication: peer="{Voter server-7d989ea5-17e5-fd78-5502-d3e54835f463 7d989ea5-17e5-fd78-5502-d3e54835f463}" -testLoggerAdapter verbose: server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: 2024-09-03T21:34:27.964Z [INFO] server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: aborting pipeline replication: peer="{Voter server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab 42887ace-6c07-eaf5-5f4a-7a1018ed91ab}" - testing.go:149: server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: 2024-09-03T21:34:27.964Z [INFO] server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: aborting pipeline replication: peer="{Voter server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab 42887ace-6c07-eaf5-5f4a-7a1018ed91ab}" -testLoggerAdapter verbose: server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: 2024-09-03T21:34:28.058Z [WARN] server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: not part of stable configuration, aborting election - testing.go:149: server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: 2024-09-03T21:34:28.058Z [WARN] server-1d30c5f5-0889-0328-0e8f-e252ad1048e1: not part of stable configuration, aborting election -testLoggerAdapter verbose: server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: 2024-09-03T21:34:28.069Z [WARN] server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: heartbeat timeout reached, starting election: last-leader=1d30c5f5-0889-0328-0e8f-e252ad1048e1 - testing.go:149: server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: 2024-09-03T21:34:28.069Z [WARN] server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: heartbeat timeout reached, starting election: last-leader=1d30c5f5-0889-0328-0e8f-e252ad1048e1 -testLoggerAdapter verbose: server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: 2024-09-03T21:34:28.069Z [INFO] server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: entering candidate state: node="Node at 42887ace-6c07-eaf5-5f4a-7a1018ed91ab [Candidate]" term=3 - testing.go:149: server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: 2024-09-03T21:34:28.069Z [INFO] server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: entering candidate state: node="Node at 42887ace-6c07-eaf5-5f4a-7a1018ed91ab [Candidate]" term=3 -testLoggerAdapter verbose: server-7d989ea5-17e5-fd78-5502-d3e54835f463: 2024-09-03T21:34:28.069Z [WARN] server-7d989ea5-17e5-fd78-5502-d3e54835f463: rejecting vote request since we have a leader: from=42887ace-6c07-eaf5-5f4a-7a1018ed91ab leader=1d30c5f5-0889-0328-0e8f-e252ad1048e1 - testing.go:149: server-7d989ea5-17e5-fd78-5502-d3e54835f463: 2024-09-03T21:34:28.069Z [WARN] server-7d989ea5-17e5-fd78-5502-d3e54835f463: rejecting vote request since we have a leader: from=42887ace-6c07-eaf5-5f4a-7a1018ed91ab leader=1d30c5f5-0889-0328-0e8f-e252ad1048e1 -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:28.069Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:28.069Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-7d989ea5-17e5-fd78-5502-d3e54835f463: 2024-09-03T21:34:28.094Z [WARN] server-7d989ea5-17e5-fd78-5502-d3e54835f463: heartbeat timeout reached, starting election: last-leader=1d30c5f5-0889-0328-0e8f-e252ad1048e1 - testing.go:149: server-7d989ea5-17e5-fd78-5502-d3e54835f463: 2024-09-03T21:34:28.094Z [WARN] server-7d989ea5-17e5-fd78-5502-d3e54835f463: heartbeat timeout reached, starting election: last-leader=1d30c5f5-0889-0328-0e8f-e252ad1048e1 -testLoggerAdapter verbose: server-7d989ea5-17e5-fd78-5502-d3e54835f463: 2024-09-03T21:34:28.094Z [INFO] server-7d989ea5-17e5-fd78-5502-d3e54835f463: entering candidate state: node="Node at 7d989ea5-17e5-fd78-5502-d3e54835f463 [Candidate]" term=3 - testing.go:149: server-7d989ea5-17e5-fd78-5502-d3e54835f463: 2024-09-03T21:34:28.094Z [INFO] server-7d989ea5-17e5-fd78-5502-d3e54835f463: entering candidate state: node="Node at 7d989ea5-17e5-fd78-5502-d3e54835f463 [Candidate]" term=3 -testLoggerAdapter verbose: server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: 2024-09-03T21:34:28.094Z [INFO] server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: duplicate requestVote for same term: term=3 - testing.go:149: server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: 2024-09-03T21:34:28.094Z [INFO] server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: duplicate requestVote for same term: term=3 -testLoggerAdapter verbose: server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: 2024-09-03T21:34:28.168Z [WARN] server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: Election timeout reached, restarting election - testing.go:149: server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: 2024-09-03T21:34:28.168Z [WARN] server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: Election timeout reached, restarting election -testLoggerAdapter verbose: server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: 2024-09-03T21:34:28.168Z [INFO] server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: entering candidate state: node="Node at 42887ace-6c07-eaf5-5f4a-7a1018ed91ab [Candidate]" term=4 - testing.go:149: server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: 2024-09-03T21:34:28.168Z [INFO] server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: entering candidate state: node="Node at 42887ace-6c07-eaf5-5f4a-7a1018ed91ab [Candidate]" term=4 -testLoggerAdapter verbose: server-7d989ea5-17e5-fd78-5502-d3e54835f463: 2024-09-03T21:34:28.168Z [INFO] server-7d989ea5-17e5-fd78-5502-d3e54835f463: entering follower state: follower="Node at 7d989ea5-17e5-fd78-5502-d3e54835f463 [Follower]" leader= - testing.go:149: server-7d989ea5-17e5-fd78-5502-d3e54835f463: 2024-09-03T21:34:28.168Z [INFO] server-7d989ea5-17e5-fd78-5502-d3e54835f463: entering follower state: follower="Node at 7d989ea5-17e5-fd78-5502-d3e54835f463 [Follower]" leader= -testLoggerAdapter verbose: server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: 2024-09-03T21:34:28.168Z [INFO] server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: election won: tally=2 - testing.go:149: server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: 2024-09-03T21:34:28.168Z [INFO] server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: election won: tally=2 -testLoggerAdapter verbose: server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: 2024-09-03T21:34:28.168Z [INFO] server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: entering leader state: leader="Node at 42887ace-6c07-eaf5-5f4a-7a1018ed91ab [Leader]" - testing.go:149: server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: 2024-09-03T21:34:28.168Z [INFO] server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: entering leader state: leader="Node at 42887ace-6c07-eaf5-5f4a-7a1018ed91ab [Leader]" -testLoggerAdapter verbose: server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: 2024-09-03T21:34:28.169Z [INFO] server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: added peer, starting replication: peer=server-7d989ea5-17e5-fd78-5502-d3e54835f463 - testing.go:149: server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: 2024-09-03T21:34:28.169Z [INFO] server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: added peer, starting replication: peer=server-7d989ea5-17e5-fd78-5502-d3e54835f463 -testLoggerAdapter verbose: server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: 2024-09-03T21:34:28.169Z [INFO] server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: pipelining replication: peer="{Voter server-7d989ea5-17e5-fd78-5502-d3e54835f463 7d989ea5-17e5-fd78-5502-d3e54835f463}" - testing.go:149: server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: 2024-09-03T21:34:28.169Z [INFO] server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: pipelining replication: peer="{Voter server-7d989ea5-17e5-fd78-5502-d3e54835f463 7d989ea5-17e5-fd78-5502-d3e54835f463}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:28.274Z [INFO] stable state for Leader reached at 2024-09-03 21:34:28.169161089 +0000 UTC m=+9.215818220 (1 nodes), 99.494587ms from start of poll, 491.297597ms from cluster start. Timeout at 2024-09-03 21:34:28.274245661 +0000 UTC m=+9.320902802, 105.084582ms after stability - testing.go:149: cluster: 2024-09-03T21:34:28.274Z [INFO] stable state for Leader reached at 2024-09-03 21:34:28.169161089 +0000 UTC m=+9.215818220 (1 nodes), 99.494587ms from start of poll, 491.297597ms from cluster start. Timeout at 2024-09-03 21:34:28.274245661 +0000 UTC m=+9.320902802, 105.084582ms after stability -testLoggerAdapter verbose: server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: 2024-09-03T21:34:28.379Z [INFO] server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: aborting pipeline replication: peer="{Voter server-7d989ea5-17e5-fd78-5502-d3e54835f463 7d989ea5-17e5-fd78-5502-d3e54835f463}" - testing.go:149: server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: 2024-09-03T21:34:28.379Z [INFO] server-42887ace-6c07-eaf5-5f4a-7a1018ed91ab: aborting pipeline replication: peer="{Voter server-7d989ea5-17e5-fd78-5502-d3e54835f463 7d989ea5-17e5-fd78-5502-d3e54835f463}" ---- PASS: TestRaft_RemoveLeader_NoShutdown (0.70s) +testLoggerAdapter verbose: server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: 2023-08-02T15:12:56.724Z [INFO] server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c Address:bf8e15c9-b2a9-d822-6159-523a1e97ec9c} {Suffrage:Voter ID:server-064a8bac-5a48-2c3c-6153-da94e6d050c7 Address:064a8bac-5a48-2c3c-6153-da94e6d050c7} {Suffrage:Voter ID:server-559cfffc-7a69-fb43-6ca4-99a924e98ca9 Address:559cfffc-7a69-fb43-6ca4-99a924e98ca9}]" + testing.go:149: server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: 2023-08-02T15:12:56.724Z [INFO] server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c Address:bf8e15c9-b2a9-d822-6159-523a1e97ec9c} {Suffrage:Voter ID:server-064a8bac-5a48-2c3c-6153-da94e6d050c7 Address:064a8bac-5a48-2c3c-6153-da94e6d050c7} {Suffrage:Voter ID:server-559cfffc-7a69-fb43-6ca4-99a924e98ca9 Address:559cfffc-7a69-fb43-6ca4-99a924e98ca9}]" +testLoggerAdapter verbose: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:56.724Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c Address:bf8e15c9-b2a9-d822-6159-523a1e97ec9c} {Suffrage:Voter ID:server-064a8bac-5a48-2c3c-6153-da94e6d050c7 Address:064a8bac-5a48-2c3c-6153-da94e6d050c7} {Suffrage:Voter ID:server-559cfffc-7a69-fb43-6ca4-99a924e98ca9 Address:559cfffc-7a69-fb43-6ca4-99a924e98ca9}]" + testing.go:149: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:56.724Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c Address:bf8e15c9-b2a9-d822-6159-523a1e97ec9c} {Suffrage:Voter ID:server-064a8bac-5a48-2c3c-6153-da94e6d050c7 Address:064a8bac-5a48-2c3c-6153-da94e6d050c7} {Suffrage:Voter ID:server-559cfffc-7a69-fb43-6ca4-99a924e98ca9 Address:559cfffc-7a69-fb43-6ca4-99a924e98ca9}]" +testLoggerAdapter verbose: server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: 2023-08-02T15:12:56.724Z [INFO] server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: entering follower state: follower="Node at bf8e15c9-b2a9-d822-6159-523a1e97ec9c [Follower]" leader= + testing.go:149: server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: 2023-08-02T15:12:56.724Z [INFO] server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: entering follower state: follower="Node at bf8e15c9-b2a9-d822-6159-523a1e97ec9c [Follower]" leader= +testLoggerAdapter verbose: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.724Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c Address:bf8e15c9-b2a9-d822-6159-523a1e97ec9c} {Suffrage:Voter ID:server-064a8bac-5a48-2c3c-6153-da94e6d050c7 Address:064a8bac-5a48-2c3c-6153-da94e6d050c7} {Suffrage:Voter ID:server-559cfffc-7a69-fb43-6ca4-99a924e98ca9 Address:559cfffc-7a69-fb43-6ca4-99a924e98ca9}]" + testing.go:149: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.724Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c Address:bf8e15c9-b2a9-d822-6159-523a1e97ec9c} {Suffrage:Voter ID:server-064a8bac-5a48-2c3c-6153-da94e6d050c7 Address:064a8bac-5a48-2c3c-6153-da94e6d050c7} {Suffrage:Voter ID:server-559cfffc-7a69-fb43-6ca4-99a924e98ca9 Address:559cfffc-7a69-fb43-6ca4-99a924e98ca9}]" +testLoggerAdapter verbose: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:56.724Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: entering follower state: follower="Node at 064a8bac-5a48-2c3c-6153-da94e6d050c7 [Follower]" leader= + testing.go:149: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:56.724Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: entering follower state: follower="Node at 064a8bac-5a48-2c3c-6153-da94e6d050c7 [Follower]" leader= +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:56.724Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-08-02T15:12:56.724Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.724Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: entering follower state: follower="Node at 559cfffc-7a69-fb43-6ca4-99a924e98ca9 [Follower]" leader= + testing.go:149: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.724Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: entering follower state: follower="Node at 559cfffc-7a69-fb43-6ca4-99a924e98ca9 [Follower]" leader= +testLoggerAdapter verbose: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.787Z [WARN] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.787Z [WARN] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.788Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: entering candidate state: node="Node at 559cfffc-7a69-fb43-6ca4-99a924e98ca9 [Candidate]" term=2 + testing.go:149: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.788Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: entering candidate state: node="Node at 559cfffc-7a69-fb43-6ca4-99a924e98ca9 [Candidate]" term=2 +testLoggerAdapter verbose: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.788Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: election won: tally=2 + testing.go:149: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.788Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: election won: tally=2 +testLoggerAdapter verbose: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.788Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: entering leader state: leader="Node at 559cfffc-7a69-fb43-6ca4-99a924e98ca9 [Leader]" + testing.go:149: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.788Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: entering leader state: leader="Node at 559cfffc-7a69-fb43-6ca4-99a924e98ca9 [Leader]" +testLoggerAdapter verbose: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.788Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: added peer, starting replication: peer=server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c + testing.go:149: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.788Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: added peer, starting replication: peer=server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c +testLoggerAdapter verbose: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.788Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: added peer, starting replication: peer=server-064a8bac-5a48-2c3c-6153-da94e6d050c7 + testing.go:149: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.788Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: added peer, starting replication: peer=server-064a8bac-5a48-2c3c-6153-da94e6d050c7 +testLoggerAdapter verbose: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.788Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: pipelining replication: peer="{Voter server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c bf8e15c9-b2a9-d822-6159-523a1e97ec9c}" + testing.go:149: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.788Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: pipelining replication: peer="{Voter server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c bf8e15c9-b2a9-d822-6159-523a1e97ec9c}" +testLoggerAdapter verbose: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.788Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: pipelining replication: peer="{Voter server-064a8bac-5a48-2c3c-6153-da94e6d050c7 064a8bac-5a48-2c3c-6153-da94e6d050c7}" + testing.go:149: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.788Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: pipelining replication: peer="{Voter server-064a8bac-5a48-2c3c-6153-da94e6d050c7 064a8bac-5a48-2c3c-6153-da94e6d050c7}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:56.893Z [INFO] stable state for Follower reached at 2023-08-02 15:12:56.788303194 +0000 UTC m=+8.884239528 (2 nodes), 63.321073ms from start of poll, 64.096025ms from cluster start. Timeout at 2023-08-02 15:12:56.893618173 +0000 UTC m=+8.989554743, 105.315215ms after stability + testing.go:149: cluster: 2023-08-02T15:12:56.893Z [INFO] stable state for Follower reached at 2023-08-02 15:12:56.788303194 +0000 UTC m=+8.884239528 (2 nodes), 63.321073ms from start of poll, 64.096025ms from cluster start. Timeout at 2023-08-02 15:12:56.893618173 +0000 UTC m=+8.989554743, 105.315215ms after stability +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:56.893Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:12:56.893Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:56.998Z [INFO] stable state for Leader reached at 2023-08-02 15:12:56.893802243 +0000 UTC m=+8.989738570 (1 nodes), 1.547µs from start of poll, 169.595067ms from cluster start. Timeout at 2023-08-02 15:12:56.998912236 +0000 UTC m=+9.094848580, 105.11001ms after stability + testing.go:149: cluster: 2023-08-02T15:12:56.998Z [INFO] stable state for Leader reached at 2023-08-02 15:12:56.893802243 +0000 UTC m=+8.989738570 (1 nodes), 1.547µs from start of poll, 169.595067ms from cluster start. Timeout at 2023-08-02 15:12:56.998912236 +0000 UTC m=+9.094848580, 105.11001ms after stability +testLoggerAdapter verbose: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:57.001Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: updating configuration: command=RemoveServer server-id=server-559cfffc-7a69-fb43-6ca4-99a924e98ca9 server-addr= servers="[{Suffrage:Voter ID:server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c Address:bf8e15c9-b2a9-d822-6159-523a1e97ec9c} {Suffrage:Voter ID:server-064a8bac-5a48-2c3c-6153-da94e6d050c7 Address:064a8bac-5a48-2c3c-6153-da94e6d050c7}]" + testing.go:149: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:57.001Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: updating configuration: command=RemoveServer server-id=server-559cfffc-7a69-fb43-6ca4-99a924e98ca9 server-addr= servers="[{Suffrage:Voter ID:server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c Address:bf8e15c9-b2a9-d822-6159-523a1e97ec9c} {Suffrage:Voter ID:server-064a8bac-5a48-2c3c-6153-da94e6d050c7 Address:064a8bac-5a48-2c3c-6153-da94e6d050c7}]" +testLoggerAdapter verbose: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:57.001Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: removed ourself, transitioning to follower + testing.go:149: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:57.001Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: removed ourself, transitioning to follower +testLoggerAdapter verbose: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:57.001Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: entering follower state: follower="Node at 559cfffc-7a69-fb43-6ca4-99a924e98ca9 [Follower]" leader= + testing.go:149: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:57.001Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: entering follower state: follower="Node at 559cfffc-7a69-fb43-6ca4-99a924e98ca9 [Follower]" leader= +testLoggerAdapter verbose: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:57.001Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: aborting pipeline replication: peer="{Voter server-064a8bac-5a48-2c3c-6153-da94e6d050c7 064a8bac-5a48-2c3c-6153-da94e6d050c7}" + testing.go:149: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:57.001Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: aborting pipeline replication: peer="{Voter server-064a8bac-5a48-2c3c-6153-da94e6d050c7 064a8bac-5a48-2c3c-6153-da94e6d050c7}" +testLoggerAdapter verbose: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:57.001Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: aborting pipeline replication: peer="{Voter server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c bf8e15c9-b2a9-d822-6159-523a1e97ec9c}" + testing.go:149: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:57.001Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: aborting pipeline replication: peer="{Voter server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c bf8e15c9-b2a9-d822-6159-523a1e97ec9c}" +testLoggerAdapter verbose: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.070Z [WARN] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: heartbeat timeout reached, starting election: last-leader=559cfffc-7a69-fb43-6ca4-99a924e98ca9 + testing.go:149: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.070Z [WARN] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: heartbeat timeout reached, starting election: last-leader=559cfffc-7a69-fb43-6ca4-99a924e98ca9 +testLoggerAdapter verbose: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.070Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: entering candidate state: node="Node at 064a8bac-5a48-2c3c-6153-da94e6d050c7 [Candidate]" term=3 + testing.go:149: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.070Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: entering candidate state: node="Node at 064a8bac-5a48-2c3c-6153-da94e6d050c7 [Candidate]" term=3 +testLoggerAdapter verbose: server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: 2023-08-02T15:12:57.070Z [WARN] server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: rejecting vote request since we have a leader: from=064a8bac-5a48-2c3c-6153-da94e6d050c7 leader=559cfffc-7a69-fb43-6ca4-99a924e98ca9 + testing.go:149: server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: 2023-08-02T15:12:57.070Z [WARN] server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: rejecting vote request since we have a leader: from=064a8bac-5a48-2c3c-6153-da94e6d050c7 leader=559cfffc-7a69-fb43-6ca4-99a924e98ca9 +testLoggerAdapter verbose: server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: 2023-08-02T15:12:57.078Z [WARN] server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: heartbeat timeout reached, starting election: last-leader=559cfffc-7a69-fb43-6ca4-99a924e98ca9 + testing.go:149: server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: 2023-08-02T15:12:57.078Z [WARN] server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: heartbeat timeout reached, starting election: last-leader=559cfffc-7a69-fb43-6ca4-99a924e98ca9 +testLoggerAdapter verbose: server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: 2023-08-02T15:12:57.078Z [INFO] server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: entering candidate state: node="Node at bf8e15c9-b2a9-d822-6159-523a1e97ec9c [Candidate]" term=3 + testing.go:149: server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: 2023-08-02T15:12:57.078Z [INFO] server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: entering candidate state: node="Node at bf8e15c9-b2a9-d822-6159-523a1e97ec9c [Candidate]" term=3 +testLoggerAdapter verbose: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.078Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: duplicate requestVote for same term: term=3 + testing.go:149: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.078Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: duplicate requestVote for same term: term=3 +testLoggerAdapter verbose: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:57.088Z [WARN] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: not part of stable configuration, aborting election + testing.go:149: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:57.088Z [WARN] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: not part of stable configuration, aborting election +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:57.107Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:12:57.107Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.132Z [WARN] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: Election timeout reached, restarting election + testing.go:149: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.132Z [WARN] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: Election timeout reached, restarting election +testLoggerAdapter verbose: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.132Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: entering candidate state: node="Node at 064a8bac-5a48-2c3c-6153-da94e6d050c7 [Candidate]" term=4 + testing.go:149: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.132Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: entering candidate state: node="Node at 064a8bac-5a48-2c3c-6153-da94e6d050c7 [Candidate]" term=4 +testLoggerAdapter verbose: server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: 2023-08-02T15:12:57.132Z [INFO] server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: entering follower state: follower="Node at bf8e15c9-b2a9-d822-6159-523a1e97ec9c [Follower]" leader= + testing.go:149: server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: 2023-08-02T15:12:57.132Z [INFO] server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: entering follower state: follower="Node at bf8e15c9-b2a9-d822-6159-523a1e97ec9c [Follower]" leader= +testLoggerAdapter verbose: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.132Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: election won: tally=2 + testing.go:149: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.132Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: election won: tally=2 +testLoggerAdapter verbose: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.132Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: entering leader state: leader="Node at 064a8bac-5a48-2c3c-6153-da94e6d050c7 [Leader]" + testing.go:149: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.132Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: entering leader state: leader="Node at 064a8bac-5a48-2c3c-6153-da94e6d050c7 [Leader]" +testLoggerAdapter verbose: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.132Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: added peer, starting replication: peer=server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c + testing.go:149: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.132Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: added peer, starting replication: peer=server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c +testLoggerAdapter verbose: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.132Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: pipelining replication: peer="{Voter server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c bf8e15c9-b2a9-d822-6159-523a1e97ec9c}" + testing.go:149: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.132Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: pipelining replication: peer="{Voter server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c bf8e15c9-b2a9-d822-6159-523a1e97ec9c}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:57.237Z [INFO] stable state for Leader reached at 2023-08-02 15:12:57.132797661 +0000 UTC m=+9.228734012 (1 nodes), 25.669304ms from start of poll, 408.590509ms from cluster start. Timeout at 2023-08-02 15:12:57.237804592 +0000 UTC m=+9.333740928, 105.006916ms after stability + testing.go:149: cluster: 2023-08-02T15:12:57.237Z [INFO] stable state for Leader reached at 2023-08-02 15:12:57.132797661 +0000 UTC m=+9.228734012 (1 nodes), 25.669304ms from start of poll, 408.590509ms from cluster start. Timeout at 2023-08-02 15:12:57.237804592 +0000 UTC m=+9.333740928, 105.006916ms after stability +testLoggerAdapter verbose: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.343Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: aborting pipeline replication: peer="{Voter server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c bf8e15c9-b2a9-d822-6159-523a1e97ec9c}" + testing.go:149: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.343Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: aborting pipeline replication: peer="{Voter server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c bf8e15c9-b2a9-d822-6159-523a1e97ec9c}" +--- PASS: TestRaft_RemoveLeader_NoShutdown (0.62s) === RUN TestRaft_RemoveFollower_SplitCluster -testLoggerAdapter verbose: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.380Z [INFO] server-2014d471-0917-43d2-0ae5-cc60457a30d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2014d471-0917-43d2-0ae5-cc60457a30d8 Address:2014d471-0917-43d2-0ae5-cc60457a30d8} {Suffrage:Voter ID:server-b31b296e-93ea-2a01-fac4-b731701596e8 Address:b31b296e-93ea-2a01-fac4-b731701596e8} {Suffrage:Voter ID:server-e320dbad-2699-6079-715c-4786cc61d6da Address:e320dbad-2699-6079-715c-4786cc61d6da} {Suffrage:Voter ID:server-18bf439f-22ed-da1c-a22f-2ced670f8fcf Address:18bf439f-22ed-da1c-a22f-2ced670f8fcf}]" - testing.go:149: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.380Z [INFO] server-2014d471-0917-43d2-0ae5-cc60457a30d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2014d471-0917-43d2-0ae5-cc60457a30d8 Address:2014d471-0917-43d2-0ae5-cc60457a30d8} {Suffrage:Voter ID:server-b31b296e-93ea-2a01-fac4-b731701596e8 Address:b31b296e-93ea-2a01-fac4-b731701596e8} {Suffrage:Voter ID:server-e320dbad-2699-6079-715c-4786cc61d6da Address:e320dbad-2699-6079-715c-4786cc61d6da} {Suffrage:Voter ID:server-18bf439f-22ed-da1c-a22f-2ced670f8fcf Address:18bf439f-22ed-da1c-a22f-2ced670f8fcf}]" -testLoggerAdapter verbose: server-b31b296e-93ea-2a01-fac4-b731701596e8: 2024-09-03T21:34:28.380Z [INFO] server-b31b296e-93ea-2a01-fac4-b731701596e8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2014d471-0917-43d2-0ae5-cc60457a30d8 Address:2014d471-0917-43d2-0ae5-cc60457a30d8} {Suffrage:Voter ID:server-b31b296e-93ea-2a01-fac4-b731701596e8 Address:b31b296e-93ea-2a01-fac4-b731701596e8} {Suffrage:Voter ID:server-e320dbad-2699-6079-715c-4786cc61d6da Address:e320dbad-2699-6079-715c-4786cc61d6da} {Suffrage:Voter ID:server-18bf439f-22ed-da1c-a22f-2ced670f8fcf Address:18bf439f-22ed-da1c-a22f-2ced670f8fcf}]" - testing.go:149: server-b31b296e-93ea-2a01-fac4-b731701596e8: 2024-09-03T21:34:28.380Z [INFO] server-b31b296e-93ea-2a01-fac4-b731701596e8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2014d471-0917-43d2-0ae5-cc60457a30d8 Address:2014d471-0917-43d2-0ae5-cc60457a30d8} {Suffrage:Voter ID:server-b31b296e-93ea-2a01-fac4-b731701596e8 Address:b31b296e-93ea-2a01-fac4-b731701596e8} {Suffrage:Voter ID:server-e320dbad-2699-6079-715c-4786cc61d6da Address:e320dbad-2699-6079-715c-4786cc61d6da} {Suffrage:Voter ID:server-18bf439f-22ed-da1c-a22f-2ced670f8fcf Address:18bf439f-22ed-da1c-a22f-2ced670f8fcf}]" -testLoggerAdapter verbose: server-e320dbad-2699-6079-715c-4786cc61d6da: 2024-09-03T21:34:28.380Z [INFO] server-e320dbad-2699-6079-715c-4786cc61d6da: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2014d471-0917-43d2-0ae5-cc60457a30d8 Address:2014d471-0917-43d2-0ae5-cc60457a30d8} {Suffrage:Voter ID:server-b31b296e-93ea-2a01-fac4-b731701596e8 Address:b31b296e-93ea-2a01-fac4-b731701596e8} {Suffrage:Voter ID:server-e320dbad-2699-6079-715c-4786cc61d6da Address:e320dbad-2699-6079-715c-4786cc61d6da} {Suffrage:Voter ID:server-18bf439f-22ed-da1c-a22f-2ced670f8fcf Address:18bf439f-22ed-da1c-a22f-2ced670f8fcf}]" - testing.go:149: server-e320dbad-2699-6079-715c-4786cc61d6da: 2024-09-03T21:34:28.380Z [INFO] server-e320dbad-2699-6079-715c-4786cc61d6da: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2014d471-0917-43d2-0ae5-cc60457a30d8 Address:2014d471-0917-43d2-0ae5-cc60457a30d8} {Suffrage:Voter ID:server-b31b296e-93ea-2a01-fac4-b731701596e8 Address:b31b296e-93ea-2a01-fac4-b731701596e8} {Suffrage:Voter ID:server-e320dbad-2699-6079-715c-4786cc61d6da Address:e320dbad-2699-6079-715c-4786cc61d6da} {Suffrage:Voter ID:server-18bf439f-22ed-da1c-a22f-2ced670f8fcf Address:18bf439f-22ed-da1c-a22f-2ced670f8fcf}]" -testLoggerAdapter verbose: server-18bf439f-22ed-da1c-a22f-2ced670f8fcf: 2024-09-03T21:34:28.380Z [INFO] server-18bf439f-22ed-da1c-a22f-2ced670f8fcf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2014d471-0917-43d2-0ae5-cc60457a30d8 Address:2014d471-0917-43d2-0ae5-cc60457a30d8} {Suffrage:Voter ID:server-b31b296e-93ea-2a01-fac4-b731701596e8 Address:b31b296e-93ea-2a01-fac4-b731701596e8} {Suffrage:Voter ID:server-e320dbad-2699-6079-715c-4786cc61d6da Address:e320dbad-2699-6079-715c-4786cc61d6da} {Suffrage:Voter ID:server-18bf439f-22ed-da1c-a22f-2ced670f8fcf Address:18bf439f-22ed-da1c-a22f-2ced670f8fcf}]" - testing.go:149: server-18bf439f-22ed-da1c-a22f-2ced670f8fcf: 2024-09-03T21:34:28.380Z [INFO] server-18bf439f-22ed-da1c-a22f-2ced670f8fcf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2014d471-0917-43d2-0ae5-cc60457a30d8 Address:2014d471-0917-43d2-0ae5-cc60457a30d8} {Suffrage:Voter ID:server-b31b296e-93ea-2a01-fac4-b731701596e8 Address:b31b296e-93ea-2a01-fac4-b731701596e8} {Suffrage:Voter ID:server-e320dbad-2699-6079-715c-4786cc61d6da Address:e320dbad-2699-6079-715c-4786cc61d6da} {Suffrage:Voter ID:server-18bf439f-22ed-da1c-a22f-2ced670f8fcf Address:18bf439f-22ed-da1c-a22f-2ced670f8fcf}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:28.380Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:28.380Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.380Z [INFO] server-2014d471-0917-43d2-0ae5-cc60457a30d8: entering follower state: follower="Node at 2014d471-0917-43d2-0ae5-cc60457a30d8 [Follower]" leader= - testing.go:149: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.380Z [INFO] server-2014d471-0917-43d2-0ae5-cc60457a30d8: entering follower state: follower="Node at 2014d471-0917-43d2-0ae5-cc60457a30d8 [Follower]" leader= -testLoggerAdapter verbose: server-b31b296e-93ea-2a01-fac4-b731701596e8: 2024-09-03T21:34:28.380Z [INFO] server-b31b296e-93ea-2a01-fac4-b731701596e8: entering follower state: follower="Node at b31b296e-93ea-2a01-fac4-b731701596e8 [Follower]" leader= - testing.go:149: server-b31b296e-93ea-2a01-fac4-b731701596e8: 2024-09-03T21:34:28.380Z [INFO] server-b31b296e-93ea-2a01-fac4-b731701596e8: entering follower state: follower="Node at b31b296e-93ea-2a01-fac4-b731701596e8 [Follower]" leader= -testLoggerAdapter verbose: server-e320dbad-2699-6079-715c-4786cc61d6da: 2024-09-03T21:34:28.380Z [INFO] server-e320dbad-2699-6079-715c-4786cc61d6da: entering follower state: follower="Node at e320dbad-2699-6079-715c-4786cc61d6da [Follower]" leader= - testing.go:149: server-e320dbad-2699-6079-715c-4786cc61d6da: 2024-09-03T21:34:28.380Z [INFO] server-e320dbad-2699-6079-715c-4786cc61d6da: entering follower state: follower="Node at e320dbad-2699-6079-715c-4786cc61d6da [Follower]" leader= -testLoggerAdapter verbose: server-18bf439f-22ed-da1c-a22f-2ced670f8fcf: 2024-09-03T21:34:28.380Z [INFO] server-18bf439f-22ed-da1c-a22f-2ced670f8fcf: entering follower state: follower="Node at 18bf439f-22ed-da1c-a22f-2ced670f8fcf [Follower]" leader= - testing.go:149: server-18bf439f-22ed-da1c-a22f-2ced670f8fcf: 2024-09-03T21:34:28.380Z [INFO] server-18bf439f-22ed-da1c-a22f-2ced670f8fcf: entering follower state: follower="Node at 18bf439f-22ed-da1c-a22f-2ced670f8fcf [Follower]" leader= -testLoggerAdapter verbose: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.434Z [WARN] server-2014d471-0917-43d2-0ae5-cc60457a30d8: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.434Z [WARN] server-2014d471-0917-43d2-0ae5-cc60457a30d8: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.435Z [INFO] server-2014d471-0917-43d2-0ae5-cc60457a30d8: entering candidate state: node="Node at 2014d471-0917-43d2-0ae5-cc60457a30d8 [Candidate]" term=2 - testing.go:149: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.435Z [INFO] server-2014d471-0917-43d2-0ae5-cc60457a30d8: entering candidate state: node="Node at 2014d471-0917-43d2-0ae5-cc60457a30d8 [Candidate]" term=2 -testLoggerAdapter verbose: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.435Z [INFO] server-2014d471-0917-43d2-0ae5-cc60457a30d8: election won: tally=3 - testing.go:149: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.435Z [INFO] server-2014d471-0917-43d2-0ae5-cc60457a30d8: election won: tally=3 -testLoggerAdapter verbose: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.435Z [INFO] server-2014d471-0917-43d2-0ae5-cc60457a30d8: entering leader state: leader="Node at 2014d471-0917-43d2-0ae5-cc60457a30d8 [Leader]" - testing.go:149: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.435Z [INFO] server-2014d471-0917-43d2-0ae5-cc60457a30d8: entering leader state: leader="Node at 2014d471-0917-43d2-0ae5-cc60457a30d8 [Leader]" -testLoggerAdapter verbose: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.435Z [INFO] server-2014d471-0917-43d2-0ae5-cc60457a30d8: added peer, starting replication: peer=server-b31b296e-93ea-2a01-fac4-b731701596e8 - testing.go:149: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.435Z [INFO] server-2014d471-0917-43d2-0ae5-cc60457a30d8: added peer, starting replication: peer=server-b31b296e-93ea-2a01-fac4-b731701596e8 -testLoggerAdapter verbose: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.435Z [INFO] server-2014d471-0917-43d2-0ae5-cc60457a30d8: added peer, starting replication: peer=server-e320dbad-2699-6079-715c-4786cc61d6da - testing.go:149: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.435Z [INFO] server-2014d471-0917-43d2-0ae5-cc60457a30d8: added peer, starting replication: peer=server-e320dbad-2699-6079-715c-4786cc61d6da -testLoggerAdapter verbose: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.435Z [INFO] server-2014d471-0917-43d2-0ae5-cc60457a30d8: added peer, starting replication: peer=server-18bf439f-22ed-da1c-a22f-2ced670f8fcf - testing.go:149: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.435Z [INFO] server-2014d471-0917-43d2-0ae5-cc60457a30d8: added peer, starting replication: peer=server-18bf439f-22ed-da1c-a22f-2ced670f8fcf -testLoggerAdapter verbose: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.435Z [INFO] server-2014d471-0917-43d2-0ae5-cc60457a30d8: pipelining replication: peer="{Voter server-18bf439f-22ed-da1c-a22f-2ced670f8fcf 18bf439f-22ed-da1c-a22f-2ced670f8fcf}" - testing.go:149: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.435Z [INFO] server-2014d471-0917-43d2-0ae5-cc60457a30d8: pipelining replication: peer="{Voter server-18bf439f-22ed-da1c-a22f-2ced670f8fcf 18bf439f-22ed-da1c-a22f-2ced670f8fcf}" -testLoggerAdapter verbose: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.435Z [INFO] server-2014d471-0917-43d2-0ae5-cc60457a30d8: pipelining replication: peer="{Voter server-b31b296e-93ea-2a01-fac4-b731701596e8 b31b296e-93ea-2a01-fac4-b731701596e8}" - testing.go:149: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.435Z [INFO] server-2014d471-0917-43d2-0ae5-cc60457a30d8: pipelining replication: peer="{Voter server-b31b296e-93ea-2a01-fac4-b731701596e8 b31b296e-93ea-2a01-fac4-b731701596e8}" -testLoggerAdapter verbose: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.435Z [INFO] server-2014d471-0917-43d2-0ae5-cc60457a30d8: pipelining replication: peer="{Voter server-e320dbad-2699-6079-715c-4786cc61d6da e320dbad-2699-6079-715c-4786cc61d6da}" - testing.go:149: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.435Z [INFO] server-2014d471-0917-43d2-0ae5-cc60457a30d8: pipelining replication: peer="{Voter server-e320dbad-2699-6079-715c-4786cc61d6da e320dbad-2699-6079-715c-4786cc61d6da}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:28.540Z [INFO] stable state for Leader reached at 2024-09-03 21:34:28.435502422 +0000 UTC m=+9.482159554 (1 nodes), 54.738778ms from start of poll, 55.227135ms from cluster start. Timeout at 2024-09-03 21:34:28.540603764 +0000 UTC m=+9.587260956, 105.101402ms after stability - testing.go:149: cluster: 2024-09-03T21:34:28.540Z [INFO] stable state for Leader reached at 2024-09-03 21:34:28.435502422 +0000 UTC m=+9.482159554 (1 nodes), 54.738778ms from start of poll, 55.227135ms from cluster start. Timeout at 2024-09-03 21:34:28.540603764 +0000 UTC m=+9.587260956, 105.101402ms after stability -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:28.645Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-09-03T21:34:28.645Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:28.751Z [INFO] stable state for Follower reached at 2024-09-03 21:34:28.645969509 +0000 UTC m=+9.692626645 (3 nodes), 1.935µs from start of poll, 265.694226ms from cluster start. Timeout at 2024-09-03 21:34:28.751046593 +0000 UTC m=+9.797703727, 105.077082ms after stability - testing.go:149: cluster: 2024-09-03T21:34:28.751Z [INFO] stable state for Follower reached at 2024-09-03 21:34:28.645969509 +0000 UTC m=+9.692626645 (3 nodes), 1.935µs from start of poll, 265.694226ms from cluster start. Timeout at 2024-09-03 21:34:28.751046593 +0000 UTC m=+9.797703727, 105.077082ms after stability -testLoggerAdapter verbose: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.751Z [INFO] server-2014d471-0917-43d2-0ae5-cc60457a30d8: updating configuration: command=RemoveServer server-id=server-18bf439f-22ed-da1c-a22f-2ced670f8fcf server-addr= servers="[{Suffrage:Voter ID:server-2014d471-0917-43d2-0ae5-cc60457a30d8 Address:2014d471-0917-43d2-0ae5-cc60457a30d8} {Suffrage:Voter ID:server-b31b296e-93ea-2a01-fac4-b731701596e8 Address:b31b296e-93ea-2a01-fac4-b731701596e8} {Suffrage:Voter ID:server-e320dbad-2699-6079-715c-4786cc61d6da Address:e320dbad-2699-6079-715c-4786cc61d6da}]" - testing.go:149: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.751Z [INFO] server-2014d471-0917-43d2-0ae5-cc60457a30d8: updating configuration: command=RemoveServer server-id=server-18bf439f-22ed-da1c-a22f-2ced670f8fcf server-addr= servers="[{Suffrage:Voter ID:server-2014d471-0917-43d2-0ae5-cc60457a30d8 Address:2014d471-0917-43d2-0ae5-cc60457a30d8} {Suffrage:Voter ID:server-b31b296e-93ea-2a01-fac4-b731701596e8 Address:b31b296e-93ea-2a01-fac4-b731701596e8} {Suffrage:Voter ID:server-e320dbad-2699-6079-715c-4786cc61d6da Address:e320dbad-2699-6079-715c-4786cc61d6da}]" -testLoggerAdapter verbose: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.751Z [INFO] server-2014d471-0917-43d2-0ae5-cc60457a30d8: removed peer, stopping replication: peer=server-18bf439f-22ed-da1c-a22f-2ced670f8fcf last-index=3 - testing.go:149: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.751Z [INFO] server-2014d471-0917-43d2-0ae5-cc60457a30d8: removed peer, stopping replication: peer=server-18bf439f-22ed-da1c-a22f-2ced670f8fcf last-index=3 -testLoggerAdapter verbose: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.751Z [ERROR] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to heartbeat to: peer=e320dbad-2699-6079-715c-4786cc61d6da error="failed to connect to peer: e320dbad-2699-6079-715c-4786cc61d6da" - testing.go:149: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.751Z [ERROR] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to heartbeat to: peer=e320dbad-2699-6079-715c-4786cc61d6da error="failed to connect to peer: e320dbad-2699-6079-715c-4786cc61d6da" -testLoggerAdapter verbose: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.751Z [ERROR] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to pipeline appendEntries: peer="{Voter server-b31b296e-93ea-2a01-fac4-b731701596e8 b31b296e-93ea-2a01-fac4-b731701596e8}" error="append pipeline closed" - testing.go:149: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.751Z [ERROR] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to pipeline appendEntries: peer="{Voter server-b31b296e-93ea-2a01-fac4-b731701596e8 b31b296e-93ea-2a01-fac4-b731701596e8}" error="append pipeline closed" -testLoggerAdapter verbose: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.751Z [INFO] server-2014d471-0917-43d2-0ae5-cc60457a30d8: aborting pipeline replication: peer="{Voter server-b31b296e-93ea-2a01-fac4-b731701596e8 b31b296e-93ea-2a01-fac4-b731701596e8}" - testing.go:149: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.751Z [INFO] server-2014d471-0917-43d2-0ae5-cc60457a30d8: aborting pipeline replication: peer="{Voter server-b31b296e-93ea-2a01-fac4-b731701596e8 b31b296e-93ea-2a01-fac4-b731701596e8}" -testLoggerAdapter verbose: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.751Z [ERROR] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to pipeline appendEntries: peer="{Voter server-e320dbad-2699-6079-715c-4786cc61d6da e320dbad-2699-6079-715c-4786cc61d6da}" error="append pipeline closed" - testing.go:149: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.751Z [ERROR] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to pipeline appendEntries: peer="{Voter server-e320dbad-2699-6079-715c-4786cc61d6da e320dbad-2699-6079-715c-4786cc61d6da}" error="append pipeline closed" -testLoggerAdapter verbose: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.751Z [INFO] server-2014d471-0917-43d2-0ae5-cc60457a30d8: aborting pipeline replication: peer="{Voter server-e320dbad-2699-6079-715c-4786cc61d6da e320dbad-2699-6079-715c-4786cc61d6da}" - testing.go:149: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.751Z [INFO] server-2014d471-0917-43d2-0ae5-cc60457a30d8: aborting pipeline replication: peer="{Voter server-e320dbad-2699-6079-715c-4786cc61d6da e320dbad-2699-6079-715c-4786cc61d6da}" -testLoggerAdapter verbose: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.751Z [INFO] server-2014d471-0917-43d2-0ae5-cc60457a30d8: aborting pipeline replication: peer="{Voter server-18bf439f-22ed-da1c-a22f-2ced670f8fcf 18bf439f-22ed-da1c-a22f-2ced670f8fcf}" - testing.go:149: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.751Z [INFO] server-2014d471-0917-43d2-0ae5-cc60457a30d8: aborting pipeline replication: peer="{Voter server-18bf439f-22ed-da1c-a22f-2ced670f8fcf 18bf439f-22ed-da1c-a22f-2ced670f8fcf}" -testLoggerAdapter verbose: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.752Z [ERROR] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to heartbeat to: peer=b31b296e-93ea-2a01-fac4-b731701596e8 error="failed to connect to peer: b31b296e-93ea-2a01-fac4-b731701596e8" - testing.go:149: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.752Z [ERROR] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to heartbeat to: peer=b31b296e-93ea-2a01-fac4-b731701596e8 error="failed to connect to peer: b31b296e-93ea-2a01-fac4-b731701596e8" -testLoggerAdapter verbose: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.756Z [ERROR] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to appendEntries to: peer="{Voter server-e320dbad-2699-6079-715c-4786cc61d6da e320dbad-2699-6079-715c-4786cc61d6da}" error="failed to connect to peer: e320dbad-2699-6079-715c-4786cc61d6da" - testing.go:149: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.756Z [ERROR] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to appendEntries to: peer="{Voter server-e320dbad-2699-6079-715c-4786cc61d6da e320dbad-2699-6079-715c-4786cc61d6da}" error="failed to connect to peer: e320dbad-2699-6079-715c-4786cc61d6da" -testLoggerAdapter verbose: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.758Z [ERROR] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to appendEntries to: peer="{Voter server-b31b296e-93ea-2a01-fac4-b731701596e8 b31b296e-93ea-2a01-fac4-b731701596e8}" error="failed to connect to peer: b31b296e-93ea-2a01-fac4-b731701596e8" - testing.go:149: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.758Z [ERROR] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to appendEntries to: peer="{Voter server-b31b296e-93ea-2a01-fac4-b731701596e8 b31b296e-93ea-2a01-fac4-b731701596e8}" error="failed to connect to peer: b31b296e-93ea-2a01-fac4-b731701596e8" -testLoggerAdapter verbose: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.767Z [ERROR] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to heartbeat to: peer=e320dbad-2699-6079-715c-4786cc61d6da error="failed to connect to peer: e320dbad-2699-6079-715c-4786cc61d6da" - testing.go:149: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.767Z [ERROR] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to heartbeat to: peer=e320dbad-2699-6079-715c-4786cc61d6da error="failed to connect to peer: e320dbad-2699-6079-715c-4786cc61d6da" -testLoggerAdapter verbose: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.770Z [ERROR] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to heartbeat to: peer=b31b296e-93ea-2a01-fac4-b731701596e8 error="failed to connect to peer: b31b296e-93ea-2a01-fac4-b731701596e8" - testing.go:149: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.770Z [ERROR] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to heartbeat to: peer=b31b296e-93ea-2a01-fac4-b731701596e8 error="failed to connect to peer: b31b296e-93ea-2a01-fac4-b731701596e8" -testLoggerAdapter verbose: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.772Z [ERROR] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to appendEntries to: peer="{Voter server-e320dbad-2699-6079-715c-4786cc61d6da e320dbad-2699-6079-715c-4786cc61d6da}" error="failed to connect to peer: e320dbad-2699-6079-715c-4786cc61d6da" - testing.go:149: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.772Z [ERROR] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to appendEntries to: peer="{Voter server-e320dbad-2699-6079-715c-4786cc61d6da e320dbad-2699-6079-715c-4786cc61d6da}" error="failed to connect to peer: e320dbad-2699-6079-715c-4786cc61d6da" -testLoggerAdapter verbose: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.773Z [ERROR] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to appendEntries to: peer="{Voter server-b31b296e-93ea-2a01-fac4-b731701596e8 b31b296e-93ea-2a01-fac4-b731701596e8}" error="failed to connect to peer: b31b296e-93ea-2a01-fac4-b731701596e8" - testing.go:149: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.773Z [ERROR] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to appendEntries to: peer="{Voter server-b31b296e-93ea-2a01-fac4-b731701596e8 b31b296e-93ea-2a01-fac4-b731701596e8}" error="failed to connect to peer: b31b296e-93ea-2a01-fac4-b731701596e8" -testLoggerAdapter verbose: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.785Z [ERROR] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to heartbeat to: peer=b31b296e-93ea-2a01-fac4-b731701596e8 error="failed to connect to peer: b31b296e-93ea-2a01-fac4-b731701596e8" - testing.go:149: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.785Z [ERROR] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to heartbeat to: peer=b31b296e-93ea-2a01-fac4-b731701596e8 error="failed to connect to peer: b31b296e-93ea-2a01-fac4-b731701596e8" -testLoggerAdapter verbose: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.786Z [ERROR] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to heartbeat to: peer=e320dbad-2699-6079-715c-4786cc61d6da error="failed to connect to peer: e320dbad-2699-6079-715c-4786cc61d6da" - testing.go:149: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.786Z [ERROR] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to heartbeat to: peer=e320dbad-2699-6079-715c-4786cc61d6da error="failed to connect to peer: e320dbad-2699-6079-715c-4786cc61d6da" -testLoggerAdapter verbose: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.790Z [ERROR] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to appendEntries to: peer="{Voter server-e320dbad-2699-6079-715c-4786cc61d6da e320dbad-2699-6079-715c-4786cc61d6da}" error="failed to connect to peer: e320dbad-2699-6079-715c-4786cc61d6da" - testing.go:149: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.790Z [ERROR] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to appendEntries to: peer="{Voter server-e320dbad-2699-6079-715c-4786cc61d6da e320dbad-2699-6079-715c-4786cc61d6da}" error="failed to connect to peer: e320dbad-2699-6079-715c-4786cc61d6da" -testLoggerAdapter verbose: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.794Z [ERROR] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to appendEntries to: peer="{Voter server-b31b296e-93ea-2a01-fac4-b731701596e8 b31b296e-93ea-2a01-fac4-b731701596e8}" error="failed to connect to peer: b31b296e-93ea-2a01-fac4-b731701596e8" - testing.go:149: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.794Z [ERROR] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to appendEntries to: peer="{Voter server-b31b296e-93ea-2a01-fac4-b731701596e8 b31b296e-93ea-2a01-fac4-b731701596e8}" error="failed to connect to peer: b31b296e-93ea-2a01-fac4-b731701596e8" -testLoggerAdapter verbose: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.798Z [WARN] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to contact: server-id=server-e320dbad-2699-6079-715c-4786cc61d6da time=50.082083ms - testing.go:149: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.798Z [WARN] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to contact: server-id=server-e320dbad-2699-6079-715c-4786cc61d6da time=50.082083ms -testLoggerAdapter verbose: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.808Z [WARN] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to contact: server-id=server-b31b296e-93ea-2a01-fac4-b731701596e8 time=57.563537ms - testing.go:149: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.808Z [WARN] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to contact: server-id=server-b31b296e-93ea-2a01-fac4-b731701596e8 time=57.563537ms -testLoggerAdapter verbose: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.808Z [WARN] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to contact: server-id=server-e320dbad-2699-6079-715c-4786cc61d6da time=60.247947ms - testing.go:149: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.808Z [WARN] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to contact: server-id=server-e320dbad-2699-6079-715c-4786cc61d6da time=60.247947ms -testLoggerAdapter verbose: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.808Z [WARN] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to contact quorum of nodes, stepping down - testing.go:149: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.808Z [WARN] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to contact quorum of nodes, stepping down -testLoggerAdapter verbose: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.808Z [INFO] server-2014d471-0917-43d2-0ae5-cc60457a30d8: entering follower state: follower="Node at 2014d471-0917-43d2-0ae5-cc60457a30d8 [Follower]" leader= - testing.go:149: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.808Z [INFO] server-2014d471-0917-43d2-0ae5-cc60457a30d8: entering follower state: follower="Node at 2014d471-0917-43d2-0ae5-cc60457a30d8 [Follower]" leader= -testLoggerAdapter verbose: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.808Z [ERROR] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to appendEntries to: peer="{Voter server-b31b296e-93ea-2a01-fac4-b731701596e8 b31b296e-93ea-2a01-fac4-b731701596e8}" error="failed to connect to peer: b31b296e-93ea-2a01-fac4-b731701596e8" - testing.go:149: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.808Z [ERROR] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to appendEntries to: peer="{Voter server-b31b296e-93ea-2a01-fac4-b731701596e8 b31b296e-93ea-2a01-fac4-b731701596e8}" error="failed to connect to peer: b31b296e-93ea-2a01-fac4-b731701596e8" -testLoggerAdapter verbose: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.808Z [ERROR] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to appendEntries to: peer="{Voter server-e320dbad-2699-6079-715c-4786cc61d6da e320dbad-2699-6079-715c-4786cc61d6da}" error="failed to connect to peer: e320dbad-2699-6079-715c-4786cc61d6da" - testing.go:149: server-2014d471-0917-43d2-0ae5-cc60457a30d8: 2024-09-03T21:34:28.808Z [ERROR] server-2014d471-0917-43d2-0ae5-cc60457a30d8: failed to appendEntries to: peer="{Voter server-e320dbad-2699-6079-715c-4786cc61d6da e320dbad-2699-6079-715c-4786cc61d6da}" error="failed to connect to peer: e320dbad-2699-6079-715c-4786cc61d6da" +testLoggerAdapter verbose: server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: 2023-08-02T15:12:57.345Z [INFO] server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 Address:2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32} {Suffrage:Voter ID:server-72d5cbac-9805-20ef-6943-1ce879c34793 Address:72d5cbac-9805-20ef-6943-1ce879c34793} {Suffrage:Voter ID:server-c352a243-aaff-1c2a-41f5-7d01e4e3980f Address:c352a243-aaff-1c2a-41f5-7d01e4e3980f} {Suffrage:Voter ID:server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 Address:af9bc9da-291e-572a-d18b-aa9cc6c426a8}]" + testing.go:149: server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: 2023-08-02T15:12:57.345Z [INFO] server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 Address:2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32} {Suffrage:Voter ID:server-72d5cbac-9805-20ef-6943-1ce879c34793 Address:72d5cbac-9805-20ef-6943-1ce879c34793} {Suffrage:Voter ID:server-c352a243-aaff-1c2a-41f5-7d01e4e3980f Address:c352a243-aaff-1c2a-41f5-7d01e4e3980f} {Suffrage:Voter ID:server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 Address:af9bc9da-291e-572a-d18b-aa9cc6c426a8}]" +testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.345Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 Address:2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32} {Suffrage:Voter ID:server-72d5cbac-9805-20ef-6943-1ce879c34793 Address:72d5cbac-9805-20ef-6943-1ce879c34793} {Suffrage:Voter ID:server-c352a243-aaff-1c2a-41f5-7d01e4e3980f Address:c352a243-aaff-1c2a-41f5-7d01e4e3980f} {Suffrage:Voter ID:server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 Address:af9bc9da-291e-572a-d18b-aa9cc6c426a8}]" + testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.345Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 Address:2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32} {Suffrage:Voter ID:server-72d5cbac-9805-20ef-6943-1ce879c34793 Address:72d5cbac-9805-20ef-6943-1ce879c34793} {Suffrage:Voter ID:server-c352a243-aaff-1c2a-41f5-7d01e4e3980f Address:c352a243-aaff-1c2a-41f5-7d01e4e3980f} {Suffrage:Voter ID:server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 Address:af9bc9da-291e-572a-d18b-aa9cc6c426a8}]" +testLoggerAdapter verbose: server-c352a243-aaff-1c2a-41f5-7d01e4e3980f: 2023-08-02T15:12:57.345Z [INFO] server-c352a243-aaff-1c2a-41f5-7d01e4e3980f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 Address:2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32} {Suffrage:Voter ID:server-72d5cbac-9805-20ef-6943-1ce879c34793 Address:72d5cbac-9805-20ef-6943-1ce879c34793} {Suffrage:Voter ID:server-c352a243-aaff-1c2a-41f5-7d01e4e3980f Address:c352a243-aaff-1c2a-41f5-7d01e4e3980f} {Suffrage:Voter ID:server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 Address:af9bc9da-291e-572a-d18b-aa9cc6c426a8}]" + testing.go:149: server-c352a243-aaff-1c2a-41f5-7d01e4e3980f: 2023-08-02T15:12:57.345Z [INFO] server-c352a243-aaff-1c2a-41f5-7d01e4e3980f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 Address:2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32} {Suffrage:Voter ID:server-72d5cbac-9805-20ef-6943-1ce879c34793 Address:72d5cbac-9805-20ef-6943-1ce879c34793} {Suffrage:Voter ID:server-c352a243-aaff-1c2a-41f5-7d01e4e3980f Address:c352a243-aaff-1c2a-41f5-7d01e4e3980f} {Suffrage:Voter ID:server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 Address:af9bc9da-291e-572a-d18b-aa9cc6c426a8}]" +testLoggerAdapter verbose: server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: 2023-08-02T15:12:57.345Z [INFO] server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: entering follower state: follower="Node at 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 [Follower]" leader= + testing.go:149: server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: 2023-08-02T15:12:57.345Z [INFO] server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: entering follower state: follower="Node at 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 [Follower]" leader= +testLoggerAdapter verbose: server-af9bc9da-291e-572a-d18b-aa9cc6c426a8: 2023-08-02T15:12:57.345Z [INFO] server-af9bc9da-291e-572a-d18b-aa9cc6c426a8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 Address:2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32} {Suffrage:Voter ID:server-72d5cbac-9805-20ef-6943-1ce879c34793 Address:72d5cbac-9805-20ef-6943-1ce879c34793} {Suffrage:Voter ID:server-c352a243-aaff-1c2a-41f5-7d01e4e3980f Address:c352a243-aaff-1c2a-41f5-7d01e4e3980f} {Suffrage:Voter ID:server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 Address:af9bc9da-291e-572a-d18b-aa9cc6c426a8}]" +testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.345Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: entering follower state: follower="Node at 72d5cbac-9805-20ef-6943-1ce879c34793 [Follower]" leader= + testing.go:149: server-af9bc9da-291e-572a-d18b-aa9cc6c426a8: 2023-08-02T15:12:57.345Z [INFO] server-af9bc9da-291e-572a-d18b-aa9cc6c426a8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 Address:2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32} {Suffrage:Voter ID:server-72d5cbac-9805-20ef-6943-1ce879c34793 Address:72d5cbac-9805-20ef-6943-1ce879c34793} {Suffrage:Voter ID:server-c352a243-aaff-1c2a-41f5-7d01e4e3980f Address:c352a243-aaff-1c2a-41f5-7d01e4e3980f} {Suffrage:Voter ID:server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 Address:af9bc9da-291e-572a-d18b-aa9cc6c426a8}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:57.345Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:12:57.345Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-c352a243-aaff-1c2a-41f5-7d01e4e3980f: 2023-08-02T15:12:57.345Z [INFO] server-c352a243-aaff-1c2a-41f5-7d01e4e3980f: entering follower state: follower="Node at c352a243-aaff-1c2a-41f5-7d01e4e3980f [Follower]" leader= + testing.go:149: server-c352a243-aaff-1c2a-41f5-7d01e4e3980f: 2023-08-02T15:12:57.345Z [INFO] server-c352a243-aaff-1c2a-41f5-7d01e4e3980f: entering follower state: follower="Node at c352a243-aaff-1c2a-41f5-7d01e4e3980f [Follower]" leader= + testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.345Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: entering follower state: follower="Node at 72d5cbac-9805-20ef-6943-1ce879c34793 [Follower]" leader= +testLoggerAdapter verbose: server-af9bc9da-291e-572a-d18b-aa9cc6c426a8: 2023-08-02T15:12:57.345Z [INFO] server-af9bc9da-291e-572a-d18b-aa9cc6c426a8: entering follower state: follower="Node at af9bc9da-291e-572a-d18b-aa9cc6c426a8 [Follower]" leader= + testing.go:149: server-af9bc9da-291e-572a-d18b-aa9cc6c426a8: 2023-08-02T15:12:57.345Z [INFO] server-af9bc9da-291e-572a-d18b-aa9cc6c426a8: entering follower state: follower="Node at af9bc9da-291e-572a-d18b-aa9cc6c426a8 [Follower]" leader= +testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.401Z [WARN] server-72d5cbac-9805-20ef-6943-1ce879c34793: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.401Z [WARN] server-72d5cbac-9805-20ef-6943-1ce879c34793: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.402Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: entering candidate state: node="Node at 72d5cbac-9805-20ef-6943-1ce879c34793 [Candidate]" term=2 + testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.402Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: entering candidate state: node="Node at 72d5cbac-9805-20ef-6943-1ce879c34793 [Candidate]" term=2 +testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.402Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: election won: tally=3 + testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.402Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: election won: tally=3 +testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.402Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: entering leader state: leader="Node at 72d5cbac-9805-20ef-6943-1ce879c34793 [Leader]" + testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.402Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: entering leader state: leader="Node at 72d5cbac-9805-20ef-6943-1ce879c34793 [Leader]" +testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.402Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: added peer, starting replication: peer=server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 + testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.402Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: added peer, starting replication: peer=server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 +testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.402Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: added peer, starting replication: peer=server-c352a243-aaff-1c2a-41f5-7d01e4e3980f + testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.402Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: added peer, starting replication: peer=server-c352a243-aaff-1c2a-41f5-7d01e4e3980f +testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.402Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: added peer, starting replication: peer=server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 + testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.402Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: added peer, starting replication: peer=server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 +testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.402Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: pipelining replication: peer="{Voter server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 af9bc9da-291e-572a-d18b-aa9cc6c426a8}" + testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.402Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: pipelining replication: peer="{Voter server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 af9bc9da-291e-572a-d18b-aa9cc6c426a8}" +testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.402Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: pipelining replication: peer="{Voter server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32}" + testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.402Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: pipelining replication: peer="{Voter server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32}" +testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.402Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: pipelining replication: peer="{Voter server-c352a243-aaff-1c2a-41f5-7d01e4e3980f c352a243-aaff-1c2a-41f5-7d01e4e3980f}" + testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.402Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: pipelining replication: peer="{Voter server-c352a243-aaff-1c2a-41f5-7d01e4e3980f c352a243-aaff-1c2a-41f5-7d01e4e3980f}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:57.507Z [INFO] stable state for Leader reached at 2023-08-02 15:12:57.402357675 +0000 UTC m=+9.498294029 (1 nodes), 56.742829ms from start of poll, 57.45646ms from cluster start. Timeout at 2023-08-02 15:12:57.507559901 +0000 UTC m=+9.603496236, 105.202207ms after stability + testing.go:149: cluster: 2023-08-02T15:12:57.507Z [INFO] stable state for Leader reached at 2023-08-02 15:12:57.402357675 +0000 UTC m=+9.498294029 (1 nodes), 56.742829ms from start of poll, 57.45646ms from cluster start. Timeout at 2023-08-02 15:12:57.507559901 +0000 UTC m=+9.603496236, 105.202207ms after stability +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:57.613Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-08-02T15:12:57.613Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:57.718Z [INFO] stable state for Follower reached at 2023-08-02 15:12:57.613119085 +0000 UTC m=+9.709055424 (3 nodes), 1.97µs from start of poll, 268.217855ms from cluster start. Timeout at 2023-08-02 15:12:57.718250306 +0000 UTC m=+9.814186639, 105.131215ms after stability + testing.go:149: cluster: 2023-08-02T15:12:57.718Z [INFO] stable state for Follower reached at 2023-08-02 15:12:57.613119085 +0000 UTC m=+9.709055424 (3 nodes), 1.97µs from start of poll, 268.217855ms from cluster start. Timeout at 2023-08-02 15:12:57.718250306 +0000 UTC m=+9.814186639, 105.131215ms after stability +testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.718Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: updating configuration: command=RemoveServer server-id=server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 server-addr= servers="[{Suffrage:Voter ID:server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 Address:2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32} {Suffrage:Voter ID:server-72d5cbac-9805-20ef-6943-1ce879c34793 Address:72d5cbac-9805-20ef-6943-1ce879c34793} {Suffrage:Voter ID:server-c352a243-aaff-1c2a-41f5-7d01e4e3980f Address:c352a243-aaff-1c2a-41f5-7d01e4e3980f}]" + testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.718Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: updating configuration: command=RemoveServer server-id=server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 server-addr= servers="[{Suffrage:Voter ID:server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 Address:2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32} {Suffrage:Voter ID:server-72d5cbac-9805-20ef-6943-1ce879c34793 Address:72d5cbac-9805-20ef-6943-1ce879c34793} {Suffrage:Voter ID:server-c352a243-aaff-1c2a-41f5-7d01e4e3980f Address:c352a243-aaff-1c2a-41f5-7d01e4e3980f}]" +testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.718Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: removed peer, stopping replication: peer=server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 last-index=3 + testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.718Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: removed peer, stopping replication: peer=server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 last-index=3 +testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.718Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to pipeline appendEntries: peer="{Voter server-c352a243-aaff-1c2a-41f5-7d01e4e3980f c352a243-aaff-1c2a-41f5-7d01e4e3980f}" error="append pipeline closed" + testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.718Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to pipeline appendEntries: peer="{Voter server-c352a243-aaff-1c2a-41f5-7d01e4e3980f c352a243-aaff-1c2a-41f5-7d01e4e3980f}" error="append pipeline closed" +testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.718Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: aborting pipeline replication: peer="{Voter server-c352a243-aaff-1c2a-41f5-7d01e4e3980f c352a243-aaff-1c2a-41f5-7d01e4e3980f}" + testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.718Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: aborting pipeline replication: peer="{Voter server-c352a243-aaff-1c2a-41f5-7d01e4e3980f c352a243-aaff-1c2a-41f5-7d01e4e3980f}" +testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.718Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to pipeline appendEntries: peer="{Voter server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32}" error="append pipeline closed" + testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.718Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to pipeline appendEntries: peer="{Voter server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32}" error="append pipeline closed" +testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.718Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: aborting pipeline replication: peer="{Voter server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32}" + testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.718Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: aborting pipeline replication: peer="{Voter server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32}" +testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.718Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to heartbeat to: peer=c352a243-aaff-1c2a-41f5-7d01e4e3980f error="failed to connect to peer: c352a243-aaff-1c2a-41f5-7d01e4e3980f" + testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.718Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to heartbeat to: peer=c352a243-aaff-1c2a-41f5-7d01e4e3980f error="failed to connect to peer: c352a243-aaff-1c2a-41f5-7d01e4e3980f" +testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.718Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: aborting pipeline replication: peer="{Voter server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 af9bc9da-291e-572a-d18b-aa9cc6c426a8}" + testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.718Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: aborting pipeline replication: peer="{Voter server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 af9bc9da-291e-572a-d18b-aa9cc6c426a8}" +testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.718Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to heartbeat to: peer=2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 error="failed to connect to peer: 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32" + testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.718Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to heartbeat to: peer=2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 error="failed to connect to peer: 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32" +testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.725Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to appendEntries to: peer="{Voter server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32}" error="failed to connect to peer: 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32" + testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.725Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to appendEntries to: peer="{Voter server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32}" error="failed to connect to peer: 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32" +testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.726Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to appendEntries to: peer="{Voter server-c352a243-aaff-1c2a-41f5-7d01e4e3980f c352a243-aaff-1c2a-41f5-7d01e4e3980f}" error="failed to connect to peer: c352a243-aaff-1c2a-41f5-7d01e4e3980f" + testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.726Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to appendEntries to: peer="{Voter server-c352a243-aaff-1c2a-41f5-7d01e4e3980f c352a243-aaff-1c2a-41f5-7d01e4e3980f}" error="failed to connect to peer: c352a243-aaff-1c2a-41f5-7d01e4e3980f" +testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.735Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to heartbeat to: peer=c352a243-aaff-1c2a-41f5-7d01e4e3980f error="failed to connect to peer: c352a243-aaff-1c2a-41f5-7d01e4e3980f" + testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.735Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to heartbeat to: peer=c352a243-aaff-1c2a-41f5-7d01e4e3980f error="failed to connect to peer: c352a243-aaff-1c2a-41f5-7d01e4e3980f" +testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.737Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to heartbeat to: peer=2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 error="failed to connect to peer: 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32" + testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.737Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to heartbeat to: peer=2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 error="failed to connect to peer: 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32" +testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.745Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to appendEntries to: peer="{Voter server-c352a243-aaff-1c2a-41f5-7d01e4e3980f c352a243-aaff-1c2a-41f5-7d01e4e3980f}" error="failed to connect to peer: c352a243-aaff-1c2a-41f5-7d01e4e3980f" + testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.745Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to appendEntries to: peer="{Voter server-c352a243-aaff-1c2a-41f5-7d01e4e3980f c352a243-aaff-1c2a-41f5-7d01e4e3980f}" error="failed to connect to peer: c352a243-aaff-1c2a-41f5-7d01e4e3980f" +testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.745Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to appendEntries to: peer="{Voter server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32}" error="failed to connect to peer: 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32" + testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.745Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to appendEntries to: peer="{Voter server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32}" error="failed to connect to peer: 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32" +testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.755Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to heartbeat to: peer=c352a243-aaff-1c2a-41f5-7d01e4e3980f error="failed to connect to peer: c352a243-aaff-1c2a-41f5-7d01e4e3980f" + testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.755Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to heartbeat to: peer=c352a243-aaff-1c2a-41f5-7d01e4e3980f error="failed to connect to peer: c352a243-aaff-1c2a-41f5-7d01e4e3980f" +testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.755Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to heartbeat to: peer=2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 error="failed to connect to peer: 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32" + testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.755Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to heartbeat to: peer=2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 error="failed to connect to peer: 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32" +testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.762Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to appendEntries to: peer="{Voter server-c352a243-aaff-1c2a-41f5-7d01e4e3980f c352a243-aaff-1c2a-41f5-7d01e4e3980f}" error="failed to connect to peer: c352a243-aaff-1c2a-41f5-7d01e4e3980f" + testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.762Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to appendEntries to: peer="{Voter server-c352a243-aaff-1c2a-41f5-7d01e4e3980f c352a243-aaff-1c2a-41f5-7d01e4e3980f}" error="failed to connect to peer: c352a243-aaff-1c2a-41f5-7d01e4e3980f" +testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.763Z [WARN] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to contact: server-id=server-c352a243-aaff-1c2a-41f5-7d01e4e3980f time=50.078705ms + testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.763Z [WARN] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to contact: server-id=server-c352a243-aaff-1c2a-41f5-7d01e4e3980f time=50.078705ms +testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.765Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to appendEntries to: peer="{Voter server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32}" error="failed to connect to peer: 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32" + testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.765Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to appendEntries to: peer="{Voter server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32}" error="failed to connect to peer: 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32" +testLoggerAdapter verbose: server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: 2023-08-02T15:12:57.772Z [WARN] server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: heartbeat timeout reached, starting election: last-leader=72d5cbac-9805-20ef-6943-1ce879c34793 + testing.go:149: server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: 2023-08-02T15:12:57.772Z [WARN] server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: heartbeat timeout reached, starting election: last-leader=72d5cbac-9805-20ef-6943-1ce879c34793 +testLoggerAdapter verbose: server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: 2023-08-02T15:12:57.772Z [INFO] server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: entering candidate state: node="Node at 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 [Candidate]" term=3 + testing.go:149: server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: 2023-08-02T15:12:57.772Z [INFO] server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: entering candidate state: node="Node at 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 [Candidate]" term=3 +testLoggerAdapter verbose: server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: 2023-08-02T15:12:57.772Z [ERROR] server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: failed to make requestVote RPC: target="{Voter server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 af9bc9da-291e-572a-d18b-aa9cc6c426a8}" error="failed to connect to peer: af9bc9da-291e-572a-d18b-aa9cc6c426a8" + testing.go:149: server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: 2023-08-02T15:12:57.772Z [ERROR] server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: failed to make requestVote RPC: target="{Voter server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 af9bc9da-291e-572a-d18b-aa9cc6c426a8}" error="failed to connect to peer: af9bc9da-291e-572a-d18b-aa9cc6c426a8" +testLoggerAdapter verbose: server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: 2023-08-02T15:12:57.772Z [ERROR] server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: failed to make requestVote RPC: target="{Voter server-72d5cbac-9805-20ef-6943-1ce879c34793 72d5cbac-9805-20ef-6943-1ce879c34793}" error="failed to connect to peer: 72d5cbac-9805-20ef-6943-1ce879c34793" + testing.go:149: server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: 2023-08-02T15:12:57.772Z [ERROR] server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: failed to make requestVote RPC: target="{Voter server-72d5cbac-9805-20ef-6943-1ce879c34793 72d5cbac-9805-20ef-6943-1ce879c34793}" error="failed to connect to peer: 72d5cbac-9805-20ef-6943-1ce879c34793" +testLoggerAdapter verbose: server-c352a243-aaff-1c2a-41f5-7d01e4e3980f: 2023-08-02T15:12:57.772Z [WARN] server-c352a243-aaff-1c2a-41f5-7d01e4e3980f: rejecting vote request since we have a leader: from=2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 leader=72d5cbac-9805-20ef-6943-1ce879c34793 + testing.go:149: server-c352a243-aaff-1c2a-41f5-7d01e4e3980f: 2023-08-02T15:12:57.772Z [WARN] server-c352a243-aaff-1c2a-41f5-7d01e4e3980f: rejecting vote request since we have a leader: from=2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 leader=72d5cbac-9805-20ef-6943-1ce879c34793 +testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.773Z [WARN] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to contact: server-id=server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 time=55.818744ms + testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.773Z [WARN] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to contact: server-id=server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 time=55.818744ms +testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.773Z [WARN] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to contact: server-id=server-c352a243-aaff-1c2a-41f5-7d01e4e3980f time=60.547117ms + testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.773Z [WARN] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to contact: server-id=server-c352a243-aaff-1c2a-41f5-7d01e4e3980f time=60.547117ms +testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.773Z [WARN] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to contact quorum of nodes, stepping down + testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.773Z [WARN] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to contact quorum of nodes, stepping down +testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.773Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: entering follower state: follower="Node at 72d5cbac-9805-20ef-6943-1ce879c34793 [Follower]" leader= + testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.773Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: entering follower state: follower="Node at 72d5cbac-9805-20ef-6943-1ce879c34793 [Follower]" leader= +testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.773Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to appendEntries to: peer="{Voter server-c352a243-aaff-1c2a-41f5-7d01e4e3980f c352a243-aaff-1c2a-41f5-7d01e4e3980f}" error="failed to connect to peer: c352a243-aaff-1c2a-41f5-7d01e4e3980f" + testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.773Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to appendEntries to: peer="{Voter server-c352a243-aaff-1c2a-41f5-7d01e4e3980f c352a243-aaff-1c2a-41f5-7d01e4e3980f}" error="failed to connect to peer: c352a243-aaff-1c2a-41f5-7d01e4e3980f" --- PASS: TestRaft_RemoveFollower_SplitCluster (0.43s) === RUN TestRaft_AddKnownPeer -testLoggerAdapter verbose: server-65d01bd5-94ce-6bd5-ede4-31438de563fb: 2024-09-03T21:34:28.809Z [INFO] server-65d01bd5-94ce-6bd5-ede4-31438de563fb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-65d01bd5-94ce-6bd5-ede4-31438de563fb Address:65d01bd5-94ce-6bd5-ede4-31438de563fb} {Suffrage:Voter ID:server-e5bcac0e-304e-3aac-0a4b-01ce64fab428 Address:e5bcac0e-304e-3aac-0a4b-01ce64fab428} {Suffrage:Voter ID:server-09447185-96cd-2301-d0d0-4d708b1fc277 Address:09447185-96cd-2301-d0d0-4d708b1fc277}]" - testing.go:149: server-65d01bd5-94ce-6bd5-ede4-31438de563fb: 2024-09-03T21:34:28.809Z [INFO] server-65d01bd5-94ce-6bd5-ede4-31438de563fb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-65d01bd5-94ce-6bd5-ede4-31438de563fb Address:65d01bd5-94ce-6bd5-ede4-31438de563fb} {Suffrage:Voter ID:server-e5bcac0e-304e-3aac-0a4b-01ce64fab428 Address:e5bcac0e-304e-3aac-0a4b-01ce64fab428} {Suffrage:Voter ID:server-09447185-96cd-2301-d0d0-4d708b1fc277 Address:09447185-96cd-2301-d0d0-4d708b1fc277}]" -testLoggerAdapter verbose: server-e5bcac0e-304e-3aac-0a4b-01ce64fab428: 2024-09-03T21:34:28.809Z [INFO] server-e5bcac0e-304e-3aac-0a4b-01ce64fab428: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-65d01bd5-94ce-6bd5-ede4-31438de563fb Address:65d01bd5-94ce-6bd5-ede4-31438de563fb} {Suffrage:Voter ID:server-e5bcac0e-304e-3aac-0a4b-01ce64fab428 Address:e5bcac0e-304e-3aac-0a4b-01ce64fab428} {Suffrage:Voter ID:server-09447185-96cd-2301-d0d0-4d708b1fc277 Address:09447185-96cd-2301-d0d0-4d708b1fc277}]" - testing.go:149: server-e5bcac0e-304e-3aac-0a4b-01ce64fab428: 2024-09-03T21:34:28.809Z [INFO] server-e5bcac0e-304e-3aac-0a4b-01ce64fab428: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-65d01bd5-94ce-6bd5-ede4-31438de563fb Address:65d01bd5-94ce-6bd5-ede4-31438de563fb} {Suffrage:Voter ID:server-e5bcac0e-304e-3aac-0a4b-01ce64fab428 Address:e5bcac0e-304e-3aac-0a4b-01ce64fab428} {Suffrage:Voter ID:server-09447185-96cd-2301-d0d0-4d708b1fc277 Address:09447185-96cd-2301-d0d0-4d708b1fc277}]" -testLoggerAdapter verbose: server-09447185-96cd-2301-d0d0-4d708b1fc277: 2024-09-03T21:34:28.809Z [INFO] server-09447185-96cd-2301-d0d0-4d708b1fc277: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-65d01bd5-94ce-6bd5-ede4-31438de563fb Address:65d01bd5-94ce-6bd5-ede4-31438de563fb} {Suffrage:Voter ID:server-e5bcac0e-304e-3aac-0a4b-01ce64fab428 Address:e5bcac0e-304e-3aac-0a4b-01ce64fab428} {Suffrage:Voter ID:server-09447185-96cd-2301-d0d0-4d708b1fc277 Address:09447185-96cd-2301-d0d0-4d708b1fc277}]" - testing.go:149: server-09447185-96cd-2301-d0d0-4d708b1fc277: 2024-09-03T21:34:28.809Z [INFO] server-09447185-96cd-2301-d0d0-4d708b1fc277: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-65d01bd5-94ce-6bd5-ede4-31438de563fb Address:65d01bd5-94ce-6bd5-ede4-31438de563fb} {Suffrage:Voter ID:server-e5bcac0e-304e-3aac-0a4b-01ce64fab428 Address:e5bcac0e-304e-3aac-0a4b-01ce64fab428} {Suffrage:Voter ID:server-09447185-96cd-2301-d0d0-4d708b1fc277 Address:09447185-96cd-2301-d0d0-4d708b1fc277}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:28.809Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:28.809Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-65d01bd5-94ce-6bd5-ede4-31438de563fb: 2024-09-03T21:34:28.810Z [INFO] server-65d01bd5-94ce-6bd5-ede4-31438de563fb: entering follower state: follower="Node at 65d01bd5-94ce-6bd5-ede4-31438de563fb [Follower]" leader= - testing.go:149: server-65d01bd5-94ce-6bd5-ede4-31438de563fb: 2024-09-03T21:34:28.810Z [INFO] server-65d01bd5-94ce-6bd5-ede4-31438de563fb: entering follower state: follower="Node at 65d01bd5-94ce-6bd5-ede4-31438de563fb [Follower]" leader= -testLoggerAdapter verbose: server-e5bcac0e-304e-3aac-0a4b-01ce64fab428: 2024-09-03T21:34:28.810Z [INFO] server-e5bcac0e-304e-3aac-0a4b-01ce64fab428: entering follower state: follower="Node at e5bcac0e-304e-3aac-0a4b-01ce64fab428 [Follower]" leader= - testing.go:149: server-e5bcac0e-304e-3aac-0a4b-01ce64fab428: 2024-09-03T21:34:28.810Z [INFO] server-e5bcac0e-304e-3aac-0a4b-01ce64fab428: entering follower state: follower="Node at e5bcac0e-304e-3aac-0a4b-01ce64fab428 [Follower]" leader= -testLoggerAdapter verbose: server-09447185-96cd-2301-d0d0-4d708b1fc277: 2024-09-03T21:34:28.810Z [INFO] server-09447185-96cd-2301-d0d0-4d708b1fc277: entering follower state: follower="Node at 09447185-96cd-2301-d0d0-4d708b1fc277 [Follower]" leader= - testing.go:149: server-09447185-96cd-2301-d0d0-4d708b1fc277: 2024-09-03T21:34:28.810Z [INFO] server-09447185-96cd-2301-d0d0-4d708b1fc277: entering follower state: follower="Node at 09447185-96cd-2301-d0d0-4d708b1fc277 [Follower]" leader= -testLoggerAdapter verbose: server-65d01bd5-94ce-6bd5-ede4-31438de563fb: 2024-09-03T21:34:28.871Z [WARN] server-65d01bd5-94ce-6bd5-ede4-31438de563fb: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-65d01bd5-94ce-6bd5-ede4-31438de563fb: 2024-09-03T21:34:28.871Z [WARN] server-65d01bd5-94ce-6bd5-ede4-31438de563fb: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-65d01bd5-94ce-6bd5-ede4-31438de563fb: 2024-09-03T21:34:28.871Z [INFO] server-65d01bd5-94ce-6bd5-ede4-31438de563fb: entering candidate state: node="Node at 65d01bd5-94ce-6bd5-ede4-31438de563fb [Candidate]" term=2 - testing.go:149: server-65d01bd5-94ce-6bd5-ede4-31438de563fb: 2024-09-03T21:34:28.871Z [INFO] server-65d01bd5-94ce-6bd5-ede4-31438de563fb: entering candidate state: node="Node at 65d01bd5-94ce-6bd5-ede4-31438de563fb [Candidate]" term=2 -testLoggerAdapter verbose: server-65d01bd5-94ce-6bd5-ede4-31438de563fb: 2024-09-03T21:34:28.871Z [INFO] server-65d01bd5-94ce-6bd5-ede4-31438de563fb: election won: tally=2 - testing.go:149: server-65d01bd5-94ce-6bd5-ede4-31438de563fb: 2024-09-03T21:34:28.871Z [INFO] server-65d01bd5-94ce-6bd5-ede4-31438de563fb: election won: tally=2 -testLoggerAdapter verbose: server-65d01bd5-94ce-6bd5-ede4-31438de563fb: 2024-09-03T21:34:28.871Z [INFO] server-65d01bd5-94ce-6bd5-ede4-31438de563fb: entering leader state: leader="Node at 65d01bd5-94ce-6bd5-ede4-31438de563fb [Leader]" - testing.go:149: server-65d01bd5-94ce-6bd5-ede4-31438de563fb: 2024-09-03T21:34:28.871Z [INFO] server-65d01bd5-94ce-6bd5-ede4-31438de563fb: entering leader state: leader="Node at 65d01bd5-94ce-6bd5-ede4-31438de563fb [Leader]" -testLoggerAdapter verbose: server-65d01bd5-94ce-6bd5-ede4-31438de563fb: 2024-09-03T21:34:28.871Z [INFO] server-65d01bd5-94ce-6bd5-ede4-31438de563fb: added peer, starting replication: peer=server-e5bcac0e-304e-3aac-0a4b-01ce64fab428 - testing.go:149: server-65d01bd5-94ce-6bd5-ede4-31438de563fb: 2024-09-03T21:34:28.871Z [INFO] server-65d01bd5-94ce-6bd5-ede4-31438de563fb: added peer, starting replication: peer=server-e5bcac0e-304e-3aac-0a4b-01ce64fab428 -testLoggerAdapter verbose: server-65d01bd5-94ce-6bd5-ede4-31438de563fb: 2024-09-03T21:34:28.871Z [INFO] server-65d01bd5-94ce-6bd5-ede4-31438de563fb: added peer, starting replication: peer=server-09447185-96cd-2301-d0d0-4d708b1fc277 - testing.go:149: server-65d01bd5-94ce-6bd5-ede4-31438de563fb: 2024-09-03T21:34:28.871Z [INFO] server-65d01bd5-94ce-6bd5-ede4-31438de563fb: added peer, starting replication: peer=server-09447185-96cd-2301-d0d0-4d708b1fc277 -testLoggerAdapter verbose: server-65d01bd5-94ce-6bd5-ede4-31438de563fb: 2024-09-03T21:34:28.871Z [INFO] server-65d01bd5-94ce-6bd5-ede4-31438de563fb: pipelining replication: peer="{Voter server-09447185-96cd-2301-d0d0-4d708b1fc277 09447185-96cd-2301-d0d0-4d708b1fc277}" - testing.go:149: server-65d01bd5-94ce-6bd5-ede4-31438de563fb: 2024-09-03T21:34:28.871Z [INFO] server-65d01bd5-94ce-6bd5-ede4-31438de563fb: pipelining replication: peer="{Voter server-09447185-96cd-2301-d0d0-4d708b1fc277 09447185-96cd-2301-d0d0-4d708b1fc277}" -testLoggerAdapter verbose: server-65d01bd5-94ce-6bd5-ede4-31438de563fb: 2024-09-03T21:34:28.872Z [INFO] server-65d01bd5-94ce-6bd5-ede4-31438de563fb: pipelining replication: peer="{Voter server-e5bcac0e-304e-3aac-0a4b-01ce64fab428 e5bcac0e-304e-3aac-0a4b-01ce64fab428}" - testing.go:149: server-65d01bd5-94ce-6bd5-ede4-31438de563fb: 2024-09-03T21:34:28.872Z [INFO] server-65d01bd5-94ce-6bd5-ede4-31438de563fb: pipelining replication: peer="{Voter server-e5bcac0e-304e-3aac-0a4b-01ce64fab428 e5bcac0e-304e-3aac-0a4b-01ce64fab428}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:28.977Z [INFO] stable state for Leader reached at 2024-09-03 21:34:28.872174643 +0000 UTC m=+9.918831788 (1 nodes), 62.1833ms from start of poll, 62.564481ms from cluster start. Timeout at 2024-09-03 21:34:28.977254922 +0000 UTC m=+10.023912057, 105.080269ms after stability - testing.go:149: cluster: 2024-09-03T21:34:28.977Z [INFO] stable state for Leader reached at 2024-09-03 21:34:28.872174643 +0000 UTC m=+9.918831788 (1 nodes), 62.1833ms from start of poll, 62.564481ms from cluster start. Timeout at 2024-09-03 21:34:28.977254922 +0000 UTC m=+10.023912057, 105.080269ms after stability -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:28.977Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-09-03T21:34:28.977Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:29.082Z [INFO] stable state for Follower reached at 2024-09-03 21:34:28.977401781 +0000 UTC m=+10.024058919 (2 nodes), 1.199µs from start of poll, 167.791612ms from cluster start. Timeout at 2024-09-03 21:34:29.082494721 +0000 UTC m=+10.129151854, 105.092935ms after stability - testing.go:149: cluster: 2024-09-03T21:34:29.082Z [INFO] stable state for Follower reached at 2024-09-03 21:34:28.977401781 +0000 UTC m=+10.024058919 (2 nodes), 1.199µs from start of poll, 167.791612ms from cluster start. Timeout at 2024-09-03 21:34:29.082494721 +0000 UTC m=+10.129151854, 105.092935ms after stability -testLoggerAdapter verbose: server-65d01bd5-94ce-6bd5-ede4-31438de563fb: 2024-09-03T21:34:29.082Z [INFO] server-65d01bd5-94ce-6bd5-ede4-31438de563fb: updating configuration: command=AddStaging server-id=server-e5bcac0e-304e-3aac-0a4b-01ce64fab428 server-addr=e5bcac0e-304e-3aac-0a4b-01ce64fab428 servers="[{Suffrage:Voter ID:server-65d01bd5-94ce-6bd5-ede4-31438de563fb Address:65d01bd5-94ce-6bd5-ede4-31438de563fb} {Suffrage:Voter ID:server-e5bcac0e-304e-3aac-0a4b-01ce64fab428 Address:e5bcac0e-304e-3aac-0a4b-01ce64fab428} {Suffrage:Voter ID:server-09447185-96cd-2301-d0d0-4d708b1fc277 Address:09447185-96cd-2301-d0d0-4d708b1fc277}]" - testing.go:149: server-65d01bd5-94ce-6bd5-ede4-31438de563fb: 2024-09-03T21:34:29.082Z [INFO] server-65d01bd5-94ce-6bd5-ede4-31438de563fb: updating configuration: command=AddStaging server-id=server-e5bcac0e-304e-3aac-0a4b-01ce64fab428 server-addr=e5bcac0e-304e-3aac-0a4b-01ce64fab428 servers="[{Suffrage:Voter ID:server-65d01bd5-94ce-6bd5-ede4-31438de563fb Address:65d01bd5-94ce-6bd5-ede4-31438de563fb} {Suffrage:Voter ID:server-e5bcac0e-304e-3aac-0a4b-01ce64fab428 Address:e5bcac0e-304e-3aac-0a4b-01ce64fab428} {Suffrage:Voter ID:server-09447185-96cd-2301-d0d0-4d708b1fc277 Address:09447185-96cd-2301-d0d0-4d708b1fc277}]" -testLoggerAdapter verbose: server-65d01bd5-94ce-6bd5-ede4-31438de563fb: 2024-09-03T21:34:29.082Z [INFO] server-65d01bd5-94ce-6bd5-ede4-31438de563fb: aborting pipeline replication: peer="{Voter server-09447185-96cd-2301-d0d0-4d708b1fc277 09447185-96cd-2301-d0d0-4d708b1fc277}" - testing.go:149: server-65d01bd5-94ce-6bd5-ede4-31438de563fb: 2024-09-03T21:34:29.082Z [INFO] server-65d01bd5-94ce-6bd5-ede4-31438de563fb: aborting pipeline replication: peer="{Voter server-09447185-96cd-2301-d0d0-4d708b1fc277 09447185-96cd-2301-d0d0-4d708b1fc277}" -testLoggerAdapter verbose: server-65d01bd5-94ce-6bd5-ede4-31438de563fb: 2024-09-03T21:34:29.082Z [INFO] server-65d01bd5-94ce-6bd5-ede4-31438de563fb: aborting pipeline replication: peer="{Voter server-e5bcac0e-304e-3aac-0a4b-01ce64fab428 e5bcac0e-304e-3aac-0a4b-01ce64fab428}" - testing.go:149: server-65d01bd5-94ce-6bd5-ede4-31438de563fb: 2024-09-03T21:34:29.082Z [INFO] server-65d01bd5-94ce-6bd5-ede4-31438de563fb: aborting pipeline replication: peer="{Voter server-e5bcac0e-304e-3aac-0a4b-01ce64fab428 e5bcac0e-304e-3aac-0a4b-01ce64fab428}" +testLoggerAdapter verbose: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.775Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968 Address:edc63f06-c6c2-f61c-4a9a-b0fb889b2968} {Suffrage:Voter ID:server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd Address:13ecbf72-8fd5-c6d3-559b-ee06ae55cddd} {Suffrage:Voter ID:server-2393508c-a264-c68d-6a01-4620c05d8d00 Address:2393508c-a264-c68d-6a01-4620c05d8d00}]" + testing.go:149: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.775Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968 Address:edc63f06-c6c2-f61c-4a9a-b0fb889b2968} {Suffrage:Voter ID:server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd Address:13ecbf72-8fd5-c6d3-559b-ee06ae55cddd} {Suffrage:Voter ID:server-2393508c-a264-c68d-6a01-4620c05d8d00 Address:2393508c-a264-c68d-6a01-4620c05d8d00}]" +testLoggerAdapter verbose: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.775Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: entering follower state: follower="Node at edc63f06-c6c2-f61c-4a9a-b0fb889b2968 [Follower]" leader= +testLoggerAdapter verbose: server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd: 2023-08-02T15:12:57.775Z [INFO] server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968 Address:edc63f06-c6c2-f61c-4a9a-b0fb889b2968} {Suffrage:Voter ID:server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd Address:13ecbf72-8fd5-c6d3-559b-ee06ae55cddd} {Suffrage:Voter ID:server-2393508c-a264-c68d-6a01-4620c05d8d00 Address:2393508c-a264-c68d-6a01-4620c05d8d00}]" + testing.go:149: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.775Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: entering follower state: follower="Node at edc63f06-c6c2-f61c-4a9a-b0fb889b2968 [Follower]" leader= + testing.go:149: server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd: 2023-08-02T15:12:57.775Z [INFO] server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968 Address:edc63f06-c6c2-f61c-4a9a-b0fb889b2968} {Suffrage:Voter ID:server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd Address:13ecbf72-8fd5-c6d3-559b-ee06ae55cddd} {Suffrage:Voter ID:server-2393508c-a264-c68d-6a01-4620c05d8d00 Address:2393508c-a264-c68d-6a01-4620c05d8d00}]" +testLoggerAdapter verbose: server-2393508c-a264-c68d-6a01-4620c05d8d00: 2023-08-02T15:12:57.776Z [INFO] server-2393508c-a264-c68d-6a01-4620c05d8d00: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968 Address:edc63f06-c6c2-f61c-4a9a-b0fb889b2968} {Suffrage:Voter ID:server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd Address:13ecbf72-8fd5-c6d3-559b-ee06ae55cddd} {Suffrage:Voter ID:server-2393508c-a264-c68d-6a01-4620c05d8d00 Address:2393508c-a264-c68d-6a01-4620c05d8d00}]" + testing.go:149: server-2393508c-a264-c68d-6a01-4620c05d8d00: 2023-08-02T15:12:57.776Z [INFO] server-2393508c-a264-c68d-6a01-4620c05d8d00: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968 Address:edc63f06-c6c2-f61c-4a9a-b0fb889b2968} {Suffrage:Voter ID:server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd Address:13ecbf72-8fd5-c6d3-559b-ee06ae55cddd} {Suffrage:Voter ID:server-2393508c-a264-c68d-6a01-4620c05d8d00 Address:2393508c-a264-c68d-6a01-4620c05d8d00}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:57.776Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:12:57.776Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd: 2023-08-02T15:12:57.776Z [INFO] server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd: entering follower state: follower="Node at 13ecbf72-8fd5-c6d3-559b-ee06ae55cddd [Follower]" leader= + testing.go:149: server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd: 2023-08-02T15:12:57.776Z [INFO] server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd: entering follower state: follower="Node at 13ecbf72-8fd5-c6d3-559b-ee06ae55cddd [Follower]" leader= +testLoggerAdapter verbose: server-2393508c-a264-c68d-6a01-4620c05d8d00: 2023-08-02T15:12:57.776Z [INFO] server-2393508c-a264-c68d-6a01-4620c05d8d00: entering follower state: follower="Node at 2393508c-a264-c68d-6a01-4620c05d8d00 [Follower]" leader= + testing.go:149: server-2393508c-a264-c68d-6a01-4620c05d8d00: 2023-08-02T15:12:57.776Z [INFO] server-2393508c-a264-c68d-6a01-4620c05d8d00: entering follower state: follower="Node at 2393508c-a264-c68d-6a01-4620c05d8d00 [Follower]" leader= +testLoggerAdapter verbose: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.835Z [WARN] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.835Z [WARN] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.835Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: entering candidate state: node="Node at edc63f06-c6c2-f61c-4a9a-b0fb889b2968 [Candidate]" term=2 + testing.go:149: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.835Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: entering candidate state: node="Node at edc63f06-c6c2-f61c-4a9a-b0fb889b2968 [Candidate]" term=2 +testLoggerAdapter verbose: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.835Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: election won: tally=2 + testing.go:149: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.835Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: election won: tally=2 +testLoggerAdapter verbose: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.835Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: entering leader state: leader="Node at edc63f06-c6c2-f61c-4a9a-b0fb889b2968 [Leader]" + testing.go:149: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.835Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: entering leader state: leader="Node at edc63f06-c6c2-f61c-4a9a-b0fb889b2968 [Leader]" +testLoggerAdapter verbose: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.835Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: added peer, starting replication: peer=server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd + testing.go:149: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.835Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: added peer, starting replication: peer=server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd +testLoggerAdapter verbose: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.835Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: added peer, starting replication: peer=server-2393508c-a264-c68d-6a01-4620c05d8d00 + testing.go:149: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.835Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: added peer, starting replication: peer=server-2393508c-a264-c68d-6a01-4620c05d8d00 +testLoggerAdapter verbose: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.835Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: pipelining replication: peer="{Voter server-2393508c-a264-c68d-6a01-4620c05d8d00 2393508c-a264-c68d-6a01-4620c05d8d00}" + testing.go:149: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.835Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: pipelining replication: peer="{Voter server-2393508c-a264-c68d-6a01-4620c05d8d00 2393508c-a264-c68d-6a01-4620c05d8d00}" +testLoggerAdapter verbose: server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd: 2023-08-02T15:12:57.835Z [WARN] server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd: rejecting vote request since our last term is greater: candidate=edc63f06-c6c2-f61c-4a9a-b0fb889b2968 last-term=2 last-candidate-term=1 + testing.go:149: server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd: 2023-08-02T15:12:57.835Z [WARN] server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd: rejecting vote request since our last term is greater: candidate=edc63f06-c6c2-f61c-4a9a-b0fb889b2968 last-term=2 last-candidate-term=1 +testLoggerAdapter verbose: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.835Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: pipelining replication: peer="{Voter server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd 13ecbf72-8fd5-c6d3-559b-ee06ae55cddd}" + testing.go:149: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.835Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: pipelining replication: peer="{Voter server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd 13ecbf72-8fd5-c6d3-559b-ee06ae55cddd}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:57.940Z [INFO] stable state for Leader reached at 2023-08-02 15:12:57.835642048 +0000 UTC m=+9.931578382 (1 nodes), 59.407415ms from start of poll, 60.744835ms from cluster start. Timeout at 2023-08-02 15:12:57.94072152 +0000 UTC m=+10.036657888, 105.079506ms after stability + testing.go:149: cluster: 2023-08-02T15:12:57.940Z [INFO] stable state for Leader reached at 2023-08-02 15:12:57.835642048 +0000 UTC m=+9.931578382 (1 nodes), 59.407415ms from start of poll, 60.744835ms from cluster start. Timeout at 2023-08-02 15:12:57.94072152 +0000 UTC m=+10.036657888, 105.079506ms after stability +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:57.940Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-08-02T15:12:57.940Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:58.046Z [INFO] stable state for Follower reached at 2023-08-02 15:12:57.940942397 +0000 UTC m=+10.036878734 (2 nodes), 1.85µs from start of poll, 166.045187ms from cluster start. Timeout at 2023-08-02 15:12:58.045974526 +0000 UTC m=+10.141910851, 105.032117ms after stability + testing.go:149: cluster: 2023-08-02T15:12:58.046Z [INFO] stable state for Follower reached at 2023-08-02 15:12:57.940942397 +0000 UTC m=+10.036878734 (2 nodes), 1.85µs from start of poll, 166.045187ms from cluster start. Timeout at 2023-08-02 15:12:58.045974526 +0000 UTC m=+10.141910851, 105.032117ms after stability +testLoggerAdapter verbose: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:58.046Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: updating configuration: command=AddStaging server-id=server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd server-addr=13ecbf72-8fd5-c6d3-559b-ee06ae55cddd servers="[{Suffrage:Voter ID:server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968 Address:edc63f06-c6c2-f61c-4a9a-b0fb889b2968} {Suffrage:Voter ID:server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd Address:13ecbf72-8fd5-c6d3-559b-ee06ae55cddd} {Suffrage:Voter ID:server-2393508c-a264-c68d-6a01-4620c05d8d00 Address:2393508c-a264-c68d-6a01-4620c05d8d00}]" + testing.go:149: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:58.046Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: updating configuration: command=AddStaging server-id=server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd server-addr=13ecbf72-8fd5-c6d3-559b-ee06ae55cddd servers="[{Suffrage:Voter ID:server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968 Address:edc63f06-c6c2-f61c-4a9a-b0fb889b2968} {Suffrage:Voter ID:server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd Address:13ecbf72-8fd5-c6d3-559b-ee06ae55cddd} {Suffrage:Voter ID:server-2393508c-a264-c68d-6a01-4620c05d8d00 Address:2393508c-a264-c68d-6a01-4620c05d8d00}]" +testLoggerAdapter verbose: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:58.046Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: aborting pipeline replication: peer="{Voter server-2393508c-a264-c68d-6a01-4620c05d8d00 2393508c-a264-c68d-6a01-4620c05d8d00}" + testing.go:149: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:58.046Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: aborting pipeline replication: peer="{Voter server-2393508c-a264-c68d-6a01-4620c05d8d00 2393508c-a264-c68d-6a01-4620c05d8d00}" +testLoggerAdapter verbose: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:58.046Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: aborting pipeline replication: peer="{Voter server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd 13ecbf72-8fd5-c6d3-559b-ee06ae55cddd}" + testing.go:149: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:58.046Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: aborting pipeline replication: peer="{Voter server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd 13ecbf72-8fd5-c6d3-559b-ee06ae55cddd}" --- PASS: TestRaft_AddKnownPeer (0.27s) === RUN TestRaft_RemoveUnknownPeer -testLoggerAdapter verbose: server-0ff7f206-9aaa-6016-437a-3026ff55f213: 2024-09-03T21:34:29.085Z [INFO] server-0ff7f206-9aaa-6016-437a-3026ff55f213: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0ff7f206-9aaa-6016-437a-3026ff55f213 Address:0ff7f206-9aaa-6016-437a-3026ff55f213} {Suffrage:Voter ID:server-56537bcc-6cab-5d3c-b914-a919fb076447 Address:56537bcc-6cab-5d3c-b914-a919fb076447} {Suffrage:Voter ID:server-fa3794bb-d2e6-2a90-2332-640f9b372bca Address:fa3794bb-d2e6-2a90-2332-640f9b372bca}]" - testing.go:149: server-0ff7f206-9aaa-6016-437a-3026ff55f213: 2024-09-03T21:34:29.085Z [INFO] server-0ff7f206-9aaa-6016-437a-3026ff55f213: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0ff7f206-9aaa-6016-437a-3026ff55f213 Address:0ff7f206-9aaa-6016-437a-3026ff55f213} {Suffrage:Voter ID:server-56537bcc-6cab-5d3c-b914-a919fb076447 Address:56537bcc-6cab-5d3c-b914-a919fb076447} {Suffrage:Voter ID:server-fa3794bb-d2e6-2a90-2332-640f9b372bca Address:fa3794bb-d2e6-2a90-2332-640f9b372bca}]" -testLoggerAdapter verbose: server-56537bcc-6cab-5d3c-b914-a919fb076447: 2024-09-03T21:34:29.085Z [INFO] server-56537bcc-6cab-5d3c-b914-a919fb076447: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0ff7f206-9aaa-6016-437a-3026ff55f213 Address:0ff7f206-9aaa-6016-437a-3026ff55f213} {Suffrage:Voter ID:server-56537bcc-6cab-5d3c-b914-a919fb076447 Address:56537bcc-6cab-5d3c-b914-a919fb076447} {Suffrage:Voter ID:server-fa3794bb-d2e6-2a90-2332-640f9b372bca Address:fa3794bb-d2e6-2a90-2332-640f9b372bca}]" -testLoggerAdapter verbose: server-0ff7f206-9aaa-6016-437a-3026ff55f213: 2024-09-03T21:34:29.085Z [INFO] server-0ff7f206-9aaa-6016-437a-3026ff55f213: entering follower state: follower="Node at 0ff7f206-9aaa-6016-437a-3026ff55f213 [Follower]" leader= - testing.go:149: server-56537bcc-6cab-5d3c-b914-a919fb076447: 2024-09-03T21:34:29.085Z [INFO] server-56537bcc-6cab-5d3c-b914-a919fb076447: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0ff7f206-9aaa-6016-437a-3026ff55f213 Address:0ff7f206-9aaa-6016-437a-3026ff55f213} {Suffrage:Voter ID:server-56537bcc-6cab-5d3c-b914-a919fb076447 Address:56537bcc-6cab-5d3c-b914-a919fb076447} {Suffrage:Voter ID:server-fa3794bb-d2e6-2a90-2332-640f9b372bca Address:fa3794bb-d2e6-2a90-2332-640f9b372bca}]" -testLoggerAdapter verbose: server-fa3794bb-d2e6-2a90-2332-640f9b372bca: 2024-09-03T21:34:29.085Z [INFO] server-fa3794bb-d2e6-2a90-2332-640f9b372bca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0ff7f206-9aaa-6016-437a-3026ff55f213 Address:0ff7f206-9aaa-6016-437a-3026ff55f213} {Suffrage:Voter ID:server-56537bcc-6cab-5d3c-b914-a919fb076447 Address:56537bcc-6cab-5d3c-b914-a919fb076447} {Suffrage:Voter ID:server-fa3794bb-d2e6-2a90-2332-640f9b372bca Address:fa3794bb-d2e6-2a90-2332-640f9b372bca}]" - testing.go:149: server-fa3794bb-d2e6-2a90-2332-640f9b372bca: 2024-09-03T21:34:29.085Z [INFO] server-fa3794bb-d2e6-2a90-2332-640f9b372bca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0ff7f206-9aaa-6016-437a-3026ff55f213 Address:0ff7f206-9aaa-6016-437a-3026ff55f213} {Suffrage:Voter ID:server-56537bcc-6cab-5d3c-b914-a919fb076447 Address:56537bcc-6cab-5d3c-b914-a919fb076447} {Suffrage:Voter ID:server-fa3794bb-d2e6-2a90-2332-640f9b372bca Address:fa3794bb-d2e6-2a90-2332-640f9b372bca}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:29.085Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:29.085Z [INFO] starting stability test: raft-state=Leader - testing.go:149: server-0ff7f206-9aaa-6016-437a-3026ff55f213: 2024-09-03T21:34:29.085Z [INFO] server-0ff7f206-9aaa-6016-437a-3026ff55f213: entering follower state: follower="Node at 0ff7f206-9aaa-6016-437a-3026ff55f213 [Follower]" leader= -testLoggerAdapter verbose: server-56537bcc-6cab-5d3c-b914-a919fb076447: 2024-09-03T21:34:29.085Z [INFO] server-56537bcc-6cab-5d3c-b914-a919fb076447: entering follower state: follower="Node at 56537bcc-6cab-5d3c-b914-a919fb076447 [Follower]" leader= - testing.go:149: server-56537bcc-6cab-5d3c-b914-a919fb076447: 2024-09-03T21:34:29.085Z [INFO] server-56537bcc-6cab-5d3c-b914-a919fb076447: entering follower state: follower="Node at 56537bcc-6cab-5d3c-b914-a919fb076447 [Follower]" leader= -testLoggerAdapter verbose: server-fa3794bb-d2e6-2a90-2332-640f9b372bca: 2024-09-03T21:34:29.085Z [INFO] server-fa3794bb-d2e6-2a90-2332-640f9b372bca: entering follower state: follower="Node at fa3794bb-d2e6-2a90-2332-640f9b372bca [Follower]" leader= - testing.go:149: server-fa3794bb-d2e6-2a90-2332-640f9b372bca: 2024-09-03T21:34:29.085Z [INFO] server-fa3794bb-d2e6-2a90-2332-640f9b372bca: entering follower state: follower="Node at fa3794bb-d2e6-2a90-2332-640f9b372bca [Follower]" leader= -testLoggerAdapter verbose: server-56537bcc-6cab-5d3c-b914-a919fb076447: 2024-09-03T21:34:29.149Z [WARN] server-56537bcc-6cab-5d3c-b914-a919fb076447: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-56537bcc-6cab-5d3c-b914-a919fb076447: 2024-09-03T21:34:29.149Z [WARN] server-56537bcc-6cab-5d3c-b914-a919fb076447: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-56537bcc-6cab-5d3c-b914-a919fb076447: 2024-09-03T21:34:29.149Z [INFO] server-56537bcc-6cab-5d3c-b914-a919fb076447: entering candidate state: node="Node at 56537bcc-6cab-5d3c-b914-a919fb076447 [Candidate]" term=2 - testing.go:149: server-56537bcc-6cab-5d3c-b914-a919fb076447: 2024-09-03T21:34:29.149Z [INFO] server-56537bcc-6cab-5d3c-b914-a919fb076447: entering candidate state: node="Node at 56537bcc-6cab-5d3c-b914-a919fb076447 [Candidate]" term=2 -testLoggerAdapter verbose: server-56537bcc-6cab-5d3c-b914-a919fb076447: 2024-09-03T21:34:29.149Z [INFO] server-56537bcc-6cab-5d3c-b914-a919fb076447: election won: tally=2 - testing.go:149: server-56537bcc-6cab-5d3c-b914-a919fb076447: 2024-09-03T21:34:29.149Z [INFO] server-56537bcc-6cab-5d3c-b914-a919fb076447: election won: tally=2 -testLoggerAdapter verbose: server-56537bcc-6cab-5d3c-b914-a919fb076447: 2024-09-03T21:34:29.149Z [INFO] server-56537bcc-6cab-5d3c-b914-a919fb076447: entering leader state: leader="Node at 56537bcc-6cab-5d3c-b914-a919fb076447 [Leader]" - testing.go:149: server-56537bcc-6cab-5d3c-b914-a919fb076447: 2024-09-03T21:34:29.149Z [INFO] server-56537bcc-6cab-5d3c-b914-a919fb076447: entering leader state: leader="Node at 56537bcc-6cab-5d3c-b914-a919fb076447 [Leader]" -testLoggerAdapter verbose: server-56537bcc-6cab-5d3c-b914-a919fb076447: 2024-09-03T21:34:29.149Z [INFO] server-56537bcc-6cab-5d3c-b914-a919fb076447: added peer, starting replication: peer=server-0ff7f206-9aaa-6016-437a-3026ff55f213 - testing.go:149: server-56537bcc-6cab-5d3c-b914-a919fb076447: 2024-09-03T21:34:29.149Z [INFO] server-56537bcc-6cab-5d3c-b914-a919fb076447: added peer, starting replication: peer=server-0ff7f206-9aaa-6016-437a-3026ff55f213 -testLoggerAdapter verbose: server-56537bcc-6cab-5d3c-b914-a919fb076447: 2024-09-03T21:34:29.149Z [INFO] server-56537bcc-6cab-5d3c-b914-a919fb076447: added peer, starting replication: peer=server-fa3794bb-d2e6-2a90-2332-640f9b372bca - testing.go:149: server-56537bcc-6cab-5d3c-b914-a919fb076447: 2024-09-03T21:34:29.149Z [INFO] server-56537bcc-6cab-5d3c-b914-a919fb076447: added peer, starting replication: peer=server-fa3794bb-d2e6-2a90-2332-640f9b372bca -testLoggerAdapter verbose: server-56537bcc-6cab-5d3c-b914-a919fb076447: 2024-09-03T21:34:29.150Z [INFO] server-56537bcc-6cab-5d3c-b914-a919fb076447: pipelining replication: peer="{Voter server-fa3794bb-d2e6-2a90-2332-640f9b372bca fa3794bb-d2e6-2a90-2332-640f9b372bca}" - testing.go:149: server-56537bcc-6cab-5d3c-b914-a919fb076447: 2024-09-03T21:34:29.150Z [INFO] server-56537bcc-6cab-5d3c-b914-a919fb076447: pipelining replication: peer="{Voter server-fa3794bb-d2e6-2a90-2332-640f9b372bca fa3794bb-d2e6-2a90-2332-640f9b372bca}" -testLoggerAdapter verbose: server-56537bcc-6cab-5d3c-b914-a919fb076447: 2024-09-03T21:34:29.150Z [INFO] server-56537bcc-6cab-5d3c-b914-a919fb076447: pipelining replication: peer="{Voter server-0ff7f206-9aaa-6016-437a-3026ff55f213 0ff7f206-9aaa-6016-437a-3026ff55f213}" - testing.go:149: server-56537bcc-6cab-5d3c-b914-a919fb076447: 2024-09-03T21:34:29.150Z [INFO] server-56537bcc-6cab-5d3c-b914-a919fb076447: pipelining replication: peer="{Voter server-0ff7f206-9aaa-6016-437a-3026ff55f213 0ff7f206-9aaa-6016-437a-3026ff55f213}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:29.255Z [INFO] stable state for Leader reached at 2024-09-03 21:34:29.150242854 +0000 UTC m=+10.196899992 (1 nodes), 64.520719ms from start of poll, 65.169512ms from cluster start. Timeout at 2024-09-03 21:34:29.255342055 +0000 UTC m=+10.301999186, 105.099194ms after stability - testing.go:149: cluster: 2024-09-03T21:34:29.255Z [INFO] stable state for Leader reached at 2024-09-03 21:34:29.150242854 +0000 UTC m=+10.196899992 (1 nodes), 64.520719ms from start of poll, 65.169512ms from cluster start. Timeout at 2024-09-03 21:34:29.255342055 +0000 UTC m=+10.301999186, 105.099194ms after stability -testLoggerAdapter verbose: server-56537bcc-6cab-5d3c-b914-a919fb076447: 2024-09-03T21:34:29.255Z [INFO] server-56537bcc-6cab-5d3c-b914-a919fb076447: updating configuration: command=RemoveServer server-id=a74a6c4d-0a3d-b6e8-e3bd-8c4eb1f2fc99 server-addr= servers="[{Suffrage:Voter ID:server-0ff7f206-9aaa-6016-437a-3026ff55f213 Address:0ff7f206-9aaa-6016-437a-3026ff55f213} {Suffrage:Voter ID:server-56537bcc-6cab-5d3c-b914-a919fb076447 Address:56537bcc-6cab-5d3c-b914-a919fb076447} {Suffrage:Voter ID:server-fa3794bb-d2e6-2a90-2332-640f9b372bca Address:fa3794bb-d2e6-2a90-2332-640f9b372bca}]" - testing.go:149: server-56537bcc-6cab-5d3c-b914-a919fb076447: 2024-09-03T21:34:29.255Z [INFO] server-56537bcc-6cab-5d3c-b914-a919fb076447: updating configuration: command=RemoveServer server-id=a74a6c4d-0a3d-b6e8-e3bd-8c4eb1f2fc99 server-addr= servers="[{Suffrage:Voter ID:server-0ff7f206-9aaa-6016-437a-3026ff55f213 Address:0ff7f206-9aaa-6016-437a-3026ff55f213} {Suffrage:Voter ID:server-56537bcc-6cab-5d3c-b914-a919fb076447 Address:56537bcc-6cab-5d3c-b914-a919fb076447} {Suffrage:Voter ID:server-fa3794bb-d2e6-2a90-2332-640f9b372bca Address:fa3794bb-d2e6-2a90-2332-640f9b372bca}]" -testLoggerAdapter verbose: server-56537bcc-6cab-5d3c-b914-a919fb076447: 2024-09-03T21:34:29.255Z [INFO] server-56537bcc-6cab-5d3c-b914-a919fb076447: aborting pipeline replication: peer="{Voter server-fa3794bb-d2e6-2a90-2332-640f9b372bca fa3794bb-d2e6-2a90-2332-640f9b372bca}" - testing.go:149: server-56537bcc-6cab-5d3c-b914-a919fb076447: 2024-09-03T21:34:29.255Z [INFO] server-56537bcc-6cab-5d3c-b914-a919fb076447: aborting pipeline replication: peer="{Voter server-fa3794bb-d2e6-2a90-2332-640f9b372bca fa3794bb-d2e6-2a90-2332-640f9b372bca}" -testLoggerAdapter verbose: server-56537bcc-6cab-5d3c-b914-a919fb076447: 2024-09-03T21:34:29.255Z [INFO] server-56537bcc-6cab-5d3c-b914-a919fb076447: aborting pipeline replication: peer="{Voter server-0ff7f206-9aaa-6016-437a-3026ff55f213 0ff7f206-9aaa-6016-437a-3026ff55f213}" - testing.go:149: server-56537bcc-6cab-5d3c-b914-a919fb076447: 2024-09-03T21:34:29.255Z [INFO] server-56537bcc-6cab-5d3c-b914-a919fb076447: aborting pipeline replication: peer="{Voter server-0ff7f206-9aaa-6016-437a-3026ff55f213 0ff7f206-9aaa-6016-437a-3026ff55f213}" ---- PASS: TestRaft_RemoveUnknownPeer (0.17s) +testLoggerAdapter verbose: server-fbc18987-2172-6726-425e-580f0ca1aabf: 2023-08-02T15:12:58.047Z [INFO] server-fbc18987-2172-6726-425e-580f0ca1aabf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fbc18987-2172-6726-425e-580f0ca1aabf Address:fbc18987-2172-6726-425e-580f0ca1aabf} {Suffrage:Voter ID:server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697 Address:ba99bbaf-1217-e8ac-c80d-3d3ff83a4697} {Suffrage:Voter ID:server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b Address:9cc529c2-6e9e-abe0-0f91-27f748f76e3b}]" + testing.go:149: server-fbc18987-2172-6726-425e-580f0ca1aabf: 2023-08-02T15:12:58.047Z [INFO] server-fbc18987-2172-6726-425e-580f0ca1aabf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fbc18987-2172-6726-425e-580f0ca1aabf Address:fbc18987-2172-6726-425e-580f0ca1aabf} {Suffrage:Voter ID:server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697 Address:ba99bbaf-1217-e8ac-c80d-3d3ff83a4697} {Suffrage:Voter ID:server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b Address:9cc529c2-6e9e-abe0-0f91-27f748f76e3b}]" +testLoggerAdapter verbose: server-fbc18987-2172-6726-425e-580f0ca1aabf: 2023-08-02T15:12:58.047Z [INFO] server-fbc18987-2172-6726-425e-580f0ca1aabf: entering follower state: follower="Node at fbc18987-2172-6726-425e-580f0ca1aabf [Follower]" leader= + testing.go:149: server-fbc18987-2172-6726-425e-580f0ca1aabf: 2023-08-02T15:12:58.047Z [INFO] server-fbc18987-2172-6726-425e-580f0ca1aabf: entering follower state: follower="Node at fbc18987-2172-6726-425e-580f0ca1aabf [Follower]" leader= +testLoggerAdapter verbose: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.047Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fbc18987-2172-6726-425e-580f0ca1aabf Address:fbc18987-2172-6726-425e-580f0ca1aabf} {Suffrage:Voter ID:server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697 Address:ba99bbaf-1217-e8ac-c80d-3d3ff83a4697} {Suffrage:Voter ID:server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b Address:9cc529c2-6e9e-abe0-0f91-27f748f76e3b}]" + testing.go:149: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.047Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fbc18987-2172-6726-425e-580f0ca1aabf Address:fbc18987-2172-6726-425e-580f0ca1aabf} {Suffrage:Voter ID:server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697 Address:ba99bbaf-1217-e8ac-c80d-3d3ff83a4697} {Suffrage:Voter ID:server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b Address:9cc529c2-6e9e-abe0-0f91-27f748f76e3b}]" +testLoggerAdapter verbose: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.048Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: entering follower state: follower="Node at ba99bbaf-1217-e8ac-c80d-3d3ff83a4697 [Follower]" leader= + testing.go:149: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.048Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: entering follower state: follower="Node at ba99bbaf-1217-e8ac-c80d-3d3ff83a4697 [Follower]" leader= +testLoggerAdapter verbose: server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b: 2023-08-02T15:12:58.048Z [INFO] server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fbc18987-2172-6726-425e-580f0ca1aabf Address:fbc18987-2172-6726-425e-580f0ca1aabf} {Suffrage:Voter ID:server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697 Address:ba99bbaf-1217-e8ac-c80d-3d3ff83a4697} {Suffrage:Voter ID:server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b Address:9cc529c2-6e9e-abe0-0f91-27f748f76e3b}]" + testing.go:149: server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b: 2023-08-02T15:12:58.048Z [INFO] server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fbc18987-2172-6726-425e-580f0ca1aabf Address:fbc18987-2172-6726-425e-580f0ca1aabf} {Suffrage:Voter ID:server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697 Address:ba99bbaf-1217-e8ac-c80d-3d3ff83a4697} {Suffrage:Voter ID:server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b Address:9cc529c2-6e9e-abe0-0f91-27f748f76e3b}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:58.048Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:12:58.048Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b: 2023-08-02T15:12:58.048Z [INFO] server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b: entering follower state: follower="Node at 9cc529c2-6e9e-abe0-0f91-27f748f76e3b [Follower]" leader= + testing.go:149: server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b: 2023-08-02T15:12:58.048Z [INFO] server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b: entering follower state: follower="Node at 9cc529c2-6e9e-abe0-0f91-27f748f76e3b [Follower]" leader= +testLoggerAdapter verbose: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.116Z [WARN] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.116Z [WARN] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.116Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: entering candidate state: node="Node at ba99bbaf-1217-e8ac-c80d-3d3ff83a4697 [Candidate]" term=2 + testing.go:149: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.116Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: entering candidate state: node="Node at ba99bbaf-1217-e8ac-c80d-3d3ff83a4697 [Candidate]" term=2 +testLoggerAdapter verbose: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.116Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: election won: tally=2 + testing.go:149: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.116Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: election won: tally=2 +testLoggerAdapter verbose: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.117Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: entering leader state: leader="Node at ba99bbaf-1217-e8ac-c80d-3d3ff83a4697 [Leader]" + testing.go:149: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.117Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: entering leader state: leader="Node at ba99bbaf-1217-e8ac-c80d-3d3ff83a4697 [Leader]" +testLoggerAdapter verbose: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.117Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: added peer, starting replication: peer=server-fbc18987-2172-6726-425e-580f0ca1aabf + testing.go:149: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.117Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: added peer, starting replication: peer=server-fbc18987-2172-6726-425e-580f0ca1aabf +testLoggerAdapter verbose: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.117Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: added peer, starting replication: peer=server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b + testing.go:149: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.117Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: added peer, starting replication: peer=server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b +testLoggerAdapter verbose: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.117Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: pipelining replication: peer="{Voter server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b 9cc529c2-6e9e-abe0-0f91-27f748f76e3b}" + testing.go:149: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.117Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: pipelining replication: peer="{Voter server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b 9cc529c2-6e9e-abe0-0f91-27f748f76e3b}" +testLoggerAdapter verbose: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.117Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: pipelining replication: peer="{Voter server-fbc18987-2172-6726-425e-580f0ca1aabf fbc18987-2172-6726-425e-580f0ca1aabf}" +testLoggerAdapter verbose: server-fbc18987-2172-6726-425e-580f0ca1aabf: 2023-08-02T15:12:58.117Z [WARN] server-fbc18987-2172-6726-425e-580f0ca1aabf: rejecting vote request since our last term is greater: candidate=ba99bbaf-1217-e8ac-c80d-3d3ff83a4697 last-term=2 last-candidate-term=1 + testing.go:149: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.117Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: pipelining replication: peer="{Voter server-fbc18987-2172-6726-425e-580f0ca1aabf fbc18987-2172-6726-425e-580f0ca1aabf}" + testing.go:149: server-fbc18987-2172-6726-425e-580f0ca1aabf: 2023-08-02T15:12:58.117Z [WARN] server-fbc18987-2172-6726-425e-580f0ca1aabf: rejecting vote request since our last term is greater: candidate=ba99bbaf-1217-e8ac-c80d-3d3ff83a4697 last-term=2 last-candidate-term=1 +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:58.222Z [INFO] stable state for Leader reached at 2023-08-02 15:12:58.117392465 +0000 UTC m=+10.213328802 (1 nodes), 69.096338ms from start of poll, 69.875016ms from cluster start. Timeout at 2023-08-02 15:12:58.222523436 +0000 UTC m=+10.318459770, 105.130968ms after stability + testing.go:149: cluster: 2023-08-02T15:12:58.222Z [INFO] stable state for Leader reached at 2023-08-02 15:12:58.117392465 +0000 UTC m=+10.213328802 (1 nodes), 69.096338ms from start of poll, 69.875016ms from cluster start. Timeout at 2023-08-02 15:12:58.222523436 +0000 UTC m=+10.318459770, 105.130968ms after stability +testLoggerAdapter verbose: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.222Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: updating configuration: command=RemoveServer server-id=9d9e9ce2-d4a5-fc88-1aaf-97f96cb011f2 server-addr= servers="[{Suffrage:Voter ID:server-fbc18987-2172-6726-425e-580f0ca1aabf Address:fbc18987-2172-6726-425e-580f0ca1aabf} {Suffrage:Voter ID:server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697 Address:ba99bbaf-1217-e8ac-c80d-3d3ff83a4697} {Suffrage:Voter ID:server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b Address:9cc529c2-6e9e-abe0-0f91-27f748f76e3b}]" + testing.go:149: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.222Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: updating configuration: command=RemoveServer server-id=9d9e9ce2-d4a5-fc88-1aaf-97f96cb011f2 server-addr= servers="[{Suffrage:Voter ID:server-fbc18987-2172-6726-425e-580f0ca1aabf Address:fbc18987-2172-6726-425e-580f0ca1aabf} {Suffrage:Voter ID:server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697 Address:ba99bbaf-1217-e8ac-c80d-3d3ff83a4697} {Suffrage:Voter ID:server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b Address:9cc529c2-6e9e-abe0-0f91-27f748f76e3b}]" +testLoggerAdapter verbose: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.223Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: aborting pipeline replication: peer="{Voter server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b 9cc529c2-6e9e-abe0-0f91-27f748f76e3b}" + testing.go:149: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.223Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: aborting pipeline replication: peer="{Voter server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b 9cc529c2-6e9e-abe0-0f91-27f748f76e3b}" +testLoggerAdapter verbose: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.223Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: aborting pipeline replication: peer="{Voter server-fbc18987-2172-6726-425e-580f0ca1aabf fbc18987-2172-6726-425e-580f0ca1aabf}" + testing.go:149: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.223Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: aborting pipeline replication: peer="{Voter server-fbc18987-2172-6726-425e-580f0ca1aabf fbc18987-2172-6726-425e-580f0ca1aabf}" +--- PASS: TestRaft_RemoveUnknownPeer (0.18s) === RUN TestRaft_SnapshotRestore -testLoggerAdapter verbose: server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: 2024-09-03T21:34:29.256Z [INFO] server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ac25132e-a4dd-b235-dabb-01a3704b1fb2 Address:ac25132e-a4dd-b235-dabb-01a3704b1fb2}]" - testing.go:149: server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: 2024-09-03T21:34:29.256Z [INFO] server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ac25132e-a4dd-b235-dabb-01a3704b1fb2 Address:ac25132e-a4dd-b235-dabb-01a3704b1fb2}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:29.256Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:29.256Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: 2024-09-03T21:34:29.256Z [INFO] server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: entering follower state: follower="Node at ac25132e-a4dd-b235-dabb-01a3704b1fb2 [Follower]" leader= - testing.go:149: server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: 2024-09-03T21:34:29.256Z [INFO] server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: entering follower state: follower="Node at ac25132e-a4dd-b235-dabb-01a3704b1fb2 [Follower]" leader= -testLoggerAdapter verbose: server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: 2024-09-03T21:34:29.330Z [WARN] server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: 2024-09-03T21:34:29.330Z [WARN] server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: 2024-09-03T21:34:29.330Z [INFO] server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: entering candidate state: node="Node at ac25132e-a4dd-b235-dabb-01a3704b1fb2 [Candidate]" term=2 - testing.go:149: server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: 2024-09-03T21:34:29.330Z [INFO] server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: entering candidate state: node="Node at ac25132e-a4dd-b235-dabb-01a3704b1fb2 [Candidate]" term=2 -testLoggerAdapter verbose: server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: 2024-09-03T21:34:29.331Z [INFO] server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: election won: tally=1 - testing.go:149: server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: 2024-09-03T21:34:29.331Z [INFO] server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: election won: tally=1 -testLoggerAdapter verbose: server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: 2024-09-03T21:34:29.331Z [INFO] server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: entering leader state: leader="Node at ac25132e-a4dd-b235-dabb-01a3704b1fb2 [Leader]" - testing.go:149: server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: 2024-09-03T21:34:29.331Z [INFO] server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: entering leader state: leader="Node at ac25132e-a4dd-b235-dabb-01a3704b1fb2 [Leader]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:29.436Z [INFO] stable state for Leader reached at 2024-09-03 21:34:29.33112311 +0000 UTC m=+10.377780241 (1 nodes), 74.605765ms from start of poll, 74.79283ms from cluster start. Timeout at 2024-09-03 21:34:29.436216831 +0000 UTC m=+10.482873963, 105.093722ms after stability - testing.go:149: cluster: 2024-09-03T21:34:29.436Z [INFO] stable state for Leader reached at 2024-09-03 21:34:29.33112311 +0000 UTC m=+10.377780241 (1 nodes), 74.605765ms from start of poll, 74.79283ms from cluster start. Timeout at 2024-09-03 21:34:29.436216831 +0000 UTC m=+10.482873963, 105.093722ms after stability -testLoggerAdapter verbose: server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: 2024-09-03T21:34:29.437Z [INFO] server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: starting snapshot up to: index=102 - testing.go:149: server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: 2024-09-03T21:34:29.437Z [INFO] server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: starting snapshot up to: index=102 - testing.go:153: 2024-09-03T21:34:29.437Z [INFO] creating new snapshot: path=/tmp/raft488787264/snapshots/2-102-1725399269437.tmp -testLoggerAdapter verbose: server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: 2024-09-03T21:34:29.437Z [INFO] server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: compacting logs: from=1 to=92 - testing.go:149: server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: 2024-09-03T21:34:29.437Z [INFO] server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: compacting logs: from=1 to=92 -testLoggerAdapter verbose: server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: 2024-09-03T21:34:29.437Z [INFO] server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: snapshot complete up to: index=102 - testing.go:149: server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: 2024-09-03T21:34:29.437Z [INFO] server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: snapshot complete up to: index=102 -testLoggerAdapter verbose: server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: 2024-09-03T21:34:29.437Z [INFO] server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: restored from snapshot: id=2-102-1725399269437 - testing.go:149: server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: 2024-09-03T21:34:29.437Z [INFO] server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: restored from snapshot: id=2-102-1725399269437 -testLoggerAdapter verbose: server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: 2024-09-03T21:34:29.437Z [INFO] server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ac25132e-a4dd-b235-dabb-01a3704b1fb2 Address:ac25132e-a4dd-b235-dabb-01a3704b1fb2}]" - testing.go:149: server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: 2024-09-03T21:34:29.437Z [INFO] server-ac25132e-a4dd-b235-dabb-01a3704b1fb2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ac25132e-a4dd-b235-dabb-01a3704b1fb2 Address:ac25132e-a4dd-b235-dabb-01a3704b1fb2}]" +testLoggerAdapter verbose: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.223Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e Address:b6cdb39d-6a85-bda7-38a7-abcb262e1d8e}]" + testing.go:149: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.223Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e Address:b6cdb39d-6a85-bda7-38a7-abcb262e1d8e}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:58.223Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:12:58.223Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.223Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: entering follower state: follower="Node at b6cdb39d-6a85-bda7-38a7-abcb262e1d8e [Follower]" leader= + testing.go:149: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.223Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: entering follower state: follower="Node at b6cdb39d-6a85-bda7-38a7-abcb262e1d8e [Follower]" leader= +testLoggerAdapter verbose: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.290Z [WARN] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.290Z [WARN] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.290Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: entering candidate state: node="Node at b6cdb39d-6a85-bda7-38a7-abcb262e1d8e [Candidate]" term=2 + testing.go:149: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.290Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: entering candidate state: node="Node at b6cdb39d-6a85-bda7-38a7-abcb262e1d8e [Candidate]" term=2 +testLoggerAdapter verbose: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.290Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: election won: tally=1 + testing.go:149: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.290Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: election won: tally=1 +testLoggerAdapter verbose: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.290Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: entering leader state: leader="Node at b6cdb39d-6a85-bda7-38a7-abcb262e1d8e [Leader]" + testing.go:149: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.290Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: entering leader state: leader="Node at b6cdb39d-6a85-bda7-38a7-abcb262e1d8e [Leader]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:58.395Z [INFO] stable state for Leader reached at 2023-08-02 15:12:58.290594284 +0000 UTC m=+10.386530617 (1 nodes), 66.688321ms from start of poll, 66.932408ms from cluster start. Timeout at 2023-08-02 15:12:58.395850695 +0000 UTC m=+10.491787030, 105.256413ms after stability + testing.go:149: cluster: 2023-08-02T15:12:58.395Z [INFO] stable state for Leader reached at 2023-08-02 15:12:58.290594284 +0000 UTC m=+10.386530617 (1 nodes), 66.688321ms from start of poll, 66.932408ms from cluster start. Timeout at 2023-08-02 15:12:58.395850695 +0000 UTC m=+10.491787030, 105.256413ms after stability +testLoggerAdapter verbose: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.397Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: starting snapshot up to: index=102 + testing.go:149: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.397Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: starting snapshot up to: index=102 + testing.go:153: 2023-08-02T15:12:58.397Z [INFO] creating new snapshot: path=/tmp/raft356815295/snapshots/2-102-1690989178397.tmp +testLoggerAdapter verbose: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.400Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: compacting logs: from=1 to=92 + testing.go:149: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.400Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: compacting logs: from=1 to=92 +testLoggerAdapter verbose: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.400Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: snapshot complete up to: index=102 + testing.go:149: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.400Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: snapshot complete up to: index=102 +testLoggerAdapter verbose: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.400Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: restored from snapshot: id=2-102-1690989178397 + testing.go:149: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.400Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: restored from snapshot: id=2-102-1690989178397 +testLoggerAdapter verbose: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.400Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e Address:b6cdb39d-6a85-bda7-38a7-abcb262e1d8e}]" + testing.go:149: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.400Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e Address:b6cdb39d-6a85-bda7-38a7-abcb262e1d8e}]" --- PASS: TestRaft_SnapshotRestore (0.18s) === RUN TestRaft_NoRestoreOnStart -testLoggerAdapter verbose: server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: 2024-09-03T21:34:29.438Z [INFO] server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02 Address:d13f29b5-bed5-aff3-f2a6-ce76ef495c02}]" - testing.go:149: server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: 2024-09-03T21:34:29.438Z [INFO] server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02 Address:d13f29b5-bed5-aff3-f2a6-ce76ef495c02}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:29.438Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:29.438Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: 2024-09-03T21:34:29.438Z [INFO] server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: entering follower state: follower="Node at d13f29b5-bed5-aff3-f2a6-ce76ef495c02 [Follower]" leader= - testing.go:149: server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: 2024-09-03T21:34:29.438Z [INFO] server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: entering follower state: follower="Node at d13f29b5-bed5-aff3-f2a6-ce76ef495c02 [Follower]" leader= -testLoggerAdapter verbose: server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: 2024-09-03T21:34:29.528Z [WARN] server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: 2024-09-03T21:34:29.528Z [WARN] server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: 2024-09-03T21:34:29.528Z [INFO] server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: entering candidate state: node="Node at d13f29b5-bed5-aff3-f2a6-ce76ef495c02 [Candidate]" term=2 - testing.go:149: server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: 2024-09-03T21:34:29.528Z [INFO] server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: entering candidate state: node="Node at d13f29b5-bed5-aff3-f2a6-ce76ef495c02 [Candidate]" term=2 -testLoggerAdapter verbose: server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: 2024-09-03T21:34:29.528Z [INFO] server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: election won: tally=1 - testing.go:149: server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: 2024-09-03T21:34:29.528Z [INFO] server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: election won: tally=1 -testLoggerAdapter verbose: server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: 2024-09-03T21:34:29.528Z [INFO] server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: entering leader state: leader="Node at d13f29b5-bed5-aff3-f2a6-ce76ef495c02 [Leader]" - testing.go:149: server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: 2024-09-03T21:34:29.528Z [INFO] server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: entering leader state: leader="Node at d13f29b5-bed5-aff3-f2a6-ce76ef495c02 [Leader]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:29.635Z [INFO] stable state for Leader reached at 2024-09-03 21:34:29.528572609 +0000 UTC m=+10.575229739 (1 nodes), 90.247225ms from start of poll, 90.381793ms from cluster start. Timeout at 2024-09-03 21:34:29.635139005 +0000 UTC m=+10.681796203, 106.566464ms after stability - testing.go:149: cluster: 2024-09-03T21:34:29.635Z [INFO] stable state for Leader reached at 2024-09-03 21:34:29.528572609 +0000 UTC m=+10.575229739 (1 nodes), 90.247225ms from start of poll, 90.381793ms from cluster start. Timeout at 2024-09-03 21:34:29.635139005 +0000 UTC m=+10.681796203, 106.566464ms after stability -testLoggerAdapter verbose: server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: 2024-09-03T21:34:29.636Z [INFO] server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: starting snapshot up to: index=102 - testing.go:149: server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: 2024-09-03T21:34:29.636Z [INFO] server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: starting snapshot up to: index=102 - testing.go:153: 2024-09-03T21:34:29.636Z [INFO] creating new snapshot: path=/tmp/raft494304882/snapshots/2-102-1725399269636.tmp -testLoggerAdapter verbose: server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: 2024-09-03T21:34:29.636Z [INFO] server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: compacting logs: from=1 to=92 - testing.go:149: server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: 2024-09-03T21:34:29.636Z [INFO] server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: compacting logs: from=1 to=92 -testLoggerAdapter verbose: server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: 2024-09-03T21:34:29.636Z [INFO] server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: snapshot complete up to: index=102 - testing.go:149: server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: 2024-09-03T21:34:29.636Z [INFO] server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: snapshot complete up to: index=102 -testLoggerAdapter verbose: server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: 2024-09-03T21:34:29.636Z [INFO] server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02 Address:d13f29b5-bed5-aff3-f2a6-ce76ef495c02}]" - testing.go:149: server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: 2024-09-03T21:34:29.636Z [INFO] server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02 Address:d13f29b5-bed5-aff3-f2a6-ce76ef495c02}]" ---- PASS: TestRaft_NoRestoreOnStart (0.20s) +testLoggerAdapter verbose: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.402Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424 Address:b4f9dcb1-6c91-d98c-2202-bf5a3a26d424}]" + testing.go:149: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.402Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424 Address:b4f9dcb1-6c91-d98c-2202-bf5a3a26d424}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:58.402Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:12:58.402Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.402Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: entering follower state: follower="Node at b4f9dcb1-6c91-d98c-2202-bf5a3a26d424 [Follower]" leader= + testing.go:149: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.402Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: entering follower state: follower="Node at b4f9dcb1-6c91-d98c-2202-bf5a3a26d424 [Follower]" leader= +testLoggerAdapter verbose: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.476Z [WARN] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.476Z [WARN] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.476Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: entering candidate state: node="Node at b4f9dcb1-6c91-d98c-2202-bf5a3a26d424 [Candidate]" term=2 + testing.go:149: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.476Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: entering candidate state: node="Node at b4f9dcb1-6c91-d98c-2202-bf5a3a26d424 [Candidate]" term=2 +testLoggerAdapter verbose: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.476Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: election won: tally=1 + testing.go:149: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.476Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: election won: tally=1 +testLoggerAdapter verbose: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.476Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: entering leader state: leader="Node at b4f9dcb1-6c91-d98c-2202-bf5a3a26d424 [Leader]" + testing.go:149: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.476Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: entering leader state: leader="Node at b4f9dcb1-6c91-d98c-2202-bf5a3a26d424 [Leader]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:58.582Z [INFO] stable state for Leader reached at 2023-08-02 15:12:58.476486081 +0000 UTC m=+10.572422420 (1 nodes), 74.199832ms from start of poll, 74.73178ms from cluster start. Timeout at 2023-08-02 15:12:58.582165972 +0000 UTC m=+10.678102323, 105.679903ms after stability + testing.go:149: cluster: 2023-08-02T15:12:58.582Z [INFO] stable state for Leader reached at 2023-08-02 15:12:58.476486081 +0000 UTC m=+10.572422420 (1 nodes), 74.199832ms from start of poll, 74.73178ms from cluster start. Timeout at 2023-08-02 15:12:58.582165972 +0000 UTC m=+10.678102323, 105.679903ms after stability +testLoggerAdapter verbose: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.583Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: starting snapshot up to: index=102 + testing.go:149: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.583Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: starting snapshot up to: index=102 + testing.go:153: 2023-08-02T15:12:58.584Z [INFO] creating new snapshot: path=/tmp/raft012387401/snapshots/2-102-1690989178583.tmp +testLoggerAdapter verbose: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.584Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: compacting logs: from=1 to=92 + testing.go:149: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.584Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: compacting logs: from=1 to=92 +testLoggerAdapter verbose: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.584Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: snapshot complete up to: index=102 + testing.go:149: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.584Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: snapshot complete up to: index=102 +testLoggerAdapter verbose: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.584Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424 Address:b4f9dcb1-6c91-d98c-2202-bf5a3a26d424}]" + testing.go:149: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.584Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424 Address:b4f9dcb1-6c91-d98c-2202-bf5a3a26d424}]" +--- PASS: TestRaft_NoRestoreOnStart (0.18s) === RUN TestRaft_SnapshotRestore_PeerChange raft_test.go:1026: Disabled unreliable test --- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s) @@ -2195,1704 +2232,1760 @@ raft_test.go:1132: Disabled unreliable test --- SKIP: TestRaft_AutoSnapshot (0.00s) === RUN TestRaft_UserSnapshot -testLoggerAdapter verbose: server-10b4d774-721d-5175-87bc-e8078f8fad48: 2024-09-03T21:34:29.637Z [INFO] server-10b4d774-721d-5175-87bc-e8078f8fad48: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-10b4d774-721d-5175-87bc-e8078f8fad48 Address:10b4d774-721d-5175-87bc-e8078f8fad48}]" - testing.go:149: server-10b4d774-721d-5175-87bc-e8078f8fad48: 2024-09-03T21:34:29.637Z [INFO] server-10b4d774-721d-5175-87bc-e8078f8fad48: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-10b4d774-721d-5175-87bc-e8078f8fad48 Address:10b4d774-721d-5175-87bc-e8078f8fad48}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:29.637Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:29.637Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: 2024-09-03T21:34:29.637Z [INFO] server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: entering follower state: follower="Node at d13f29b5-bed5-aff3-f2a6-ce76ef495c02 [Follower]" leader= -testLoggerAdapter verbose: server-10b4d774-721d-5175-87bc-e8078f8fad48: 2024-09-03T21:34:29.637Z [INFO] server-10b4d774-721d-5175-87bc-e8078f8fad48: entering follower state: follower="Node at 10b4d774-721d-5175-87bc-e8078f8fad48 [Follower]" leader= - testing.go:149: server-10b4d774-721d-5175-87bc-e8078f8fad48: 2024-09-03T21:34:29.637Z [INFO] server-10b4d774-721d-5175-87bc-e8078f8fad48: entering follower state: follower="Node at 10b4d774-721d-5175-87bc-e8078f8fad48 [Follower]" leader= -testLoggerAdapter verbose: server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: 2024-09-03T21:34:29.709Z [WARN] server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: 2024-09-03T21:34:29.710Z [INFO] server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: entering candidate state: node="Node at d13f29b5-bed5-aff3-f2a6-ce76ef495c02 [Candidate]" term=3 -testLoggerAdapter verbose: server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: 2024-09-03T21:34:29.710Z [INFO] server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: election won: tally=1 -testLoggerAdapter verbose: server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: 2024-09-03T21:34:29.710Z [INFO] server-d13f29b5-bed5-aff3-f2a6-ce76ef495c02: entering leader state: leader="Node at d13f29b5-bed5-aff3-f2a6-ce76ef495c02 [Leader]" -testLoggerAdapter verbose: server-10b4d774-721d-5175-87bc-e8078f8fad48: 2024-09-03T21:34:29.716Z [WARN] server-10b4d774-721d-5175-87bc-e8078f8fad48: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-10b4d774-721d-5175-87bc-e8078f8fad48: 2024-09-03T21:34:29.716Z [WARN] server-10b4d774-721d-5175-87bc-e8078f8fad48: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-10b4d774-721d-5175-87bc-e8078f8fad48: 2024-09-03T21:34:29.716Z [INFO] server-10b4d774-721d-5175-87bc-e8078f8fad48: entering candidate state: node="Node at 10b4d774-721d-5175-87bc-e8078f8fad48 [Candidate]" term=2 - testing.go:149: server-10b4d774-721d-5175-87bc-e8078f8fad48: 2024-09-03T21:34:29.716Z [INFO] server-10b4d774-721d-5175-87bc-e8078f8fad48: entering candidate state: node="Node at 10b4d774-721d-5175-87bc-e8078f8fad48 [Candidate]" term=2 -testLoggerAdapter verbose: server-10b4d774-721d-5175-87bc-e8078f8fad48: 2024-09-03T21:34:29.716Z [INFO] server-10b4d774-721d-5175-87bc-e8078f8fad48: election won: tally=1 - testing.go:149: server-10b4d774-721d-5175-87bc-e8078f8fad48: 2024-09-03T21:34:29.716Z [INFO] server-10b4d774-721d-5175-87bc-e8078f8fad48: election won: tally=1 -testLoggerAdapter verbose: server-10b4d774-721d-5175-87bc-e8078f8fad48: 2024-09-03T21:34:29.716Z [INFO] server-10b4d774-721d-5175-87bc-e8078f8fad48: entering leader state: leader="Node at 10b4d774-721d-5175-87bc-e8078f8fad48 [Leader]" - testing.go:149: server-10b4d774-721d-5175-87bc-e8078f8fad48: 2024-09-03T21:34:29.716Z [INFO] server-10b4d774-721d-5175-87bc-e8078f8fad48: entering leader state: leader="Node at 10b4d774-721d-5175-87bc-e8078f8fad48 [Leader]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:29.822Z [INFO] stable state for Leader reached at 2024-09-03 21:34:29.716902459 +0000 UTC m=+10.763559590 (1 nodes), 79.790204ms from start of poll, 79.947852ms from cluster start. Timeout at 2024-09-03 21:34:29.821999392 +0000 UTC m=+10.868656524, 105.096934ms after stability - testing.go:149: cluster: 2024-09-03T21:34:29.822Z [INFO] stable state for Leader reached at 2024-09-03 21:34:29.716902459 +0000 UTC m=+10.763559590 (1 nodes), 79.790204ms from start of poll, 79.947852ms from cluster start. Timeout at 2024-09-03 21:34:29.821999392 +0000 UTC m=+10.868656524, 105.096934ms after stability -testLoggerAdapter verbose: server-10b4d774-721d-5175-87bc-e8078f8fad48: 2024-09-03T21:34:29.822Z [ERROR] server-10b4d774-721d-5175-87bc-e8078f8fad48: failed to take snapshot: error="nothing new to snapshot" - testing.go:149: server-10b4d774-721d-5175-87bc-e8078f8fad48: 2024-09-03T21:34:29.822Z [ERROR] server-10b4d774-721d-5175-87bc-e8078f8fad48: failed to take snapshot: error="nothing new to snapshot" -testLoggerAdapter verbose: server-10b4d774-721d-5175-87bc-e8078f8fad48: 2024-09-03T21:34:29.822Z [INFO] server-10b4d774-721d-5175-87bc-e8078f8fad48: starting snapshot up to: index=12 - testing.go:149: server-10b4d774-721d-5175-87bc-e8078f8fad48: 2024-09-03T21:34:29.822Z [INFO] server-10b4d774-721d-5175-87bc-e8078f8fad48: starting snapshot up to: index=12 - testing.go:153: 2024-09-03T21:34:29.822Z [INFO] creating new snapshot: path=/tmp/raft024136308/snapshots/2-12-1725399269822.tmp -testLoggerAdapter verbose: server-10b4d774-721d-5175-87bc-e8078f8fad48: 2024-09-03T21:34:29.822Z [INFO] server-10b4d774-721d-5175-87bc-e8078f8fad48: compacting logs: from=1 to=2 - testing.go:149: server-10b4d774-721d-5175-87bc-e8078f8fad48: 2024-09-03T21:34:29.822Z [INFO] server-10b4d774-721d-5175-87bc-e8078f8fad48: compacting logs: from=1 to=2 -testLoggerAdapter verbose: server-10b4d774-721d-5175-87bc-e8078f8fad48: 2024-09-03T21:34:29.822Z [INFO] server-10b4d774-721d-5175-87bc-e8078f8fad48: snapshot complete up to: index=12 - testing.go:149: server-10b4d774-721d-5175-87bc-e8078f8fad48: 2024-09-03T21:34:29.822Z [INFO] server-10b4d774-721d-5175-87bc-e8078f8fad48: snapshot complete up to: index=12 +testLoggerAdapter verbose: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.585Z [INFO] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-902566b9-8890-c51f-12ae-ef78a06fa7e3 Address:902566b9-8890-c51f-12ae-ef78a06fa7e3}]" + testing.go:149: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.585Z [INFO] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-902566b9-8890-c51f-12ae-ef78a06fa7e3 Address:902566b9-8890-c51f-12ae-ef78a06fa7e3}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:58.585Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:12:58.585Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.585Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: entering follower state: follower="Node at b4f9dcb1-6c91-d98c-2202-bf5a3a26d424 [Follower]" leader= +testLoggerAdapter verbose: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.585Z [INFO] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: entering follower state: follower="Node at 902566b9-8890-c51f-12ae-ef78a06fa7e3 [Follower]" leader= + testing.go:149: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.585Z [INFO] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: entering follower state: follower="Node at 902566b9-8890-c51f-12ae-ef78a06fa7e3 [Follower]" leader= +testLoggerAdapter verbose: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.667Z [WARN] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.667Z [WARN] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.668Z [INFO] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: entering candidate state: node="Node at 902566b9-8890-c51f-12ae-ef78a06fa7e3 [Candidate]" term=2 + testing.go:149: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.668Z [INFO] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: entering candidate state: node="Node at 902566b9-8890-c51f-12ae-ef78a06fa7e3 [Candidate]" term=2 +testLoggerAdapter verbose: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.668Z [INFO] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: election won: tally=1 + testing.go:149: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.668Z [INFO] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: election won: tally=1 +testLoggerAdapter verbose: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.668Z [INFO] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: entering leader state: leader="Node at 902566b9-8890-c51f-12ae-ef78a06fa7e3 [Leader]" + testing.go:149: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.668Z [INFO] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: entering leader state: leader="Node at 902566b9-8890-c51f-12ae-ef78a06fa7e3 [Leader]" +testLoggerAdapter verbose: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.679Z [WARN] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.679Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: entering candidate state: node="Node at b4f9dcb1-6c91-d98c-2202-bf5a3a26d424 [Candidate]" term=3 +testLoggerAdapter verbose: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.679Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: election won: tally=1 +testLoggerAdapter verbose: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.679Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: entering leader state: leader="Node at b4f9dcb1-6c91-d98c-2202-bf5a3a26d424 [Leader]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:58.773Z [INFO] stable state for Leader reached at 2023-08-02 15:12:58.668285644 +0000 UTC m=+10.764222010 (1 nodes), 83.123386ms from start of poll, 83.288678ms from cluster start. Timeout at 2023-08-02 15:12:58.77373504 +0000 UTC m=+10.869671410, 105.4494ms after stability + testing.go:149: cluster: 2023-08-02T15:12:58.773Z [INFO] stable state for Leader reached at 2023-08-02 15:12:58.668285644 +0000 UTC m=+10.764222010 (1 nodes), 83.123386ms from start of poll, 83.288678ms from cluster start. Timeout at 2023-08-02 15:12:58.77373504 +0000 UTC m=+10.869671410, 105.4494ms after stability +testLoggerAdapter verbose: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.773Z [ERROR] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: failed to take snapshot: error="nothing new to snapshot" + testing.go:149: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.773Z [ERROR] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: failed to take snapshot: error="nothing new to snapshot" +testLoggerAdapter verbose: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.774Z [INFO] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: starting snapshot up to: index=12 + testing.go:149: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.774Z [INFO] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: starting snapshot up to: index=12 + testing.go:153: 2023-08-02T15:12:58.774Z [INFO] creating new snapshot: path=/tmp/raft815413603/snapshots/2-12-1690989178774.tmp +testLoggerAdapter verbose: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.774Z [INFO] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: compacting logs: from=1 to=2 + testing.go:149: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.774Z [INFO] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: compacting logs: from=1 to=2 +testLoggerAdapter verbose: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.774Z [INFO] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: snapshot complete up to: index=12 + testing.go:149: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.774Z [INFO] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: snapshot complete up to: index=12 --- PASS: TestRaft_UserSnapshot (0.19s) === RUN TestRaft_UserRestore -testLoggerAdapter verbose: server-b6c3026e-f376-206a-28b9-143e83a7515f: 2024-09-03T21:34:29.823Z [INFO] server-b6c3026e-f376-206a-28b9-143e83a7515f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6c3026e-f376-206a-28b9-143e83a7515f Address:b6c3026e-f376-206a-28b9-143e83a7515f} {Suffrage:Voter ID:server-b254b9e4-80f0-d1b8-3845-a52230688881 Address:b254b9e4-80f0-d1b8-3845-a52230688881} {Suffrage:Voter ID:server-4aaed739-7e6f-1846-02f8-95a420a09f91 Address:4aaed739-7e6f-1846-02f8-95a420a09f91}]" - testing.go:149: server-b6c3026e-f376-206a-28b9-143e83a7515f: 2024-09-03T21:34:29.823Z [INFO] server-b6c3026e-f376-206a-28b9-143e83a7515f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6c3026e-f376-206a-28b9-143e83a7515f Address:b6c3026e-f376-206a-28b9-143e83a7515f} {Suffrage:Voter ID:server-b254b9e4-80f0-d1b8-3845-a52230688881 Address:b254b9e4-80f0-d1b8-3845-a52230688881} {Suffrage:Voter ID:server-4aaed739-7e6f-1846-02f8-95a420a09f91 Address:4aaed739-7e6f-1846-02f8-95a420a09f91}]" -testLoggerAdapter verbose: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.823Z [INFO] server-b254b9e4-80f0-d1b8-3845-a52230688881: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6c3026e-f376-206a-28b9-143e83a7515f Address:b6c3026e-f376-206a-28b9-143e83a7515f} {Suffrage:Voter ID:server-b254b9e4-80f0-d1b8-3845-a52230688881 Address:b254b9e4-80f0-d1b8-3845-a52230688881} {Suffrage:Voter ID:server-4aaed739-7e6f-1846-02f8-95a420a09f91 Address:4aaed739-7e6f-1846-02f8-95a420a09f91}]" - testing.go:149: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.823Z [INFO] server-b254b9e4-80f0-d1b8-3845-a52230688881: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6c3026e-f376-206a-28b9-143e83a7515f Address:b6c3026e-f376-206a-28b9-143e83a7515f} {Suffrage:Voter ID:server-b254b9e4-80f0-d1b8-3845-a52230688881 Address:b254b9e4-80f0-d1b8-3845-a52230688881} {Suffrage:Voter ID:server-4aaed739-7e6f-1846-02f8-95a420a09f91 Address:4aaed739-7e6f-1846-02f8-95a420a09f91}]" -testLoggerAdapter verbose: server-4aaed739-7e6f-1846-02f8-95a420a09f91: 2024-09-03T21:34:29.823Z [INFO] server-4aaed739-7e6f-1846-02f8-95a420a09f91: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6c3026e-f376-206a-28b9-143e83a7515f Address:b6c3026e-f376-206a-28b9-143e83a7515f} {Suffrage:Voter ID:server-b254b9e4-80f0-d1b8-3845-a52230688881 Address:b254b9e4-80f0-d1b8-3845-a52230688881} {Suffrage:Voter ID:server-4aaed739-7e6f-1846-02f8-95a420a09f91 Address:4aaed739-7e6f-1846-02f8-95a420a09f91}]" - testing.go:149: server-4aaed739-7e6f-1846-02f8-95a420a09f91: 2024-09-03T21:34:29.823Z [INFO] server-4aaed739-7e6f-1846-02f8-95a420a09f91: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6c3026e-f376-206a-28b9-143e83a7515f Address:b6c3026e-f376-206a-28b9-143e83a7515f} {Suffrage:Voter ID:server-b254b9e4-80f0-d1b8-3845-a52230688881 Address:b254b9e4-80f0-d1b8-3845-a52230688881} {Suffrage:Voter ID:server-4aaed739-7e6f-1846-02f8-95a420a09f91 Address:4aaed739-7e6f-1846-02f8-95a420a09f91}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:29.823Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:29.823Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-b6c3026e-f376-206a-28b9-143e83a7515f: 2024-09-03T21:34:29.823Z [INFO] server-b6c3026e-f376-206a-28b9-143e83a7515f: entering follower state: follower="Node at b6c3026e-f376-206a-28b9-143e83a7515f [Follower]" leader= - testing.go:149: server-b6c3026e-f376-206a-28b9-143e83a7515f: 2024-09-03T21:34:29.823Z [INFO] server-b6c3026e-f376-206a-28b9-143e83a7515f: entering follower state: follower="Node at b6c3026e-f376-206a-28b9-143e83a7515f [Follower]" leader= -testLoggerAdapter verbose: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.823Z [INFO] server-b254b9e4-80f0-d1b8-3845-a52230688881: entering follower state: follower="Node at b254b9e4-80f0-d1b8-3845-a52230688881 [Follower]" leader= - testing.go:149: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.823Z [INFO] server-b254b9e4-80f0-d1b8-3845-a52230688881: entering follower state: follower="Node at b254b9e4-80f0-d1b8-3845-a52230688881 [Follower]" leader= -testLoggerAdapter verbose: server-4aaed739-7e6f-1846-02f8-95a420a09f91: 2024-09-03T21:34:29.823Z [INFO] server-4aaed739-7e6f-1846-02f8-95a420a09f91: entering follower state: follower="Node at 4aaed739-7e6f-1846-02f8-95a420a09f91 [Follower]" leader= - testing.go:149: server-4aaed739-7e6f-1846-02f8-95a420a09f91: 2024-09-03T21:34:29.823Z [INFO] server-4aaed739-7e6f-1846-02f8-95a420a09f91: entering follower state: follower="Node at 4aaed739-7e6f-1846-02f8-95a420a09f91 [Follower]" leader= -testLoggerAdapter verbose: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.883Z [WARN] server-b254b9e4-80f0-d1b8-3845-a52230688881: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.883Z [WARN] server-b254b9e4-80f0-d1b8-3845-a52230688881: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.883Z [INFO] server-b254b9e4-80f0-d1b8-3845-a52230688881: entering candidate state: node="Node at b254b9e4-80f0-d1b8-3845-a52230688881 [Candidate]" term=2 - testing.go:149: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.883Z [INFO] server-b254b9e4-80f0-d1b8-3845-a52230688881: entering candidate state: node="Node at b254b9e4-80f0-d1b8-3845-a52230688881 [Candidate]" term=2 -testLoggerAdapter verbose: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.883Z [INFO] server-b254b9e4-80f0-d1b8-3845-a52230688881: election won: tally=2 - testing.go:149: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.883Z [INFO] server-b254b9e4-80f0-d1b8-3845-a52230688881: election won: tally=2 -testLoggerAdapter verbose: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.883Z [INFO] server-b254b9e4-80f0-d1b8-3845-a52230688881: entering leader state: leader="Node at b254b9e4-80f0-d1b8-3845-a52230688881 [Leader]" - testing.go:149: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.883Z [INFO] server-b254b9e4-80f0-d1b8-3845-a52230688881: entering leader state: leader="Node at b254b9e4-80f0-d1b8-3845-a52230688881 [Leader]" -testLoggerAdapter verbose: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.883Z [INFO] server-b254b9e4-80f0-d1b8-3845-a52230688881: added peer, starting replication: peer=server-b6c3026e-f376-206a-28b9-143e83a7515f - testing.go:149: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.883Z [INFO] server-b254b9e4-80f0-d1b8-3845-a52230688881: added peer, starting replication: peer=server-b6c3026e-f376-206a-28b9-143e83a7515f -testLoggerAdapter verbose: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.883Z [INFO] server-b254b9e4-80f0-d1b8-3845-a52230688881: added peer, starting replication: peer=server-4aaed739-7e6f-1846-02f8-95a420a09f91 - testing.go:149: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.883Z [INFO] server-b254b9e4-80f0-d1b8-3845-a52230688881: added peer, starting replication: peer=server-4aaed739-7e6f-1846-02f8-95a420a09f91 -testLoggerAdapter verbose: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.883Z [INFO] server-b254b9e4-80f0-d1b8-3845-a52230688881: pipelining replication: peer="{Voter server-4aaed739-7e6f-1846-02f8-95a420a09f91 4aaed739-7e6f-1846-02f8-95a420a09f91}" - testing.go:149: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.883Z [INFO] server-b254b9e4-80f0-d1b8-3845-a52230688881: pipelining replication: peer="{Voter server-4aaed739-7e6f-1846-02f8-95a420a09f91 4aaed739-7e6f-1846-02f8-95a420a09f91}" -testLoggerAdapter verbose: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.883Z [INFO] server-b254b9e4-80f0-d1b8-3845-a52230688881: pipelining replication: peer="{Voter server-b6c3026e-f376-206a-28b9-143e83a7515f b6c3026e-f376-206a-28b9-143e83a7515f}" - testing.go:149: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.883Z [INFO] server-b254b9e4-80f0-d1b8-3845-a52230688881: pipelining replication: peer="{Voter server-b6c3026e-f376-206a-28b9-143e83a7515f b6c3026e-f376-206a-28b9-143e83a7515f}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:29.988Z [INFO] stable state for Leader reached at 2024-09-03 21:34:29.883536574 +0000 UTC m=+10.930193710 (1 nodes), 59.916271ms from start of poll, 60.268588ms from cluster start. Timeout at 2024-09-03 21:34:29.988633631 +0000 UTC m=+11.035290789, 105.097079ms after stability - testing.go:149: cluster: 2024-09-03T21:34:29.988Z [INFO] stable state for Leader reached at 2024-09-03 21:34:29.883536574 +0000 UTC m=+10.930193710 (1 nodes), 59.916271ms from start of poll, 60.268588ms from cluster start. Timeout at 2024-09-03 21:34:29.988633631 +0000 UTC m=+11.035290789, 105.097079ms after stability -testLoggerAdapter verbose: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.989Z [INFO] server-b254b9e4-80f0-d1b8-3845-a52230688881: starting snapshot up to: index=12 - testing.go:149: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.989Z [INFO] server-b254b9e4-80f0-d1b8-3845-a52230688881: starting snapshot up to: index=12 - testing.go:153: 2024-09-03T21:34:29.989Z [INFO] creating new snapshot: path=/tmp/raft463276776/snapshots/2-12-1725399269989.tmp -testLoggerAdapter verbose: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.989Z [INFO] server-b254b9e4-80f0-d1b8-3845-a52230688881: snapshot complete up to: index=12 - testing.go:149: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.989Z [INFO] server-b254b9e4-80f0-d1b8-3845-a52230688881: snapshot complete up to: index=12 - testing.go:153: 2024-09-03T21:34:29.990Z [INFO] creating new snapshot: path=/tmp/raft463276776/snapshots/2-23-1725399269990.tmp -testLoggerAdapter verbose: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.990Z [INFO] server-b254b9e4-80f0-d1b8-3845-a52230688881: copied to local snapshot: bytes=71 - testing.go:149: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.990Z [INFO] server-b254b9e4-80f0-d1b8-3845-a52230688881: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.990Z [INFO] server-b254b9e4-80f0-d1b8-3845-a52230688881: restored user snapshot: index=1 - testing.go:149: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.990Z [INFO] server-b254b9e4-80f0-d1b8-3845-a52230688881: restored user snapshot: index=1 -testLoggerAdapter verbose: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.991Z [ERROR] server-b254b9e4-80f0-d1b8-3845-a52230688881: failed to get log: index=23 error="log not found" - testing.go:149: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.991Z [ERROR] server-b254b9e4-80f0-d1b8-3845-a52230688881: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.991Z [INFO] server-b254b9e4-80f0-d1b8-3845-a52230688881: aborting pipeline replication: peer="{Voter server-4aaed739-7e6f-1846-02f8-95a420a09f91 4aaed739-7e6f-1846-02f8-95a420a09f91}" - testing.go:149: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.991Z [INFO] server-b254b9e4-80f0-d1b8-3845-a52230688881: aborting pipeline replication: peer="{Voter server-4aaed739-7e6f-1846-02f8-95a420a09f91 4aaed739-7e6f-1846-02f8-95a420a09f91}" -testLoggerAdapter verbose: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.991Z [ERROR] server-b254b9e4-80f0-d1b8-3845-a52230688881: failed to get log: index=23 error="log not found" - testing.go:149: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.991Z [ERROR] server-b254b9e4-80f0-d1b8-3845-a52230688881: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.991Z [INFO] server-b254b9e4-80f0-d1b8-3845-a52230688881: aborting pipeline replication: peer="{Voter server-b6c3026e-f376-206a-28b9-143e83a7515f b6c3026e-f376-206a-28b9-143e83a7515f}" - testing.go:149: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.991Z [INFO] server-b254b9e4-80f0-d1b8-3845-a52230688881: aborting pipeline replication: peer="{Voter server-b6c3026e-f376-206a-28b9-143e83a7515f b6c3026e-f376-206a-28b9-143e83a7515f}" -testLoggerAdapter verbose: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.996Z [ERROR] server-b254b9e4-80f0-d1b8-3845-a52230688881: failed to get log: index=23 error="log not found" - testing.go:149: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.996Z [ERROR] server-b254b9e4-80f0-d1b8-3845-a52230688881: failed to get log: index=23 error="log not found" - testing.go:153: 2024-09-03T21:34:29.997Z [INFO] creating new snapshot: path=/tmp/raft353939398/snapshots/2-23-1725399269997.tmp -testLoggerAdapter verbose: server-b6c3026e-f376-206a-28b9-143e83a7515f: 2024-09-03T21:34:29.997Z [INFO] server-b6c3026e-f376-206a-28b9-143e83a7515f: copied to local snapshot: bytes=71 - testing.go:149: server-b6c3026e-f376-206a-28b9-143e83a7515f: 2024-09-03T21:34:29.997Z [INFO] server-b6c3026e-f376-206a-28b9-143e83a7515f: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-b6c3026e-f376-206a-28b9-143e83a7515f: 2024-09-03T21:34:29.997Z [INFO] server-b6c3026e-f376-206a-28b9-143e83a7515f: Installed remote snapshot - testing.go:149: server-b6c3026e-f376-206a-28b9-143e83a7515f: 2024-09-03T21:34:29.997Z [INFO] server-b6c3026e-f376-206a-28b9-143e83a7515f: Installed remote snapshot -testLoggerAdapter verbose: server-b6c3026e-f376-206a-28b9-143e83a7515f: 2024-09-03T21:34:29.997Z [WARN] server-b6c3026e-f376-206a-28b9-143e83a7515f: skipping application of old log: index=22 - testing.go:149: server-b6c3026e-f376-206a-28b9-143e83a7515f: 2024-09-03T21:34:29.997Z [WARN] server-b6c3026e-f376-206a-28b9-143e83a7515f: skipping application of old log: index=22 -testLoggerAdapter verbose: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.997Z [INFO] server-b254b9e4-80f0-d1b8-3845-a52230688881: pipelining replication: peer="{Voter server-b6c3026e-f376-206a-28b9-143e83a7515f b6c3026e-f376-206a-28b9-143e83a7515f}" - testing.go:149: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.997Z [INFO] server-b254b9e4-80f0-d1b8-3845-a52230688881: pipelining replication: peer="{Voter server-b6c3026e-f376-206a-28b9-143e83a7515f b6c3026e-f376-206a-28b9-143e83a7515f}" -testLoggerAdapter verbose: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.997Z [ERROR] server-b254b9e4-80f0-d1b8-3845-a52230688881: failed to get log: index=23 error="log not found" - testing.go:149: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.997Z [ERROR] server-b254b9e4-80f0-d1b8-3845-a52230688881: failed to get log: index=23 error="log not found" - testing.go:153: 2024-09-03T21:34:29.998Z [INFO] creating new snapshot: path=/tmp/raft851981914/snapshots/2-23-1725399269998.tmp -testLoggerAdapter verbose: server-4aaed739-7e6f-1846-02f8-95a420a09f91: 2024-09-03T21:34:29.998Z [INFO] server-4aaed739-7e6f-1846-02f8-95a420a09f91: copied to local snapshot: bytes=71 - testing.go:149: server-4aaed739-7e6f-1846-02f8-95a420a09f91: 2024-09-03T21:34:29.998Z [INFO] server-4aaed739-7e6f-1846-02f8-95a420a09f91: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-4aaed739-7e6f-1846-02f8-95a420a09f91: 2024-09-03T21:34:29.998Z [INFO] server-4aaed739-7e6f-1846-02f8-95a420a09f91: Installed remote snapshot - testing.go:149: server-4aaed739-7e6f-1846-02f8-95a420a09f91: 2024-09-03T21:34:29.998Z [INFO] server-4aaed739-7e6f-1846-02f8-95a420a09f91: Installed remote snapshot -testLoggerAdapter verbose: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.998Z [INFO] server-b254b9e4-80f0-d1b8-3845-a52230688881: pipelining replication: peer="{Voter server-4aaed739-7e6f-1846-02f8-95a420a09f91 4aaed739-7e6f-1846-02f8-95a420a09f91}" - testing.go:149: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:29.998Z [INFO] server-b254b9e4-80f0-d1b8-3845-a52230688881: pipelining replication: peer="{Voter server-4aaed739-7e6f-1846-02f8-95a420a09f91 4aaed739-7e6f-1846-02f8-95a420a09f91}" -testLoggerAdapter verbose: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:30.013Z [INFO] server-b254b9e4-80f0-d1b8-3845-a52230688881: aborting pipeline replication: peer="{Voter server-4aaed739-7e6f-1846-02f8-95a420a09f91 4aaed739-7e6f-1846-02f8-95a420a09f91}" - testing.go:149: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:30.013Z [INFO] server-b254b9e4-80f0-d1b8-3845-a52230688881: aborting pipeline replication: peer="{Voter server-4aaed739-7e6f-1846-02f8-95a420a09f91 4aaed739-7e6f-1846-02f8-95a420a09f91}" -testLoggerAdapter verbose: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:30.013Z [INFO] server-b254b9e4-80f0-d1b8-3845-a52230688881: aborting pipeline replication: peer="{Voter server-b6c3026e-f376-206a-28b9-143e83a7515f b6c3026e-f376-206a-28b9-143e83a7515f}" - testing.go:149: server-b254b9e4-80f0-d1b8-3845-a52230688881: 2024-09-03T21:34:30.013Z [INFO] server-b254b9e4-80f0-d1b8-3845-a52230688881: aborting pipeline replication: peer="{Voter server-b6c3026e-f376-206a-28b9-143e83a7515f b6c3026e-f376-206a-28b9-143e83a7515f}" -testLoggerAdapter verbose: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.015Z [INFO] server-e79138c5-a779-8127-b620-e549c338c59d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e79138c5-a779-8127-b620-e549c338c59d Address:e79138c5-a779-8127-b620-e549c338c59d} {Suffrage:Voter ID:server-5c52c69b-0582-e32b-32f2-0967d523dda4 Address:5c52c69b-0582-e32b-32f2-0967d523dda4} {Suffrage:Voter ID:server-8d78b689-c6a6-a4ea-f468-22caeedd7211 Address:8d78b689-c6a6-a4ea-f468-22caeedd7211}]" - testing.go:149: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.015Z [INFO] server-e79138c5-a779-8127-b620-e549c338c59d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e79138c5-a779-8127-b620-e549c338c59d Address:e79138c5-a779-8127-b620-e549c338c59d} {Suffrage:Voter ID:server-5c52c69b-0582-e32b-32f2-0967d523dda4 Address:5c52c69b-0582-e32b-32f2-0967d523dda4} {Suffrage:Voter ID:server-8d78b689-c6a6-a4ea-f468-22caeedd7211 Address:8d78b689-c6a6-a4ea-f468-22caeedd7211}]" -testLoggerAdapter verbose: server-5c52c69b-0582-e32b-32f2-0967d523dda4: 2024-09-03T21:34:30.015Z [INFO] server-5c52c69b-0582-e32b-32f2-0967d523dda4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e79138c5-a779-8127-b620-e549c338c59d Address:e79138c5-a779-8127-b620-e549c338c59d} {Suffrage:Voter ID:server-5c52c69b-0582-e32b-32f2-0967d523dda4 Address:5c52c69b-0582-e32b-32f2-0967d523dda4} {Suffrage:Voter ID:server-8d78b689-c6a6-a4ea-f468-22caeedd7211 Address:8d78b689-c6a6-a4ea-f468-22caeedd7211}]" - testing.go:149: server-5c52c69b-0582-e32b-32f2-0967d523dda4: 2024-09-03T21:34:30.015Z [INFO] server-5c52c69b-0582-e32b-32f2-0967d523dda4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e79138c5-a779-8127-b620-e549c338c59d Address:e79138c5-a779-8127-b620-e549c338c59d} {Suffrage:Voter ID:server-5c52c69b-0582-e32b-32f2-0967d523dda4 Address:5c52c69b-0582-e32b-32f2-0967d523dda4} {Suffrage:Voter ID:server-8d78b689-c6a6-a4ea-f468-22caeedd7211 Address:8d78b689-c6a6-a4ea-f468-22caeedd7211}]" -testLoggerAdapter verbose: server-8d78b689-c6a6-a4ea-f468-22caeedd7211: 2024-09-03T21:34:30.015Z [INFO] server-8d78b689-c6a6-a4ea-f468-22caeedd7211: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e79138c5-a779-8127-b620-e549c338c59d Address:e79138c5-a779-8127-b620-e549c338c59d} {Suffrage:Voter ID:server-5c52c69b-0582-e32b-32f2-0967d523dda4 Address:5c52c69b-0582-e32b-32f2-0967d523dda4} {Suffrage:Voter ID:server-8d78b689-c6a6-a4ea-f468-22caeedd7211 Address:8d78b689-c6a6-a4ea-f468-22caeedd7211}]" - testing.go:149: server-8d78b689-c6a6-a4ea-f468-22caeedd7211: 2024-09-03T21:34:30.015Z [INFO] server-8d78b689-c6a6-a4ea-f468-22caeedd7211: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e79138c5-a779-8127-b620-e549c338c59d Address:e79138c5-a779-8127-b620-e549c338c59d} {Suffrage:Voter ID:server-5c52c69b-0582-e32b-32f2-0967d523dda4 Address:5c52c69b-0582-e32b-32f2-0967d523dda4} {Suffrage:Voter ID:server-8d78b689-c6a6-a4ea-f468-22caeedd7211 Address:8d78b689-c6a6-a4ea-f468-22caeedd7211}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:30.015Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:30.015Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-5c52c69b-0582-e32b-32f2-0967d523dda4: 2024-09-03T21:34:30.016Z [INFO] server-5c52c69b-0582-e32b-32f2-0967d523dda4: entering follower state: follower="Node at 5c52c69b-0582-e32b-32f2-0967d523dda4 [Follower]" leader= - testing.go:149: server-5c52c69b-0582-e32b-32f2-0967d523dda4: 2024-09-03T21:34:30.016Z [INFO] server-5c52c69b-0582-e32b-32f2-0967d523dda4: entering follower state: follower="Node at 5c52c69b-0582-e32b-32f2-0967d523dda4 [Follower]" leader= -testLoggerAdapter verbose: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.017Z [INFO] server-e79138c5-a779-8127-b620-e549c338c59d: entering follower state: follower="Node at e79138c5-a779-8127-b620-e549c338c59d [Follower]" leader= - testing.go:149: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.017Z [INFO] server-e79138c5-a779-8127-b620-e549c338c59d: entering follower state: follower="Node at e79138c5-a779-8127-b620-e549c338c59d [Follower]" leader= -testLoggerAdapter verbose: server-8d78b689-c6a6-a4ea-f468-22caeedd7211: 2024-09-03T21:34:30.017Z [INFO] server-8d78b689-c6a6-a4ea-f468-22caeedd7211: entering follower state: follower="Node at 8d78b689-c6a6-a4ea-f468-22caeedd7211 [Follower]" leader= - testing.go:149: server-8d78b689-c6a6-a4ea-f468-22caeedd7211: 2024-09-03T21:34:30.017Z [INFO] server-8d78b689-c6a6-a4ea-f468-22caeedd7211: entering follower state: follower="Node at 8d78b689-c6a6-a4ea-f468-22caeedd7211 [Follower]" leader= -testLoggerAdapter verbose: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.088Z [WARN] server-e79138c5-a779-8127-b620-e549c338c59d: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.088Z [WARN] server-e79138c5-a779-8127-b620-e549c338c59d: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.088Z [INFO] server-e79138c5-a779-8127-b620-e549c338c59d: entering candidate state: node="Node at e79138c5-a779-8127-b620-e549c338c59d [Candidate]" term=2 - testing.go:149: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.088Z [INFO] server-e79138c5-a779-8127-b620-e549c338c59d: entering candidate state: node="Node at e79138c5-a779-8127-b620-e549c338c59d [Candidate]" term=2 -testLoggerAdapter verbose: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.089Z [INFO] server-e79138c5-a779-8127-b620-e549c338c59d: election won: tally=2 - testing.go:149: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.089Z [INFO] server-e79138c5-a779-8127-b620-e549c338c59d: election won: tally=2 -testLoggerAdapter verbose: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.089Z [INFO] server-e79138c5-a779-8127-b620-e549c338c59d: entering leader state: leader="Node at e79138c5-a779-8127-b620-e549c338c59d [Leader]" - testing.go:149: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.089Z [INFO] server-e79138c5-a779-8127-b620-e549c338c59d: entering leader state: leader="Node at e79138c5-a779-8127-b620-e549c338c59d [Leader]" -testLoggerAdapter verbose: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.089Z [INFO] server-e79138c5-a779-8127-b620-e549c338c59d: added peer, starting replication: peer=server-5c52c69b-0582-e32b-32f2-0967d523dda4 - testing.go:149: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.089Z [INFO] server-e79138c5-a779-8127-b620-e549c338c59d: added peer, starting replication: peer=server-5c52c69b-0582-e32b-32f2-0967d523dda4 -testLoggerAdapter verbose: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.089Z [INFO] server-e79138c5-a779-8127-b620-e549c338c59d: added peer, starting replication: peer=server-8d78b689-c6a6-a4ea-f468-22caeedd7211 - testing.go:149: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.089Z [INFO] server-e79138c5-a779-8127-b620-e549c338c59d: added peer, starting replication: peer=server-8d78b689-c6a6-a4ea-f468-22caeedd7211 -testLoggerAdapter verbose: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.089Z [INFO] server-e79138c5-a779-8127-b620-e549c338c59d: pipelining replication: peer="{Voter server-8d78b689-c6a6-a4ea-f468-22caeedd7211 8d78b689-c6a6-a4ea-f468-22caeedd7211}" - testing.go:149: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.089Z [INFO] server-e79138c5-a779-8127-b620-e549c338c59d: pipelining replication: peer="{Voter server-8d78b689-c6a6-a4ea-f468-22caeedd7211 8d78b689-c6a6-a4ea-f468-22caeedd7211}" -testLoggerAdapter verbose: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.089Z [INFO] server-e79138c5-a779-8127-b620-e549c338c59d: pipelining replication: peer="{Voter server-5c52c69b-0582-e32b-32f2-0967d523dda4 5c52c69b-0582-e32b-32f2-0967d523dda4}" - testing.go:149: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.089Z [INFO] server-e79138c5-a779-8127-b620-e549c338c59d: pipelining replication: peer="{Voter server-5c52c69b-0582-e32b-32f2-0967d523dda4 5c52c69b-0582-e32b-32f2-0967d523dda4}" -testLoggerAdapter verbose: server-5c52c69b-0582-e32b-32f2-0967d523dda4: 2024-09-03T21:34:30.089Z [WARN] server-5c52c69b-0582-e32b-32f2-0967d523dda4: rejecting vote request since our last term is greater: candidate=e79138c5-a779-8127-b620-e549c338c59d last-term=2 last-candidate-term=1 - testing.go:149: server-5c52c69b-0582-e32b-32f2-0967d523dda4: 2024-09-03T21:34:30.089Z [WARN] server-5c52c69b-0582-e32b-32f2-0967d523dda4: rejecting vote request since our last term is greater: candidate=e79138c5-a779-8127-b620-e549c338c59d last-term=2 last-candidate-term=1 -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:30.194Z [INFO] stable state for Leader reached at 2024-09-03 21:34:30.089344548 +0000 UTC m=+11.136001678 (1 nodes), 73.428776ms from start of poll, 74.105269ms from cluster start. Timeout at 2024-09-03 21:34:30.194436615 +0000 UTC m=+11.241093745, 105.092067ms after stability - testing.go:149: cluster: 2024-09-03T21:34:30.194Z [INFO] stable state for Leader reached at 2024-09-03 21:34:30.089344548 +0000 UTC m=+11.136001678 (1 nodes), 73.428776ms from start of poll, 74.105269ms from cluster start. Timeout at 2024-09-03 21:34:30.194436615 +0000 UTC m=+11.241093745, 105.092067ms after stability -testLoggerAdapter verbose: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.194Z [INFO] server-e79138c5-a779-8127-b620-e549c338c59d: starting snapshot up to: index=12 - testing.go:149: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.194Z [INFO] server-e79138c5-a779-8127-b620-e549c338c59d: starting snapshot up to: index=12 - testing.go:153: 2024-09-03T21:34:30.194Z [INFO] creating new snapshot: path=/tmp/raft239277724/snapshots/2-12-1725399270194.tmp -testLoggerAdapter verbose: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.195Z [INFO] server-e79138c5-a779-8127-b620-e549c338c59d: snapshot complete up to: index=12 - testing.go:149: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.195Z [INFO] server-e79138c5-a779-8127-b620-e549c338c59d: snapshot complete up to: index=12 - testing.go:153: 2024-09-03T21:34:30.195Z [INFO] creating new snapshot: path=/tmp/raft239277724/snapshots/2-23-1725399270195.tmp -testLoggerAdapter verbose: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.195Z [INFO] server-e79138c5-a779-8127-b620-e549c338c59d: copied to local snapshot: bytes=71 - testing.go:149: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.195Z [INFO] server-e79138c5-a779-8127-b620-e549c338c59d: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.196Z [INFO] server-e79138c5-a779-8127-b620-e549c338c59d: restored user snapshot: index=1 - testing.go:149: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.196Z [INFO] server-e79138c5-a779-8127-b620-e549c338c59d: restored user snapshot: index=1 -testLoggerAdapter verbose: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.196Z [ERROR] server-e79138c5-a779-8127-b620-e549c338c59d: failed to get log: index=23 error="log not found" - testing.go:149: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.196Z [ERROR] server-e79138c5-a779-8127-b620-e549c338c59d: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.196Z [INFO] server-e79138c5-a779-8127-b620-e549c338c59d: aborting pipeline replication: peer="{Voter server-8d78b689-c6a6-a4ea-f468-22caeedd7211 8d78b689-c6a6-a4ea-f468-22caeedd7211}" - testing.go:149: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.196Z [INFO] server-e79138c5-a779-8127-b620-e549c338c59d: aborting pipeline replication: peer="{Voter server-8d78b689-c6a6-a4ea-f468-22caeedd7211 8d78b689-c6a6-a4ea-f468-22caeedd7211}" -testLoggerAdapter verbose: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.196Z [ERROR] server-e79138c5-a779-8127-b620-e549c338c59d: failed to get log: index=23 error="log not found" - testing.go:149: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.196Z [ERROR] server-e79138c5-a779-8127-b620-e549c338c59d: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.196Z [INFO] server-e79138c5-a779-8127-b620-e549c338c59d: aborting pipeline replication: peer="{Voter server-5c52c69b-0582-e32b-32f2-0967d523dda4 5c52c69b-0582-e32b-32f2-0967d523dda4}" - testing.go:149: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.196Z [INFO] server-e79138c5-a779-8127-b620-e549c338c59d: aborting pipeline replication: peer="{Voter server-5c52c69b-0582-e32b-32f2-0967d523dda4 5c52c69b-0582-e32b-32f2-0967d523dda4}" -testLoggerAdapter verbose: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.205Z [ERROR] server-e79138c5-a779-8127-b620-e549c338c59d: failed to get log: index=23 error="log not found" - testing.go:149: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.205Z [ERROR] server-e79138c5-a779-8127-b620-e549c338c59d: failed to get log: index=23 error="log not found" - testing.go:153: 2024-09-03T21:34:30.205Z [INFO] creating new snapshot: path=/tmp/raft947985296/snapshots/2-23-1725399270205.tmp -testLoggerAdapter verbose: server-8d78b689-c6a6-a4ea-f468-22caeedd7211: 2024-09-03T21:34:30.205Z [INFO] server-8d78b689-c6a6-a4ea-f468-22caeedd7211: copied to local snapshot: bytes=71 - testing.go:149: server-8d78b689-c6a6-a4ea-f468-22caeedd7211: 2024-09-03T21:34:30.205Z [INFO] server-8d78b689-c6a6-a4ea-f468-22caeedd7211: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-8d78b689-c6a6-a4ea-f468-22caeedd7211: 2024-09-03T21:34:30.205Z [INFO] server-8d78b689-c6a6-a4ea-f468-22caeedd7211: Installed remote snapshot - testing.go:149: server-8d78b689-c6a6-a4ea-f468-22caeedd7211: 2024-09-03T21:34:30.205Z [INFO] server-8d78b689-c6a6-a4ea-f468-22caeedd7211: Installed remote snapshot -testLoggerAdapter verbose: server-8d78b689-c6a6-a4ea-f468-22caeedd7211: 2024-09-03T21:34:30.205Z [WARN] server-8d78b689-c6a6-a4ea-f468-22caeedd7211: skipping application of old log: index=22 - testing.go:149: server-8d78b689-c6a6-a4ea-f468-22caeedd7211: 2024-09-03T21:34:30.205Z [WARN] server-8d78b689-c6a6-a4ea-f468-22caeedd7211: skipping application of old log: index=22 -testLoggerAdapter verbose: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.206Z [INFO] server-e79138c5-a779-8127-b620-e549c338c59d: pipelining replication: peer="{Voter server-8d78b689-c6a6-a4ea-f468-22caeedd7211 8d78b689-c6a6-a4ea-f468-22caeedd7211}" - testing.go:149: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.206Z [INFO] server-e79138c5-a779-8127-b620-e549c338c59d: pipelining replication: peer="{Voter server-8d78b689-c6a6-a4ea-f468-22caeedd7211 8d78b689-c6a6-a4ea-f468-22caeedd7211}" -testLoggerAdapter verbose: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.206Z [ERROR] server-e79138c5-a779-8127-b620-e549c338c59d: failed to get log: index=23 error="log not found" - testing.go:149: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.206Z [ERROR] server-e79138c5-a779-8127-b620-e549c338c59d: failed to get log: index=23 error="log not found" - testing.go:153: 2024-09-03T21:34:30.206Z [INFO] creating new snapshot: path=/tmp/raft285853742/snapshots/2-23-1725399270206.tmp -testLoggerAdapter verbose: server-5c52c69b-0582-e32b-32f2-0967d523dda4: 2024-09-03T21:34:30.206Z [INFO] server-5c52c69b-0582-e32b-32f2-0967d523dda4: copied to local snapshot: bytes=71 - testing.go:149: server-5c52c69b-0582-e32b-32f2-0967d523dda4: 2024-09-03T21:34:30.206Z [INFO] server-5c52c69b-0582-e32b-32f2-0967d523dda4: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-5c52c69b-0582-e32b-32f2-0967d523dda4: 2024-09-03T21:34:30.206Z [INFO] server-5c52c69b-0582-e32b-32f2-0967d523dda4: Installed remote snapshot - testing.go:149: server-5c52c69b-0582-e32b-32f2-0967d523dda4: 2024-09-03T21:34:30.206Z [INFO] server-5c52c69b-0582-e32b-32f2-0967d523dda4: Installed remote snapshot -testLoggerAdapter verbose: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.206Z [INFO] server-e79138c5-a779-8127-b620-e549c338c59d: pipelining replication: peer="{Voter server-5c52c69b-0582-e32b-32f2-0967d523dda4 5c52c69b-0582-e32b-32f2-0967d523dda4}" - testing.go:149: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.206Z [INFO] server-e79138c5-a779-8127-b620-e549c338c59d: pipelining replication: peer="{Voter server-5c52c69b-0582-e32b-32f2-0967d523dda4 5c52c69b-0582-e32b-32f2-0967d523dda4}" -testLoggerAdapter verbose: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.221Z [INFO] server-e79138c5-a779-8127-b620-e549c338c59d: aborting pipeline replication: peer="{Voter server-5c52c69b-0582-e32b-32f2-0967d523dda4 5c52c69b-0582-e32b-32f2-0967d523dda4}" - testing.go:149: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.221Z [INFO] server-e79138c5-a779-8127-b620-e549c338c59d: aborting pipeline replication: peer="{Voter server-5c52c69b-0582-e32b-32f2-0967d523dda4 5c52c69b-0582-e32b-32f2-0967d523dda4}" -testLoggerAdapter verbose: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.221Z [INFO] server-e79138c5-a779-8127-b620-e549c338c59d: aborting pipeline replication: peer="{Voter server-8d78b689-c6a6-a4ea-f468-22caeedd7211 8d78b689-c6a6-a4ea-f468-22caeedd7211}" - testing.go:149: server-e79138c5-a779-8127-b620-e549c338c59d: 2024-09-03T21:34:30.221Z [INFO] server-e79138c5-a779-8127-b620-e549c338c59d: aborting pipeline replication: peer="{Voter server-8d78b689-c6a6-a4ea-f468-22caeedd7211 8d78b689-c6a6-a4ea-f468-22caeedd7211}" -testLoggerAdapter verbose: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.222Z [INFO] server-69672c51-c090-7959-3b75-caa4251589da: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-69672c51-c090-7959-3b75-caa4251589da Address:69672c51-c090-7959-3b75-caa4251589da} {Suffrage:Voter ID:server-dccc9681-70a5-65b1-5ad4-136537c657c6 Address:dccc9681-70a5-65b1-5ad4-136537c657c6} {Suffrage:Voter ID:server-4caa6cad-6b90-683e-9f83-0dc355252cb9 Address:4caa6cad-6b90-683e-9f83-0dc355252cb9}]" - testing.go:149: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.222Z [INFO] server-69672c51-c090-7959-3b75-caa4251589da: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-69672c51-c090-7959-3b75-caa4251589da Address:69672c51-c090-7959-3b75-caa4251589da} {Suffrage:Voter ID:server-dccc9681-70a5-65b1-5ad4-136537c657c6 Address:dccc9681-70a5-65b1-5ad4-136537c657c6} {Suffrage:Voter ID:server-4caa6cad-6b90-683e-9f83-0dc355252cb9 Address:4caa6cad-6b90-683e-9f83-0dc355252cb9}]" -testLoggerAdapter verbose: server-dccc9681-70a5-65b1-5ad4-136537c657c6: 2024-09-03T21:34:30.222Z [INFO] server-dccc9681-70a5-65b1-5ad4-136537c657c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-69672c51-c090-7959-3b75-caa4251589da Address:69672c51-c090-7959-3b75-caa4251589da} {Suffrage:Voter ID:server-dccc9681-70a5-65b1-5ad4-136537c657c6 Address:dccc9681-70a5-65b1-5ad4-136537c657c6} {Suffrage:Voter ID:server-4caa6cad-6b90-683e-9f83-0dc355252cb9 Address:4caa6cad-6b90-683e-9f83-0dc355252cb9}]" - testing.go:149: server-dccc9681-70a5-65b1-5ad4-136537c657c6: 2024-09-03T21:34:30.222Z [INFO] server-dccc9681-70a5-65b1-5ad4-136537c657c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-69672c51-c090-7959-3b75-caa4251589da Address:69672c51-c090-7959-3b75-caa4251589da} {Suffrage:Voter ID:server-dccc9681-70a5-65b1-5ad4-136537c657c6 Address:dccc9681-70a5-65b1-5ad4-136537c657c6} {Suffrage:Voter ID:server-4caa6cad-6b90-683e-9f83-0dc355252cb9 Address:4caa6cad-6b90-683e-9f83-0dc355252cb9}]" -testLoggerAdapter verbose: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.222Z [INFO] server-69672c51-c090-7959-3b75-caa4251589da: entering follower state: follower="Node at 69672c51-c090-7959-3b75-caa4251589da [Follower]" leader= - testing.go:149: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.222Z [INFO] server-69672c51-c090-7959-3b75-caa4251589da: entering follower state: follower="Node at 69672c51-c090-7959-3b75-caa4251589da [Follower]" leader= -testLoggerAdapter verbose: server-dccc9681-70a5-65b1-5ad4-136537c657c6: 2024-09-03T21:34:30.222Z [INFO] server-dccc9681-70a5-65b1-5ad4-136537c657c6: entering follower state: follower="Node at dccc9681-70a5-65b1-5ad4-136537c657c6 [Follower]" leader= - testing.go:149: server-dccc9681-70a5-65b1-5ad4-136537c657c6: 2024-09-03T21:34:30.222Z [INFO] server-dccc9681-70a5-65b1-5ad4-136537c657c6: entering follower state: follower="Node at dccc9681-70a5-65b1-5ad4-136537c657c6 [Follower]" leader= -testLoggerAdapter verbose: server-4caa6cad-6b90-683e-9f83-0dc355252cb9: 2024-09-03T21:34:30.222Z [INFO] server-4caa6cad-6b90-683e-9f83-0dc355252cb9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-69672c51-c090-7959-3b75-caa4251589da Address:69672c51-c090-7959-3b75-caa4251589da} {Suffrage:Voter ID:server-dccc9681-70a5-65b1-5ad4-136537c657c6 Address:dccc9681-70a5-65b1-5ad4-136537c657c6} {Suffrage:Voter ID:server-4caa6cad-6b90-683e-9f83-0dc355252cb9 Address:4caa6cad-6b90-683e-9f83-0dc355252cb9}]" - testing.go:149: server-4caa6cad-6b90-683e-9f83-0dc355252cb9: 2024-09-03T21:34:30.222Z [INFO] server-4caa6cad-6b90-683e-9f83-0dc355252cb9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-69672c51-c090-7959-3b75-caa4251589da Address:69672c51-c090-7959-3b75-caa4251589da} {Suffrage:Voter ID:server-dccc9681-70a5-65b1-5ad4-136537c657c6 Address:dccc9681-70a5-65b1-5ad4-136537c657c6} {Suffrage:Voter ID:server-4caa6cad-6b90-683e-9f83-0dc355252cb9 Address:4caa6cad-6b90-683e-9f83-0dc355252cb9}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:30.223Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:30.223Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-4caa6cad-6b90-683e-9f83-0dc355252cb9: 2024-09-03T21:34:30.223Z [INFO] server-4caa6cad-6b90-683e-9f83-0dc355252cb9: entering follower state: follower="Node at 4caa6cad-6b90-683e-9f83-0dc355252cb9 [Follower]" leader= - testing.go:149: server-4caa6cad-6b90-683e-9f83-0dc355252cb9: 2024-09-03T21:34:30.223Z [INFO] server-4caa6cad-6b90-683e-9f83-0dc355252cb9: entering follower state: follower="Node at 4caa6cad-6b90-683e-9f83-0dc355252cb9 [Follower]" leader= -testLoggerAdapter verbose: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.282Z [WARN] server-69672c51-c090-7959-3b75-caa4251589da: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.282Z [WARN] server-69672c51-c090-7959-3b75-caa4251589da: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.282Z [INFO] server-69672c51-c090-7959-3b75-caa4251589da: entering candidate state: node="Node at 69672c51-c090-7959-3b75-caa4251589da [Candidate]" term=2 - testing.go:149: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.282Z [INFO] server-69672c51-c090-7959-3b75-caa4251589da: entering candidate state: node="Node at 69672c51-c090-7959-3b75-caa4251589da [Candidate]" term=2 -testLoggerAdapter verbose: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.282Z [INFO] server-69672c51-c090-7959-3b75-caa4251589da: election won: tally=2 - testing.go:149: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.282Z [INFO] server-69672c51-c090-7959-3b75-caa4251589da: election won: tally=2 -testLoggerAdapter verbose: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.282Z [INFO] server-69672c51-c090-7959-3b75-caa4251589da: entering leader state: leader="Node at 69672c51-c090-7959-3b75-caa4251589da [Leader]" - testing.go:149: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.282Z [INFO] server-69672c51-c090-7959-3b75-caa4251589da: entering leader state: leader="Node at 69672c51-c090-7959-3b75-caa4251589da [Leader]" -testLoggerAdapter verbose: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.282Z [INFO] server-69672c51-c090-7959-3b75-caa4251589da: added peer, starting replication: peer=server-dccc9681-70a5-65b1-5ad4-136537c657c6 - testing.go:149: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.282Z [INFO] server-69672c51-c090-7959-3b75-caa4251589da: added peer, starting replication: peer=server-dccc9681-70a5-65b1-5ad4-136537c657c6 -testLoggerAdapter verbose: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.282Z [INFO] server-69672c51-c090-7959-3b75-caa4251589da: added peer, starting replication: peer=server-4caa6cad-6b90-683e-9f83-0dc355252cb9 - testing.go:149: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.282Z [INFO] server-69672c51-c090-7959-3b75-caa4251589da: added peer, starting replication: peer=server-4caa6cad-6b90-683e-9f83-0dc355252cb9 -testLoggerAdapter verbose: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.282Z [INFO] server-69672c51-c090-7959-3b75-caa4251589da: pipelining replication: peer="{Voter server-dccc9681-70a5-65b1-5ad4-136537c657c6 dccc9681-70a5-65b1-5ad4-136537c657c6}" - testing.go:149: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.282Z [INFO] server-69672c51-c090-7959-3b75-caa4251589da: pipelining replication: peer="{Voter server-dccc9681-70a5-65b1-5ad4-136537c657c6 dccc9681-70a5-65b1-5ad4-136537c657c6}" -testLoggerAdapter verbose: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.282Z [INFO] server-69672c51-c090-7959-3b75-caa4251589da: pipelining replication: peer="{Voter server-4caa6cad-6b90-683e-9f83-0dc355252cb9 4caa6cad-6b90-683e-9f83-0dc355252cb9}" - testing.go:149: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.282Z [INFO] server-69672c51-c090-7959-3b75-caa4251589da: pipelining replication: peer="{Voter server-4caa6cad-6b90-683e-9f83-0dc355252cb9 4caa6cad-6b90-683e-9f83-0dc355252cb9}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:30.387Z [INFO] stable state for Leader reached at 2024-09-03 21:34:30.282747 +0000 UTC m=+11.329404130 (1 nodes), 59.673935ms from start of poll, 60.114149ms from cluster start. Timeout at 2024-09-03 21:34:30.387812807 +0000 UTC m=+11.434469937, 105.065807ms after stability - testing.go:149: cluster: 2024-09-03T21:34:30.387Z [INFO] stable state for Leader reached at 2024-09-03 21:34:30.282747 +0000 UTC m=+11.329404130 (1 nodes), 59.673935ms from start of poll, 60.114149ms from cluster start. Timeout at 2024-09-03 21:34:30.387812807 +0000 UTC m=+11.434469937, 105.065807ms after stability -testLoggerAdapter verbose: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.388Z [INFO] server-69672c51-c090-7959-3b75-caa4251589da: starting snapshot up to: index=12 - testing.go:149: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.388Z [INFO] server-69672c51-c090-7959-3b75-caa4251589da: starting snapshot up to: index=12 - testing.go:153: 2024-09-03T21:34:30.388Z [INFO] creating new snapshot: path=/tmp/raft465204546/snapshots/2-12-1725399270388.tmp -testLoggerAdapter verbose: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.388Z [INFO] server-69672c51-c090-7959-3b75-caa4251589da: snapshot complete up to: index=12 - testing.go:149: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.388Z [INFO] server-69672c51-c090-7959-3b75-caa4251589da: snapshot complete up to: index=12 - testing.go:153: 2024-09-03T21:34:30.389Z [INFO] creating new snapshot: path=/tmp/raft465204546/snapshots/2-23-1725399270389.tmp -testLoggerAdapter verbose: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.389Z [INFO] server-69672c51-c090-7959-3b75-caa4251589da: copied to local snapshot: bytes=71 - testing.go:149: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.389Z [INFO] server-69672c51-c090-7959-3b75-caa4251589da: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.390Z [INFO] server-69672c51-c090-7959-3b75-caa4251589da: restored user snapshot: index=1 - testing.go:149: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.390Z [INFO] server-69672c51-c090-7959-3b75-caa4251589da: restored user snapshot: index=1 -testLoggerAdapter verbose: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.390Z [ERROR] server-69672c51-c090-7959-3b75-caa4251589da: failed to get log: index=23 error="log not found" - testing.go:149: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.390Z [ERROR] server-69672c51-c090-7959-3b75-caa4251589da: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.390Z [INFO] server-69672c51-c090-7959-3b75-caa4251589da: aborting pipeline replication: peer="{Voter server-4caa6cad-6b90-683e-9f83-0dc355252cb9 4caa6cad-6b90-683e-9f83-0dc355252cb9}" - testing.go:149: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.390Z [INFO] server-69672c51-c090-7959-3b75-caa4251589da: aborting pipeline replication: peer="{Voter server-4caa6cad-6b90-683e-9f83-0dc355252cb9 4caa6cad-6b90-683e-9f83-0dc355252cb9}" -testLoggerAdapter verbose: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.391Z [ERROR] server-69672c51-c090-7959-3b75-caa4251589da: failed to get log: index=23 error="log not found" - testing.go:149: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.391Z [ERROR] server-69672c51-c090-7959-3b75-caa4251589da: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.391Z [INFO] server-69672c51-c090-7959-3b75-caa4251589da: aborting pipeline replication: peer="{Voter server-dccc9681-70a5-65b1-5ad4-136537c657c6 dccc9681-70a5-65b1-5ad4-136537c657c6}" - testing.go:149: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.391Z [INFO] server-69672c51-c090-7959-3b75-caa4251589da: aborting pipeline replication: peer="{Voter server-dccc9681-70a5-65b1-5ad4-136537c657c6 dccc9681-70a5-65b1-5ad4-136537c657c6}" -testLoggerAdapter verbose: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.399Z [ERROR] server-69672c51-c090-7959-3b75-caa4251589da: failed to get log: index=23 error="log not found" - testing.go:149: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.399Z [ERROR] server-69672c51-c090-7959-3b75-caa4251589da: failed to get log: index=23 error="log not found" - testing.go:153: 2024-09-03T21:34:30.399Z [INFO] creating new snapshot: path=/tmp/raft570130838/snapshots/2-23-1725399270399.tmp -testLoggerAdapter verbose: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.400Z [ERROR] server-69672c51-c090-7959-3b75-caa4251589da: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-4caa6cad-6b90-683e-9f83-0dc355252cb9: 2024-09-03T21:34:30.400Z [INFO] server-4caa6cad-6b90-683e-9f83-0dc355252cb9: copied to local snapshot: bytes=71 - testing.go:149: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.400Z [ERROR] server-69672c51-c090-7959-3b75-caa4251589da: failed to get log: index=23 error="log not found" - testing.go:149: server-4caa6cad-6b90-683e-9f83-0dc355252cb9: 2024-09-03T21:34:30.400Z [INFO] server-4caa6cad-6b90-683e-9f83-0dc355252cb9: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-4caa6cad-6b90-683e-9f83-0dc355252cb9: 2024-09-03T21:34:30.400Z [INFO] server-4caa6cad-6b90-683e-9f83-0dc355252cb9: Installed remote snapshot - testing.go:149: server-4caa6cad-6b90-683e-9f83-0dc355252cb9: 2024-09-03T21:34:30.400Z [INFO] server-4caa6cad-6b90-683e-9f83-0dc355252cb9: Installed remote snapshot -testLoggerAdapter verbose: server-4caa6cad-6b90-683e-9f83-0dc355252cb9: 2024-09-03T21:34:30.400Z [WARN] server-4caa6cad-6b90-683e-9f83-0dc355252cb9: skipping application of old log: index=22 - testing.go:149: server-4caa6cad-6b90-683e-9f83-0dc355252cb9: 2024-09-03T21:34:30.400Z [WARN] server-4caa6cad-6b90-683e-9f83-0dc355252cb9: skipping application of old log: index=22 -testLoggerAdapter verbose: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.400Z [INFO] server-69672c51-c090-7959-3b75-caa4251589da: pipelining replication: peer="{Voter server-4caa6cad-6b90-683e-9f83-0dc355252cb9 4caa6cad-6b90-683e-9f83-0dc355252cb9}" - testing.go:149: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.400Z [INFO] server-69672c51-c090-7959-3b75-caa4251589da: pipelining replication: peer="{Voter server-4caa6cad-6b90-683e-9f83-0dc355252cb9 4caa6cad-6b90-683e-9f83-0dc355252cb9}" - testing.go:153: 2024-09-03T21:34:30.400Z [INFO] creating new snapshot: path=/tmp/raft737888708/snapshots/2-23-1725399270400.tmp -testLoggerAdapter verbose: server-dccc9681-70a5-65b1-5ad4-136537c657c6: 2024-09-03T21:34:30.400Z [INFO] server-dccc9681-70a5-65b1-5ad4-136537c657c6: copied to local snapshot: bytes=71 - testing.go:149: server-dccc9681-70a5-65b1-5ad4-136537c657c6: 2024-09-03T21:34:30.400Z [INFO] server-dccc9681-70a5-65b1-5ad4-136537c657c6: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-dccc9681-70a5-65b1-5ad4-136537c657c6: 2024-09-03T21:34:30.400Z [INFO] server-dccc9681-70a5-65b1-5ad4-136537c657c6: Installed remote snapshot - testing.go:149: server-dccc9681-70a5-65b1-5ad4-136537c657c6: 2024-09-03T21:34:30.400Z [INFO] server-dccc9681-70a5-65b1-5ad4-136537c657c6: Installed remote snapshot -testLoggerAdapter verbose: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.401Z [INFO] server-69672c51-c090-7959-3b75-caa4251589da: pipelining replication: peer="{Voter server-dccc9681-70a5-65b1-5ad4-136537c657c6 dccc9681-70a5-65b1-5ad4-136537c657c6}" - testing.go:149: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.401Z [INFO] server-69672c51-c090-7959-3b75-caa4251589da: pipelining replication: peer="{Voter server-dccc9681-70a5-65b1-5ad4-136537c657c6 dccc9681-70a5-65b1-5ad4-136537c657c6}" -testLoggerAdapter verbose: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.416Z [INFO] server-69672c51-c090-7959-3b75-caa4251589da: aborting pipeline replication: peer="{Voter server-4caa6cad-6b90-683e-9f83-0dc355252cb9 4caa6cad-6b90-683e-9f83-0dc355252cb9}" - testing.go:149: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.416Z [INFO] server-69672c51-c090-7959-3b75-caa4251589da: aborting pipeline replication: peer="{Voter server-4caa6cad-6b90-683e-9f83-0dc355252cb9 4caa6cad-6b90-683e-9f83-0dc355252cb9}" -testLoggerAdapter verbose: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.416Z [INFO] server-69672c51-c090-7959-3b75-caa4251589da: aborting pipeline replication: peer="{Voter server-dccc9681-70a5-65b1-5ad4-136537c657c6 dccc9681-70a5-65b1-5ad4-136537c657c6}" - testing.go:149: server-69672c51-c090-7959-3b75-caa4251589da: 2024-09-03T21:34:30.416Z [INFO] server-69672c51-c090-7959-3b75-caa4251589da: aborting pipeline replication: peer="{Voter server-dccc9681-70a5-65b1-5ad4-136537c657c6 dccc9681-70a5-65b1-5ad4-136537c657c6}" -testLoggerAdapter verbose: server-edfcba5d-710c-d8ec-160c-ff854b1c4957: 2024-09-03T21:34:30.417Z [INFO] server-edfcba5d-710c-d8ec-160c-ff854b1c4957: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-edfcba5d-710c-d8ec-160c-ff854b1c4957 Address:edfcba5d-710c-d8ec-160c-ff854b1c4957} {Suffrage:Voter ID:server-bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01 Address:bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01} {Suffrage:Voter ID:server-8a2e283e-e8f6-4812-3e83-8c2463e83db2 Address:8a2e283e-e8f6-4812-3e83-8c2463e83db2}]" - testing.go:149: server-edfcba5d-710c-d8ec-160c-ff854b1c4957: 2024-09-03T21:34:30.417Z [INFO] server-edfcba5d-710c-d8ec-160c-ff854b1c4957: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-edfcba5d-710c-d8ec-160c-ff854b1c4957 Address:edfcba5d-710c-d8ec-160c-ff854b1c4957} {Suffrage:Voter ID:server-bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01 Address:bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01} {Suffrage:Voter ID:server-8a2e283e-e8f6-4812-3e83-8c2463e83db2 Address:8a2e283e-e8f6-4812-3e83-8c2463e83db2}]" -testLoggerAdapter verbose: server-bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01: 2024-09-03T21:34:30.417Z [INFO] server-bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-edfcba5d-710c-d8ec-160c-ff854b1c4957 Address:edfcba5d-710c-d8ec-160c-ff854b1c4957} {Suffrage:Voter ID:server-bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01 Address:bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01} {Suffrage:Voter ID:server-8a2e283e-e8f6-4812-3e83-8c2463e83db2 Address:8a2e283e-e8f6-4812-3e83-8c2463e83db2}]" - testing.go:149: server-bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01: 2024-09-03T21:34:30.417Z [INFO] server-bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-edfcba5d-710c-d8ec-160c-ff854b1c4957 Address:edfcba5d-710c-d8ec-160c-ff854b1c4957} {Suffrage:Voter ID:server-bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01 Address:bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01} {Suffrage:Voter ID:server-8a2e283e-e8f6-4812-3e83-8c2463e83db2 Address:8a2e283e-e8f6-4812-3e83-8c2463e83db2}]" -testLoggerAdapter verbose: server-edfcba5d-710c-d8ec-160c-ff854b1c4957: 2024-09-03T21:34:30.417Z [INFO] server-edfcba5d-710c-d8ec-160c-ff854b1c4957: entering follower state: follower="Node at edfcba5d-710c-d8ec-160c-ff854b1c4957 [Follower]" leader= - testing.go:149: server-edfcba5d-710c-d8ec-160c-ff854b1c4957: 2024-09-03T21:34:30.417Z [INFO] server-edfcba5d-710c-d8ec-160c-ff854b1c4957: entering follower state: follower="Node at edfcba5d-710c-d8ec-160c-ff854b1c4957 [Follower]" leader= -testLoggerAdapter verbose: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.417Z [INFO] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-edfcba5d-710c-d8ec-160c-ff854b1c4957 Address:edfcba5d-710c-d8ec-160c-ff854b1c4957} {Suffrage:Voter ID:server-bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01 Address:bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01} {Suffrage:Voter ID:server-8a2e283e-e8f6-4812-3e83-8c2463e83db2 Address:8a2e283e-e8f6-4812-3e83-8c2463e83db2}]" - testing.go:149: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.417Z [INFO] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-edfcba5d-710c-d8ec-160c-ff854b1c4957 Address:edfcba5d-710c-d8ec-160c-ff854b1c4957} {Suffrage:Voter ID:server-bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01 Address:bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01} {Suffrage:Voter ID:server-8a2e283e-e8f6-4812-3e83-8c2463e83db2 Address:8a2e283e-e8f6-4812-3e83-8c2463e83db2}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:30.417Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:30.417Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01: 2024-09-03T21:34:30.417Z [INFO] server-bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01: entering follower state: follower="Node at bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01 [Follower]" leader= - testing.go:149: server-bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01: 2024-09-03T21:34:30.417Z [INFO] server-bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01: entering follower state: follower="Node at bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01 [Follower]" leader= -testLoggerAdapter verbose: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.417Z [INFO] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: entering follower state: follower="Node at 8a2e283e-e8f6-4812-3e83-8c2463e83db2 [Follower]" leader= - testing.go:149: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.417Z [INFO] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: entering follower state: follower="Node at 8a2e283e-e8f6-4812-3e83-8c2463e83db2 [Follower]" leader= -testLoggerAdapter verbose: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.476Z [WARN] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.476Z [WARN] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.476Z [INFO] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: entering candidate state: node="Node at 8a2e283e-e8f6-4812-3e83-8c2463e83db2 [Candidate]" term=2 - testing.go:149: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.476Z [INFO] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: entering candidate state: node="Node at 8a2e283e-e8f6-4812-3e83-8c2463e83db2 [Candidate]" term=2 -testLoggerAdapter verbose: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.476Z [INFO] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: election won: tally=2 - testing.go:149: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.476Z [INFO] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: election won: tally=2 -testLoggerAdapter verbose: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.477Z [INFO] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: entering leader state: leader="Node at 8a2e283e-e8f6-4812-3e83-8c2463e83db2 [Leader]" - testing.go:149: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.477Z [INFO] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: entering leader state: leader="Node at 8a2e283e-e8f6-4812-3e83-8c2463e83db2 [Leader]" -testLoggerAdapter verbose: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.477Z [INFO] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: added peer, starting replication: peer=server-edfcba5d-710c-d8ec-160c-ff854b1c4957 - testing.go:149: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.477Z [INFO] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: added peer, starting replication: peer=server-edfcba5d-710c-d8ec-160c-ff854b1c4957 -testLoggerAdapter verbose: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.477Z [INFO] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: added peer, starting replication: peer=server-bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01 - testing.go:149: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.477Z [INFO] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: added peer, starting replication: peer=server-bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01 -testLoggerAdapter verbose: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.477Z [INFO] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: pipelining replication: peer="{Voter server-edfcba5d-710c-d8ec-160c-ff854b1c4957 edfcba5d-710c-d8ec-160c-ff854b1c4957}" - testing.go:149: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.477Z [INFO] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: pipelining replication: peer="{Voter server-edfcba5d-710c-d8ec-160c-ff854b1c4957 edfcba5d-710c-d8ec-160c-ff854b1c4957}" -testLoggerAdapter verbose: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.477Z [INFO] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: pipelining replication: peer="{Voter server-bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01 bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01}" - testing.go:149: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.477Z [INFO] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: pipelining replication: peer="{Voter server-bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01 bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:30.582Z [INFO] stable state for Leader reached at 2024-09-03 21:34:30.477059489 +0000 UTC m=+11.523716638 (1 nodes), 59.760097ms from start of poll, 60.129966ms from cluster start. Timeout at 2024-09-03 21:34:30.582092186 +0000 UTC m=+11.628749319, 105.032681ms after stability - testing.go:149: cluster: 2024-09-03T21:34:30.582Z [INFO] stable state for Leader reached at 2024-09-03 21:34:30.477059489 +0000 UTC m=+11.523716638 (1 nodes), 59.760097ms from start of poll, 60.129966ms from cluster start. Timeout at 2024-09-03 21:34:30.582092186 +0000 UTC m=+11.628749319, 105.032681ms after stability -testLoggerAdapter verbose: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.582Z [INFO] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: starting snapshot up to: index=12 - testing.go:149: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.582Z [INFO] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: starting snapshot up to: index=12 - testing.go:153: 2024-09-03T21:34:30.582Z [INFO] creating new snapshot: path=/tmp/raft045529580/snapshots/2-12-1725399270582.tmp -testLoggerAdapter verbose: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.583Z [INFO] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: snapshot complete up to: index=12 - testing.go:149: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.583Z [INFO] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: snapshot complete up to: index=12 - testing.go:153: 2024-09-03T21:34:30.583Z [INFO] creating new snapshot: path=/tmp/raft045529580/snapshots/2-113-1725399270583.tmp -testLoggerAdapter verbose: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.583Z [INFO] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: copied to local snapshot: bytes=71 - testing.go:149: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.583Z [INFO] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.583Z [INFO] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: restored user snapshot: index=1 - testing.go:149: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.583Z [INFO] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: restored user snapshot: index=1 -testLoggerAdapter verbose: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.583Z [ERROR] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: failed to get log: index=23 error="log not found" - testing.go:149: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.583Z [ERROR] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.583Z [INFO] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: aborting pipeline replication: peer="{Voter server-bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01 bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01}" - testing.go:149: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.583Z [INFO] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: aborting pipeline replication: peer="{Voter server-bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01 bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01}" -testLoggerAdapter verbose: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.583Z [ERROR] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: failed to get log: index=23 error="log not found" - testing.go:149: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.583Z [ERROR] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.584Z [INFO] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: aborting pipeline replication: peer="{Voter server-edfcba5d-710c-d8ec-160c-ff854b1c4957 edfcba5d-710c-d8ec-160c-ff854b1c4957}" - testing.go:149: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.584Z [INFO] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: aborting pipeline replication: peer="{Voter server-edfcba5d-710c-d8ec-160c-ff854b1c4957 edfcba5d-710c-d8ec-160c-ff854b1c4957}" -testLoggerAdapter verbose: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.593Z [ERROR] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: failed to get log: index=23 error="log not found" - testing.go:149: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.593Z [ERROR] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: failed to get log: index=23 error="log not found" - testing.go:153: 2024-09-03T21:34:30.593Z [INFO] creating new snapshot: path=/tmp/raft360838968/snapshots/2-113-1725399270593.tmp -testLoggerAdapter verbose: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.593Z [ERROR] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-edfcba5d-710c-d8ec-160c-ff854b1c4957: 2024-09-03T21:34:30.593Z [INFO] server-edfcba5d-710c-d8ec-160c-ff854b1c4957: copied to local snapshot: bytes=71 - testing.go:149: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.593Z [ERROR] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: failed to get log: index=23 error="log not found" - testing.go:149: server-edfcba5d-710c-d8ec-160c-ff854b1c4957: 2024-09-03T21:34:30.593Z [INFO] server-edfcba5d-710c-d8ec-160c-ff854b1c4957: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-edfcba5d-710c-d8ec-160c-ff854b1c4957: 2024-09-03T21:34:30.594Z [INFO] server-edfcba5d-710c-d8ec-160c-ff854b1c4957: Installed remote snapshot - testing.go:149: server-edfcba5d-710c-d8ec-160c-ff854b1c4957: 2024-09-03T21:34:30.594Z [INFO] server-edfcba5d-710c-d8ec-160c-ff854b1c4957: Installed remote snapshot -testLoggerAdapter verbose: server-edfcba5d-710c-d8ec-160c-ff854b1c4957: 2024-09-03T21:34:30.594Z [WARN] server-edfcba5d-710c-d8ec-160c-ff854b1c4957: skipping application of old log: index=22 - testing.go:149: server-edfcba5d-710c-d8ec-160c-ff854b1c4957: 2024-09-03T21:34:30.594Z [WARN] server-edfcba5d-710c-d8ec-160c-ff854b1c4957: skipping application of old log: index=22 -testLoggerAdapter verbose: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.594Z [INFO] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: pipelining replication: peer="{Voter server-edfcba5d-710c-d8ec-160c-ff854b1c4957 edfcba5d-710c-d8ec-160c-ff854b1c4957}" - testing.go:149: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.594Z [INFO] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: pipelining replication: peer="{Voter server-edfcba5d-710c-d8ec-160c-ff854b1c4957 edfcba5d-710c-d8ec-160c-ff854b1c4957}" - testing.go:153: 2024-09-03T21:34:30.594Z [INFO] creating new snapshot: path=/tmp/raft261731114/snapshots/2-113-1725399270594.tmp -testLoggerAdapter verbose: server-bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01: 2024-09-03T21:34:30.594Z [INFO] server-bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01: copied to local snapshot: bytes=71 - testing.go:149: server-bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01: 2024-09-03T21:34:30.594Z [INFO] server-bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01: 2024-09-03T21:34:30.595Z [INFO] server-bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01: Installed remote snapshot - testing.go:149: server-bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01: 2024-09-03T21:34:30.595Z [INFO] server-bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01: Installed remote snapshot -testLoggerAdapter verbose: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.595Z [INFO] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: pipelining replication: peer="{Voter server-bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01 bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01}" - testing.go:149: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.595Z [INFO] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: pipelining replication: peer="{Voter server-bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01 bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01}" -testLoggerAdapter verbose: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.611Z [INFO] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: aborting pipeline replication: peer="{Voter server-edfcba5d-710c-d8ec-160c-ff854b1c4957 edfcba5d-710c-d8ec-160c-ff854b1c4957}" - testing.go:149: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.611Z [INFO] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: aborting pipeline replication: peer="{Voter server-edfcba5d-710c-d8ec-160c-ff854b1c4957 edfcba5d-710c-d8ec-160c-ff854b1c4957}" -testLoggerAdapter verbose: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.611Z [INFO] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: aborting pipeline replication: peer="{Voter server-bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01 bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01}" - testing.go:149: server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: 2024-09-03T21:34:30.611Z [INFO] server-8a2e283e-e8f6-4812-3e83-8c2463e83db2: aborting pipeline replication: peer="{Voter server-bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01 bc3ad72e-770e-4b4c-dbf2-e6a2c7ff8d01}" -testLoggerAdapter verbose: server-873f3b2f-b1cb-1bb3-b771-4f5734ac35c8: 2024-09-03T21:34:30.613Z [INFO] server-873f3b2f-b1cb-1bb3-b771-4f5734ac35c8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-873f3b2f-b1cb-1bb3-b771-4f5734ac35c8 Address:873f3b2f-b1cb-1bb3-b771-4f5734ac35c8} {Suffrage:Voter ID:server-e123b81f-3697-1f35-665f-a1a10c831d8c Address:e123b81f-3697-1f35-665f-a1a10c831d8c} {Suffrage:Voter ID:server-7a0d4876-814a-0720-778d-195f118db6cb Address:7a0d4876-814a-0720-778d-195f118db6cb}]" - testing.go:149: server-873f3b2f-b1cb-1bb3-b771-4f5734ac35c8: 2024-09-03T21:34:30.613Z [INFO] server-873f3b2f-b1cb-1bb3-b771-4f5734ac35c8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-873f3b2f-b1cb-1bb3-b771-4f5734ac35c8 Address:873f3b2f-b1cb-1bb3-b771-4f5734ac35c8} {Suffrage:Voter ID:server-e123b81f-3697-1f35-665f-a1a10c831d8c Address:e123b81f-3697-1f35-665f-a1a10c831d8c} {Suffrage:Voter ID:server-7a0d4876-814a-0720-778d-195f118db6cb Address:7a0d4876-814a-0720-778d-195f118db6cb}]" -testLoggerAdapter verbose: server-873f3b2f-b1cb-1bb3-b771-4f5734ac35c8: 2024-09-03T21:34:30.613Z [INFO] server-873f3b2f-b1cb-1bb3-b771-4f5734ac35c8: entering follower state: follower="Node at 873f3b2f-b1cb-1bb3-b771-4f5734ac35c8 [Follower]" leader= -testLoggerAdapter verbose: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.613Z [INFO] server-e123b81f-3697-1f35-665f-a1a10c831d8c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-873f3b2f-b1cb-1bb3-b771-4f5734ac35c8 Address:873f3b2f-b1cb-1bb3-b771-4f5734ac35c8} {Suffrage:Voter ID:server-e123b81f-3697-1f35-665f-a1a10c831d8c Address:e123b81f-3697-1f35-665f-a1a10c831d8c} {Suffrage:Voter ID:server-7a0d4876-814a-0720-778d-195f118db6cb Address:7a0d4876-814a-0720-778d-195f118db6cb}]" - testing.go:149: server-873f3b2f-b1cb-1bb3-b771-4f5734ac35c8: 2024-09-03T21:34:30.613Z [INFO] server-873f3b2f-b1cb-1bb3-b771-4f5734ac35c8: entering follower state: follower="Node at 873f3b2f-b1cb-1bb3-b771-4f5734ac35c8 [Follower]" leader= - testing.go:149: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.613Z [INFO] server-e123b81f-3697-1f35-665f-a1a10c831d8c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-873f3b2f-b1cb-1bb3-b771-4f5734ac35c8 Address:873f3b2f-b1cb-1bb3-b771-4f5734ac35c8} {Suffrage:Voter ID:server-e123b81f-3697-1f35-665f-a1a10c831d8c Address:e123b81f-3697-1f35-665f-a1a10c831d8c} {Suffrage:Voter ID:server-7a0d4876-814a-0720-778d-195f118db6cb Address:7a0d4876-814a-0720-778d-195f118db6cb}]" -testLoggerAdapter verbose: server-7a0d4876-814a-0720-778d-195f118db6cb: 2024-09-03T21:34:30.614Z [INFO] server-7a0d4876-814a-0720-778d-195f118db6cb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-873f3b2f-b1cb-1bb3-b771-4f5734ac35c8 Address:873f3b2f-b1cb-1bb3-b771-4f5734ac35c8} {Suffrage:Voter ID:server-e123b81f-3697-1f35-665f-a1a10c831d8c Address:e123b81f-3697-1f35-665f-a1a10c831d8c} {Suffrage:Voter ID:server-7a0d4876-814a-0720-778d-195f118db6cb Address:7a0d4876-814a-0720-778d-195f118db6cb}]" - testing.go:149: server-7a0d4876-814a-0720-778d-195f118db6cb: 2024-09-03T21:34:30.614Z [INFO] server-7a0d4876-814a-0720-778d-195f118db6cb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-873f3b2f-b1cb-1bb3-b771-4f5734ac35c8 Address:873f3b2f-b1cb-1bb3-b771-4f5734ac35c8} {Suffrage:Voter ID:server-e123b81f-3697-1f35-665f-a1a10c831d8c Address:e123b81f-3697-1f35-665f-a1a10c831d8c} {Suffrage:Voter ID:server-7a0d4876-814a-0720-778d-195f118db6cb Address:7a0d4876-814a-0720-778d-195f118db6cb}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:30.614Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:30.614Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-7a0d4876-814a-0720-778d-195f118db6cb: 2024-09-03T21:34:30.614Z [INFO] server-7a0d4876-814a-0720-778d-195f118db6cb: entering follower state: follower="Node at 7a0d4876-814a-0720-778d-195f118db6cb [Follower]" leader= - testing.go:149: server-7a0d4876-814a-0720-778d-195f118db6cb: 2024-09-03T21:34:30.614Z [INFO] server-7a0d4876-814a-0720-778d-195f118db6cb: entering follower state: follower="Node at 7a0d4876-814a-0720-778d-195f118db6cb [Follower]" leader= -testLoggerAdapter verbose: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.614Z [INFO] server-e123b81f-3697-1f35-665f-a1a10c831d8c: entering follower state: follower="Node at e123b81f-3697-1f35-665f-a1a10c831d8c [Follower]" leader= - testing.go:149: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.614Z [INFO] server-e123b81f-3697-1f35-665f-a1a10c831d8c: entering follower state: follower="Node at e123b81f-3697-1f35-665f-a1a10c831d8c [Follower]" leader= -testLoggerAdapter verbose: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.669Z [WARN] server-e123b81f-3697-1f35-665f-a1a10c831d8c: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.669Z [WARN] server-e123b81f-3697-1f35-665f-a1a10c831d8c: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.669Z [INFO] server-e123b81f-3697-1f35-665f-a1a10c831d8c: entering candidate state: node="Node at e123b81f-3697-1f35-665f-a1a10c831d8c [Candidate]" term=2 - testing.go:149: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.669Z [INFO] server-e123b81f-3697-1f35-665f-a1a10c831d8c: entering candidate state: node="Node at e123b81f-3697-1f35-665f-a1a10c831d8c [Candidate]" term=2 -testLoggerAdapter verbose: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.669Z [INFO] server-e123b81f-3697-1f35-665f-a1a10c831d8c: election won: tally=2 - testing.go:149: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.669Z [INFO] server-e123b81f-3697-1f35-665f-a1a10c831d8c: election won: tally=2 -testLoggerAdapter verbose: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.669Z [INFO] server-e123b81f-3697-1f35-665f-a1a10c831d8c: entering leader state: leader="Node at e123b81f-3697-1f35-665f-a1a10c831d8c [Leader]" - testing.go:149: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.669Z [INFO] server-e123b81f-3697-1f35-665f-a1a10c831d8c: entering leader state: leader="Node at e123b81f-3697-1f35-665f-a1a10c831d8c [Leader]" -testLoggerAdapter verbose: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.669Z [INFO] server-e123b81f-3697-1f35-665f-a1a10c831d8c: added peer, starting replication: peer=server-873f3b2f-b1cb-1bb3-b771-4f5734ac35c8 - testing.go:149: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.669Z [INFO] server-e123b81f-3697-1f35-665f-a1a10c831d8c: added peer, starting replication: peer=server-873f3b2f-b1cb-1bb3-b771-4f5734ac35c8 -testLoggerAdapter verbose: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.669Z [INFO] server-e123b81f-3697-1f35-665f-a1a10c831d8c: added peer, starting replication: peer=server-7a0d4876-814a-0720-778d-195f118db6cb - testing.go:149: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.669Z [INFO] server-e123b81f-3697-1f35-665f-a1a10c831d8c: added peer, starting replication: peer=server-7a0d4876-814a-0720-778d-195f118db6cb -testLoggerAdapter verbose: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.669Z [INFO] server-e123b81f-3697-1f35-665f-a1a10c831d8c: pipelining replication: peer="{Voter server-7a0d4876-814a-0720-778d-195f118db6cb 7a0d4876-814a-0720-778d-195f118db6cb}" - testing.go:149: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.669Z [INFO] server-e123b81f-3697-1f35-665f-a1a10c831d8c: pipelining replication: peer="{Voter server-7a0d4876-814a-0720-778d-195f118db6cb 7a0d4876-814a-0720-778d-195f118db6cb}" -testLoggerAdapter verbose: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.669Z [INFO] server-e123b81f-3697-1f35-665f-a1a10c831d8c: pipelining replication: peer="{Voter server-873f3b2f-b1cb-1bb3-b771-4f5734ac35c8 873f3b2f-b1cb-1bb3-b771-4f5734ac35c8}" - testing.go:149: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.669Z [INFO] server-e123b81f-3697-1f35-665f-a1a10c831d8c: pipelining replication: peer="{Voter server-873f3b2f-b1cb-1bb3-b771-4f5734ac35c8 873f3b2f-b1cb-1bb3-b771-4f5734ac35c8}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:30.774Z [INFO] stable state for Leader reached at 2024-09-03 21:34:30.669826475 +0000 UTC m=+11.716483608 (1 nodes), 55.529015ms from start of poll, 57.833357ms from cluster start. Timeout at 2024-09-03 21:34:30.774923093 +0000 UTC m=+11.821580244, 105.096636ms after stability - testing.go:149: cluster: 2024-09-03T21:34:30.774Z [INFO] stable state for Leader reached at 2024-09-03 21:34:30.669826475 +0000 UTC m=+11.716483608 (1 nodes), 55.529015ms from start of poll, 57.833357ms from cluster start. Timeout at 2024-09-03 21:34:30.774923093 +0000 UTC m=+11.821580244, 105.096636ms after stability -testLoggerAdapter verbose: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.775Z [INFO] server-e123b81f-3697-1f35-665f-a1a10c831d8c: starting snapshot up to: index=12 - testing.go:149: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.775Z [INFO] server-e123b81f-3697-1f35-665f-a1a10c831d8c: starting snapshot up to: index=12 - testing.go:153: 2024-09-03T21:34:30.775Z [INFO] creating new snapshot: path=/tmp/raft868343776/snapshots/2-12-1725399270775.tmp -testLoggerAdapter verbose: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.775Z [INFO] server-e123b81f-3697-1f35-665f-a1a10c831d8c: snapshot complete up to: index=12 - testing.go:149: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.775Z [INFO] server-e123b81f-3697-1f35-665f-a1a10c831d8c: snapshot complete up to: index=12 - testing.go:153: 2024-09-03T21:34:30.776Z [INFO] creating new snapshot: path=/tmp/raft868343776/snapshots/2-1013-1725399270776.tmp -testLoggerAdapter verbose: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.776Z [INFO] server-e123b81f-3697-1f35-665f-a1a10c831d8c: copied to local snapshot: bytes=71 - testing.go:149: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.776Z [INFO] server-e123b81f-3697-1f35-665f-a1a10c831d8c: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.776Z [INFO] server-e123b81f-3697-1f35-665f-a1a10c831d8c: restored user snapshot: index=1 - testing.go:149: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.776Z [INFO] server-e123b81f-3697-1f35-665f-a1a10c831d8c: restored user snapshot: index=1 -testLoggerAdapter verbose: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.776Z [ERROR] server-e123b81f-3697-1f35-665f-a1a10c831d8c: failed to get log: index=23 error="log not found" - testing.go:149: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.776Z [ERROR] server-e123b81f-3697-1f35-665f-a1a10c831d8c: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.776Z [INFO] server-e123b81f-3697-1f35-665f-a1a10c831d8c: aborting pipeline replication: peer="{Voter server-7a0d4876-814a-0720-778d-195f118db6cb 7a0d4876-814a-0720-778d-195f118db6cb}" - testing.go:149: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.776Z [INFO] server-e123b81f-3697-1f35-665f-a1a10c831d8c: aborting pipeline replication: peer="{Voter server-7a0d4876-814a-0720-778d-195f118db6cb 7a0d4876-814a-0720-778d-195f118db6cb}" -testLoggerAdapter verbose: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.776Z [ERROR] server-e123b81f-3697-1f35-665f-a1a10c831d8c: failed to get log: index=23 error="log not found" - testing.go:149: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.776Z [ERROR] server-e123b81f-3697-1f35-665f-a1a10c831d8c: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.776Z [INFO] server-e123b81f-3697-1f35-665f-a1a10c831d8c: aborting pipeline replication: peer="{Voter server-873f3b2f-b1cb-1bb3-b771-4f5734ac35c8 873f3b2f-b1cb-1bb3-b771-4f5734ac35c8}" - testing.go:149: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.776Z [INFO] server-e123b81f-3697-1f35-665f-a1a10c831d8c: aborting pipeline replication: peer="{Voter server-873f3b2f-b1cb-1bb3-b771-4f5734ac35c8 873f3b2f-b1cb-1bb3-b771-4f5734ac35c8}" -testLoggerAdapter verbose: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.782Z [ERROR] server-e123b81f-3697-1f35-665f-a1a10c831d8c: failed to get log: index=23 error="log not found" - testing.go:149: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.782Z [ERROR] server-e123b81f-3697-1f35-665f-a1a10c831d8c: failed to get log: index=23 error="log not found" - testing.go:153: 2024-09-03T21:34:30.782Z [INFO] creating new snapshot: path=/tmp/raft038216722/snapshots/2-1013-1725399270782.tmp -testLoggerAdapter verbose: server-7a0d4876-814a-0720-778d-195f118db6cb: 2024-09-03T21:34:30.782Z [INFO] server-7a0d4876-814a-0720-778d-195f118db6cb: copied to local snapshot: bytes=71 - testing.go:149: server-7a0d4876-814a-0720-778d-195f118db6cb: 2024-09-03T21:34:30.782Z [INFO] server-7a0d4876-814a-0720-778d-195f118db6cb: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-7a0d4876-814a-0720-778d-195f118db6cb: 2024-09-03T21:34:30.783Z [INFO] server-7a0d4876-814a-0720-778d-195f118db6cb: Installed remote snapshot - testing.go:149: server-7a0d4876-814a-0720-778d-195f118db6cb: 2024-09-03T21:34:30.783Z [INFO] server-7a0d4876-814a-0720-778d-195f118db6cb: Installed remote snapshot -testLoggerAdapter verbose: server-7a0d4876-814a-0720-778d-195f118db6cb: 2024-09-03T21:34:30.783Z [WARN] server-7a0d4876-814a-0720-778d-195f118db6cb: skipping application of old log: index=22 - testing.go:149: server-7a0d4876-814a-0720-778d-195f118db6cb: 2024-09-03T21:34:30.783Z [WARN] server-7a0d4876-814a-0720-778d-195f118db6cb: skipping application of old log: index=22 -testLoggerAdapter verbose: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.783Z [INFO] server-e123b81f-3697-1f35-665f-a1a10c831d8c: pipelining replication: peer="{Voter server-7a0d4876-814a-0720-778d-195f118db6cb 7a0d4876-814a-0720-778d-195f118db6cb}" - testing.go:149: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.783Z [INFO] server-e123b81f-3697-1f35-665f-a1a10c831d8c: pipelining replication: peer="{Voter server-7a0d4876-814a-0720-778d-195f118db6cb 7a0d4876-814a-0720-778d-195f118db6cb}" -testLoggerAdapter verbose: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.784Z [ERROR] server-e123b81f-3697-1f35-665f-a1a10c831d8c: failed to get log: index=23 error="log not found" - testing.go:149: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.784Z [ERROR] server-e123b81f-3697-1f35-665f-a1a10c831d8c: failed to get log: index=23 error="log not found" - testing.go:153: 2024-09-03T21:34:30.785Z [INFO] creating new snapshot: path=/tmp/raft132404222/snapshots/2-1013-1725399270785.tmp -testLoggerAdapter verbose: server-873f3b2f-b1cb-1bb3-b771-4f5734ac35c8: 2024-09-03T21:34:30.785Z [INFO] server-873f3b2f-b1cb-1bb3-b771-4f5734ac35c8: copied to local snapshot: bytes=71 - testing.go:149: server-873f3b2f-b1cb-1bb3-b771-4f5734ac35c8: 2024-09-03T21:34:30.785Z [INFO] server-873f3b2f-b1cb-1bb3-b771-4f5734ac35c8: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-873f3b2f-b1cb-1bb3-b771-4f5734ac35c8: 2024-09-03T21:34:30.785Z [INFO] server-873f3b2f-b1cb-1bb3-b771-4f5734ac35c8: Installed remote snapshot - testing.go:149: server-873f3b2f-b1cb-1bb3-b771-4f5734ac35c8: 2024-09-03T21:34:30.785Z [INFO] server-873f3b2f-b1cb-1bb3-b771-4f5734ac35c8: Installed remote snapshot -testLoggerAdapter verbose: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.785Z [INFO] server-e123b81f-3697-1f35-665f-a1a10c831d8c: pipelining replication: peer="{Voter server-873f3b2f-b1cb-1bb3-b771-4f5734ac35c8 873f3b2f-b1cb-1bb3-b771-4f5734ac35c8}" - testing.go:149: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.785Z [INFO] server-e123b81f-3697-1f35-665f-a1a10c831d8c: pipelining replication: peer="{Voter server-873f3b2f-b1cb-1bb3-b771-4f5734ac35c8 873f3b2f-b1cb-1bb3-b771-4f5734ac35c8}" -testLoggerAdapter verbose: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.799Z [INFO] server-e123b81f-3697-1f35-665f-a1a10c831d8c: aborting pipeline replication: peer="{Voter server-7a0d4876-814a-0720-778d-195f118db6cb 7a0d4876-814a-0720-778d-195f118db6cb}" - testing.go:149: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.799Z [INFO] server-e123b81f-3697-1f35-665f-a1a10c831d8c: aborting pipeline replication: peer="{Voter server-7a0d4876-814a-0720-778d-195f118db6cb 7a0d4876-814a-0720-778d-195f118db6cb}" -testLoggerAdapter verbose: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.799Z [INFO] server-e123b81f-3697-1f35-665f-a1a10c831d8c: aborting pipeline replication: peer="{Voter server-873f3b2f-b1cb-1bb3-b771-4f5734ac35c8 873f3b2f-b1cb-1bb3-b771-4f5734ac35c8}" - testing.go:149: server-e123b81f-3697-1f35-665f-a1a10c831d8c: 2024-09-03T21:34:30.799Z [INFO] server-e123b81f-3697-1f35-665f-a1a10c831d8c: aborting pipeline replication: peer="{Voter server-873f3b2f-b1cb-1bb3-b771-4f5734ac35c8 873f3b2f-b1cb-1bb3-b771-4f5734ac35c8}" -testLoggerAdapter verbose: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.800Z [INFO] server-df1e9c40-9cd3-a959-5992-292be6374c73: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-df1e9c40-9cd3-a959-5992-292be6374c73 Address:df1e9c40-9cd3-a959-5992-292be6374c73} {Suffrage:Voter ID:server-45019901-7e48-49dd-213b-ec2501ae4040 Address:45019901-7e48-49dd-213b-ec2501ae4040} {Suffrage:Voter ID:server-ddb252ef-e2aa-db0b-b073-270ff1d6466f Address:ddb252ef-e2aa-db0b-b073-270ff1d6466f}]" - testing.go:149: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.800Z [INFO] server-df1e9c40-9cd3-a959-5992-292be6374c73: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-df1e9c40-9cd3-a959-5992-292be6374c73 Address:df1e9c40-9cd3-a959-5992-292be6374c73} {Suffrage:Voter ID:server-45019901-7e48-49dd-213b-ec2501ae4040 Address:45019901-7e48-49dd-213b-ec2501ae4040} {Suffrage:Voter ID:server-ddb252ef-e2aa-db0b-b073-270ff1d6466f Address:ddb252ef-e2aa-db0b-b073-270ff1d6466f}]" -testLoggerAdapter verbose: server-45019901-7e48-49dd-213b-ec2501ae4040: 2024-09-03T21:34:30.800Z [INFO] server-45019901-7e48-49dd-213b-ec2501ae4040: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-df1e9c40-9cd3-a959-5992-292be6374c73 Address:df1e9c40-9cd3-a959-5992-292be6374c73} {Suffrage:Voter ID:server-45019901-7e48-49dd-213b-ec2501ae4040 Address:45019901-7e48-49dd-213b-ec2501ae4040} {Suffrage:Voter ID:server-ddb252ef-e2aa-db0b-b073-270ff1d6466f Address:ddb252ef-e2aa-db0b-b073-270ff1d6466f}]" - testing.go:149: server-45019901-7e48-49dd-213b-ec2501ae4040: 2024-09-03T21:34:30.800Z [INFO] server-45019901-7e48-49dd-213b-ec2501ae4040: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-df1e9c40-9cd3-a959-5992-292be6374c73 Address:df1e9c40-9cd3-a959-5992-292be6374c73} {Suffrage:Voter ID:server-45019901-7e48-49dd-213b-ec2501ae4040 Address:45019901-7e48-49dd-213b-ec2501ae4040} {Suffrage:Voter ID:server-ddb252ef-e2aa-db0b-b073-270ff1d6466f Address:ddb252ef-e2aa-db0b-b073-270ff1d6466f}]" -testLoggerAdapter verbose: server-ddb252ef-e2aa-db0b-b073-270ff1d6466f: 2024-09-03T21:34:30.800Z [INFO] server-ddb252ef-e2aa-db0b-b073-270ff1d6466f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-df1e9c40-9cd3-a959-5992-292be6374c73 Address:df1e9c40-9cd3-a959-5992-292be6374c73} {Suffrage:Voter ID:server-45019901-7e48-49dd-213b-ec2501ae4040 Address:45019901-7e48-49dd-213b-ec2501ae4040} {Suffrage:Voter ID:server-ddb252ef-e2aa-db0b-b073-270ff1d6466f Address:ddb252ef-e2aa-db0b-b073-270ff1d6466f}]" - testing.go:149: server-ddb252ef-e2aa-db0b-b073-270ff1d6466f: 2024-09-03T21:34:30.800Z [INFO] server-ddb252ef-e2aa-db0b-b073-270ff1d6466f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-df1e9c40-9cd3-a959-5992-292be6374c73 Address:df1e9c40-9cd3-a959-5992-292be6374c73} {Suffrage:Voter ID:server-45019901-7e48-49dd-213b-ec2501ae4040 Address:45019901-7e48-49dd-213b-ec2501ae4040} {Suffrage:Voter ID:server-ddb252ef-e2aa-db0b-b073-270ff1d6466f Address:ddb252ef-e2aa-db0b-b073-270ff1d6466f}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:30.800Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:30.800Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.800Z [INFO] server-df1e9c40-9cd3-a959-5992-292be6374c73: entering follower state: follower="Node at df1e9c40-9cd3-a959-5992-292be6374c73 [Follower]" leader= - testing.go:149: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.800Z [INFO] server-df1e9c40-9cd3-a959-5992-292be6374c73: entering follower state: follower="Node at df1e9c40-9cd3-a959-5992-292be6374c73 [Follower]" leader= -testLoggerAdapter verbose: server-45019901-7e48-49dd-213b-ec2501ae4040: 2024-09-03T21:34:30.800Z [INFO] server-45019901-7e48-49dd-213b-ec2501ae4040: entering follower state: follower="Node at 45019901-7e48-49dd-213b-ec2501ae4040 [Follower]" leader= - testing.go:149: server-45019901-7e48-49dd-213b-ec2501ae4040: 2024-09-03T21:34:30.800Z [INFO] server-45019901-7e48-49dd-213b-ec2501ae4040: entering follower state: follower="Node at 45019901-7e48-49dd-213b-ec2501ae4040 [Follower]" leader= -testLoggerAdapter verbose: server-ddb252ef-e2aa-db0b-b073-270ff1d6466f: 2024-09-03T21:34:30.800Z [INFO] server-ddb252ef-e2aa-db0b-b073-270ff1d6466f: entering follower state: follower="Node at ddb252ef-e2aa-db0b-b073-270ff1d6466f [Follower]" leader= - testing.go:149: server-ddb252ef-e2aa-db0b-b073-270ff1d6466f: 2024-09-03T21:34:30.800Z [INFO] server-ddb252ef-e2aa-db0b-b073-270ff1d6466f: entering follower state: follower="Node at ddb252ef-e2aa-db0b-b073-270ff1d6466f [Follower]" leader= -testLoggerAdapter verbose: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.857Z [WARN] server-df1e9c40-9cd3-a959-5992-292be6374c73: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.857Z [WARN] server-df1e9c40-9cd3-a959-5992-292be6374c73: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.857Z [INFO] server-df1e9c40-9cd3-a959-5992-292be6374c73: entering candidate state: node="Node at df1e9c40-9cd3-a959-5992-292be6374c73 [Candidate]" term=2 - testing.go:149: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.857Z [INFO] server-df1e9c40-9cd3-a959-5992-292be6374c73: entering candidate state: node="Node at df1e9c40-9cd3-a959-5992-292be6374c73 [Candidate]" term=2 -testLoggerAdapter verbose: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.857Z [INFO] server-df1e9c40-9cd3-a959-5992-292be6374c73: election won: tally=2 - testing.go:149: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.857Z [INFO] server-df1e9c40-9cd3-a959-5992-292be6374c73: election won: tally=2 -testLoggerAdapter verbose: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.857Z [INFO] server-df1e9c40-9cd3-a959-5992-292be6374c73: entering leader state: leader="Node at df1e9c40-9cd3-a959-5992-292be6374c73 [Leader]" - testing.go:149: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.857Z [INFO] server-df1e9c40-9cd3-a959-5992-292be6374c73: entering leader state: leader="Node at df1e9c40-9cd3-a959-5992-292be6374c73 [Leader]" -testLoggerAdapter verbose: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.857Z [INFO] server-df1e9c40-9cd3-a959-5992-292be6374c73: added peer, starting replication: peer=server-45019901-7e48-49dd-213b-ec2501ae4040 - testing.go:149: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.857Z [INFO] server-df1e9c40-9cd3-a959-5992-292be6374c73: added peer, starting replication: peer=server-45019901-7e48-49dd-213b-ec2501ae4040 -testLoggerAdapter verbose: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.857Z [INFO] server-df1e9c40-9cd3-a959-5992-292be6374c73: added peer, starting replication: peer=server-ddb252ef-e2aa-db0b-b073-270ff1d6466f - testing.go:149: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.857Z [INFO] server-df1e9c40-9cd3-a959-5992-292be6374c73: added peer, starting replication: peer=server-ddb252ef-e2aa-db0b-b073-270ff1d6466f -testLoggerAdapter verbose: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.857Z [INFO] server-df1e9c40-9cd3-a959-5992-292be6374c73: pipelining replication: peer="{Voter server-ddb252ef-e2aa-db0b-b073-270ff1d6466f ddb252ef-e2aa-db0b-b073-270ff1d6466f}" - testing.go:149: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.857Z [INFO] server-df1e9c40-9cd3-a959-5992-292be6374c73: pipelining replication: peer="{Voter server-ddb252ef-e2aa-db0b-b073-270ff1d6466f ddb252ef-e2aa-db0b-b073-270ff1d6466f}" -testLoggerAdapter verbose: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.858Z [INFO] server-df1e9c40-9cd3-a959-5992-292be6374c73: pipelining replication: peer="{Voter server-45019901-7e48-49dd-213b-ec2501ae4040 45019901-7e48-49dd-213b-ec2501ae4040}" - testing.go:149: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.858Z [INFO] server-df1e9c40-9cd3-a959-5992-292be6374c73: pipelining replication: peer="{Voter server-45019901-7e48-49dd-213b-ec2501ae4040 45019901-7e48-49dd-213b-ec2501ae4040}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:30.967Z [INFO] stable state for Leader reached at 2024-09-03 21:34:30.857831047 +0000 UTC m=+11.904488195 (1 nodes), 57.457589ms from start of poll, 57.806528ms from cluster start. Timeout at 2024-09-03 21:34:30.967058879 +0000 UTC m=+12.013716014, 109.227819ms after stability - testing.go:149: cluster: 2024-09-03T21:34:30.967Z [INFO] stable state for Leader reached at 2024-09-03 21:34:30.857831047 +0000 UTC m=+11.904488195 (1 nodes), 57.457589ms from start of poll, 57.806528ms from cluster start. Timeout at 2024-09-03 21:34:30.967058879 +0000 UTC m=+12.013716014, 109.227819ms after stability -testLoggerAdapter verbose: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.967Z [INFO] server-df1e9c40-9cd3-a959-5992-292be6374c73: starting snapshot up to: index=12 - testing.go:149: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.967Z [INFO] server-df1e9c40-9cd3-a959-5992-292be6374c73: starting snapshot up to: index=12 - testing.go:153: 2024-09-03T21:34:30.967Z [INFO] creating new snapshot: path=/tmp/raft485012756/snapshots/2-12-1725399270967.tmp -testLoggerAdapter verbose: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.967Z [INFO] server-df1e9c40-9cd3-a959-5992-292be6374c73: snapshot complete up to: index=12 - testing.go:149: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.967Z [INFO] server-df1e9c40-9cd3-a959-5992-292be6374c73: snapshot complete up to: index=12 - testing.go:153: 2024-09-03T21:34:30.968Z [INFO] creating new snapshot: path=/tmp/raft485012756/snapshots/2-10013-1725399270968.tmp -testLoggerAdapter verbose: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.968Z [INFO] server-df1e9c40-9cd3-a959-5992-292be6374c73: copied to local snapshot: bytes=71 - testing.go:149: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.968Z [INFO] server-df1e9c40-9cd3-a959-5992-292be6374c73: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.968Z [INFO] server-df1e9c40-9cd3-a959-5992-292be6374c73: restored user snapshot: index=1 - testing.go:149: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.968Z [INFO] server-df1e9c40-9cd3-a959-5992-292be6374c73: restored user snapshot: index=1 -testLoggerAdapter verbose: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.968Z [ERROR] server-df1e9c40-9cd3-a959-5992-292be6374c73: failed to get log: index=23 error="log not found" - testing.go:149: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.968Z [ERROR] server-df1e9c40-9cd3-a959-5992-292be6374c73: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.968Z [INFO] server-df1e9c40-9cd3-a959-5992-292be6374c73: aborting pipeline replication: peer="{Voter server-ddb252ef-e2aa-db0b-b073-270ff1d6466f ddb252ef-e2aa-db0b-b073-270ff1d6466f}" - testing.go:149: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.968Z [INFO] server-df1e9c40-9cd3-a959-5992-292be6374c73: aborting pipeline replication: peer="{Voter server-ddb252ef-e2aa-db0b-b073-270ff1d6466f ddb252ef-e2aa-db0b-b073-270ff1d6466f}" -testLoggerAdapter verbose: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.968Z [ERROR] server-df1e9c40-9cd3-a959-5992-292be6374c73: failed to get log: index=23 error="log not found" - testing.go:149: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.968Z [ERROR] server-df1e9c40-9cd3-a959-5992-292be6374c73: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.968Z [INFO] server-df1e9c40-9cd3-a959-5992-292be6374c73: aborting pipeline replication: peer="{Voter server-45019901-7e48-49dd-213b-ec2501ae4040 45019901-7e48-49dd-213b-ec2501ae4040}" - testing.go:149: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.968Z [INFO] server-df1e9c40-9cd3-a959-5992-292be6374c73: aborting pipeline replication: peer="{Voter server-45019901-7e48-49dd-213b-ec2501ae4040 45019901-7e48-49dd-213b-ec2501ae4040}" -testLoggerAdapter verbose: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.968Z [ERROR] server-df1e9c40-9cd3-a959-5992-292be6374c73: failed to get log: index=23 error="log not found" - testing.go:149: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.968Z [ERROR] server-df1e9c40-9cd3-a959-5992-292be6374c73: failed to get log: index=23 error="log not found" - testing.go:153: 2024-09-03T21:34:30.969Z [INFO] creating new snapshot: path=/tmp/raft418687334/snapshots/2-10013-1725399270969.tmp -testLoggerAdapter verbose: server-45019901-7e48-49dd-213b-ec2501ae4040: 2024-09-03T21:34:30.969Z [INFO] server-45019901-7e48-49dd-213b-ec2501ae4040: copied to local snapshot: bytes=71 - testing.go:149: server-45019901-7e48-49dd-213b-ec2501ae4040: 2024-09-03T21:34:30.969Z [INFO] server-45019901-7e48-49dd-213b-ec2501ae4040: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-45019901-7e48-49dd-213b-ec2501ae4040: 2024-09-03T21:34:30.969Z [INFO] server-45019901-7e48-49dd-213b-ec2501ae4040: Installed remote snapshot - testing.go:149: server-45019901-7e48-49dd-213b-ec2501ae4040: 2024-09-03T21:34:30.969Z [INFO] server-45019901-7e48-49dd-213b-ec2501ae4040: Installed remote snapshot -testLoggerAdapter verbose: server-45019901-7e48-49dd-213b-ec2501ae4040: 2024-09-03T21:34:30.969Z [WARN] server-45019901-7e48-49dd-213b-ec2501ae4040: skipping application of old log: index=22 - testing.go:149: server-45019901-7e48-49dd-213b-ec2501ae4040: 2024-09-03T21:34:30.969Z [WARN] server-45019901-7e48-49dd-213b-ec2501ae4040: skipping application of old log: index=22 -testLoggerAdapter verbose: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.969Z [INFO] server-df1e9c40-9cd3-a959-5992-292be6374c73: pipelining replication: peer="{Voter server-45019901-7e48-49dd-213b-ec2501ae4040 45019901-7e48-49dd-213b-ec2501ae4040}" - testing.go:149: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.969Z [INFO] server-df1e9c40-9cd3-a959-5992-292be6374c73: pipelining replication: peer="{Voter server-45019901-7e48-49dd-213b-ec2501ae4040 45019901-7e48-49dd-213b-ec2501ae4040}" -testLoggerAdapter verbose: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.978Z [ERROR] server-df1e9c40-9cd3-a959-5992-292be6374c73: failed to get log: index=23 error="log not found" - testing.go:149: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.978Z [ERROR] server-df1e9c40-9cd3-a959-5992-292be6374c73: failed to get log: index=23 error="log not found" - testing.go:153: 2024-09-03T21:34:30.980Z [INFO] creating new snapshot: path=/tmp/raft965958024/snapshots/2-10013-1725399270980.tmp -testLoggerAdapter verbose: server-ddb252ef-e2aa-db0b-b073-270ff1d6466f: 2024-09-03T21:34:30.980Z [INFO] server-ddb252ef-e2aa-db0b-b073-270ff1d6466f: copied to local snapshot: bytes=71 - testing.go:149: server-ddb252ef-e2aa-db0b-b073-270ff1d6466f: 2024-09-03T21:34:30.980Z [INFO] server-ddb252ef-e2aa-db0b-b073-270ff1d6466f: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-ddb252ef-e2aa-db0b-b073-270ff1d6466f: 2024-09-03T21:34:30.981Z [INFO] server-ddb252ef-e2aa-db0b-b073-270ff1d6466f: Installed remote snapshot - testing.go:149: server-ddb252ef-e2aa-db0b-b073-270ff1d6466f: 2024-09-03T21:34:30.981Z [INFO] server-ddb252ef-e2aa-db0b-b073-270ff1d6466f: Installed remote snapshot -testLoggerAdapter verbose: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.981Z [INFO] server-df1e9c40-9cd3-a959-5992-292be6374c73: pipelining replication: peer="{Voter server-ddb252ef-e2aa-db0b-b073-270ff1d6466f ddb252ef-e2aa-db0b-b073-270ff1d6466f}" - testing.go:149: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.981Z [INFO] server-df1e9c40-9cd3-a959-5992-292be6374c73: pipelining replication: peer="{Voter server-ddb252ef-e2aa-db0b-b073-270ff1d6466f ddb252ef-e2aa-db0b-b073-270ff1d6466f}" -testLoggerAdapter verbose: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.996Z [INFO] server-df1e9c40-9cd3-a959-5992-292be6374c73: aborting pipeline replication: peer="{Voter server-ddb252ef-e2aa-db0b-b073-270ff1d6466f ddb252ef-e2aa-db0b-b073-270ff1d6466f}" - testing.go:149: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.996Z [INFO] server-df1e9c40-9cd3-a959-5992-292be6374c73: aborting pipeline replication: peer="{Voter server-ddb252ef-e2aa-db0b-b073-270ff1d6466f ddb252ef-e2aa-db0b-b073-270ff1d6466f}" -testLoggerAdapter verbose: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.996Z [INFO] server-df1e9c40-9cd3-a959-5992-292be6374c73: aborting pipeline replication: peer="{Voter server-45019901-7e48-49dd-213b-ec2501ae4040 45019901-7e48-49dd-213b-ec2501ae4040}" - testing.go:149: server-df1e9c40-9cd3-a959-5992-292be6374c73: 2024-09-03T21:34:30.996Z [INFO] server-df1e9c40-9cd3-a959-5992-292be6374c73: aborting pipeline replication: peer="{Voter server-45019901-7e48-49dd-213b-ec2501ae4040 45019901-7e48-49dd-213b-ec2501ae4040}" ---- PASS: TestRaft_UserRestore (1.17s) +testLoggerAdapter verbose: server-f8811921-2735-d284-f23e-ce90eafcb10a: 2023-08-02T15:12:58.775Z [INFO] server-f8811921-2735-d284-f23e-ce90eafcb10a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f8811921-2735-d284-f23e-ce90eafcb10a Address:f8811921-2735-d284-f23e-ce90eafcb10a} {Suffrage:Voter ID:server-f25432d1-1355-b829-f9d5-1c675504ab50 Address:f25432d1-1355-b829-f9d5-1c675504ab50} {Suffrage:Voter ID:server-11a0d01c-4a46-e06a-e67b-844791818f99 Address:11a0d01c-4a46-e06a-e67b-844791818f99}]" + testing.go:149: server-f8811921-2735-d284-f23e-ce90eafcb10a: 2023-08-02T15:12:58.775Z [INFO] server-f8811921-2735-d284-f23e-ce90eafcb10a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f8811921-2735-d284-f23e-ce90eafcb10a Address:f8811921-2735-d284-f23e-ce90eafcb10a} {Suffrage:Voter ID:server-f25432d1-1355-b829-f9d5-1c675504ab50 Address:f25432d1-1355-b829-f9d5-1c675504ab50} {Suffrage:Voter ID:server-11a0d01c-4a46-e06a-e67b-844791818f99 Address:11a0d01c-4a46-e06a-e67b-844791818f99}]" +testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.775Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f8811921-2735-d284-f23e-ce90eafcb10a Address:f8811921-2735-d284-f23e-ce90eafcb10a} {Suffrage:Voter ID:server-f25432d1-1355-b829-f9d5-1c675504ab50 Address:f25432d1-1355-b829-f9d5-1c675504ab50} {Suffrage:Voter ID:server-11a0d01c-4a46-e06a-e67b-844791818f99 Address:11a0d01c-4a46-e06a-e67b-844791818f99}]" + testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.775Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f8811921-2735-d284-f23e-ce90eafcb10a Address:f8811921-2735-d284-f23e-ce90eafcb10a} {Suffrage:Voter ID:server-f25432d1-1355-b829-f9d5-1c675504ab50 Address:f25432d1-1355-b829-f9d5-1c675504ab50} {Suffrage:Voter ID:server-11a0d01c-4a46-e06a-e67b-844791818f99 Address:11a0d01c-4a46-e06a-e67b-844791818f99}]" +testLoggerAdapter verbose: server-11a0d01c-4a46-e06a-e67b-844791818f99: 2023-08-02T15:12:58.776Z [INFO] server-11a0d01c-4a46-e06a-e67b-844791818f99: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f8811921-2735-d284-f23e-ce90eafcb10a Address:f8811921-2735-d284-f23e-ce90eafcb10a} {Suffrage:Voter ID:server-f25432d1-1355-b829-f9d5-1c675504ab50 Address:f25432d1-1355-b829-f9d5-1c675504ab50} {Suffrage:Voter ID:server-11a0d01c-4a46-e06a-e67b-844791818f99 Address:11a0d01c-4a46-e06a-e67b-844791818f99}]" + testing.go:149: server-11a0d01c-4a46-e06a-e67b-844791818f99: 2023-08-02T15:12:58.776Z [INFO] server-11a0d01c-4a46-e06a-e67b-844791818f99: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f8811921-2735-d284-f23e-ce90eafcb10a Address:f8811921-2735-d284-f23e-ce90eafcb10a} {Suffrage:Voter ID:server-f25432d1-1355-b829-f9d5-1c675504ab50 Address:f25432d1-1355-b829-f9d5-1c675504ab50} {Suffrage:Voter ID:server-11a0d01c-4a46-e06a-e67b-844791818f99 Address:11a0d01c-4a46-e06a-e67b-844791818f99}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:58.776Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:12:58.776Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-f8811921-2735-d284-f23e-ce90eafcb10a: 2023-08-02T15:12:58.776Z [INFO] server-f8811921-2735-d284-f23e-ce90eafcb10a: entering follower state: follower="Node at f8811921-2735-d284-f23e-ce90eafcb10a [Follower]" leader= + testing.go:149: server-f8811921-2735-d284-f23e-ce90eafcb10a: 2023-08-02T15:12:58.776Z [INFO] server-f8811921-2735-d284-f23e-ce90eafcb10a: entering follower state: follower="Node at f8811921-2735-d284-f23e-ce90eafcb10a [Follower]" leader= +testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.776Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: entering follower state: follower="Node at f25432d1-1355-b829-f9d5-1c675504ab50 [Follower]" leader= + testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.776Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: entering follower state: follower="Node at f25432d1-1355-b829-f9d5-1c675504ab50 [Follower]" leader= +testLoggerAdapter verbose: server-11a0d01c-4a46-e06a-e67b-844791818f99: 2023-08-02T15:12:58.776Z [INFO] server-11a0d01c-4a46-e06a-e67b-844791818f99: entering follower state: follower="Node at 11a0d01c-4a46-e06a-e67b-844791818f99 [Follower]" leader= + testing.go:149: server-11a0d01c-4a46-e06a-e67b-844791818f99: 2023-08-02T15:12:58.776Z [INFO] server-11a0d01c-4a46-e06a-e67b-844791818f99: entering follower state: follower="Node at 11a0d01c-4a46-e06a-e67b-844791818f99 [Follower]" leader= +testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.834Z [WARN] server-f25432d1-1355-b829-f9d5-1c675504ab50: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.834Z [WARN] server-f25432d1-1355-b829-f9d5-1c675504ab50: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.834Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: entering candidate state: node="Node at f25432d1-1355-b829-f9d5-1c675504ab50 [Candidate]" term=2 + testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.834Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: entering candidate state: node="Node at f25432d1-1355-b829-f9d5-1c675504ab50 [Candidate]" term=2 +testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.835Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: election won: tally=2 + testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.835Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: election won: tally=2 +testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.835Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: entering leader state: leader="Node at f25432d1-1355-b829-f9d5-1c675504ab50 [Leader]" + testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.835Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: entering leader state: leader="Node at f25432d1-1355-b829-f9d5-1c675504ab50 [Leader]" +testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.835Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: added peer, starting replication: peer=server-f8811921-2735-d284-f23e-ce90eafcb10a + testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.835Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: added peer, starting replication: peer=server-f8811921-2735-d284-f23e-ce90eafcb10a +testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.835Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: added peer, starting replication: peer=server-11a0d01c-4a46-e06a-e67b-844791818f99 + testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.835Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: added peer, starting replication: peer=server-11a0d01c-4a46-e06a-e67b-844791818f99 +testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.835Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: pipelining replication: peer="{Voter server-f8811921-2735-d284-f23e-ce90eafcb10a f8811921-2735-d284-f23e-ce90eafcb10a}" + testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.835Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: pipelining replication: peer="{Voter server-f8811921-2735-d284-f23e-ce90eafcb10a f8811921-2735-d284-f23e-ce90eafcb10a}" +testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.835Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: pipelining replication: peer="{Voter server-11a0d01c-4a46-e06a-e67b-844791818f99 11a0d01c-4a46-e06a-e67b-844791818f99}" + testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.835Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: pipelining replication: peer="{Voter server-11a0d01c-4a46-e06a-e67b-844791818f99 11a0d01c-4a46-e06a-e67b-844791818f99}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:58.940Z [INFO] stable state for Leader reached at 2023-08-02 15:12:58.835315302 +0000 UTC m=+10.931251660 (1 nodes), 59.064405ms from start of poll, 59.752137ms from cluster start. Timeout at 2023-08-02 15:12:58.940436599 +0000 UTC m=+11.036372988, 105.121328ms after stability + testing.go:149: cluster: 2023-08-02T15:12:58.940Z [INFO] stable state for Leader reached at 2023-08-02 15:12:58.835315302 +0000 UTC m=+10.931251660 (1 nodes), 59.064405ms from start of poll, 59.752137ms from cluster start. Timeout at 2023-08-02 15:12:58.940436599 +0000 UTC m=+11.036372988, 105.121328ms after stability +testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.941Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: starting snapshot up to: index=12 + testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.941Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: starting snapshot up to: index=12 + testing.go:153: 2023-08-02T15:12:58.941Z [INFO] creating new snapshot: path=/tmp/raft142344519/snapshots/2-12-1690989178941.tmp +testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.941Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: snapshot complete up to: index=12 + testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.941Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: snapshot complete up to: index=12 + testing.go:153: 2023-08-02T15:12:58.942Z [INFO] creating new snapshot: path=/tmp/raft142344519/snapshots/2-23-1690989178942.tmp +testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.942Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: copied to local snapshot: bytes=71 + testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.942Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.942Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: restored user snapshot: index=1 + testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.942Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: restored user snapshot: index=1 +testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.942Z [ERROR] server-f25432d1-1355-b829-f9d5-1c675504ab50: failed to get log: index=23 error="log not found" + testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.942Z [ERROR] server-f25432d1-1355-b829-f9d5-1c675504ab50: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.942Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: aborting pipeline replication: peer="{Voter server-11a0d01c-4a46-e06a-e67b-844791818f99 11a0d01c-4a46-e06a-e67b-844791818f99}" + testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.942Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: aborting pipeline replication: peer="{Voter server-11a0d01c-4a46-e06a-e67b-844791818f99 11a0d01c-4a46-e06a-e67b-844791818f99}" +testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.942Z [ERROR] server-f25432d1-1355-b829-f9d5-1c675504ab50: failed to get log: index=23 error="log not found" + testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.942Z [ERROR] server-f25432d1-1355-b829-f9d5-1c675504ab50: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.942Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: aborting pipeline replication: peer="{Voter server-f8811921-2735-d284-f23e-ce90eafcb10a f8811921-2735-d284-f23e-ce90eafcb10a}" + testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.942Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: aborting pipeline replication: peer="{Voter server-f8811921-2735-d284-f23e-ce90eafcb10a f8811921-2735-d284-f23e-ce90eafcb10a}" +testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.942Z [ERROR] server-f25432d1-1355-b829-f9d5-1c675504ab50: failed to get log: index=23 error="log not found" + testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.942Z [ERROR] server-f25432d1-1355-b829-f9d5-1c675504ab50: failed to get log: index=23 error="log not found" + testing.go:153: 2023-08-02T15:12:58.943Z [INFO] creating new snapshot: path=/tmp/raft013424013/snapshots/2-23-1690989178943.tmp +testLoggerAdapter verbose: server-f8811921-2735-d284-f23e-ce90eafcb10a: 2023-08-02T15:12:58.943Z [INFO] server-f8811921-2735-d284-f23e-ce90eafcb10a: copied to local snapshot: bytes=71 + testing.go:149: server-f8811921-2735-d284-f23e-ce90eafcb10a: 2023-08-02T15:12:58.943Z [INFO] server-f8811921-2735-d284-f23e-ce90eafcb10a: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-f8811921-2735-d284-f23e-ce90eafcb10a: 2023-08-02T15:12:58.944Z [INFO] server-f8811921-2735-d284-f23e-ce90eafcb10a: Installed remote snapshot + testing.go:149: server-f8811921-2735-d284-f23e-ce90eafcb10a: 2023-08-02T15:12:58.944Z [INFO] server-f8811921-2735-d284-f23e-ce90eafcb10a: Installed remote snapshot +testLoggerAdapter verbose: server-f8811921-2735-d284-f23e-ce90eafcb10a: 2023-08-02T15:12:58.944Z [WARN] server-f8811921-2735-d284-f23e-ce90eafcb10a: skipping application of old log: index=22 + testing.go:149: server-f8811921-2735-d284-f23e-ce90eafcb10a: 2023-08-02T15:12:58.944Z [WARN] server-f8811921-2735-d284-f23e-ce90eafcb10a: skipping application of old log: index=22 +testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.944Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: pipelining replication: peer="{Voter server-f8811921-2735-d284-f23e-ce90eafcb10a f8811921-2735-d284-f23e-ce90eafcb10a}" + testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.944Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: pipelining replication: peer="{Voter server-f8811921-2735-d284-f23e-ce90eafcb10a f8811921-2735-d284-f23e-ce90eafcb10a}" +testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.952Z [ERROR] server-f25432d1-1355-b829-f9d5-1c675504ab50: failed to get log: index=23 error="log not found" + testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.952Z [ERROR] server-f25432d1-1355-b829-f9d5-1c675504ab50: failed to get log: index=23 error="log not found" + testing.go:153: 2023-08-02T15:12:58.952Z [INFO] creating new snapshot: path=/tmp/raft242716433/snapshots/2-23-1690989178952.tmp +testLoggerAdapter verbose: server-11a0d01c-4a46-e06a-e67b-844791818f99: 2023-08-02T15:12:58.953Z [INFO] server-11a0d01c-4a46-e06a-e67b-844791818f99: copied to local snapshot: bytes=71 + testing.go:149: server-11a0d01c-4a46-e06a-e67b-844791818f99: 2023-08-02T15:12:58.953Z [INFO] server-11a0d01c-4a46-e06a-e67b-844791818f99: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-11a0d01c-4a46-e06a-e67b-844791818f99: 2023-08-02T15:12:58.953Z [INFO] server-11a0d01c-4a46-e06a-e67b-844791818f99: Installed remote snapshot + testing.go:149: server-11a0d01c-4a46-e06a-e67b-844791818f99: 2023-08-02T15:12:58.953Z [INFO] server-11a0d01c-4a46-e06a-e67b-844791818f99: Installed remote snapshot +testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.953Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: pipelining replication: peer="{Voter server-11a0d01c-4a46-e06a-e67b-844791818f99 11a0d01c-4a46-e06a-e67b-844791818f99}" + testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.953Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: pipelining replication: peer="{Voter server-11a0d01c-4a46-e06a-e67b-844791818f99 11a0d01c-4a46-e06a-e67b-844791818f99}" +testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.965Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: aborting pipeline replication: peer="{Voter server-11a0d01c-4a46-e06a-e67b-844791818f99 11a0d01c-4a46-e06a-e67b-844791818f99}" + testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.965Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: aborting pipeline replication: peer="{Voter server-11a0d01c-4a46-e06a-e67b-844791818f99 11a0d01c-4a46-e06a-e67b-844791818f99}" +testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.965Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: aborting pipeline replication: peer="{Voter server-f8811921-2735-d284-f23e-ce90eafcb10a f8811921-2735-d284-f23e-ce90eafcb10a}" + testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.965Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: aborting pipeline replication: peer="{Voter server-f8811921-2735-d284-f23e-ce90eafcb10a f8811921-2735-d284-f23e-ce90eafcb10a}" +testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:58.966Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b137adde-4720-880c-9538-d85b60b2a59b Address:b137adde-4720-880c-9538-d85b60b2a59b} {Suffrage:Voter ID:server-cff6bc78-1346-3f29-6e01-90cee53fe183 Address:cff6bc78-1346-3f29-6e01-90cee53fe183} {Suffrage:Voter ID:server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d Address:5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d}]" + testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:58.966Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b137adde-4720-880c-9538-d85b60b2a59b Address:b137adde-4720-880c-9538-d85b60b2a59b} {Suffrage:Voter ID:server-cff6bc78-1346-3f29-6e01-90cee53fe183 Address:cff6bc78-1346-3f29-6e01-90cee53fe183} {Suffrage:Voter ID:server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d Address:5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d}]" +testLoggerAdapter verbose: server-cff6bc78-1346-3f29-6e01-90cee53fe183: 2023-08-02T15:12:58.966Z [INFO] server-cff6bc78-1346-3f29-6e01-90cee53fe183: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b137adde-4720-880c-9538-d85b60b2a59b Address:b137adde-4720-880c-9538-d85b60b2a59b} {Suffrage:Voter ID:server-cff6bc78-1346-3f29-6e01-90cee53fe183 Address:cff6bc78-1346-3f29-6e01-90cee53fe183} {Suffrage:Voter ID:server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d Address:5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d}]" + testing.go:149: server-cff6bc78-1346-3f29-6e01-90cee53fe183: 2023-08-02T15:12:58.966Z [INFO] server-cff6bc78-1346-3f29-6e01-90cee53fe183: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b137adde-4720-880c-9538-d85b60b2a59b Address:b137adde-4720-880c-9538-d85b60b2a59b} {Suffrage:Voter ID:server-cff6bc78-1346-3f29-6e01-90cee53fe183 Address:cff6bc78-1346-3f29-6e01-90cee53fe183} {Suffrage:Voter ID:server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d Address:5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d}]" +testLoggerAdapter verbose: server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d: 2023-08-02T15:12:58.966Z [INFO] server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b137adde-4720-880c-9538-d85b60b2a59b Address:b137adde-4720-880c-9538-d85b60b2a59b} {Suffrage:Voter ID:server-cff6bc78-1346-3f29-6e01-90cee53fe183 Address:cff6bc78-1346-3f29-6e01-90cee53fe183} {Suffrage:Voter ID:server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d Address:5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d}]" + testing.go:149: server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d: 2023-08-02T15:12:58.966Z [INFO] server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b137adde-4720-880c-9538-d85b60b2a59b Address:b137adde-4720-880c-9538-d85b60b2a59b} {Suffrage:Voter ID:server-cff6bc78-1346-3f29-6e01-90cee53fe183 Address:cff6bc78-1346-3f29-6e01-90cee53fe183} {Suffrage:Voter ID:server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d Address:5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:58.967Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:12:58.967Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:58.967Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: entering follower state: follower="Node at b137adde-4720-880c-9538-d85b60b2a59b [Follower]" leader= + testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:58.967Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: entering follower state: follower="Node at b137adde-4720-880c-9538-d85b60b2a59b [Follower]" leader= +testLoggerAdapter verbose: server-cff6bc78-1346-3f29-6e01-90cee53fe183: 2023-08-02T15:12:58.967Z [INFO] server-cff6bc78-1346-3f29-6e01-90cee53fe183: entering follower state: follower="Node at cff6bc78-1346-3f29-6e01-90cee53fe183 [Follower]" leader= + testing.go:149: server-cff6bc78-1346-3f29-6e01-90cee53fe183: 2023-08-02T15:12:58.967Z [INFO] server-cff6bc78-1346-3f29-6e01-90cee53fe183: entering follower state: follower="Node at cff6bc78-1346-3f29-6e01-90cee53fe183 [Follower]" leader= +testLoggerAdapter verbose: server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d: 2023-08-02T15:12:58.967Z [INFO] server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d: entering follower state: follower="Node at 5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d [Follower]" leader= + testing.go:149: server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d: 2023-08-02T15:12:58.967Z [INFO] server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d: entering follower state: follower="Node at 5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d [Follower]" leader= +testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.021Z [WARN] server-b137adde-4720-880c-9538-d85b60b2a59b: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.021Z [WARN] server-b137adde-4720-880c-9538-d85b60b2a59b: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.021Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: entering candidate state: node="Node at b137adde-4720-880c-9538-d85b60b2a59b [Candidate]" term=2 + testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.021Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: entering candidate state: node="Node at b137adde-4720-880c-9538-d85b60b2a59b [Candidate]" term=2 +testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.021Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: election won: tally=2 + testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.021Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: election won: tally=2 +testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.022Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: entering leader state: leader="Node at b137adde-4720-880c-9538-d85b60b2a59b [Leader]" + testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.022Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: entering leader state: leader="Node at b137adde-4720-880c-9538-d85b60b2a59b [Leader]" +testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.022Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: added peer, starting replication: peer=server-cff6bc78-1346-3f29-6e01-90cee53fe183 + testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.022Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: added peer, starting replication: peer=server-cff6bc78-1346-3f29-6e01-90cee53fe183 +testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.022Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: added peer, starting replication: peer=server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d + testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.022Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: added peer, starting replication: peer=server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d +testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.022Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: pipelining replication: peer="{Voter server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d 5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d}" + testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.022Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: pipelining replication: peer="{Voter server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d 5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d}" +testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.022Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: pipelining replication: peer="{Voter server-cff6bc78-1346-3f29-6e01-90cee53fe183 cff6bc78-1346-3f29-6e01-90cee53fe183}" + testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.022Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: pipelining replication: peer="{Voter server-cff6bc78-1346-3f29-6e01-90cee53fe183 cff6bc78-1346-3f29-6e01-90cee53fe183}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:59.127Z [INFO] stable state for Leader reached at 2023-08-02 15:12:59.022308266 +0000 UTC m=+11.118244598 (1 nodes), 55.256349ms from start of poll, 55.776229ms from cluster start. Timeout at 2023-08-02 15:12:59.127443147 +0000 UTC m=+11.223379516, 105.134918ms after stability + testing.go:149: cluster: 2023-08-02T15:12:59.127Z [INFO] stable state for Leader reached at 2023-08-02 15:12:59.022308266 +0000 UTC m=+11.118244598 (1 nodes), 55.256349ms from start of poll, 55.776229ms from cluster start. Timeout at 2023-08-02 15:12:59.127443147 +0000 UTC m=+11.223379516, 105.134918ms after stability +testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.128Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: starting snapshot up to: index=12 + testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.128Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: starting snapshot up to: index=12 + testing.go:153: 2023-08-02T15:12:59.128Z [INFO] creating new snapshot: path=/tmp/raft464897387/snapshots/2-12-1690989179128.tmp +testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.128Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: snapshot complete up to: index=12 + testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.128Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: snapshot complete up to: index=12 + testing.go:153: 2023-08-02T15:12:59.129Z [INFO] creating new snapshot: path=/tmp/raft464897387/snapshots/2-23-1690989179129.tmp +testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.129Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: copied to local snapshot: bytes=71 + testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.129Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.129Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: restored user snapshot: index=1 + testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.129Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: restored user snapshot: index=1 +testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.129Z [ERROR] server-b137adde-4720-880c-9538-d85b60b2a59b: failed to get log: index=23 error="log not found" + testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.129Z [ERROR] server-b137adde-4720-880c-9538-d85b60b2a59b: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.129Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: aborting pipeline replication: peer="{Voter server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d 5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d}" + testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.129Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: aborting pipeline replication: peer="{Voter server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d 5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d}" +testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.129Z [ERROR] server-b137adde-4720-880c-9538-d85b60b2a59b: failed to get log: index=23 error="log not found" + testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.129Z [ERROR] server-b137adde-4720-880c-9538-d85b60b2a59b: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.129Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: aborting pipeline replication: peer="{Voter server-cff6bc78-1346-3f29-6e01-90cee53fe183 cff6bc78-1346-3f29-6e01-90cee53fe183}" + testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.129Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: aborting pipeline replication: peer="{Voter server-cff6bc78-1346-3f29-6e01-90cee53fe183 cff6bc78-1346-3f29-6e01-90cee53fe183}" +testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.137Z [ERROR] server-b137adde-4720-880c-9538-d85b60b2a59b: failed to get log: index=23 error="log not found" + testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.137Z [ERROR] server-b137adde-4720-880c-9538-d85b60b2a59b: failed to get log: index=23 error="log not found" + testing.go:153: 2023-08-02T15:12:59.138Z [INFO] creating new snapshot: path=/tmp/raft980704469/snapshots/2-23-1690989179138.tmp +testLoggerAdapter verbose: server-cff6bc78-1346-3f29-6e01-90cee53fe183: 2023-08-02T15:12:59.138Z [INFO] server-cff6bc78-1346-3f29-6e01-90cee53fe183: copied to local snapshot: bytes=71 + testing.go:149: server-cff6bc78-1346-3f29-6e01-90cee53fe183: 2023-08-02T15:12:59.138Z [INFO] server-cff6bc78-1346-3f29-6e01-90cee53fe183: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-cff6bc78-1346-3f29-6e01-90cee53fe183: 2023-08-02T15:12:59.139Z [INFO] server-cff6bc78-1346-3f29-6e01-90cee53fe183: Installed remote snapshot + testing.go:149: server-cff6bc78-1346-3f29-6e01-90cee53fe183: 2023-08-02T15:12:59.139Z [INFO] server-cff6bc78-1346-3f29-6e01-90cee53fe183: Installed remote snapshot +testLoggerAdapter verbose: server-cff6bc78-1346-3f29-6e01-90cee53fe183: 2023-08-02T15:12:59.139Z [WARN] server-cff6bc78-1346-3f29-6e01-90cee53fe183: skipping application of old log: index=22 + testing.go:149: server-cff6bc78-1346-3f29-6e01-90cee53fe183: 2023-08-02T15:12:59.139Z [WARN] server-cff6bc78-1346-3f29-6e01-90cee53fe183: skipping application of old log: index=22 +testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.139Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: pipelining replication: peer="{Voter server-cff6bc78-1346-3f29-6e01-90cee53fe183 cff6bc78-1346-3f29-6e01-90cee53fe183}" + testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.139Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: pipelining replication: peer="{Voter server-cff6bc78-1346-3f29-6e01-90cee53fe183 cff6bc78-1346-3f29-6e01-90cee53fe183}" +testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.139Z [ERROR] server-b137adde-4720-880c-9538-d85b60b2a59b: failed to get log: index=23 error="log not found" + testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.139Z [ERROR] server-b137adde-4720-880c-9538-d85b60b2a59b: failed to get log: index=23 error="log not found" + testing.go:153: 2023-08-02T15:12:59.142Z [INFO] creating new snapshot: path=/tmp/raft956950479/snapshots/2-23-1690989179142.tmp +testLoggerAdapter verbose: server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d: 2023-08-02T15:12:59.143Z [INFO] server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d: copied to local snapshot: bytes=71 + testing.go:149: server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d: 2023-08-02T15:12:59.143Z [INFO] server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d: 2023-08-02T15:12:59.143Z [INFO] server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d: Installed remote snapshot + testing.go:149: server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d: 2023-08-02T15:12:59.143Z [INFO] server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d: Installed remote snapshot +testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.143Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: pipelining replication: peer="{Voter server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d 5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d}" + testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.143Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: pipelining replication: peer="{Voter server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d 5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d}" +testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.155Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: aborting pipeline replication: peer="{Voter server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d 5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d}" + testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.155Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: aborting pipeline replication: peer="{Voter server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d 5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d}" +testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.155Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: aborting pipeline replication: peer="{Voter server-cff6bc78-1346-3f29-6e01-90cee53fe183 cff6bc78-1346-3f29-6e01-90cee53fe183}" + testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.155Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: aborting pipeline replication: peer="{Voter server-cff6bc78-1346-3f29-6e01-90cee53fe183 cff6bc78-1346-3f29-6e01-90cee53fe183}" +testLoggerAdapter verbose: server-97b6deac-af23-d668-a303-38e9fd8a9ad7: 2023-08-02T15:12:59.156Z [INFO] server-97b6deac-af23-d668-a303-38e9fd8a9ad7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-97b6deac-af23-d668-a303-38e9fd8a9ad7 Address:97b6deac-af23-d668-a303-38e9fd8a9ad7} {Suffrage:Voter ID:server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9 Address:52f0d966-03d3-98ea-7f23-a0c6d2a743a9} {Suffrage:Voter ID:server-2d4e71af-f3b4-de58-aecf-42726d85262c Address:2d4e71af-f3b4-de58-aecf-42726d85262c}]" + testing.go:149: server-97b6deac-af23-d668-a303-38e9fd8a9ad7: 2023-08-02T15:12:59.156Z [INFO] server-97b6deac-af23-d668-a303-38e9fd8a9ad7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-97b6deac-af23-d668-a303-38e9fd8a9ad7 Address:97b6deac-af23-d668-a303-38e9fd8a9ad7} {Suffrage:Voter ID:server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9 Address:52f0d966-03d3-98ea-7f23-a0c6d2a743a9} {Suffrage:Voter ID:server-2d4e71af-f3b4-de58-aecf-42726d85262c Address:2d4e71af-f3b4-de58-aecf-42726d85262c}]" +testLoggerAdapter verbose: server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: 2023-08-02T15:12:59.156Z [INFO] server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-97b6deac-af23-d668-a303-38e9fd8a9ad7 Address:97b6deac-af23-d668-a303-38e9fd8a9ad7} {Suffrage:Voter ID:server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9 Address:52f0d966-03d3-98ea-7f23-a0c6d2a743a9} {Suffrage:Voter ID:server-2d4e71af-f3b4-de58-aecf-42726d85262c Address:2d4e71af-f3b4-de58-aecf-42726d85262c}]" + testing.go:149: server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: 2023-08-02T15:12:59.156Z [INFO] server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-97b6deac-af23-d668-a303-38e9fd8a9ad7 Address:97b6deac-af23-d668-a303-38e9fd8a9ad7} {Suffrage:Voter ID:server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9 Address:52f0d966-03d3-98ea-7f23-a0c6d2a743a9} {Suffrage:Voter ID:server-2d4e71af-f3b4-de58-aecf-42726d85262c Address:2d4e71af-f3b4-de58-aecf-42726d85262c}]" +testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.156Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-97b6deac-af23-d668-a303-38e9fd8a9ad7 Address:97b6deac-af23-d668-a303-38e9fd8a9ad7} {Suffrage:Voter ID:server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9 Address:52f0d966-03d3-98ea-7f23-a0c6d2a743a9} {Suffrage:Voter ID:server-2d4e71af-f3b4-de58-aecf-42726d85262c Address:2d4e71af-f3b4-de58-aecf-42726d85262c}]" + testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.156Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-97b6deac-af23-d668-a303-38e9fd8a9ad7 Address:97b6deac-af23-d668-a303-38e9fd8a9ad7} {Suffrage:Voter ID:server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9 Address:52f0d966-03d3-98ea-7f23-a0c6d2a743a9} {Suffrage:Voter ID:server-2d4e71af-f3b4-de58-aecf-42726d85262c Address:2d4e71af-f3b4-de58-aecf-42726d85262c}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:59.157Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:12:59.157Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-97b6deac-af23-d668-a303-38e9fd8a9ad7: 2023-08-02T15:12:59.157Z [INFO] server-97b6deac-af23-d668-a303-38e9fd8a9ad7: entering follower state: follower="Node at 97b6deac-af23-d668-a303-38e9fd8a9ad7 [Follower]" leader= + testing.go:149: server-97b6deac-af23-d668-a303-38e9fd8a9ad7: 2023-08-02T15:12:59.157Z [INFO] server-97b6deac-af23-d668-a303-38e9fd8a9ad7: entering follower state: follower="Node at 97b6deac-af23-d668-a303-38e9fd8a9ad7 [Follower]" leader= +testLoggerAdapter verbose: server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: 2023-08-02T15:12:59.157Z [INFO] server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: entering follower state: follower="Node at 52f0d966-03d3-98ea-7f23-a0c6d2a743a9 [Follower]" leader= + testing.go:149: server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: 2023-08-02T15:12:59.157Z [INFO] server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: entering follower state: follower="Node at 52f0d966-03d3-98ea-7f23-a0c6d2a743a9 [Follower]" leader= +testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.157Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: entering follower state: follower="Node at 2d4e71af-f3b4-de58-aecf-42726d85262c [Follower]" leader= + testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.157Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: entering follower state: follower="Node at 2d4e71af-f3b4-de58-aecf-42726d85262c [Follower]" leader= +testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.210Z [WARN] server-2d4e71af-f3b4-de58-aecf-42726d85262c: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.210Z [WARN] server-2d4e71af-f3b4-de58-aecf-42726d85262c: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.211Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: entering candidate state: node="Node at 2d4e71af-f3b4-de58-aecf-42726d85262c [Candidate]" term=2 + testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.211Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: entering candidate state: node="Node at 2d4e71af-f3b4-de58-aecf-42726d85262c [Candidate]" term=2 +testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.211Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: election won: tally=2 + testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.211Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: election won: tally=2 +testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.211Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: entering leader state: leader="Node at 2d4e71af-f3b4-de58-aecf-42726d85262c [Leader]" + testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.211Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: entering leader state: leader="Node at 2d4e71af-f3b4-de58-aecf-42726d85262c [Leader]" +testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.211Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: added peer, starting replication: peer=server-97b6deac-af23-d668-a303-38e9fd8a9ad7 + testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.211Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: added peer, starting replication: peer=server-97b6deac-af23-d668-a303-38e9fd8a9ad7 +testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.211Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: added peer, starting replication: peer=server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9 + testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.211Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: added peer, starting replication: peer=server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9 +testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.211Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: pipelining replication: peer="{Voter server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9 52f0d966-03d3-98ea-7f23-a0c6d2a743a9}" + testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.211Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: pipelining replication: peer="{Voter server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9 52f0d966-03d3-98ea-7f23-a0c6d2a743a9}" +testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.211Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: pipelining replication: peer="{Voter server-97b6deac-af23-d668-a303-38e9fd8a9ad7 97b6deac-af23-d668-a303-38e9fd8a9ad7}" + testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.211Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: pipelining replication: peer="{Voter server-97b6deac-af23-d668-a303-38e9fd8a9ad7 97b6deac-af23-d668-a303-38e9fd8a9ad7}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:59.316Z [INFO] stable state for Leader reached at 2023-08-02 15:12:59.211589405 +0000 UTC m=+11.307525733 (1 nodes), 54.532045ms from start of poll, 55.12247ms from cluster start. Timeout at 2023-08-02 15:12:59.316686956 +0000 UTC m=+11.412623283, 105.09755ms after stability + testing.go:149: cluster: 2023-08-02T15:12:59.316Z [INFO] stable state for Leader reached at 2023-08-02 15:12:59.211589405 +0000 UTC m=+11.307525733 (1 nodes), 54.532045ms from start of poll, 55.12247ms from cluster start. Timeout at 2023-08-02 15:12:59.316686956 +0000 UTC m=+11.412623283, 105.09755ms after stability +testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.317Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: starting snapshot up to: index=12 + testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.317Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: starting snapshot up to: index=12 + testing.go:153: 2023-08-02T15:12:59.317Z [INFO] creating new snapshot: path=/tmp/raft515795741/snapshots/2-12-1690989179317.tmp +testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.317Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: snapshot complete up to: index=12 + testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.317Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: snapshot complete up to: index=12 + testing.go:153: 2023-08-02T15:12:59.318Z [INFO] creating new snapshot: path=/tmp/raft515795741/snapshots/2-23-1690989179318.tmp +testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.318Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: copied to local snapshot: bytes=71 + testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.318Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.318Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: restored user snapshot: index=1 + testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.318Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: restored user snapshot: index=1 +testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.318Z [ERROR] server-2d4e71af-f3b4-de58-aecf-42726d85262c: failed to get log: index=23 error="log not found" + testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.318Z [ERROR] server-2d4e71af-f3b4-de58-aecf-42726d85262c: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.319Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: aborting pipeline replication: peer="{Voter server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9 52f0d966-03d3-98ea-7f23-a0c6d2a743a9}" + testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.319Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: aborting pipeline replication: peer="{Voter server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9 52f0d966-03d3-98ea-7f23-a0c6d2a743a9}" +testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.318Z [ERROR] server-2d4e71af-f3b4-de58-aecf-42726d85262c: failed to get log: index=23 error="log not found" + testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.318Z [ERROR] server-2d4e71af-f3b4-de58-aecf-42726d85262c: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.319Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: aborting pipeline replication: peer="{Voter server-97b6deac-af23-d668-a303-38e9fd8a9ad7 97b6deac-af23-d668-a303-38e9fd8a9ad7}" + testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.319Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: aborting pipeline replication: peer="{Voter server-97b6deac-af23-d668-a303-38e9fd8a9ad7 97b6deac-af23-d668-a303-38e9fd8a9ad7}" +testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.326Z [ERROR] server-2d4e71af-f3b4-de58-aecf-42726d85262c: failed to get log: index=23 error="log not found" + testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.326Z [ERROR] server-2d4e71af-f3b4-de58-aecf-42726d85262c: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.326Z [ERROR] server-2d4e71af-f3b4-de58-aecf-42726d85262c: failed to get log: index=23 error="log not found" + testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.326Z [ERROR] server-2d4e71af-f3b4-de58-aecf-42726d85262c: failed to get log: index=23 error="log not found" + testing.go:153: 2023-08-02T15:12:59.326Z [INFO] creating new snapshot: path=/tmp/raft130860761/snapshots/2-23-1690989179326.tmp + testing.go:153: 2023-08-02T15:12:59.326Z [INFO] creating new snapshot: path=/tmp/raft861224563/snapshots/2-23-1690989179326.tmp +testLoggerAdapter verbose: server-97b6deac-af23-d668-a303-38e9fd8a9ad7: 2023-08-02T15:12:59.327Z [INFO] server-97b6deac-af23-d668-a303-38e9fd8a9ad7: copied to local snapshot: bytes=71 + testing.go:149: server-97b6deac-af23-d668-a303-38e9fd8a9ad7: 2023-08-02T15:12:59.327Z [INFO] server-97b6deac-af23-d668-a303-38e9fd8a9ad7: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: 2023-08-02T15:12:59.327Z [INFO] server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: copied to local snapshot: bytes=71 + testing.go:149: server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: 2023-08-02T15:12:59.327Z [INFO] server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-97b6deac-af23-d668-a303-38e9fd8a9ad7: 2023-08-02T15:12:59.327Z [INFO] server-97b6deac-af23-d668-a303-38e9fd8a9ad7: Installed remote snapshot + testing.go:149: server-97b6deac-af23-d668-a303-38e9fd8a9ad7: 2023-08-02T15:12:59.327Z [INFO] server-97b6deac-af23-d668-a303-38e9fd8a9ad7: Installed remote snapshot +testLoggerAdapter verbose: server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: 2023-08-02T15:12:59.327Z [INFO] server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: Installed remote snapshot +testLoggerAdapter verbose: server-97b6deac-af23-d668-a303-38e9fd8a9ad7: 2023-08-02T15:12:59.327Z [WARN] server-97b6deac-af23-d668-a303-38e9fd8a9ad7: skipping application of old log: index=22 + testing.go:149: server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: 2023-08-02T15:12:59.327Z [INFO] server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: Installed remote snapshot + testing.go:149: server-97b6deac-af23-d668-a303-38e9fd8a9ad7: 2023-08-02T15:12:59.327Z [WARN] server-97b6deac-af23-d668-a303-38e9fd8a9ad7: skipping application of old log: index=22 +testLoggerAdapter verbose: server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: 2023-08-02T15:12:59.327Z [WARN] server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: skipping application of old log: index=23 + testing.go:149: server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: 2023-08-02T15:12:59.327Z [WARN] server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: skipping application of old log: index=23 +testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.327Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: pipelining replication: peer="{Voter server-97b6deac-af23-d668-a303-38e9fd8a9ad7 97b6deac-af23-d668-a303-38e9fd8a9ad7}" + testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.327Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: pipelining replication: peer="{Voter server-97b6deac-af23-d668-a303-38e9fd8a9ad7 97b6deac-af23-d668-a303-38e9fd8a9ad7}" +testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.327Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: pipelining replication: peer="{Voter server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9 52f0d966-03d3-98ea-7f23-a0c6d2a743a9}" + testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.327Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: pipelining replication: peer="{Voter server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9 52f0d966-03d3-98ea-7f23-a0c6d2a743a9}" +testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.338Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: aborting pipeline replication: peer="{Voter server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9 52f0d966-03d3-98ea-7f23-a0c6d2a743a9}" + testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.338Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: aborting pipeline replication: peer="{Voter server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9 52f0d966-03d3-98ea-7f23-a0c6d2a743a9}" +testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.338Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: aborting pipeline replication: peer="{Voter server-97b6deac-af23-d668-a303-38e9fd8a9ad7 97b6deac-af23-d668-a303-38e9fd8a9ad7}" + testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.338Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: aborting pipeline replication: peer="{Voter server-97b6deac-af23-d668-a303-38e9fd8a9ad7 97b6deac-af23-d668-a303-38e9fd8a9ad7}" +testLoggerAdapter verbose: server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2: 2023-08-02T15:12:59.339Z [INFO] server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2 Address:abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2} {Suffrage:Voter ID:server-009a1e34-c01e-0280-4511-05dabf8f55db Address:009a1e34-c01e-0280-4511-05dabf8f55db} {Suffrage:Voter ID:server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9 Address:5afb8d1a-261a-64dc-db92-bc1cbc2525b9}]" + testing.go:149: server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2: 2023-08-02T15:12:59.339Z [INFO] server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2 Address:abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2} {Suffrage:Voter ID:server-009a1e34-c01e-0280-4511-05dabf8f55db Address:009a1e34-c01e-0280-4511-05dabf8f55db} {Suffrage:Voter ID:server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9 Address:5afb8d1a-261a-64dc-db92-bc1cbc2525b9}]" +testLoggerAdapter verbose: server-009a1e34-c01e-0280-4511-05dabf8f55db: 2023-08-02T15:12:59.339Z [INFO] server-009a1e34-c01e-0280-4511-05dabf8f55db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2 Address:abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2} {Suffrage:Voter ID:server-009a1e34-c01e-0280-4511-05dabf8f55db Address:009a1e34-c01e-0280-4511-05dabf8f55db} {Suffrage:Voter ID:server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9 Address:5afb8d1a-261a-64dc-db92-bc1cbc2525b9}]" + testing.go:149: server-009a1e34-c01e-0280-4511-05dabf8f55db: 2023-08-02T15:12:59.339Z [INFO] server-009a1e34-c01e-0280-4511-05dabf8f55db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2 Address:abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2} {Suffrage:Voter ID:server-009a1e34-c01e-0280-4511-05dabf8f55db Address:009a1e34-c01e-0280-4511-05dabf8f55db} {Suffrage:Voter ID:server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9 Address:5afb8d1a-261a-64dc-db92-bc1cbc2525b9}]" +testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.339Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2 Address:abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2} {Suffrage:Voter ID:server-009a1e34-c01e-0280-4511-05dabf8f55db Address:009a1e34-c01e-0280-4511-05dabf8f55db} {Suffrage:Voter ID:server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9 Address:5afb8d1a-261a-64dc-db92-bc1cbc2525b9}]" + testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.339Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2 Address:abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2} {Suffrage:Voter ID:server-009a1e34-c01e-0280-4511-05dabf8f55db Address:009a1e34-c01e-0280-4511-05dabf8f55db} {Suffrage:Voter ID:server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9 Address:5afb8d1a-261a-64dc-db92-bc1cbc2525b9}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:59.339Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:12:59.339Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2: 2023-08-02T15:12:59.339Z [INFO] server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2: entering follower state: follower="Node at abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2 [Follower]" leader= + testing.go:149: server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2: 2023-08-02T15:12:59.339Z [INFO] server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2: entering follower state: follower="Node at abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2 [Follower]" leader= +testLoggerAdapter verbose: server-009a1e34-c01e-0280-4511-05dabf8f55db: 2023-08-02T15:12:59.340Z [INFO] server-009a1e34-c01e-0280-4511-05dabf8f55db: entering follower state: follower="Node at 009a1e34-c01e-0280-4511-05dabf8f55db [Follower]" leader= + testing.go:149: server-009a1e34-c01e-0280-4511-05dabf8f55db: 2023-08-02T15:12:59.340Z [INFO] server-009a1e34-c01e-0280-4511-05dabf8f55db: entering follower state: follower="Node at 009a1e34-c01e-0280-4511-05dabf8f55db [Follower]" leader= +testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.340Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: entering follower state: follower="Node at 5afb8d1a-261a-64dc-db92-bc1cbc2525b9 [Follower]" leader= + testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.340Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: entering follower state: follower="Node at 5afb8d1a-261a-64dc-db92-bc1cbc2525b9 [Follower]" leader= +testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.406Z [WARN] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.406Z [WARN] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.407Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: entering candidate state: node="Node at 5afb8d1a-261a-64dc-db92-bc1cbc2525b9 [Candidate]" term=2 + testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.407Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: entering candidate state: node="Node at 5afb8d1a-261a-64dc-db92-bc1cbc2525b9 [Candidate]" term=2 +testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.407Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: election won: tally=2 + testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.407Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: election won: tally=2 +testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.407Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: entering leader state: leader="Node at 5afb8d1a-261a-64dc-db92-bc1cbc2525b9 [Leader]" + testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.407Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: entering leader state: leader="Node at 5afb8d1a-261a-64dc-db92-bc1cbc2525b9 [Leader]" +testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.407Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: added peer, starting replication: peer=server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2 + testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.407Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: added peer, starting replication: peer=server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2 +testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.407Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: added peer, starting replication: peer=server-009a1e34-c01e-0280-4511-05dabf8f55db + testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.407Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: added peer, starting replication: peer=server-009a1e34-c01e-0280-4511-05dabf8f55db +testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.408Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: pipelining replication: peer="{Voter server-009a1e34-c01e-0280-4511-05dabf8f55db 009a1e34-c01e-0280-4511-05dabf8f55db}" + testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.408Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: pipelining replication: peer="{Voter server-009a1e34-c01e-0280-4511-05dabf8f55db 009a1e34-c01e-0280-4511-05dabf8f55db}" +testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.408Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: pipelining replication: peer="{Voter server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2 abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2}" + testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.408Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: pipelining replication: peer="{Voter server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2 abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:59.513Z [INFO] stable state for Leader reached at 2023-08-02 15:12:59.408227435 +0000 UTC m=+11.504163780 (1 nodes), 68.284665ms from start of poll, 68.787649ms from cluster start. Timeout at 2023-08-02 15:12:59.513295383 +0000 UTC m=+11.609231960, 105.06818ms after stability + testing.go:149: cluster: 2023-08-02T15:12:59.513Z [INFO] stable state for Leader reached at 2023-08-02 15:12:59.408227435 +0000 UTC m=+11.504163780 (1 nodes), 68.284665ms from start of poll, 68.787649ms from cluster start. Timeout at 2023-08-02 15:12:59.513295383 +0000 UTC m=+11.609231960, 105.06818ms after stability +testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.514Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: starting snapshot up to: index=12 + testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.514Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: starting snapshot up to: index=12 + testing.go:153: 2023-08-02T15:12:59.514Z [INFO] creating new snapshot: path=/tmp/raft565098107/snapshots/2-12-1690989179514.tmp +testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.514Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: snapshot complete up to: index=12 + testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.514Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: snapshot complete up to: index=12 + testing.go:153: 2023-08-02T15:12:59.515Z [INFO] creating new snapshot: path=/tmp/raft565098107/snapshots/2-113-1690989179515.tmp +testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.516Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: copied to local snapshot: bytes=71 + testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.516Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.516Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: restored user snapshot: index=1 + testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.516Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: restored user snapshot: index=1 +testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.516Z [ERROR] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: failed to get log: index=23 error="log not found" + testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.516Z [ERROR] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.516Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: aborting pipeline replication: peer="{Voter server-009a1e34-c01e-0280-4511-05dabf8f55db 009a1e34-c01e-0280-4511-05dabf8f55db}" + testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.516Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: aborting pipeline replication: peer="{Voter server-009a1e34-c01e-0280-4511-05dabf8f55db 009a1e34-c01e-0280-4511-05dabf8f55db}" +testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.516Z [ERROR] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: failed to get log: index=23 error="log not found" + testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.516Z [ERROR] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.516Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: aborting pipeline replication: peer="{Voter server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2 abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2}" + testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.516Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: aborting pipeline replication: peer="{Voter server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2 abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2}" +testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.523Z [ERROR] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: failed to get log: index=23 error="log not found" + testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.523Z [ERROR] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.523Z [ERROR] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: failed to get log: index=23 error="log not found" + testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.523Z [ERROR] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: failed to get log: index=23 error="log not found" + testing.go:153: 2023-08-02T15:12:59.524Z [INFO] creating new snapshot: path=/tmp/raft368743841/snapshots/2-113-1690989179524.tmp + testing.go:153: 2023-08-02T15:12:59.524Z [INFO] creating new snapshot: path=/tmp/raft182143831/snapshots/2-113-1690989179524.tmp +testLoggerAdapter verbose: server-009a1e34-c01e-0280-4511-05dabf8f55db: 2023-08-02T15:12:59.525Z [INFO] server-009a1e34-c01e-0280-4511-05dabf8f55db: copied to local snapshot: bytes=71 + testing.go:149: server-009a1e34-c01e-0280-4511-05dabf8f55db: 2023-08-02T15:12:59.525Z [INFO] server-009a1e34-c01e-0280-4511-05dabf8f55db: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2: 2023-08-02T15:12:59.525Z [INFO] server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2: copied to local snapshot: bytes=71 + testing.go:149: server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2: 2023-08-02T15:12:59.525Z [INFO] server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-009a1e34-c01e-0280-4511-05dabf8f55db: 2023-08-02T15:12:59.525Z [INFO] server-009a1e34-c01e-0280-4511-05dabf8f55db: Installed remote snapshot + testing.go:149: server-009a1e34-c01e-0280-4511-05dabf8f55db: 2023-08-02T15:12:59.525Z [INFO] server-009a1e34-c01e-0280-4511-05dabf8f55db: Installed remote snapshot +testLoggerAdapter verbose: server-009a1e34-c01e-0280-4511-05dabf8f55db: 2023-08-02T15:12:59.525Z [WARN] server-009a1e34-c01e-0280-4511-05dabf8f55db: skipping application of old log: index=22 + testing.go:149: server-009a1e34-c01e-0280-4511-05dabf8f55db: 2023-08-02T15:12:59.525Z [WARN] server-009a1e34-c01e-0280-4511-05dabf8f55db: skipping application of old log: index=22 +testLoggerAdapter verbose: server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2: 2023-08-02T15:12:59.525Z [INFO] server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2: Installed remote snapshot +testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.525Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: pipelining replication: peer="{Voter server-009a1e34-c01e-0280-4511-05dabf8f55db 009a1e34-c01e-0280-4511-05dabf8f55db}" + testing.go:149: server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2: 2023-08-02T15:12:59.525Z [INFO] server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2: Installed remote snapshot + testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.525Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: pipelining replication: peer="{Voter server-009a1e34-c01e-0280-4511-05dabf8f55db 009a1e34-c01e-0280-4511-05dabf8f55db}" +testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.525Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: pipelining replication: peer="{Voter server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2 abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2}" + testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.525Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: pipelining replication: peer="{Voter server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2 abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2}" +testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.536Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: aborting pipeline replication: peer="{Voter server-009a1e34-c01e-0280-4511-05dabf8f55db 009a1e34-c01e-0280-4511-05dabf8f55db}" + testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.536Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: aborting pipeline replication: peer="{Voter server-009a1e34-c01e-0280-4511-05dabf8f55db 009a1e34-c01e-0280-4511-05dabf8f55db}" +testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.536Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: aborting pipeline replication: peer="{Voter server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2 abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2}" + testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.536Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: aborting pipeline replication: peer="{Voter server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2 abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2}" +testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.551Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced Address:70ef33c8-8204-4afb-87b1-6fabdcbe3ced} {Suffrage:Voter ID:server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9 Address:0dad452b-68e0-c2e4-90d2-3f90baeab6c9} {Suffrage:Voter ID:server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4 Address:2b5444f2-c2c5-d465-f6d5-c4f4551af8d4}]" + testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.551Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced Address:70ef33c8-8204-4afb-87b1-6fabdcbe3ced} {Suffrage:Voter ID:server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9 Address:0dad452b-68e0-c2e4-90d2-3f90baeab6c9} {Suffrage:Voter ID:server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4 Address:2b5444f2-c2c5-d465-f6d5-c4f4551af8d4}]" +testLoggerAdapter verbose: server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9: 2023-08-02T15:12:59.551Z [INFO] server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced Address:70ef33c8-8204-4afb-87b1-6fabdcbe3ced} {Suffrage:Voter ID:server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9 Address:0dad452b-68e0-c2e4-90d2-3f90baeab6c9} {Suffrage:Voter ID:server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4 Address:2b5444f2-c2c5-d465-f6d5-c4f4551af8d4}]" + testing.go:149: server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9: 2023-08-02T15:12:59.551Z [INFO] server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced Address:70ef33c8-8204-4afb-87b1-6fabdcbe3ced} {Suffrage:Voter ID:server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9 Address:0dad452b-68e0-c2e4-90d2-3f90baeab6c9} {Suffrage:Voter ID:server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4 Address:2b5444f2-c2c5-d465-f6d5-c4f4551af8d4}]" +testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.551Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: entering follower state: follower="Node at 70ef33c8-8204-4afb-87b1-6fabdcbe3ced [Follower]" leader= + testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.551Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: entering follower state: follower="Node at 70ef33c8-8204-4afb-87b1-6fabdcbe3ced [Follower]" leader= +testLoggerAdapter verbose: server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: 2023-08-02T15:12:59.551Z [INFO] server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced Address:70ef33c8-8204-4afb-87b1-6fabdcbe3ced} {Suffrage:Voter ID:server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9 Address:0dad452b-68e0-c2e4-90d2-3f90baeab6c9} {Suffrage:Voter ID:server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4 Address:2b5444f2-c2c5-d465-f6d5-c4f4551af8d4}]" +testLoggerAdapter verbose: server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9: 2023-08-02T15:12:59.551Z [INFO] server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9: entering follower state: follower="Node at 0dad452b-68e0-c2e4-90d2-3f90baeab6c9 [Follower]" leader= + testing.go:149: server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: 2023-08-02T15:12:59.551Z [INFO] server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced Address:70ef33c8-8204-4afb-87b1-6fabdcbe3ced} {Suffrage:Voter ID:server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9 Address:0dad452b-68e0-c2e4-90d2-3f90baeab6c9} {Suffrage:Voter ID:server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4 Address:2b5444f2-c2c5-d465-f6d5-c4f4551af8d4}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:59.551Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:12:59.551Z [INFO] starting stability test: raft-state=Leader + testing.go:149: server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9: 2023-08-02T15:12:59.551Z [INFO] server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9: entering follower state: follower="Node at 0dad452b-68e0-c2e4-90d2-3f90baeab6c9 [Follower]" leader= +testLoggerAdapter verbose: server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: 2023-08-02T15:12:59.551Z [INFO] server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: entering follower state: follower="Node at 2b5444f2-c2c5-d465-f6d5-c4f4551af8d4 [Follower]" leader= + testing.go:149: server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: 2023-08-02T15:12:59.551Z [INFO] server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: entering follower state: follower="Node at 2b5444f2-c2c5-d465-f6d5-c4f4551af8d4 [Follower]" leader= +testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.607Z [WARN] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.607Z [WARN] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.607Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: entering candidate state: node="Node at 70ef33c8-8204-4afb-87b1-6fabdcbe3ced [Candidate]" term=2 + testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.607Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: entering candidate state: node="Node at 70ef33c8-8204-4afb-87b1-6fabdcbe3ced [Candidate]" term=2 +testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.608Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: election won: tally=2 + testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.608Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: election won: tally=2 +testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.608Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: entering leader state: leader="Node at 70ef33c8-8204-4afb-87b1-6fabdcbe3ced [Leader]" + testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.608Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: entering leader state: leader="Node at 70ef33c8-8204-4afb-87b1-6fabdcbe3ced [Leader]" +testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.608Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: added peer, starting replication: peer=server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9 + testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.608Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: added peer, starting replication: peer=server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9 +testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.608Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: added peer, starting replication: peer=server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4 + testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.608Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: added peer, starting replication: peer=server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4 +testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.608Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: pipelining replication: peer="{Voter server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4 2b5444f2-c2c5-d465-f6d5-c4f4551af8d4}" + testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.608Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: pipelining replication: peer="{Voter server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4 2b5444f2-c2c5-d465-f6d5-c4f4551af8d4}" +testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.608Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: pipelining replication: peer="{Voter server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9 0dad452b-68e0-c2e4-90d2-3f90baeab6c9}" + testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.608Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: pipelining replication: peer="{Voter server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9 0dad452b-68e0-c2e4-90d2-3f90baeab6c9}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:59.713Z [INFO] stable state for Leader reached at 2023-08-02 15:12:59.608344187 +0000 UTC m=+11.704280554 (1 nodes), 56.54741ms from start of poll, 57.359673ms from cluster start. Timeout at 2023-08-02 15:12:59.713425547 +0000 UTC m=+11.809361898, 105.081344ms after stability + testing.go:149: cluster: 2023-08-02T15:12:59.713Z [INFO] stable state for Leader reached at 2023-08-02 15:12:59.608344187 +0000 UTC m=+11.704280554 (1 nodes), 56.54741ms from start of poll, 57.359673ms from cluster start. Timeout at 2023-08-02 15:12:59.713425547 +0000 UTC m=+11.809361898, 105.081344ms after stability +testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.714Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: starting snapshot up to: index=12 + testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.714Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: starting snapshot up to: index=12 + testing.go:153: 2023-08-02T15:12:59.714Z [INFO] creating new snapshot: path=/tmp/raft860841061/snapshots/2-12-1690989179714.tmp +testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.714Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: snapshot complete up to: index=12 + testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.714Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: snapshot complete up to: index=12 + testing.go:153: 2023-08-02T15:12:59.715Z [INFO] creating new snapshot: path=/tmp/raft860841061/snapshots/2-1013-1690989179715.tmp +testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.715Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: copied to local snapshot: bytes=71 + testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.715Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.720Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: restored user snapshot: index=1 + testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.720Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: restored user snapshot: index=1 +testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.720Z [ERROR] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: failed to get log: index=23 error="log not found" + testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.720Z [ERROR] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.720Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: aborting pipeline replication: peer="{Voter server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4 2b5444f2-c2c5-d465-f6d5-c4f4551af8d4}" + testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.720Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: aborting pipeline replication: peer="{Voter server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4 2b5444f2-c2c5-d465-f6d5-c4f4551af8d4}" +testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.720Z [ERROR] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: failed to get log: index=23 error="log not found" + testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.720Z [ERROR] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.721Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: aborting pipeline replication: peer="{Voter server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9 0dad452b-68e0-c2e4-90d2-3f90baeab6c9}" + testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.721Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: aborting pipeline replication: peer="{Voter server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9 0dad452b-68e0-c2e4-90d2-3f90baeab6c9}" +testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.727Z [ERROR] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: failed to get log: index=23 error="log not found" + testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.727Z [ERROR] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: failed to get log: index=23 error="log not found" + testing.go:153: 2023-08-02T15:12:59.728Z [INFO] creating new snapshot: path=/tmp/raft211612521/snapshots/2-1013-1690989179728.tmp +testLoggerAdapter verbose: server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: 2023-08-02T15:12:59.729Z [INFO] server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: copied to local snapshot: bytes=71 + testing.go:149: server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: 2023-08-02T15:12:59.729Z [INFO] server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: 2023-08-02T15:12:59.729Z [INFO] server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: Installed remote snapshot + testing.go:149: server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: 2023-08-02T15:12:59.729Z [INFO] server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: Installed remote snapshot +testLoggerAdapter verbose: server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: 2023-08-02T15:12:59.730Z [WARN] server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: skipping application of old log: index=22 + testing.go:149: server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: 2023-08-02T15:12:59.730Z [WARN] server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: skipping application of old log: index=22 +testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.730Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: pipelining replication: peer="{Voter server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4 2b5444f2-c2c5-d465-f6d5-c4f4551af8d4}" + testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.730Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: pipelining replication: peer="{Voter server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4 2b5444f2-c2c5-d465-f6d5-c4f4551af8d4}" +testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.730Z [ERROR] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: failed to get log: index=23 error="log not found" + testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.730Z [ERROR] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: failed to get log: index=23 error="log not found" + testing.go:153: 2023-08-02T15:12:59.730Z [INFO] creating new snapshot: path=/tmp/raft786216927/snapshots/2-1013-1690989179730.tmp +testLoggerAdapter verbose: server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9: 2023-08-02T15:12:59.731Z [INFO] server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9: copied to local snapshot: bytes=71 + testing.go:149: server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9: 2023-08-02T15:12:59.731Z [INFO] server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9: 2023-08-02T15:12:59.731Z [INFO] server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9: Installed remote snapshot + testing.go:149: server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9: 2023-08-02T15:12:59.731Z [INFO] server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9: Installed remote snapshot +testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.731Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: pipelining replication: peer="{Voter server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9 0dad452b-68e0-c2e4-90d2-3f90baeab6c9}" + testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.731Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: pipelining replication: peer="{Voter server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9 0dad452b-68e0-c2e4-90d2-3f90baeab6c9}" +testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.745Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: aborting pipeline replication: peer="{Voter server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4 2b5444f2-c2c5-d465-f6d5-c4f4551af8d4}" + testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.745Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: aborting pipeline replication: peer="{Voter server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4 2b5444f2-c2c5-d465-f6d5-c4f4551af8d4}" +testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.745Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: aborting pipeline replication: peer="{Voter server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9 0dad452b-68e0-c2e4-90d2-3f90baeab6c9}" + testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.745Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: aborting pipeline replication: peer="{Voter server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9 0dad452b-68e0-c2e4-90d2-3f90baeab6c9}" +testLoggerAdapter verbose: server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee: 2023-08-02T15:12:59.747Z [INFO] server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee Address:a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee} {Suffrage:Voter ID:server-fcba1207-da01-9c09-257d-4ce22241f775 Address:fcba1207-da01-9c09-257d-4ce22241f775} {Suffrage:Voter ID:server-7f10c0ec-9758-2914-9b53-04ccadf5b75c Address:7f10c0ec-9758-2914-9b53-04ccadf5b75c}]" + testing.go:149: server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee: 2023-08-02T15:12:59.747Z [INFO] server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee Address:a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee} {Suffrage:Voter ID:server-fcba1207-da01-9c09-257d-4ce22241f775 Address:fcba1207-da01-9c09-257d-4ce22241f775} {Suffrage:Voter ID:server-7f10c0ec-9758-2914-9b53-04ccadf5b75c Address:7f10c0ec-9758-2914-9b53-04ccadf5b75c}]" +testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.747Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee Address:a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee} {Suffrage:Voter ID:server-fcba1207-da01-9c09-257d-4ce22241f775 Address:fcba1207-da01-9c09-257d-4ce22241f775} {Suffrage:Voter ID:server-7f10c0ec-9758-2914-9b53-04ccadf5b75c Address:7f10c0ec-9758-2914-9b53-04ccadf5b75c}]" + testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.747Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee Address:a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee} {Suffrage:Voter ID:server-fcba1207-da01-9c09-257d-4ce22241f775 Address:fcba1207-da01-9c09-257d-4ce22241f775} {Suffrage:Voter ID:server-7f10c0ec-9758-2914-9b53-04ccadf5b75c Address:7f10c0ec-9758-2914-9b53-04ccadf5b75c}]" +testLoggerAdapter verbose: server-7f10c0ec-9758-2914-9b53-04ccadf5b75c: 2023-08-02T15:12:59.747Z [INFO] server-7f10c0ec-9758-2914-9b53-04ccadf5b75c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee Address:a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee} {Suffrage:Voter ID:server-fcba1207-da01-9c09-257d-4ce22241f775 Address:fcba1207-da01-9c09-257d-4ce22241f775} {Suffrage:Voter ID:server-7f10c0ec-9758-2914-9b53-04ccadf5b75c Address:7f10c0ec-9758-2914-9b53-04ccadf5b75c}]" + testing.go:149: server-7f10c0ec-9758-2914-9b53-04ccadf5b75c: 2023-08-02T15:12:59.747Z [INFO] server-7f10c0ec-9758-2914-9b53-04ccadf5b75c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee Address:a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee} {Suffrage:Voter ID:server-fcba1207-da01-9c09-257d-4ce22241f775 Address:fcba1207-da01-9c09-257d-4ce22241f775} {Suffrage:Voter ID:server-7f10c0ec-9758-2914-9b53-04ccadf5b75c Address:7f10c0ec-9758-2914-9b53-04ccadf5b75c}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:59.747Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:12:59.747Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.747Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: entering follower state: follower="Node at fcba1207-da01-9c09-257d-4ce22241f775 [Follower]" leader= + testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.747Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: entering follower state: follower="Node at fcba1207-da01-9c09-257d-4ce22241f775 [Follower]" leader= +testLoggerAdapter verbose: server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee: 2023-08-02T15:12:59.747Z [INFO] server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee: entering follower state: follower="Node at a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee [Follower]" leader= + testing.go:149: server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee: 2023-08-02T15:12:59.747Z [INFO] server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee: entering follower state: follower="Node at a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee [Follower]" leader= +testLoggerAdapter verbose: server-7f10c0ec-9758-2914-9b53-04ccadf5b75c: 2023-08-02T15:12:59.747Z [INFO] server-7f10c0ec-9758-2914-9b53-04ccadf5b75c: entering follower state: follower="Node at 7f10c0ec-9758-2914-9b53-04ccadf5b75c [Follower]" leader= + testing.go:149: server-7f10c0ec-9758-2914-9b53-04ccadf5b75c: 2023-08-02T15:12:59.747Z [INFO] server-7f10c0ec-9758-2914-9b53-04ccadf5b75c: entering follower state: follower="Node at 7f10c0ec-9758-2914-9b53-04ccadf5b75c [Follower]" leader= +testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.820Z [WARN] server-fcba1207-da01-9c09-257d-4ce22241f775: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.820Z [WARN] server-fcba1207-da01-9c09-257d-4ce22241f775: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.821Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: entering candidate state: node="Node at fcba1207-da01-9c09-257d-4ce22241f775 [Candidate]" term=2 + testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.821Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: entering candidate state: node="Node at fcba1207-da01-9c09-257d-4ce22241f775 [Candidate]" term=2 +testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.821Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: election won: tally=2 + testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.821Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: election won: tally=2 +testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.821Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: entering leader state: leader="Node at fcba1207-da01-9c09-257d-4ce22241f775 [Leader]" + testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.821Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: entering leader state: leader="Node at fcba1207-da01-9c09-257d-4ce22241f775 [Leader]" +testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.821Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: added peer, starting replication: peer=server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee + testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.821Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: added peer, starting replication: peer=server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee +testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.821Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: added peer, starting replication: peer=server-7f10c0ec-9758-2914-9b53-04ccadf5b75c + testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.821Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: added peer, starting replication: peer=server-7f10c0ec-9758-2914-9b53-04ccadf5b75c +testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.821Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: pipelining replication: peer="{Voter server-7f10c0ec-9758-2914-9b53-04ccadf5b75c 7f10c0ec-9758-2914-9b53-04ccadf5b75c}" + testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.821Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: pipelining replication: peer="{Voter server-7f10c0ec-9758-2914-9b53-04ccadf5b75c 7f10c0ec-9758-2914-9b53-04ccadf5b75c}" +testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.821Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: pipelining replication: peer="{Voter server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee}" + testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.821Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: pipelining replication: peer="{Voter server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:59.926Z [INFO] stable state for Leader reached at 2023-08-02 15:12:59.821294997 +0000 UTC m=+11.917231331 (1 nodes), 73.794891ms from start of poll, 74.205928ms from cluster start. Timeout at 2023-08-02 15:12:59.926323397 +0000 UTC m=+12.022259732, 105.028401ms after stability + testing.go:149: cluster: 2023-08-02T15:12:59.926Z [INFO] stable state for Leader reached at 2023-08-02 15:12:59.821294997 +0000 UTC m=+11.917231331 (1 nodes), 73.794891ms from start of poll, 74.205928ms from cluster start. Timeout at 2023-08-02 15:12:59.926323397 +0000 UTC m=+12.022259732, 105.028401ms after stability +testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.926Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: starting snapshot up to: index=12 + testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.926Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: starting snapshot up to: index=12 + testing.go:153: 2023-08-02T15:12:59.926Z [INFO] creating new snapshot: path=/tmp/raft370563757/snapshots/2-12-1690989179926.tmp +testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.927Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: snapshot complete up to: index=12 + testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.927Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: snapshot complete up to: index=12 + testing.go:153: 2023-08-02T15:12:59.927Z [INFO] creating new snapshot: path=/tmp/raft370563757/snapshots/2-10013-1690989179927.tmp +testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.928Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: copied to local snapshot: bytes=71 + testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.928Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.928Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: restored user snapshot: index=1 + testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.928Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: restored user snapshot: index=1 +testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.928Z [ERROR] server-fcba1207-da01-9c09-257d-4ce22241f775: failed to get log: index=23 error="log not found" + testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.928Z [ERROR] server-fcba1207-da01-9c09-257d-4ce22241f775: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.928Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: aborting pipeline replication: peer="{Voter server-7f10c0ec-9758-2914-9b53-04ccadf5b75c 7f10c0ec-9758-2914-9b53-04ccadf5b75c}" + testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.928Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: aborting pipeline replication: peer="{Voter server-7f10c0ec-9758-2914-9b53-04ccadf5b75c 7f10c0ec-9758-2914-9b53-04ccadf5b75c}" +testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.928Z [ERROR] server-fcba1207-da01-9c09-257d-4ce22241f775: failed to get log: index=23 error="log not found" + testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.928Z [ERROR] server-fcba1207-da01-9c09-257d-4ce22241f775: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.928Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: aborting pipeline replication: peer="{Voter server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee}" + testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.928Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: aborting pipeline replication: peer="{Voter server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee}" +testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.936Z [ERROR] server-fcba1207-da01-9c09-257d-4ce22241f775: failed to get log: index=23 error="log not found" + testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.936Z [ERROR] server-fcba1207-da01-9c09-257d-4ce22241f775: failed to get log: index=23 error="log not found" + testing.go:153: 2023-08-02T15:12:59.937Z [INFO] creating new snapshot: path=/tmp/raft659919235/snapshots/2-10013-1690989179937.tmp +testLoggerAdapter verbose: server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee: 2023-08-02T15:12:59.937Z [INFO] server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee: copied to local snapshot: bytes=71 + testing.go:149: server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee: 2023-08-02T15:12:59.937Z [INFO] server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee: 2023-08-02T15:12:59.937Z [INFO] server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee: Installed remote snapshot + testing.go:149: server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee: 2023-08-02T15:12:59.937Z [INFO] server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee: Installed remote snapshot +testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.937Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: pipelining replication: peer="{Voter server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee}" + testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.937Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: pipelining replication: peer="{Voter server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee}" +testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.938Z [ERROR] server-fcba1207-da01-9c09-257d-4ce22241f775: failed to get log: index=23 error="log not found" + testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.938Z [ERROR] server-fcba1207-da01-9c09-257d-4ce22241f775: failed to get log: index=23 error="log not found" + testing.go:153: 2023-08-02T15:12:59.939Z [INFO] creating new snapshot: path=/tmp/raft166029159/snapshots/2-10013-1690989179939.tmp +testLoggerAdapter verbose: server-7f10c0ec-9758-2914-9b53-04ccadf5b75c: 2023-08-02T15:12:59.939Z [INFO] server-7f10c0ec-9758-2914-9b53-04ccadf5b75c: copied to local snapshot: bytes=71 + testing.go:149: server-7f10c0ec-9758-2914-9b53-04ccadf5b75c: 2023-08-02T15:12:59.939Z [INFO] server-7f10c0ec-9758-2914-9b53-04ccadf5b75c: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-7f10c0ec-9758-2914-9b53-04ccadf5b75c: 2023-08-02T15:12:59.939Z [INFO] server-7f10c0ec-9758-2914-9b53-04ccadf5b75c: Installed remote snapshot + testing.go:149: server-7f10c0ec-9758-2914-9b53-04ccadf5b75c: 2023-08-02T15:12:59.939Z [INFO] server-7f10c0ec-9758-2914-9b53-04ccadf5b75c: Installed remote snapshot +testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.939Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: pipelining replication: peer="{Voter server-7f10c0ec-9758-2914-9b53-04ccadf5b75c 7f10c0ec-9758-2914-9b53-04ccadf5b75c}" + testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.939Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: pipelining replication: peer="{Voter server-7f10c0ec-9758-2914-9b53-04ccadf5b75c 7f10c0ec-9758-2914-9b53-04ccadf5b75c}" +testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.954Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: aborting pipeline replication: peer="{Voter server-7f10c0ec-9758-2914-9b53-04ccadf5b75c 7f10c0ec-9758-2914-9b53-04ccadf5b75c}" + testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.954Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: aborting pipeline replication: peer="{Voter server-7f10c0ec-9758-2914-9b53-04ccadf5b75c 7f10c0ec-9758-2914-9b53-04ccadf5b75c}" +testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.954Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: aborting pipeline replication: peer="{Voter server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee}" + testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.954Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: aborting pipeline replication: peer="{Voter server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee}" +--- PASS: TestRaft_UserRestore (1.18s) === RUN TestRaft_SendSnapshotFollower -testLoggerAdapter verbose: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:30.997Z [INFO] server-cafb45fc-fcc8-9960-5632-ff7c74639861: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cafb45fc-fcc8-9960-5632-ff7c74639861 Address:cafb45fc-fcc8-9960-5632-ff7c74639861} {Suffrage:Voter ID:server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f Address:fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f} {Suffrage:Voter ID:server-cbf3b14d-d4cd-b754-961b-ff206a801833 Address:cbf3b14d-d4cd-b754-961b-ff206a801833}]" - testing.go:149: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:30.997Z [INFO] server-cafb45fc-fcc8-9960-5632-ff7c74639861: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cafb45fc-fcc8-9960-5632-ff7c74639861 Address:cafb45fc-fcc8-9960-5632-ff7c74639861} {Suffrage:Voter ID:server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f Address:fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f} {Suffrage:Voter ID:server-cbf3b14d-d4cd-b754-961b-ff206a801833 Address:cbf3b14d-d4cd-b754-961b-ff206a801833}]" -testLoggerAdapter verbose: server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f: 2024-09-03T21:34:30.997Z [INFO] server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cafb45fc-fcc8-9960-5632-ff7c74639861 Address:cafb45fc-fcc8-9960-5632-ff7c74639861} {Suffrage:Voter ID:server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f Address:fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f} {Suffrage:Voter ID:server-cbf3b14d-d4cd-b754-961b-ff206a801833 Address:cbf3b14d-d4cd-b754-961b-ff206a801833}]" - testing.go:149: server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f: 2024-09-03T21:34:30.997Z [INFO] server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cafb45fc-fcc8-9960-5632-ff7c74639861 Address:cafb45fc-fcc8-9960-5632-ff7c74639861} {Suffrage:Voter ID:server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f Address:fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f} {Suffrage:Voter ID:server-cbf3b14d-d4cd-b754-961b-ff206a801833 Address:cbf3b14d-d4cd-b754-961b-ff206a801833}]" -testLoggerAdapter verbose: server-cbf3b14d-d4cd-b754-961b-ff206a801833: 2024-09-03T21:34:30.997Z [INFO] server-cbf3b14d-d4cd-b754-961b-ff206a801833: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cafb45fc-fcc8-9960-5632-ff7c74639861 Address:cafb45fc-fcc8-9960-5632-ff7c74639861} {Suffrage:Voter ID:server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f Address:fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f} {Suffrage:Voter ID:server-cbf3b14d-d4cd-b754-961b-ff206a801833 Address:cbf3b14d-d4cd-b754-961b-ff206a801833}]" - testing.go:149: server-cbf3b14d-d4cd-b754-961b-ff206a801833: 2024-09-03T21:34:30.997Z [INFO] server-cbf3b14d-d4cd-b754-961b-ff206a801833: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cafb45fc-fcc8-9960-5632-ff7c74639861 Address:cafb45fc-fcc8-9960-5632-ff7c74639861} {Suffrage:Voter ID:server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f Address:fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f} {Suffrage:Voter ID:server-cbf3b14d-d4cd-b754-961b-ff206a801833 Address:cbf3b14d-d4cd-b754-961b-ff206a801833}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:30.997Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-09-03T21:34:30.997Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:30.997Z [INFO] server-cafb45fc-fcc8-9960-5632-ff7c74639861: entering follower state: follower="Node at cafb45fc-fcc8-9960-5632-ff7c74639861 [Follower]" leader= - testing.go:149: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:30.997Z [INFO] server-cafb45fc-fcc8-9960-5632-ff7c74639861: entering follower state: follower="Node at cafb45fc-fcc8-9960-5632-ff7c74639861 [Follower]" leader= -testLoggerAdapter verbose: server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f: 2024-09-03T21:34:30.997Z [INFO] server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f: entering follower state: follower="Node at fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f [Follower]" leader= - testing.go:149: server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f: 2024-09-03T21:34:30.997Z [INFO] server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f: entering follower state: follower="Node at fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f [Follower]" leader= -testLoggerAdapter verbose: server-cbf3b14d-d4cd-b754-961b-ff206a801833: 2024-09-03T21:34:30.997Z [INFO] server-cbf3b14d-d4cd-b754-961b-ff206a801833: entering follower state: follower="Node at cbf3b14d-d4cd-b754-961b-ff206a801833 [Follower]" leader= - testing.go:149: server-cbf3b14d-d4cd-b754-961b-ff206a801833: 2024-09-03T21:34:30.997Z [INFO] server-cbf3b14d-d4cd-b754-961b-ff206a801833: entering follower state: follower="Node at cbf3b14d-d4cd-b754-961b-ff206a801833 [Follower]" leader= -testLoggerAdapter verbose: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:31.075Z [WARN] server-cafb45fc-fcc8-9960-5632-ff7c74639861: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:31.075Z [WARN] server-cafb45fc-fcc8-9960-5632-ff7c74639861: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:31.075Z [INFO] server-cafb45fc-fcc8-9960-5632-ff7c74639861: entering candidate state: node="Node at cafb45fc-fcc8-9960-5632-ff7c74639861 [Candidate]" term=2 - testing.go:149: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:31.075Z [INFO] server-cafb45fc-fcc8-9960-5632-ff7c74639861: entering candidate state: node="Node at cafb45fc-fcc8-9960-5632-ff7c74639861 [Candidate]" term=2 -testLoggerAdapter verbose: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:31.075Z [INFO] server-cafb45fc-fcc8-9960-5632-ff7c74639861: election won: tally=2 - testing.go:149: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:31.075Z [INFO] server-cafb45fc-fcc8-9960-5632-ff7c74639861: election won: tally=2 -testLoggerAdapter verbose: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:31.075Z [INFO] server-cafb45fc-fcc8-9960-5632-ff7c74639861: entering leader state: leader="Node at cafb45fc-fcc8-9960-5632-ff7c74639861 [Leader]" - testing.go:149: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:31.075Z [INFO] server-cafb45fc-fcc8-9960-5632-ff7c74639861: entering leader state: leader="Node at cafb45fc-fcc8-9960-5632-ff7c74639861 [Leader]" -testLoggerAdapter verbose: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:31.075Z [INFO] server-cafb45fc-fcc8-9960-5632-ff7c74639861: added peer, starting replication: peer=server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f - testing.go:149: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:31.075Z [INFO] server-cafb45fc-fcc8-9960-5632-ff7c74639861: added peer, starting replication: peer=server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f -testLoggerAdapter verbose: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:31.075Z [INFO] server-cafb45fc-fcc8-9960-5632-ff7c74639861: added peer, starting replication: peer=server-cbf3b14d-d4cd-b754-961b-ff206a801833 - testing.go:149: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:31.075Z [INFO] server-cafb45fc-fcc8-9960-5632-ff7c74639861: added peer, starting replication: peer=server-cbf3b14d-d4cd-b754-961b-ff206a801833 -testLoggerAdapter verbose: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:31.075Z [INFO] server-cafb45fc-fcc8-9960-5632-ff7c74639861: pipelining replication: peer="{Voter server-cbf3b14d-d4cd-b754-961b-ff206a801833 cbf3b14d-d4cd-b754-961b-ff206a801833}" - testing.go:149: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:31.075Z [INFO] server-cafb45fc-fcc8-9960-5632-ff7c74639861: pipelining replication: peer="{Voter server-cbf3b14d-d4cd-b754-961b-ff206a801833 cbf3b14d-d4cd-b754-961b-ff206a801833}" -testLoggerAdapter verbose: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:31.075Z [INFO] server-cafb45fc-fcc8-9960-5632-ff7c74639861: pipelining replication: peer="{Voter server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f}" - testing.go:149: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:31.075Z [INFO] server-cafb45fc-fcc8-9960-5632-ff7c74639861: pipelining replication: peer="{Voter server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:31.180Z [INFO] stable state for Follower reached at 2024-09-03 21:34:31.075801287 +0000 UTC m=+12.122458420 (2 nodes), 78.533933ms from start of poll, 78.878711ms from cluster start. Timeout at 2024-09-03 21:34:31.180901495 +0000 UTC m=+12.227558625, 105.100205ms after stability - testing.go:149: cluster: 2024-09-03T21:34:31.180Z [INFO] stable state for Follower reached at 2024-09-03 21:34:31.075801287 +0000 UTC m=+12.122458420 (2 nodes), 78.533933ms from start of poll, 78.878711ms from cluster start. Timeout at 2024-09-03 21:34:31.180901495 +0000 UTC m=+12.227558625, 105.100205ms after stability -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:31.181Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:31.181Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:31.286Z [INFO] stable state for Leader reached at 2024-09-03 21:34:31.181054908 +0000 UTC m=+12.227712050 (1 nodes), 1.038µs from start of poll, 184.132341ms from cluster start. Timeout at 2024-09-03 21:34:31.286133935 +0000 UTC m=+12.332791069, 105.079019ms after stability - testing.go:149: cluster: 2024-09-03T21:34:31.286Z [INFO] stable state for Leader reached at 2024-09-03 21:34:31.181054908 +0000 UTC m=+12.227712050 (1 nodes), 1.038µs from start of poll, 184.132341ms from cluster start. Timeout at 2024-09-03 21:34:31.286133935 +0000 UTC m=+12.332791069, 105.079019ms after stability -testLoggerAdapter verbose: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:31.286Z [ERROR] server-cafb45fc-fcc8-9960-5632-ff7c74639861: failed to pipeline appendEntries: peer="{Voter server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f}" error="append pipeline closed" - testing.go:149: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:31.286Z [ERROR] server-cafb45fc-fcc8-9960-5632-ff7c74639861: failed to pipeline appendEntries: peer="{Voter server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f}" error="append pipeline closed" -testLoggerAdapter verbose: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:31.286Z [INFO] server-cafb45fc-fcc8-9960-5632-ff7c74639861: aborting pipeline replication: peer="{Voter server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f}" - testing.go:149: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:31.286Z [INFO] server-cafb45fc-fcc8-9960-5632-ff7c74639861: aborting pipeline replication: peer="{Voter server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f}" -testLoggerAdapter verbose: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:31.286Z [ERROR] server-cafb45fc-fcc8-9960-5632-ff7c74639861: failed to appendEntries to: peer="{Voter server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f}" error="failed to connect to peer: fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f" - testing.go:149: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:31.286Z [ERROR] server-cafb45fc-fcc8-9960-5632-ff7c74639861: failed to appendEntries to: peer="{Voter server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f}" error="failed to connect to peer: fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f" +testLoggerAdapter verbose: server-90180b34-5b90-ebe3-c828-5fe59d8e596a: 2023-08-02T15:12:59.956Z [INFO] server-90180b34-5b90-ebe3-c828-5fe59d8e596a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90180b34-5b90-ebe3-c828-5fe59d8e596a Address:90180b34-5b90-ebe3-c828-5fe59d8e596a} {Suffrage:Voter ID:server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680 Address:5e1a394b-e983-b1a6-50ae-99f0a0ab8680} {Suffrage:Voter ID:server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4 Address:4dd7916e-4fcc-1112-eaed-08c8cc08fef4}]" + testing.go:149: server-90180b34-5b90-ebe3-c828-5fe59d8e596a: 2023-08-02T15:12:59.956Z [INFO] server-90180b34-5b90-ebe3-c828-5fe59d8e596a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90180b34-5b90-ebe3-c828-5fe59d8e596a Address:90180b34-5b90-ebe3-c828-5fe59d8e596a} {Suffrage:Voter ID:server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680 Address:5e1a394b-e983-b1a6-50ae-99f0a0ab8680} {Suffrage:Voter ID:server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4 Address:4dd7916e-4fcc-1112-eaed-08c8cc08fef4}]" +testLoggerAdapter verbose: server-90180b34-5b90-ebe3-c828-5fe59d8e596a: 2023-08-02T15:12:59.957Z [INFO] server-90180b34-5b90-ebe3-c828-5fe59d8e596a: entering follower state: follower="Node at 90180b34-5b90-ebe3-c828-5fe59d8e596a [Follower]" leader= + testing.go:149: server-90180b34-5b90-ebe3-c828-5fe59d8e596a: 2023-08-02T15:12:59.957Z [INFO] server-90180b34-5b90-ebe3-c828-5fe59d8e596a: entering follower state: follower="Node at 90180b34-5b90-ebe3-c828-5fe59d8e596a [Follower]" leader= +testLoggerAdapter verbose: server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: 2023-08-02T15:12:59.957Z [INFO] server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90180b34-5b90-ebe3-c828-5fe59d8e596a Address:90180b34-5b90-ebe3-c828-5fe59d8e596a} {Suffrage:Voter ID:server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680 Address:5e1a394b-e983-b1a6-50ae-99f0a0ab8680} {Suffrage:Voter ID:server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4 Address:4dd7916e-4fcc-1112-eaed-08c8cc08fef4}]" + testing.go:149: server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: 2023-08-02T15:12:59.957Z [INFO] server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90180b34-5b90-ebe3-c828-5fe59d8e596a Address:90180b34-5b90-ebe3-c828-5fe59d8e596a} {Suffrage:Voter ID:server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680 Address:5e1a394b-e983-b1a6-50ae-99f0a0ab8680} {Suffrage:Voter ID:server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4 Address:4dd7916e-4fcc-1112-eaed-08c8cc08fef4}]" +testLoggerAdapter verbose: server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: 2023-08-02T15:12:59.957Z [INFO] server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: entering follower state: follower="Node at 5e1a394b-e983-b1a6-50ae-99f0a0ab8680 [Follower]" leader= + testing.go:149: server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: 2023-08-02T15:12:59.957Z [INFO] server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: entering follower state: follower="Node at 5e1a394b-e983-b1a6-50ae-99f0a0ab8680 [Follower]" leader= +testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:12:59.957Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90180b34-5b90-ebe3-c828-5fe59d8e596a Address:90180b34-5b90-ebe3-c828-5fe59d8e596a} {Suffrage:Voter ID:server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680 Address:5e1a394b-e983-b1a6-50ae-99f0a0ab8680} {Suffrage:Voter ID:server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4 Address:4dd7916e-4fcc-1112-eaed-08c8cc08fef4}]" + testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:12:59.957Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90180b34-5b90-ebe3-c828-5fe59d8e596a Address:90180b34-5b90-ebe3-c828-5fe59d8e596a} {Suffrage:Voter ID:server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680 Address:5e1a394b-e983-b1a6-50ae-99f0a0ab8680} {Suffrage:Voter ID:server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4 Address:4dd7916e-4fcc-1112-eaed-08c8cc08fef4}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:12:59.957Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-08-02T15:12:59.957Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:12:59.957Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: entering follower state: follower="Node at 4dd7916e-4fcc-1112-eaed-08c8cc08fef4 [Follower]" leader= + testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:12:59.957Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: entering follower state: follower="Node at 4dd7916e-4fcc-1112-eaed-08c8cc08fef4 [Follower]" leader= +testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.042Z [WARN] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.042Z [WARN] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.042Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: entering candidate state: node="Node at 4dd7916e-4fcc-1112-eaed-08c8cc08fef4 [Candidate]" term=2 + testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.042Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: entering candidate state: node="Node at 4dd7916e-4fcc-1112-eaed-08c8cc08fef4 [Candidate]" term=2 +testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.042Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: election won: tally=2 + testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.042Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: election won: tally=2 +testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.042Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: entering leader state: leader="Node at 4dd7916e-4fcc-1112-eaed-08c8cc08fef4 [Leader]" + testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.042Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: entering leader state: leader="Node at 4dd7916e-4fcc-1112-eaed-08c8cc08fef4 [Leader]" +testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.042Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: added peer, starting replication: peer=server-90180b34-5b90-ebe3-c828-5fe59d8e596a + testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.042Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: added peer, starting replication: peer=server-90180b34-5b90-ebe3-c828-5fe59d8e596a +testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.042Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: added peer, starting replication: peer=server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680 + testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.042Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: added peer, starting replication: peer=server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680 +testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.042Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: pipelining replication: peer="{Voter server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680 5e1a394b-e983-b1a6-50ae-99f0a0ab8680}" + testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.042Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: pipelining replication: peer="{Voter server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680 5e1a394b-e983-b1a6-50ae-99f0a0ab8680}" +testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.043Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: pipelining replication: peer="{Voter server-90180b34-5b90-ebe3-c828-5fe59d8e596a 90180b34-5b90-ebe3-c828-5fe59d8e596a}" + testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.043Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: pipelining replication: peer="{Voter server-90180b34-5b90-ebe3-c828-5fe59d8e596a 90180b34-5b90-ebe3-c828-5fe59d8e596a}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:00.149Z [INFO] stable state for Follower reached at 2023-08-02 15:13:00.043136359 +0000 UTC m=+12.139072697 (2 nodes), 85.47898ms from start of poll, 86.773127ms from cluster start. Timeout at 2023-08-02 15:13:00.149135193 +0000 UTC m=+12.245071708, 105.999011ms after stability + testing.go:149: cluster: 2023-08-02T15:13:00.149Z [INFO] stable state for Follower reached at 2023-08-02 15:13:00.043136359 +0000 UTC m=+12.139072697 (2 nodes), 85.47898ms from start of poll, 86.773127ms from cluster start. Timeout at 2023-08-02 15:13:00.149135193 +0000 UTC m=+12.245071708, 105.999011ms after stability +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:00.149Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:13:00.149Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:00.254Z [INFO] stable state for Leader reached at 2023-08-02 15:13:00.149356647 +0000 UTC m=+12.245292975 (1 nodes), 2.1µs from start of poll, 192.993405ms from cluster start. Timeout at 2023-08-02 15:13:00.25444801 +0000 UTC m=+12.350384354, 105.091379ms after stability + testing.go:149: cluster: 2023-08-02T15:13:00.254Z [INFO] stable state for Leader reached at 2023-08-02 15:13:00.149356647 +0000 UTC m=+12.245292975 (1 nodes), 2.1µs from start of poll, 192.993405ms from cluster start. Timeout at 2023-08-02 15:13:00.25444801 +0000 UTC m=+12.350384354, 105.091379ms after stability +testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.254Z [ERROR] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: failed to pipeline appendEntries: peer="{Voter server-90180b34-5b90-ebe3-c828-5fe59d8e596a 90180b34-5b90-ebe3-c828-5fe59d8e596a}" error="append pipeline closed" + testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.254Z [ERROR] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: failed to pipeline appendEntries: peer="{Voter server-90180b34-5b90-ebe3-c828-5fe59d8e596a 90180b34-5b90-ebe3-c828-5fe59d8e596a}" error="append pipeline closed" +testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.254Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: aborting pipeline replication: peer="{Voter server-90180b34-5b90-ebe3-c828-5fe59d8e596a 90180b34-5b90-ebe3-c828-5fe59d8e596a}" + testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.254Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: aborting pipeline replication: peer="{Voter server-90180b34-5b90-ebe3-c828-5fe59d8e596a 90180b34-5b90-ebe3-c828-5fe59d8e596a}" +testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.254Z [ERROR] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: failed to appendEntries to: peer="{Voter server-90180b34-5b90-ebe3-c828-5fe59d8e596a 90180b34-5b90-ebe3-c828-5fe59d8e596a}" error="failed to connect to peer: 90180b34-5b90-ebe3-c828-5fe59d8e596a" + testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.254Z [ERROR] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: failed to appendEntries to: peer="{Voter server-90180b34-5b90-ebe3-c828-5fe59d8e596a 90180b34-5b90-ebe3-c828-5fe59d8e596a}" error="failed to connect to peer: 90180b34-5b90-ebe3-c828-5fe59d8e596a" +testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.255Z [ERROR] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: failed to heartbeat to: peer=90180b34-5b90-ebe3-c828-5fe59d8e596a error="failed to connect to peer: 90180b34-5b90-ebe3-c828-5fe59d8e596a" + testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.255Z [ERROR] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: failed to heartbeat to: peer=90180b34-5b90-ebe3-c828-5fe59d8e596a error="failed to connect to peer: 90180b34-5b90-ebe3-c828-5fe59d8e596a" raft_test.go:1320: [INFO] Finished apply without behind follower -testLoggerAdapter verbose: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:31.287Z [INFO] server-cafb45fc-fcc8-9960-5632-ff7c74639861: starting snapshot up to: index=102 - testing.go:149: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:31.287Z [INFO] server-cafb45fc-fcc8-9960-5632-ff7c74639861: starting snapshot up to: index=102 - testing.go:153: 2024-09-03T21:34:31.287Z [INFO] creating new snapshot: path=/tmp/raft193143802/snapshots/2-102-1725399271287.tmp -testLoggerAdapter verbose: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:31.287Z [INFO] server-cafb45fc-fcc8-9960-5632-ff7c74639861: compacting logs: from=1 to=92 - testing.go:149: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:31.287Z [INFO] server-cafb45fc-fcc8-9960-5632-ff7c74639861: compacting logs: from=1 to=92 -testLoggerAdapter verbose: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:31.288Z [INFO] server-cafb45fc-fcc8-9960-5632-ff7c74639861: snapshot complete up to: index=102 - testing.go:149: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:31.288Z [INFO] server-cafb45fc-fcc8-9960-5632-ff7c74639861: snapshot complete up to: index=102 -testLoggerAdapter verbose: server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f: 2024-09-03T21:34:31.288Z [ERROR] server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f: failed to take snapshot: error="nothing new to snapshot" - testing.go:149: server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f: 2024-09-03T21:34:31.288Z [ERROR] server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f: failed to take snapshot: error="nothing new to snapshot" -testLoggerAdapter verbose: server-cbf3b14d-d4cd-b754-961b-ff206a801833: 2024-09-03T21:34:31.288Z [INFO] server-cbf3b14d-d4cd-b754-961b-ff206a801833: starting snapshot up to: index=6 - testing.go:149: server-cbf3b14d-d4cd-b754-961b-ff206a801833: 2024-09-03T21:34:31.288Z [INFO] server-cbf3b14d-d4cd-b754-961b-ff206a801833: starting snapshot up to: index=6 - testing.go:153: 2024-09-03T21:34:31.288Z [INFO] creating new snapshot: path=/tmp/raft356941774/snapshots/2-6-1725399271288.tmp -testLoggerAdapter verbose: server-cbf3b14d-d4cd-b754-961b-ff206a801833: 2024-09-03T21:34:31.288Z [INFO] server-cbf3b14d-d4cd-b754-961b-ff206a801833: compacting logs: from=1 to=6 - testing.go:149: server-cbf3b14d-d4cd-b754-961b-ff206a801833: 2024-09-03T21:34:31.288Z [INFO] server-cbf3b14d-d4cd-b754-961b-ff206a801833: compacting logs: from=1 to=6 -testLoggerAdapter verbose: server-cbf3b14d-d4cd-b754-961b-ff206a801833: 2024-09-03T21:34:31.288Z [INFO] server-cbf3b14d-d4cd-b754-961b-ff206a801833: snapshot complete up to: index=6 - testing.go:149: server-cbf3b14d-d4cd-b754-961b-ff206a801833: 2024-09-03T21:34:31.288Z [INFO] server-cbf3b14d-d4cd-b754-961b-ff206a801833: snapshot complete up to: index=6 -testLoggerAdapter verbose: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:31.296Z [ERROR] server-cafb45fc-fcc8-9960-5632-ff7c74639861: failed to get log: index=2 error="log not found" - testing.go:149: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:31.296Z [ERROR] server-cafb45fc-fcc8-9960-5632-ff7c74639861: failed to get log: index=2 error="log not found" - testing.go:153: 2024-09-03T21:34:31.296Z [INFO] creating new snapshot: path=/tmp/raft451278652/snapshots/2-102-1725399271296.tmp -testLoggerAdapter verbose: server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f: 2024-09-03T21:34:31.297Z [INFO] server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f: copied to local snapshot: bytes=693 - testing.go:149: server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f: 2024-09-03T21:34:31.297Z [INFO] server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f: copied to local snapshot: bytes=693 -testLoggerAdapter verbose: server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f: 2024-09-03T21:34:31.297Z [INFO] server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f: Installed remote snapshot - testing.go:149: server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f: 2024-09-03T21:34:31.297Z [INFO] server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f: Installed remote snapshot -testLoggerAdapter verbose: server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f: 2024-09-03T21:34:31.297Z [WARN] server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f: skipping application of old log: index=102 - testing.go:149: server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f: 2024-09-03T21:34:31.297Z [WARN] server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f: skipping application of old log: index=102 -testLoggerAdapter verbose: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:31.297Z [INFO] server-cafb45fc-fcc8-9960-5632-ff7c74639861: pipelining replication: peer="{Voter server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f}" - testing.go:149: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:31.297Z [INFO] server-cafb45fc-fcc8-9960-5632-ff7c74639861: pipelining replication: peer="{Voter server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f}" -testLoggerAdapter verbose: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:31.298Z [INFO] server-cafb45fc-fcc8-9960-5632-ff7c74639861: aborting pipeline replication: peer="{Voter server-cbf3b14d-d4cd-b754-961b-ff206a801833 cbf3b14d-d4cd-b754-961b-ff206a801833}" - testing.go:149: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:31.298Z [INFO] server-cafb45fc-fcc8-9960-5632-ff7c74639861: aborting pipeline replication: peer="{Voter server-cbf3b14d-d4cd-b754-961b-ff206a801833 cbf3b14d-d4cd-b754-961b-ff206a801833}" -testLoggerAdapter verbose: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:31.298Z [INFO] server-cafb45fc-fcc8-9960-5632-ff7c74639861: aborting pipeline replication: peer="{Voter server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f}" - testing.go:149: server-cafb45fc-fcc8-9960-5632-ff7c74639861: 2024-09-03T21:34:31.298Z [INFO] server-cafb45fc-fcc8-9960-5632-ff7c74639861: aborting pipeline replication: peer="{Voter server-fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f fd5ab4cc-5bcb-70ed-48a8-4e6e759c4a4f}" ---- PASS: TestRaft_SendSnapshotFollower (0.30s) +testLoggerAdapter verbose: server-90180b34-5b90-ebe3-c828-5fe59d8e596a: 2023-08-02T15:13:00.257Z [ERROR] server-90180b34-5b90-ebe3-c828-5fe59d8e596a: failed to take snapshot: error="nothing new to snapshot" + testing.go:149: server-90180b34-5b90-ebe3-c828-5fe59d8e596a: 2023-08-02T15:13:00.257Z [ERROR] server-90180b34-5b90-ebe3-c828-5fe59d8e596a: failed to take snapshot: error="nothing new to snapshot" +testLoggerAdapter verbose: server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: 2023-08-02T15:13:00.257Z [INFO] server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: starting snapshot up to: index=98 + testing.go:149: server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: 2023-08-02T15:13:00.257Z [INFO] server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: starting snapshot up to: index=98 + testing.go:153: 2023-08-02T15:13:00.257Z [INFO] creating new snapshot: path=/tmp/raft916630411/snapshots/2-98-1690989180257.tmp +testLoggerAdapter verbose: server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: 2023-08-02T15:13:00.258Z [INFO] server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: compacting logs: from=1 to=92 + testing.go:149: server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: 2023-08-02T15:13:00.258Z [INFO] server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: compacting logs: from=1 to=92 +testLoggerAdapter verbose: server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: 2023-08-02T15:13:00.258Z [INFO] server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: snapshot complete up to: index=98 + testing.go:149: server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: 2023-08-02T15:13:00.258Z [INFO] server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: snapshot complete up to: index=98 +testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.258Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: starting snapshot up to: index=102 + testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.258Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: starting snapshot up to: index=102 + testing.go:153: 2023-08-02T15:13:00.258Z [INFO] creating new snapshot: path=/tmp/raft626003957/snapshots/2-102-1690989180258.tmp +testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.258Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: compacting logs: from=1 to=92 + testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.258Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: compacting logs: from=1 to=92 +testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.258Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: snapshot complete up to: index=102 + testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.258Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: snapshot complete up to: index=102 +testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.264Z [ERROR] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: failed to get log: index=2 error="log not found" + testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.264Z [ERROR] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: failed to get log: index=2 error="log not found" + testing.go:153: 2023-08-02T15:13:00.265Z [INFO] creating new snapshot: path=/tmp/raft245301297/snapshots/2-102-1690989180265.tmp +testLoggerAdapter verbose: server-90180b34-5b90-ebe3-c828-5fe59d8e596a: 2023-08-02T15:13:00.265Z [INFO] server-90180b34-5b90-ebe3-c828-5fe59d8e596a: copied to local snapshot: bytes=693 + testing.go:149: server-90180b34-5b90-ebe3-c828-5fe59d8e596a: 2023-08-02T15:13:00.265Z [INFO] server-90180b34-5b90-ebe3-c828-5fe59d8e596a: copied to local snapshot: bytes=693 +testLoggerAdapter verbose: server-90180b34-5b90-ebe3-c828-5fe59d8e596a: 2023-08-02T15:13:00.266Z [INFO] server-90180b34-5b90-ebe3-c828-5fe59d8e596a: Installed remote snapshot + testing.go:149: server-90180b34-5b90-ebe3-c828-5fe59d8e596a: 2023-08-02T15:13:00.266Z [INFO] server-90180b34-5b90-ebe3-c828-5fe59d8e596a: Installed remote snapshot +testLoggerAdapter verbose: server-90180b34-5b90-ebe3-c828-5fe59d8e596a: 2023-08-02T15:13:00.266Z [WARN] server-90180b34-5b90-ebe3-c828-5fe59d8e596a: skipping application of old log: index=102 + testing.go:149: server-90180b34-5b90-ebe3-c828-5fe59d8e596a: 2023-08-02T15:13:00.266Z [WARN] server-90180b34-5b90-ebe3-c828-5fe59d8e596a: skipping application of old log: index=102 +testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.266Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: pipelining replication: peer="{Voter server-90180b34-5b90-ebe3-c828-5fe59d8e596a 90180b34-5b90-ebe3-c828-5fe59d8e596a}" + testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.266Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: pipelining replication: peer="{Voter server-90180b34-5b90-ebe3-c828-5fe59d8e596a 90180b34-5b90-ebe3-c828-5fe59d8e596a}" +testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.269Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: aborting pipeline replication: peer="{Voter server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680 5e1a394b-e983-b1a6-50ae-99f0a0ab8680}" + testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.269Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: aborting pipeline replication: peer="{Voter server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680 5e1a394b-e983-b1a6-50ae-99f0a0ab8680}" +testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.269Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: aborting pipeline replication: peer="{Voter server-90180b34-5b90-ebe3-c828-5fe59d8e596a 90180b34-5b90-ebe3-c828-5fe59d8e596a}" + testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.269Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: aborting pipeline replication: peer="{Voter server-90180b34-5b90-ebe3-c828-5fe59d8e596a 90180b34-5b90-ebe3-c828-5fe59d8e596a}" +--- PASS: TestRaft_SendSnapshotFollower (0.31s) === RUN TestRaft_SendSnapshotAndLogsFollower -testLoggerAdapter verbose: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.299Z [INFO] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b4bb9333-f99e-9d13-a209-7b34ed10bbad Address:b4bb9333-f99e-9d13-a209-7b34ed10bbad} {Suffrage:Voter ID:server-e303cef5-f705-9e9f-ee17-085610cf102b Address:e303cef5-f705-9e9f-ee17-085610cf102b} {Suffrage:Voter ID:server-9144bf6d-a055-ab6d-a145-5bf0176f99f3 Address:9144bf6d-a055-ab6d-a145-5bf0176f99f3}]" - testing.go:149: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.299Z [INFO] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b4bb9333-f99e-9d13-a209-7b34ed10bbad Address:b4bb9333-f99e-9d13-a209-7b34ed10bbad} {Suffrage:Voter ID:server-e303cef5-f705-9e9f-ee17-085610cf102b Address:e303cef5-f705-9e9f-ee17-085610cf102b} {Suffrage:Voter ID:server-9144bf6d-a055-ab6d-a145-5bf0176f99f3 Address:9144bf6d-a055-ab6d-a145-5bf0176f99f3}]" -testLoggerAdapter verbose: server-e303cef5-f705-9e9f-ee17-085610cf102b: 2024-09-03T21:34:31.299Z [INFO] server-e303cef5-f705-9e9f-ee17-085610cf102b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b4bb9333-f99e-9d13-a209-7b34ed10bbad Address:b4bb9333-f99e-9d13-a209-7b34ed10bbad} {Suffrage:Voter ID:server-e303cef5-f705-9e9f-ee17-085610cf102b Address:e303cef5-f705-9e9f-ee17-085610cf102b} {Suffrage:Voter ID:server-9144bf6d-a055-ab6d-a145-5bf0176f99f3 Address:9144bf6d-a055-ab6d-a145-5bf0176f99f3}]" - testing.go:149: server-e303cef5-f705-9e9f-ee17-085610cf102b: 2024-09-03T21:34:31.299Z [INFO] server-e303cef5-f705-9e9f-ee17-085610cf102b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b4bb9333-f99e-9d13-a209-7b34ed10bbad Address:b4bb9333-f99e-9d13-a209-7b34ed10bbad} {Suffrage:Voter ID:server-e303cef5-f705-9e9f-ee17-085610cf102b Address:e303cef5-f705-9e9f-ee17-085610cf102b} {Suffrage:Voter ID:server-9144bf6d-a055-ab6d-a145-5bf0176f99f3 Address:9144bf6d-a055-ab6d-a145-5bf0176f99f3}]" -testLoggerAdapter verbose: server-9144bf6d-a055-ab6d-a145-5bf0176f99f3: 2024-09-03T21:34:31.299Z [INFO] server-9144bf6d-a055-ab6d-a145-5bf0176f99f3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b4bb9333-f99e-9d13-a209-7b34ed10bbad Address:b4bb9333-f99e-9d13-a209-7b34ed10bbad} {Suffrage:Voter ID:server-e303cef5-f705-9e9f-ee17-085610cf102b Address:e303cef5-f705-9e9f-ee17-085610cf102b} {Suffrage:Voter ID:server-9144bf6d-a055-ab6d-a145-5bf0176f99f3 Address:9144bf6d-a055-ab6d-a145-5bf0176f99f3}]" - testing.go:149: server-9144bf6d-a055-ab6d-a145-5bf0176f99f3: 2024-09-03T21:34:31.299Z [INFO] server-9144bf6d-a055-ab6d-a145-5bf0176f99f3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b4bb9333-f99e-9d13-a209-7b34ed10bbad Address:b4bb9333-f99e-9d13-a209-7b34ed10bbad} {Suffrage:Voter ID:server-e303cef5-f705-9e9f-ee17-085610cf102b Address:e303cef5-f705-9e9f-ee17-085610cf102b} {Suffrage:Voter ID:server-9144bf6d-a055-ab6d-a145-5bf0176f99f3 Address:9144bf6d-a055-ab6d-a145-5bf0176f99f3}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:31.300Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-09-03T21:34:31.300Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.300Z [INFO] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: entering follower state: follower="Node at b4bb9333-f99e-9d13-a209-7b34ed10bbad [Follower]" leader= - testing.go:149: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.300Z [INFO] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: entering follower state: follower="Node at b4bb9333-f99e-9d13-a209-7b34ed10bbad [Follower]" leader= -testLoggerAdapter verbose: server-e303cef5-f705-9e9f-ee17-085610cf102b: 2024-09-03T21:34:31.300Z [INFO] server-e303cef5-f705-9e9f-ee17-085610cf102b: entering follower state: follower="Node at e303cef5-f705-9e9f-ee17-085610cf102b [Follower]" leader= - testing.go:149: server-e303cef5-f705-9e9f-ee17-085610cf102b: 2024-09-03T21:34:31.300Z [INFO] server-e303cef5-f705-9e9f-ee17-085610cf102b: entering follower state: follower="Node at e303cef5-f705-9e9f-ee17-085610cf102b [Follower]" leader= -testLoggerAdapter verbose: server-9144bf6d-a055-ab6d-a145-5bf0176f99f3: 2024-09-03T21:34:31.300Z [INFO] server-9144bf6d-a055-ab6d-a145-5bf0176f99f3: entering follower state: follower="Node at 9144bf6d-a055-ab6d-a145-5bf0176f99f3 [Follower]" leader= - testing.go:149: server-9144bf6d-a055-ab6d-a145-5bf0176f99f3: 2024-09-03T21:34:31.300Z [INFO] server-9144bf6d-a055-ab6d-a145-5bf0176f99f3: entering follower state: follower="Node at 9144bf6d-a055-ab6d-a145-5bf0176f99f3 [Follower]" leader= -testLoggerAdapter verbose: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.365Z [WARN] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.365Z [WARN] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.365Z [INFO] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: entering candidate state: node="Node at b4bb9333-f99e-9d13-a209-7b34ed10bbad [Candidate]" term=2 - testing.go:149: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.365Z [INFO] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: entering candidate state: node="Node at b4bb9333-f99e-9d13-a209-7b34ed10bbad [Candidate]" term=2 -testLoggerAdapter verbose: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.365Z [INFO] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: election won: tally=2 - testing.go:149: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.365Z [INFO] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: election won: tally=2 -testLoggerAdapter verbose: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.365Z [INFO] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: entering leader state: leader="Node at b4bb9333-f99e-9d13-a209-7b34ed10bbad [Leader]" - testing.go:149: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.365Z [INFO] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: entering leader state: leader="Node at b4bb9333-f99e-9d13-a209-7b34ed10bbad [Leader]" -testLoggerAdapter verbose: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.365Z [INFO] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: added peer, starting replication: peer=server-e303cef5-f705-9e9f-ee17-085610cf102b - testing.go:149: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.365Z [INFO] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: added peer, starting replication: peer=server-e303cef5-f705-9e9f-ee17-085610cf102b -testLoggerAdapter verbose: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.365Z [INFO] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: added peer, starting replication: peer=server-9144bf6d-a055-ab6d-a145-5bf0176f99f3 - testing.go:149: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.365Z [INFO] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: added peer, starting replication: peer=server-9144bf6d-a055-ab6d-a145-5bf0176f99f3 -testLoggerAdapter verbose: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.365Z [INFO] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: pipelining replication: peer="{Voter server-9144bf6d-a055-ab6d-a145-5bf0176f99f3 9144bf6d-a055-ab6d-a145-5bf0176f99f3}" - testing.go:149: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.365Z [INFO] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: pipelining replication: peer="{Voter server-9144bf6d-a055-ab6d-a145-5bf0176f99f3 9144bf6d-a055-ab6d-a145-5bf0176f99f3}" -testLoggerAdapter verbose: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.365Z [INFO] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: pipelining replication: peer="{Voter server-e303cef5-f705-9e9f-ee17-085610cf102b e303cef5-f705-9e9f-ee17-085610cf102b}" - testing.go:149: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.365Z [INFO] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: pipelining replication: peer="{Voter server-e303cef5-f705-9e9f-ee17-085610cf102b e303cef5-f705-9e9f-ee17-085610cf102b}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:31.470Z [INFO] stable state for Follower reached at 2024-09-03 21:34:31.365841084 +0000 UTC m=+12.412498228 (2 nodes), 65.791811ms from start of poll, 66.211292ms from cluster start. Timeout at 2024-09-03 21:34:31.47091444 +0000 UTC m=+12.517571571, 105.073343ms after stability - testing.go:149: cluster: 2024-09-03T21:34:31.470Z [INFO] stable state for Follower reached at 2024-09-03 21:34:31.365841084 +0000 UTC m=+12.412498228 (2 nodes), 65.791811ms from start of poll, 66.211292ms from cluster start. Timeout at 2024-09-03 21:34:31.47091444 +0000 UTC m=+12.517571571, 105.073343ms after stability -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:31.471Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:31.471Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:31.576Z [INFO] stable state for Leader reached at 2024-09-03 21:34:31.471070146 +0000 UTC m=+12.517727279 (1 nodes), 982ns from start of poll, 171.440343ms from cluster start. Timeout at 2024-09-03 21:34:31.576149423 +0000 UTC m=+12.622806556, 105.079277ms after stability - testing.go:149: cluster: 2024-09-03T21:34:31.576Z [INFO] stable state for Leader reached at 2024-09-03 21:34:31.471070146 +0000 UTC m=+12.517727279 (1 nodes), 982ns from start of poll, 171.440343ms from cluster start. Timeout at 2024-09-03 21:34:31.576149423 +0000 UTC m=+12.622806556, 105.079277ms after stability -testLoggerAdapter verbose: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.576Z [ERROR] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: failed to pipeline appendEntries: peer="{Voter server-e303cef5-f705-9e9f-ee17-085610cf102b e303cef5-f705-9e9f-ee17-085610cf102b}" error="append pipeline closed" - testing.go:149: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.576Z [ERROR] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: failed to pipeline appendEntries: peer="{Voter server-e303cef5-f705-9e9f-ee17-085610cf102b e303cef5-f705-9e9f-ee17-085610cf102b}" error="append pipeline closed" -testLoggerAdapter verbose: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.576Z [INFO] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: aborting pipeline replication: peer="{Voter server-e303cef5-f705-9e9f-ee17-085610cf102b e303cef5-f705-9e9f-ee17-085610cf102b}" - testing.go:149: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.576Z [INFO] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: aborting pipeline replication: peer="{Voter server-e303cef5-f705-9e9f-ee17-085610cf102b e303cef5-f705-9e9f-ee17-085610cf102b}" -testLoggerAdapter verbose: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.576Z [ERROR] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: failed to appendEntries to: peer="{Voter server-e303cef5-f705-9e9f-ee17-085610cf102b e303cef5-f705-9e9f-ee17-085610cf102b}" error="failed to connect to peer: e303cef5-f705-9e9f-ee17-085610cf102b" - testing.go:149: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.576Z [ERROR] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: failed to appendEntries to: peer="{Voter server-e303cef5-f705-9e9f-ee17-085610cf102b e303cef5-f705-9e9f-ee17-085610cf102b}" error="failed to connect to peer: e303cef5-f705-9e9f-ee17-085610cf102b" +testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.270Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-781112b6-6781-e0d3-2e7a-fb39add31117 Address:781112b6-6781-e0d3-2e7a-fb39add31117} {Suffrage:Voter ID:server-80b68dcd-3486-a063-a2f4-474aebdf95c1 Address:80b68dcd-3486-a063-a2f4-474aebdf95c1} {Suffrage:Voter ID:server-6a85796e-de9a-002e-9695-01b9be4344c6 Address:6a85796e-de9a-002e-9695-01b9be4344c6}]" + testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.270Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-781112b6-6781-e0d3-2e7a-fb39add31117 Address:781112b6-6781-e0d3-2e7a-fb39add31117} {Suffrage:Voter ID:server-80b68dcd-3486-a063-a2f4-474aebdf95c1 Address:80b68dcd-3486-a063-a2f4-474aebdf95c1} {Suffrage:Voter ID:server-6a85796e-de9a-002e-9695-01b9be4344c6 Address:6a85796e-de9a-002e-9695-01b9be4344c6}]" +testLoggerAdapter verbose: server-80b68dcd-3486-a063-a2f4-474aebdf95c1: 2023-08-02T15:13:00.271Z [INFO] server-80b68dcd-3486-a063-a2f4-474aebdf95c1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-781112b6-6781-e0d3-2e7a-fb39add31117 Address:781112b6-6781-e0d3-2e7a-fb39add31117} {Suffrage:Voter ID:server-80b68dcd-3486-a063-a2f4-474aebdf95c1 Address:80b68dcd-3486-a063-a2f4-474aebdf95c1} {Suffrage:Voter ID:server-6a85796e-de9a-002e-9695-01b9be4344c6 Address:6a85796e-de9a-002e-9695-01b9be4344c6}]" +testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.271Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: entering follower state: follower="Node at 781112b6-6781-e0d3-2e7a-fb39add31117 [Follower]" leader= + testing.go:149: server-80b68dcd-3486-a063-a2f4-474aebdf95c1: 2023-08-02T15:13:00.271Z [INFO] server-80b68dcd-3486-a063-a2f4-474aebdf95c1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-781112b6-6781-e0d3-2e7a-fb39add31117 Address:781112b6-6781-e0d3-2e7a-fb39add31117} {Suffrage:Voter ID:server-80b68dcd-3486-a063-a2f4-474aebdf95c1 Address:80b68dcd-3486-a063-a2f4-474aebdf95c1} {Suffrage:Voter ID:server-6a85796e-de9a-002e-9695-01b9be4344c6 Address:6a85796e-de9a-002e-9695-01b9be4344c6}]" +testLoggerAdapter verbose: server-80b68dcd-3486-a063-a2f4-474aebdf95c1: 2023-08-02T15:13:00.271Z [INFO] server-80b68dcd-3486-a063-a2f4-474aebdf95c1: entering follower state: follower="Node at 80b68dcd-3486-a063-a2f4-474aebdf95c1 [Follower]" leader= + testing.go:149: server-80b68dcd-3486-a063-a2f4-474aebdf95c1: 2023-08-02T15:13:00.271Z [INFO] server-80b68dcd-3486-a063-a2f4-474aebdf95c1: entering follower state: follower="Node at 80b68dcd-3486-a063-a2f4-474aebdf95c1 [Follower]" leader= + testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.271Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: entering follower state: follower="Node at 781112b6-6781-e0d3-2e7a-fb39add31117 [Follower]" leader= +testLoggerAdapter verbose: server-6a85796e-de9a-002e-9695-01b9be4344c6: 2023-08-02T15:13:00.271Z [INFO] server-6a85796e-de9a-002e-9695-01b9be4344c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-781112b6-6781-e0d3-2e7a-fb39add31117 Address:781112b6-6781-e0d3-2e7a-fb39add31117} {Suffrage:Voter ID:server-80b68dcd-3486-a063-a2f4-474aebdf95c1 Address:80b68dcd-3486-a063-a2f4-474aebdf95c1} {Suffrage:Voter ID:server-6a85796e-de9a-002e-9695-01b9be4344c6 Address:6a85796e-de9a-002e-9695-01b9be4344c6}]" + testing.go:149: server-6a85796e-de9a-002e-9695-01b9be4344c6: 2023-08-02T15:13:00.271Z [INFO] server-6a85796e-de9a-002e-9695-01b9be4344c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-781112b6-6781-e0d3-2e7a-fb39add31117 Address:781112b6-6781-e0d3-2e7a-fb39add31117} {Suffrage:Voter ID:server-80b68dcd-3486-a063-a2f4-474aebdf95c1 Address:80b68dcd-3486-a063-a2f4-474aebdf95c1} {Suffrage:Voter ID:server-6a85796e-de9a-002e-9695-01b9be4344c6 Address:6a85796e-de9a-002e-9695-01b9be4344c6}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:00.271Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-08-02T15:13:00.271Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-6a85796e-de9a-002e-9695-01b9be4344c6: 2023-08-02T15:13:00.271Z [INFO] server-6a85796e-de9a-002e-9695-01b9be4344c6: entering follower state: follower="Node at 6a85796e-de9a-002e-9695-01b9be4344c6 [Follower]" leader= + testing.go:149: server-6a85796e-de9a-002e-9695-01b9be4344c6: 2023-08-02T15:13:00.271Z [INFO] server-6a85796e-de9a-002e-9695-01b9be4344c6: entering follower state: follower="Node at 6a85796e-de9a-002e-9695-01b9be4344c6 [Follower]" leader= +testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.324Z [WARN] server-781112b6-6781-e0d3-2e7a-fb39add31117: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.324Z [WARN] server-781112b6-6781-e0d3-2e7a-fb39add31117: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.324Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: entering candidate state: node="Node at 781112b6-6781-e0d3-2e7a-fb39add31117 [Candidate]" term=2 + testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.324Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: entering candidate state: node="Node at 781112b6-6781-e0d3-2e7a-fb39add31117 [Candidate]" term=2 +testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.324Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: election won: tally=2 + testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.324Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: election won: tally=2 +testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.325Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: entering leader state: leader="Node at 781112b6-6781-e0d3-2e7a-fb39add31117 [Leader]" + testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.325Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: entering leader state: leader="Node at 781112b6-6781-e0d3-2e7a-fb39add31117 [Leader]" +testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.325Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: added peer, starting replication: peer=server-80b68dcd-3486-a063-a2f4-474aebdf95c1 + testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.325Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: added peer, starting replication: peer=server-80b68dcd-3486-a063-a2f4-474aebdf95c1 +testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.325Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: added peer, starting replication: peer=server-6a85796e-de9a-002e-9695-01b9be4344c6 + testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.325Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: added peer, starting replication: peer=server-6a85796e-de9a-002e-9695-01b9be4344c6 +testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.325Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: pipelining replication: peer="{Voter server-6a85796e-de9a-002e-9695-01b9be4344c6 6a85796e-de9a-002e-9695-01b9be4344c6}" + testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.325Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: pipelining replication: peer="{Voter server-6a85796e-de9a-002e-9695-01b9be4344c6 6a85796e-de9a-002e-9695-01b9be4344c6}" +testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.325Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: pipelining replication: peer="{Voter server-80b68dcd-3486-a063-a2f4-474aebdf95c1 80b68dcd-3486-a063-a2f4-474aebdf95c1}" + testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.325Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: pipelining replication: peer="{Voter server-80b68dcd-3486-a063-a2f4-474aebdf95c1 80b68dcd-3486-a063-a2f4-474aebdf95c1}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:00.430Z [INFO] stable state for Follower reached at 2023-08-02 15:13:00.325446619 +0000 UTC m=+12.421382958 (2 nodes), 53.969814ms from start of poll, 54.752205ms from cluster start. Timeout at 2023-08-02 15:13:00.430498896 +0000 UTC m=+12.526435241, 105.052283ms after stability + testing.go:149: cluster: 2023-08-02T15:13:00.430Z [INFO] stable state for Follower reached at 2023-08-02 15:13:00.325446619 +0000 UTC m=+12.421382958 (2 nodes), 53.969814ms from start of poll, 54.752205ms from cluster start. Timeout at 2023-08-02 15:13:00.430498896 +0000 UTC m=+12.526435241, 105.052283ms after stability +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:00.430Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:13:00.430Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:00.536Z [INFO] stable state for Leader reached at 2023-08-02 15:13:00.430685202 +0000 UTC m=+12.526621546 (1 nodes), 1.747µs from start of poll, 159.990793ms from cluster start. Timeout at 2023-08-02 15:13:00.536009117 +0000 UTC m=+12.631945456, 105.32391ms after stability + testing.go:149: cluster: 2023-08-02T15:13:00.536Z [INFO] stable state for Leader reached at 2023-08-02 15:13:00.430685202 +0000 UTC m=+12.526621546 (1 nodes), 1.747µs from start of poll, 159.990793ms from cluster start. Timeout at 2023-08-02 15:13:00.536009117 +0000 UTC m=+12.631945456, 105.32391ms after stability +testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.536Z [ERROR] server-781112b6-6781-e0d3-2e7a-fb39add31117: failed to pipeline appendEntries: peer="{Voter server-80b68dcd-3486-a063-a2f4-474aebdf95c1 80b68dcd-3486-a063-a2f4-474aebdf95c1}" error="append pipeline closed" + testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.536Z [ERROR] server-781112b6-6781-e0d3-2e7a-fb39add31117: failed to pipeline appendEntries: peer="{Voter server-80b68dcd-3486-a063-a2f4-474aebdf95c1 80b68dcd-3486-a063-a2f4-474aebdf95c1}" error="append pipeline closed" +testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.536Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: aborting pipeline replication: peer="{Voter server-80b68dcd-3486-a063-a2f4-474aebdf95c1 80b68dcd-3486-a063-a2f4-474aebdf95c1}" + testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.536Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: aborting pipeline replication: peer="{Voter server-80b68dcd-3486-a063-a2f4-474aebdf95c1 80b68dcd-3486-a063-a2f4-474aebdf95c1}" +testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.536Z [ERROR] server-781112b6-6781-e0d3-2e7a-fb39add31117: failed to appendEntries to: peer="{Voter server-80b68dcd-3486-a063-a2f4-474aebdf95c1 80b68dcd-3486-a063-a2f4-474aebdf95c1}" error="failed to connect to peer: 80b68dcd-3486-a063-a2f4-474aebdf95c1" + testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.536Z [ERROR] server-781112b6-6781-e0d3-2e7a-fb39add31117: failed to appendEntries to: peer="{Voter server-80b68dcd-3486-a063-a2f4-474aebdf95c1 80b68dcd-3486-a063-a2f4-474aebdf95c1}" error="failed to connect to peer: 80b68dcd-3486-a063-a2f4-474aebdf95c1" raft_test.go:1362: [INFO] Finished apply without behind follower -testLoggerAdapter verbose: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.578Z [INFO] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: starting snapshot up to: index=102 - testing.go:149: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.578Z [INFO] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: starting snapshot up to: index=102 - testing.go:153: 2024-09-03T21:34:31.578Z [INFO] creating new snapshot: path=/tmp/raft235216944/snapshots/2-102-1725399271578.tmp -testLoggerAdapter verbose: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.578Z [INFO] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: compacting logs: from=1 to=92 - testing.go:149: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.578Z [INFO] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: compacting logs: from=1 to=92 -testLoggerAdapter verbose: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.578Z [INFO] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: snapshot complete up to: index=102 - testing.go:149: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.578Z [INFO] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: snapshot complete up to: index=102 -testLoggerAdapter verbose: server-e303cef5-f705-9e9f-ee17-085610cf102b: 2024-09-03T21:34:31.579Z [ERROR] server-e303cef5-f705-9e9f-ee17-085610cf102b: failed to take snapshot: error="nothing new to snapshot" - testing.go:149: server-e303cef5-f705-9e9f-ee17-085610cf102b: 2024-09-03T21:34:31.579Z [ERROR] server-e303cef5-f705-9e9f-ee17-085610cf102b: failed to take snapshot: error="nothing new to snapshot" -testLoggerAdapter verbose: server-9144bf6d-a055-ab6d-a145-5bf0176f99f3: 2024-09-03T21:34:31.579Z [INFO] server-9144bf6d-a055-ab6d-a145-5bf0176f99f3: starting snapshot up to: index=101 - testing.go:149: server-9144bf6d-a055-ab6d-a145-5bf0176f99f3: 2024-09-03T21:34:31.579Z [INFO] server-9144bf6d-a055-ab6d-a145-5bf0176f99f3: starting snapshot up to: index=101 - testing.go:153: 2024-09-03T21:34:31.579Z [INFO] creating new snapshot: path=/tmp/raft193874020/snapshots/2-101-1725399271579.tmp -testLoggerAdapter verbose: server-9144bf6d-a055-ab6d-a145-5bf0176f99f3: 2024-09-03T21:34:31.579Z [INFO] server-9144bf6d-a055-ab6d-a145-5bf0176f99f3: compacting logs: from=1 to=92 - testing.go:149: server-9144bf6d-a055-ab6d-a145-5bf0176f99f3: 2024-09-03T21:34:31.579Z [INFO] server-9144bf6d-a055-ab6d-a145-5bf0176f99f3: compacting logs: from=1 to=92 -testLoggerAdapter verbose: server-9144bf6d-a055-ab6d-a145-5bf0176f99f3: 2024-09-03T21:34:31.579Z [INFO] server-9144bf6d-a055-ab6d-a145-5bf0176f99f3: snapshot complete up to: index=101 - testing.go:149: server-9144bf6d-a055-ab6d-a145-5bf0176f99f3: 2024-09-03T21:34:31.579Z [INFO] server-9144bf6d-a055-ab6d-a145-5bf0176f99f3: snapshot complete up to: index=101 -testLoggerAdapter verbose: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.580Z [ERROR] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: failed to heartbeat to: peer=e303cef5-f705-9e9f-ee17-085610cf102b error="failed to connect to peer: e303cef5-f705-9e9f-ee17-085610cf102b" - testing.go:149: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.580Z [ERROR] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: failed to heartbeat to: peer=e303cef5-f705-9e9f-ee17-085610cf102b error="failed to connect to peer: e303cef5-f705-9e9f-ee17-085610cf102b" +testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.538Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: starting snapshot up to: index=102 + testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.538Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: starting snapshot up to: index=102 + testing.go:153: 2023-08-02T15:13:00.539Z [INFO] creating new snapshot: path=/tmp/raft796835311/snapshots/2-102-1690989180539.tmp +testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.539Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: compacting logs: from=1 to=92 + testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.539Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: compacting logs: from=1 to=92 +testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.539Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: snapshot complete up to: index=102 + testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.539Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: snapshot complete up to: index=102 +testLoggerAdapter verbose: server-80b68dcd-3486-a063-a2f4-474aebdf95c1: 2023-08-02T15:13:00.539Z [ERROR] server-80b68dcd-3486-a063-a2f4-474aebdf95c1: failed to take snapshot: error="nothing new to snapshot" + testing.go:149: server-80b68dcd-3486-a063-a2f4-474aebdf95c1: 2023-08-02T15:13:00.539Z [ERROR] server-80b68dcd-3486-a063-a2f4-474aebdf95c1: failed to take snapshot: error="nothing new to snapshot" +testLoggerAdapter verbose: server-6a85796e-de9a-002e-9695-01b9be4344c6: 2023-08-02T15:13:00.539Z [INFO] server-6a85796e-de9a-002e-9695-01b9be4344c6: starting snapshot up to: index=88 + testing.go:149: server-6a85796e-de9a-002e-9695-01b9be4344c6: 2023-08-02T15:13:00.539Z [INFO] server-6a85796e-de9a-002e-9695-01b9be4344c6: starting snapshot up to: index=88 + testing.go:153: 2023-08-02T15:13:00.539Z [INFO] creating new snapshot: path=/tmp/raft471634067/snapshots/2-88-1690989180539.tmp +testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.540Z [ERROR] server-781112b6-6781-e0d3-2e7a-fb39add31117: failed to heartbeat to: peer=80b68dcd-3486-a063-a2f4-474aebdf95c1 error="failed to connect to peer: 80b68dcd-3486-a063-a2f4-474aebdf95c1" + testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.540Z [ERROR] server-781112b6-6781-e0d3-2e7a-fb39add31117: failed to heartbeat to: peer=80b68dcd-3486-a063-a2f4-474aebdf95c1 error="failed to connect to peer: 80b68dcd-3486-a063-a2f4-474aebdf95c1" +testLoggerAdapter verbose: server-6a85796e-de9a-002e-9695-01b9be4344c6: 2023-08-02T15:13:00.541Z [INFO] server-6a85796e-de9a-002e-9695-01b9be4344c6: compacting logs: from=1 to=88 + testing.go:149: server-6a85796e-de9a-002e-9695-01b9be4344c6: 2023-08-02T15:13:00.541Z [INFO] server-6a85796e-de9a-002e-9695-01b9be4344c6: compacting logs: from=1 to=88 +testLoggerAdapter verbose: server-6a85796e-de9a-002e-9695-01b9be4344c6: 2023-08-02T15:13:00.541Z [INFO] server-6a85796e-de9a-002e-9695-01b9be4344c6: snapshot complete up to: index=88 + testing.go:149: server-6a85796e-de9a-002e-9695-01b9be4344c6: 2023-08-02T15:13:00.541Z [INFO] server-6a85796e-de9a-002e-9695-01b9be4344c6: snapshot complete up to: index=88 raft_test.go:1383: [INFO] Finished apply without behind follower -testLoggerAdapter verbose: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.586Z [ERROR] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: failed to get log: index=2 error="log not found" - testing.go:149: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.586Z [ERROR] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: failed to get log: index=2 error="log not found" - testing.go:153: 2024-09-03T21:34:31.589Z [INFO] creating new snapshot: path=/tmp/raft834315490/snapshots/2-102-1725399271589.tmp -testLoggerAdapter verbose: server-e303cef5-f705-9e9f-ee17-085610cf102b: 2024-09-03T21:34:31.590Z [INFO] server-e303cef5-f705-9e9f-ee17-085610cf102b: copied to local snapshot: bytes=693 - testing.go:149: server-e303cef5-f705-9e9f-ee17-085610cf102b: 2024-09-03T21:34:31.590Z [INFO] server-e303cef5-f705-9e9f-ee17-085610cf102b: copied to local snapshot: bytes=693 -testLoggerAdapter verbose: server-e303cef5-f705-9e9f-ee17-085610cf102b: 2024-09-03T21:34:31.590Z [INFO] server-e303cef5-f705-9e9f-ee17-085610cf102b: Installed remote snapshot - testing.go:149: server-e303cef5-f705-9e9f-ee17-085610cf102b: 2024-09-03T21:34:31.590Z [INFO] server-e303cef5-f705-9e9f-ee17-085610cf102b: Installed remote snapshot -testLoggerAdapter verbose: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.590Z [INFO] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: pipelining replication: peer="{Voter server-e303cef5-f705-9e9f-ee17-085610cf102b e303cef5-f705-9e9f-ee17-085610cf102b}" - testing.go:149: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.590Z [INFO] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: pipelining replication: peer="{Voter server-e303cef5-f705-9e9f-ee17-085610cf102b e303cef5-f705-9e9f-ee17-085610cf102b}" -testLoggerAdapter verbose: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.591Z [INFO] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: aborting pipeline replication: peer="{Voter server-9144bf6d-a055-ab6d-a145-5bf0176f99f3 9144bf6d-a055-ab6d-a145-5bf0176f99f3}" - testing.go:149: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.591Z [INFO] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: aborting pipeline replication: peer="{Voter server-9144bf6d-a055-ab6d-a145-5bf0176f99f3 9144bf6d-a055-ab6d-a145-5bf0176f99f3}" -testLoggerAdapter verbose: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.591Z [INFO] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: aborting pipeline replication: peer="{Voter server-e303cef5-f705-9e9f-ee17-085610cf102b e303cef5-f705-9e9f-ee17-085610cf102b}" - testing.go:149: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.591Z [INFO] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: aborting pipeline replication: peer="{Voter server-e303cef5-f705-9e9f-ee17-085610cf102b e303cef5-f705-9e9f-ee17-085610cf102b}" -testLoggerAdapter verbose: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.641Z [ERROR] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: failed to heartbeat to: peer=9144bf6d-a055-ab6d-a145-5bf0176f99f3 error="command timed out" - testing.go:149: server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: 2024-09-03T21:34:31.641Z [ERROR] server-b4bb9333-f99e-9d13-a209-7b34ed10bbad: failed to heartbeat to: peer=9144bf6d-a055-ab6d-a145-5bf0176f99f3 error="command timed out" ---- PASS: TestRaft_SendSnapshotAndLogsFollower (0.34s) +testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.550Z [ERROR] server-781112b6-6781-e0d3-2e7a-fb39add31117: failed to get log: index=2 error="log not found" + testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.550Z [ERROR] server-781112b6-6781-e0d3-2e7a-fb39add31117: failed to get log: index=2 error="log not found" + testing.go:153: 2023-08-02T15:13:00.551Z [INFO] creating new snapshot: path=/tmp/raft864382969/snapshots/2-102-1690989180551.tmp +testLoggerAdapter verbose: server-80b68dcd-3486-a063-a2f4-474aebdf95c1: 2023-08-02T15:13:00.551Z [INFO] server-80b68dcd-3486-a063-a2f4-474aebdf95c1: copied to local snapshot: bytes=693 + testing.go:149: server-80b68dcd-3486-a063-a2f4-474aebdf95c1: 2023-08-02T15:13:00.551Z [INFO] server-80b68dcd-3486-a063-a2f4-474aebdf95c1: copied to local snapshot: bytes=693 +testLoggerAdapter verbose: server-80b68dcd-3486-a063-a2f4-474aebdf95c1: 2023-08-02T15:13:00.551Z [INFO] server-80b68dcd-3486-a063-a2f4-474aebdf95c1: Installed remote snapshot + testing.go:149: server-80b68dcd-3486-a063-a2f4-474aebdf95c1: 2023-08-02T15:13:00.551Z [INFO] server-80b68dcd-3486-a063-a2f4-474aebdf95c1: Installed remote snapshot +testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.552Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: pipelining replication: peer="{Voter server-80b68dcd-3486-a063-a2f4-474aebdf95c1 80b68dcd-3486-a063-a2f4-474aebdf95c1}" + testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.552Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: pipelining replication: peer="{Voter server-80b68dcd-3486-a063-a2f4-474aebdf95c1 80b68dcd-3486-a063-a2f4-474aebdf95c1}" +testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.561Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: aborting pipeline replication: peer="{Voter server-6a85796e-de9a-002e-9695-01b9be4344c6 6a85796e-de9a-002e-9695-01b9be4344c6}" + testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.561Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: aborting pipeline replication: peer="{Voter server-6a85796e-de9a-002e-9695-01b9be4344c6 6a85796e-de9a-002e-9695-01b9be4344c6}" +testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.561Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: aborting pipeline replication: peer="{Voter server-80b68dcd-3486-a063-a2f4-474aebdf95c1 80b68dcd-3486-a063-a2f4-474aebdf95c1}" + testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.561Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: aborting pipeline replication: peer="{Voter server-80b68dcd-3486-a063-a2f4-474aebdf95c1 80b68dcd-3486-a063-a2f4-474aebdf95c1}" +--- PASS: TestRaft_SendSnapshotAndLogsFollower (0.29s) === RUN TestRaft_ReJoinFollower -testLoggerAdapter verbose: server-b667d949-7421-bd1e-db11-08abfca00c4a: 2024-09-03T21:34:31.643Z [INFO] server-b667d949-7421-bd1e-db11-08abfca00c4a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b667d949-7421-bd1e-db11-08abfca00c4a Address:b667d949-7421-bd1e-db11-08abfca00c4a} {Suffrage:Voter ID:server-d230261e-52ee-9f48-2e93-d683c22ff6ab Address:d230261e-52ee-9f48-2e93-d683c22ff6ab} {Suffrage:Voter ID:server-dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7 Address:dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7}]" - testing.go:149: server-b667d949-7421-bd1e-db11-08abfca00c4a: 2024-09-03T21:34:31.643Z [INFO] server-b667d949-7421-bd1e-db11-08abfca00c4a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b667d949-7421-bd1e-db11-08abfca00c4a Address:b667d949-7421-bd1e-db11-08abfca00c4a} {Suffrage:Voter ID:server-d230261e-52ee-9f48-2e93-d683c22ff6ab Address:d230261e-52ee-9f48-2e93-d683c22ff6ab} {Suffrage:Voter ID:server-dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7 Address:dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7}]" -testLoggerAdapter verbose: server-d230261e-52ee-9f48-2e93-d683c22ff6ab: 2024-09-03T21:34:31.643Z [INFO] server-d230261e-52ee-9f48-2e93-d683c22ff6ab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b667d949-7421-bd1e-db11-08abfca00c4a Address:b667d949-7421-bd1e-db11-08abfca00c4a} {Suffrage:Voter ID:server-d230261e-52ee-9f48-2e93-d683c22ff6ab Address:d230261e-52ee-9f48-2e93-d683c22ff6ab} {Suffrage:Voter ID:server-dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7 Address:dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7}]" - testing.go:149: server-d230261e-52ee-9f48-2e93-d683c22ff6ab: 2024-09-03T21:34:31.643Z [INFO] server-d230261e-52ee-9f48-2e93-d683c22ff6ab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b667d949-7421-bd1e-db11-08abfca00c4a Address:b667d949-7421-bd1e-db11-08abfca00c4a} {Suffrage:Voter ID:server-d230261e-52ee-9f48-2e93-d683c22ff6ab Address:d230261e-52ee-9f48-2e93-d683c22ff6ab} {Suffrage:Voter ID:server-dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7 Address:dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7}]" -testLoggerAdapter verbose: server-b667d949-7421-bd1e-db11-08abfca00c4a: 2024-09-03T21:34:31.643Z [INFO] server-b667d949-7421-bd1e-db11-08abfca00c4a: entering follower state: follower="Node at b667d949-7421-bd1e-db11-08abfca00c4a [Follower]" leader= - testing.go:149: server-b667d949-7421-bd1e-db11-08abfca00c4a: 2024-09-03T21:34:31.643Z [INFO] server-b667d949-7421-bd1e-db11-08abfca00c4a: entering follower state: follower="Node at b667d949-7421-bd1e-db11-08abfca00c4a [Follower]" leader= -testLoggerAdapter verbose: server-dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7: 2024-09-03T21:34:31.643Z [INFO] server-dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b667d949-7421-bd1e-db11-08abfca00c4a Address:b667d949-7421-bd1e-db11-08abfca00c4a} {Suffrage:Voter ID:server-d230261e-52ee-9f48-2e93-d683c22ff6ab Address:d230261e-52ee-9f48-2e93-d683c22ff6ab} {Suffrage:Voter ID:server-dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7 Address:dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7}]" - testing.go:149: server-dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7: 2024-09-03T21:34:31.643Z [INFO] server-dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b667d949-7421-bd1e-db11-08abfca00c4a Address:b667d949-7421-bd1e-db11-08abfca00c4a} {Suffrage:Voter ID:server-d230261e-52ee-9f48-2e93-d683c22ff6ab Address:d230261e-52ee-9f48-2e93-d683c22ff6ab} {Suffrage:Voter ID:server-dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7 Address:dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7}]" -testLoggerAdapter verbose: server-d230261e-52ee-9f48-2e93-d683c22ff6ab: 2024-09-03T21:34:31.643Z [INFO] server-d230261e-52ee-9f48-2e93-d683c22ff6ab: entering follower state: follower="Node at d230261e-52ee-9f48-2e93-d683c22ff6ab [Follower]" leader= -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:31.643Z [INFO] starting stability test: raft-state=Leader - testing.go:149: server-d230261e-52ee-9f48-2e93-d683c22ff6ab: 2024-09-03T21:34:31.643Z [INFO] server-d230261e-52ee-9f48-2e93-d683c22ff6ab: entering follower state: follower="Node at d230261e-52ee-9f48-2e93-d683c22ff6ab [Follower]" leader= - testing.go:149: cluster: 2024-09-03T21:34:31.643Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7: 2024-09-03T21:34:31.643Z [INFO] server-dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7: entering follower state: follower="Node at dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7 [Follower]" leader= - testing.go:149: server-dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7: 2024-09-03T21:34:31.643Z [INFO] server-dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7: entering follower state: follower="Node at dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7 [Follower]" leader= -testLoggerAdapter verbose: server-d230261e-52ee-9f48-2e93-d683c22ff6ab: 2024-09-03T21:34:31.723Z [WARN] server-d230261e-52ee-9f48-2e93-d683c22ff6ab: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-d230261e-52ee-9f48-2e93-d683c22ff6ab: 2024-09-03T21:34:31.723Z [WARN] server-d230261e-52ee-9f48-2e93-d683c22ff6ab: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-d230261e-52ee-9f48-2e93-d683c22ff6ab: 2024-09-03T21:34:31.723Z [INFO] server-d230261e-52ee-9f48-2e93-d683c22ff6ab: entering candidate state: node="Node at d230261e-52ee-9f48-2e93-d683c22ff6ab [Candidate]" term=2 - testing.go:149: server-d230261e-52ee-9f48-2e93-d683c22ff6ab: 2024-09-03T21:34:31.723Z [INFO] server-d230261e-52ee-9f48-2e93-d683c22ff6ab: entering candidate state: node="Node at d230261e-52ee-9f48-2e93-d683c22ff6ab [Candidate]" term=2 -testLoggerAdapter verbose: server-d230261e-52ee-9f48-2e93-d683c22ff6ab: 2024-09-03T21:34:31.723Z [INFO] server-d230261e-52ee-9f48-2e93-d683c22ff6ab: election won: tally=2 - testing.go:149: server-d230261e-52ee-9f48-2e93-d683c22ff6ab: 2024-09-03T21:34:31.723Z [INFO] server-d230261e-52ee-9f48-2e93-d683c22ff6ab: election won: tally=2 -testLoggerAdapter verbose: server-d230261e-52ee-9f48-2e93-d683c22ff6ab: 2024-09-03T21:34:31.723Z [INFO] server-d230261e-52ee-9f48-2e93-d683c22ff6ab: entering leader state: leader="Node at d230261e-52ee-9f48-2e93-d683c22ff6ab [Leader]" - testing.go:149: server-d230261e-52ee-9f48-2e93-d683c22ff6ab: 2024-09-03T21:34:31.723Z [INFO] server-d230261e-52ee-9f48-2e93-d683c22ff6ab: entering leader state: leader="Node at d230261e-52ee-9f48-2e93-d683c22ff6ab [Leader]" -testLoggerAdapter verbose: server-d230261e-52ee-9f48-2e93-d683c22ff6ab: 2024-09-03T21:34:31.723Z [INFO] server-d230261e-52ee-9f48-2e93-d683c22ff6ab: added peer, starting replication: peer=server-b667d949-7421-bd1e-db11-08abfca00c4a - testing.go:149: server-d230261e-52ee-9f48-2e93-d683c22ff6ab: 2024-09-03T21:34:31.723Z [INFO] server-d230261e-52ee-9f48-2e93-d683c22ff6ab: added peer, starting replication: peer=server-b667d949-7421-bd1e-db11-08abfca00c4a -testLoggerAdapter verbose: server-d230261e-52ee-9f48-2e93-d683c22ff6ab: 2024-09-03T21:34:31.723Z [INFO] server-d230261e-52ee-9f48-2e93-d683c22ff6ab: added peer, starting replication: peer=server-dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7 - testing.go:149: server-d230261e-52ee-9f48-2e93-d683c22ff6ab: 2024-09-03T21:34:31.723Z [INFO] server-d230261e-52ee-9f48-2e93-d683c22ff6ab: added peer, starting replication: peer=server-dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7 -testLoggerAdapter verbose: server-d230261e-52ee-9f48-2e93-d683c22ff6ab: 2024-09-03T21:34:31.723Z [INFO] server-d230261e-52ee-9f48-2e93-d683c22ff6ab: pipelining replication: peer="{Voter server-dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7 dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7}" - testing.go:149: server-d230261e-52ee-9f48-2e93-d683c22ff6ab: 2024-09-03T21:34:31.723Z [INFO] server-d230261e-52ee-9f48-2e93-d683c22ff6ab: pipelining replication: peer="{Voter server-dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7 dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7}" -testLoggerAdapter verbose: server-d230261e-52ee-9f48-2e93-d683c22ff6ab: 2024-09-03T21:34:31.723Z [INFO] server-d230261e-52ee-9f48-2e93-d683c22ff6ab: pipelining replication: peer="{Voter server-b667d949-7421-bd1e-db11-08abfca00c4a b667d949-7421-bd1e-db11-08abfca00c4a}" - testing.go:149: server-d230261e-52ee-9f48-2e93-d683c22ff6ab: 2024-09-03T21:34:31.723Z [INFO] server-d230261e-52ee-9f48-2e93-d683c22ff6ab: pipelining replication: peer="{Voter server-b667d949-7421-bd1e-db11-08abfca00c4a b667d949-7421-bd1e-db11-08abfca00c4a}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:31.828Z [INFO] stable state for Leader reached at 2024-09-03 21:34:31.723589506 +0000 UTC m=+12.770246656 (1 nodes), 80.026032ms from start of poll, 80.746813ms from cluster start. Timeout at 2024-09-03 21:34:31.828672614 +0000 UTC m=+12.875329751, 105.083095ms after stability - testing.go:149: cluster: 2024-09-03T21:34:31.828Z [INFO] stable state for Leader reached at 2024-09-03 21:34:31.723589506 +0000 UTC m=+12.770246656 (1 nodes), 80.026032ms from start of poll, 80.746813ms from cluster start. Timeout at 2024-09-03 21:34:31.828672614 +0000 UTC m=+12.875329751, 105.083095ms after stability -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:31.833Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-09-03T21:34:31.833Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:31.939Z [INFO] stable state for Follower reached at 2024-09-03 21:34:31.833943181 +0000 UTC m=+12.880600335 (2 nodes), 1.525µs from start of poll, 191.100492ms from cluster start. Timeout at 2024-09-03 21:34:31.939023909 +0000 UTC m=+12.985681041, 105.080706ms after stability - testing.go:149: cluster: 2024-09-03T21:34:31.939Z [INFO] stable state for Follower reached at 2024-09-03 21:34:31.833943181 +0000 UTC m=+12.880600335 (2 nodes), 1.525µs from start of poll, 191.100492ms from cluster start. Timeout at 2024-09-03 21:34:31.939023909 +0000 UTC m=+12.985681041, 105.080706ms after stability -testLoggerAdapter verbose: server-d230261e-52ee-9f48-2e93-d683c22ff6ab: 2024-09-03T21:34:31.939Z [INFO] server-d230261e-52ee-9f48-2e93-d683c22ff6ab: updating configuration: command=RemoveServer server-id=server-b667d949-7421-bd1e-db11-08abfca00c4a server-addr= servers="[{Suffrage:Voter ID:server-d230261e-52ee-9f48-2e93-d683c22ff6ab Address:d230261e-52ee-9f48-2e93-d683c22ff6ab} {Suffrage:Voter ID:server-dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7 Address:dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7}]" - testing.go:149: server-d230261e-52ee-9f48-2e93-d683c22ff6ab: 2024-09-03T21:34:31.939Z [INFO] server-d230261e-52ee-9f48-2e93-d683c22ff6ab: updating configuration: command=RemoveServer server-id=server-b667d949-7421-bd1e-db11-08abfca00c4a server-addr= servers="[{Suffrage:Voter ID:server-d230261e-52ee-9f48-2e93-d683c22ff6ab Address:d230261e-52ee-9f48-2e93-d683c22ff6ab} {Suffrage:Voter ID:server-dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7 Address:dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7}]" -testLoggerAdapter verbose: server-d230261e-52ee-9f48-2e93-d683c22ff6ab: 2024-09-03T21:34:31.939Z [INFO] server-d230261e-52ee-9f48-2e93-d683c22ff6ab: removed peer, stopping replication: peer=server-b667d949-7421-bd1e-db11-08abfca00c4a last-index=3 - testing.go:149: server-d230261e-52ee-9f48-2e93-d683c22ff6ab: 2024-09-03T21:34:31.939Z [INFO] server-d230261e-52ee-9f48-2e93-d683c22ff6ab: removed peer, stopping replication: peer=server-b667d949-7421-bd1e-db11-08abfca00c4a last-index=3 -testLoggerAdapter verbose: server-d230261e-52ee-9f48-2e93-d683c22ff6ab: 2024-09-03T21:34:31.939Z [INFO] server-d230261e-52ee-9f48-2e93-d683c22ff6ab: aborting pipeline replication: peer="{Voter server-b667d949-7421-bd1e-db11-08abfca00c4a b667d949-7421-bd1e-db11-08abfca00c4a}" - testing.go:149: server-d230261e-52ee-9f48-2e93-d683c22ff6ab: 2024-09-03T21:34:31.939Z [INFO] server-d230261e-52ee-9f48-2e93-d683c22ff6ab: aborting pipeline replication: peer="{Voter server-b667d949-7421-bd1e-db11-08abfca00c4a b667d949-7421-bd1e-db11-08abfca00c4a}" -testLoggerAdapter verbose: server-d230261e-52ee-9f48-2e93-d683c22ff6ab: 2024-09-03T21:34:32.049Z [INFO] server-d230261e-52ee-9f48-2e93-d683c22ff6ab: updating configuration: command=AddStaging server-id=server-b667d949-7421-bd1e-db11-08abfca00c4a server-addr=b667d949-7421-bd1e-db11-08abfca00c4a servers="[{Suffrage:Voter ID:server-d230261e-52ee-9f48-2e93-d683c22ff6ab Address:d230261e-52ee-9f48-2e93-d683c22ff6ab} {Suffrage:Voter ID:server-dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7 Address:dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7} {Suffrage:Voter ID:server-b667d949-7421-bd1e-db11-08abfca00c4a Address:b667d949-7421-bd1e-db11-08abfca00c4a}]" - testing.go:149: server-d230261e-52ee-9f48-2e93-d683c22ff6ab: 2024-09-03T21:34:32.049Z [INFO] server-d230261e-52ee-9f48-2e93-d683c22ff6ab: updating configuration: command=AddStaging server-id=server-b667d949-7421-bd1e-db11-08abfca00c4a server-addr=b667d949-7421-bd1e-db11-08abfca00c4a servers="[{Suffrage:Voter ID:server-d230261e-52ee-9f48-2e93-d683c22ff6ab Address:d230261e-52ee-9f48-2e93-d683c22ff6ab} {Suffrage:Voter ID:server-dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7 Address:dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7} {Suffrage:Voter ID:server-b667d949-7421-bd1e-db11-08abfca00c4a Address:b667d949-7421-bd1e-db11-08abfca00c4a}]" -testLoggerAdapter verbose: server-d230261e-52ee-9f48-2e93-d683c22ff6ab: 2024-09-03T21:34:32.049Z [INFO] server-d230261e-52ee-9f48-2e93-d683c22ff6ab: added peer, starting replication: peer=server-b667d949-7421-bd1e-db11-08abfca00c4a - testing.go:149: server-d230261e-52ee-9f48-2e93-d683c22ff6ab: 2024-09-03T21:34:32.049Z [INFO] server-d230261e-52ee-9f48-2e93-d683c22ff6ab: added peer, starting replication: peer=server-b667d949-7421-bd1e-db11-08abfca00c4a -testLoggerAdapter verbose: server-b667d949-7421-bd1e-db11-08abfca00c4a: 2024-09-03T21:34:32.049Z [WARN] server-b667d949-7421-bd1e-db11-08abfca00c4a: failed to get previous log: previous-index=4 last-index=3 error="log not found" - testing.go:149: server-b667d949-7421-bd1e-db11-08abfca00c4a: 2024-09-03T21:34:32.049Z [WARN] server-b667d949-7421-bd1e-db11-08abfca00c4a: failed to get previous log: previous-index=4 last-index=3 error="log not found" -testLoggerAdapter verbose: server-d230261e-52ee-9f48-2e93-d683c22ff6ab: 2024-09-03T21:34:32.049Z [WARN] server-d230261e-52ee-9f48-2e93-d683c22ff6ab: appendEntries rejected, sending older logs: peer="{Voter server-b667d949-7421-bd1e-db11-08abfca00c4a b667d949-7421-bd1e-db11-08abfca00c4a}" next=4 - testing.go:149: server-d230261e-52ee-9f48-2e93-d683c22ff6ab: 2024-09-03T21:34:32.049Z [WARN] server-d230261e-52ee-9f48-2e93-d683c22ff6ab: appendEntries rejected, sending older logs: peer="{Voter server-b667d949-7421-bd1e-db11-08abfca00c4a b667d949-7421-bd1e-db11-08abfca00c4a}" next=4 -testLoggerAdapter verbose: server-d230261e-52ee-9f48-2e93-d683c22ff6ab: 2024-09-03T21:34:32.049Z [INFO] server-d230261e-52ee-9f48-2e93-d683c22ff6ab: pipelining replication: peer="{Voter server-b667d949-7421-bd1e-db11-08abfca00c4a b667d949-7421-bd1e-db11-08abfca00c4a}" - testing.go:149: server-d230261e-52ee-9f48-2e93-d683c22ff6ab: 2024-09-03T21:34:32.049Z [INFO] server-d230261e-52ee-9f48-2e93-d683c22ff6ab: pipelining replication: peer="{Voter server-b667d949-7421-bd1e-db11-08abfca00c4a b667d949-7421-bd1e-db11-08abfca00c4a}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:32.049Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:32.049Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:32.155Z [INFO] stable state for Leader reached at 2024-09-03 21:34:32.050002595 +0000 UTC m=+13.096659737 (1 nodes), 337ns from start of poll, 407.159894ms from cluster start. Timeout at 2024-09-03 21:34:32.155079393 +0000 UTC m=+13.201736527, 105.07679ms after stability - testing.go:149: cluster: 2024-09-03T21:34:32.155Z [INFO] stable state for Leader reached at 2024-09-03 21:34:32.050002595 +0000 UTC m=+13.096659737 (1 nodes), 337ns from start of poll, 407.159894ms from cluster start. Timeout at 2024-09-03 21:34:32.155079393 +0000 UTC m=+13.201736527, 105.07679ms after stability -testLoggerAdapter verbose: server-d230261e-52ee-9f48-2e93-d683c22ff6ab: 2024-09-03T21:34:32.155Z [INFO] server-d230261e-52ee-9f48-2e93-d683c22ff6ab: aborting pipeline replication: peer="{Voter server-dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7 dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7}" - testing.go:149: server-d230261e-52ee-9f48-2e93-d683c22ff6ab: 2024-09-03T21:34:32.155Z [INFO] server-d230261e-52ee-9f48-2e93-d683c22ff6ab: aborting pipeline replication: peer="{Voter server-dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7 dd27aa65-8e7c-c510-8fc4-0d32e6ce85f7}" -testLoggerAdapter verbose: server-d230261e-52ee-9f48-2e93-d683c22ff6ab: 2024-09-03T21:34:32.155Z [INFO] server-d230261e-52ee-9f48-2e93-d683c22ff6ab: aborting pipeline replication: peer="{Voter server-b667d949-7421-bd1e-db11-08abfca00c4a b667d949-7421-bd1e-db11-08abfca00c4a}" - testing.go:149: server-d230261e-52ee-9f48-2e93-d683c22ff6ab: 2024-09-03T21:34:32.155Z [INFO] server-d230261e-52ee-9f48-2e93-d683c22ff6ab: aborting pipeline replication: peer="{Voter server-b667d949-7421-bd1e-db11-08abfca00c4a b667d949-7421-bd1e-db11-08abfca00c4a}" ---- PASS: TestRaft_ReJoinFollower (0.51s) +testLoggerAdapter verbose: server-2eb76803-14cb-629a-acf5-13a64eb4363d: 2023-08-02T15:13:00.562Z [INFO] server-2eb76803-14cb-629a-acf5-13a64eb4363d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2eb76803-14cb-629a-acf5-13a64eb4363d Address:2eb76803-14cb-629a-acf5-13a64eb4363d} {Suffrage:Voter ID:server-52547b59-7eae-c603-5b0b-827290183393 Address:52547b59-7eae-c603-5b0b-827290183393} {Suffrage:Voter ID:server-121f24e6-86e6-79bf-62ed-7d3216c32cfb Address:121f24e6-86e6-79bf-62ed-7d3216c32cfb}]" + testing.go:149: server-2eb76803-14cb-629a-acf5-13a64eb4363d: 2023-08-02T15:13:00.562Z [INFO] server-2eb76803-14cb-629a-acf5-13a64eb4363d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2eb76803-14cb-629a-acf5-13a64eb4363d Address:2eb76803-14cb-629a-acf5-13a64eb4363d} {Suffrage:Voter ID:server-52547b59-7eae-c603-5b0b-827290183393 Address:52547b59-7eae-c603-5b0b-827290183393} {Suffrage:Voter ID:server-121f24e6-86e6-79bf-62ed-7d3216c32cfb Address:121f24e6-86e6-79bf-62ed-7d3216c32cfb}]" +testLoggerAdapter verbose: server-52547b59-7eae-c603-5b0b-827290183393: 2023-08-02T15:13:00.562Z [INFO] server-52547b59-7eae-c603-5b0b-827290183393: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2eb76803-14cb-629a-acf5-13a64eb4363d Address:2eb76803-14cb-629a-acf5-13a64eb4363d} {Suffrage:Voter ID:server-52547b59-7eae-c603-5b0b-827290183393 Address:52547b59-7eae-c603-5b0b-827290183393} {Suffrage:Voter ID:server-121f24e6-86e6-79bf-62ed-7d3216c32cfb Address:121f24e6-86e6-79bf-62ed-7d3216c32cfb}]" + testing.go:149: server-52547b59-7eae-c603-5b0b-827290183393: 2023-08-02T15:13:00.562Z [INFO] server-52547b59-7eae-c603-5b0b-827290183393: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2eb76803-14cb-629a-acf5-13a64eb4363d Address:2eb76803-14cb-629a-acf5-13a64eb4363d} {Suffrage:Voter ID:server-52547b59-7eae-c603-5b0b-827290183393 Address:52547b59-7eae-c603-5b0b-827290183393} {Suffrage:Voter ID:server-121f24e6-86e6-79bf-62ed-7d3216c32cfb Address:121f24e6-86e6-79bf-62ed-7d3216c32cfb}]" +testLoggerAdapter verbose: server-2eb76803-14cb-629a-acf5-13a64eb4363d: 2023-08-02T15:13:00.562Z [INFO] server-2eb76803-14cb-629a-acf5-13a64eb4363d: entering follower state: follower="Node at 2eb76803-14cb-629a-acf5-13a64eb4363d [Follower]" leader= + testing.go:149: server-2eb76803-14cb-629a-acf5-13a64eb4363d: 2023-08-02T15:13:00.562Z [INFO] server-2eb76803-14cb-629a-acf5-13a64eb4363d: entering follower state: follower="Node at 2eb76803-14cb-629a-acf5-13a64eb4363d [Follower]" leader= +testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.563Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2eb76803-14cb-629a-acf5-13a64eb4363d Address:2eb76803-14cb-629a-acf5-13a64eb4363d} {Suffrage:Voter ID:server-52547b59-7eae-c603-5b0b-827290183393 Address:52547b59-7eae-c603-5b0b-827290183393} {Suffrage:Voter ID:server-121f24e6-86e6-79bf-62ed-7d3216c32cfb Address:121f24e6-86e6-79bf-62ed-7d3216c32cfb}]" + testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.563Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2eb76803-14cb-629a-acf5-13a64eb4363d Address:2eb76803-14cb-629a-acf5-13a64eb4363d} {Suffrage:Voter ID:server-52547b59-7eae-c603-5b0b-827290183393 Address:52547b59-7eae-c603-5b0b-827290183393} {Suffrage:Voter ID:server-121f24e6-86e6-79bf-62ed-7d3216c32cfb Address:121f24e6-86e6-79bf-62ed-7d3216c32cfb}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:00.563Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:13:00.563Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-52547b59-7eae-c603-5b0b-827290183393: 2023-08-02T15:13:00.563Z [INFO] server-52547b59-7eae-c603-5b0b-827290183393: entering follower state: follower="Node at 52547b59-7eae-c603-5b0b-827290183393 [Follower]" leader= + testing.go:149: server-52547b59-7eae-c603-5b0b-827290183393: 2023-08-02T15:13:00.563Z [INFO] server-52547b59-7eae-c603-5b0b-827290183393: entering follower state: follower="Node at 52547b59-7eae-c603-5b0b-827290183393 [Follower]" leader= +testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.563Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: entering follower state: follower="Node at 121f24e6-86e6-79bf-62ed-7d3216c32cfb [Follower]" leader= + testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.563Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: entering follower state: follower="Node at 121f24e6-86e6-79bf-62ed-7d3216c32cfb [Follower]" leader= +testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.619Z [WARN] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.619Z [WARN] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.619Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: entering candidate state: node="Node at 121f24e6-86e6-79bf-62ed-7d3216c32cfb [Candidate]" term=2 + testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.619Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: entering candidate state: node="Node at 121f24e6-86e6-79bf-62ed-7d3216c32cfb [Candidate]" term=2 +testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.619Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: election won: tally=2 + testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.619Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: election won: tally=2 +testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.619Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: entering leader state: leader="Node at 121f24e6-86e6-79bf-62ed-7d3216c32cfb [Leader]" + testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.619Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: entering leader state: leader="Node at 121f24e6-86e6-79bf-62ed-7d3216c32cfb [Leader]" +testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.619Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: added peer, starting replication: peer=server-2eb76803-14cb-629a-acf5-13a64eb4363d + testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.619Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: added peer, starting replication: peer=server-2eb76803-14cb-629a-acf5-13a64eb4363d +testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.619Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: added peer, starting replication: peer=server-52547b59-7eae-c603-5b0b-827290183393 + testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.619Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: added peer, starting replication: peer=server-52547b59-7eae-c603-5b0b-827290183393 +testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.619Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: pipelining replication: peer="{Voter server-52547b59-7eae-c603-5b0b-827290183393 52547b59-7eae-c603-5b0b-827290183393}" + testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.619Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: pipelining replication: peer="{Voter server-52547b59-7eae-c603-5b0b-827290183393 52547b59-7eae-c603-5b0b-827290183393}" +testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.620Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: pipelining replication: peer="{Voter server-2eb76803-14cb-629a-acf5-13a64eb4363d 2eb76803-14cb-629a-acf5-13a64eb4363d}" + testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.620Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: pipelining replication: peer="{Voter server-2eb76803-14cb-629a-acf5-13a64eb4363d 2eb76803-14cb-629a-acf5-13a64eb4363d}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:00.725Z [INFO] stable state for Leader reached at 2023-08-02 15:13:00.620109742 +0000 UTC m=+12.716046069 (1 nodes), 56.967285ms from start of poll, 57.408444ms from cluster start. Timeout at 2023-08-02 15:13:00.72521219 +0000 UTC m=+12.821148523, 105.102454ms after stability + testing.go:149: cluster: 2023-08-02T15:13:00.725Z [INFO] stable state for Leader reached at 2023-08-02 15:13:00.620109742 +0000 UTC m=+12.716046069 (1 nodes), 56.967285ms from start of poll, 57.408444ms from cluster start. Timeout at 2023-08-02 15:13:00.72521219 +0000 UTC m=+12.821148523, 105.102454ms after stability +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:00.730Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-08-02T15:13:00.730Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:00.835Z [INFO] stable state for Follower reached at 2023-08-02 15:13:00.730583341 +0000 UTC m=+12.826519669 (2 nodes), 2.917µs from start of poll, 167.882044ms from cluster start. Timeout at 2023-08-02 15:13:00.835626419 +0000 UTC m=+12.931562753, 105.043084ms after stability + testing.go:149: cluster: 2023-08-02T15:13:00.835Z [INFO] stable state for Follower reached at 2023-08-02 15:13:00.730583341 +0000 UTC m=+12.826519669 (2 nodes), 2.917µs from start of poll, 167.882044ms from cluster start. Timeout at 2023-08-02 15:13:00.835626419 +0000 UTC m=+12.931562753, 105.043084ms after stability +testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.835Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: updating configuration: command=RemoveServer server-id=server-2eb76803-14cb-629a-acf5-13a64eb4363d server-addr= servers="[{Suffrage:Voter ID:server-52547b59-7eae-c603-5b0b-827290183393 Address:52547b59-7eae-c603-5b0b-827290183393} {Suffrage:Voter ID:server-121f24e6-86e6-79bf-62ed-7d3216c32cfb Address:121f24e6-86e6-79bf-62ed-7d3216c32cfb}]" + testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.835Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: updating configuration: command=RemoveServer server-id=server-2eb76803-14cb-629a-acf5-13a64eb4363d server-addr= servers="[{Suffrage:Voter ID:server-52547b59-7eae-c603-5b0b-827290183393 Address:52547b59-7eae-c603-5b0b-827290183393} {Suffrage:Voter ID:server-121f24e6-86e6-79bf-62ed-7d3216c32cfb Address:121f24e6-86e6-79bf-62ed-7d3216c32cfb}]" +testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.835Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: removed peer, stopping replication: peer=server-2eb76803-14cb-629a-acf5-13a64eb4363d last-index=3 + testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.835Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: removed peer, stopping replication: peer=server-2eb76803-14cb-629a-acf5-13a64eb4363d last-index=3 +testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.835Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: aborting pipeline replication: peer="{Voter server-2eb76803-14cb-629a-acf5-13a64eb4363d 2eb76803-14cb-629a-acf5-13a64eb4363d}" + testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.835Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: aborting pipeline replication: peer="{Voter server-2eb76803-14cb-629a-acf5-13a64eb4363d 2eb76803-14cb-629a-acf5-13a64eb4363d}" +testLoggerAdapter verbose: server-2eb76803-14cb-629a-acf5-13a64eb4363d: 2023-08-02T15:13:00.921Z [WARN] server-2eb76803-14cb-629a-acf5-13a64eb4363d: heartbeat timeout reached, starting election: last-leader=121f24e6-86e6-79bf-62ed-7d3216c32cfb + testing.go:149: server-2eb76803-14cb-629a-acf5-13a64eb4363d: 2023-08-02T15:13:00.921Z [WARN] server-2eb76803-14cb-629a-acf5-13a64eb4363d: heartbeat timeout reached, starting election: last-leader=121f24e6-86e6-79bf-62ed-7d3216c32cfb +testLoggerAdapter verbose: server-2eb76803-14cb-629a-acf5-13a64eb4363d: 2023-08-02T15:13:00.922Z [INFO] server-2eb76803-14cb-629a-acf5-13a64eb4363d: entering candidate state: node="Node at 2eb76803-14cb-629a-acf5-13a64eb4363d [Candidate]" term=3 + testing.go:149: server-2eb76803-14cb-629a-acf5-13a64eb4363d: 2023-08-02T15:13:00.922Z [INFO] server-2eb76803-14cb-629a-acf5-13a64eb4363d: entering candidate state: node="Node at 2eb76803-14cb-629a-acf5-13a64eb4363d [Candidate]" term=3 +testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.922Z [WARN] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: rejecting vote request since we have a leader: from=2eb76803-14cb-629a-acf5-13a64eb4363d leader=121f24e6-86e6-79bf-62ed-7d3216c32cfb + testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.922Z [WARN] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: rejecting vote request since we have a leader: from=2eb76803-14cb-629a-acf5-13a64eb4363d leader=121f24e6-86e6-79bf-62ed-7d3216c32cfb +testLoggerAdapter verbose: server-52547b59-7eae-c603-5b0b-827290183393: 2023-08-02T15:13:00.922Z [WARN] server-52547b59-7eae-c603-5b0b-827290183393: rejecting vote request since we have a leader: from=2eb76803-14cb-629a-acf5-13a64eb4363d leader=121f24e6-86e6-79bf-62ed-7d3216c32cfb + testing.go:149: server-52547b59-7eae-c603-5b0b-827290183393: 2023-08-02T15:13:00.922Z [WARN] server-52547b59-7eae-c603-5b0b-827290183393: rejecting vote request since we have a leader: from=2eb76803-14cb-629a-acf5-13a64eb4363d leader=121f24e6-86e6-79bf-62ed-7d3216c32cfb +testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.941Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: updating configuration: command=AddStaging server-id=server-2eb76803-14cb-629a-acf5-13a64eb4363d server-addr=2eb76803-14cb-629a-acf5-13a64eb4363d servers="[{Suffrage:Voter ID:server-52547b59-7eae-c603-5b0b-827290183393 Address:52547b59-7eae-c603-5b0b-827290183393} {Suffrage:Voter ID:server-121f24e6-86e6-79bf-62ed-7d3216c32cfb Address:121f24e6-86e6-79bf-62ed-7d3216c32cfb} {Suffrage:Voter ID:server-2eb76803-14cb-629a-acf5-13a64eb4363d Address:2eb76803-14cb-629a-acf5-13a64eb4363d}]" + testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.941Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: updating configuration: command=AddStaging server-id=server-2eb76803-14cb-629a-acf5-13a64eb4363d server-addr=2eb76803-14cb-629a-acf5-13a64eb4363d servers="[{Suffrage:Voter ID:server-52547b59-7eae-c603-5b0b-827290183393 Address:52547b59-7eae-c603-5b0b-827290183393} {Suffrage:Voter ID:server-121f24e6-86e6-79bf-62ed-7d3216c32cfb Address:121f24e6-86e6-79bf-62ed-7d3216c32cfb} {Suffrage:Voter ID:server-2eb76803-14cb-629a-acf5-13a64eb4363d Address:2eb76803-14cb-629a-acf5-13a64eb4363d}]" +testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.941Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: added peer, starting replication: peer=server-2eb76803-14cb-629a-acf5-13a64eb4363d + testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.941Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: added peer, starting replication: peer=server-2eb76803-14cb-629a-acf5-13a64eb4363d +testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.941Z [ERROR] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: peer has newer term, stopping replication: peer="{Voter server-2eb76803-14cb-629a-acf5-13a64eb4363d 2eb76803-14cb-629a-acf5-13a64eb4363d}" + testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.941Z [ERROR] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: peer has newer term, stopping replication: peer="{Voter server-2eb76803-14cb-629a-acf5-13a64eb4363d 2eb76803-14cb-629a-acf5-13a64eb4363d}" +testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.941Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: entering follower state: follower="Node at 121f24e6-86e6-79bf-62ed-7d3216c32cfb [Follower]" leader= + testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.941Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: entering follower state: follower="Node at 121f24e6-86e6-79bf-62ed-7d3216c32cfb [Follower]" leader= +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:00.941Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:13:00.941Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.941Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: aborting pipeline replication: peer="{Voter server-52547b59-7eae-c603-5b0b-827290183393 52547b59-7eae-c603-5b0b-827290183393}" + testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.941Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: aborting pipeline replication: peer="{Voter server-52547b59-7eae-c603-5b0b-827290183393 52547b59-7eae-c603-5b0b-827290183393}" +testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.006Z [WARN] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.006Z [WARN] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.006Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: entering candidate state: node="Node at 121f24e6-86e6-79bf-62ed-7d3216c32cfb [Candidate]" term=3 + testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.006Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: entering candidate state: node="Node at 121f24e6-86e6-79bf-62ed-7d3216c32cfb [Candidate]" term=3 +testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.006Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: election won: tally=2 + testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.006Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: election won: tally=2 +testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.006Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: entering leader state: leader="Node at 121f24e6-86e6-79bf-62ed-7d3216c32cfb [Leader]" + testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.006Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: entering leader state: leader="Node at 121f24e6-86e6-79bf-62ed-7d3216c32cfb [Leader]" +testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.006Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: added peer, starting replication: peer=server-52547b59-7eae-c603-5b0b-827290183393 + testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.006Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: added peer, starting replication: peer=server-52547b59-7eae-c603-5b0b-827290183393 +testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.006Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: added peer, starting replication: peer=server-2eb76803-14cb-629a-acf5-13a64eb4363d + testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.006Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: added peer, starting replication: peer=server-2eb76803-14cb-629a-acf5-13a64eb4363d +testLoggerAdapter verbose: server-2eb76803-14cb-629a-acf5-13a64eb4363d: 2023-08-02T15:13:01.007Z [WARN] server-2eb76803-14cb-629a-acf5-13a64eb4363d: failed to get previous log: previous-index=4 last-index=3 error="log not found" + testing.go:149: server-2eb76803-14cb-629a-acf5-13a64eb4363d: 2023-08-02T15:13:01.007Z [WARN] server-2eb76803-14cb-629a-acf5-13a64eb4363d: failed to get previous log: previous-index=4 last-index=3 error="log not found" +testLoggerAdapter verbose: server-2eb76803-14cb-629a-acf5-13a64eb4363d: 2023-08-02T15:13:01.007Z [INFO] server-2eb76803-14cb-629a-acf5-13a64eb4363d: entering follower state: follower="Node at 2eb76803-14cb-629a-acf5-13a64eb4363d [Follower]" leader=121f24e6-86e6-79bf-62ed-7d3216c32cfb + testing.go:149: server-2eb76803-14cb-629a-acf5-13a64eb4363d: 2023-08-02T15:13:01.007Z [INFO] server-2eb76803-14cb-629a-acf5-13a64eb4363d: entering follower state: follower="Node at 2eb76803-14cb-629a-acf5-13a64eb4363d [Follower]" leader=121f24e6-86e6-79bf-62ed-7d3216c32cfb +testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.007Z [WARN] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: appendEntries rejected, sending older logs: peer="{Voter server-2eb76803-14cb-629a-acf5-13a64eb4363d 2eb76803-14cb-629a-acf5-13a64eb4363d}" next=4 + testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.007Z [WARN] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: appendEntries rejected, sending older logs: peer="{Voter server-2eb76803-14cb-629a-acf5-13a64eb4363d 2eb76803-14cb-629a-acf5-13a64eb4363d}" next=4 +testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.007Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: pipelining replication: peer="{Voter server-52547b59-7eae-c603-5b0b-827290183393 52547b59-7eae-c603-5b0b-827290183393}" + testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.007Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: pipelining replication: peer="{Voter server-52547b59-7eae-c603-5b0b-827290183393 52547b59-7eae-c603-5b0b-827290183393}" +testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.007Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: pipelining replication: peer="{Voter server-2eb76803-14cb-629a-acf5-13a64eb4363d 2eb76803-14cb-629a-acf5-13a64eb4363d}" + testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.007Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: pipelining replication: peer="{Voter server-2eb76803-14cb-629a-acf5-13a64eb4363d 2eb76803-14cb-629a-acf5-13a64eb4363d}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:01.112Z [INFO] stable state for Leader reached at 2023-08-02 15:13:01.007107292 +0000 UTC m=+13.103043620 (1 nodes), 65.281287ms from start of poll, 444.405995ms from cluster start. Timeout at 2023-08-02 15:13:01.112207411 +0000 UTC m=+13.208143750, 105.10013ms after stability + testing.go:149: cluster: 2023-08-02T15:13:01.112Z [INFO] stable state for Leader reached at 2023-08-02 15:13:01.007107292 +0000 UTC m=+13.103043620 (1 nodes), 65.281287ms from start of poll, 444.405995ms from cluster start. Timeout at 2023-08-02 15:13:01.112207411 +0000 UTC m=+13.208143750, 105.10013ms after stability +testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.112Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: aborting pipeline replication: peer="{Voter server-2eb76803-14cb-629a-acf5-13a64eb4363d 2eb76803-14cb-629a-acf5-13a64eb4363d}" + testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.112Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: aborting pipeline replication: peer="{Voter server-2eb76803-14cb-629a-acf5-13a64eb4363d 2eb76803-14cb-629a-acf5-13a64eb4363d}" +testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.112Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: aborting pipeline replication: peer="{Voter server-52547b59-7eae-c603-5b0b-827290183393 52547b59-7eae-c603-5b0b-827290183393}" + testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.112Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: aborting pipeline replication: peer="{Voter server-52547b59-7eae-c603-5b0b-827290183393 52547b59-7eae-c603-5b0b-827290183393}" +--- PASS: TestRaft_ReJoinFollower (0.55s) === RUN TestRaft_LeaderLeaseExpire -testLoggerAdapter verbose: server-412e69ce-d0f3-7c78-822e-b23444158b37: 2024-09-03T21:34:32.156Z [INFO] server-412e69ce-d0f3-7c78-822e-b23444158b37: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-412e69ce-d0f3-7c78-822e-b23444158b37 Address:412e69ce-d0f3-7c78-822e-b23444158b37} {Suffrage:Voter ID:server-f34e1492-862a-67b2-22e6-48bccfe84443 Address:f34e1492-862a-67b2-22e6-48bccfe84443}]" - testing.go:149: server-412e69ce-d0f3-7c78-822e-b23444158b37: 2024-09-03T21:34:32.156Z [INFO] server-412e69ce-d0f3-7c78-822e-b23444158b37: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-412e69ce-d0f3-7c78-822e-b23444158b37 Address:412e69ce-d0f3-7c78-822e-b23444158b37} {Suffrage:Voter ID:server-f34e1492-862a-67b2-22e6-48bccfe84443 Address:f34e1492-862a-67b2-22e6-48bccfe84443}]" -testLoggerAdapter verbose: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.156Z [INFO] server-f34e1492-862a-67b2-22e6-48bccfe84443: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-412e69ce-d0f3-7c78-822e-b23444158b37 Address:412e69ce-d0f3-7c78-822e-b23444158b37} {Suffrage:Voter ID:server-f34e1492-862a-67b2-22e6-48bccfe84443 Address:f34e1492-862a-67b2-22e6-48bccfe84443}]" - testing.go:149: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.156Z [INFO] server-f34e1492-862a-67b2-22e6-48bccfe84443: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-412e69ce-d0f3-7c78-822e-b23444158b37 Address:412e69ce-d0f3-7c78-822e-b23444158b37} {Suffrage:Voter ID:server-f34e1492-862a-67b2-22e6-48bccfe84443 Address:f34e1492-862a-67b2-22e6-48bccfe84443}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:32.156Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:32.156Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-412e69ce-d0f3-7c78-822e-b23444158b37: 2024-09-03T21:34:32.156Z [INFO] server-412e69ce-d0f3-7c78-822e-b23444158b37: entering follower state: follower="Node at 412e69ce-d0f3-7c78-822e-b23444158b37 [Follower]" leader= - testing.go:149: server-412e69ce-d0f3-7c78-822e-b23444158b37: 2024-09-03T21:34:32.156Z [INFO] server-412e69ce-d0f3-7c78-822e-b23444158b37: entering follower state: follower="Node at 412e69ce-d0f3-7c78-822e-b23444158b37 [Follower]" leader= -testLoggerAdapter verbose: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.156Z [INFO] server-f34e1492-862a-67b2-22e6-48bccfe84443: entering follower state: follower="Node at f34e1492-862a-67b2-22e6-48bccfe84443 [Follower]" leader= - testing.go:149: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.156Z [INFO] server-f34e1492-862a-67b2-22e6-48bccfe84443: entering follower state: follower="Node at f34e1492-862a-67b2-22e6-48bccfe84443 [Follower]" leader= -testLoggerAdapter verbose: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.242Z [WARN] server-f34e1492-862a-67b2-22e6-48bccfe84443: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.242Z [WARN] server-f34e1492-862a-67b2-22e6-48bccfe84443: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.242Z [INFO] server-f34e1492-862a-67b2-22e6-48bccfe84443: entering candidate state: node="Node at f34e1492-862a-67b2-22e6-48bccfe84443 [Candidate]" term=2 - testing.go:149: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.242Z [INFO] server-f34e1492-862a-67b2-22e6-48bccfe84443: entering candidate state: node="Node at f34e1492-862a-67b2-22e6-48bccfe84443 [Candidate]" term=2 -testLoggerAdapter verbose: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.242Z [INFO] server-f34e1492-862a-67b2-22e6-48bccfe84443: election won: tally=2 - testing.go:149: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.242Z [INFO] server-f34e1492-862a-67b2-22e6-48bccfe84443: election won: tally=2 -testLoggerAdapter verbose: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.243Z [INFO] server-f34e1492-862a-67b2-22e6-48bccfe84443: entering leader state: leader="Node at f34e1492-862a-67b2-22e6-48bccfe84443 [Leader]" - testing.go:149: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.243Z [INFO] server-f34e1492-862a-67b2-22e6-48bccfe84443: entering leader state: leader="Node at f34e1492-862a-67b2-22e6-48bccfe84443 [Leader]" -testLoggerAdapter verbose: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.243Z [INFO] server-f34e1492-862a-67b2-22e6-48bccfe84443: added peer, starting replication: peer=server-412e69ce-d0f3-7c78-822e-b23444158b37 - testing.go:149: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.243Z [INFO] server-f34e1492-862a-67b2-22e6-48bccfe84443: added peer, starting replication: peer=server-412e69ce-d0f3-7c78-822e-b23444158b37 -testLoggerAdapter verbose: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.243Z [INFO] server-f34e1492-862a-67b2-22e6-48bccfe84443: pipelining replication: peer="{Voter server-412e69ce-d0f3-7c78-822e-b23444158b37 412e69ce-d0f3-7c78-822e-b23444158b37}" - testing.go:149: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.243Z [INFO] server-f34e1492-862a-67b2-22e6-48bccfe84443: pipelining replication: peer="{Voter server-412e69ce-d0f3-7c78-822e-b23444158b37 412e69ce-d0f3-7c78-822e-b23444158b37}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:32.348Z [INFO] stable state for Leader reached at 2024-09-03 21:34:32.243257359 +0000 UTC m=+13.289914492 (1 nodes), 86.992449ms from start of poll, 87.333821ms from cluster start. Timeout at 2024-09-03 21:34:32.348339621 +0000 UTC m=+13.394996755, 105.082263ms after stability - testing.go:149: cluster: 2024-09-03T21:34:32.348Z [INFO] stable state for Leader reached at 2024-09-03 21:34:32.243257359 +0000 UTC m=+13.289914492 (1 nodes), 86.992449ms from start of poll, 87.333821ms from cluster start. Timeout at 2024-09-03 21:34:32.348339621 +0000 UTC m=+13.394996755, 105.082263ms after stability -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:32.353Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-09-03T21:34:32.353Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:32.458Z [INFO] stable state for Follower reached at 2024-09-03 21:34:32.353610415 +0000 UTC m=+13.400267558 (1 nodes), 1.08µs from start of poll, 197.686887ms from cluster start. Timeout at 2024-09-03 21:34:32.458692947 +0000 UTC m=+13.505350081, 105.082523ms after stability - testing.go:149: cluster: 2024-09-03T21:34:32.458Z [INFO] stable state for Follower reached at 2024-09-03 21:34:32.353610415 +0000 UTC m=+13.400267558 (1 nodes), 1.08µs from start of poll, 197.686887ms from cluster start. Timeout at 2024-09-03 21:34:32.458692947 +0000 UTC m=+13.505350081, 105.082523ms after stability - raft_test.go:1492: [INFO] Disconnecting Node at 412e69ce-d0f3-7c78-822e-b23444158b37 [Follower] -testLoggerAdapter verbose: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.463Z [ERROR] server-f34e1492-862a-67b2-22e6-48bccfe84443: failed to pipeline appendEntries: peer="{Voter server-412e69ce-d0f3-7c78-822e-b23444158b37 412e69ce-d0f3-7c78-822e-b23444158b37}" error="append pipeline closed" - testing.go:149: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.463Z [ERROR] server-f34e1492-862a-67b2-22e6-48bccfe84443: failed to pipeline appendEntries: peer="{Voter server-412e69ce-d0f3-7c78-822e-b23444158b37 412e69ce-d0f3-7c78-822e-b23444158b37}" error="append pipeline closed" -testLoggerAdapter verbose: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.463Z [INFO] server-f34e1492-862a-67b2-22e6-48bccfe84443: aborting pipeline replication: peer="{Voter server-412e69ce-d0f3-7c78-822e-b23444158b37 412e69ce-d0f3-7c78-822e-b23444158b37}" - testing.go:149: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.463Z [INFO] server-f34e1492-862a-67b2-22e6-48bccfe84443: aborting pipeline replication: peer="{Voter server-412e69ce-d0f3-7c78-822e-b23444158b37 412e69ce-d0f3-7c78-822e-b23444158b37}" -testLoggerAdapter verbose: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.467Z [ERROR] server-f34e1492-862a-67b2-22e6-48bccfe84443: failed to heartbeat to: peer=412e69ce-d0f3-7c78-822e-b23444158b37 error="failed to connect to peer: 412e69ce-d0f3-7c78-822e-b23444158b37" - testing.go:149: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.467Z [ERROR] server-f34e1492-862a-67b2-22e6-48bccfe84443: failed to heartbeat to: peer=412e69ce-d0f3-7c78-822e-b23444158b37 error="failed to connect to peer: 412e69ce-d0f3-7c78-822e-b23444158b37" -testLoggerAdapter verbose: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.469Z [ERROR] server-f34e1492-862a-67b2-22e6-48bccfe84443: failed to appendEntries to: peer="{Voter server-412e69ce-d0f3-7c78-822e-b23444158b37 412e69ce-d0f3-7c78-822e-b23444158b37}" error="failed to connect to peer: 412e69ce-d0f3-7c78-822e-b23444158b37" - testing.go:149: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.469Z [ERROR] server-f34e1492-862a-67b2-22e6-48bccfe84443: failed to appendEntries to: peer="{Voter server-412e69ce-d0f3-7c78-822e-b23444158b37 412e69ce-d0f3-7c78-822e-b23444158b37}" error="failed to connect to peer: 412e69ce-d0f3-7c78-822e-b23444158b37" -testLoggerAdapter verbose: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.485Z [ERROR] server-f34e1492-862a-67b2-22e6-48bccfe84443: failed to heartbeat to: peer=412e69ce-d0f3-7c78-822e-b23444158b37 error="failed to connect to peer: 412e69ce-d0f3-7c78-822e-b23444158b37" - testing.go:149: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.485Z [ERROR] server-f34e1492-862a-67b2-22e6-48bccfe84443: failed to heartbeat to: peer=412e69ce-d0f3-7c78-822e-b23444158b37 error="failed to connect to peer: 412e69ce-d0f3-7c78-822e-b23444158b37" -testLoggerAdapter verbose: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.486Z [ERROR] server-f34e1492-862a-67b2-22e6-48bccfe84443: failed to appendEntries to: peer="{Voter server-412e69ce-d0f3-7c78-822e-b23444158b37 412e69ce-d0f3-7c78-822e-b23444158b37}" error="failed to connect to peer: 412e69ce-d0f3-7c78-822e-b23444158b37" - testing.go:149: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.486Z [ERROR] server-f34e1492-862a-67b2-22e6-48bccfe84443: failed to appendEntries to: peer="{Voter server-412e69ce-d0f3-7c78-822e-b23444158b37 412e69ce-d0f3-7c78-822e-b23444158b37}" error="failed to connect to peer: 412e69ce-d0f3-7c78-822e-b23444158b37" -testLoggerAdapter verbose: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.501Z [ERROR] server-f34e1492-862a-67b2-22e6-48bccfe84443: failed to appendEntries to: peer="{Voter server-412e69ce-d0f3-7c78-822e-b23444158b37 412e69ce-d0f3-7c78-822e-b23444158b37}" error="failed to connect to peer: 412e69ce-d0f3-7c78-822e-b23444158b37" - testing.go:149: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.501Z [ERROR] server-f34e1492-862a-67b2-22e6-48bccfe84443: failed to appendEntries to: peer="{Voter server-412e69ce-d0f3-7c78-822e-b23444158b37 412e69ce-d0f3-7c78-822e-b23444158b37}" error="failed to connect to peer: 412e69ce-d0f3-7c78-822e-b23444158b37" -testLoggerAdapter verbose: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.503Z [ERROR] server-f34e1492-862a-67b2-22e6-48bccfe84443: failed to heartbeat to: peer=412e69ce-d0f3-7c78-822e-b23444158b37 error="failed to connect to peer: 412e69ce-d0f3-7c78-822e-b23444158b37" - testing.go:149: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.503Z [ERROR] server-f34e1492-862a-67b2-22e6-48bccfe84443: failed to heartbeat to: peer=412e69ce-d0f3-7c78-822e-b23444158b37 error="failed to connect to peer: 412e69ce-d0f3-7c78-822e-b23444158b37" -testLoggerAdapter verbose: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.507Z [WARN] server-f34e1492-862a-67b2-22e6-48bccfe84443: failed to contact: server-id=server-412e69ce-d0f3-7c78-822e-b23444158b37 time=50.097602ms - testing.go:149: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.507Z [WARN] server-f34e1492-862a-67b2-22e6-48bccfe84443: failed to contact: server-id=server-412e69ce-d0f3-7c78-822e-b23444158b37 time=50.097602ms -testLoggerAdapter verbose: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.508Z [WARN] server-f34e1492-862a-67b2-22e6-48bccfe84443: failed to contact quorum of nodes, stepping down - testing.go:149: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.508Z [WARN] server-f34e1492-862a-67b2-22e6-48bccfe84443: failed to contact quorum of nodes, stepping down -testLoggerAdapter verbose: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.508Z [INFO] server-f34e1492-862a-67b2-22e6-48bccfe84443: entering follower state: follower="Node at f34e1492-862a-67b2-22e6-48bccfe84443 [Follower]" leader= - testing.go:149: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.508Z [INFO] server-f34e1492-862a-67b2-22e6-48bccfe84443: entering follower state: follower="Node at f34e1492-862a-67b2-22e6-48bccfe84443 [Follower]" leader= -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:32.508Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:32.508Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-412e69ce-d0f3-7c78-822e-b23444158b37: 2024-09-03T21:34:32.526Z [WARN] server-412e69ce-d0f3-7c78-822e-b23444158b37: heartbeat timeout reached, starting election: last-leader=f34e1492-862a-67b2-22e6-48bccfe84443 - testing.go:149: server-412e69ce-d0f3-7c78-822e-b23444158b37: 2024-09-03T21:34:32.526Z [WARN] server-412e69ce-d0f3-7c78-822e-b23444158b37: heartbeat timeout reached, starting election: last-leader=f34e1492-862a-67b2-22e6-48bccfe84443 -testLoggerAdapter verbose: server-412e69ce-d0f3-7c78-822e-b23444158b37: 2024-09-03T21:34:32.527Z [INFO] server-412e69ce-d0f3-7c78-822e-b23444158b37: entering candidate state: node="Node at 412e69ce-d0f3-7c78-822e-b23444158b37 [Candidate]" term=3 - testing.go:149: server-412e69ce-d0f3-7c78-822e-b23444158b37: 2024-09-03T21:34:32.527Z [INFO] server-412e69ce-d0f3-7c78-822e-b23444158b37: entering candidate state: node="Node at 412e69ce-d0f3-7c78-822e-b23444158b37 [Candidate]" term=3 -testLoggerAdapter verbose: server-412e69ce-d0f3-7c78-822e-b23444158b37: 2024-09-03T21:34:32.527Z [ERROR] server-412e69ce-d0f3-7c78-822e-b23444158b37: failed to make requestVote RPC: target="{Voter server-f34e1492-862a-67b2-22e6-48bccfe84443 f34e1492-862a-67b2-22e6-48bccfe84443}" error="failed to connect to peer: f34e1492-862a-67b2-22e6-48bccfe84443" - testing.go:149: server-412e69ce-d0f3-7c78-822e-b23444158b37: 2024-09-03T21:34:32.527Z [ERROR] server-412e69ce-d0f3-7c78-822e-b23444158b37: failed to make requestVote RPC: target="{Voter server-f34e1492-862a-67b2-22e6-48bccfe84443 f34e1492-862a-67b2-22e6-48bccfe84443}" error="failed to connect to peer: f34e1492-862a-67b2-22e6-48bccfe84443" -testLoggerAdapter verbose: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.528Z [ERROR] server-f34e1492-862a-67b2-22e6-48bccfe84443: failed to appendEntries to: peer="{Voter server-412e69ce-d0f3-7c78-822e-b23444158b37 412e69ce-d0f3-7c78-822e-b23444158b37}" error="failed to connect to peer: 412e69ce-d0f3-7c78-822e-b23444158b37" - testing.go:149: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.528Z [ERROR] server-f34e1492-862a-67b2-22e6-48bccfe84443: failed to appendEntries to: peer="{Voter server-412e69ce-d0f3-7c78-822e-b23444158b37 412e69ce-d0f3-7c78-822e-b23444158b37}" error="failed to connect to peer: 412e69ce-d0f3-7c78-822e-b23444158b37" -testLoggerAdapter verbose: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.562Z [WARN] server-f34e1492-862a-67b2-22e6-48bccfe84443: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.562Z [WARN] server-f34e1492-862a-67b2-22e6-48bccfe84443: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.562Z [INFO] server-f34e1492-862a-67b2-22e6-48bccfe84443: entering candidate state: node="Node at f34e1492-862a-67b2-22e6-48bccfe84443 [Candidate]" term=3 - testing.go:149: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.562Z [INFO] server-f34e1492-862a-67b2-22e6-48bccfe84443: entering candidate state: node="Node at f34e1492-862a-67b2-22e6-48bccfe84443 [Candidate]" term=3 -testLoggerAdapter verbose: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.562Z [ERROR] server-f34e1492-862a-67b2-22e6-48bccfe84443: failed to make requestVote RPC: target="{Voter server-412e69ce-d0f3-7c78-822e-b23444158b37 412e69ce-d0f3-7c78-822e-b23444158b37}" error="failed to connect to peer: 412e69ce-d0f3-7c78-822e-b23444158b37" - testing.go:149: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.562Z [ERROR] server-f34e1492-862a-67b2-22e6-48bccfe84443: failed to make requestVote RPC: target="{Voter server-412e69ce-d0f3-7c78-822e-b23444158b37 412e69ce-d0f3-7c78-822e-b23444158b37}" error="failed to connect to peer: 412e69ce-d0f3-7c78-822e-b23444158b37" -testLoggerAdapter verbose: server-412e69ce-d0f3-7c78-822e-b23444158b37: 2024-09-03T21:34:32.625Z [WARN] server-412e69ce-d0f3-7c78-822e-b23444158b37: Election timeout reached, restarting election - testing.go:149: server-412e69ce-d0f3-7c78-822e-b23444158b37: 2024-09-03T21:34:32.625Z [WARN] server-412e69ce-d0f3-7c78-822e-b23444158b37: Election timeout reached, restarting election -testLoggerAdapter verbose: server-412e69ce-d0f3-7c78-822e-b23444158b37: 2024-09-03T21:34:32.625Z [INFO] server-412e69ce-d0f3-7c78-822e-b23444158b37: entering candidate state: node="Node at 412e69ce-d0f3-7c78-822e-b23444158b37 [Candidate]" term=4 - testing.go:149: server-412e69ce-d0f3-7c78-822e-b23444158b37: 2024-09-03T21:34:32.625Z [INFO] server-412e69ce-d0f3-7c78-822e-b23444158b37: entering candidate state: node="Node at 412e69ce-d0f3-7c78-822e-b23444158b37 [Candidate]" term=4 -testLoggerAdapter verbose: server-412e69ce-d0f3-7c78-822e-b23444158b37: 2024-09-03T21:34:32.625Z [ERROR] server-412e69ce-d0f3-7c78-822e-b23444158b37: failed to make requestVote RPC: target="{Voter server-f34e1492-862a-67b2-22e6-48bccfe84443 f34e1492-862a-67b2-22e6-48bccfe84443}" error="failed to connect to peer: f34e1492-862a-67b2-22e6-48bccfe84443" - testing.go:149: server-412e69ce-d0f3-7c78-822e-b23444158b37: 2024-09-03T21:34:32.625Z [ERROR] server-412e69ce-d0f3-7c78-822e-b23444158b37: failed to make requestVote RPC: target="{Voter server-f34e1492-862a-67b2-22e6-48bccfe84443 f34e1492-862a-67b2-22e6-48bccfe84443}" error="failed to connect to peer: f34e1492-862a-67b2-22e6-48bccfe84443" -testLoggerAdapter verbose: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.642Z [WARN] server-f34e1492-862a-67b2-22e6-48bccfe84443: Election timeout reached, restarting election - testing.go:149: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.642Z [WARN] server-f34e1492-862a-67b2-22e6-48bccfe84443: Election timeout reached, restarting election -testLoggerAdapter verbose: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.642Z [INFO] server-f34e1492-862a-67b2-22e6-48bccfe84443: entering candidate state: node="Node at f34e1492-862a-67b2-22e6-48bccfe84443 [Candidate]" term=4 - testing.go:149: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.642Z [INFO] server-f34e1492-862a-67b2-22e6-48bccfe84443: entering candidate state: node="Node at f34e1492-862a-67b2-22e6-48bccfe84443 [Candidate]" term=4 -testLoggerAdapter verbose: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.642Z [ERROR] server-f34e1492-862a-67b2-22e6-48bccfe84443: failed to make requestVote RPC: target="{Voter server-412e69ce-d0f3-7c78-822e-b23444158b37 412e69ce-d0f3-7c78-822e-b23444158b37}" error="failed to connect to peer: 412e69ce-d0f3-7c78-822e-b23444158b37" - testing.go:149: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.642Z [ERROR] server-f34e1492-862a-67b2-22e6-48bccfe84443: failed to make requestVote RPC: target="{Voter server-412e69ce-d0f3-7c78-822e-b23444158b37 412e69ce-d0f3-7c78-822e-b23444158b37}" error="failed to connect to peer: 412e69ce-d0f3-7c78-822e-b23444158b37" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:32.667Z [INFO] stable state for Leader reached at 2024-09-03 21:34:32.562637196 +0000 UTC m=+13.609294344 (0 nodes), 54.491015ms from start of poll, 406.713673ms from cluster start. Timeout at 2024-09-03 21:34:32.667741846 +0000 UTC m=+13.714398986, 105.104642ms after stability - testing.go:149: cluster: 2024-09-03T21:34:32.667Z [INFO] stable state for Leader reached at 2024-09-03 21:34:32.562637196 +0000 UTC m=+13.609294344 (0 nodes), 54.491015ms from start of poll, 406.713673ms from cluster start. Timeout at 2024-09-03 21:34:32.667741846 +0000 UTC m=+13.714398986, 105.104642ms after stability -testLoggerAdapter verbose: server-412e69ce-d0f3-7c78-822e-b23444158b37: 2024-09-03T21:34:32.696Z [WARN] server-412e69ce-d0f3-7c78-822e-b23444158b37: Election timeout reached, restarting election - testing.go:149: server-412e69ce-d0f3-7c78-822e-b23444158b37: 2024-09-03T21:34:32.696Z [WARN] server-412e69ce-d0f3-7c78-822e-b23444158b37: Election timeout reached, restarting election -testLoggerAdapter verbose: server-412e69ce-d0f3-7c78-822e-b23444158b37: 2024-09-03T21:34:32.696Z [INFO] server-412e69ce-d0f3-7c78-822e-b23444158b37: entering candidate state: node="Node at 412e69ce-d0f3-7c78-822e-b23444158b37 [Candidate]" term=5 - testing.go:149: server-412e69ce-d0f3-7c78-822e-b23444158b37: 2024-09-03T21:34:32.696Z [INFO] server-412e69ce-d0f3-7c78-822e-b23444158b37: entering candidate state: node="Node at 412e69ce-d0f3-7c78-822e-b23444158b37 [Candidate]" term=5 -testLoggerAdapter verbose: server-412e69ce-d0f3-7c78-822e-b23444158b37: 2024-09-03T21:34:32.696Z [ERROR] server-412e69ce-d0f3-7c78-822e-b23444158b37: failed to make requestVote RPC: target="{Voter server-f34e1492-862a-67b2-22e6-48bccfe84443 f34e1492-862a-67b2-22e6-48bccfe84443}" error="failed to connect to peer: f34e1492-862a-67b2-22e6-48bccfe84443" - testing.go:149: server-412e69ce-d0f3-7c78-822e-b23444158b37: 2024-09-03T21:34:32.696Z [ERROR] server-412e69ce-d0f3-7c78-822e-b23444158b37: failed to make requestVote RPC: target="{Voter server-f34e1492-862a-67b2-22e6-48bccfe84443 f34e1492-862a-67b2-22e6-48bccfe84443}" error="failed to connect to peer: f34e1492-862a-67b2-22e6-48bccfe84443" -testLoggerAdapter verbose: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.709Z [WARN] server-f34e1492-862a-67b2-22e6-48bccfe84443: Election timeout reached, restarting election - testing.go:149: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.709Z [WARN] server-f34e1492-862a-67b2-22e6-48bccfe84443: Election timeout reached, restarting election -testLoggerAdapter verbose: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.709Z [INFO] server-f34e1492-862a-67b2-22e6-48bccfe84443: entering candidate state: node="Node at f34e1492-862a-67b2-22e6-48bccfe84443 [Candidate]" term=5 - testing.go:149: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.709Z [INFO] server-f34e1492-862a-67b2-22e6-48bccfe84443: entering candidate state: node="Node at f34e1492-862a-67b2-22e6-48bccfe84443 [Candidate]" term=5 -testLoggerAdapter verbose: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.709Z [ERROR] server-f34e1492-862a-67b2-22e6-48bccfe84443: failed to make requestVote RPC: target="{Voter server-412e69ce-d0f3-7c78-822e-b23444158b37 412e69ce-d0f3-7c78-822e-b23444158b37}" error="failed to connect to peer: 412e69ce-d0f3-7c78-822e-b23444158b37" - testing.go:149: server-f34e1492-862a-67b2-22e6-48bccfe84443: 2024-09-03T21:34:32.709Z [ERROR] server-f34e1492-862a-67b2-22e6-48bccfe84443: failed to make requestVote RPC: target="{Voter server-412e69ce-d0f3-7c78-822e-b23444158b37 412e69ce-d0f3-7c78-822e-b23444158b37}" error="failed to connect to peer: 412e69ce-d0f3-7c78-822e-b23444158b37" -testLoggerAdapter verbose: server-412e69ce-d0f3-7c78-822e-b23444158b37: 2024-09-03T21:34:32.751Z [WARN] server-412e69ce-d0f3-7c78-822e-b23444158b37: Election timeout reached, restarting election - testing.go:149: server-412e69ce-d0f3-7c78-822e-b23444158b37: 2024-09-03T21:34:32.751Z [WARN] server-412e69ce-d0f3-7c78-822e-b23444158b37: Election timeout reached, restarting election -testLoggerAdapter verbose: server-412e69ce-d0f3-7c78-822e-b23444158b37: 2024-09-03T21:34:32.751Z [INFO] server-412e69ce-d0f3-7c78-822e-b23444158b37: entering candidate state: node="Node at 412e69ce-d0f3-7c78-822e-b23444158b37 [Candidate]" term=6 - testing.go:149: server-412e69ce-d0f3-7c78-822e-b23444158b37: 2024-09-03T21:34:32.751Z [INFO] server-412e69ce-d0f3-7c78-822e-b23444158b37: entering candidate state: node="Node at 412e69ce-d0f3-7c78-822e-b23444158b37 [Candidate]" term=6 -testLoggerAdapter verbose: server-412e69ce-d0f3-7c78-822e-b23444158b37: 2024-09-03T21:34:32.751Z [ERROR] server-412e69ce-d0f3-7c78-822e-b23444158b37: failed to make requestVote RPC: target="{Voter server-f34e1492-862a-67b2-22e6-48bccfe84443 f34e1492-862a-67b2-22e6-48bccfe84443}" error="failed to connect to peer: f34e1492-862a-67b2-22e6-48bccfe84443" - testing.go:149: server-412e69ce-d0f3-7c78-822e-b23444158b37: 2024-09-03T21:34:32.751Z [ERROR] server-412e69ce-d0f3-7c78-822e-b23444158b37: failed to make requestVote RPC: target="{Voter server-f34e1492-862a-67b2-22e6-48bccfe84443 f34e1492-862a-67b2-22e6-48bccfe84443}" error="failed to connect to peer: f34e1492-862a-67b2-22e6-48bccfe84443" ---- PASS: TestRaft_LeaderLeaseExpire (0.62s) +testLoggerAdapter verbose: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.113Z [INFO] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-412e45c2-b294-66dc-33a9-f062c8ae3eea Address:412e45c2-b294-66dc-33a9-f062c8ae3eea} {Suffrage:Voter ID:server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f Address:f153d14b-dc13-123e-9a82-a2ac10ac8f3f}]" + testing.go:149: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.113Z [INFO] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-412e45c2-b294-66dc-33a9-f062c8ae3eea Address:412e45c2-b294-66dc-33a9-f062c8ae3eea} {Suffrage:Voter ID:server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f Address:f153d14b-dc13-123e-9a82-a2ac10ac8f3f}]" +testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.113Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-412e45c2-b294-66dc-33a9-f062c8ae3eea Address:412e45c2-b294-66dc-33a9-f062c8ae3eea} {Suffrage:Voter ID:server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f Address:f153d14b-dc13-123e-9a82-a2ac10ac8f3f}]" + testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.113Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-412e45c2-b294-66dc-33a9-f062c8ae3eea Address:412e45c2-b294-66dc-33a9-f062c8ae3eea} {Suffrage:Voter ID:server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f Address:f153d14b-dc13-123e-9a82-a2ac10ac8f3f}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:01.113Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:13:01.113Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.113Z [INFO] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: entering follower state: follower="Node at 412e45c2-b294-66dc-33a9-f062c8ae3eea [Follower]" leader= + testing.go:149: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.113Z [INFO] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: entering follower state: follower="Node at 412e45c2-b294-66dc-33a9-f062c8ae3eea [Follower]" leader= +testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.113Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: entering follower state: follower="Node at f153d14b-dc13-123e-9a82-a2ac10ac8f3f [Follower]" leader= + testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.113Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: entering follower state: follower="Node at f153d14b-dc13-123e-9a82-a2ac10ac8f3f [Follower]" leader= +testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.183Z [WARN] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.183Z [WARN] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.183Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: entering candidate state: node="Node at f153d14b-dc13-123e-9a82-a2ac10ac8f3f [Candidate]" term=2 + testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.183Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: entering candidate state: node="Node at f153d14b-dc13-123e-9a82-a2ac10ac8f3f [Candidate]" term=2 +testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.183Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: election won: tally=2 + testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.183Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: election won: tally=2 +testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.183Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: entering leader state: leader="Node at f153d14b-dc13-123e-9a82-a2ac10ac8f3f [Leader]" + testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.183Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: entering leader state: leader="Node at f153d14b-dc13-123e-9a82-a2ac10ac8f3f [Leader]" +testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.183Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: added peer, starting replication: peer=server-412e45c2-b294-66dc-33a9-f062c8ae3eea + testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.183Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: added peer, starting replication: peer=server-412e45c2-b294-66dc-33a9-f062c8ae3eea +testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.183Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: pipelining replication: peer="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" + testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.183Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: pipelining replication: peer="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:01.288Z [INFO] stable state for Leader reached at 2023-08-02 15:13:01.18362594 +0000 UTC m=+13.279562275 (1 nodes), 69.965836ms from start of poll, 70.345877ms from cluster start. Timeout at 2023-08-02 15:13:01.288766989 +0000 UTC m=+13.384703343, 105.141068ms after stability + testing.go:149: cluster: 2023-08-02T15:13:01.288Z [INFO] stable state for Leader reached at 2023-08-02 15:13:01.18362594 +0000 UTC m=+13.279562275 (1 nodes), 69.965836ms from start of poll, 70.345877ms from cluster start. Timeout at 2023-08-02 15:13:01.288766989 +0000 UTC m=+13.384703343, 105.141068ms after stability +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:01.294Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-08-02T15:13:01.294Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:01.399Z [INFO] stable state for Follower reached at 2023-08-02 15:13:01.294225947 +0000 UTC m=+13.390162282 (1 nodes), 1.592µs from start of poll, 180.945884ms from cluster start. Timeout at 2023-08-02 15:13:01.399333228 +0000 UTC m=+13.495269575, 105.107293ms after stability + testing.go:149: cluster: 2023-08-02T15:13:01.399Z [INFO] stable state for Follower reached at 2023-08-02 15:13:01.294225947 +0000 UTC m=+13.390162282 (1 nodes), 1.592µs from start of poll, 180.945884ms from cluster start. Timeout at 2023-08-02 15:13:01.399333228 +0000 UTC m=+13.495269575, 105.107293ms after stability + raft_test.go:1492: [INFO] Disconnecting Node at 412e45c2-b294-66dc-33a9-f062c8ae3eea [Follower] +testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.400Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to pipeline appendEntries: peer="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" error="append pipeline closed" + testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.400Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to pipeline appendEntries: peer="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" error="append pipeline closed" +testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.400Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: aborting pipeline replication: peer="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" + testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.400Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: aborting pipeline replication: peer="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" +testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.403Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to heartbeat to: peer=412e45c2-b294-66dc-33a9-f062c8ae3eea error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" + testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.403Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to heartbeat to: peer=412e45c2-b294-66dc-33a9-f062c8ae3eea error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" +testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.409Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to appendEntries to: peer="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" + testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.409Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to appendEntries to: peer="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" +testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.423Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to heartbeat to: peer=412e45c2-b294-66dc-33a9-f062c8ae3eea error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" + testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.423Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to heartbeat to: peer=412e45c2-b294-66dc-33a9-f062c8ae3eea error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" +testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.430Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to appendEntries to: peer="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" + testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.430Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to appendEntries to: peer="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" +testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.439Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to heartbeat to: peer=412e45c2-b294-66dc-33a9-f062c8ae3eea error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" + testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.439Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to heartbeat to: peer=412e45c2-b294-66dc-33a9-f062c8ae3eea error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" +testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.444Z [WARN] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to contact: server-id=server-412e45c2-b294-66dc-33a9-f062c8ae3eea time=50.11397ms + testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.444Z [WARN] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to contact: server-id=server-412e45c2-b294-66dc-33a9-f062c8ae3eea time=50.11397ms +testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.444Z [WARN] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to contact quorum of nodes, stepping down + testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.444Z [WARN] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to contact quorum of nodes, stepping down +testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.445Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: entering follower state: follower="Node at f153d14b-dc13-123e-9a82-a2ac10ac8f3f [Follower]" leader= + testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.445Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: entering follower state: follower="Node at f153d14b-dc13-123e-9a82-a2ac10ac8f3f [Follower]" leader= +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:01.445Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:13:01.445Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.451Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to appendEntries to: peer="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" + testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.451Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to appendEntries to: peer="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" +testLoggerAdapter verbose: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.510Z [WARN] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: heartbeat timeout reached, starting election: last-leader=f153d14b-dc13-123e-9a82-a2ac10ac8f3f + testing.go:149: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.510Z [WARN] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: heartbeat timeout reached, starting election: last-leader=f153d14b-dc13-123e-9a82-a2ac10ac8f3f +testLoggerAdapter verbose: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.510Z [INFO] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: entering candidate state: node="Node at 412e45c2-b294-66dc-33a9-f062c8ae3eea [Candidate]" term=3 + testing.go:149: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.510Z [INFO] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: entering candidate state: node="Node at 412e45c2-b294-66dc-33a9-f062c8ae3eea [Candidate]" term=3 +testLoggerAdapter verbose: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.510Z [ERROR] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: failed to make requestVote RPC: target="{Voter server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f f153d14b-dc13-123e-9a82-a2ac10ac8f3f}" error="failed to connect to peer: f153d14b-dc13-123e-9a82-a2ac10ac8f3f" + testing.go:149: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.510Z [ERROR] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: failed to make requestVote RPC: target="{Voter server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f f153d14b-dc13-123e-9a82-a2ac10ac8f3f}" error="failed to connect to peer: f153d14b-dc13-123e-9a82-a2ac10ac8f3f" +testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.541Z [WARN] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.541Z [WARN] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.542Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: entering candidate state: node="Node at f153d14b-dc13-123e-9a82-a2ac10ac8f3f [Candidate]" term=3 + testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.542Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: entering candidate state: node="Node at f153d14b-dc13-123e-9a82-a2ac10ac8f3f [Candidate]" term=3 +testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.542Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to make requestVote RPC: target="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" + testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.542Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to make requestVote RPC: target="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" +testLoggerAdapter verbose: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.598Z [WARN] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: Election timeout reached, restarting election + testing.go:149: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.598Z [WARN] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: Election timeout reached, restarting election +testLoggerAdapter verbose: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.599Z [INFO] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: entering candidate state: node="Node at 412e45c2-b294-66dc-33a9-f062c8ae3eea [Candidate]" term=4 + testing.go:149: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.599Z [INFO] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: entering candidate state: node="Node at 412e45c2-b294-66dc-33a9-f062c8ae3eea [Candidate]" term=4 +testLoggerAdapter verbose: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.599Z [ERROR] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: failed to make requestVote RPC: target="{Voter server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f f153d14b-dc13-123e-9a82-a2ac10ac8f3f}" error="failed to connect to peer: f153d14b-dc13-123e-9a82-a2ac10ac8f3f" + testing.go:149: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.599Z [ERROR] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: failed to make requestVote RPC: target="{Voter server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f f153d14b-dc13-123e-9a82-a2ac10ac8f3f}" error="failed to connect to peer: f153d14b-dc13-123e-9a82-a2ac10ac8f3f" +testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.612Z [WARN] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: Election timeout reached, restarting election + testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.612Z [WARN] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: Election timeout reached, restarting election +testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.612Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: entering candidate state: node="Node at f153d14b-dc13-123e-9a82-a2ac10ac8f3f [Candidate]" term=4 + testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.612Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: entering candidate state: node="Node at f153d14b-dc13-123e-9a82-a2ac10ac8f3f [Candidate]" term=4 +testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.613Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to make requestVote RPC: target="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" + testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.613Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to make requestVote RPC: target="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:01.647Z [INFO] stable state for Leader reached at 2023-08-02 15:13:01.542220035 +0000 UTC m=+13.638156380 (0 nodes), 97.065321ms from start of poll, 428.939982ms from cluster start. Timeout at 2023-08-02 15:13:01.647336274 +0000 UTC m=+13.743272616, 105.116236ms after stability + testing.go:149: cluster: 2023-08-02T15:13:01.647Z [INFO] stable state for Leader reached at 2023-08-02 15:13:01.542220035 +0000 UTC m=+13.638156380 (0 nodes), 97.065321ms from start of poll, 428.939982ms from cluster start. Timeout at 2023-08-02 15:13:01.647336274 +0000 UTC m=+13.743272616, 105.116236ms after stability +testLoggerAdapter verbose: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.653Z [WARN] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: Election timeout reached, restarting election + testing.go:149: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.653Z [WARN] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: Election timeout reached, restarting election +testLoggerAdapter verbose: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.653Z [INFO] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: entering candidate state: node="Node at 412e45c2-b294-66dc-33a9-f062c8ae3eea [Candidate]" term=5 + testing.go:149: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.653Z [INFO] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: entering candidate state: node="Node at 412e45c2-b294-66dc-33a9-f062c8ae3eea [Candidate]" term=5 +testLoggerAdapter verbose: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.653Z [ERROR] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: failed to make requestVote RPC: target="{Voter server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f f153d14b-dc13-123e-9a82-a2ac10ac8f3f}" error="failed to connect to peer: f153d14b-dc13-123e-9a82-a2ac10ac8f3f" + testing.go:149: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.653Z [ERROR] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: failed to make requestVote RPC: target="{Voter server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f f153d14b-dc13-123e-9a82-a2ac10ac8f3f}" error="failed to connect to peer: f153d14b-dc13-123e-9a82-a2ac10ac8f3f" +testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.681Z [WARN] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: Election timeout reached, restarting election + testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.681Z [WARN] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: Election timeout reached, restarting election +testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.681Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: entering candidate state: node="Node at f153d14b-dc13-123e-9a82-a2ac10ac8f3f [Candidate]" term=5 + testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.681Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: entering candidate state: node="Node at f153d14b-dc13-123e-9a82-a2ac10ac8f3f [Candidate]" term=5 +testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.682Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to make requestVote RPC: target="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" + testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.682Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to make requestVote RPC: target="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" +testLoggerAdapter verbose: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.720Z [WARN] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: Election timeout reached, restarting election + testing.go:149: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.720Z [WARN] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: Election timeout reached, restarting election +testLoggerAdapter verbose: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.720Z [INFO] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: entering candidate state: node="Node at 412e45c2-b294-66dc-33a9-f062c8ae3eea [Candidate]" term=6 + testing.go:149: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.720Z [INFO] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: entering candidate state: node="Node at 412e45c2-b294-66dc-33a9-f062c8ae3eea [Candidate]" term=6 +testLoggerAdapter verbose: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.720Z [ERROR] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: failed to make requestVote RPC: target="{Voter server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f f153d14b-dc13-123e-9a82-a2ac10ac8f3f}" error="failed to connect to peer: f153d14b-dc13-123e-9a82-a2ac10ac8f3f" + testing.go:149: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.720Z [ERROR] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: failed to make requestVote RPC: target="{Voter server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f f153d14b-dc13-123e-9a82-a2ac10ac8f3f}" error="failed to connect to peer: f153d14b-dc13-123e-9a82-a2ac10ac8f3f" +testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.738Z [WARN] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: Election timeout reached, restarting election + testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.738Z [WARN] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: Election timeout reached, restarting election +testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.738Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: entering candidate state: node="Node at f153d14b-dc13-123e-9a82-a2ac10ac8f3f [Candidate]" term=6 + testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.738Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: entering candidate state: node="Node at f153d14b-dc13-123e-9a82-a2ac10ac8f3f [Candidate]" term=6 +testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.738Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to make requestVote RPC: target="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" + testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.738Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to make requestVote RPC: target="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" +--- PASS: TestRaft_LeaderLeaseExpire (0.64s) === RUN TestRaft_Barrier -testLoggerAdapter verbose: server-1835165e-c69d-1704-6aad-e51773d1c881: 2024-09-03T21:34:32.773Z [INFO] server-1835165e-c69d-1704-6aad-e51773d1c881: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1835165e-c69d-1704-6aad-e51773d1c881 Address:1835165e-c69d-1704-6aad-e51773d1c881} {Suffrage:Voter ID:server-847c7117-ad27-b387-63ff-cbac045153a3 Address:847c7117-ad27-b387-63ff-cbac045153a3} {Suffrage:Voter ID:server-e0d9932c-be3c-2e37-e70a-b70fda45206e Address:e0d9932c-be3c-2e37-e70a-b70fda45206e}]" - testing.go:149: server-1835165e-c69d-1704-6aad-e51773d1c881: 2024-09-03T21:34:32.773Z [INFO] server-1835165e-c69d-1704-6aad-e51773d1c881: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1835165e-c69d-1704-6aad-e51773d1c881 Address:1835165e-c69d-1704-6aad-e51773d1c881} {Suffrage:Voter ID:server-847c7117-ad27-b387-63ff-cbac045153a3 Address:847c7117-ad27-b387-63ff-cbac045153a3} {Suffrage:Voter ID:server-e0d9932c-be3c-2e37-e70a-b70fda45206e Address:e0d9932c-be3c-2e37-e70a-b70fda45206e}]" -testLoggerAdapter verbose: server-847c7117-ad27-b387-63ff-cbac045153a3: 2024-09-03T21:34:32.773Z [INFO] server-847c7117-ad27-b387-63ff-cbac045153a3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1835165e-c69d-1704-6aad-e51773d1c881 Address:1835165e-c69d-1704-6aad-e51773d1c881} {Suffrage:Voter ID:server-847c7117-ad27-b387-63ff-cbac045153a3 Address:847c7117-ad27-b387-63ff-cbac045153a3} {Suffrage:Voter ID:server-e0d9932c-be3c-2e37-e70a-b70fda45206e Address:e0d9932c-be3c-2e37-e70a-b70fda45206e}]" - testing.go:149: server-847c7117-ad27-b387-63ff-cbac045153a3: 2024-09-03T21:34:32.773Z [INFO] server-847c7117-ad27-b387-63ff-cbac045153a3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1835165e-c69d-1704-6aad-e51773d1c881 Address:1835165e-c69d-1704-6aad-e51773d1c881} {Suffrage:Voter ID:server-847c7117-ad27-b387-63ff-cbac045153a3 Address:847c7117-ad27-b387-63ff-cbac045153a3} {Suffrage:Voter ID:server-e0d9932c-be3c-2e37-e70a-b70fda45206e Address:e0d9932c-be3c-2e37-e70a-b70fda45206e}]" -testLoggerAdapter verbose: server-e0d9932c-be3c-2e37-e70a-b70fda45206e: 2024-09-03T21:34:32.774Z [INFO] server-e0d9932c-be3c-2e37-e70a-b70fda45206e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1835165e-c69d-1704-6aad-e51773d1c881 Address:1835165e-c69d-1704-6aad-e51773d1c881} {Suffrage:Voter ID:server-847c7117-ad27-b387-63ff-cbac045153a3 Address:847c7117-ad27-b387-63ff-cbac045153a3} {Suffrage:Voter ID:server-e0d9932c-be3c-2e37-e70a-b70fda45206e Address:e0d9932c-be3c-2e37-e70a-b70fda45206e}]" - testing.go:149: server-e0d9932c-be3c-2e37-e70a-b70fda45206e: 2024-09-03T21:34:32.774Z [INFO] server-e0d9932c-be3c-2e37-e70a-b70fda45206e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1835165e-c69d-1704-6aad-e51773d1c881 Address:1835165e-c69d-1704-6aad-e51773d1c881} {Suffrage:Voter ID:server-847c7117-ad27-b387-63ff-cbac045153a3 Address:847c7117-ad27-b387-63ff-cbac045153a3} {Suffrage:Voter ID:server-e0d9932c-be3c-2e37-e70a-b70fda45206e Address:e0d9932c-be3c-2e37-e70a-b70fda45206e}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:32.774Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:32.774Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-1835165e-c69d-1704-6aad-e51773d1c881: 2024-09-03T21:34:32.774Z [INFO] server-1835165e-c69d-1704-6aad-e51773d1c881: entering follower state: follower="Node at 1835165e-c69d-1704-6aad-e51773d1c881 [Follower]" leader= - testing.go:149: server-1835165e-c69d-1704-6aad-e51773d1c881: 2024-09-03T21:34:32.774Z [INFO] server-1835165e-c69d-1704-6aad-e51773d1c881: entering follower state: follower="Node at 1835165e-c69d-1704-6aad-e51773d1c881 [Follower]" leader= -testLoggerAdapter verbose: server-847c7117-ad27-b387-63ff-cbac045153a3: 2024-09-03T21:34:32.774Z [INFO] server-847c7117-ad27-b387-63ff-cbac045153a3: entering follower state: follower="Node at 847c7117-ad27-b387-63ff-cbac045153a3 [Follower]" leader= - testing.go:149: server-847c7117-ad27-b387-63ff-cbac045153a3: 2024-09-03T21:34:32.774Z [INFO] server-847c7117-ad27-b387-63ff-cbac045153a3: entering follower state: follower="Node at 847c7117-ad27-b387-63ff-cbac045153a3 [Follower]" leader= -testLoggerAdapter verbose: server-e0d9932c-be3c-2e37-e70a-b70fda45206e: 2024-09-03T21:34:32.774Z [INFO] server-e0d9932c-be3c-2e37-e70a-b70fda45206e: entering follower state: follower="Node at e0d9932c-be3c-2e37-e70a-b70fda45206e [Follower]" leader= - testing.go:149: server-e0d9932c-be3c-2e37-e70a-b70fda45206e: 2024-09-03T21:34:32.774Z [INFO] server-e0d9932c-be3c-2e37-e70a-b70fda45206e: entering follower state: follower="Node at e0d9932c-be3c-2e37-e70a-b70fda45206e [Follower]" leader= -testLoggerAdapter verbose: server-1835165e-c69d-1704-6aad-e51773d1c881: 2024-09-03T21:34:32.831Z [WARN] server-1835165e-c69d-1704-6aad-e51773d1c881: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-1835165e-c69d-1704-6aad-e51773d1c881: 2024-09-03T21:34:32.831Z [WARN] server-1835165e-c69d-1704-6aad-e51773d1c881: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-1835165e-c69d-1704-6aad-e51773d1c881: 2024-09-03T21:34:32.831Z [INFO] server-1835165e-c69d-1704-6aad-e51773d1c881: entering candidate state: node="Node at 1835165e-c69d-1704-6aad-e51773d1c881 [Candidate]" term=2 - testing.go:149: server-1835165e-c69d-1704-6aad-e51773d1c881: 2024-09-03T21:34:32.831Z [INFO] server-1835165e-c69d-1704-6aad-e51773d1c881: entering candidate state: node="Node at 1835165e-c69d-1704-6aad-e51773d1c881 [Candidate]" term=2 -testLoggerAdapter verbose: server-1835165e-c69d-1704-6aad-e51773d1c881: 2024-09-03T21:34:32.831Z [INFO] server-1835165e-c69d-1704-6aad-e51773d1c881: election won: tally=2 - testing.go:149: server-1835165e-c69d-1704-6aad-e51773d1c881: 2024-09-03T21:34:32.831Z [INFO] server-1835165e-c69d-1704-6aad-e51773d1c881: election won: tally=2 -testLoggerAdapter verbose: server-1835165e-c69d-1704-6aad-e51773d1c881: 2024-09-03T21:34:32.831Z [INFO] server-1835165e-c69d-1704-6aad-e51773d1c881: entering leader state: leader="Node at 1835165e-c69d-1704-6aad-e51773d1c881 [Leader]" - testing.go:149: server-1835165e-c69d-1704-6aad-e51773d1c881: 2024-09-03T21:34:32.831Z [INFO] server-1835165e-c69d-1704-6aad-e51773d1c881: entering leader state: leader="Node at 1835165e-c69d-1704-6aad-e51773d1c881 [Leader]" -testLoggerAdapter verbose: server-1835165e-c69d-1704-6aad-e51773d1c881: 2024-09-03T21:34:32.831Z [INFO] server-1835165e-c69d-1704-6aad-e51773d1c881: added peer, starting replication: peer=server-847c7117-ad27-b387-63ff-cbac045153a3 - testing.go:149: server-1835165e-c69d-1704-6aad-e51773d1c881: 2024-09-03T21:34:32.831Z [INFO] server-1835165e-c69d-1704-6aad-e51773d1c881: added peer, starting replication: peer=server-847c7117-ad27-b387-63ff-cbac045153a3 -testLoggerAdapter verbose: server-1835165e-c69d-1704-6aad-e51773d1c881: 2024-09-03T21:34:32.831Z [INFO] server-1835165e-c69d-1704-6aad-e51773d1c881: added peer, starting replication: peer=server-e0d9932c-be3c-2e37-e70a-b70fda45206e - testing.go:149: server-1835165e-c69d-1704-6aad-e51773d1c881: 2024-09-03T21:34:32.831Z [INFO] server-1835165e-c69d-1704-6aad-e51773d1c881: added peer, starting replication: peer=server-e0d9932c-be3c-2e37-e70a-b70fda45206e -testLoggerAdapter verbose: server-1835165e-c69d-1704-6aad-e51773d1c881: 2024-09-03T21:34:32.831Z [INFO] server-1835165e-c69d-1704-6aad-e51773d1c881: pipelining replication: peer="{Voter server-e0d9932c-be3c-2e37-e70a-b70fda45206e e0d9932c-be3c-2e37-e70a-b70fda45206e}" - testing.go:149: server-1835165e-c69d-1704-6aad-e51773d1c881: 2024-09-03T21:34:32.831Z [INFO] server-1835165e-c69d-1704-6aad-e51773d1c881: pipelining replication: peer="{Voter server-e0d9932c-be3c-2e37-e70a-b70fda45206e e0d9932c-be3c-2e37-e70a-b70fda45206e}" -testLoggerAdapter verbose: server-1835165e-c69d-1704-6aad-e51773d1c881: 2024-09-03T21:34:32.832Z [INFO] server-1835165e-c69d-1704-6aad-e51773d1c881: pipelining replication: peer="{Voter server-847c7117-ad27-b387-63ff-cbac045153a3 847c7117-ad27-b387-63ff-cbac045153a3}" - testing.go:149: server-1835165e-c69d-1704-6aad-e51773d1c881: 2024-09-03T21:34:32.832Z [INFO] server-1835165e-c69d-1704-6aad-e51773d1c881: pipelining replication: peer="{Voter server-847c7117-ad27-b387-63ff-cbac045153a3 847c7117-ad27-b387-63ff-cbac045153a3}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:32.937Z [INFO] stable state for Leader reached at 2024-09-03 21:34:32.832080588 +0000 UTC m=+13.878737741 (1 nodes), 57.891091ms from start of poll, 58.415732ms from cluster start. Timeout at 2024-09-03 21:34:32.937125353 +0000 UTC m=+13.983782481, 105.04474ms after stability - testing.go:149: cluster: 2024-09-03T21:34:32.937Z [INFO] stable state for Leader reached at 2024-09-03 21:34:32.832080588 +0000 UTC m=+13.878737741 (1 nodes), 57.891091ms from start of poll, 58.415732ms from cluster start. Timeout at 2024-09-03 21:34:32.937125353 +0000 UTC m=+13.983782481, 105.04474ms after stability -testLoggerAdapter verbose: server-1835165e-c69d-1704-6aad-e51773d1c881: 2024-09-03T21:34:32.949Z [INFO] server-1835165e-c69d-1704-6aad-e51773d1c881: aborting pipeline replication: peer="{Voter server-e0d9932c-be3c-2e37-e70a-b70fda45206e e0d9932c-be3c-2e37-e70a-b70fda45206e}" - testing.go:149: server-1835165e-c69d-1704-6aad-e51773d1c881: 2024-09-03T21:34:32.949Z [INFO] server-1835165e-c69d-1704-6aad-e51773d1c881: aborting pipeline replication: peer="{Voter server-e0d9932c-be3c-2e37-e70a-b70fda45206e e0d9932c-be3c-2e37-e70a-b70fda45206e}" -testLoggerAdapter verbose: server-1835165e-c69d-1704-6aad-e51773d1c881: 2024-09-03T21:34:32.951Z [INFO] server-1835165e-c69d-1704-6aad-e51773d1c881: aborting pipeline replication: peer="{Voter server-847c7117-ad27-b387-63ff-cbac045153a3 847c7117-ad27-b387-63ff-cbac045153a3}" - testing.go:149: server-1835165e-c69d-1704-6aad-e51773d1c881: 2024-09-03T21:34:32.951Z [INFO] server-1835165e-c69d-1704-6aad-e51773d1c881: aborting pipeline replication: peer="{Voter server-847c7117-ad27-b387-63ff-cbac045153a3 847c7117-ad27-b387-63ff-cbac045153a3}" +testLoggerAdapter verbose: server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5: 2023-08-02T15:13:01.754Z [INFO] server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5 Address:fcaa246d-0e1b-6eba-1533-44e9d7ec79f5} {Suffrage:Voter ID:server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8 Address:7f9f7a1a-cb9b-484f-e407-7713638e3ab8} {Suffrage:Voter ID:server-948f5509-fc1f-3029-1799-1b3b8018197f Address:948f5509-fc1f-3029-1799-1b3b8018197f}]" + testing.go:149: server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5: 2023-08-02T15:13:01.754Z [INFO] server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5 Address:fcaa246d-0e1b-6eba-1533-44e9d7ec79f5} {Suffrage:Voter ID:server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8 Address:7f9f7a1a-cb9b-484f-e407-7713638e3ab8} {Suffrage:Voter ID:server-948f5509-fc1f-3029-1799-1b3b8018197f Address:948f5509-fc1f-3029-1799-1b3b8018197f}]" +testLoggerAdapter verbose: server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8: 2023-08-02T15:13:01.754Z [INFO] server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5 Address:fcaa246d-0e1b-6eba-1533-44e9d7ec79f5} {Suffrage:Voter ID:server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8 Address:7f9f7a1a-cb9b-484f-e407-7713638e3ab8} {Suffrage:Voter ID:server-948f5509-fc1f-3029-1799-1b3b8018197f Address:948f5509-fc1f-3029-1799-1b3b8018197f}]" + testing.go:149: server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8: 2023-08-02T15:13:01.754Z [INFO] server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5 Address:fcaa246d-0e1b-6eba-1533-44e9d7ec79f5} {Suffrage:Voter ID:server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8 Address:7f9f7a1a-cb9b-484f-e407-7713638e3ab8} {Suffrage:Voter ID:server-948f5509-fc1f-3029-1799-1b3b8018197f Address:948f5509-fc1f-3029-1799-1b3b8018197f}]" +testLoggerAdapter verbose: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.754Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5 Address:fcaa246d-0e1b-6eba-1533-44e9d7ec79f5} {Suffrage:Voter ID:server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8 Address:7f9f7a1a-cb9b-484f-e407-7713638e3ab8} {Suffrage:Voter ID:server-948f5509-fc1f-3029-1799-1b3b8018197f Address:948f5509-fc1f-3029-1799-1b3b8018197f}]" + testing.go:149: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.754Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5 Address:fcaa246d-0e1b-6eba-1533-44e9d7ec79f5} {Suffrage:Voter ID:server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8 Address:7f9f7a1a-cb9b-484f-e407-7713638e3ab8} {Suffrage:Voter ID:server-948f5509-fc1f-3029-1799-1b3b8018197f Address:948f5509-fc1f-3029-1799-1b3b8018197f}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:01.754Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8: 2023-08-02T15:13:01.754Z [INFO] server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8: entering follower state: follower="Node at 7f9f7a1a-cb9b-484f-e407-7713638e3ab8 [Follower]" leader= + testing.go:149: server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8: 2023-08-02T15:13:01.754Z [INFO] server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8: entering follower state: follower="Node at 7f9f7a1a-cb9b-484f-e407-7713638e3ab8 [Follower]" leader= +testLoggerAdapter verbose: server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5: 2023-08-02T15:13:01.754Z [INFO] server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5: entering follower state: follower="Node at fcaa246d-0e1b-6eba-1533-44e9d7ec79f5 [Follower]" leader= + testing.go:149: server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5: 2023-08-02T15:13:01.754Z [INFO] server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5: entering follower state: follower="Node at fcaa246d-0e1b-6eba-1533-44e9d7ec79f5 [Follower]" leader= +testLoggerAdapter verbose: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.754Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: entering follower state: follower="Node at 948f5509-fc1f-3029-1799-1b3b8018197f [Follower]" leader= + testing.go:149: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.754Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: entering follower state: follower="Node at 948f5509-fc1f-3029-1799-1b3b8018197f [Follower]" leader= + testing.go:149: cluster: 2023-08-02T15:13:01.754Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.810Z [WARN] server-948f5509-fc1f-3029-1799-1b3b8018197f: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.810Z [WARN] server-948f5509-fc1f-3029-1799-1b3b8018197f: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.810Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: entering candidate state: node="Node at 948f5509-fc1f-3029-1799-1b3b8018197f [Candidate]" term=2 + testing.go:149: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.810Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: entering candidate state: node="Node at 948f5509-fc1f-3029-1799-1b3b8018197f [Candidate]" term=2 +testLoggerAdapter verbose: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.810Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: election won: tally=2 + testing.go:149: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.810Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: election won: tally=2 +testLoggerAdapter verbose: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.810Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: entering leader state: leader="Node at 948f5509-fc1f-3029-1799-1b3b8018197f [Leader]" + testing.go:149: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.810Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: entering leader state: leader="Node at 948f5509-fc1f-3029-1799-1b3b8018197f [Leader]" +testLoggerAdapter verbose: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.810Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: added peer, starting replication: peer=server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5 + testing.go:149: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.810Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: added peer, starting replication: peer=server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5 +testLoggerAdapter verbose: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.810Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: added peer, starting replication: peer=server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8 + testing.go:149: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.810Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: added peer, starting replication: peer=server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8 +testLoggerAdapter verbose: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.810Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: pipelining replication: peer="{Voter server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8 7f9f7a1a-cb9b-484f-e407-7713638e3ab8}" + testing.go:149: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.810Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: pipelining replication: peer="{Voter server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8 7f9f7a1a-cb9b-484f-e407-7713638e3ab8}" +testLoggerAdapter verbose: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.810Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: pipelining replication: peer="{Voter server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5 fcaa246d-0e1b-6eba-1533-44e9d7ec79f5}" + testing.go:149: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.810Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: pipelining replication: peer="{Voter server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5 fcaa246d-0e1b-6eba-1533-44e9d7ec79f5}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:01.915Z [INFO] stable state for Leader reached at 2023-08-02 15:13:01.810605981 +0000 UTC m=+13.906542315 (1 nodes), 55.672251ms from start of poll, 56.650316ms from cluster start. Timeout at 2023-08-02 15:13:01.915819577 +0000 UTC m=+14.011755914, 105.213599ms after stability + testing.go:149: cluster: 2023-08-02T15:13:01.915Z [INFO] stable state for Leader reached at 2023-08-02 15:13:01.810605981 +0000 UTC m=+13.906542315 (1 nodes), 55.672251ms from start of poll, 56.650316ms from cluster start. Timeout at 2023-08-02 15:13:01.915819577 +0000 UTC m=+14.011755914, 105.213599ms after stability +testLoggerAdapter verbose: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.929Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: aborting pipeline replication: peer="{Voter server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8 7f9f7a1a-cb9b-484f-e407-7713638e3ab8}" + testing.go:149: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.929Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: aborting pipeline replication: peer="{Voter server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8 7f9f7a1a-cb9b-484f-e407-7713638e3ab8}" +testLoggerAdapter verbose: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.929Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: aborting pipeline replication: peer="{Voter server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5 fcaa246d-0e1b-6eba-1533-44e9d7ec79f5}" + testing.go:149: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.929Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: aborting pipeline replication: peer="{Voter server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5 fcaa246d-0e1b-6eba-1533-44e9d7ec79f5}" --- PASS: TestRaft_Barrier (0.18s) === RUN TestRaft_VerifyLeader -testLoggerAdapter verbose: server-0d3f372d-63f7-945c-b01c-a2a1fdf9b5e5: 2024-09-03T21:34:32.952Z [INFO] server-0d3f372d-63f7-945c-b01c-a2a1fdf9b5e5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d3f372d-63f7-945c-b01c-a2a1fdf9b5e5 Address:0d3f372d-63f7-945c-b01c-a2a1fdf9b5e5} {Suffrage:Voter ID:server-007c3d35-086a-f373-539b-400d34860b90 Address:007c3d35-086a-f373-539b-400d34860b90} {Suffrage:Voter ID:server-6992b02d-919e-866e-9ef4-74e77aee2b6d Address:6992b02d-919e-866e-9ef4-74e77aee2b6d}]" - testing.go:149: server-0d3f372d-63f7-945c-b01c-a2a1fdf9b5e5: 2024-09-03T21:34:32.952Z [INFO] server-0d3f372d-63f7-945c-b01c-a2a1fdf9b5e5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d3f372d-63f7-945c-b01c-a2a1fdf9b5e5 Address:0d3f372d-63f7-945c-b01c-a2a1fdf9b5e5} {Suffrage:Voter ID:server-007c3d35-086a-f373-539b-400d34860b90 Address:007c3d35-086a-f373-539b-400d34860b90} {Suffrage:Voter ID:server-6992b02d-919e-866e-9ef4-74e77aee2b6d Address:6992b02d-919e-866e-9ef4-74e77aee2b6d}]" -testLoggerAdapter verbose: server-007c3d35-086a-f373-539b-400d34860b90: 2024-09-03T21:34:32.952Z [INFO] server-007c3d35-086a-f373-539b-400d34860b90: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d3f372d-63f7-945c-b01c-a2a1fdf9b5e5 Address:0d3f372d-63f7-945c-b01c-a2a1fdf9b5e5} {Suffrage:Voter ID:server-007c3d35-086a-f373-539b-400d34860b90 Address:007c3d35-086a-f373-539b-400d34860b90} {Suffrage:Voter ID:server-6992b02d-919e-866e-9ef4-74e77aee2b6d Address:6992b02d-919e-866e-9ef4-74e77aee2b6d}]" - testing.go:149: server-007c3d35-086a-f373-539b-400d34860b90: 2024-09-03T21:34:32.952Z [INFO] server-007c3d35-086a-f373-539b-400d34860b90: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d3f372d-63f7-945c-b01c-a2a1fdf9b5e5 Address:0d3f372d-63f7-945c-b01c-a2a1fdf9b5e5} {Suffrage:Voter ID:server-007c3d35-086a-f373-539b-400d34860b90 Address:007c3d35-086a-f373-539b-400d34860b90} {Suffrage:Voter ID:server-6992b02d-919e-866e-9ef4-74e77aee2b6d Address:6992b02d-919e-866e-9ef4-74e77aee2b6d}]" -testLoggerAdapter verbose: server-6992b02d-919e-866e-9ef4-74e77aee2b6d: 2024-09-03T21:34:32.953Z [INFO] server-6992b02d-919e-866e-9ef4-74e77aee2b6d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d3f372d-63f7-945c-b01c-a2a1fdf9b5e5 Address:0d3f372d-63f7-945c-b01c-a2a1fdf9b5e5} {Suffrage:Voter ID:server-007c3d35-086a-f373-539b-400d34860b90 Address:007c3d35-086a-f373-539b-400d34860b90} {Suffrage:Voter ID:server-6992b02d-919e-866e-9ef4-74e77aee2b6d Address:6992b02d-919e-866e-9ef4-74e77aee2b6d}]" - testing.go:149: server-6992b02d-919e-866e-9ef4-74e77aee2b6d: 2024-09-03T21:34:32.953Z [INFO] server-6992b02d-919e-866e-9ef4-74e77aee2b6d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d3f372d-63f7-945c-b01c-a2a1fdf9b5e5 Address:0d3f372d-63f7-945c-b01c-a2a1fdf9b5e5} {Suffrage:Voter ID:server-007c3d35-086a-f373-539b-400d34860b90 Address:007c3d35-086a-f373-539b-400d34860b90} {Suffrage:Voter ID:server-6992b02d-919e-866e-9ef4-74e77aee2b6d Address:6992b02d-919e-866e-9ef4-74e77aee2b6d}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:32.953Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:32.953Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-0d3f372d-63f7-945c-b01c-a2a1fdf9b5e5: 2024-09-03T21:34:32.953Z [INFO] server-0d3f372d-63f7-945c-b01c-a2a1fdf9b5e5: entering follower state: follower="Node at 0d3f372d-63f7-945c-b01c-a2a1fdf9b5e5 [Follower]" leader= - testing.go:149: server-0d3f372d-63f7-945c-b01c-a2a1fdf9b5e5: 2024-09-03T21:34:32.953Z [INFO] server-0d3f372d-63f7-945c-b01c-a2a1fdf9b5e5: entering follower state: follower="Node at 0d3f372d-63f7-945c-b01c-a2a1fdf9b5e5 [Follower]" leader= -testLoggerAdapter verbose: server-007c3d35-086a-f373-539b-400d34860b90: 2024-09-03T21:34:32.953Z [INFO] server-007c3d35-086a-f373-539b-400d34860b90: entering follower state: follower="Node at 007c3d35-086a-f373-539b-400d34860b90 [Follower]" leader= - testing.go:149: server-007c3d35-086a-f373-539b-400d34860b90: 2024-09-03T21:34:32.953Z [INFO] server-007c3d35-086a-f373-539b-400d34860b90: entering follower state: follower="Node at 007c3d35-086a-f373-539b-400d34860b90 [Follower]" leader= -testLoggerAdapter verbose: server-6992b02d-919e-866e-9ef4-74e77aee2b6d: 2024-09-03T21:34:32.953Z [INFO] server-6992b02d-919e-866e-9ef4-74e77aee2b6d: entering follower state: follower="Node at 6992b02d-919e-866e-9ef4-74e77aee2b6d [Follower]" leader= - testing.go:149: server-6992b02d-919e-866e-9ef4-74e77aee2b6d: 2024-09-03T21:34:32.953Z [INFO] server-6992b02d-919e-866e-9ef4-74e77aee2b6d: entering follower state: follower="Node at 6992b02d-919e-866e-9ef4-74e77aee2b6d [Follower]" leader= -testLoggerAdapter verbose: server-6992b02d-919e-866e-9ef4-74e77aee2b6d: 2024-09-03T21:34:33.006Z [WARN] server-6992b02d-919e-866e-9ef4-74e77aee2b6d: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-6992b02d-919e-866e-9ef4-74e77aee2b6d: 2024-09-03T21:34:33.006Z [WARN] server-6992b02d-919e-866e-9ef4-74e77aee2b6d: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-6992b02d-919e-866e-9ef4-74e77aee2b6d: 2024-09-03T21:34:33.007Z [INFO] server-6992b02d-919e-866e-9ef4-74e77aee2b6d: entering candidate state: node="Node at 6992b02d-919e-866e-9ef4-74e77aee2b6d [Candidate]" term=2 - testing.go:149: server-6992b02d-919e-866e-9ef4-74e77aee2b6d: 2024-09-03T21:34:33.007Z [INFO] server-6992b02d-919e-866e-9ef4-74e77aee2b6d: entering candidate state: node="Node at 6992b02d-919e-866e-9ef4-74e77aee2b6d [Candidate]" term=2 -testLoggerAdapter verbose: server-6992b02d-919e-866e-9ef4-74e77aee2b6d: 2024-09-03T21:34:33.007Z [INFO] server-6992b02d-919e-866e-9ef4-74e77aee2b6d: election won: tally=2 - testing.go:149: server-6992b02d-919e-866e-9ef4-74e77aee2b6d: 2024-09-03T21:34:33.007Z [INFO] server-6992b02d-919e-866e-9ef4-74e77aee2b6d: election won: tally=2 -testLoggerAdapter verbose: server-6992b02d-919e-866e-9ef4-74e77aee2b6d: 2024-09-03T21:34:33.007Z [INFO] server-6992b02d-919e-866e-9ef4-74e77aee2b6d: entering leader state: leader="Node at 6992b02d-919e-866e-9ef4-74e77aee2b6d [Leader]" - testing.go:149: server-6992b02d-919e-866e-9ef4-74e77aee2b6d: 2024-09-03T21:34:33.007Z [INFO] server-6992b02d-919e-866e-9ef4-74e77aee2b6d: entering leader state: leader="Node at 6992b02d-919e-866e-9ef4-74e77aee2b6d [Leader]" -testLoggerAdapter verbose: server-6992b02d-919e-866e-9ef4-74e77aee2b6d: 2024-09-03T21:34:33.007Z [INFO] server-6992b02d-919e-866e-9ef4-74e77aee2b6d: added peer, starting replication: peer=server-0d3f372d-63f7-945c-b01c-a2a1fdf9b5e5 - testing.go:149: server-6992b02d-919e-866e-9ef4-74e77aee2b6d: 2024-09-03T21:34:33.007Z [INFO] server-6992b02d-919e-866e-9ef4-74e77aee2b6d: added peer, starting replication: peer=server-0d3f372d-63f7-945c-b01c-a2a1fdf9b5e5 -testLoggerAdapter verbose: server-6992b02d-919e-866e-9ef4-74e77aee2b6d: 2024-09-03T21:34:33.007Z [INFO] server-6992b02d-919e-866e-9ef4-74e77aee2b6d: added peer, starting replication: peer=server-007c3d35-086a-f373-539b-400d34860b90 - testing.go:149: server-6992b02d-919e-866e-9ef4-74e77aee2b6d: 2024-09-03T21:34:33.007Z [INFO] server-6992b02d-919e-866e-9ef4-74e77aee2b6d: added peer, starting replication: peer=server-007c3d35-086a-f373-539b-400d34860b90 -testLoggerAdapter verbose: server-6992b02d-919e-866e-9ef4-74e77aee2b6d: 2024-09-03T21:34:33.007Z [INFO] server-6992b02d-919e-866e-9ef4-74e77aee2b6d: pipelining replication: peer="{Voter server-007c3d35-086a-f373-539b-400d34860b90 007c3d35-086a-f373-539b-400d34860b90}" - testing.go:149: server-6992b02d-919e-866e-9ef4-74e77aee2b6d: 2024-09-03T21:34:33.007Z [INFO] server-6992b02d-919e-866e-9ef4-74e77aee2b6d: pipelining replication: peer="{Voter server-007c3d35-086a-f373-539b-400d34860b90 007c3d35-086a-f373-539b-400d34860b90}" -testLoggerAdapter verbose: server-6992b02d-919e-866e-9ef4-74e77aee2b6d: 2024-09-03T21:34:33.007Z [INFO] server-6992b02d-919e-866e-9ef4-74e77aee2b6d: pipelining replication: peer="{Voter server-0d3f372d-63f7-945c-b01c-a2a1fdf9b5e5 0d3f372d-63f7-945c-b01c-a2a1fdf9b5e5}" - testing.go:149: server-6992b02d-919e-866e-9ef4-74e77aee2b6d: 2024-09-03T21:34:33.007Z [INFO] server-6992b02d-919e-866e-9ef4-74e77aee2b6d: pipelining replication: peer="{Voter server-0d3f372d-63f7-945c-b01c-a2a1fdf9b5e5 0d3f372d-63f7-945c-b01c-a2a1fdf9b5e5}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:33.112Z [INFO] stable state for Leader reached at 2024-09-03 21:34:33.007461274 +0000 UTC m=+14.054118411 (1 nodes), 54.313917ms from start of poll, 54.838515ms from cluster start. Timeout at 2024-09-03 21:34:33.112513455 +0000 UTC m=+14.159170585, 105.052174ms after stability - testing.go:149: cluster: 2024-09-03T21:34:33.112Z [INFO] stable state for Leader reached at 2024-09-03 21:34:33.007461274 +0000 UTC m=+14.054118411 (1 nodes), 54.313917ms from start of poll, 54.838515ms from cluster start. Timeout at 2024-09-03 21:34:33.112513455 +0000 UTC m=+14.159170585, 105.052174ms after stability -testLoggerAdapter verbose: server-6992b02d-919e-866e-9ef4-74e77aee2b6d: 2024-09-03T21:34:33.112Z [INFO] server-6992b02d-919e-866e-9ef4-74e77aee2b6d: aborting pipeline replication: peer="{Voter server-007c3d35-086a-f373-539b-400d34860b90 007c3d35-086a-f373-539b-400d34860b90}" - testing.go:149: server-6992b02d-919e-866e-9ef4-74e77aee2b6d: 2024-09-03T21:34:33.112Z [INFO] server-6992b02d-919e-866e-9ef4-74e77aee2b6d: aborting pipeline replication: peer="{Voter server-007c3d35-086a-f373-539b-400d34860b90 007c3d35-086a-f373-539b-400d34860b90}" -testLoggerAdapter verbose: server-6992b02d-919e-866e-9ef4-74e77aee2b6d: 2024-09-03T21:34:33.112Z [INFO] server-6992b02d-919e-866e-9ef4-74e77aee2b6d: aborting pipeline replication: peer="{Voter server-0d3f372d-63f7-945c-b01c-a2a1fdf9b5e5 0d3f372d-63f7-945c-b01c-a2a1fdf9b5e5}" - testing.go:149: server-6992b02d-919e-866e-9ef4-74e77aee2b6d: 2024-09-03T21:34:33.112Z [INFO] server-6992b02d-919e-866e-9ef4-74e77aee2b6d: aborting pipeline replication: peer="{Voter server-0d3f372d-63f7-945c-b01c-a2a1fdf9b5e5 0d3f372d-63f7-945c-b01c-a2a1fdf9b5e5}" ---- PASS: TestRaft_VerifyLeader (0.16s) +testLoggerAdapter verbose: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.930Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce240121-2598-81a5-cbf8-0215ef3bc5ec Address:ce240121-2598-81a5-cbf8-0215ef3bc5ec} {Suffrage:Voter ID:server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb Address:9e31aa83-a216-b7a1-1a6c-25d0da1699eb} {Suffrage:Voter ID:server-b2a27528-b3be-e9e3-6130-3a02f3c122fd Address:b2a27528-b3be-e9e3-6130-3a02f3c122fd}]" + testing.go:149: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.930Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce240121-2598-81a5-cbf8-0215ef3bc5ec Address:ce240121-2598-81a5-cbf8-0215ef3bc5ec} {Suffrage:Voter ID:server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb Address:9e31aa83-a216-b7a1-1a6c-25d0da1699eb} {Suffrage:Voter ID:server-b2a27528-b3be-e9e3-6130-3a02f3c122fd Address:b2a27528-b3be-e9e3-6130-3a02f3c122fd}]" +testLoggerAdapter verbose: server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: 2023-08-02T15:13:01.930Z [INFO] server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce240121-2598-81a5-cbf8-0215ef3bc5ec Address:ce240121-2598-81a5-cbf8-0215ef3bc5ec} {Suffrage:Voter ID:server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb Address:9e31aa83-a216-b7a1-1a6c-25d0da1699eb} {Suffrage:Voter ID:server-b2a27528-b3be-e9e3-6130-3a02f3c122fd Address:b2a27528-b3be-e9e3-6130-3a02f3c122fd}]" + testing.go:149: server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: 2023-08-02T15:13:01.930Z [INFO] server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce240121-2598-81a5-cbf8-0215ef3bc5ec Address:ce240121-2598-81a5-cbf8-0215ef3bc5ec} {Suffrage:Voter ID:server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb Address:9e31aa83-a216-b7a1-1a6c-25d0da1699eb} {Suffrage:Voter ID:server-b2a27528-b3be-e9e3-6130-3a02f3c122fd Address:b2a27528-b3be-e9e3-6130-3a02f3c122fd}]" +testLoggerAdapter verbose: server-b2a27528-b3be-e9e3-6130-3a02f3c122fd: 2023-08-02T15:13:01.931Z [INFO] server-b2a27528-b3be-e9e3-6130-3a02f3c122fd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce240121-2598-81a5-cbf8-0215ef3bc5ec Address:ce240121-2598-81a5-cbf8-0215ef3bc5ec} {Suffrage:Voter ID:server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb Address:9e31aa83-a216-b7a1-1a6c-25d0da1699eb} {Suffrage:Voter ID:server-b2a27528-b3be-e9e3-6130-3a02f3c122fd Address:b2a27528-b3be-e9e3-6130-3a02f3c122fd}]" + testing.go:149: server-b2a27528-b3be-e9e3-6130-3a02f3c122fd: 2023-08-02T15:13:01.931Z [INFO] server-b2a27528-b3be-e9e3-6130-3a02f3c122fd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce240121-2598-81a5-cbf8-0215ef3bc5ec Address:ce240121-2598-81a5-cbf8-0215ef3bc5ec} {Suffrage:Voter ID:server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb Address:9e31aa83-a216-b7a1-1a6c-25d0da1699eb} {Suffrage:Voter ID:server-b2a27528-b3be-e9e3-6130-3a02f3c122fd Address:b2a27528-b3be-e9e3-6130-3a02f3c122fd}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:01.931Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:13:01.931Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.931Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: entering follower state: follower="Node at ce240121-2598-81a5-cbf8-0215ef3bc5ec [Follower]" leader= + testing.go:149: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.931Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: entering follower state: follower="Node at ce240121-2598-81a5-cbf8-0215ef3bc5ec [Follower]" leader= +testLoggerAdapter verbose: server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: 2023-08-02T15:13:01.931Z [INFO] server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: entering follower state: follower="Node at 9e31aa83-a216-b7a1-1a6c-25d0da1699eb [Follower]" leader= + testing.go:149: server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: 2023-08-02T15:13:01.931Z [INFO] server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: entering follower state: follower="Node at 9e31aa83-a216-b7a1-1a6c-25d0da1699eb [Follower]" leader= +testLoggerAdapter verbose: server-b2a27528-b3be-e9e3-6130-3a02f3c122fd: 2023-08-02T15:13:01.931Z [INFO] server-b2a27528-b3be-e9e3-6130-3a02f3c122fd: entering follower state: follower="Node at b2a27528-b3be-e9e3-6130-3a02f3c122fd [Follower]" leader= + testing.go:149: server-b2a27528-b3be-e9e3-6130-3a02f3c122fd: 2023-08-02T15:13:01.931Z [INFO] server-b2a27528-b3be-e9e3-6130-3a02f3c122fd: entering follower state: follower="Node at b2a27528-b3be-e9e3-6130-3a02f3c122fd [Follower]" leader= +testLoggerAdapter verbose: server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: 2023-08-02T15:13:01.985Z [WARN] server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: 2023-08-02T15:13:01.985Z [WARN] server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: 2023-08-02T15:13:01.985Z [INFO] server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: entering candidate state: node="Node at 9e31aa83-a216-b7a1-1a6c-25d0da1699eb [Candidate]" term=2 + testing.go:149: server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: 2023-08-02T15:13:01.985Z [INFO] server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: entering candidate state: node="Node at 9e31aa83-a216-b7a1-1a6c-25d0da1699eb [Candidate]" term=2 +testLoggerAdapter verbose: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.985Z [WARN] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.985Z [WARN] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.985Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: entering candidate state: node="Node at ce240121-2598-81a5-cbf8-0215ef3bc5ec [Candidate]" term=2 + testing.go:149: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.985Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: entering candidate state: node="Node at ce240121-2598-81a5-cbf8-0215ef3bc5ec [Candidate]" term=2 +testLoggerAdapter verbose: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.986Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: election won: tally=2 + testing.go:149: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.986Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: election won: tally=2 +testLoggerAdapter verbose: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.986Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: entering leader state: leader="Node at ce240121-2598-81a5-cbf8-0215ef3bc5ec [Leader]" + testing.go:149: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.986Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: entering leader state: leader="Node at ce240121-2598-81a5-cbf8-0215ef3bc5ec [Leader]" +testLoggerAdapter verbose: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.986Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: added peer, starting replication: peer=server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb + testing.go:149: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.986Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: added peer, starting replication: peer=server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb +testLoggerAdapter verbose: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.986Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: added peer, starting replication: peer=server-b2a27528-b3be-e9e3-6130-3a02f3c122fd + testing.go:149: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.986Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: added peer, starting replication: peer=server-b2a27528-b3be-e9e3-6130-3a02f3c122fd +testLoggerAdapter verbose: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.986Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: pipelining replication: peer="{Voter server-b2a27528-b3be-e9e3-6130-3a02f3c122fd b2a27528-b3be-e9e3-6130-3a02f3c122fd}" + testing.go:149: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.986Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: pipelining replication: peer="{Voter server-b2a27528-b3be-e9e3-6130-3a02f3c122fd b2a27528-b3be-e9e3-6130-3a02f3c122fd}" +testLoggerAdapter verbose: server-b2a27528-b3be-e9e3-6130-3a02f3c122fd: 2023-08-02T15:13:01.986Z [WARN] server-b2a27528-b3be-e9e3-6130-3a02f3c122fd: rejecting vote request since we have a leader: from=9e31aa83-a216-b7a1-1a6c-25d0da1699eb leader=ce240121-2598-81a5-cbf8-0215ef3bc5ec + testing.go:149: server-b2a27528-b3be-e9e3-6130-3a02f3c122fd: 2023-08-02T15:13:01.986Z [WARN] server-b2a27528-b3be-e9e3-6130-3a02f3c122fd: rejecting vote request since we have a leader: from=9e31aa83-a216-b7a1-1a6c-25d0da1699eb leader=ce240121-2598-81a5-cbf8-0215ef3bc5ec +testLoggerAdapter verbose: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.986Z [WARN] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: rejecting vote request since we have a leader: from=9e31aa83-a216-b7a1-1a6c-25d0da1699eb leader=ce240121-2598-81a5-cbf8-0215ef3bc5ec + testing.go:149: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.986Z [WARN] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: rejecting vote request since we have a leader: from=9e31aa83-a216-b7a1-1a6c-25d0da1699eb leader=ce240121-2598-81a5-cbf8-0215ef3bc5ec +testLoggerAdapter verbose: server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: 2023-08-02T15:13:01.986Z [INFO] server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: duplicate requestVote for same term: term=2 + testing.go:149: server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: 2023-08-02T15:13:01.986Z [INFO] server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: duplicate requestVote for same term: term=2 +testLoggerAdapter verbose: server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: 2023-08-02T15:13:01.986Z [INFO] server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: entering follower state: follower="Node at 9e31aa83-a216-b7a1-1a6c-25d0da1699eb [Follower]" leader=ce240121-2598-81a5-cbf8-0215ef3bc5ec + testing.go:149: server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: 2023-08-02T15:13:01.986Z [INFO] server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: entering follower state: follower="Node at 9e31aa83-a216-b7a1-1a6c-25d0da1699eb [Follower]" leader=ce240121-2598-81a5-cbf8-0215ef3bc5ec +testLoggerAdapter verbose: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.986Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: pipelining replication: peer="{Voter server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb 9e31aa83-a216-b7a1-1a6c-25d0da1699eb}" + testing.go:149: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.986Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: pipelining replication: peer="{Voter server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb 9e31aa83-a216-b7a1-1a6c-25d0da1699eb}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:02.092Z [INFO] stable state for Leader reached at 2023-08-02 15:13:01.986783382 +0000 UTC m=+14.082719733 (1 nodes), 55.632441ms from start of poll, 56.281044ms from cluster start. Timeout at 2023-08-02 15:13:02.092067611 +0000 UTC m=+14.188003951, 105.284218ms after stability + testing.go:149: cluster: 2023-08-02T15:13:02.092Z [INFO] stable state for Leader reached at 2023-08-02 15:13:01.986783382 +0000 UTC m=+14.082719733 (1 nodes), 55.632441ms from start of poll, 56.281044ms from cluster start. Timeout at 2023-08-02 15:13:02.092067611 +0000 UTC m=+14.188003951, 105.284218ms after stability +testLoggerAdapter verbose: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:02.092Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: aborting pipeline replication: peer="{Voter server-b2a27528-b3be-e9e3-6130-3a02f3c122fd b2a27528-b3be-e9e3-6130-3a02f3c122fd}" + testing.go:149: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:02.092Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: aborting pipeline replication: peer="{Voter server-b2a27528-b3be-e9e3-6130-3a02f3c122fd b2a27528-b3be-e9e3-6130-3a02f3c122fd}" +testLoggerAdapter verbose: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:02.092Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: aborting pipeline replication: peer="{Voter server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb 9e31aa83-a216-b7a1-1a6c-25d0da1699eb}" + testing.go:149: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:02.092Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: aborting pipeline replication: peer="{Voter server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb 9e31aa83-a216-b7a1-1a6c-25d0da1699eb}" +testLoggerAdapter verbose: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:02.142Z [ERROR] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: failed to heartbeat to: peer=b2a27528-b3be-e9e3-6130-3a02f3c122fd error="command timed out" + testing.go:149: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:02.142Z [ERROR] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: failed to heartbeat to: peer=b2a27528-b3be-e9e3-6130-3a02f3c122fd error="command timed out" +--- PASS: TestRaft_VerifyLeader (0.21s) === RUN TestRaft_VerifyLeader_Single -testLoggerAdapter verbose: server-300dde89-61db-7f0d-301d-ae764d6b2b95: 2024-09-03T21:34:33.113Z [INFO] server-300dde89-61db-7f0d-301d-ae764d6b2b95: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-300dde89-61db-7f0d-301d-ae764d6b2b95 Address:300dde89-61db-7f0d-301d-ae764d6b2b95}]" - testing.go:149: server-300dde89-61db-7f0d-301d-ae764d6b2b95: 2024-09-03T21:34:33.113Z [INFO] server-300dde89-61db-7f0d-301d-ae764d6b2b95: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-300dde89-61db-7f0d-301d-ae764d6b2b95 Address:300dde89-61db-7f0d-301d-ae764d6b2b95}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:33.113Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:33.113Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-300dde89-61db-7f0d-301d-ae764d6b2b95: 2024-09-03T21:34:33.113Z [INFO] server-300dde89-61db-7f0d-301d-ae764d6b2b95: entering follower state: follower="Node at 300dde89-61db-7f0d-301d-ae764d6b2b95 [Follower]" leader= - testing.go:149: server-300dde89-61db-7f0d-301d-ae764d6b2b95: 2024-09-03T21:34:33.113Z [INFO] server-300dde89-61db-7f0d-301d-ae764d6b2b95: entering follower state: follower="Node at 300dde89-61db-7f0d-301d-ae764d6b2b95 [Follower]" leader= -testLoggerAdapter verbose: server-300dde89-61db-7f0d-301d-ae764d6b2b95: 2024-09-03T21:34:33.164Z [WARN] server-300dde89-61db-7f0d-301d-ae764d6b2b95: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-300dde89-61db-7f0d-301d-ae764d6b2b95: 2024-09-03T21:34:33.164Z [WARN] server-300dde89-61db-7f0d-301d-ae764d6b2b95: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-300dde89-61db-7f0d-301d-ae764d6b2b95: 2024-09-03T21:34:33.164Z [INFO] server-300dde89-61db-7f0d-301d-ae764d6b2b95: entering candidate state: node="Node at 300dde89-61db-7f0d-301d-ae764d6b2b95 [Candidate]" term=2 - testing.go:149: server-300dde89-61db-7f0d-301d-ae764d6b2b95: 2024-09-03T21:34:33.164Z [INFO] server-300dde89-61db-7f0d-301d-ae764d6b2b95: entering candidate state: node="Node at 300dde89-61db-7f0d-301d-ae764d6b2b95 [Candidate]" term=2 -testLoggerAdapter verbose: server-300dde89-61db-7f0d-301d-ae764d6b2b95: 2024-09-03T21:34:33.164Z [INFO] server-300dde89-61db-7f0d-301d-ae764d6b2b95: election won: tally=1 - testing.go:149: server-300dde89-61db-7f0d-301d-ae764d6b2b95: 2024-09-03T21:34:33.164Z [INFO] server-300dde89-61db-7f0d-301d-ae764d6b2b95: election won: tally=1 -testLoggerAdapter verbose: server-300dde89-61db-7f0d-301d-ae764d6b2b95: 2024-09-03T21:34:33.164Z [INFO] server-300dde89-61db-7f0d-301d-ae764d6b2b95: entering leader state: leader="Node at 300dde89-61db-7f0d-301d-ae764d6b2b95 [Leader]" - testing.go:149: server-300dde89-61db-7f0d-301d-ae764d6b2b95: 2024-09-03T21:34:33.164Z [INFO] server-300dde89-61db-7f0d-301d-ae764d6b2b95: entering leader state: leader="Node at 300dde89-61db-7f0d-301d-ae764d6b2b95 [Leader]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:33.269Z [INFO] stable state for Leader reached at 2024-09-03 21:34:33.164801238 +0000 UTC m=+14.211458383 (1 nodes), 51.349659ms from start of poll, 51.530783ms from cluster start. Timeout at 2024-09-03 21:34:33.269897669 +0000 UTC m=+14.316554802, 105.096419ms after stability - testing.go:149: cluster: 2024-09-03T21:34:33.269Z [INFO] stable state for Leader reached at 2024-09-03 21:34:33.164801238 +0000 UTC m=+14.211458383 (1 nodes), 51.349659ms from start of poll, 51.530783ms from cluster start. Timeout at 2024-09-03 21:34:33.269897669 +0000 UTC m=+14.316554802, 105.096419ms after stability +testLoggerAdapter verbose: server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: 2023-08-02T15:13:02.143Z [INFO] server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f Address:f4221d0f-3e37-6c31-8f2b-efef8b0bc38f}]" + testing.go:149: server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: 2023-08-02T15:13:02.143Z [INFO] server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f Address:f4221d0f-3e37-6c31-8f2b-efef8b0bc38f}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:02.143Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:13:02.143Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: 2023-08-02T15:13:02.143Z [INFO] server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: entering follower state: follower="Node at f4221d0f-3e37-6c31-8f2b-efef8b0bc38f [Follower]" leader= + testing.go:149: server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: 2023-08-02T15:13:02.143Z [INFO] server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: entering follower state: follower="Node at f4221d0f-3e37-6c31-8f2b-efef8b0bc38f [Follower]" leader= +testLoggerAdapter verbose: server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: 2023-08-02T15:13:02.193Z [WARN] server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: 2023-08-02T15:13:02.193Z [WARN] server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: 2023-08-02T15:13:02.193Z [INFO] server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: entering candidate state: node="Node at f4221d0f-3e37-6c31-8f2b-efef8b0bc38f [Candidate]" term=2 + testing.go:149: server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: 2023-08-02T15:13:02.193Z [INFO] server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: entering candidate state: node="Node at f4221d0f-3e37-6c31-8f2b-efef8b0bc38f [Candidate]" term=2 +testLoggerAdapter verbose: server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: 2023-08-02T15:13:02.194Z [INFO] server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: election won: tally=1 + testing.go:149: server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: 2023-08-02T15:13:02.194Z [INFO] server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: election won: tally=1 +testLoggerAdapter verbose: server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: 2023-08-02T15:13:02.194Z [INFO] server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: entering leader state: leader="Node at f4221d0f-3e37-6c31-8f2b-efef8b0bc38f [Leader]" + testing.go:149: server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: 2023-08-02T15:13:02.194Z [INFO] server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: entering leader state: leader="Node at f4221d0f-3e37-6c31-8f2b-efef8b0bc38f [Leader]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:02.302Z [INFO] stable state for Leader reached at 2023-08-02 15:13:02.194081154 +0000 UTC m=+14.290017500 (1 nodes), 50.656475ms from start of poll, 50.903098ms from cluster start. Timeout at 2023-08-02 15:13:02.302631008 +0000 UTC m=+14.398567421, 108.549921ms after stability + testing.go:149: cluster: 2023-08-02T15:13:02.302Z [INFO] stable state for Leader reached at 2023-08-02 15:13:02.194081154 +0000 UTC m=+14.290017500 (1 nodes), 50.656475ms from start of poll, 50.903098ms from cluster start. Timeout at 2023-08-02 15:13:02.302631008 +0000 UTC m=+14.398567421, 108.549921ms after stability --- PASS: TestRaft_VerifyLeader_Single (0.16s) === RUN TestRaft_VerifyLeader_Fail -testLoggerAdapter verbose: server-654494cb-a38a-433d-0d73-96fd53dac5c0: 2024-09-03T21:34:33.270Z [INFO] server-654494cb-a38a-433d-0d73-96fd53dac5c0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-654494cb-a38a-433d-0d73-96fd53dac5c0 Address:654494cb-a38a-433d-0d73-96fd53dac5c0} {Suffrage:Voter ID:server-f795e4d7-ed3d-7e96-8506-e2531ff11de7 Address:f795e4d7-ed3d-7e96-8506-e2531ff11de7}]" - testing.go:149: server-654494cb-a38a-433d-0d73-96fd53dac5c0: 2024-09-03T21:34:33.270Z [INFO] server-654494cb-a38a-433d-0d73-96fd53dac5c0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-654494cb-a38a-433d-0d73-96fd53dac5c0 Address:654494cb-a38a-433d-0d73-96fd53dac5c0} {Suffrage:Voter ID:server-f795e4d7-ed3d-7e96-8506-e2531ff11de7 Address:f795e4d7-ed3d-7e96-8506-e2531ff11de7}]" -testLoggerAdapter verbose: server-f795e4d7-ed3d-7e96-8506-e2531ff11de7: 2024-09-03T21:34:33.270Z [INFO] server-f795e4d7-ed3d-7e96-8506-e2531ff11de7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-654494cb-a38a-433d-0d73-96fd53dac5c0 Address:654494cb-a38a-433d-0d73-96fd53dac5c0} {Suffrage:Voter ID:server-f795e4d7-ed3d-7e96-8506-e2531ff11de7 Address:f795e4d7-ed3d-7e96-8506-e2531ff11de7}]" - testing.go:149: server-f795e4d7-ed3d-7e96-8506-e2531ff11de7: 2024-09-03T21:34:33.270Z [INFO] server-f795e4d7-ed3d-7e96-8506-e2531ff11de7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-654494cb-a38a-433d-0d73-96fd53dac5c0 Address:654494cb-a38a-433d-0d73-96fd53dac5c0} {Suffrage:Voter ID:server-f795e4d7-ed3d-7e96-8506-e2531ff11de7 Address:f795e4d7-ed3d-7e96-8506-e2531ff11de7}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:33.270Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:33.270Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-654494cb-a38a-433d-0d73-96fd53dac5c0: 2024-09-03T21:34:33.270Z [INFO] server-654494cb-a38a-433d-0d73-96fd53dac5c0: entering follower state: follower="Node at 654494cb-a38a-433d-0d73-96fd53dac5c0 [Follower]" leader= - testing.go:149: server-654494cb-a38a-433d-0d73-96fd53dac5c0: 2024-09-03T21:34:33.270Z [INFO] server-654494cb-a38a-433d-0d73-96fd53dac5c0: entering follower state: follower="Node at 654494cb-a38a-433d-0d73-96fd53dac5c0 [Follower]" leader= -testLoggerAdapter verbose: server-f795e4d7-ed3d-7e96-8506-e2531ff11de7: 2024-09-03T21:34:33.270Z [INFO] server-f795e4d7-ed3d-7e96-8506-e2531ff11de7: entering follower state: follower="Node at f795e4d7-ed3d-7e96-8506-e2531ff11de7 [Follower]" leader= - testing.go:149: server-f795e4d7-ed3d-7e96-8506-e2531ff11de7: 2024-09-03T21:34:33.270Z [INFO] server-f795e4d7-ed3d-7e96-8506-e2531ff11de7: entering follower state: follower="Node at f795e4d7-ed3d-7e96-8506-e2531ff11de7 [Follower]" leader= -testLoggerAdapter verbose: server-654494cb-a38a-433d-0d73-96fd53dac5c0: 2024-09-03T21:34:33.334Z [WARN] server-654494cb-a38a-433d-0d73-96fd53dac5c0: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-654494cb-a38a-433d-0d73-96fd53dac5c0: 2024-09-03T21:34:33.334Z [WARN] server-654494cb-a38a-433d-0d73-96fd53dac5c0: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-654494cb-a38a-433d-0d73-96fd53dac5c0: 2024-09-03T21:34:33.334Z [INFO] server-654494cb-a38a-433d-0d73-96fd53dac5c0: entering candidate state: node="Node at 654494cb-a38a-433d-0d73-96fd53dac5c0 [Candidate]" term=2 - testing.go:149: server-654494cb-a38a-433d-0d73-96fd53dac5c0: 2024-09-03T21:34:33.334Z [INFO] server-654494cb-a38a-433d-0d73-96fd53dac5c0: entering candidate state: node="Node at 654494cb-a38a-433d-0d73-96fd53dac5c0 [Candidate]" term=2 -testLoggerAdapter verbose: server-654494cb-a38a-433d-0d73-96fd53dac5c0: 2024-09-03T21:34:33.334Z [INFO] server-654494cb-a38a-433d-0d73-96fd53dac5c0: election won: tally=2 - testing.go:149: server-654494cb-a38a-433d-0d73-96fd53dac5c0: 2024-09-03T21:34:33.334Z [INFO] server-654494cb-a38a-433d-0d73-96fd53dac5c0: election won: tally=2 -testLoggerAdapter verbose: server-654494cb-a38a-433d-0d73-96fd53dac5c0: 2024-09-03T21:34:33.334Z [INFO] server-654494cb-a38a-433d-0d73-96fd53dac5c0: entering leader state: leader="Node at 654494cb-a38a-433d-0d73-96fd53dac5c0 [Leader]" - testing.go:149: server-654494cb-a38a-433d-0d73-96fd53dac5c0: 2024-09-03T21:34:33.334Z [INFO] server-654494cb-a38a-433d-0d73-96fd53dac5c0: entering leader state: leader="Node at 654494cb-a38a-433d-0d73-96fd53dac5c0 [Leader]" -testLoggerAdapter verbose: server-654494cb-a38a-433d-0d73-96fd53dac5c0: 2024-09-03T21:34:33.334Z [INFO] server-654494cb-a38a-433d-0d73-96fd53dac5c0: added peer, starting replication: peer=server-f795e4d7-ed3d-7e96-8506-e2531ff11de7 - testing.go:149: server-654494cb-a38a-433d-0d73-96fd53dac5c0: 2024-09-03T21:34:33.334Z [INFO] server-654494cb-a38a-433d-0d73-96fd53dac5c0: added peer, starting replication: peer=server-f795e4d7-ed3d-7e96-8506-e2531ff11de7 -testLoggerAdapter verbose: server-654494cb-a38a-433d-0d73-96fd53dac5c0: 2024-09-03T21:34:33.334Z [INFO] server-654494cb-a38a-433d-0d73-96fd53dac5c0: pipelining replication: peer="{Voter server-f795e4d7-ed3d-7e96-8506-e2531ff11de7 f795e4d7-ed3d-7e96-8506-e2531ff11de7}" - testing.go:149: server-654494cb-a38a-433d-0d73-96fd53dac5c0: 2024-09-03T21:34:33.334Z [INFO] server-654494cb-a38a-433d-0d73-96fd53dac5c0: pipelining replication: peer="{Voter server-f795e4d7-ed3d-7e96-8506-e2531ff11de7 f795e4d7-ed3d-7e96-8506-e2531ff11de7}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:33.439Z [INFO] stable state for Leader reached at 2024-09-03 21:34:33.334495735 +0000 UTC m=+14.381152880 (1 nodes), 63.769349ms from start of poll, 64.073822ms from cluster start. Timeout at 2024-09-03 21:34:33.439572101 +0000 UTC m=+14.486229232, 105.076352ms after stability - testing.go:149: cluster: 2024-09-03T21:34:33.439Z [INFO] stable state for Leader reached at 2024-09-03 21:34:33.334495735 +0000 UTC m=+14.381152880 (1 nodes), 63.769349ms from start of poll, 64.073822ms from cluster start. Timeout at 2024-09-03 21:34:33.439572101 +0000 UTC m=+14.486229232, 105.076352ms after stability -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:33.439Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-09-03T21:34:33.439Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:33.544Z [INFO] stable state for Follower reached at 2024-09-03 21:34:33.439709261 +0000 UTC m=+14.486366392 (1 nodes), 870ns from start of poll, 169.287334ms from cluster start. Timeout at 2024-09-03 21:34:33.544793493 +0000 UTC m=+14.591450633, 105.084241ms after stability - testing.go:149: cluster: 2024-09-03T21:34:33.544Z [INFO] stable state for Follower reached at 2024-09-03 21:34:33.439709261 +0000 UTC m=+14.486366392 (1 nodes), 870ns from start of poll, 169.287334ms from cluster start. Timeout at 2024-09-03 21:34:33.544793493 +0000 UTC m=+14.591450633, 105.084241ms after stability -testLoggerAdapter verbose: server-654494cb-a38a-433d-0d73-96fd53dac5c0: 2024-09-03T21:34:33.544Z [WARN] server-654494cb-a38a-433d-0d73-96fd53dac5c0: new leader elected, stepping down - testing.go:149: server-654494cb-a38a-433d-0d73-96fd53dac5c0: 2024-09-03T21:34:33.544Z [WARN] server-654494cb-a38a-433d-0d73-96fd53dac5c0: new leader elected, stepping down -testLoggerAdapter verbose: server-654494cb-a38a-433d-0d73-96fd53dac5c0: 2024-09-03T21:34:33.545Z [INFO] server-654494cb-a38a-433d-0d73-96fd53dac5c0: entering follower state: follower="Node at 654494cb-a38a-433d-0d73-96fd53dac5c0 [Follower]" leader= - testing.go:149: server-654494cb-a38a-433d-0d73-96fd53dac5c0: 2024-09-03T21:34:33.545Z [INFO] server-654494cb-a38a-433d-0d73-96fd53dac5c0: entering follower state: follower="Node at 654494cb-a38a-433d-0d73-96fd53dac5c0 [Follower]" leader= -testLoggerAdapter verbose: server-654494cb-a38a-433d-0d73-96fd53dac5c0: 2024-09-03T21:34:33.545Z [INFO] server-654494cb-a38a-433d-0d73-96fd53dac5c0: aborting pipeline replication: peer="{Voter server-f795e4d7-ed3d-7e96-8506-e2531ff11de7 f795e4d7-ed3d-7e96-8506-e2531ff11de7}" - testing.go:149: server-654494cb-a38a-433d-0d73-96fd53dac5c0: 2024-09-03T21:34:33.545Z [INFO] server-654494cb-a38a-433d-0d73-96fd53dac5c0: aborting pipeline replication: peer="{Voter server-f795e4d7-ed3d-7e96-8506-e2531ff11de7 f795e4d7-ed3d-7e96-8506-e2531ff11de7}" +testLoggerAdapter verbose: server-7c654af0-419b-3eb9-1d7d-8b0ee8d572f0: 2023-08-02T15:13:02.303Z [INFO] server-7c654af0-419b-3eb9-1d7d-8b0ee8d572f0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7c654af0-419b-3eb9-1d7d-8b0ee8d572f0 Address:7c654af0-419b-3eb9-1d7d-8b0ee8d572f0} {Suffrage:Voter ID:server-a68448d6-81fd-6fdd-9450-70ba17177931 Address:a68448d6-81fd-6fdd-9450-70ba17177931}]" + testing.go:149: server-7c654af0-419b-3eb9-1d7d-8b0ee8d572f0: 2023-08-02T15:13:02.303Z [INFO] server-7c654af0-419b-3eb9-1d7d-8b0ee8d572f0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7c654af0-419b-3eb9-1d7d-8b0ee8d572f0 Address:7c654af0-419b-3eb9-1d7d-8b0ee8d572f0} {Suffrage:Voter ID:server-a68448d6-81fd-6fdd-9450-70ba17177931 Address:a68448d6-81fd-6fdd-9450-70ba17177931}]" +testLoggerAdapter verbose: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.304Z [INFO] server-a68448d6-81fd-6fdd-9450-70ba17177931: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7c654af0-419b-3eb9-1d7d-8b0ee8d572f0 Address:7c654af0-419b-3eb9-1d7d-8b0ee8d572f0} {Suffrage:Voter ID:server-a68448d6-81fd-6fdd-9450-70ba17177931 Address:a68448d6-81fd-6fdd-9450-70ba17177931}]" + testing.go:149: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.304Z [INFO] server-a68448d6-81fd-6fdd-9450-70ba17177931: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7c654af0-419b-3eb9-1d7d-8b0ee8d572f0 Address:7c654af0-419b-3eb9-1d7d-8b0ee8d572f0} {Suffrage:Voter ID:server-a68448d6-81fd-6fdd-9450-70ba17177931 Address:a68448d6-81fd-6fdd-9450-70ba17177931}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:02.304Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:13:02.304Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-7c654af0-419b-3eb9-1d7d-8b0ee8d572f0: 2023-08-02T15:13:02.304Z [INFO] server-7c654af0-419b-3eb9-1d7d-8b0ee8d572f0: entering follower state: follower="Node at 7c654af0-419b-3eb9-1d7d-8b0ee8d572f0 [Follower]" leader= + testing.go:149: server-7c654af0-419b-3eb9-1d7d-8b0ee8d572f0: 2023-08-02T15:13:02.304Z [INFO] server-7c654af0-419b-3eb9-1d7d-8b0ee8d572f0: entering follower state: follower="Node at 7c654af0-419b-3eb9-1d7d-8b0ee8d572f0 [Follower]" leader= +testLoggerAdapter verbose: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.304Z [INFO] server-a68448d6-81fd-6fdd-9450-70ba17177931: entering follower state: follower="Node at a68448d6-81fd-6fdd-9450-70ba17177931 [Follower]" leader= + testing.go:149: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.304Z [INFO] server-a68448d6-81fd-6fdd-9450-70ba17177931: entering follower state: follower="Node at a68448d6-81fd-6fdd-9450-70ba17177931 [Follower]" leader= +testLoggerAdapter verbose: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.372Z [WARN] server-a68448d6-81fd-6fdd-9450-70ba17177931: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.372Z [WARN] server-a68448d6-81fd-6fdd-9450-70ba17177931: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.372Z [INFO] server-a68448d6-81fd-6fdd-9450-70ba17177931: entering candidate state: node="Node at a68448d6-81fd-6fdd-9450-70ba17177931 [Candidate]" term=2 + testing.go:149: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.372Z [INFO] server-a68448d6-81fd-6fdd-9450-70ba17177931: entering candidate state: node="Node at a68448d6-81fd-6fdd-9450-70ba17177931 [Candidate]" term=2 +testLoggerAdapter verbose: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.372Z [INFO] server-a68448d6-81fd-6fdd-9450-70ba17177931: election won: tally=2 + testing.go:149: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.372Z [INFO] server-a68448d6-81fd-6fdd-9450-70ba17177931: election won: tally=2 +testLoggerAdapter verbose: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.372Z [INFO] server-a68448d6-81fd-6fdd-9450-70ba17177931: entering leader state: leader="Node at a68448d6-81fd-6fdd-9450-70ba17177931 [Leader]" + testing.go:149: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.372Z [INFO] server-a68448d6-81fd-6fdd-9450-70ba17177931: entering leader state: leader="Node at a68448d6-81fd-6fdd-9450-70ba17177931 [Leader]" +testLoggerAdapter verbose: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.372Z [INFO] server-a68448d6-81fd-6fdd-9450-70ba17177931: added peer, starting replication: peer=server-7c654af0-419b-3eb9-1d7d-8b0ee8d572f0 + testing.go:149: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.372Z [INFO] server-a68448d6-81fd-6fdd-9450-70ba17177931: added peer, starting replication: peer=server-7c654af0-419b-3eb9-1d7d-8b0ee8d572f0 +testLoggerAdapter verbose: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.372Z [INFO] server-a68448d6-81fd-6fdd-9450-70ba17177931: pipelining replication: peer="{Voter server-7c654af0-419b-3eb9-1d7d-8b0ee8d572f0 7c654af0-419b-3eb9-1d7d-8b0ee8d572f0}" + testing.go:149: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.372Z [INFO] server-a68448d6-81fd-6fdd-9450-70ba17177931: pipelining replication: peer="{Voter server-7c654af0-419b-3eb9-1d7d-8b0ee8d572f0 7c654af0-419b-3eb9-1d7d-8b0ee8d572f0}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:02.477Z [INFO] stable state for Leader reached at 2023-08-02 15:13:02.372485667 +0000 UTC m=+14.468421996 (1 nodes), 68.251859ms from start of poll, 68.848709ms from cluster start. Timeout at 2023-08-02 15:13:02.477602556 +0000 UTC m=+14.573538891, 105.116895ms after stability + testing.go:149: cluster: 2023-08-02T15:13:02.477Z [INFO] stable state for Leader reached at 2023-08-02 15:13:02.372485667 +0000 UTC m=+14.468421996 (1 nodes), 68.251859ms from start of poll, 68.848709ms from cluster start. Timeout at 2023-08-02 15:13:02.477602556 +0000 UTC m=+14.573538891, 105.116895ms after stability +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:02.477Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-08-02T15:13:02.477Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:02.582Z [INFO] stable state for Follower reached at 2023-08-02 15:13:02.477833216 +0000 UTC m=+14.573769554 (1 nodes), 4.277µs from start of poll, 174.196267ms from cluster start. Timeout at 2023-08-02 15:13:02.582944698 +0000 UTC m=+14.678881026, 105.111472ms after stability + testing.go:149: cluster: 2023-08-02T15:13:02.582Z [INFO] stable state for Follower reached at 2023-08-02 15:13:02.477833216 +0000 UTC m=+14.573769554 (1 nodes), 4.277µs from start of poll, 174.196267ms from cluster start. Timeout at 2023-08-02 15:13:02.582944698 +0000 UTC m=+14.678881026, 105.111472ms after stability +testLoggerAdapter verbose: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.583Z [WARN] server-a68448d6-81fd-6fdd-9450-70ba17177931: new leader elected, stepping down + testing.go:149: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.583Z [WARN] server-a68448d6-81fd-6fdd-9450-70ba17177931: new leader elected, stepping down +testLoggerAdapter verbose: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.583Z [INFO] server-a68448d6-81fd-6fdd-9450-70ba17177931: entering follower state: follower="Node at a68448d6-81fd-6fdd-9450-70ba17177931 [Follower]" leader= + testing.go:149: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.583Z [INFO] server-a68448d6-81fd-6fdd-9450-70ba17177931: entering follower state: follower="Node at a68448d6-81fd-6fdd-9450-70ba17177931 [Follower]" leader= +testLoggerAdapter verbose: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.583Z [INFO] server-a68448d6-81fd-6fdd-9450-70ba17177931: aborting pipeline replication: peer="{Voter server-7c654af0-419b-3eb9-1d7d-8b0ee8d572f0 7c654af0-419b-3eb9-1d7d-8b0ee8d572f0}" + testing.go:149: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.583Z [INFO] server-a68448d6-81fd-6fdd-9450-70ba17177931: aborting pipeline replication: peer="{Voter server-7c654af0-419b-3eb9-1d7d-8b0ee8d572f0 7c654af0-419b-3eb9-1d7d-8b0ee8d572f0}" --- PASS: TestRaft_VerifyLeader_Fail (0.28s) === RUN TestRaft_VerifyLeader_PartialConnect -testLoggerAdapter verbose: server-cdf84696-fe2c-2be8-6636-db2d2091c2e5: 2024-09-03T21:34:33.545Z [INFO] server-cdf84696-fe2c-2be8-6636-db2d2091c2e5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cdf84696-fe2c-2be8-6636-db2d2091c2e5 Address:cdf84696-fe2c-2be8-6636-db2d2091c2e5} {Suffrage:Voter ID:server-43c0a050-aa04-3472-d072-4c10812ae288 Address:43c0a050-aa04-3472-d072-4c10812ae288} {Suffrage:Voter ID:server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3 Address:692f1afc-f84a-68ef-8afa-2e63d89b2ec3}]" - testing.go:149: server-cdf84696-fe2c-2be8-6636-db2d2091c2e5: 2024-09-03T21:34:33.545Z [INFO] server-cdf84696-fe2c-2be8-6636-db2d2091c2e5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cdf84696-fe2c-2be8-6636-db2d2091c2e5 Address:cdf84696-fe2c-2be8-6636-db2d2091c2e5} {Suffrage:Voter ID:server-43c0a050-aa04-3472-d072-4c10812ae288 Address:43c0a050-aa04-3472-d072-4c10812ae288} {Suffrage:Voter ID:server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3 Address:692f1afc-f84a-68ef-8afa-2e63d89b2ec3}]" -testLoggerAdapter verbose: server-43c0a050-aa04-3472-d072-4c10812ae288: 2024-09-03T21:34:33.545Z [INFO] server-43c0a050-aa04-3472-d072-4c10812ae288: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cdf84696-fe2c-2be8-6636-db2d2091c2e5 Address:cdf84696-fe2c-2be8-6636-db2d2091c2e5} {Suffrage:Voter ID:server-43c0a050-aa04-3472-d072-4c10812ae288 Address:43c0a050-aa04-3472-d072-4c10812ae288} {Suffrage:Voter ID:server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3 Address:692f1afc-f84a-68ef-8afa-2e63d89b2ec3}]" - testing.go:149: server-43c0a050-aa04-3472-d072-4c10812ae288: 2024-09-03T21:34:33.545Z [INFO] server-43c0a050-aa04-3472-d072-4c10812ae288: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cdf84696-fe2c-2be8-6636-db2d2091c2e5 Address:cdf84696-fe2c-2be8-6636-db2d2091c2e5} {Suffrage:Voter ID:server-43c0a050-aa04-3472-d072-4c10812ae288 Address:43c0a050-aa04-3472-d072-4c10812ae288} {Suffrage:Voter ID:server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3 Address:692f1afc-f84a-68ef-8afa-2e63d89b2ec3}]" -testLoggerAdapter verbose: server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: 2024-09-03T21:34:33.545Z [INFO] server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cdf84696-fe2c-2be8-6636-db2d2091c2e5 Address:cdf84696-fe2c-2be8-6636-db2d2091c2e5} {Suffrage:Voter ID:server-43c0a050-aa04-3472-d072-4c10812ae288 Address:43c0a050-aa04-3472-d072-4c10812ae288} {Suffrage:Voter ID:server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3 Address:692f1afc-f84a-68ef-8afa-2e63d89b2ec3}]" - testing.go:149: server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: 2024-09-03T21:34:33.545Z [INFO] server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cdf84696-fe2c-2be8-6636-db2d2091c2e5 Address:cdf84696-fe2c-2be8-6636-db2d2091c2e5} {Suffrage:Voter ID:server-43c0a050-aa04-3472-d072-4c10812ae288 Address:43c0a050-aa04-3472-d072-4c10812ae288} {Suffrage:Voter ID:server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3 Address:692f1afc-f84a-68ef-8afa-2e63d89b2ec3}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:33.546Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:33.546Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-cdf84696-fe2c-2be8-6636-db2d2091c2e5: 2024-09-03T21:34:33.546Z [INFO] server-cdf84696-fe2c-2be8-6636-db2d2091c2e5: entering follower state: follower="Node at cdf84696-fe2c-2be8-6636-db2d2091c2e5 [Follower]" leader= - testing.go:149: server-cdf84696-fe2c-2be8-6636-db2d2091c2e5: 2024-09-03T21:34:33.546Z [INFO] server-cdf84696-fe2c-2be8-6636-db2d2091c2e5: entering follower state: follower="Node at cdf84696-fe2c-2be8-6636-db2d2091c2e5 [Follower]" leader= -testLoggerAdapter verbose: server-43c0a050-aa04-3472-d072-4c10812ae288: 2024-09-03T21:34:33.546Z [INFO] server-43c0a050-aa04-3472-d072-4c10812ae288: entering follower state: follower="Node at 43c0a050-aa04-3472-d072-4c10812ae288 [Follower]" leader= - testing.go:149: server-43c0a050-aa04-3472-d072-4c10812ae288: 2024-09-03T21:34:33.546Z [INFO] server-43c0a050-aa04-3472-d072-4c10812ae288: entering follower state: follower="Node at 43c0a050-aa04-3472-d072-4c10812ae288 [Follower]" leader= -testLoggerAdapter verbose: server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: 2024-09-03T21:34:33.546Z [INFO] server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: entering follower state: follower="Node at 692f1afc-f84a-68ef-8afa-2e63d89b2ec3 [Follower]" leader= - testing.go:149: server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: 2024-09-03T21:34:33.546Z [INFO] server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: entering follower state: follower="Node at 692f1afc-f84a-68ef-8afa-2e63d89b2ec3 [Follower]" leader= -testLoggerAdapter verbose: server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: 2024-09-03T21:34:33.596Z [WARN] server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: 2024-09-03T21:34:33.596Z [WARN] server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: 2024-09-03T21:34:33.596Z [INFO] server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: entering candidate state: node="Node at 692f1afc-f84a-68ef-8afa-2e63d89b2ec3 [Candidate]" term=2 - testing.go:149: server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: 2024-09-03T21:34:33.596Z [INFO] server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: entering candidate state: node="Node at 692f1afc-f84a-68ef-8afa-2e63d89b2ec3 [Candidate]" term=2 -testLoggerAdapter verbose: server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: 2024-09-03T21:34:33.596Z [INFO] server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: election won: tally=2 - testing.go:149: server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: 2024-09-03T21:34:33.596Z [INFO] server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: election won: tally=2 -testLoggerAdapter verbose: server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: 2024-09-03T21:34:33.597Z [INFO] server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: entering leader state: leader="Node at 692f1afc-f84a-68ef-8afa-2e63d89b2ec3 [Leader]" - testing.go:149: server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: 2024-09-03T21:34:33.597Z [INFO] server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: entering leader state: leader="Node at 692f1afc-f84a-68ef-8afa-2e63d89b2ec3 [Leader]" -testLoggerAdapter verbose: server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: 2024-09-03T21:34:33.597Z [INFO] server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: added peer, starting replication: peer=server-cdf84696-fe2c-2be8-6636-db2d2091c2e5 - testing.go:149: server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: 2024-09-03T21:34:33.597Z [INFO] server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: added peer, starting replication: peer=server-cdf84696-fe2c-2be8-6636-db2d2091c2e5 -testLoggerAdapter verbose: server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: 2024-09-03T21:34:33.597Z [INFO] server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: added peer, starting replication: peer=server-43c0a050-aa04-3472-d072-4c10812ae288 - testing.go:149: server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: 2024-09-03T21:34:33.597Z [INFO] server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: added peer, starting replication: peer=server-43c0a050-aa04-3472-d072-4c10812ae288 -testLoggerAdapter verbose: server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: 2024-09-03T21:34:33.597Z [INFO] server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: pipelining replication: peer="{Voter server-43c0a050-aa04-3472-d072-4c10812ae288 43c0a050-aa04-3472-d072-4c10812ae288}" - testing.go:149: server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: 2024-09-03T21:34:33.597Z [INFO] server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: pipelining replication: peer="{Voter server-43c0a050-aa04-3472-d072-4c10812ae288 43c0a050-aa04-3472-d072-4c10812ae288}" -testLoggerAdapter verbose: server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: 2024-09-03T21:34:33.597Z [INFO] server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: pipelining replication: peer="{Voter server-cdf84696-fe2c-2be8-6636-db2d2091c2e5 cdf84696-fe2c-2be8-6636-db2d2091c2e5}" - testing.go:149: server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: 2024-09-03T21:34:33.597Z [INFO] server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: pipelining replication: peer="{Voter server-cdf84696-fe2c-2be8-6636-db2d2091c2e5 cdf84696-fe2c-2be8-6636-db2d2091c2e5}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:33.702Z [INFO] stable state for Leader reached at 2024-09-03 21:34:33.597343251 +0000 UTC m=+14.644000382 (1 nodes), 51.311263ms from start of poll, 51.72232ms from cluster start. Timeout at 2024-09-03 21:34:33.702435343 +0000 UTC m=+14.749092482, 105.0921ms after stability - testing.go:149: cluster: 2024-09-03T21:34:33.702Z [INFO] stable state for Leader reached at 2024-09-03 21:34:33.597343251 +0000 UTC m=+14.644000382 (1 nodes), 51.311263ms from start of poll, 51.72232ms from cluster start. Timeout at 2024-09-03 21:34:33.702435343 +0000 UTC m=+14.749092482, 105.0921ms after stability -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:33.707Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-09-03T21:34:33.707Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:33.812Z [INFO] stable state for Follower reached at 2024-09-03 21:34:33.707722439 +0000 UTC m=+14.754379580 (2 nodes), 1.579µs from start of poll, 162.101518ms from cluster start. Timeout at 2024-09-03 21:34:33.812816108 +0000 UTC m=+14.859473245, 105.093665ms after stability - testing.go:149: cluster: 2024-09-03T21:34:33.812Z [INFO] stable state for Follower reached at 2024-09-03 21:34:33.707722439 +0000 UTC m=+14.754379580 (2 nodes), 1.579µs from start of poll, 162.101518ms from cluster start. Timeout at 2024-09-03 21:34:33.812816108 +0000 UTC m=+14.859473245, 105.093665ms after stability - raft_test.go:1647: [INFO] Disconnecting Node at cdf84696-fe2c-2be8-6636-db2d2091c2e5 [Follower] -testLoggerAdapter verbose: server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: 2024-09-03T21:34:33.813Z [ERROR] server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: failed to heartbeat to: peer=cdf84696-fe2c-2be8-6636-db2d2091c2e5 error="failed to connect to peer: cdf84696-fe2c-2be8-6636-db2d2091c2e5" - testing.go:149: server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: 2024-09-03T21:34:33.813Z [ERROR] server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: failed to heartbeat to: peer=cdf84696-fe2c-2be8-6636-db2d2091c2e5 error="failed to connect to peer: cdf84696-fe2c-2be8-6636-db2d2091c2e5" -testLoggerAdapter verbose: server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: 2024-09-03T21:34:33.813Z [INFO] server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: aborting pipeline replication: peer="{Voter server-cdf84696-fe2c-2be8-6636-db2d2091c2e5 cdf84696-fe2c-2be8-6636-db2d2091c2e5}" - testing.go:149: server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: 2024-09-03T21:34:33.813Z [INFO] server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: aborting pipeline replication: peer="{Voter server-cdf84696-fe2c-2be8-6636-db2d2091c2e5 cdf84696-fe2c-2be8-6636-db2d2091c2e5}" -testLoggerAdapter verbose: server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: 2024-09-03T21:34:33.813Z [INFO] server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: aborting pipeline replication: peer="{Voter server-43c0a050-aa04-3472-d072-4c10812ae288 43c0a050-aa04-3472-d072-4c10812ae288}" - testing.go:149: server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: 2024-09-03T21:34:33.813Z [INFO] server-692f1afc-f84a-68ef-8afa-2e63d89b2ec3: aborting pipeline replication: peer="{Voter server-43c0a050-aa04-3472-d072-4c10812ae288 43c0a050-aa04-3472-d072-4c10812ae288}" ---- PASS: TestRaft_VerifyLeader_PartialConnect (0.27s) +testLoggerAdapter verbose: server-3645a2fd-2c05-2806-6e81-ed5f4763e766: 2023-08-02T15:13:02.584Z [INFO] server-3645a2fd-2c05-2806-6e81-ed5f4763e766: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3645a2fd-2c05-2806-6e81-ed5f4763e766 Address:3645a2fd-2c05-2806-6e81-ed5f4763e766} {Suffrage:Voter ID:server-4d34c2e8-a619-4768-eded-14353e29d27b Address:4d34c2e8-a619-4768-eded-14353e29d27b} {Suffrage:Voter ID:server-b3744a4e-f1ec-98db-8b97-0efc209868ca Address:b3744a4e-f1ec-98db-8b97-0efc209868ca}]" + testing.go:149: server-3645a2fd-2c05-2806-6e81-ed5f4763e766: 2023-08-02T15:13:02.584Z [INFO] server-3645a2fd-2c05-2806-6e81-ed5f4763e766: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3645a2fd-2c05-2806-6e81-ed5f4763e766 Address:3645a2fd-2c05-2806-6e81-ed5f4763e766} {Suffrage:Voter ID:server-4d34c2e8-a619-4768-eded-14353e29d27b Address:4d34c2e8-a619-4768-eded-14353e29d27b} {Suffrage:Voter ID:server-b3744a4e-f1ec-98db-8b97-0efc209868ca Address:b3744a4e-f1ec-98db-8b97-0efc209868ca}]" +testLoggerAdapter verbose: server-3645a2fd-2c05-2806-6e81-ed5f4763e766: 2023-08-02T15:13:02.584Z [INFO] server-3645a2fd-2c05-2806-6e81-ed5f4763e766: entering follower state: follower="Node at 3645a2fd-2c05-2806-6e81-ed5f4763e766 [Follower]" leader= + testing.go:149: server-3645a2fd-2c05-2806-6e81-ed5f4763e766: 2023-08-02T15:13:02.584Z [INFO] server-3645a2fd-2c05-2806-6e81-ed5f4763e766: entering follower state: follower="Node at 3645a2fd-2c05-2806-6e81-ed5f4763e766 [Follower]" leader= +testLoggerAdapter verbose: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.584Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3645a2fd-2c05-2806-6e81-ed5f4763e766 Address:3645a2fd-2c05-2806-6e81-ed5f4763e766} {Suffrage:Voter ID:server-4d34c2e8-a619-4768-eded-14353e29d27b Address:4d34c2e8-a619-4768-eded-14353e29d27b} {Suffrage:Voter ID:server-b3744a4e-f1ec-98db-8b97-0efc209868ca Address:b3744a4e-f1ec-98db-8b97-0efc209868ca}]" + testing.go:149: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.584Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3645a2fd-2c05-2806-6e81-ed5f4763e766 Address:3645a2fd-2c05-2806-6e81-ed5f4763e766} {Suffrage:Voter ID:server-4d34c2e8-a619-4768-eded-14353e29d27b Address:4d34c2e8-a619-4768-eded-14353e29d27b} {Suffrage:Voter ID:server-b3744a4e-f1ec-98db-8b97-0efc209868ca Address:b3744a4e-f1ec-98db-8b97-0efc209868ca}]" +testLoggerAdapter verbose: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.584Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: entering follower state: follower="Node at 4d34c2e8-a619-4768-eded-14353e29d27b [Follower]" leader= + testing.go:149: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.584Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: entering follower state: follower="Node at 4d34c2e8-a619-4768-eded-14353e29d27b [Follower]" leader= +testLoggerAdapter verbose: server-b3744a4e-f1ec-98db-8b97-0efc209868ca: 2023-08-02T15:13:02.584Z [INFO] server-b3744a4e-f1ec-98db-8b97-0efc209868ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3645a2fd-2c05-2806-6e81-ed5f4763e766 Address:3645a2fd-2c05-2806-6e81-ed5f4763e766} {Suffrage:Voter ID:server-4d34c2e8-a619-4768-eded-14353e29d27b Address:4d34c2e8-a619-4768-eded-14353e29d27b} {Suffrage:Voter ID:server-b3744a4e-f1ec-98db-8b97-0efc209868ca Address:b3744a4e-f1ec-98db-8b97-0efc209868ca}]" + testing.go:149: server-b3744a4e-f1ec-98db-8b97-0efc209868ca: 2023-08-02T15:13:02.584Z [INFO] server-b3744a4e-f1ec-98db-8b97-0efc209868ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3645a2fd-2c05-2806-6e81-ed5f4763e766 Address:3645a2fd-2c05-2806-6e81-ed5f4763e766} {Suffrage:Voter ID:server-4d34c2e8-a619-4768-eded-14353e29d27b Address:4d34c2e8-a619-4768-eded-14353e29d27b} {Suffrage:Voter ID:server-b3744a4e-f1ec-98db-8b97-0efc209868ca Address:b3744a4e-f1ec-98db-8b97-0efc209868ca}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:02.584Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:13:02.584Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-b3744a4e-f1ec-98db-8b97-0efc209868ca: 2023-08-02T15:13:02.584Z [INFO] server-b3744a4e-f1ec-98db-8b97-0efc209868ca: entering follower state: follower="Node at b3744a4e-f1ec-98db-8b97-0efc209868ca [Follower]" leader= + testing.go:149: server-b3744a4e-f1ec-98db-8b97-0efc209868ca: 2023-08-02T15:13:02.584Z [INFO] server-b3744a4e-f1ec-98db-8b97-0efc209868ca: entering follower state: follower="Node at b3744a4e-f1ec-98db-8b97-0efc209868ca [Follower]" leader= +testLoggerAdapter verbose: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.666Z [WARN] server-4d34c2e8-a619-4768-eded-14353e29d27b: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.666Z [WARN] server-4d34c2e8-a619-4768-eded-14353e29d27b: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.666Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: entering candidate state: node="Node at 4d34c2e8-a619-4768-eded-14353e29d27b [Candidate]" term=2 + testing.go:149: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.666Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: entering candidate state: node="Node at 4d34c2e8-a619-4768-eded-14353e29d27b [Candidate]" term=2 +testLoggerAdapter verbose: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.666Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: election won: tally=2 + testing.go:149: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.666Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: election won: tally=2 +testLoggerAdapter verbose: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.666Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: entering leader state: leader="Node at 4d34c2e8-a619-4768-eded-14353e29d27b [Leader]" + testing.go:149: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.666Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: entering leader state: leader="Node at 4d34c2e8-a619-4768-eded-14353e29d27b [Leader]" +testLoggerAdapter verbose: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.666Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: added peer, starting replication: peer=server-3645a2fd-2c05-2806-6e81-ed5f4763e766 + testing.go:149: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.666Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: added peer, starting replication: peer=server-3645a2fd-2c05-2806-6e81-ed5f4763e766 +testLoggerAdapter verbose: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.666Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: added peer, starting replication: peer=server-b3744a4e-f1ec-98db-8b97-0efc209868ca + testing.go:149: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.666Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: added peer, starting replication: peer=server-b3744a4e-f1ec-98db-8b97-0efc209868ca +testLoggerAdapter verbose: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.666Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: pipelining replication: peer="{Voter server-b3744a4e-f1ec-98db-8b97-0efc209868ca b3744a4e-f1ec-98db-8b97-0efc209868ca}" + testing.go:149: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.666Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: pipelining replication: peer="{Voter server-b3744a4e-f1ec-98db-8b97-0efc209868ca b3744a4e-f1ec-98db-8b97-0efc209868ca}" +testLoggerAdapter verbose: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.666Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: pipelining replication: peer="{Voter server-3645a2fd-2c05-2806-6e81-ed5f4763e766 3645a2fd-2c05-2806-6e81-ed5f4763e766}" + testing.go:149: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.666Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: pipelining replication: peer="{Voter server-3645a2fd-2c05-2806-6e81-ed5f4763e766 3645a2fd-2c05-2806-6e81-ed5f4763e766}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:02.771Z [INFO] stable state for Leader reached at 2023-08-02 15:13:02.66678072 +0000 UTC m=+14.762717072 (1 nodes), 82.250255ms from start of poll, 82.816725ms from cluster start. Timeout at 2023-08-02 15:13:02.771921304 +0000 UTC m=+14.867857637, 105.140565ms after stability + testing.go:149: cluster: 2023-08-02T15:13:02.771Z [INFO] stable state for Leader reached at 2023-08-02 15:13:02.66678072 +0000 UTC m=+14.762717072 (1 nodes), 82.250255ms from start of poll, 82.816725ms from cluster start. Timeout at 2023-08-02 15:13:02.771921304 +0000 UTC m=+14.867857637, 105.140565ms after stability +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:02.777Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-08-02T15:13:02.777Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:02.883Z [INFO] stable state for Follower reached at 2023-08-02 15:13:02.777263003 +0000 UTC m=+14.873199329 (2 nodes), 1.546µs from start of poll, 193.298982ms from cluster start. Timeout at 2023-08-02 15:13:02.883318944 +0000 UTC m=+14.979255287, 106.055958ms after stability + testing.go:149: cluster: 2023-08-02T15:13:02.883Z [INFO] stable state for Follower reached at 2023-08-02 15:13:02.777263003 +0000 UTC m=+14.873199329 (2 nodes), 1.546µs from start of poll, 193.298982ms from cluster start. Timeout at 2023-08-02 15:13:02.883318944 +0000 UTC m=+14.979255287, 106.055958ms after stability + raft_test.go:1647: [INFO] Disconnecting Node at 3645a2fd-2c05-2806-6e81-ed5f4763e766 [Follower] +testLoggerAdapter verbose: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.883Z [ERROR] server-4d34c2e8-a619-4768-eded-14353e29d27b: failed to heartbeat to: peer=3645a2fd-2c05-2806-6e81-ed5f4763e766 error="failed to connect to peer: 3645a2fd-2c05-2806-6e81-ed5f4763e766" + testing.go:149: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.883Z [ERROR] server-4d34c2e8-a619-4768-eded-14353e29d27b: failed to heartbeat to: peer=3645a2fd-2c05-2806-6e81-ed5f4763e766 error="failed to connect to peer: 3645a2fd-2c05-2806-6e81-ed5f4763e766" +testLoggerAdapter verbose: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.883Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: aborting pipeline replication: peer="{Voter server-b3744a4e-f1ec-98db-8b97-0efc209868ca b3744a4e-f1ec-98db-8b97-0efc209868ca}" + testing.go:149: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.883Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: aborting pipeline replication: peer="{Voter server-b3744a4e-f1ec-98db-8b97-0efc209868ca b3744a4e-f1ec-98db-8b97-0efc209868ca}" +testLoggerAdapter verbose: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.883Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: aborting pipeline replication: peer="{Voter server-3645a2fd-2c05-2806-6e81-ed5f4763e766 3645a2fd-2c05-2806-6e81-ed5f4763e766}" + testing.go:149: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.883Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: aborting pipeline replication: peer="{Voter server-3645a2fd-2c05-2806-6e81-ed5f4763e766 3645a2fd-2c05-2806-6e81-ed5f4763e766}" +--- PASS: TestRaft_VerifyLeader_PartialConnect (0.30s) === RUN TestRaft_StartAsLeader -testLoggerAdapter verbose: server-389af727-720b-73fd-bc35-70b7a7db5333: 2024-09-03T21:34:33.813Z [INFO] server-389af727-720b-73fd-bc35-70b7a7db5333: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-389af727-720b-73fd-bc35-70b7a7db5333 Address:389af727-720b-73fd-bc35-70b7a7db5333}]" - testing.go:149: server-389af727-720b-73fd-bc35-70b7a7db5333: 2024-09-03T21:34:33.813Z [INFO] server-389af727-720b-73fd-bc35-70b7a7db5333: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-389af727-720b-73fd-bc35-70b7a7db5333 Address:389af727-720b-73fd-bc35-70b7a7db5333}]" -testLoggerAdapter verbose: server-389af727-720b-73fd-bc35-70b7a7db5333: 2024-09-03T21:34:33.813Z [INFO] server-389af727-720b-73fd-bc35-70b7a7db5333: entering leader state: leader="Node at 389af727-720b-73fd-bc35-70b7a7db5333 [Leader]" - testing.go:149: server-389af727-720b-73fd-bc35-70b7a7db5333: 2024-09-03T21:34:33.813Z [INFO] server-389af727-720b-73fd-bc35-70b7a7db5333: entering leader state: leader="Node at 389af727-720b-73fd-bc35-70b7a7db5333 [Leader]" +testLoggerAdapter verbose: server-fb46439f-cfb6-a2c1-77e9-dd5a747192ec: 2023-08-02T15:13:02.885Z [INFO] server-fb46439f-cfb6-a2c1-77e9-dd5a747192ec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fb46439f-cfb6-a2c1-77e9-dd5a747192ec Address:fb46439f-cfb6-a2c1-77e9-dd5a747192ec}]" + testing.go:149: server-fb46439f-cfb6-a2c1-77e9-dd5a747192ec: 2023-08-02T15:13:02.885Z [INFO] server-fb46439f-cfb6-a2c1-77e9-dd5a747192ec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fb46439f-cfb6-a2c1-77e9-dd5a747192ec Address:fb46439f-cfb6-a2c1-77e9-dd5a747192ec}]" +testLoggerAdapter verbose: server-fb46439f-cfb6-a2c1-77e9-dd5a747192ec: 2023-08-02T15:13:02.885Z [INFO] server-fb46439f-cfb6-a2c1-77e9-dd5a747192ec: entering leader state: leader="Node at fb46439f-cfb6-a2c1-77e9-dd5a747192ec [Leader]" + testing.go:149: server-fb46439f-cfb6-a2c1-77e9-dd5a747192ec: 2023-08-02T15:13:02.885Z [INFO] server-fb46439f-cfb6-a2c1-77e9-dd5a747192ec: entering leader state: leader="Node at fb46439f-cfb6-a2c1-77e9-dd5a747192ec [Leader]" --- PASS: TestRaft_StartAsLeader (0.00s) === RUN TestRaft_NotifyCh -testLoggerAdapter verbose: server-0572e0e0-cc4f-7f4a-dc8d-dfdb3fcbceb7: 2024-09-03T21:34:33.814Z [INFO] server-0572e0e0-cc4f-7f4a-dc8d-dfdb3fcbceb7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0572e0e0-cc4f-7f4a-dc8d-dfdb3fcbceb7 Address:0572e0e0-cc4f-7f4a-dc8d-dfdb3fcbceb7}]" - testing.go:149: server-0572e0e0-cc4f-7f4a-dc8d-dfdb3fcbceb7: 2024-09-03T21:34:33.814Z [INFO] server-0572e0e0-cc4f-7f4a-dc8d-dfdb3fcbceb7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0572e0e0-cc4f-7f4a-dc8d-dfdb3fcbceb7 Address:0572e0e0-cc4f-7f4a-dc8d-dfdb3fcbceb7}]" -testLoggerAdapter verbose: server-0572e0e0-cc4f-7f4a-dc8d-dfdb3fcbceb7: 2024-09-03T21:34:33.814Z [INFO] server-0572e0e0-cc4f-7f4a-dc8d-dfdb3fcbceb7: entering follower state: follower="Node at 0572e0e0-cc4f-7f4a-dc8d-dfdb3fcbceb7 [Follower]" leader= - testing.go:149: server-0572e0e0-cc4f-7f4a-dc8d-dfdb3fcbceb7: 2024-09-03T21:34:33.814Z [INFO] server-0572e0e0-cc4f-7f4a-dc8d-dfdb3fcbceb7: entering follower state: follower="Node at 0572e0e0-cc4f-7f4a-dc8d-dfdb3fcbceb7 [Follower]" leader= -testLoggerAdapter verbose: server-0572e0e0-cc4f-7f4a-dc8d-dfdb3fcbceb7: 2024-09-03T21:34:33.874Z [WARN] server-0572e0e0-cc4f-7f4a-dc8d-dfdb3fcbceb7: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-0572e0e0-cc4f-7f4a-dc8d-dfdb3fcbceb7: 2024-09-03T21:34:33.874Z [WARN] server-0572e0e0-cc4f-7f4a-dc8d-dfdb3fcbceb7: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-0572e0e0-cc4f-7f4a-dc8d-dfdb3fcbceb7: 2024-09-03T21:34:33.874Z [INFO] server-0572e0e0-cc4f-7f4a-dc8d-dfdb3fcbceb7: entering candidate state: node="Node at 0572e0e0-cc4f-7f4a-dc8d-dfdb3fcbceb7 [Candidate]" term=2 - testing.go:149: server-0572e0e0-cc4f-7f4a-dc8d-dfdb3fcbceb7: 2024-09-03T21:34:33.874Z [INFO] server-0572e0e0-cc4f-7f4a-dc8d-dfdb3fcbceb7: entering candidate state: node="Node at 0572e0e0-cc4f-7f4a-dc8d-dfdb3fcbceb7 [Candidate]" term=2 -testLoggerAdapter verbose: server-0572e0e0-cc4f-7f4a-dc8d-dfdb3fcbceb7: 2024-09-03T21:34:33.874Z [INFO] server-0572e0e0-cc4f-7f4a-dc8d-dfdb3fcbceb7: election won: tally=1 - testing.go:149: server-0572e0e0-cc4f-7f4a-dc8d-dfdb3fcbceb7: 2024-09-03T21:34:33.874Z [INFO] server-0572e0e0-cc4f-7f4a-dc8d-dfdb3fcbceb7: election won: tally=1 -testLoggerAdapter verbose: server-0572e0e0-cc4f-7f4a-dc8d-dfdb3fcbceb7: 2024-09-03T21:34:33.874Z [INFO] server-0572e0e0-cc4f-7f4a-dc8d-dfdb3fcbceb7: entering leader state: leader="Node at 0572e0e0-cc4f-7f4a-dc8d-dfdb3fcbceb7 [Leader]" - testing.go:149: server-0572e0e0-cc4f-7f4a-dc8d-dfdb3fcbceb7: 2024-09-03T21:34:33.874Z [INFO] server-0572e0e0-cc4f-7f4a-dc8d-dfdb3fcbceb7: entering leader state: leader="Node at 0572e0e0-cc4f-7f4a-dc8d-dfdb3fcbceb7 [Leader]" ---- PASS: TestRaft_NotifyCh (0.06s) +testLoggerAdapter verbose: server-03feaa6a-da93-368d-6f1c-f91e4480848d: 2023-08-02T15:13:02.886Z [INFO] server-03feaa6a-da93-368d-6f1c-f91e4480848d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-03feaa6a-da93-368d-6f1c-f91e4480848d Address:03feaa6a-da93-368d-6f1c-f91e4480848d}]" + testing.go:149: server-03feaa6a-da93-368d-6f1c-f91e4480848d: 2023-08-02T15:13:02.886Z [INFO] server-03feaa6a-da93-368d-6f1c-f91e4480848d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-03feaa6a-da93-368d-6f1c-f91e4480848d Address:03feaa6a-da93-368d-6f1c-f91e4480848d}]" +testLoggerAdapter verbose: server-03feaa6a-da93-368d-6f1c-f91e4480848d: 2023-08-02T15:13:02.886Z [INFO] server-03feaa6a-da93-368d-6f1c-f91e4480848d: entering follower state: follower="Node at 03feaa6a-da93-368d-6f1c-f91e4480848d [Follower]" leader= + testing.go:149: server-03feaa6a-da93-368d-6f1c-f91e4480848d: 2023-08-02T15:13:02.886Z [INFO] server-03feaa6a-da93-368d-6f1c-f91e4480848d: entering follower state: follower="Node at 03feaa6a-da93-368d-6f1c-f91e4480848d [Follower]" leader= +testLoggerAdapter verbose: server-03feaa6a-da93-368d-6f1c-f91e4480848d: 2023-08-02T15:13:02.971Z [WARN] server-03feaa6a-da93-368d-6f1c-f91e4480848d: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-03feaa6a-da93-368d-6f1c-f91e4480848d: 2023-08-02T15:13:02.971Z [WARN] server-03feaa6a-da93-368d-6f1c-f91e4480848d: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-03feaa6a-da93-368d-6f1c-f91e4480848d: 2023-08-02T15:13:02.971Z [INFO] server-03feaa6a-da93-368d-6f1c-f91e4480848d: entering candidate state: node="Node at 03feaa6a-da93-368d-6f1c-f91e4480848d [Candidate]" term=2 + testing.go:149: server-03feaa6a-da93-368d-6f1c-f91e4480848d: 2023-08-02T15:13:02.971Z [INFO] server-03feaa6a-da93-368d-6f1c-f91e4480848d: entering candidate state: node="Node at 03feaa6a-da93-368d-6f1c-f91e4480848d [Candidate]" term=2 +testLoggerAdapter verbose: server-03feaa6a-da93-368d-6f1c-f91e4480848d: 2023-08-02T15:13:02.971Z [INFO] server-03feaa6a-da93-368d-6f1c-f91e4480848d: election won: tally=1 + testing.go:149: server-03feaa6a-da93-368d-6f1c-f91e4480848d: 2023-08-02T15:13:02.971Z [INFO] server-03feaa6a-da93-368d-6f1c-f91e4480848d: election won: tally=1 +testLoggerAdapter verbose: server-03feaa6a-da93-368d-6f1c-f91e4480848d: 2023-08-02T15:13:02.971Z [INFO] server-03feaa6a-da93-368d-6f1c-f91e4480848d: entering leader state: leader="Node at 03feaa6a-da93-368d-6f1c-f91e4480848d [Leader]" + testing.go:149: server-03feaa6a-da93-368d-6f1c-f91e4480848d: 2023-08-02T15:13:02.971Z [INFO] server-03feaa6a-da93-368d-6f1c-f91e4480848d: entering leader state: leader="Node at 03feaa6a-da93-368d-6f1c-f91e4480848d [Leader]" +--- PASS: TestRaft_NotifyCh (0.09s) === RUN TestRaft_Voting -testLoggerAdapter verbose: server-54af44b7-abfc-5006-8895-08b18152f039: 2024-09-03T21:34:33.875Z [INFO] server-54af44b7-abfc-5006-8895-08b18152f039: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-54af44b7-abfc-5006-8895-08b18152f039 Address:54af44b7-abfc-5006-8895-08b18152f039} {Suffrage:Voter ID:server-09ef5e60-0e37-3720-4638-6e1124f0513e Address:09ef5e60-0e37-3720-4638-6e1124f0513e} {Suffrage:Voter ID:server-ee869883-a18d-9fc7-dda8-ba64e87063d6 Address:ee869883-a18d-9fc7-dda8-ba64e87063d6}]" - testing.go:149: server-54af44b7-abfc-5006-8895-08b18152f039: 2024-09-03T21:34:33.875Z [INFO] server-54af44b7-abfc-5006-8895-08b18152f039: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-54af44b7-abfc-5006-8895-08b18152f039 Address:54af44b7-abfc-5006-8895-08b18152f039} {Suffrage:Voter ID:server-09ef5e60-0e37-3720-4638-6e1124f0513e Address:09ef5e60-0e37-3720-4638-6e1124f0513e} {Suffrage:Voter ID:server-ee869883-a18d-9fc7-dda8-ba64e87063d6 Address:ee869883-a18d-9fc7-dda8-ba64e87063d6}]" -testLoggerAdapter verbose: server-09ef5e60-0e37-3720-4638-6e1124f0513e: 2024-09-03T21:34:33.875Z [INFO] server-09ef5e60-0e37-3720-4638-6e1124f0513e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-54af44b7-abfc-5006-8895-08b18152f039 Address:54af44b7-abfc-5006-8895-08b18152f039} {Suffrage:Voter ID:server-09ef5e60-0e37-3720-4638-6e1124f0513e Address:09ef5e60-0e37-3720-4638-6e1124f0513e} {Suffrage:Voter ID:server-ee869883-a18d-9fc7-dda8-ba64e87063d6 Address:ee869883-a18d-9fc7-dda8-ba64e87063d6}]" - testing.go:149: server-09ef5e60-0e37-3720-4638-6e1124f0513e: 2024-09-03T21:34:33.875Z [INFO] server-09ef5e60-0e37-3720-4638-6e1124f0513e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-54af44b7-abfc-5006-8895-08b18152f039 Address:54af44b7-abfc-5006-8895-08b18152f039} {Suffrage:Voter ID:server-09ef5e60-0e37-3720-4638-6e1124f0513e Address:09ef5e60-0e37-3720-4638-6e1124f0513e} {Suffrage:Voter ID:server-ee869883-a18d-9fc7-dda8-ba64e87063d6 Address:ee869883-a18d-9fc7-dda8-ba64e87063d6}]" -testLoggerAdapter verbose: server-ee869883-a18d-9fc7-dda8-ba64e87063d6: 2024-09-03T21:34:33.875Z [INFO] server-ee869883-a18d-9fc7-dda8-ba64e87063d6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-54af44b7-abfc-5006-8895-08b18152f039 Address:54af44b7-abfc-5006-8895-08b18152f039} {Suffrage:Voter ID:server-09ef5e60-0e37-3720-4638-6e1124f0513e Address:09ef5e60-0e37-3720-4638-6e1124f0513e} {Suffrage:Voter ID:server-ee869883-a18d-9fc7-dda8-ba64e87063d6 Address:ee869883-a18d-9fc7-dda8-ba64e87063d6}]" - testing.go:149: server-ee869883-a18d-9fc7-dda8-ba64e87063d6: 2024-09-03T21:34:33.875Z [INFO] server-ee869883-a18d-9fc7-dda8-ba64e87063d6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-54af44b7-abfc-5006-8895-08b18152f039 Address:54af44b7-abfc-5006-8895-08b18152f039} {Suffrage:Voter ID:server-09ef5e60-0e37-3720-4638-6e1124f0513e Address:09ef5e60-0e37-3720-4638-6e1124f0513e} {Suffrage:Voter ID:server-ee869883-a18d-9fc7-dda8-ba64e87063d6 Address:ee869883-a18d-9fc7-dda8-ba64e87063d6}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:33.875Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-09-03T21:34:33.875Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-54af44b7-abfc-5006-8895-08b18152f039: 2024-09-03T21:34:33.875Z [INFO] server-54af44b7-abfc-5006-8895-08b18152f039: entering follower state: follower="Node at 54af44b7-abfc-5006-8895-08b18152f039 [Follower]" leader= - testing.go:149: server-54af44b7-abfc-5006-8895-08b18152f039: 2024-09-03T21:34:33.875Z [INFO] server-54af44b7-abfc-5006-8895-08b18152f039: entering follower state: follower="Node at 54af44b7-abfc-5006-8895-08b18152f039 [Follower]" leader= -testLoggerAdapter verbose: server-09ef5e60-0e37-3720-4638-6e1124f0513e: 2024-09-03T21:34:33.875Z [INFO] server-09ef5e60-0e37-3720-4638-6e1124f0513e: entering follower state: follower="Node at 09ef5e60-0e37-3720-4638-6e1124f0513e [Follower]" leader= - testing.go:149: server-09ef5e60-0e37-3720-4638-6e1124f0513e: 2024-09-03T21:34:33.875Z [INFO] server-09ef5e60-0e37-3720-4638-6e1124f0513e: entering follower state: follower="Node at 09ef5e60-0e37-3720-4638-6e1124f0513e [Follower]" leader= -testLoggerAdapter verbose: server-ee869883-a18d-9fc7-dda8-ba64e87063d6: 2024-09-03T21:34:33.875Z [INFO] server-ee869883-a18d-9fc7-dda8-ba64e87063d6: entering follower state: follower="Node at ee869883-a18d-9fc7-dda8-ba64e87063d6 [Follower]" leader= - testing.go:149: server-ee869883-a18d-9fc7-dda8-ba64e87063d6: 2024-09-03T21:34:33.875Z [INFO] server-ee869883-a18d-9fc7-dda8-ba64e87063d6: entering follower state: follower="Node at ee869883-a18d-9fc7-dda8-ba64e87063d6 [Follower]" leader= -testLoggerAdapter verbose: server-ee869883-a18d-9fc7-dda8-ba64e87063d6: 2024-09-03T21:34:33.941Z [WARN] server-ee869883-a18d-9fc7-dda8-ba64e87063d6: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-ee869883-a18d-9fc7-dda8-ba64e87063d6: 2024-09-03T21:34:33.941Z [WARN] server-ee869883-a18d-9fc7-dda8-ba64e87063d6: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-ee869883-a18d-9fc7-dda8-ba64e87063d6: 2024-09-03T21:34:33.942Z [INFO] server-ee869883-a18d-9fc7-dda8-ba64e87063d6: entering candidate state: node="Node at ee869883-a18d-9fc7-dda8-ba64e87063d6 [Candidate]" term=2 - testing.go:149: server-ee869883-a18d-9fc7-dda8-ba64e87063d6: 2024-09-03T21:34:33.942Z [INFO] server-ee869883-a18d-9fc7-dda8-ba64e87063d6: entering candidate state: node="Node at ee869883-a18d-9fc7-dda8-ba64e87063d6 [Candidate]" term=2 -testLoggerAdapter verbose: server-ee869883-a18d-9fc7-dda8-ba64e87063d6: 2024-09-03T21:34:33.942Z [INFO] server-ee869883-a18d-9fc7-dda8-ba64e87063d6: election won: tally=2 - testing.go:149: server-ee869883-a18d-9fc7-dda8-ba64e87063d6: 2024-09-03T21:34:33.942Z [INFO] server-ee869883-a18d-9fc7-dda8-ba64e87063d6: election won: tally=2 -testLoggerAdapter verbose: server-ee869883-a18d-9fc7-dda8-ba64e87063d6: 2024-09-03T21:34:33.942Z [INFO] server-ee869883-a18d-9fc7-dda8-ba64e87063d6: entering leader state: leader="Node at ee869883-a18d-9fc7-dda8-ba64e87063d6 [Leader]" - testing.go:149: server-ee869883-a18d-9fc7-dda8-ba64e87063d6: 2024-09-03T21:34:33.942Z [INFO] server-ee869883-a18d-9fc7-dda8-ba64e87063d6: entering leader state: leader="Node at ee869883-a18d-9fc7-dda8-ba64e87063d6 [Leader]" -testLoggerAdapter verbose: server-ee869883-a18d-9fc7-dda8-ba64e87063d6: 2024-09-03T21:34:33.942Z [INFO] server-ee869883-a18d-9fc7-dda8-ba64e87063d6: added peer, starting replication: peer=server-54af44b7-abfc-5006-8895-08b18152f039 - testing.go:149: server-ee869883-a18d-9fc7-dda8-ba64e87063d6: 2024-09-03T21:34:33.942Z [INFO] server-ee869883-a18d-9fc7-dda8-ba64e87063d6: added peer, starting replication: peer=server-54af44b7-abfc-5006-8895-08b18152f039 -testLoggerAdapter verbose: server-ee869883-a18d-9fc7-dda8-ba64e87063d6: 2024-09-03T21:34:33.942Z [INFO] server-ee869883-a18d-9fc7-dda8-ba64e87063d6: added peer, starting replication: peer=server-09ef5e60-0e37-3720-4638-6e1124f0513e - testing.go:149: server-ee869883-a18d-9fc7-dda8-ba64e87063d6: 2024-09-03T21:34:33.942Z [INFO] server-ee869883-a18d-9fc7-dda8-ba64e87063d6: added peer, starting replication: peer=server-09ef5e60-0e37-3720-4638-6e1124f0513e -testLoggerAdapter verbose: server-ee869883-a18d-9fc7-dda8-ba64e87063d6: 2024-09-03T21:34:33.942Z [INFO] server-ee869883-a18d-9fc7-dda8-ba64e87063d6: pipelining replication: peer="{Voter server-09ef5e60-0e37-3720-4638-6e1124f0513e 09ef5e60-0e37-3720-4638-6e1124f0513e}" - testing.go:149: server-ee869883-a18d-9fc7-dda8-ba64e87063d6: 2024-09-03T21:34:33.942Z [INFO] server-ee869883-a18d-9fc7-dda8-ba64e87063d6: pipelining replication: peer="{Voter server-09ef5e60-0e37-3720-4638-6e1124f0513e 09ef5e60-0e37-3720-4638-6e1124f0513e}" -testLoggerAdapter verbose: server-ee869883-a18d-9fc7-dda8-ba64e87063d6: 2024-09-03T21:34:33.942Z [INFO] server-ee869883-a18d-9fc7-dda8-ba64e87063d6: pipelining replication: peer="{Voter server-54af44b7-abfc-5006-8895-08b18152f039 54af44b7-abfc-5006-8895-08b18152f039}" - testing.go:149: server-ee869883-a18d-9fc7-dda8-ba64e87063d6: 2024-09-03T21:34:33.942Z [INFO] server-ee869883-a18d-9fc7-dda8-ba64e87063d6: pipelining replication: peer="{Voter server-54af44b7-abfc-5006-8895-08b18152f039 54af44b7-abfc-5006-8895-08b18152f039}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:34.047Z [INFO] stable state for Follower reached at 2024-09-03 21:34:33.942639027 +0000 UTC m=+14.989296179 (2 nodes), 67.259213ms from start of poll, 67.655923ms from cluster start. Timeout at 2024-09-03 21:34:34.047731992 +0000 UTC m=+15.094389124, 105.092945ms after stability - testing.go:149: cluster: 2024-09-03T21:34:34.047Z [INFO] stable state for Follower reached at 2024-09-03 21:34:33.942639027 +0000 UTC m=+14.989296179 (2 nodes), 67.259213ms from start of poll, 67.655923ms from cluster start. Timeout at 2024-09-03 21:34:34.047731992 +0000 UTC m=+15.094389124, 105.092945ms after stability -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:34.047Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:34.047Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:34.152Z [INFO] stable state for Leader reached at 2024-09-03 21:34:34.047876421 +0000 UTC m=+15.094533551 (1 nodes), 951ns from start of poll, 172.893295ms from cluster start. Timeout at 2024-09-03 21:34:34.15292965 +0000 UTC m=+15.199586781, 105.05323ms after stability - testing.go:149: cluster: 2024-09-03T21:34:34.152Z [INFO] stable state for Leader reached at 2024-09-03 21:34:34.047876421 +0000 UTC m=+15.094533551 (1 nodes), 951ns from start of poll, 172.893295ms from cluster start. Timeout at 2024-09-03 21:34:34.15292965 +0000 UTC m=+15.199586781, 105.05323ms after stability -testLoggerAdapter verbose: server-09ef5e60-0e37-3720-4638-6e1124f0513e: 2024-09-03T21:34:34.153Z [WARN] server-09ef5e60-0e37-3720-4638-6e1124f0513e: rejecting vote request since we have a leader: from=54af44b7-abfc-5006-8895-08b18152f039 leader=ee869883-a18d-9fc7-dda8-ba64e87063d6 - testing.go:149: server-09ef5e60-0e37-3720-4638-6e1124f0513e: 2024-09-03T21:34:34.153Z [WARN] server-09ef5e60-0e37-3720-4638-6e1124f0513e: rejecting vote request since we have a leader: from=54af44b7-abfc-5006-8895-08b18152f039 leader=ee869883-a18d-9fc7-dda8-ba64e87063d6 -testLoggerAdapter verbose: server-ee869883-a18d-9fc7-dda8-ba64e87063d6: 2024-09-03T21:34:34.153Z [INFO] server-ee869883-a18d-9fc7-dda8-ba64e87063d6: aborting pipeline replication: peer="{Voter server-09ef5e60-0e37-3720-4638-6e1124f0513e 09ef5e60-0e37-3720-4638-6e1124f0513e}" - testing.go:149: server-ee869883-a18d-9fc7-dda8-ba64e87063d6: 2024-09-03T21:34:34.153Z [INFO] server-ee869883-a18d-9fc7-dda8-ba64e87063d6: aborting pipeline replication: peer="{Voter server-09ef5e60-0e37-3720-4638-6e1124f0513e 09ef5e60-0e37-3720-4638-6e1124f0513e}" -testLoggerAdapter verbose: server-ee869883-a18d-9fc7-dda8-ba64e87063d6: 2024-09-03T21:34:34.153Z [INFO] server-ee869883-a18d-9fc7-dda8-ba64e87063d6: aborting pipeline replication: peer="{Voter server-54af44b7-abfc-5006-8895-08b18152f039 54af44b7-abfc-5006-8895-08b18152f039}" - testing.go:149: server-ee869883-a18d-9fc7-dda8-ba64e87063d6: 2024-09-03T21:34:34.153Z [INFO] server-ee869883-a18d-9fc7-dda8-ba64e87063d6: aborting pipeline replication: peer="{Voter server-54af44b7-abfc-5006-8895-08b18152f039 54af44b7-abfc-5006-8895-08b18152f039}" ---- PASS: TestRaft_Voting (0.28s) +testLoggerAdapter verbose: server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44: 2023-08-02T15:13:02.972Z [INFO] server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44 Address:84a510be-8c77-a26a-a8ab-1b5f6cc96c44} {Suffrage:Voter ID:server-4315480e-881b-0df7-4949-932968f11d4e Address:4315480e-881b-0df7-4949-932968f11d4e} {Suffrage:Voter ID:server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535 Address:6a9bcf8f-e132-323e-d5dc-c3e34ddf8535}]" + testing.go:149: server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44: 2023-08-02T15:13:02.972Z [INFO] server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44 Address:84a510be-8c77-a26a-a8ab-1b5f6cc96c44} {Suffrage:Voter ID:server-4315480e-881b-0df7-4949-932968f11d4e Address:4315480e-881b-0df7-4949-932968f11d4e} {Suffrage:Voter ID:server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535 Address:6a9bcf8f-e132-323e-d5dc-c3e34ddf8535}]" +testLoggerAdapter verbose: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:02.972Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44 Address:84a510be-8c77-a26a-a8ab-1b5f6cc96c44} {Suffrage:Voter ID:server-4315480e-881b-0df7-4949-932968f11d4e Address:4315480e-881b-0df7-4949-932968f11d4e} {Suffrage:Voter ID:server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535 Address:6a9bcf8f-e132-323e-d5dc-c3e34ddf8535}]" + testing.go:149: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:02.972Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44 Address:84a510be-8c77-a26a-a8ab-1b5f6cc96c44} {Suffrage:Voter ID:server-4315480e-881b-0df7-4949-932968f11d4e Address:4315480e-881b-0df7-4949-932968f11d4e} {Suffrage:Voter ID:server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535 Address:6a9bcf8f-e132-323e-d5dc-c3e34ddf8535}]" +testLoggerAdapter verbose: server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535: 2023-08-02T15:13:02.972Z [INFO] server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44 Address:84a510be-8c77-a26a-a8ab-1b5f6cc96c44} {Suffrage:Voter ID:server-4315480e-881b-0df7-4949-932968f11d4e Address:4315480e-881b-0df7-4949-932968f11d4e} {Suffrage:Voter ID:server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535 Address:6a9bcf8f-e132-323e-d5dc-c3e34ddf8535}]" + testing.go:149: server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535: 2023-08-02T15:13:02.972Z [INFO] server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44 Address:84a510be-8c77-a26a-a8ab-1b5f6cc96c44} {Suffrage:Voter ID:server-4315480e-881b-0df7-4949-932968f11d4e Address:4315480e-881b-0df7-4949-932968f11d4e} {Suffrage:Voter ID:server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535 Address:6a9bcf8f-e132-323e-d5dc-c3e34ddf8535}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:02.972Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-08-02T15:13:02.972Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44: 2023-08-02T15:13:02.972Z [INFO] server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44: entering follower state: follower="Node at 84a510be-8c77-a26a-a8ab-1b5f6cc96c44 [Follower]" leader= + testing.go:149: server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44: 2023-08-02T15:13:02.972Z [INFO] server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44: entering follower state: follower="Node at 84a510be-8c77-a26a-a8ab-1b5f6cc96c44 [Follower]" leader= +testLoggerAdapter verbose: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:02.972Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: entering follower state: follower="Node at 4315480e-881b-0df7-4949-932968f11d4e [Follower]" leader= + testing.go:149: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:02.972Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: entering follower state: follower="Node at 4315480e-881b-0df7-4949-932968f11d4e [Follower]" leader= +testLoggerAdapter verbose: server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535: 2023-08-02T15:13:02.972Z [INFO] server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535: entering follower state: follower="Node at 6a9bcf8f-e132-323e-d5dc-c3e34ddf8535 [Follower]" leader= + testing.go:149: server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535: 2023-08-02T15:13:02.972Z [INFO] server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535: entering follower state: follower="Node at 6a9bcf8f-e132-323e-d5dc-c3e34ddf8535 [Follower]" leader= +testLoggerAdapter verbose: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.023Z [WARN] server-4315480e-881b-0df7-4949-932968f11d4e: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.023Z [WARN] server-4315480e-881b-0df7-4949-932968f11d4e: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.023Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: entering candidate state: node="Node at 4315480e-881b-0df7-4949-932968f11d4e [Candidate]" term=2 + testing.go:149: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.023Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: entering candidate state: node="Node at 4315480e-881b-0df7-4949-932968f11d4e [Candidate]" term=2 +testLoggerAdapter verbose: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.024Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: election won: tally=2 + testing.go:149: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.024Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: election won: tally=2 +testLoggerAdapter verbose: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.024Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: entering leader state: leader="Node at 4315480e-881b-0df7-4949-932968f11d4e [Leader]" + testing.go:149: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.024Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: entering leader state: leader="Node at 4315480e-881b-0df7-4949-932968f11d4e [Leader]" +testLoggerAdapter verbose: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.024Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: added peer, starting replication: peer=server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44 + testing.go:149: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.024Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: added peer, starting replication: peer=server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44 +testLoggerAdapter verbose: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.024Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: added peer, starting replication: peer=server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535 + testing.go:149: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.024Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: added peer, starting replication: peer=server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535 +testLoggerAdapter verbose: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.024Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: pipelining replication: peer="{Voter server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44 84a510be-8c77-a26a-a8ab-1b5f6cc96c44}" + testing.go:149: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.024Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: pipelining replication: peer="{Voter server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44 84a510be-8c77-a26a-a8ab-1b5f6cc96c44}" +testLoggerAdapter verbose: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.024Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: pipelining replication: peer="{Voter server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535 6a9bcf8f-e132-323e-d5dc-c3e34ddf8535}" + testing.go:149: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.024Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: pipelining replication: peer="{Voter server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535 6a9bcf8f-e132-323e-d5dc-c3e34ddf8535}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:03.129Z [INFO] stable state for Follower reached at 2023-08-02 15:13:03.024198385 +0000 UTC m=+15.120134719 (2 nodes), 51.358472ms from start of poll, 51.762294ms from cluster start. Timeout at 2023-08-02 15:13:03.129289345 +0000 UTC m=+15.225225678, 105.090959ms after stability + testing.go:149: cluster: 2023-08-02T15:13:03.129Z [INFO] stable state for Follower reached at 2023-08-02 15:13:03.024198385 +0000 UTC m=+15.120134719 (2 nodes), 51.358472ms from start of poll, 51.762294ms from cluster start. Timeout at 2023-08-02 15:13:03.129289345 +0000 UTC m=+15.225225678, 105.090959ms after stability +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:03.129Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:13:03.129Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:03.234Z [INFO] stable state for Leader reached at 2023-08-02 15:13:03.129470464 +0000 UTC m=+15.225406790 (1 nodes), 1.189µs from start of poll, 157.034365ms from cluster start. Timeout at 2023-08-02 15:13:03.234593345 +0000 UTC m=+15.330529680, 105.12289ms after stability + testing.go:149: cluster: 2023-08-02T15:13:03.234Z [INFO] stable state for Leader reached at 2023-08-02 15:13:03.129470464 +0000 UTC m=+15.225406790 (1 nodes), 1.189µs from start of poll, 157.034365ms from cluster start. Timeout at 2023-08-02 15:13:03.234593345 +0000 UTC m=+15.330529680, 105.12289ms after stability +testLoggerAdapter verbose: server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535: 2023-08-02T15:13:03.234Z [WARN] server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535: rejecting vote request since we have a leader: from=84a510be-8c77-a26a-a8ab-1b5f6cc96c44 leader=4315480e-881b-0df7-4949-932968f11d4e + testing.go:149: server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535: 2023-08-02T15:13:03.234Z [WARN] server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535: rejecting vote request since we have a leader: from=84a510be-8c77-a26a-a8ab-1b5f6cc96c44 leader=4315480e-881b-0df7-4949-932968f11d4e +testLoggerAdapter verbose: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.234Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: aborting pipeline replication: peer="{Voter server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535 6a9bcf8f-e132-323e-d5dc-c3e34ddf8535}" + testing.go:149: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.234Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: aborting pipeline replication: peer="{Voter server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535 6a9bcf8f-e132-323e-d5dc-c3e34ddf8535}" +testLoggerAdapter verbose: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.234Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: aborting pipeline replication: peer="{Voter server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44 84a510be-8c77-a26a-a8ab-1b5f6cc96c44}" + testing.go:149: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.234Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: aborting pipeline replication: peer="{Voter server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44 84a510be-8c77-a26a-a8ab-1b5f6cc96c44}" +--- PASS: TestRaft_Voting (0.26s) === RUN TestRaft_ProtocolVersion_RejectRPC -testLoggerAdapter verbose: server-1ef10b59-f14b-55ac-3fd2-c2c3cb08d7dc: 2024-09-03T21:34:34.153Z [INFO] server-1ef10b59-f14b-55ac-3fd2-c2c3cb08d7dc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ef10b59-f14b-55ac-3fd2-c2c3cb08d7dc Address:1ef10b59-f14b-55ac-3fd2-c2c3cb08d7dc} {Suffrage:Voter ID:server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10 Address:149f50b9-7b02-5ed7-e32f-fa1d64cc1e10} {Suffrage:Voter ID:server-184b1691-7353-cd0d-639c-a4c1733a5acf Address:184b1691-7353-cd0d-639c-a4c1733a5acf}]" - testing.go:149: server-1ef10b59-f14b-55ac-3fd2-c2c3cb08d7dc: 2024-09-03T21:34:34.153Z [INFO] server-1ef10b59-f14b-55ac-3fd2-c2c3cb08d7dc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ef10b59-f14b-55ac-3fd2-c2c3cb08d7dc Address:1ef10b59-f14b-55ac-3fd2-c2c3cb08d7dc} {Suffrage:Voter ID:server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10 Address:149f50b9-7b02-5ed7-e32f-fa1d64cc1e10} {Suffrage:Voter ID:server-184b1691-7353-cd0d-639c-a4c1733a5acf Address:184b1691-7353-cd0d-639c-a4c1733a5acf}]" -testLoggerAdapter verbose: server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: 2024-09-03T21:34:34.154Z [INFO] server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ef10b59-f14b-55ac-3fd2-c2c3cb08d7dc Address:1ef10b59-f14b-55ac-3fd2-c2c3cb08d7dc} {Suffrage:Voter ID:server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10 Address:149f50b9-7b02-5ed7-e32f-fa1d64cc1e10} {Suffrage:Voter ID:server-184b1691-7353-cd0d-639c-a4c1733a5acf Address:184b1691-7353-cd0d-639c-a4c1733a5acf}]" - testing.go:149: server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: 2024-09-03T21:34:34.154Z [INFO] server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ef10b59-f14b-55ac-3fd2-c2c3cb08d7dc Address:1ef10b59-f14b-55ac-3fd2-c2c3cb08d7dc} {Suffrage:Voter ID:server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10 Address:149f50b9-7b02-5ed7-e32f-fa1d64cc1e10} {Suffrage:Voter ID:server-184b1691-7353-cd0d-639c-a4c1733a5acf Address:184b1691-7353-cd0d-639c-a4c1733a5acf}]" -testLoggerAdapter verbose: server-184b1691-7353-cd0d-639c-a4c1733a5acf: 2024-09-03T21:34:34.154Z [INFO] server-184b1691-7353-cd0d-639c-a4c1733a5acf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ef10b59-f14b-55ac-3fd2-c2c3cb08d7dc Address:1ef10b59-f14b-55ac-3fd2-c2c3cb08d7dc} {Suffrage:Voter ID:server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10 Address:149f50b9-7b02-5ed7-e32f-fa1d64cc1e10} {Suffrage:Voter ID:server-184b1691-7353-cd0d-639c-a4c1733a5acf Address:184b1691-7353-cd0d-639c-a4c1733a5acf}]" - testing.go:149: server-184b1691-7353-cd0d-639c-a4c1733a5acf: 2024-09-03T21:34:34.154Z [INFO] server-184b1691-7353-cd0d-639c-a4c1733a5acf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ef10b59-f14b-55ac-3fd2-c2c3cb08d7dc Address:1ef10b59-f14b-55ac-3fd2-c2c3cb08d7dc} {Suffrage:Voter ID:server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10 Address:149f50b9-7b02-5ed7-e32f-fa1d64cc1e10} {Suffrage:Voter ID:server-184b1691-7353-cd0d-639c-a4c1733a5acf Address:184b1691-7353-cd0d-639c-a4c1733a5acf}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:34.154Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-09-03T21:34:34.154Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-1ef10b59-f14b-55ac-3fd2-c2c3cb08d7dc: 2024-09-03T21:34:34.154Z [INFO] server-1ef10b59-f14b-55ac-3fd2-c2c3cb08d7dc: entering follower state: follower="Node at 1ef10b59-f14b-55ac-3fd2-c2c3cb08d7dc [Follower]" leader= - testing.go:149: server-1ef10b59-f14b-55ac-3fd2-c2c3cb08d7dc: 2024-09-03T21:34:34.154Z [INFO] server-1ef10b59-f14b-55ac-3fd2-c2c3cb08d7dc: entering follower state: follower="Node at 1ef10b59-f14b-55ac-3fd2-c2c3cb08d7dc [Follower]" leader= -testLoggerAdapter verbose: server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: 2024-09-03T21:34:34.154Z [INFO] server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: entering follower state: follower="Node at 149f50b9-7b02-5ed7-e32f-fa1d64cc1e10 [Follower]" leader= - testing.go:149: server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: 2024-09-03T21:34:34.154Z [INFO] server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: entering follower state: follower="Node at 149f50b9-7b02-5ed7-e32f-fa1d64cc1e10 [Follower]" leader= -testLoggerAdapter verbose: server-184b1691-7353-cd0d-639c-a4c1733a5acf: 2024-09-03T21:34:34.154Z [INFO] server-184b1691-7353-cd0d-639c-a4c1733a5acf: entering follower state: follower="Node at 184b1691-7353-cd0d-639c-a4c1733a5acf [Follower]" leader= - testing.go:149: server-184b1691-7353-cd0d-639c-a4c1733a5acf: 2024-09-03T21:34:34.154Z [INFO] server-184b1691-7353-cd0d-639c-a4c1733a5acf: entering follower state: follower="Node at 184b1691-7353-cd0d-639c-a4c1733a5acf [Follower]" leader= -testLoggerAdapter verbose: server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: 2024-09-03T21:34:34.214Z [WARN] server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: 2024-09-03T21:34:34.214Z [WARN] server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: 2024-09-03T21:34:34.214Z [INFO] server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: entering candidate state: node="Node at 149f50b9-7b02-5ed7-e32f-fa1d64cc1e10 [Candidate]" term=2 - testing.go:149: server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: 2024-09-03T21:34:34.214Z [INFO] server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: entering candidate state: node="Node at 149f50b9-7b02-5ed7-e32f-fa1d64cc1e10 [Candidate]" term=2 -testLoggerAdapter verbose: server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: 2024-09-03T21:34:34.215Z [INFO] server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: election won: tally=2 - testing.go:149: server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: 2024-09-03T21:34:34.215Z [INFO] server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: election won: tally=2 -testLoggerAdapter verbose: server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: 2024-09-03T21:34:34.215Z [INFO] server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: entering leader state: leader="Node at 149f50b9-7b02-5ed7-e32f-fa1d64cc1e10 [Leader]" - testing.go:149: server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: 2024-09-03T21:34:34.215Z [INFO] server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: entering leader state: leader="Node at 149f50b9-7b02-5ed7-e32f-fa1d64cc1e10 [Leader]" -testLoggerAdapter verbose: server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: 2024-09-03T21:34:34.215Z [INFO] server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: added peer, starting replication: peer=server-1ef10b59-f14b-55ac-3fd2-c2c3cb08d7dc - testing.go:149: server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: 2024-09-03T21:34:34.215Z [INFO] server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: added peer, starting replication: peer=server-1ef10b59-f14b-55ac-3fd2-c2c3cb08d7dc -testLoggerAdapter verbose: server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: 2024-09-03T21:34:34.215Z [INFO] server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: added peer, starting replication: peer=server-184b1691-7353-cd0d-639c-a4c1733a5acf - testing.go:149: server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: 2024-09-03T21:34:34.215Z [INFO] server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: added peer, starting replication: peer=server-184b1691-7353-cd0d-639c-a4c1733a5acf -testLoggerAdapter verbose: server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: 2024-09-03T21:34:34.215Z [INFO] server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: pipelining replication: peer="{Voter server-184b1691-7353-cd0d-639c-a4c1733a5acf 184b1691-7353-cd0d-639c-a4c1733a5acf}" - testing.go:149: server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: 2024-09-03T21:34:34.215Z [INFO] server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: pipelining replication: peer="{Voter server-184b1691-7353-cd0d-639c-a4c1733a5acf 184b1691-7353-cd0d-639c-a4c1733a5acf}" -testLoggerAdapter verbose: server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: 2024-09-03T21:34:34.215Z [INFO] server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: pipelining replication: peer="{Voter server-1ef10b59-f14b-55ac-3fd2-c2c3cb08d7dc 1ef10b59-f14b-55ac-3fd2-c2c3cb08d7dc}" - testing.go:149: server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: 2024-09-03T21:34:34.215Z [INFO] server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: pipelining replication: peer="{Voter server-1ef10b59-f14b-55ac-3fd2-c2c3cb08d7dc 1ef10b59-f14b-55ac-3fd2-c2c3cb08d7dc}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:34.320Z [INFO] stable state for Follower reached at 2024-09-03 21:34:34.215242946 +0000 UTC m=+15.261900109 (2 nodes), 60.953681ms from start of poll, 61.375124ms from cluster start. Timeout at 2024-09-03 21:34:34.320323952 +0000 UTC m=+15.366981084, 105.080975ms after stability - testing.go:149: cluster: 2024-09-03T21:34:34.320Z [INFO] stable state for Follower reached at 2024-09-03 21:34:34.215242946 +0000 UTC m=+15.261900109 (2 nodes), 60.953681ms from start of poll, 61.375124ms from cluster start. Timeout at 2024-09-03 21:34:34.320323952 +0000 UTC m=+15.366981084, 105.080975ms after stability -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:34.320Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:34.320Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:34.425Z [INFO] stable state for Leader reached at 2024-09-03 21:34:34.320485479 +0000 UTC m=+15.367142612 (1 nodes), 1.029µs from start of poll, 166.617627ms from cluster start. Timeout at 2024-09-03 21:34:34.425586146 +0000 UTC m=+15.472243281, 105.100669ms after stability - testing.go:149: cluster: 2024-09-03T21:34:34.425Z [INFO] stable state for Leader reached at 2024-09-03 21:34:34.320485479 +0000 UTC m=+15.367142612 (1 nodes), 1.029µs from start of poll, 166.617627ms from cluster start. Timeout at 2024-09-03 21:34:34.425586146 +0000 UTC m=+15.472243281, 105.100669ms after stability -testLoggerAdapter verbose: server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: 2024-09-03T21:34:34.427Z [INFO] server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: aborting pipeline replication: peer="{Voter server-1ef10b59-f14b-55ac-3fd2-c2c3cb08d7dc 1ef10b59-f14b-55ac-3fd2-c2c3cb08d7dc}" - testing.go:149: server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: 2024-09-03T21:34:34.427Z [INFO] server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: aborting pipeline replication: peer="{Voter server-1ef10b59-f14b-55ac-3fd2-c2c3cb08d7dc 1ef10b59-f14b-55ac-3fd2-c2c3cb08d7dc}" -testLoggerAdapter verbose: server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: 2024-09-03T21:34:34.427Z [INFO] server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: aborting pipeline replication: peer="{Voter server-184b1691-7353-cd0d-639c-a4c1733a5acf 184b1691-7353-cd0d-639c-a4c1733a5acf}" - testing.go:149: server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: 2024-09-03T21:34:34.427Z [INFO] server-149f50b9-7b02-5ed7-e32f-fa1d64cc1e10: aborting pipeline replication: peer="{Voter server-184b1691-7353-cd0d-639c-a4c1733a5acf 184b1691-7353-cd0d-639c-a4c1733a5acf}" +testLoggerAdapter verbose: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.236Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68d0c667-b6c6-93be-7c92-5c727698721a Address:68d0c667-b6c6-93be-7c92-5c727698721a} {Suffrage:Voter ID:server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a Address:fb9428dc-383b-ae4d-0fb8-6c45b88e459a} {Suffrage:Voter ID:server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1 Address:3e9df4c1-56f7-e878-ed42-5ba6850bafb1}]" + testing.go:149: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.236Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68d0c667-b6c6-93be-7c92-5c727698721a Address:68d0c667-b6c6-93be-7c92-5c727698721a} {Suffrage:Voter ID:server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a Address:fb9428dc-383b-ae4d-0fb8-6c45b88e459a} {Suffrage:Voter ID:server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1 Address:3e9df4c1-56f7-e878-ed42-5ba6850bafb1}]" +testLoggerAdapter verbose: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.236Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: entering follower state: follower="Node at 68d0c667-b6c6-93be-7c92-5c727698721a [Follower]" leader= +testLoggerAdapter verbose: server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a: 2023-08-02T15:13:03.236Z [INFO] server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68d0c667-b6c6-93be-7c92-5c727698721a Address:68d0c667-b6c6-93be-7c92-5c727698721a} {Suffrage:Voter ID:server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a Address:fb9428dc-383b-ae4d-0fb8-6c45b88e459a} {Suffrage:Voter ID:server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1 Address:3e9df4c1-56f7-e878-ed42-5ba6850bafb1}]" + testing.go:149: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.236Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: entering follower state: follower="Node at 68d0c667-b6c6-93be-7c92-5c727698721a [Follower]" leader= + testing.go:149: server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a: 2023-08-02T15:13:03.236Z [INFO] server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68d0c667-b6c6-93be-7c92-5c727698721a Address:68d0c667-b6c6-93be-7c92-5c727698721a} {Suffrage:Voter ID:server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a Address:fb9428dc-383b-ae4d-0fb8-6c45b88e459a} {Suffrage:Voter ID:server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1 Address:3e9df4c1-56f7-e878-ed42-5ba6850bafb1}]" +testLoggerAdapter verbose: server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a: 2023-08-02T15:13:03.236Z [INFO] server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a: entering follower state: follower="Node at fb9428dc-383b-ae4d-0fb8-6c45b88e459a [Follower]" leader= + testing.go:149: server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a: 2023-08-02T15:13:03.236Z [INFO] server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a: entering follower state: follower="Node at fb9428dc-383b-ae4d-0fb8-6c45b88e459a [Follower]" leader= +testLoggerAdapter verbose: server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1: 2023-08-02T15:13:03.236Z [INFO] server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68d0c667-b6c6-93be-7c92-5c727698721a Address:68d0c667-b6c6-93be-7c92-5c727698721a} {Suffrage:Voter ID:server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a Address:fb9428dc-383b-ae4d-0fb8-6c45b88e459a} {Suffrage:Voter ID:server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1 Address:3e9df4c1-56f7-e878-ed42-5ba6850bafb1}]" + testing.go:149: server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1: 2023-08-02T15:13:03.236Z [INFO] server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68d0c667-b6c6-93be-7c92-5c727698721a Address:68d0c667-b6c6-93be-7c92-5c727698721a} {Suffrage:Voter ID:server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a Address:fb9428dc-383b-ae4d-0fb8-6c45b88e459a} {Suffrage:Voter ID:server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1 Address:3e9df4c1-56f7-e878-ed42-5ba6850bafb1}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:03.236Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-08-02T15:13:03.236Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1: 2023-08-02T15:13:03.236Z [INFO] server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1: entering follower state: follower="Node at 3e9df4c1-56f7-e878-ed42-5ba6850bafb1 [Follower]" leader= + testing.go:149: server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1: 2023-08-02T15:13:03.236Z [INFO] server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1: entering follower state: follower="Node at 3e9df4c1-56f7-e878-ed42-5ba6850bafb1 [Follower]" leader= +testLoggerAdapter verbose: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.291Z [WARN] server-68d0c667-b6c6-93be-7c92-5c727698721a: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.291Z [WARN] server-68d0c667-b6c6-93be-7c92-5c727698721a: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.291Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: entering candidate state: node="Node at 68d0c667-b6c6-93be-7c92-5c727698721a [Candidate]" term=2 + testing.go:149: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.291Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: entering candidate state: node="Node at 68d0c667-b6c6-93be-7c92-5c727698721a [Candidate]" term=2 +testLoggerAdapter verbose: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.291Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: election won: tally=2 + testing.go:149: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.291Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: election won: tally=2 +testLoggerAdapter verbose: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.291Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: entering leader state: leader="Node at 68d0c667-b6c6-93be-7c92-5c727698721a [Leader]" + testing.go:149: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.291Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: entering leader state: leader="Node at 68d0c667-b6c6-93be-7c92-5c727698721a [Leader]" +testLoggerAdapter verbose: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.291Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: added peer, starting replication: peer=server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a + testing.go:149: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.291Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: added peer, starting replication: peer=server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a +testLoggerAdapter verbose: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.291Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: added peer, starting replication: peer=server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1 + testing.go:149: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.291Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: added peer, starting replication: peer=server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1 +testLoggerAdapter verbose: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.291Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: pipelining replication: peer="{Voter server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1 3e9df4c1-56f7-e878-ed42-5ba6850bafb1}" + testing.go:149: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.291Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: pipelining replication: peer="{Voter server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1 3e9df4c1-56f7-e878-ed42-5ba6850bafb1}" +testLoggerAdapter verbose: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.291Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: pipelining replication: peer="{Voter server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a fb9428dc-383b-ae4d-0fb8-6c45b88e459a}" + testing.go:149: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.291Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: pipelining replication: peer="{Voter server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a fb9428dc-383b-ae4d-0fb8-6c45b88e459a}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:03.396Z [INFO] stable state for Follower reached at 2023-08-02 15:13:03.291704946 +0000 UTC m=+15.387641273 (2 nodes), 55.009122ms from start of poll, 55.796773ms from cluster start. Timeout at 2023-08-02 15:13:03.396725276 +0000 UTC m=+15.492661617, 105.020344ms after stability + testing.go:149: cluster: 2023-08-02T15:13:03.396Z [INFO] stable state for Follower reached at 2023-08-02 15:13:03.291704946 +0000 UTC m=+15.387641273 (2 nodes), 55.009122ms from start of poll, 55.796773ms from cluster start. Timeout at 2023-08-02 15:13:03.396725276 +0000 UTC m=+15.492661617, 105.020344ms after stability +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:03.396Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:13:03.396Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:03.502Z [INFO] stable state for Leader reached at 2023-08-02 15:13:03.396885163 +0000 UTC m=+15.492821491 (1 nodes), 1.322µs from start of poll, 160.976991ms from cluster start. Timeout at 2023-08-02 15:13:03.502566897 +0000 UTC m=+15.598503230, 105.681739ms after stability + testing.go:149: cluster: 2023-08-02T15:13:03.502Z [INFO] stable state for Leader reached at 2023-08-02 15:13:03.396885163 +0000 UTC m=+15.492821491 (1 nodes), 1.322µs from start of poll, 160.976991ms from cluster start. Timeout at 2023-08-02 15:13:03.502566897 +0000 UTC m=+15.598503230, 105.681739ms after stability +testLoggerAdapter verbose: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.502Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: aborting pipeline replication: peer="{Voter server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1 3e9df4c1-56f7-e878-ed42-5ba6850bafb1}" + testing.go:149: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.502Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: aborting pipeline replication: peer="{Voter server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1 3e9df4c1-56f7-e878-ed42-5ba6850bafb1}" +testLoggerAdapter verbose: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.502Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: aborting pipeline replication: peer="{Voter server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a fb9428dc-383b-ae4d-0fb8-6c45b88e459a}" + testing.go:149: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.502Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: aborting pipeline replication: peer="{Voter server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a fb9428dc-383b-ae4d-0fb8-6c45b88e459a}" --- PASS: TestRaft_ProtocolVersion_RejectRPC (0.27s) === RUN TestRaft_ProtocolVersion_Upgrade_1_2 -testLoggerAdapter verbose: 1073db2c-9668-25e9-3d8d-c519e9646df9: 2024-09-03T21:34:34.428Z [INFO] 1073db2c-9668-25e9-3d8d-c519e9646df9: initial configuration: index=1 servers="[{Suffrage:Voter ID:1073db2c-9668-25e9-3d8d-c519e9646df9 Address:1073db2c-9668-25e9-3d8d-c519e9646df9} {Suffrage:Voter ID:52c81689-fc69-5792-6f19-063f8f5e6bf7 Address:52c81689-fc69-5792-6f19-063f8f5e6bf7}]" - testing.go:149: 1073db2c-9668-25e9-3d8d-c519e9646df9: 2024-09-03T21:34:34.428Z [INFO] 1073db2c-9668-25e9-3d8d-c519e9646df9: initial configuration: index=1 servers="[{Suffrage:Voter ID:1073db2c-9668-25e9-3d8d-c519e9646df9 Address:1073db2c-9668-25e9-3d8d-c519e9646df9} {Suffrage:Voter ID:52c81689-fc69-5792-6f19-063f8f5e6bf7 Address:52c81689-fc69-5792-6f19-063f8f5e6bf7}]" -testLoggerAdapter verbose: 52c81689-fc69-5792-6f19-063f8f5e6bf7: 2024-09-03T21:34:34.429Z [INFO] 52c81689-fc69-5792-6f19-063f8f5e6bf7: initial configuration: index=1 servers="[{Suffrage:Voter ID:1073db2c-9668-25e9-3d8d-c519e9646df9 Address:1073db2c-9668-25e9-3d8d-c519e9646df9} {Suffrage:Voter ID:52c81689-fc69-5792-6f19-063f8f5e6bf7 Address:52c81689-fc69-5792-6f19-063f8f5e6bf7}]" - testing.go:149: 52c81689-fc69-5792-6f19-063f8f5e6bf7: 2024-09-03T21:34:34.429Z [INFO] 52c81689-fc69-5792-6f19-063f8f5e6bf7: initial configuration: index=1 servers="[{Suffrage:Voter ID:1073db2c-9668-25e9-3d8d-c519e9646df9 Address:1073db2c-9668-25e9-3d8d-c519e9646df9} {Suffrage:Voter ID:52c81689-fc69-5792-6f19-063f8f5e6bf7 Address:52c81689-fc69-5792-6f19-063f8f5e6bf7}]" -testLoggerAdapter verbose: 1073db2c-9668-25e9-3d8d-c519e9646df9: 2024-09-03T21:34:34.429Z [INFO] 1073db2c-9668-25e9-3d8d-c519e9646df9: entering follower state: follower="Node at 1073db2c-9668-25e9-3d8d-c519e9646df9 [Follower]" leader= - testing.go:149: 1073db2c-9668-25e9-3d8d-c519e9646df9: 2024-09-03T21:34:34.429Z [INFO] 1073db2c-9668-25e9-3d8d-c519e9646df9: entering follower state: follower="Node at 1073db2c-9668-25e9-3d8d-c519e9646df9 [Follower]" leader= -testLoggerAdapter verbose: d241bedd-34a0-ee1b-0fbc-3298b402e4c1: 2024-09-03T21:34:34.429Z [INFO] d241bedd-34a0-ee1b-0fbc-3298b402e4c1: initial configuration: index=0 servers=[] - testing.go:149: d241bedd-34a0-ee1b-0fbc-3298b402e4c1: 2024-09-03T21:34:34.429Z [INFO] d241bedd-34a0-ee1b-0fbc-3298b402e4c1: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:34.429Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: 52c81689-fc69-5792-6f19-063f8f5e6bf7: 2024-09-03T21:34:34.429Z [INFO] 52c81689-fc69-5792-6f19-063f8f5e6bf7: entering follower state: follower="Node at 52c81689-fc69-5792-6f19-063f8f5e6bf7 [Follower]" leader= - testing.go:149: cluster: 2024-09-03T21:34:34.429Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: d241bedd-34a0-ee1b-0fbc-3298b402e4c1: 2024-09-03T21:34:34.429Z [INFO] d241bedd-34a0-ee1b-0fbc-3298b402e4c1: entering follower state: follower="Node at d241bedd-34a0-ee1b-0fbc-3298b402e4c1 [Follower]" leader= - testing.go:149: d241bedd-34a0-ee1b-0fbc-3298b402e4c1: 2024-09-03T21:34:34.429Z [INFO] d241bedd-34a0-ee1b-0fbc-3298b402e4c1: entering follower state: follower="Node at d241bedd-34a0-ee1b-0fbc-3298b402e4c1 [Follower]" leader= - testing.go:149: 52c81689-fc69-5792-6f19-063f8f5e6bf7: 2024-09-03T21:34:34.429Z [INFO] 52c81689-fc69-5792-6f19-063f8f5e6bf7: entering follower state: follower="Node at 52c81689-fc69-5792-6f19-063f8f5e6bf7 [Follower]" leader= -testLoggerAdapter verbose: 1073db2c-9668-25e9-3d8d-c519e9646df9: 2024-09-03T21:34:34.486Z [WARN] 1073db2c-9668-25e9-3d8d-c519e9646df9: heartbeat timeout reached, starting election: last-leader= - testing.go:149: 1073db2c-9668-25e9-3d8d-c519e9646df9: 2024-09-03T21:34:34.486Z [WARN] 1073db2c-9668-25e9-3d8d-c519e9646df9: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: 1073db2c-9668-25e9-3d8d-c519e9646df9: 2024-09-03T21:34:34.486Z [INFO] 1073db2c-9668-25e9-3d8d-c519e9646df9: entering candidate state: node="Node at 1073db2c-9668-25e9-3d8d-c519e9646df9 [Candidate]" term=2 - testing.go:149: 1073db2c-9668-25e9-3d8d-c519e9646df9: 2024-09-03T21:34:34.486Z [INFO] 1073db2c-9668-25e9-3d8d-c519e9646df9: entering candidate state: node="Node at 1073db2c-9668-25e9-3d8d-c519e9646df9 [Candidate]" term=2 -testLoggerAdapter verbose: 1073db2c-9668-25e9-3d8d-c519e9646df9: 2024-09-03T21:34:34.486Z [INFO] 1073db2c-9668-25e9-3d8d-c519e9646df9: election won: tally=2 - testing.go:149: 1073db2c-9668-25e9-3d8d-c519e9646df9: 2024-09-03T21:34:34.486Z [INFO] 1073db2c-9668-25e9-3d8d-c519e9646df9: election won: tally=2 -testLoggerAdapter verbose: 1073db2c-9668-25e9-3d8d-c519e9646df9: 2024-09-03T21:34:34.486Z [INFO] 1073db2c-9668-25e9-3d8d-c519e9646df9: entering leader state: leader="Node at 1073db2c-9668-25e9-3d8d-c519e9646df9 [Leader]" - testing.go:149: 1073db2c-9668-25e9-3d8d-c519e9646df9: 2024-09-03T21:34:34.486Z [INFO] 1073db2c-9668-25e9-3d8d-c519e9646df9: entering leader state: leader="Node at 1073db2c-9668-25e9-3d8d-c519e9646df9 [Leader]" -testLoggerAdapter verbose: 1073db2c-9668-25e9-3d8d-c519e9646df9: 2024-09-03T21:34:34.486Z [INFO] 1073db2c-9668-25e9-3d8d-c519e9646df9: added peer, starting replication: peer=52c81689-fc69-5792-6f19-063f8f5e6bf7 - testing.go:149: 1073db2c-9668-25e9-3d8d-c519e9646df9: 2024-09-03T21:34:34.486Z [INFO] 1073db2c-9668-25e9-3d8d-c519e9646df9: added peer, starting replication: peer=52c81689-fc69-5792-6f19-063f8f5e6bf7 -testLoggerAdapter verbose: 1073db2c-9668-25e9-3d8d-c519e9646df9: 2024-09-03T21:34:34.486Z [INFO] 1073db2c-9668-25e9-3d8d-c519e9646df9: pipelining replication: peer="{Voter 52c81689-fc69-5792-6f19-063f8f5e6bf7 52c81689-fc69-5792-6f19-063f8f5e6bf7}" - testing.go:149: 1073db2c-9668-25e9-3d8d-c519e9646df9: 2024-09-03T21:34:34.486Z [INFO] 1073db2c-9668-25e9-3d8d-c519e9646df9: pipelining replication: peer="{Voter 52c81689-fc69-5792-6f19-063f8f5e6bf7 52c81689-fc69-5792-6f19-063f8f5e6bf7}" -testLoggerAdapter verbose: d241bedd-34a0-ee1b-0fbc-3298b402e4c1: 2024-09-03T21:34:34.523Z [WARN] d241bedd-34a0-ee1b-0fbc-3298b402e4c1: no known peers, aborting election - testing.go:149: d241bedd-34a0-ee1b-0fbc-3298b402e4c1: 2024-09-03T21:34:34.523Z [WARN] d241bedd-34a0-ee1b-0fbc-3298b402e4c1: no known peers, aborting election -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:34.594Z [INFO] stable state for Leader reached at 2024-09-03 21:34:34.486612452 +0000 UTC m=+15.533269596 (1 nodes), 57.281096ms from start of poll, 57.766097ms from cluster start. Timeout at 2024-09-03 21:34:34.594190549 +0000 UTC m=+15.640847683, 107.578087ms after stability - testing.go:149: cluster: 2024-09-03T21:34:34.594Z [INFO] stable state for Leader reached at 2024-09-03 21:34:34.486612452 +0000 UTC m=+15.533269596 (1 nodes), 57.281096ms from start of poll, 57.766097ms from cluster start. Timeout at 2024-09-03 21:34:34.594190549 +0000 UTC m=+15.640847683, 107.578087ms after stability -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:34.594Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:34.594Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:34.702Z [INFO] stable state for Leader reached at 2024-09-03 21:34:34.594365376 +0000 UTC m=+15.641022509 (1 nodes), 4.108µs from start of poll, 165.51901ms from cluster start. Timeout at 2024-09-03 21:34:34.702517785 +0000 UTC m=+15.749174935, 108.152426ms after stability - testing.go:149: cluster: 2024-09-03T21:34:34.702Z [INFO] stable state for Leader reached at 2024-09-03 21:34:34.594365376 +0000 UTC m=+15.641022509 (1 nodes), 4.108µs from start of poll, 165.51901ms from cluster start. Timeout at 2024-09-03 21:34:34.702517785 +0000 UTC m=+15.749174935, 108.152426ms after stability -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:34.702Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:34.702Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:34.807Z [INFO] stable state for Leader reached at 2024-09-03 21:34:34.702671187 +0000 UTC m=+15.749328322 (1 nodes), 1.019µs from start of poll, 273.824823ms from cluster start. Timeout at 2024-09-03 21:34:34.807753702 +0000 UTC m=+15.854410837, 105.082515ms after stability - testing.go:149: cluster: 2024-09-03T21:34:34.807Z [INFO] stable state for Leader reached at 2024-09-03 21:34:34.702671187 +0000 UTC m=+15.749328322 (1 nodes), 1.019µs from start of poll, 273.824823ms from cluster start. Timeout at 2024-09-03 21:34:34.807753702 +0000 UTC m=+15.854410837, 105.082515ms after stability -testLoggerAdapter verbose: 1073db2c-9668-25e9-3d8d-c519e9646df9: 2024-09-03T21:34:34.807Z [INFO] 1073db2c-9668-25e9-3d8d-c519e9646df9: updating configuration: command=AddStaging server-id=d241bedd-34a0-ee1b-0fbc-3298b402e4c1 server-addr=d241bedd-34a0-ee1b-0fbc-3298b402e4c1 servers="[{Suffrage:Voter ID:1073db2c-9668-25e9-3d8d-c519e9646df9 Address:1073db2c-9668-25e9-3d8d-c519e9646df9} {Suffrage:Voter ID:52c81689-fc69-5792-6f19-063f8f5e6bf7 Address:52c81689-fc69-5792-6f19-063f8f5e6bf7} {Suffrage:Voter ID:d241bedd-34a0-ee1b-0fbc-3298b402e4c1 Address:d241bedd-34a0-ee1b-0fbc-3298b402e4c1}]" - testing.go:149: 1073db2c-9668-25e9-3d8d-c519e9646df9: 2024-09-03T21:34:34.807Z [INFO] 1073db2c-9668-25e9-3d8d-c519e9646df9: updating configuration: command=AddStaging server-id=d241bedd-34a0-ee1b-0fbc-3298b402e4c1 server-addr=d241bedd-34a0-ee1b-0fbc-3298b402e4c1 servers="[{Suffrage:Voter ID:1073db2c-9668-25e9-3d8d-c519e9646df9 Address:1073db2c-9668-25e9-3d8d-c519e9646df9} {Suffrage:Voter ID:52c81689-fc69-5792-6f19-063f8f5e6bf7 Address:52c81689-fc69-5792-6f19-063f8f5e6bf7} {Suffrage:Voter ID:d241bedd-34a0-ee1b-0fbc-3298b402e4c1 Address:d241bedd-34a0-ee1b-0fbc-3298b402e4c1}]" -testLoggerAdapter verbose: 1073db2c-9668-25e9-3d8d-c519e9646df9: 2024-09-03T21:34:34.808Z [INFO] 1073db2c-9668-25e9-3d8d-c519e9646df9: added peer, starting replication: peer=d241bedd-34a0-ee1b-0fbc-3298b402e4c1 - testing.go:149: 1073db2c-9668-25e9-3d8d-c519e9646df9: 2024-09-03T21:34:34.808Z [INFO] 1073db2c-9668-25e9-3d8d-c519e9646df9: added peer, starting replication: peer=d241bedd-34a0-ee1b-0fbc-3298b402e4c1 -testLoggerAdapter verbose: d241bedd-34a0-ee1b-0fbc-3298b402e4c1: 2024-09-03T21:34:34.808Z [WARN] d241bedd-34a0-ee1b-0fbc-3298b402e4c1: failed to get previous log: previous-index=3 last-index=0 error="log not found" - testing.go:149: d241bedd-34a0-ee1b-0fbc-3298b402e4c1: 2024-09-03T21:34:34.808Z [WARN] d241bedd-34a0-ee1b-0fbc-3298b402e4c1: failed to get previous log: previous-index=3 last-index=0 error="log not found" -testLoggerAdapter verbose: 1073db2c-9668-25e9-3d8d-c519e9646df9: 2024-09-03T21:34:34.808Z [WARN] 1073db2c-9668-25e9-3d8d-c519e9646df9: appendEntries rejected, sending older logs: peer="{Voter d241bedd-34a0-ee1b-0fbc-3298b402e4c1 d241bedd-34a0-ee1b-0fbc-3298b402e4c1}" next=1 - testing.go:149: 1073db2c-9668-25e9-3d8d-c519e9646df9: 2024-09-03T21:34:34.808Z [WARN] 1073db2c-9668-25e9-3d8d-c519e9646df9: appendEntries rejected, sending older logs: peer="{Voter d241bedd-34a0-ee1b-0fbc-3298b402e4c1 d241bedd-34a0-ee1b-0fbc-3298b402e4c1}" next=1 -testLoggerAdapter verbose: 1073db2c-9668-25e9-3d8d-c519e9646df9: 2024-09-03T21:34:34.808Z [INFO] 1073db2c-9668-25e9-3d8d-c519e9646df9: pipelining replication: peer="{Voter d241bedd-34a0-ee1b-0fbc-3298b402e4c1 d241bedd-34a0-ee1b-0fbc-3298b402e4c1}" - testing.go:149: 1073db2c-9668-25e9-3d8d-c519e9646df9: 2024-09-03T21:34:34.808Z [INFO] 1073db2c-9668-25e9-3d8d-c519e9646df9: pipelining replication: peer="{Voter d241bedd-34a0-ee1b-0fbc-3298b402e4c1 d241bedd-34a0-ee1b-0fbc-3298b402e4c1}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:34.808Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:34.808Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:34.913Z [INFO] stable state for Leader reached at 2024-09-03 21:34:34.808478373 +0000 UTC m=+15.855135509 (1 nodes), 619ns from start of poll, 379.63201ms from cluster start. Timeout at 2024-09-03 21:34:34.913571801 +0000 UTC m=+15.960228934, 105.093425ms after stability - testing.go:149: cluster: 2024-09-03T21:34:34.913Z [INFO] stable state for Leader reached at 2024-09-03 21:34:34.808478373 +0000 UTC m=+15.855135509 (1 nodes), 619ns from start of poll, 379.63201ms from cluster start. Timeout at 2024-09-03 21:34:34.913571801 +0000 UTC m=+15.960228934, 105.093425ms after stability -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:34.913Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:34.913Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:35.018Z [INFO] stable state for Leader reached at 2024-09-03 21:34:34.91373167 +0000 UTC m=+15.960388815 (1 nodes), 891ns from start of poll, 484.885316ms from cluster start. Timeout at 2024-09-03 21:34:35.018742733 +0000 UTC m=+16.065399877, 105.011062ms after stability - testing.go:149: cluster: 2024-09-03T21:34:35.018Z [INFO] stable state for Leader reached at 2024-09-03 21:34:34.91373167 +0000 UTC m=+15.960388815 (1 nodes), 891ns from start of poll, 484.885316ms from cluster start. Timeout at 2024-09-03 21:34:35.018742733 +0000 UTC m=+16.065399877, 105.011062ms after stability -testLoggerAdapter verbose: 1073db2c-9668-25e9-3d8d-c519e9646df9: 2024-09-03T21:34:35.018Z [INFO] 1073db2c-9668-25e9-3d8d-c519e9646df9: updating configuration: command=RemoveServer server-id=d241bedd-34a0-ee1b-0fbc-3298b402e4c1 server-addr= servers="[{Suffrage:Voter ID:1073db2c-9668-25e9-3d8d-c519e9646df9 Address:1073db2c-9668-25e9-3d8d-c519e9646df9} {Suffrage:Voter ID:52c81689-fc69-5792-6f19-063f8f5e6bf7 Address:52c81689-fc69-5792-6f19-063f8f5e6bf7}]" - testing.go:149: 1073db2c-9668-25e9-3d8d-c519e9646df9: 2024-09-03T21:34:35.018Z [INFO] 1073db2c-9668-25e9-3d8d-c519e9646df9: updating configuration: command=RemoveServer server-id=d241bedd-34a0-ee1b-0fbc-3298b402e4c1 server-addr= servers="[{Suffrage:Voter ID:1073db2c-9668-25e9-3d8d-c519e9646df9 Address:1073db2c-9668-25e9-3d8d-c519e9646df9} {Suffrage:Voter ID:52c81689-fc69-5792-6f19-063f8f5e6bf7 Address:52c81689-fc69-5792-6f19-063f8f5e6bf7}]" -testLoggerAdapter verbose: 1073db2c-9668-25e9-3d8d-c519e9646df9: 2024-09-03T21:34:35.019Z [INFO] 1073db2c-9668-25e9-3d8d-c519e9646df9: removed peer, stopping replication: peer=d241bedd-34a0-ee1b-0fbc-3298b402e4c1 last-index=4 - testing.go:149: 1073db2c-9668-25e9-3d8d-c519e9646df9: 2024-09-03T21:34:35.019Z [INFO] 1073db2c-9668-25e9-3d8d-c519e9646df9: removed peer, stopping replication: peer=d241bedd-34a0-ee1b-0fbc-3298b402e4c1 last-index=4 -testLoggerAdapter verbose: 1073db2c-9668-25e9-3d8d-c519e9646df9: 2024-09-03T21:34:35.019Z [INFO] 1073db2c-9668-25e9-3d8d-c519e9646df9: aborting pipeline replication: peer="{Voter d241bedd-34a0-ee1b-0fbc-3298b402e4c1 d241bedd-34a0-ee1b-0fbc-3298b402e4c1}" - testing.go:149: 1073db2c-9668-25e9-3d8d-c519e9646df9: 2024-09-03T21:34:35.019Z [INFO] 1073db2c-9668-25e9-3d8d-c519e9646df9: aborting pipeline replication: peer="{Voter d241bedd-34a0-ee1b-0fbc-3298b402e4c1 d241bedd-34a0-ee1b-0fbc-3298b402e4c1}" -testLoggerAdapter verbose: 1073db2c-9668-25e9-3d8d-c519e9646df9: 2024-09-03T21:34:35.019Z [INFO] 1073db2c-9668-25e9-3d8d-c519e9646df9: aborting pipeline replication: peer="{Voter 52c81689-fc69-5792-6f19-063f8f5e6bf7 52c81689-fc69-5792-6f19-063f8f5e6bf7}" - testing.go:149: 1073db2c-9668-25e9-3d8d-c519e9646df9: 2024-09-03T21:34:35.019Z [INFO] 1073db2c-9668-25e9-3d8d-c519e9646df9: aborting pipeline replication: peer="{Voter 52c81689-fc69-5792-6f19-063f8f5e6bf7 52c81689-fc69-5792-6f19-063f8f5e6bf7}" +testLoggerAdapter verbose: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.503Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: initial configuration: index=1 servers="[{Suffrage:Voter ID:ae3e7bcb-aec5-d032-06c8-d90ab73ce914 Address:ae3e7bcb-aec5-d032-06c8-d90ab73ce914} {Suffrage:Voter ID:db56337b-e877-27f1-3b4a-764bfa81b234 Address:db56337b-e877-27f1-3b4a-764bfa81b234}]" + testing.go:149: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.503Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: initial configuration: index=1 servers="[{Suffrage:Voter ID:ae3e7bcb-aec5-d032-06c8-d90ab73ce914 Address:ae3e7bcb-aec5-d032-06c8-d90ab73ce914} {Suffrage:Voter ID:db56337b-e877-27f1-3b4a-764bfa81b234 Address:db56337b-e877-27f1-3b4a-764bfa81b234}]" +testLoggerAdapter verbose: db56337b-e877-27f1-3b4a-764bfa81b234: 2023-08-02T15:13:03.503Z [INFO] db56337b-e877-27f1-3b4a-764bfa81b234: initial configuration: index=1 servers="[{Suffrage:Voter ID:ae3e7bcb-aec5-d032-06c8-d90ab73ce914 Address:ae3e7bcb-aec5-d032-06c8-d90ab73ce914} {Suffrage:Voter ID:db56337b-e877-27f1-3b4a-764bfa81b234 Address:db56337b-e877-27f1-3b4a-764bfa81b234}]" + testing.go:149: db56337b-e877-27f1-3b4a-764bfa81b234: 2023-08-02T15:13:03.503Z [INFO] db56337b-e877-27f1-3b4a-764bfa81b234: initial configuration: index=1 servers="[{Suffrage:Voter ID:ae3e7bcb-aec5-d032-06c8-d90ab73ce914 Address:ae3e7bcb-aec5-d032-06c8-d90ab73ce914} {Suffrage:Voter ID:db56337b-e877-27f1-3b4a-764bfa81b234 Address:db56337b-e877-27f1-3b4a-764bfa81b234}]" +testLoggerAdapter verbose: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.503Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: entering follower state: follower="Node at ae3e7bcb-aec5-d032-06c8-d90ab73ce914 [Follower]" leader= + testing.go:149: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.503Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: entering follower state: follower="Node at ae3e7bcb-aec5-d032-06c8-d90ab73ce914 [Follower]" leader= +testLoggerAdapter verbose: db56337b-e877-27f1-3b4a-764bfa81b234: 2023-08-02T15:13:03.503Z [INFO] db56337b-e877-27f1-3b4a-764bfa81b234: entering follower state: follower="Node at db56337b-e877-27f1-3b4a-764bfa81b234 [Follower]" leader= + testing.go:149: db56337b-e877-27f1-3b4a-764bfa81b234: 2023-08-02T15:13:03.503Z [INFO] db56337b-e877-27f1-3b4a-764bfa81b234: entering follower state: follower="Node at db56337b-e877-27f1-3b4a-764bfa81b234 [Follower]" leader= +testLoggerAdapter verbose: 39a60e0c-77b5-4ca1-ccc4-6b716f84331a: 2023-08-02T15:13:03.503Z [INFO] 39a60e0c-77b5-4ca1-ccc4-6b716f84331a: initial configuration: index=0 servers=[] + testing.go:149: 39a60e0c-77b5-4ca1-ccc4-6b716f84331a: 2023-08-02T15:13:03.503Z [INFO] 39a60e0c-77b5-4ca1-ccc4-6b716f84331a: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:03.503Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:13:03.503Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: 39a60e0c-77b5-4ca1-ccc4-6b716f84331a: 2023-08-02T15:13:03.504Z [INFO] 39a60e0c-77b5-4ca1-ccc4-6b716f84331a: entering follower state: follower="Node at 39a60e0c-77b5-4ca1-ccc4-6b716f84331a [Follower]" leader= + testing.go:149: 39a60e0c-77b5-4ca1-ccc4-6b716f84331a: 2023-08-02T15:13:03.504Z [INFO] 39a60e0c-77b5-4ca1-ccc4-6b716f84331a: entering follower state: follower="Node at 39a60e0c-77b5-4ca1-ccc4-6b716f84331a [Follower]" leader= +testLoggerAdapter verbose: 39a60e0c-77b5-4ca1-ccc4-6b716f84331a: 2023-08-02T15:13:03.562Z [WARN] 39a60e0c-77b5-4ca1-ccc4-6b716f84331a: no known peers, aborting election + testing.go:149: 39a60e0c-77b5-4ca1-ccc4-6b716f84331a: 2023-08-02T15:13:03.562Z [WARN] 39a60e0c-77b5-4ca1-ccc4-6b716f84331a: no known peers, aborting election +testLoggerAdapter verbose: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.565Z [WARN] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: heartbeat timeout reached, starting election: last-leader= + testing.go:149: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.565Z [WARN] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.565Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: entering candidate state: node="Node at ae3e7bcb-aec5-d032-06c8-d90ab73ce914 [Candidate]" term=2 + testing.go:149: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.565Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: entering candidate state: node="Node at ae3e7bcb-aec5-d032-06c8-d90ab73ce914 [Candidate]" term=2 +testLoggerAdapter verbose: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.566Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: election won: tally=2 + testing.go:149: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.566Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: election won: tally=2 +testLoggerAdapter verbose: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.566Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: entering leader state: leader="Node at ae3e7bcb-aec5-d032-06c8-d90ab73ce914 [Leader]" + testing.go:149: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.566Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: entering leader state: leader="Node at ae3e7bcb-aec5-d032-06c8-d90ab73ce914 [Leader]" +testLoggerAdapter verbose: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.566Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: added peer, starting replication: peer=db56337b-e877-27f1-3b4a-764bfa81b234 + testing.go:149: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.566Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: added peer, starting replication: peer=db56337b-e877-27f1-3b4a-764bfa81b234 +testLoggerAdapter verbose: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.566Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: pipelining replication: peer="{Voter db56337b-e877-27f1-3b4a-764bfa81b234 db56337b-e877-27f1-3b4a-764bfa81b234}" + testing.go:149: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.566Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: pipelining replication: peer="{Voter db56337b-e877-27f1-3b4a-764bfa81b234 db56337b-e877-27f1-3b4a-764bfa81b234}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:03.671Z [INFO] stable state for Leader reached at 2023-08-02 15:13:03.566250029 +0000 UTC m=+15.662186356 (1 nodes), 62.22204ms from start of poll, 62.730003ms from cluster start. Timeout at 2023-08-02 15:13:03.67135306 +0000 UTC m=+15.767289390, 105.103034ms after stability + testing.go:149: cluster: 2023-08-02T15:13:03.671Z [INFO] stable state for Leader reached at 2023-08-02 15:13:03.566250029 +0000 UTC m=+15.662186356 (1 nodes), 62.22204ms from start of poll, 62.730003ms from cluster start. Timeout at 2023-08-02 15:13:03.67135306 +0000 UTC m=+15.767289390, 105.103034ms after stability +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:03.671Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:13:03.671Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:03.776Z [INFO] stable state for Leader reached at 2023-08-02 15:13:03.671497672 +0000 UTC m=+15.767433999 (1 nodes), 1.078µs from start of poll, 167.977646ms from cluster start. Timeout at 2023-08-02 15:13:03.776797502 +0000 UTC m=+15.872733836, 105.299837ms after stability + testing.go:149: cluster: 2023-08-02T15:13:03.776Z [INFO] stable state for Leader reached at 2023-08-02 15:13:03.671497672 +0000 UTC m=+15.767433999 (1 nodes), 1.078µs from start of poll, 167.977646ms from cluster start. Timeout at 2023-08-02 15:13:03.776797502 +0000 UTC m=+15.872733836, 105.299837ms after stability +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:03.776Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:13:03.776Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:03.882Z [INFO] stable state for Leader reached at 2023-08-02 15:13:03.776985016 +0000 UTC m=+15.872921353 (1 nodes), 1.149µs from start of poll, 273.465ms from cluster start. Timeout at 2023-08-02 15:13:03.882067591 +0000 UTC m=+15.978003917, 105.082564ms after stability + testing.go:149: cluster: 2023-08-02T15:13:03.882Z [INFO] stable state for Leader reached at 2023-08-02 15:13:03.776985016 +0000 UTC m=+15.872921353 (1 nodes), 1.149µs from start of poll, 273.465ms from cluster start. Timeout at 2023-08-02 15:13:03.882067591 +0000 UTC m=+15.978003917, 105.082564ms after stability +testLoggerAdapter verbose: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.882Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: updating configuration: command=AddStaging server-id=39a60e0c-77b5-4ca1-ccc4-6b716f84331a server-addr=39a60e0c-77b5-4ca1-ccc4-6b716f84331a servers="[{Suffrage:Voter ID:ae3e7bcb-aec5-d032-06c8-d90ab73ce914 Address:ae3e7bcb-aec5-d032-06c8-d90ab73ce914} {Suffrage:Voter ID:db56337b-e877-27f1-3b4a-764bfa81b234 Address:db56337b-e877-27f1-3b4a-764bfa81b234} {Suffrage:Voter ID:39a60e0c-77b5-4ca1-ccc4-6b716f84331a Address:39a60e0c-77b5-4ca1-ccc4-6b716f84331a}]" + testing.go:149: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.882Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: updating configuration: command=AddStaging server-id=39a60e0c-77b5-4ca1-ccc4-6b716f84331a server-addr=39a60e0c-77b5-4ca1-ccc4-6b716f84331a servers="[{Suffrage:Voter ID:ae3e7bcb-aec5-d032-06c8-d90ab73ce914 Address:ae3e7bcb-aec5-d032-06c8-d90ab73ce914} {Suffrage:Voter ID:db56337b-e877-27f1-3b4a-764bfa81b234 Address:db56337b-e877-27f1-3b4a-764bfa81b234} {Suffrage:Voter ID:39a60e0c-77b5-4ca1-ccc4-6b716f84331a Address:39a60e0c-77b5-4ca1-ccc4-6b716f84331a}]" +testLoggerAdapter verbose: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.882Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: added peer, starting replication: peer=39a60e0c-77b5-4ca1-ccc4-6b716f84331a + testing.go:149: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.882Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: added peer, starting replication: peer=39a60e0c-77b5-4ca1-ccc4-6b716f84331a +testLoggerAdapter verbose: 39a60e0c-77b5-4ca1-ccc4-6b716f84331a: 2023-08-02T15:13:03.882Z [WARN] 39a60e0c-77b5-4ca1-ccc4-6b716f84331a: failed to get previous log: previous-index=3 last-index=0 error="log not found" + testing.go:149: 39a60e0c-77b5-4ca1-ccc4-6b716f84331a: 2023-08-02T15:13:03.882Z [WARN] 39a60e0c-77b5-4ca1-ccc4-6b716f84331a: failed to get previous log: previous-index=3 last-index=0 error="log not found" +testLoggerAdapter verbose: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.882Z [WARN] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: appendEntries rejected, sending older logs: peer="{Voter 39a60e0c-77b5-4ca1-ccc4-6b716f84331a 39a60e0c-77b5-4ca1-ccc4-6b716f84331a}" next=1 + testing.go:149: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.882Z [WARN] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: appendEntries rejected, sending older logs: peer="{Voter 39a60e0c-77b5-4ca1-ccc4-6b716f84331a 39a60e0c-77b5-4ca1-ccc4-6b716f84331a}" next=1 +testLoggerAdapter verbose: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.882Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: pipelining replication: peer="{Voter 39a60e0c-77b5-4ca1-ccc4-6b716f84331a 39a60e0c-77b5-4ca1-ccc4-6b716f84331a}" + testing.go:149: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.882Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: pipelining replication: peer="{Voter 39a60e0c-77b5-4ca1-ccc4-6b716f84331a 39a60e0c-77b5-4ca1-ccc4-6b716f84331a}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:03.882Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:13:03.882Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:03.987Z [INFO] stable state for Leader reached at 2023-08-02 15:13:03.882682506 +0000 UTC m=+15.978618839 (1 nodes), 756ns from start of poll, 379.162486ms from cluster start. Timeout at 2023-08-02 15:13:03.987735051 +0000 UTC m=+16.083671386, 105.052547ms after stability + testing.go:149: cluster: 2023-08-02T15:13:03.987Z [INFO] stable state for Leader reached at 2023-08-02 15:13:03.882682506 +0000 UTC m=+15.978618839 (1 nodes), 756ns from start of poll, 379.162486ms from cluster start. Timeout at 2023-08-02 15:13:03.987735051 +0000 UTC m=+16.083671386, 105.052547ms after stability +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:03.987Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:13:03.987Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:04.093Z [INFO] stable state for Leader reached at 2023-08-02 15:13:03.987932063 +0000 UTC m=+16.083868389 (1 nodes), 1.09µs from start of poll, 484.412036ms from cluster start. Timeout at 2023-08-02 15:13:04.093544787 +0000 UTC m=+16.189481124, 105.612735ms after stability + testing.go:149: cluster: 2023-08-02T15:13:04.093Z [INFO] stable state for Leader reached at 2023-08-02 15:13:03.987932063 +0000 UTC m=+16.083868389 (1 nodes), 1.09µs from start of poll, 484.412036ms from cluster start. Timeout at 2023-08-02 15:13:04.093544787 +0000 UTC m=+16.189481124, 105.612735ms after stability +testLoggerAdapter verbose: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:04.093Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: updating configuration: command=RemoveServer server-id=39a60e0c-77b5-4ca1-ccc4-6b716f84331a server-addr= servers="[{Suffrage:Voter ID:ae3e7bcb-aec5-d032-06c8-d90ab73ce914 Address:ae3e7bcb-aec5-d032-06c8-d90ab73ce914} {Suffrage:Voter ID:db56337b-e877-27f1-3b4a-764bfa81b234 Address:db56337b-e877-27f1-3b4a-764bfa81b234}]" + testing.go:149: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:04.093Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: updating configuration: command=RemoveServer server-id=39a60e0c-77b5-4ca1-ccc4-6b716f84331a server-addr= servers="[{Suffrage:Voter ID:ae3e7bcb-aec5-d032-06c8-d90ab73ce914 Address:ae3e7bcb-aec5-d032-06c8-d90ab73ce914} {Suffrage:Voter ID:db56337b-e877-27f1-3b4a-764bfa81b234 Address:db56337b-e877-27f1-3b4a-764bfa81b234}]" +testLoggerAdapter verbose: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:04.093Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: removed peer, stopping replication: peer=39a60e0c-77b5-4ca1-ccc4-6b716f84331a last-index=4 + testing.go:149: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:04.093Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: removed peer, stopping replication: peer=39a60e0c-77b5-4ca1-ccc4-6b716f84331a last-index=4 +testLoggerAdapter verbose: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:04.093Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: aborting pipeline replication: peer="{Voter 39a60e0c-77b5-4ca1-ccc4-6b716f84331a 39a60e0c-77b5-4ca1-ccc4-6b716f84331a}" + testing.go:149: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:04.093Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: aborting pipeline replication: peer="{Voter 39a60e0c-77b5-4ca1-ccc4-6b716f84331a 39a60e0c-77b5-4ca1-ccc4-6b716f84331a}" +testLoggerAdapter verbose: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:04.094Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: aborting pipeline replication: peer="{Voter db56337b-e877-27f1-3b4a-764bfa81b234 db56337b-e877-27f1-3b4a-764bfa81b234}" + testing.go:149: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:04.094Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: aborting pipeline replication: peer="{Voter db56337b-e877-27f1-3b4a-764bfa81b234 db56337b-e877-27f1-3b4a-764bfa81b234}" --- PASS: TestRaft_ProtocolVersion_Upgrade_1_2 (0.59s) === RUN TestRaft_ProtocolVersion_Upgrade_2_3 -testLoggerAdapter verbose: 02b9cf67-0de4-2563-5880-5f6e258a4016: 2024-09-03T21:34:35.019Z [INFO] 02b9cf67-0de4-2563-5880-5f6e258a4016: initial configuration: index=1 servers="[{Suffrage:Voter ID:02b9cf67-0de4-2563-5880-5f6e258a4016 Address:02b9cf67-0de4-2563-5880-5f6e258a4016} {Suffrage:Voter ID:30dcdab0-c001-1e95-aecd-133fdbcc3d32 Address:30dcdab0-c001-1e95-aecd-133fdbcc3d32}]" - testing.go:149: 02b9cf67-0de4-2563-5880-5f6e258a4016: 2024-09-03T21:34:35.019Z [INFO] 02b9cf67-0de4-2563-5880-5f6e258a4016: initial configuration: index=1 servers="[{Suffrage:Voter ID:02b9cf67-0de4-2563-5880-5f6e258a4016 Address:02b9cf67-0de4-2563-5880-5f6e258a4016} {Suffrage:Voter ID:30dcdab0-c001-1e95-aecd-133fdbcc3d32 Address:30dcdab0-c001-1e95-aecd-133fdbcc3d32}]" -testLoggerAdapter verbose: 30dcdab0-c001-1e95-aecd-133fdbcc3d32: 2024-09-03T21:34:35.019Z [INFO] 30dcdab0-c001-1e95-aecd-133fdbcc3d32: initial configuration: index=1 servers="[{Suffrage:Voter ID:02b9cf67-0de4-2563-5880-5f6e258a4016 Address:02b9cf67-0de4-2563-5880-5f6e258a4016} {Suffrage:Voter ID:30dcdab0-c001-1e95-aecd-133fdbcc3d32 Address:30dcdab0-c001-1e95-aecd-133fdbcc3d32}]" - testing.go:149: 30dcdab0-c001-1e95-aecd-133fdbcc3d32: 2024-09-03T21:34:35.019Z [INFO] 30dcdab0-c001-1e95-aecd-133fdbcc3d32: initial configuration: index=1 servers="[{Suffrage:Voter ID:02b9cf67-0de4-2563-5880-5f6e258a4016 Address:02b9cf67-0de4-2563-5880-5f6e258a4016} {Suffrage:Voter ID:30dcdab0-c001-1e95-aecd-133fdbcc3d32 Address:30dcdab0-c001-1e95-aecd-133fdbcc3d32}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:35.020Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-09-03T21:34:35.020Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: 02b9cf67-0de4-2563-5880-5f6e258a4016: 2024-09-03T21:34:35.020Z [INFO] 02b9cf67-0de4-2563-5880-5f6e258a4016: entering follower state: follower="Node at 02b9cf67-0de4-2563-5880-5f6e258a4016 [Follower]" leader= - testing.go:149: 02b9cf67-0de4-2563-5880-5f6e258a4016: 2024-09-03T21:34:35.020Z [INFO] 02b9cf67-0de4-2563-5880-5f6e258a4016: entering follower state: follower="Node at 02b9cf67-0de4-2563-5880-5f6e258a4016 [Follower]" leader= -testLoggerAdapter verbose: 30dcdab0-c001-1e95-aecd-133fdbcc3d32: 2024-09-03T21:34:35.020Z [INFO] 30dcdab0-c001-1e95-aecd-133fdbcc3d32: entering follower state: follower="Node at 30dcdab0-c001-1e95-aecd-133fdbcc3d32 [Follower]" leader= - testing.go:149: 30dcdab0-c001-1e95-aecd-133fdbcc3d32: 2024-09-03T21:34:35.020Z [INFO] 30dcdab0-c001-1e95-aecd-133fdbcc3d32: entering follower state: follower="Node at 30dcdab0-c001-1e95-aecd-133fdbcc3d32 [Follower]" leader= -testLoggerAdapter verbose: 30dcdab0-c001-1e95-aecd-133fdbcc3d32: 2024-09-03T21:34:35.115Z [WARN] 30dcdab0-c001-1e95-aecd-133fdbcc3d32: heartbeat timeout reached, starting election: last-leader= - testing.go:149: 30dcdab0-c001-1e95-aecd-133fdbcc3d32: 2024-09-03T21:34:35.115Z [WARN] 30dcdab0-c001-1e95-aecd-133fdbcc3d32: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: 30dcdab0-c001-1e95-aecd-133fdbcc3d32: 2024-09-03T21:34:35.115Z [INFO] 30dcdab0-c001-1e95-aecd-133fdbcc3d32: entering candidate state: node="Node at 30dcdab0-c001-1e95-aecd-133fdbcc3d32 [Candidate]" term=2 - testing.go:149: 30dcdab0-c001-1e95-aecd-133fdbcc3d32: 2024-09-03T21:34:35.115Z [INFO] 30dcdab0-c001-1e95-aecd-133fdbcc3d32: entering candidate state: node="Node at 30dcdab0-c001-1e95-aecd-133fdbcc3d32 [Candidate]" term=2 -testLoggerAdapter verbose: 30dcdab0-c001-1e95-aecd-133fdbcc3d32: 2024-09-03T21:34:35.115Z [INFO] 30dcdab0-c001-1e95-aecd-133fdbcc3d32: election won: tally=2 - testing.go:149: 30dcdab0-c001-1e95-aecd-133fdbcc3d32: 2024-09-03T21:34:35.115Z [INFO] 30dcdab0-c001-1e95-aecd-133fdbcc3d32: election won: tally=2 -testLoggerAdapter verbose: 30dcdab0-c001-1e95-aecd-133fdbcc3d32: 2024-09-03T21:34:35.115Z [INFO] 30dcdab0-c001-1e95-aecd-133fdbcc3d32: entering leader state: leader="Node at 30dcdab0-c001-1e95-aecd-133fdbcc3d32 [Leader]" - testing.go:149: 30dcdab0-c001-1e95-aecd-133fdbcc3d32: 2024-09-03T21:34:35.115Z [INFO] 30dcdab0-c001-1e95-aecd-133fdbcc3d32: entering leader state: leader="Node at 30dcdab0-c001-1e95-aecd-133fdbcc3d32 [Leader]" -testLoggerAdapter verbose: 30dcdab0-c001-1e95-aecd-133fdbcc3d32: 2024-09-03T21:34:35.115Z [INFO] 30dcdab0-c001-1e95-aecd-133fdbcc3d32: added peer, starting replication: peer=02b9cf67-0de4-2563-5880-5f6e258a4016 - testing.go:149: 30dcdab0-c001-1e95-aecd-133fdbcc3d32: 2024-09-03T21:34:35.115Z [INFO] 30dcdab0-c001-1e95-aecd-133fdbcc3d32: added peer, starting replication: peer=02b9cf67-0de4-2563-5880-5f6e258a4016 -testLoggerAdapter verbose: 30dcdab0-c001-1e95-aecd-133fdbcc3d32: 2024-09-03T21:34:35.115Z [INFO] 30dcdab0-c001-1e95-aecd-133fdbcc3d32: pipelining replication: peer="{Voter 02b9cf67-0de4-2563-5880-5f6e258a4016 02b9cf67-0de4-2563-5880-5f6e258a4016}" - testing.go:149: 30dcdab0-c001-1e95-aecd-133fdbcc3d32: 2024-09-03T21:34:35.115Z [INFO] 30dcdab0-c001-1e95-aecd-133fdbcc3d32: pipelining replication: peer="{Voter 02b9cf67-0de4-2563-5880-5f6e258a4016 02b9cf67-0de4-2563-5880-5f6e258a4016}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:35.220Z [INFO] stable state for Follower reached at 2024-09-03 21:34:35.115685654 +0000 UTC m=+16.162342792 (1 nodes), 95.647185ms from start of poll, 95.897899ms from cluster start. Timeout at 2024-09-03 21:34:35.220763536 +0000 UTC m=+16.267420672, 105.07788ms after stability - testing.go:149: cluster: 2024-09-03T21:34:35.220Z [INFO] stable state for Follower reached at 2024-09-03 21:34:35.115685654 +0000 UTC m=+16.162342792 (1 nodes), 95.647185ms from start of poll, 95.897899ms from cluster start. Timeout at 2024-09-03 21:34:35.220763536 +0000 UTC m=+16.267420672, 105.07788ms after stability -testLoggerAdapter verbose: server-15bd09e8-59a7-5b1b-e064-577b992ffcd5: 2024-09-03T21:34:35.221Z [INFO] server-15bd09e8-59a7-5b1b-e064-577b992ffcd5: initial configuration: index=0 servers=[] - testing.go:149: server-15bd09e8-59a7-5b1b-e064-577b992ffcd5: 2024-09-03T21:34:35.221Z [INFO] server-15bd09e8-59a7-5b1b-e064-577b992ffcd5: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:35.221Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:35.221Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-15bd09e8-59a7-5b1b-e064-577b992ffcd5: 2024-09-03T21:34:35.221Z [INFO] server-15bd09e8-59a7-5b1b-e064-577b992ffcd5: entering follower state: follower="Node at 15bd09e8-59a7-5b1b-e064-577b992ffcd5 [Follower]" leader= - testing.go:149: server-15bd09e8-59a7-5b1b-e064-577b992ffcd5: 2024-09-03T21:34:35.221Z [INFO] server-15bd09e8-59a7-5b1b-e064-577b992ffcd5: entering follower state: follower="Node at 15bd09e8-59a7-5b1b-e064-577b992ffcd5 [Follower]" leader= -testLoggerAdapter verbose: server-15bd09e8-59a7-5b1b-e064-577b992ffcd5: 2024-09-03T21:34:35.289Z [WARN] server-15bd09e8-59a7-5b1b-e064-577b992ffcd5: no known peers, aborting election - testing.go:149: server-15bd09e8-59a7-5b1b-e064-577b992ffcd5: 2024-09-03T21:34:35.289Z [WARN] server-15bd09e8-59a7-5b1b-e064-577b992ffcd5: no known peers, aborting election -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:35.326Z [INFO] stable state for Leader reached at 2024-09-03 21:34:35.221197693 +0000 UTC m=+16.267854830 (1 nodes), 555ns from start of poll, 201.409937ms from cluster start. Timeout at 2024-09-03 21:34:35.326273338 +0000 UTC m=+16.372930473, 105.075643ms after stability - testing.go:149: cluster: 2024-09-03T21:34:35.326Z [INFO] stable state for Leader reached at 2024-09-03 21:34:35.221197693 +0000 UTC m=+16.267854830 (1 nodes), 555ns from start of poll, 201.409937ms from cluster start. Timeout at 2024-09-03 21:34:35.326273338 +0000 UTC m=+16.372930473, 105.075643ms after stability -testLoggerAdapter verbose: 30dcdab0-c001-1e95-aecd-133fdbcc3d32: 2024-09-03T21:34:35.326Z [INFO] 30dcdab0-c001-1e95-aecd-133fdbcc3d32: updating configuration: command=AddStaging server-id=server-15bd09e8-59a7-5b1b-e064-577b992ffcd5 server-addr=15bd09e8-59a7-5b1b-e064-577b992ffcd5 servers="[{Suffrage:Voter ID:02b9cf67-0de4-2563-5880-5f6e258a4016 Address:02b9cf67-0de4-2563-5880-5f6e258a4016} {Suffrage:Voter ID:30dcdab0-c001-1e95-aecd-133fdbcc3d32 Address:30dcdab0-c001-1e95-aecd-133fdbcc3d32} {Suffrage:Voter ID:server-15bd09e8-59a7-5b1b-e064-577b992ffcd5 Address:15bd09e8-59a7-5b1b-e064-577b992ffcd5}]" - testing.go:149: 30dcdab0-c001-1e95-aecd-133fdbcc3d32: 2024-09-03T21:34:35.326Z [INFO] 30dcdab0-c001-1e95-aecd-133fdbcc3d32: updating configuration: command=AddStaging server-id=server-15bd09e8-59a7-5b1b-e064-577b992ffcd5 server-addr=15bd09e8-59a7-5b1b-e064-577b992ffcd5 servers="[{Suffrage:Voter ID:02b9cf67-0de4-2563-5880-5f6e258a4016 Address:02b9cf67-0de4-2563-5880-5f6e258a4016} {Suffrage:Voter ID:30dcdab0-c001-1e95-aecd-133fdbcc3d32 Address:30dcdab0-c001-1e95-aecd-133fdbcc3d32} {Suffrage:Voter ID:server-15bd09e8-59a7-5b1b-e064-577b992ffcd5 Address:15bd09e8-59a7-5b1b-e064-577b992ffcd5}]" -testLoggerAdapter verbose: 30dcdab0-c001-1e95-aecd-133fdbcc3d32: 2024-09-03T21:34:35.326Z [INFO] 30dcdab0-c001-1e95-aecd-133fdbcc3d32: added peer, starting replication: peer=server-15bd09e8-59a7-5b1b-e064-577b992ffcd5 - testing.go:149: 30dcdab0-c001-1e95-aecd-133fdbcc3d32: 2024-09-03T21:34:35.326Z [INFO] 30dcdab0-c001-1e95-aecd-133fdbcc3d32: added peer, starting replication: peer=server-15bd09e8-59a7-5b1b-e064-577b992ffcd5 -testLoggerAdapter verbose: server-15bd09e8-59a7-5b1b-e064-577b992ffcd5: 2024-09-03T21:34:35.326Z [WARN] server-15bd09e8-59a7-5b1b-e064-577b992ffcd5: failed to get previous log: previous-index=3 last-index=0 error="log not found" - testing.go:149: server-15bd09e8-59a7-5b1b-e064-577b992ffcd5: 2024-09-03T21:34:35.326Z [WARN] server-15bd09e8-59a7-5b1b-e064-577b992ffcd5: failed to get previous log: previous-index=3 last-index=0 error="log not found" -testLoggerAdapter verbose: 30dcdab0-c001-1e95-aecd-133fdbcc3d32: 2024-09-03T21:34:35.326Z [WARN] 30dcdab0-c001-1e95-aecd-133fdbcc3d32: appendEntries rejected, sending older logs: peer="{Voter server-15bd09e8-59a7-5b1b-e064-577b992ffcd5 15bd09e8-59a7-5b1b-e064-577b992ffcd5}" next=1 - testing.go:149: 30dcdab0-c001-1e95-aecd-133fdbcc3d32: 2024-09-03T21:34:35.326Z [WARN] 30dcdab0-c001-1e95-aecd-133fdbcc3d32: appendEntries rejected, sending older logs: peer="{Voter server-15bd09e8-59a7-5b1b-e064-577b992ffcd5 15bd09e8-59a7-5b1b-e064-577b992ffcd5}" next=1 -testLoggerAdapter verbose: 30dcdab0-c001-1e95-aecd-133fdbcc3d32: 2024-09-03T21:34:35.326Z [INFO] 30dcdab0-c001-1e95-aecd-133fdbcc3d32: pipelining replication: peer="{Voter server-15bd09e8-59a7-5b1b-e064-577b992ffcd5 15bd09e8-59a7-5b1b-e064-577b992ffcd5}" - testing.go:149: 30dcdab0-c001-1e95-aecd-133fdbcc3d32: 2024-09-03T21:34:35.326Z [INFO] 30dcdab0-c001-1e95-aecd-133fdbcc3d32: pipelining replication: peer="{Voter server-15bd09e8-59a7-5b1b-e064-577b992ffcd5 15bd09e8-59a7-5b1b-e064-577b992ffcd5}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:35.326Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:35.326Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:35.432Z [INFO] stable state for Leader reached at 2024-09-03 21:34:35.326931237 +0000 UTC m=+16.373588372 (1 nodes), 719ns from start of poll, 307.143479ms from cluster start. Timeout at 2024-09-03 21:34:35.43204243 +0000 UTC m=+16.478699564, 105.111192ms after stability - testing.go:149: cluster: 2024-09-03T21:34:35.432Z [INFO] stable state for Leader reached at 2024-09-03 21:34:35.326931237 +0000 UTC m=+16.373588372 (1 nodes), 719ns from start of poll, 307.143479ms from cluster start. Timeout at 2024-09-03 21:34:35.43204243 +0000 UTC m=+16.478699564, 105.111192ms after stability -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:35.432Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:35.432Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:35.537Z [INFO] stable state for Leader reached at 2024-09-03 21:34:35.432182545 +0000 UTC m=+16.478839681 (1 nodes), 943ns from start of poll, 412.394788ms from cluster start. Timeout at 2024-09-03 21:34:35.537299294 +0000 UTC m=+16.583956436, 105.116755ms after stability - testing.go:149: cluster: 2024-09-03T21:34:35.537Z [INFO] stable state for Leader reached at 2024-09-03 21:34:35.432182545 +0000 UTC m=+16.478839681 (1 nodes), 943ns from start of poll, 412.394788ms from cluster start. Timeout at 2024-09-03 21:34:35.537299294 +0000 UTC m=+16.583956436, 105.116755ms after stability -testLoggerAdapter verbose: 30dcdab0-c001-1e95-aecd-133fdbcc3d32: 2024-09-03T21:34:35.537Z [INFO] 30dcdab0-c001-1e95-aecd-133fdbcc3d32: updating configuration: command=RemoveServer server-id=02b9cf67-0de4-2563-5880-5f6e258a4016 server-addr= servers="[{Suffrage:Voter ID:30dcdab0-c001-1e95-aecd-133fdbcc3d32 Address:30dcdab0-c001-1e95-aecd-133fdbcc3d32} {Suffrage:Voter ID:server-15bd09e8-59a7-5b1b-e064-577b992ffcd5 Address:15bd09e8-59a7-5b1b-e064-577b992ffcd5}]" - testing.go:149: 30dcdab0-c001-1e95-aecd-133fdbcc3d32: 2024-09-03T21:34:35.537Z [INFO] 30dcdab0-c001-1e95-aecd-133fdbcc3d32: updating configuration: command=RemoveServer server-id=02b9cf67-0de4-2563-5880-5f6e258a4016 server-addr= servers="[{Suffrage:Voter ID:30dcdab0-c001-1e95-aecd-133fdbcc3d32 Address:30dcdab0-c001-1e95-aecd-133fdbcc3d32} {Suffrage:Voter ID:server-15bd09e8-59a7-5b1b-e064-577b992ffcd5 Address:15bd09e8-59a7-5b1b-e064-577b992ffcd5}]" -testLoggerAdapter verbose: 30dcdab0-c001-1e95-aecd-133fdbcc3d32: 2024-09-03T21:34:35.537Z [INFO] 30dcdab0-c001-1e95-aecd-133fdbcc3d32: removed peer, stopping replication: peer=02b9cf67-0de4-2563-5880-5f6e258a4016 last-index=4 - testing.go:149: 30dcdab0-c001-1e95-aecd-133fdbcc3d32: 2024-09-03T21:34:35.537Z [INFO] 30dcdab0-c001-1e95-aecd-133fdbcc3d32: removed peer, stopping replication: peer=02b9cf67-0de4-2563-5880-5f6e258a4016 last-index=4 -testLoggerAdapter verbose: 30dcdab0-c001-1e95-aecd-133fdbcc3d32: 2024-09-03T21:34:35.537Z [INFO] 30dcdab0-c001-1e95-aecd-133fdbcc3d32: aborting pipeline replication: peer="{Voter 02b9cf67-0de4-2563-5880-5f6e258a4016 02b9cf67-0de4-2563-5880-5f6e258a4016}" - testing.go:149: 30dcdab0-c001-1e95-aecd-133fdbcc3d32: 2024-09-03T21:34:35.537Z [INFO] 30dcdab0-c001-1e95-aecd-133fdbcc3d32: aborting pipeline replication: peer="{Voter 02b9cf67-0de4-2563-5880-5f6e258a4016 02b9cf67-0de4-2563-5880-5f6e258a4016}" -testLoggerAdapter verbose: 30dcdab0-c001-1e95-aecd-133fdbcc3d32: 2024-09-03T21:34:35.537Z [INFO] 30dcdab0-c001-1e95-aecd-133fdbcc3d32: aborting pipeline replication: peer="{Voter server-15bd09e8-59a7-5b1b-e064-577b992ffcd5 15bd09e8-59a7-5b1b-e064-577b992ffcd5}" - testing.go:149: 30dcdab0-c001-1e95-aecd-133fdbcc3d32: 2024-09-03T21:34:35.537Z [INFO] 30dcdab0-c001-1e95-aecd-133fdbcc3d32: aborting pipeline replication: peer="{Voter server-15bd09e8-59a7-5b1b-e064-577b992ffcd5 15bd09e8-59a7-5b1b-e064-577b992ffcd5}" ---- PASS: TestRaft_ProtocolVersion_Upgrade_2_3 (0.52s) +testLoggerAdapter verbose: 04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba: 2023-08-02T15:13:04.098Z [INFO] 04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba: initial configuration: index=1 servers="[{Suffrage:Voter ID:04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba Address:04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba} {Suffrage:Voter ID:814bc0ee-8056-34a8-4670-5bc20d0207e2 Address:814bc0ee-8056-34a8-4670-5bc20d0207e2}]" + testing.go:149: 04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba: 2023-08-02T15:13:04.098Z [INFO] 04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba: initial configuration: index=1 servers="[{Suffrage:Voter ID:04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba Address:04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba} {Suffrage:Voter ID:814bc0ee-8056-34a8-4670-5bc20d0207e2 Address:814bc0ee-8056-34a8-4670-5bc20d0207e2}]" +testLoggerAdapter verbose: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.098Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: initial configuration: index=1 servers="[{Suffrage:Voter ID:04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba Address:04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba} {Suffrage:Voter ID:814bc0ee-8056-34a8-4670-5bc20d0207e2 Address:814bc0ee-8056-34a8-4670-5bc20d0207e2}]" + testing.go:149: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.098Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: initial configuration: index=1 servers="[{Suffrage:Voter ID:04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba Address:04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba} {Suffrage:Voter ID:814bc0ee-8056-34a8-4670-5bc20d0207e2 Address:814bc0ee-8056-34a8-4670-5bc20d0207e2}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:04.098Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-08-02T15:13:04.098Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: 04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba: 2023-08-02T15:13:04.098Z [INFO] 04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba: entering follower state: follower="Node at 04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba [Follower]" leader= + testing.go:149: 04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba: 2023-08-02T15:13:04.098Z [INFO] 04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba: entering follower state: follower="Node at 04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba [Follower]" leader= +testLoggerAdapter verbose: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.098Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: entering follower state: follower="Node at 814bc0ee-8056-34a8-4670-5bc20d0207e2 [Follower]" leader= + testing.go:149: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.098Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: entering follower state: follower="Node at 814bc0ee-8056-34a8-4670-5bc20d0207e2 [Follower]" leader= +testLoggerAdapter verbose: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.162Z [WARN] 814bc0ee-8056-34a8-4670-5bc20d0207e2: heartbeat timeout reached, starting election: last-leader= + testing.go:149: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.162Z [WARN] 814bc0ee-8056-34a8-4670-5bc20d0207e2: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.162Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: entering candidate state: node="Node at 814bc0ee-8056-34a8-4670-5bc20d0207e2 [Candidate]" term=2 + testing.go:149: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.162Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: entering candidate state: node="Node at 814bc0ee-8056-34a8-4670-5bc20d0207e2 [Candidate]" term=2 +testLoggerAdapter verbose: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.162Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: election won: tally=2 + testing.go:149: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.162Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: election won: tally=2 +testLoggerAdapter verbose: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.162Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: entering leader state: leader="Node at 814bc0ee-8056-34a8-4670-5bc20d0207e2 [Leader]" + testing.go:149: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.162Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: entering leader state: leader="Node at 814bc0ee-8056-34a8-4670-5bc20d0207e2 [Leader]" +testLoggerAdapter verbose: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.162Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: added peer, starting replication: peer=04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba + testing.go:149: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.162Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: added peer, starting replication: peer=04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba +testLoggerAdapter verbose: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.163Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: pipelining replication: peer="{Voter 04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba 04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba}" + testing.go:149: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.163Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: pipelining replication: peer="{Voter 04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba 04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:04.268Z [INFO] stable state for Follower reached at 2023-08-02 15:13:04.16297693 +0000 UTC m=+16.258913266 (1 nodes), 64.292947ms from start of poll, 64.767257ms from cluster start. Timeout at 2023-08-02 15:13:04.268014184 +0000 UTC m=+16.363950657, 105.037391ms after stability + testing.go:149: cluster: 2023-08-02T15:13:04.268Z [INFO] stable state for Follower reached at 2023-08-02 15:13:04.16297693 +0000 UTC m=+16.258913266 (1 nodes), 64.292947ms from start of poll, 64.767257ms from cluster start. Timeout at 2023-08-02 15:13:04.268014184 +0000 UTC m=+16.363950657, 105.037391ms after stability +testLoggerAdapter verbose: server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f: 2023-08-02T15:13:04.268Z [INFO] server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f: initial configuration: index=0 servers=[] + testing.go:149: server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f: 2023-08-02T15:13:04.268Z [INFO] server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:04.268Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:13:04.268Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f: 2023-08-02T15:13:04.268Z [INFO] server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f: entering follower state: follower="Node at 5cad1438-9cdf-8bb0-9425-cab2bf1b077f [Follower]" leader= + testing.go:149: server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f: 2023-08-02T15:13:04.268Z [INFO] server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f: entering follower state: follower="Node at 5cad1438-9cdf-8bb0-9425-cab2bf1b077f [Follower]" leader= +testLoggerAdapter verbose: server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f: 2023-08-02T15:13:04.329Z [WARN] server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f: no known peers, aborting election + testing.go:149: server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f: 2023-08-02T15:13:04.329Z [WARN] server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f: no known peers, aborting election +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:04.373Z [INFO] stable state for Leader reached at 2023-08-02 15:13:04.268464324 +0000 UTC m=+16.364400667 (1 nodes), 931ns from start of poll, 170.254658ms from cluster start. Timeout at 2023-08-02 15:13:04.373597537 +0000 UTC m=+16.469533875, 105.133208ms after stability + testing.go:149: cluster: 2023-08-02T15:13:04.373Z [INFO] stable state for Leader reached at 2023-08-02 15:13:04.268464324 +0000 UTC m=+16.364400667 (1 nodes), 931ns from start of poll, 170.254658ms from cluster start. Timeout at 2023-08-02 15:13:04.373597537 +0000 UTC m=+16.469533875, 105.133208ms after stability +testLoggerAdapter verbose: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.373Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: updating configuration: command=AddStaging server-id=server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f server-addr=5cad1438-9cdf-8bb0-9425-cab2bf1b077f servers="[{Suffrage:Voter ID:04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba Address:04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba} {Suffrage:Voter ID:814bc0ee-8056-34a8-4670-5bc20d0207e2 Address:814bc0ee-8056-34a8-4670-5bc20d0207e2} {Suffrage:Voter ID:server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f Address:5cad1438-9cdf-8bb0-9425-cab2bf1b077f}]" + testing.go:149: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.373Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: updating configuration: command=AddStaging server-id=server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f server-addr=5cad1438-9cdf-8bb0-9425-cab2bf1b077f servers="[{Suffrage:Voter ID:04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba Address:04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba} {Suffrage:Voter ID:814bc0ee-8056-34a8-4670-5bc20d0207e2 Address:814bc0ee-8056-34a8-4670-5bc20d0207e2} {Suffrage:Voter ID:server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f Address:5cad1438-9cdf-8bb0-9425-cab2bf1b077f}]" +testLoggerAdapter verbose: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.373Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: added peer, starting replication: peer=server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f + testing.go:149: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.373Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: added peer, starting replication: peer=server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f +testLoggerAdapter verbose: server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f: 2023-08-02T15:13:04.374Z [WARN] server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f: failed to get previous log: previous-index=3 last-index=0 error="log not found" + testing.go:149: server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f: 2023-08-02T15:13:04.374Z [WARN] server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f: failed to get previous log: previous-index=3 last-index=0 error="log not found" +testLoggerAdapter verbose: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.374Z [WARN] 814bc0ee-8056-34a8-4670-5bc20d0207e2: appendEntries rejected, sending older logs: peer="{Voter server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f 5cad1438-9cdf-8bb0-9425-cab2bf1b077f}" next=1 + testing.go:149: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.374Z [WARN] 814bc0ee-8056-34a8-4670-5bc20d0207e2: appendEntries rejected, sending older logs: peer="{Voter server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f 5cad1438-9cdf-8bb0-9425-cab2bf1b077f}" next=1 +testLoggerAdapter verbose: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.374Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: pipelining replication: peer="{Voter server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f 5cad1438-9cdf-8bb0-9425-cab2bf1b077f}" + testing.go:149: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.374Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: pipelining replication: peer="{Voter server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f 5cad1438-9cdf-8bb0-9425-cab2bf1b077f}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:04.374Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:13:04.374Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:04.479Z [INFO] stable state for Leader reached at 2023-08-02 15:13:04.374400785 +0000 UTC m=+16.470337120 (1 nodes), 1.117µs from start of poll, 276.191111ms from cluster start. Timeout at 2023-08-02 15:13:04.479815172 +0000 UTC m=+16.575751509, 105.414389ms after stability + testing.go:149: cluster: 2023-08-02T15:13:04.479Z [INFO] stable state for Leader reached at 2023-08-02 15:13:04.374400785 +0000 UTC m=+16.470337120 (1 nodes), 1.117µs from start of poll, 276.191111ms from cluster start. Timeout at 2023-08-02 15:13:04.479815172 +0000 UTC m=+16.575751509, 105.414389ms after stability +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:04.479Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:13:04.479Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:04.586Z [INFO] stable state for Leader reached at 2023-08-02 15:13:04.480014834 +0000 UTC m=+16.575951161 (1 nodes), 1.162µs from start of poll, 381.805152ms from cluster start. Timeout at 2023-08-02 15:13:04.586653775 +0000 UTC m=+16.682590101, 106.63894ms after stability + testing.go:149: cluster: 2023-08-02T15:13:04.586Z [INFO] stable state for Leader reached at 2023-08-02 15:13:04.480014834 +0000 UTC m=+16.575951161 (1 nodes), 1.162µs from start of poll, 381.805152ms from cluster start. Timeout at 2023-08-02 15:13:04.586653775 +0000 UTC m=+16.682590101, 106.63894ms after stability +testLoggerAdapter verbose: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.586Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: updating configuration: command=RemoveServer server-id=04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba server-addr= servers="[{Suffrage:Voter ID:814bc0ee-8056-34a8-4670-5bc20d0207e2 Address:814bc0ee-8056-34a8-4670-5bc20d0207e2} {Suffrage:Voter ID:server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f Address:5cad1438-9cdf-8bb0-9425-cab2bf1b077f}]" + testing.go:149: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.586Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: updating configuration: command=RemoveServer server-id=04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba server-addr= servers="[{Suffrage:Voter ID:814bc0ee-8056-34a8-4670-5bc20d0207e2 Address:814bc0ee-8056-34a8-4670-5bc20d0207e2} {Suffrage:Voter ID:server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f Address:5cad1438-9cdf-8bb0-9425-cab2bf1b077f}]" +testLoggerAdapter verbose: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.586Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: removed peer, stopping replication: peer=04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba last-index=4 + testing.go:149: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.586Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: removed peer, stopping replication: peer=04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba last-index=4 +testLoggerAdapter verbose: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.586Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: aborting pipeline replication: peer="{Voter 04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba 04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba}" + testing.go:149: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.586Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: aborting pipeline replication: peer="{Voter 04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba 04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba}" +testLoggerAdapter verbose: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.587Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: aborting pipeline replication: peer="{Voter server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f 5cad1438-9cdf-8bb0-9425-cab2bf1b077f}" + testing.go:149: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.587Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: aborting pipeline replication: peer="{Voter server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f 5cad1438-9cdf-8bb0-9425-cab2bf1b077f}" +--- PASS: TestRaft_ProtocolVersion_Upgrade_2_3 (0.49s) === RUN TestRaft_LeadershipTransferInProgress --- PASS: TestRaft_LeadershipTransferInProgress (0.00s) === RUN TestRaft_LeadershipTransferPickServer --- PASS: TestRaft_LeadershipTransferPickServer (0.00s) === RUN TestRaft_LeadershipTransfer -testLoggerAdapter verbose: server-803f54fe-8263-7522-e9ca-8631451a1c81: 2024-09-03T21:34:35.538Z [INFO] server-803f54fe-8263-7522-e9ca-8631451a1c81: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-803f54fe-8263-7522-e9ca-8631451a1c81 Address:803f54fe-8263-7522-e9ca-8631451a1c81} {Suffrage:Voter ID:server-446790a7-6465-825e-4105-1925b54284ff Address:446790a7-6465-825e-4105-1925b54284ff} {Suffrage:Voter ID:server-9238d787-19c7-29f3-9a14-a50e4fc89246 Address:9238d787-19c7-29f3-9a14-a50e4fc89246}]" - testing.go:149: server-803f54fe-8263-7522-e9ca-8631451a1c81: 2024-09-03T21:34:35.538Z [INFO] server-803f54fe-8263-7522-e9ca-8631451a1c81: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-803f54fe-8263-7522-e9ca-8631451a1c81 Address:803f54fe-8263-7522-e9ca-8631451a1c81} {Suffrage:Voter ID:server-446790a7-6465-825e-4105-1925b54284ff Address:446790a7-6465-825e-4105-1925b54284ff} {Suffrage:Voter ID:server-9238d787-19c7-29f3-9a14-a50e4fc89246 Address:9238d787-19c7-29f3-9a14-a50e4fc89246}]" -testLoggerAdapter verbose: server-446790a7-6465-825e-4105-1925b54284ff: 2024-09-03T21:34:35.538Z [INFO] server-446790a7-6465-825e-4105-1925b54284ff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-803f54fe-8263-7522-e9ca-8631451a1c81 Address:803f54fe-8263-7522-e9ca-8631451a1c81} {Suffrage:Voter ID:server-446790a7-6465-825e-4105-1925b54284ff Address:446790a7-6465-825e-4105-1925b54284ff} {Suffrage:Voter ID:server-9238d787-19c7-29f3-9a14-a50e4fc89246 Address:9238d787-19c7-29f3-9a14-a50e4fc89246}]" - testing.go:149: server-446790a7-6465-825e-4105-1925b54284ff: 2024-09-03T21:34:35.538Z [INFO] server-446790a7-6465-825e-4105-1925b54284ff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-803f54fe-8263-7522-e9ca-8631451a1c81 Address:803f54fe-8263-7522-e9ca-8631451a1c81} {Suffrage:Voter ID:server-446790a7-6465-825e-4105-1925b54284ff Address:446790a7-6465-825e-4105-1925b54284ff} {Suffrage:Voter ID:server-9238d787-19c7-29f3-9a14-a50e4fc89246 Address:9238d787-19c7-29f3-9a14-a50e4fc89246}]" -testLoggerAdapter verbose: server-9238d787-19c7-29f3-9a14-a50e4fc89246: 2024-09-03T21:34:35.538Z [INFO] server-9238d787-19c7-29f3-9a14-a50e4fc89246: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-803f54fe-8263-7522-e9ca-8631451a1c81 Address:803f54fe-8263-7522-e9ca-8631451a1c81} {Suffrage:Voter ID:server-446790a7-6465-825e-4105-1925b54284ff Address:446790a7-6465-825e-4105-1925b54284ff} {Suffrage:Voter ID:server-9238d787-19c7-29f3-9a14-a50e4fc89246 Address:9238d787-19c7-29f3-9a14-a50e4fc89246}]" - testing.go:149: server-9238d787-19c7-29f3-9a14-a50e4fc89246: 2024-09-03T21:34:35.538Z [INFO] server-9238d787-19c7-29f3-9a14-a50e4fc89246: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-803f54fe-8263-7522-e9ca-8631451a1c81 Address:803f54fe-8263-7522-e9ca-8631451a1c81} {Suffrage:Voter ID:server-446790a7-6465-825e-4105-1925b54284ff Address:446790a7-6465-825e-4105-1925b54284ff} {Suffrage:Voter ID:server-9238d787-19c7-29f3-9a14-a50e4fc89246 Address:9238d787-19c7-29f3-9a14-a50e4fc89246}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:35.538Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:35.538Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-803f54fe-8263-7522-e9ca-8631451a1c81: 2024-09-03T21:34:35.538Z [INFO] server-803f54fe-8263-7522-e9ca-8631451a1c81: entering follower state: follower="Node at 803f54fe-8263-7522-e9ca-8631451a1c81 [Follower]" leader= - testing.go:149: server-803f54fe-8263-7522-e9ca-8631451a1c81: 2024-09-03T21:34:35.538Z [INFO] server-803f54fe-8263-7522-e9ca-8631451a1c81: entering follower state: follower="Node at 803f54fe-8263-7522-e9ca-8631451a1c81 [Follower]" leader= -testLoggerAdapter verbose: server-446790a7-6465-825e-4105-1925b54284ff: 2024-09-03T21:34:35.538Z [INFO] server-446790a7-6465-825e-4105-1925b54284ff: entering follower state: follower="Node at 446790a7-6465-825e-4105-1925b54284ff [Follower]" leader= - testing.go:149: server-446790a7-6465-825e-4105-1925b54284ff: 2024-09-03T21:34:35.538Z [INFO] server-446790a7-6465-825e-4105-1925b54284ff: entering follower state: follower="Node at 446790a7-6465-825e-4105-1925b54284ff [Follower]" leader= -testLoggerAdapter verbose: server-9238d787-19c7-29f3-9a14-a50e4fc89246: 2024-09-03T21:34:35.539Z [INFO] server-9238d787-19c7-29f3-9a14-a50e4fc89246: entering follower state: follower="Node at 9238d787-19c7-29f3-9a14-a50e4fc89246 [Follower]" leader= - testing.go:149: server-9238d787-19c7-29f3-9a14-a50e4fc89246: 2024-09-03T21:34:35.539Z [INFO] server-9238d787-19c7-29f3-9a14-a50e4fc89246: entering follower state: follower="Node at 9238d787-19c7-29f3-9a14-a50e4fc89246 [Follower]" leader= -testLoggerAdapter verbose: server-803f54fe-8263-7522-e9ca-8631451a1c81: 2024-09-03T21:34:35.612Z [WARN] server-803f54fe-8263-7522-e9ca-8631451a1c81: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-803f54fe-8263-7522-e9ca-8631451a1c81: 2024-09-03T21:34:35.612Z [WARN] server-803f54fe-8263-7522-e9ca-8631451a1c81: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-803f54fe-8263-7522-e9ca-8631451a1c81: 2024-09-03T21:34:35.612Z [INFO] server-803f54fe-8263-7522-e9ca-8631451a1c81: entering candidate state: node="Node at 803f54fe-8263-7522-e9ca-8631451a1c81 [Candidate]" term=2 - testing.go:149: server-803f54fe-8263-7522-e9ca-8631451a1c81: 2024-09-03T21:34:35.612Z [INFO] server-803f54fe-8263-7522-e9ca-8631451a1c81: entering candidate state: node="Node at 803f54fe-8263-7522-e9ca-8631451a1c81 [Candidate]" term=2 -testLoggerAdapter verbose: server-803f54fe-8263-7522-e9ca-8631451a1c81: 2024-09-03T21:34:35.612Z [INFO] server-803f54fe-8263-7522-e9ca-8631451a1c81: election won: tally=2 - testing.go:149: server-803f54fe-8263-7522-e9ca-8631451a1c81: 2024-09-03T21:34:35.612Z [INFO] server-803f54fe-8263-7522-e9ca-8631451a1c81: election won: tally=2 -testLoggerAdapter verbose: server-803f54fe-8263-7522-e9ca-8631451a1c81: 2024-09-03T21:34:35.612Z [INFO] server-803f54fe-8263-7522-e9ca-8631451a1c81: entering leader state: leader="Node at 803f54fe-8263-7522-e9ca-8631451a1c81 [Leader]" - testing.go:149: server-803f54fe-8263-7522-e9ca-8631451a1c81: 2024-09-03T21:34:35.612Z [INFO] server-803f54fe-8263-7522-e9ca-8631451a1c81: entering leader state: leader="Node at 803f54fe-8263-7522-e9ca-8631451a1c81 [Leader]" -testLoggerAdapter verbose: server-803f54fe-8263-7522-e9ca-8631451a1c81: 2024-09-03T21:34:35.612Z [INFO] server-803f54fe-8263-7522-e9ca-8631451a1c81: added peer, starting replication: peer=server-446790a7-6465-825e-4105-1925b54284ff - testing.go:149: server-803f54fe-8263-7522-e9ca-8631451a1c81: 2024-09-03T21:34:35.612Z [INFO] server-803f54fe-8263-7522-e9ca-8631451a1c81: added peer, starting replication: peer=server-446790a7-6465-825e-4105-1925b54284ff -testLoggerAdapter verbose: server-803f54fe-8263-7522-e9ca-8631451a1c81: 2024-09-03T21:34:35.612Z [INFO] server-803f54fe-8263-7522-e9ca-8631451a1c81: added peer, starting replication: peer=server-9238d787-19c7-29f3-9a14-a50e4fc89246 - testing.go:149: server-803f54fe-8263-7522-e9ca-8631451a1c81: 2024-09-03T21:34:35.612Z [INFO] server-803f54fe-8263-7522-e9ca-8631451a1c81: added peer, starting replication: peer=server-9238d787-19c7-29f3-9a14-a50e4fc89246 -testLoggerAdapter verbose: server-803f54fe-8263-7522-e9ca-8631451a1c81: 2024-09-03T21:34:35.613Z [INFO] server-803f54fe-8263-7522-e9ca-8631451a1c81: pipelining replication: peer="{Voter server-9238d787-19c7-29f3-9a14-a50e4fc89246 9238d787-19c7-29f3-9a14-a50e4fc89246}" - testing.go:149: server-803f54fe-8263-7522-e9ca-8631451a1c81: 2024-09-03T21:34:35.613Z [INFO] server-803f54fe-8263-7522-e9ca-8631451a1c81: pipelining replication: peer="{Voter server-9238d787-19c7-29f3-9a14-a50e4fc89246 9238d787-19c7-29f3-9a14-a50e4fc89246}" -testLoggerAdapter verbose: server-803f54fe-8263-7522-e9ca-8631451a1c81: 2024-09-03T21:34:35.613Z [INFO] server-803f54fe-8263-7522-e9ca-8631451a1c81: pipelining replication: peer="{Voter server-446790a7-6465-825e-4105-1925b54284ff 446790a7-6465-825e-4105-1925b54284ff}" - testing.go:149: server-803f54fe-8263-7522-e9ca-8631451a1c81: 2024-09-03T21:34:35.613Z [INFO] server-803f54fe-8263-7522-e9ca-8631451a1c81: pipelining replication: peer="{Voter server-446790a7-6465-825e-4105-1925b54284ff 446790a7-6465-825e-4105-1925b54284ff}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:35.718Z [INFO] stable state for Leader reached at 2024-09-03 21:34:35.6131921 +0000 UTC m=+16.659849230 (1 nodes), 74.332023ms from start of poll, 74.724594ms from cluster start. Timeout at 2024-09-03 21:34:35.718267906 +0000 UTC m=+16.764925040, 105.07581ms after stability - testing.go:149: cluster: 2024-09-03T21:34:35.718Z [INFO] stable state for Leader reached at 2024-09-03 21:34:35.6131921 +0000 UTC m=+16.659849230 (1 nodes), 74.332023ms from start of poll, 74.724594ms from cluster start. Timeout at 2024-09-03 21:34:35.718267906 +0000 UTC m=+16.764925040, 105.07581ms after stability -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:35.718Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:35.718Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:35.823Z [INFO] stable state for Leader reached at 2024-09-03 21:34:35.718415402 +0000 UTC m=+16.765072545 (1 nodes), 930ns from start of poll, 179.947909ms from cluster start. Timeout at 2024-09-03 21:34:35.823512249 +0000 UTC m=+16.870169392, 105.096847ms after stability - testing.go:149: cluster: 2024-09-03T21:34:35.823Z [INFO] stable state for Leader reached at 2024-09-03 21:34:35.718415402 +0000 UTC m=+16.765072545 (1 nodes), 930ns from start of poll, 179.947909ms from cluster start. Timeout at 2024-09-03 21:34:35.823512249 +0000 UTC m=+16.870169392, 105.096847ms after stability -testLoggerAdapter verbose: server-446790a7-6465-825e-4105-1925b54284ff: 2024-09-03T21:34:35.823Z [INFO] server-446790a7-6465-825e-4105-1925b54284ff: entering candidate state: node="Node at 446790a7-6465-825e-4105-1925b54284ff [Candidate]" term=3 - testing.go:149: server-446790a7-6465-825e-4105-1925b54284ff: 2024-09-03T21:34:35.823Z [INFO] server-446790a7-6465-825e-4105-1925b54284ff: entering candidate state: node="Node at 446790a7-6465-825e-4105-1925b54284ff [Candidate]" term=3 -testLoggerAdapter verbose: server-446790a7-6465-825e-4105-1925b54284ff: 2024-09-03T21:34:35.823Z [INFO] server-446790a7-6465-825e-4105-1925b54284ff: election won: tally=2 - testing.go:149: server-446790a7-6465-825e-4105-1925b54284ff: 2024-09-03T21:34:35.823Z [INFO] server-446790a7-6465-825e-4105-1925b54284ff: election won: tally=2 -testLoggerAdapter verbose: server-446790a7-6465-825e-4105-1925b54284ff: 2024-09-03T21:34:35.823Z [INFO] server-446790a7-6465-825e-4105-1925b54284ff: entering leader state: leader="Node at 446790a7-6465-825e-4105-1925b54284ff [Leader]" - testing.go:149: server-446790a7-6465-825e-4105-1925b54284ff: 2024-09-03T21:34:35.823Z [INFO] server-446790a7-6465-825e-4105-1925b54284ff: entering leader state: leader="Node at 446790a7-6465-825e-4105-1925b54284ff [Leader]" -testLoggerAdapter verbose: server-446790a7-6465-825e-4105-1925b54284ff: 2024-09-03T21:34:35.823Z [INFO] server-446790a7-6465-825e-4105-1925b54284ff: added peer, starting replication: peer=server-803f54fe-8263-7522-e9ca-8631451a1c81 - testing.go:149: server-446790a7-6465-825e-4105-1925b54284ff: 2024-09-03T21:34:35.823Z [INFO] server-446790a7-6465-825e-4105-1925b54284ff: added peer, starting replication: peer=server-803f54fe-8263-7522-e9ca-8631451a1c81 -testLoggerAdapter verbose: server-446790a7-6465-825e-4105-1925b54284ff: 2024-09-03T21:34:35.823Z [INFO] server-446790a7-6465-825e-4105-1925b54284ff: added peer, starting replication: peer=server-9238d787-19c7-29f3-9a14-a50e4fc89246 - testing.go:149: server-446790a7-6465-825e-4105-1925b54284ff: 2024-09-03T21:34:35.823Z [INFO] server-446790a7-6465-825e-4105-1925b54284ff: added peer, starting replication: peer=server-9238d787-19c7-29f3-9a14-a50e4fc89246 -testLoggerAdapter verbose: server-446790a7-6465-825e-4105-1925b54284ff: 2024-09-03T21:34:35.824Z [INFO] server-446790a7-6465-825e-4105-1925b54284ff: pipelining replication: peer="{Voter server-9238d787-19c7-29f3-9a14-a50e4fc89246 9238d787-19c7-29f3-9a14-a50e4fc89246}" - testing.go:149: server-446790a7-6465-825e-4105-1925b54284ff: 2024-09-03T21:34:35.824Z [INFO] server-446790a7-6465-825e-4105-1925b54284ff: pipelining replication: peer="{Voter server-9238d787-19c7-29f3-9a14-a50e4fc89246 9238d787-19c7-29f3-9a14-a50e4fc89246}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:35.824Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:35.824Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-803f54fe-8263-7522-e9ca-8631451a1c81: 2024-09-03T21:34:35.824Z [INFO] server-803f54fe-8263-7522-e9ca-8631451a1c81: entering follower state: follower="Node at 803f54fe-8263-7522-e9ca-8631451a1c81 [Follower]" leader= - testing.go:149: server-803f54fe-8263-7522-e9ca-8631451a1c81: 2024-09-03T21:34:35.824Z [INFO] server-803f54fe-8263-7522-e9ca-8631451a1c81: entering follower state: follower="Node at 803f54fe-8263-7522-e9ca-8631451a1c81 [Follower]" leader= -testLoggerAdapter verbose: server-803f54fe-8263-7522-e9ca-8631451a1c81: 2024-09-03T21:34:35.824Z [INFO] server-803f54fe-8263-7522-e9ca-8631451a1c81: aborting pipeline replication: peer="{Voter server-9238d787-19c7-29f3-9a14-a50e4fc89246 9238d787-19c7-29f3-9a14-a50e4fc89246}" - testing.go:149: server-803f54fe-8263-7522-e9ca-8631451a1c81: 2024-09-03T21:34:35.824Z [INFO] server-803f54fe-8263-7522-e9ca-8631451a1c81: aborting pipeline replication: peer="{Voter server-9238d787-19c7-29f3-9a14-a50e4fc89246 9238d787-19c7-29f3-9a14-a50e4fc89246}" -testLoggerAdapter verbose: server-446790a7-6465-825e-4105-1925b54284ff: 2024-09-03T21:34:35.824Z [INFO] server-446790a7-6465-825e-4105-1925b54284ff: pipelining replication: peer="{Voter server-803f54fe-8263-7522-e9ca-8631451a1c81 803f54fe-8263-7522-e9ca-8631451a1c81}" - testing.go:149: server-446790a7-6465-825e-4105-1925b54284ff: 2024-09-03T21:34:35.824Z [INFO] server-446790a7-6465-825e-4105-1925b54284ff: pipelining replication: peer="{Voter server-803f54fe-8263-7522-e9ca-8631451a1c81 803f54fe-8263-7522-e9ca-8631451a1c81}" -testLoggerAdapter verbose: server-803f54fe-8263-7522-e9ca-8631451a1c81: 2024-09-03T21:34:35.824Z [INFO] server-803f54fe-8263-7522-e9ca-8631451a1c81: aborting pipeline replication: peer="{Voter server-446790a7-6465-825e-4105-1925b54284ff 446790a7-6465-825e-4105-1925b54284ff}" - testing.go:149: server-803f54fe-8263-7522-e9ca-8631451a1c81: 2024-09-03T21:34:35.824Z [INFO] server-803f54fe-8263-7522-e9ca-8631451a1c81: aborting pipeline replication: peer="{Voter server-446790a7-6465-825e-4105-1925b54284ff 446790a7-6465-825e-4105-1925b54284ff}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:35.929Z [INFO] stable state for Leader reached at 2024-09-03 21:34:35.824387837 +0000 UTC m=+16.871044983 (1 nodes), 200.895µs from start of poll, 285.920347ms from cluster start. Timeout at 2024-09-03 21:34:35.929391378 +0000 UTC m=+16.976048521, 105.003538ms after stability - testing.go:149: cluster: 2024-09-03T21:34:35.929Z [INFO] stable state for Leader reached at 2024-09-03 21:34:35.824387837 +0000 UTC m=+16.871044983 (1 nodes), 200.895µs from start of poll, 285.920347ms from cluster start. Timeout at 2024-09-03 21:34:35.929391378 +0000 UTC m=+16.976048521, 105.003538ms after stability -testLoggerAdapter verbose: server-446790a7-6465-825e-4105-1925b54284ff: 2024-09-03T21:34:35.929Z [INFO] server-446790a7-6465-825e-4105-1925b54284ff: aborting pipeline replication: peer="{Voter server-9238d787-19c7-29f3-9a14-a50e4fc89246 9238d787-19c7-29f3-9a14-a50e4fc89246}" - testing.go:149: server-446790a7-6465-825e-4105-1925b54284ff: 2024-09-03T21:34:35.929Z [INFO] server-446790a7-6465-825e-4105-1925b54284ff: aborting pipeline replication: peer="{Voter server-9238d787-19c7-29f3-9a14-a50e4fc89246 9238d787-19c7-29f3-9a14-a50e4fc89246}" -testLoggerAdapter verbose: server-446790a7-6465-825e-4105-1925b54284ff: 2024-09-03T21:34:35.929Z [INFO] server-446790a7-6465-825e-4105-1925b54284ff: aborting pipeline replication: peer="{Voter server-803f54fe-8263-7522-e9ca-8631451a1c81 803f54fe-8263-7522-e9ca-8631451a1c81}" - testing.go:149: server-446790a7-6465-825e-4105-1925b54284ff: 2024-09-03T21:34:35.929Z [INFO] server-446790a7-6465-825e-4105-1925b54284ff: aborting pipeline replication: peer="{Voter server-803f54fe-8263-7522-e9ca-8631451a1c81 803f54fe-8263-7522-e9ca-8631451a1c81}" +testLoggerAdapter verbose: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.588Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-00c30f88-6bbe-af60-55fc-a32a9becb5d6 Address:00c30f88-6bbe-af60-55fc-a32a9becb5d6} {Suffrage:Voter ID:server-dd168698-36f7-4ac1-79e0-e76b3df6e677 Address:dd168698-36f7-4ac1-79e0-e76b3df6e677} {Suffrage:Voter ID:server-7c31cd11-c815-8385-066c-72f1002d3639 Address:7c31cd11-c815-8385-066c-72f1002d3639}]" + testing.go:149: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.588Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-00c30f88-6bbe-af60-55fc-a32a9becb5d6 Address:00c30f88-6bbe-af60-55fc-a32a9becb5d6} {Suffrage:Voter ID:server-dd168698-36f7-4ac1-79e0-e76b3df6e677 Address:dd168698-36f7-4ac1-79e0-e76b3df6e677} {Suffrage:Voter ID:server-7c31cd11-c815-8385-066c-72f1002d3639 Address:7c31cd11-c815-8385-066c-72f1002d3639}]" +testLoggerAdapter verbose: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.588Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-00c30f88-6bbe-af60-55fc-a32a9becb5d6 Address:00c30f88-6bbe-af60-55fc-a32a9becb5d6} {Suffrage:Voter ID:server-dd168698-36f7-4ac1-79e0-e76b3df6e677 Address:dd168698-36f7-4ac1-79e0-e76b3df6e677} {Suffrage:Voter ID:server-7c31cd11-c815-8385-066c-72f1002d3639 Address:7c31cd11-c815-8385-066c-72f1002d3639}]" + testing.go:149: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.588Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-00c30f88-6bbe-af60-55fc-a32a9becb5d6 Address:00c30f88-6bbe-af60-55fc-a32a9becb5d6} {Suffrage:Voter ID:server-dd168698-36f7-4ac1-79e0-e76b3df6e677 Address:dd168698-36f7-4ac1-79e0-e76b3df6e677} {Suffrage:Voter ID:server-7c31cd11-c815-8385-066c-72f1002d3639 Address:7c31cd11-c815-8385-066c-72f1002d3639}]" +testLoggerAdapter verbose: server-7c31cd11-c815-8385-066c-72f1002d3639: 2023-08-02T15:13:04.588Z [INFO] server-7c31cd11-c815-8385-066c-72f1002d3639: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-00c30f88-6bbe-af60-55fc-a32a9becb5d6 Address:00c30f88-6bbe-af60-55fc-a32a9becb5d6} {Suffrage:Voter ID:server-dd168698-36f7-4ac1-79e0-e76b3df6e677 Address:dd168698-36f7-4ac1-79e0-e76b3df6e677} {Suffrage:Voter ID:server-7c31cd11-c815-8385-066c-72f1002d3639 Address:7c31cd11-c815-8385-066c-72f1002d3639}]" + testing.go:149: server-7c31cd11-c815-8385-066c-72f1002d3639: 2023-08-02T15:13:04.588Z [INFO] server-7c31cd11-c815-8385-066c-72f1002d3639: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-00c30f88-6bbe-af60-55fc-a32a9becb5d6 Address:00c30f88-6bbe-af60-55fc-a32a9becb5d6} {Suffrage:Voter ID:server-dd168698-36f7-4ac1-79e0-e76b3df6e677 Address:dd168698-36f7-4ac1-79e0-e76b3df6e677} {Suffrage:Voter ID:server-7c31cd11-c815-8385-066c-72f1002d3639 Address:7c31cd11-c815-8385-066c-72f1002d3639}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:04.588Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:13:04.588Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.588Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: entering follower state: follower="Node at 00c30f88-6bbe-af60-55fc-a32a9becb5d6 [Follower]" leader= + testing.go:149: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.588Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: entering follower state: follower="Node at 00c30f88-6bbe-af60-55fc-a32a9becb5d6 [Follower]" leader= +testLoggerAdapter verbose: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.588Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: entering follower state: follower="Node at dd168698-36f7-4ac1-79e0-e76b3df6e677 [Follower]" leader= + testing.go:149: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.588Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: entering follower state: follower="Node at dd168698-36f7-4ac1-79e0-e76b3df6e677 [Follower]" leader= +testLoggerAdapter verbose: server-7c31cd11-c815-8385-066c-72f1002d3639: 2023-08-02T15:13:04.589Z [INFO] server-7c31cd11-c815-8385-066c-72f1002d3639: entering follower state: follower="Node at 7c31cd11-c815-8385-066c-72f1002d3639 [Follower]" leader= + testing.go:149: server-7c31cd11-c815-8385-066c-72f1002d3639: 2023-08-02T15:13:04.589Z [INFO] server-7c31cd11-c815-8385-066c-72f1002d3639: entering follower state: follower="Node at 7c31cd11-c815-8385-066c-72f1002d3639 [Follower]" leader= +testLoggerAdapter verbose: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.656Z [WARN] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.656Z [WARN] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.657Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: entering candidate state: node="Node at dd168698-36f7-4ac1-79e0-e76b3df6e677 [Candidate]" term=2 + testing.go:149: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.657Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: entering candidate state: node="Node at dd168698-36f7-4ac1-79e0-e76b3df6e677 [Candidate]" term=2 +testLoggerAdapter verbose: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.657Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: election won: tally=2 + testing.go:149: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.657Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: election won: tally=2 +testLoggerAdapter verbose: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.657Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: entering leader state: leader="Node at dd168698-36f7-4ac1-79e0-e76b3df6e677 [Leader]" + testing.go:149: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.657Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: entering leader state: leader="Node at dd168698-36f7-4ac1-79e0-e76b3df6e677 [Leader]" +testLoggerAdapter verbose: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.657Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: added peer, starting replication: peer=server-00c30f88-6bbe-af60-55fc-a32a9becb5d6 + testing.go:149: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.657Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: added peer, starting replication: peer=server-00c30f88-6bbe-af60-55fc-a32a9becb5d6 +testLoggerAdapter verbose: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.657Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: added peer, starting replication: peer=server-7c31cd11-c815-8385-066c-72f1002d3639 + testing.go:149: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.657Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: added peer, starting replication: peer=server-7c31cd11-c815-8385-066c-72f1002d3639 +testLoggerAdapter verbose: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.657Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: pipelining replication: peer="{Voter server-7c31cd11-c815-8385-066c-72f1002d3639 7c31cd11-c815-8385-066c-72f1002d3639}" + testing.go:149: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.657Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: pipelining replication: peer="{Voter server-7c31cd11-c815-8385-066c-72f1002d3639 7c31cd11-c815-8385-066c-72f1002d3639}" +testLoggerAdapter verbose: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.657Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: pipelining replication: peer="{Voter server-00c30f88-6bbe-af60-55fc-a32a9becb5d6 00c30f88-6bbe-af60-55fc-a32a9becb5d6}" + testing.go:149: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.657Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: pipelining replication: peer="{Voter server-00c30f88-6bbe-af60-55fc-a32a9becb5d6 00c30f88-6bbe-af60-55fc-a32a9becb5d6}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:04.762Z [INFO] stable state for Leader reached at 2023-08-02 15:13:04.657614099 +0000 UTC m=+16.753550449 (1 nodes), 68.735688ms from start of poll, 69.305716ms from cluster start. Timeout at 2023-08-02 15:13:04.76272211 +0000 UTC m=+16.858658449, 105.108ms after stability + testing.go:149: cluster: 2023-08-02T15:13:04.762Z [INFO] stable state for Leader reached at 2023-08-02 15:13:04.657614099 +0000 UTC m=+16.753550449 (1 nodes), 68.735688ms from start of poll, 69.305716ms from cluster start. Timeout at 2023-08-02 15:13:04.76272211 +0000 UTC m=+16.858658449, 105.108ms after stability +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:04.762Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:13:04.762Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:04.867Z [INFO] stable state for Leader reached at 2023-08-02 15:13:04.762905299 +0000 UTC m=+16.858841623 (1 nodes), 1.375µs from start of poll, 174.59689ms from cluster start. Timeout at 2023-08-02 15:13:04.867933101 +0000 UTC m=+16.963869428, 105.027805ms after stability + testing.go:149: cluster: 2023-08-02T15:13:04.867Z [INFO] stable state for Leader reached at 2023-08-02 15:13:04.762905299 +0000 UTC m=+16.858841623 (1 nodes), 1.375µs from start of poll, 174.59689ms from cluster start. Timeout at 2023-08-02 15:13:04.867933101 +0000 UTC m=+16.963869428, 105.027805ms after stability +testLoggerAdapter verbose: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.868Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: entering candidate state: node="Node at 00c30f88-6bbe-af60-55fc-a32a9becb5d6 [Candidate]" term=3 + testing.go:149: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.868Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: entering candidate state: node="Node at 00c30f88-6bbe-af60-55fc-a32a9becb5d6 [Candidate]" term=3 +testLoggerAdapter verbose: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.868Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: election won: tally=2 + testing.go:149: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.868Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: election won: tally=2 +testLoggerAdapter verbose: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.868Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: entering leader state: leader="Node at 00c30f88-6bbe-af60-55fc-a32a9becb5d6 [Leader]" + testing.go:149: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.868Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: entering leader state: leader="Node at 00c30f88-6bbe-af60-55fc-a32a9becb5d6 [Leader]" +testLoggerAdapter verbose: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.868Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: added peer, starting replication: peer=server-dd168698-36f7-4ac1-79e0-e76b3df6e677 + testing.go:149: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.868Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: added peer, starting replication: peer=server-dd168698-36f7-4ac1-79e0-e76b3df6e677 +testLoggerAdapter verbose: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.868Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: added peer, starting replication: peer=server-7c31cd11-c815-8385-066c-72f1002d3639 + testing.go:149: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.868Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: added peer, starting replication: peer=server-7c31cd11-c815-8385-066c-72f1002d3639 +testLoggerAdapter verbose: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.868Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: pipelining replication: peer="{Voter server-7c31cd11-c815-8385-066c-72f1002d3639 7c31cd11-c815-8385-066c-72f1002d3639}" + testing.go:149: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.868Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: pipelining replication: peer="{Voter server-7c31cd11-c815-8385-066c-72f1002d3639 7c31cd11-c815-8385-066c-72f1002d3639}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:04.868Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:13:04.868Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.868Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: entering follower state: follower="Node at dd168698-36f7-4ac1-79e0-e76b3df6e677 [Follower]" leader= + testing.go:149: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.868Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: entering follower state: follower="Node at dd168698-36f7-4ac1-79e0-e76b3df6e677 [Follower]" leader= +testLoggerAdapter verbose: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.868Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: aborting pipeline replication: peer="{Voter server-7c31cd11-c815-8385-066c-72f1002d3639 7c31cd11-c815-8385-066c-72f1002d3639}" + testing.go:149: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.868Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: aborting pipeline replication: peer="{Voter server-7c31cd11-c815-8385-066c-72f1002d3639 7c31cd11-c815-8385-066c-72f1002d3639}" +testLoggerAdapter verbose: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.868Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: pipelining replication: peer="{Voter server-dd168698-36f7-4ac1-79e0-e76b3df6e677 dd168698-36f7-4ac1-79e0-e76b3df6e677}" + testing.go:149: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.868Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: pipelining replication: peer="{Voter server-dd168698-36f7-4ac1-79e0-e76b3df6e677 dd168698-36f7-4ac1-79e0-e76b3df6e677}" +testLoggerAdapter verbose: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.868Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: aborting pipeline replication: peer="{Voter server-00c30f88-6bbe-af60-55fc-a32a9becb5d6 00c30f88-6bbe-af60-55fc-a32a9becb5d6}" + testing.go:149: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.868Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: aborting pipeline replication: peer="{Voter server-00c30f88-6bbe-af60-55fc-a32a9becb5d6 00c30f88-6bbe-af60-55fc-a32a9becb5d6}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:04.973Z [INFO] stable state for Leader reached at 2023-08-02 15:13:04.868673705 +0000 UTC m=+16.964610040 (1 nodes), 231.317µs from start of poll, 280.365307ms from cluster start. Timeout at 2023-08-02 15:13:04.973740457 +0000 UTC m=+17.069676817, 105.066777ms after stability + testing.go:149: cluster: 2023-08-02T15:13:04.973Z [INFO] stable state for Leader reached at 2023-08-02 15:13:04.868673705 +0000 UTC m=+16.964610040 (1 nodes), 231.317µs from start of poll, 280.365307ms from cluster start. Timeout at 2023-08-02 15:13:04.973740457 +0000 UTC m=+17.069676817, 105.066777ms after stability +testLoggerAdapter verbose: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.973Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: aborting pipeline replication: peer="{Voter server-7c31cd11-c815-8385-066c-72f1002d3639 7c31cd11-c815-8385-066c-72f1002d3639}" + testing.go:149: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.973Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: aborting pipeline replication: peer="{Voter server-7c31cd11-c815-8385-066c-72f1002d3639 7c31cd11-c815-8385-066c-72f1002d3639}" +testLoggerAdapter verbose: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.974Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: aborting pipeline replication: peer="{Voter server-dd168698-36f7-4ac1-79e0-e76b3df6e677 dd168698-36f7-4ac1-79e0-e76b3df6e677}" + testing.go:149: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.974Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: aborting pipeline replication: peer="{Voter server-dd168698-36f7-4ac1-79e0-e76b3df6e677 dd168698-36f7-4ac1-79e0-e76b3df6e677}" --- PASS: TestRaft_LeadershipTransfer (0.39s) === RUN TestRaft_LeadershipTransferWithOneNode -testLoggerAdapter verbose: server-ee03442d-c6e3-c252-b256-305da4fa79bf: 2024-09-03T21:34:35.930Z [INFO] server-ee03442d-c6e3-c252-b256-305da4fa79bf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ee03442d-c6e3-c252-b256-305da4fa79bf Address:ee03442d-c6e3-c252-b256-305da4fa79bf}]" - testing.go:149: server-ee03442d-c6e3-c252-b256-305da4fa79bf: 2024-09-03T21:34:35.930Z [INFO] server-ee03442d-c6e3-c252-b256-305da4fa79bf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ee03442d-c6e3-c252-b256-305da4fa79bf Address:ee03442d-c6e3-c252-b256-305da4fa79bf}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:35.930Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:35.930Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-ee03442d-c6e3-c252-b256-305da4fa79bf: 2024-09-03T21:34:35.930Z [INFO] server-ee03442d-c6e3-c252-b256-305da4fa79bf: entering follower state: follower="Node at ee03442d-c6e3-c252-b256-305da4fa79bf [Follower]" leader= - testing.go:149: server-ee03442d-c6e3-c252-b256-305da4fa79bf: 2024-09-03T21:34:35.930Z [INFO] server-ee03442d-c6e3-c252-b256-305da4fa79bf: entering follower state: follower="Node at ee03442d-c6e3-c252-b256-305da4fa79bf [Follower]" leader= -testLoggerAdapter verbose: server-ee03442d-c6e3-c252-b256-305da4fa79bf: 2024-09-03T21:34:36.009Z [WARN] server-ee03442d-c6e3-c252-b256-305da4fa79bf: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-ee03442d-c6e3-c252-b256-305da4fa79bf: 2024-09-03T21:34:36.009Z [WARN] server-ee03442d-c6e3-c252-b256-305da4fa79bf: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-ee03442d-c6e3-c252-b256-305da4fa79bf: 2024-09-03T21:34:36.009Z [INFO] server-ee03442d-c6e3-c252-b256-305da4fa79bf: entering candidate state: node="Node at ee03442d-c6e3-c252-b256-305da4fa79bf [Candidate]" term=2 - testing.go:149: server-ee03442d-c6e3-c252-b256-305da4fa79bf: 2024-09-03T21:34:36.009Z [INFO] server-ee03442d-c6e3-c252-b256-305da4fa79bf: entering candidate state: node="Node at ee03442d-c6e3-c252-b256-305da4fa79bf [Candidate]" term=2 -testLoggerAdapter verbose: server-ee03442d-c6e3-c252-b256-305da4fa79bf: 2024-09-03T21:34:36.009Z [INFO] server-ee03442d-c6e3-c252-b256-305da4fa79bf: election won: tally=1 - testing.go:149: server-ee03442d-c6e3-c252-b256-305da4fa79bf: 2024-09-03T21:34:36.009Z [INFO] server-ee03442d-c6e3-c252-b256-305da4fa79bf: election won: tally=1 -testLoggerAdapter verbose: server-ee03442d-c6e3-c252-b256-305da4fa79bf: 2024-09-03T21:34:36.009Z [INFO] server-ee03442d-c6e3-c252-b256-305da4fa79bf: entering leader state: leader="Node at ee03442d-c6e3-c252-b256-305da4fa79bf [Leader]" - testing.go:149: server-ee03442d-c6e3-c252-b256-305da4fa79bf: 2024-09-03T21:34:36.009Z [INFO] server-ee03442d-c6e3-c252-b256-305da4fa79bf: entering leader state: leader="Node at ee03442d-c6e3-c252-b256-305da4fa79bf [Leader]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:36.115Z [INFO] stable state for Leader reached at 2024-09-03 21:34:36.010012821 +0000 UTC m=+17.056669953 (1 nodes), 79.714438ms from start of poll, 79.888535ms from cluster start. Timeout at 2024-09-03 21:34:36.115093756 +0000 UTC m=+17.161750901, 105.080948ms after stability - testing.go:149: cluster: 2024-09-03T21:34:36.115Z [INFO] stable state for Leader reached at 2024-09-03 21:34:36.010012821 +0000 UTC m=+17.056669953 (1 nodes), 79.714438ms from start of poll, 79.888535ms from cluster start. Timeout at 2024-09-03 21:34:36.115093756 +0000 UTC m=+17.161750901, 105.080948ms after stability +testLoggerAdapter verbose: server-49667621-7452-7d38-0601-b1f5e9ec633d: 2023-08-02T15:13:04.974Z [INFO] server-49667621-7452-7d38-0601-b1f5e9ec633d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49667621-7452-7d38-0601-b1f5e9ec633d Address:49667621-7452-7d38-0601-b1f5e9ec633d}]" + testing.go:149: server-49667621-7452-7d38-0601-b1f5e9ec633d: 2023-08-02T15:13:04.974Z [INFO] server-49667621-7452-7d38-0601-b1f5e9ec633d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49667621-7452-7d38-0601-b1f5e9ec633d Address:49667621-7452-7d38-0601-b1f5e9ec633d}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:04.974Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:13:04.974Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-49667621-7452-7d38-0601-b1f5e9ec633d: 2023-08-02T15:13:04.975Z [INFO] server-49667621-7452-7d38-0601-b1f5e9ec633d: entering follower state: follower="Node at 49667621-7452-7d38-0601-b1f5e9ec633d [Follower]" leader= + testing.go:149: server-49667621-7452-7d38-0601-b1f5e9ec633d: 2023-08-02T15:13:04.975Z [INFO] server-49667621-7452-7d38-0601-b1f5e9ec633d: entering follower state: follower="Node at 49667621-7452-7d38-0601-b1f5e9ec633d [Follower]" leader= +testLoggerAdapter verbose: server-49667621-7452-7d38-0601-b1f5e9ec633d: 2023-08-02T15:13:05.062Z [WARN] server-49667621-7452-7d38-0601-b1f5e9ec633d: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-49667621-7452-7d38-0601-b1f5e9ec633d: 2023-08-02T15:13:05.062Z [WARN] server-49667621-7452-7d38-0601-b1f5e9ec633d: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-49667621-7452-7d38-0601-b1f5e9ec633d: 2023-08-02T15:13:05.062Z [INFO] server-49667621-7452-7d38-0601-b1f5e9ec633d: entering candidate state: node="Node at 49667621-7452-7d38-0601-b1f5e9ec633d [Candidate]" term=2 + testing.go:149: server-49667621-7452-7d38-0601-b1f5e9ec633d: 2023-08-02T15:13:05.062Z [INFO] server-49667621-7452-7d38-0601-b1f5e9ec633d: entering candidate state: node="Node at 49667621-7452-7d38-0601-b1f5e9ec633d [Candidate]" term=2 +testLoggerAdapter verbose: server-49667621-7452-7d38-0601-b1f5e9ec633d: 2023-08-02T15:13:05.062Z [INFO] server-49667621-7452-7d38-0601-b1f5e9ec633d: election won: tally=1 + testing.go:149: server-49667621-7452-7d38-0601-b1f5e9ec633d: 2023-08-02T15:13:05.062Z [INFO] server-49667621-7452-7d38-0601-b1f5e9ec633d: election won: tally=1 +testLoggerAdapter verbose: server-49667621-7452-7d38-0601-b1f5e9ec633d: 2023-08-02T15:13:05.062Z [INFO] server-49667621-7452-7d38-0601-b1f5e9ec633d: entering leader state: leader="Node at 49667621-7452-7d38-0601-b1f5e9ec633d [Leader]" + testing.go:149: server-49667621-7452-7d38-0601-b1f5e9ec633d: 2023-08-02T15:13:05.062Z [INFO] server-49667621-7452-7d38-0601-b1f5e9ec633d: entering leader state: leader="Node at 49667621-7452-7d38-0601-b1f5e9ec633d [Leader]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:05.168Z [INFO] stable state for Leader reached at 2023-08-02 15:13:05.062802441 +0000 UTC m=+17.158738773 (1 nodes), 87.784567ms from start of poll, 88.086061ms from cluster start. Timeout at 2023-08-02 15:13:05.168717691 +0000 UTC m=+17.264654016, 105.915243ms after stability + testing.go:149: cluster: 2023-08-02T15:13:05.168Z [INFO] stable state for Leader reached at 2023-08-02 15:13:05.062802441 +0000 UTC m=+17.158738773 (1 nodes), 87.784567ms from start of poll, 88.086061ms from cluster start. Timeout at 2023-08-02 15:13:05.168717691 +0000 UTC m=+17.264654016, 105.915243ms after stability --- PASS: TestRaft_LeadershipTransferWithOneNode (0.19s) === RUN TestRaft_LeadershipTransferWithSevenNodes -testLoggerAdapter verbose: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.118Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-12a9c79e-5a12-f014-115b-422de1be8ed9 Address:12a9c79e-5a12-f014-115b-422de1be8ed9} {Suffrage:Voter ID:server-5939a590-b9d0-5f82-4521-a2c243521e3e Address:5939a590-b9d0-5f82-4521-a2c243521e3e} {Suffrage:Voter ID:server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b Address:6eedb621-0cfe-58c0-f11c-90d9e3d9e56b} {Suffrage:Voter ID:server-6577524e-5102-c512-ed2b-ee8108ec66fa Address:6577524e-5102-c512-ed2b-ee8108ec66fa} {Suffrage:Voter ID:server-b861555d-211f-cb23-d74e-592c82f26832 Address:b861555d-211f-cb23-d74e-592c82f26832} {Suffrage:Voter ID:server-afa0751b-4c32-c2fc-791e-b84ad64fab14 Address:afa0751b-4c32-c2fc-791e-b84ad64fab14} {Suffrage:Voter ID:server-38e08373-3b93-60c8-f46e-04d3969c9a8b Address:38e08373-3b93-60c8-f46e-04d3969c9a8b}]" - testing.go:149: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.118Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-12a9c79e-5a12-f014-115b-422de1be8ed9 Address:12a9c79e-5a12-f014-115b-422de1be8ed9} {Suffrage:Voter ID:server-5939a590-b9d0-5f82-4521-a2c243521e3e Address:5939a590-b9d0-5f82-4521-a2c243521e3e} {Suffrage:Voter ID:server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b Address:6eedb621-0cfe-58c0-f11c-90d9e3d9e56b} {Suffrage:Voter ID:server-6577524e-5102-c512-ed2b-ee8108ec66fa Address:6577524e-5102-c512-ed2b-ee8108ec66fa} {Suffrage:Voter ID:server-b861555d-211f-cb23-d74e-592c82f26832 Address:b861555d-211f-cb23-d74e-592c82f26832} {Suffrage:Voter ID:server-afa0751b-4c32-c2fc-791e-b84ad64fab14 Address:afa0751b-4c32-c2fc-791e-b84ad64fab14} {Suffrage:Voter ID:server-38e08373-3b93-60c8-f46e-04d3969c9a8b Address:38e08373-3b93-60c8-f46e-04d3969c9a8b}]" -testLoggerAdapter verbose: server-5939a590-b9d0-5f82-4521-a2c243521e3e: 2024-09-03T21:34:36.118Z [INFO] server-5939a590-b9d0-5f82-4521-a2c243521e3e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-12a9c79e-5a12-f014-115b-422de1be8ed9 Address:12a9c79e-5a12-f014-115b-422de1be8ed9} {Suffrage:Voter ID:server-5939a590-b9d0-5f82-4521-a2c243521e3e Address:5939a590-b9d0-5f82-4521-a2c243521e3e} {Suffrage:Voter ID:server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b Address:6eedb621-0cfe-58c0-f11c-90d9e3d9e56b} {Suffrage:Voter ID:server-6577524e-5102-c512-ed2b-ee8108ec66fa Address:6577524e-5102-c512-ed2b-ee8108ec66fa} {Suffrage:Voter ID:server-b861555d-211f-cb23-d74e-592c82f26832 Address:b861555d-211f-cb23-d74e-592c82f26832} {Suffrage:Voter ID:server-afa0751b-4c32-c2fc-791e-b84ad64fab14 Address:afa0751b-4c32-c2fc-791e-b84ad64fab14} {Suffrage:Voter ID:server-38e08373-3b93-60c8-f46e-04d3969c9a8b Address:38e08373-3b93-60c8-f46e-04d3969c9a8b}]" - testing.go:149: server-5939a590-b9d0-5f82-4521-a2c243521e3e: 2024-09-03T21:34:36.118Z [INFO] server-5939a590-b9d0-5f82-4521-a2c243521e3e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-12a9c79e-5a12-f014-115b-422de1be8ed9 Address:12a9c79e-5a12-f014-115b-422de1be8ed9} {Suffrage:Voter ID:server-5939a590-b9d0-5f82-4521-a2c243521e3e Address:5939a590-b9d0-5f82-4521-a2c243521e3e} {Suffrage:Voter ID:server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b Address:6eedb621-0cfe-58c0-f11c-90d9e3d9e56b} {Suffrage:Voter ID:server-6577524e-5102-c512-ed2b-ee8108ec66fa Address:6577524e-5102-c512-ed2b-ee8108ec66fa} {Suffrage:Voter ID:server-b861555d-211f-cb23-d74e-592c82f26832 Address:b861555d-211f-cb23-d74e-592c82f26832} {Suffrage:Voter ID:server-afa0751b-4c32-c2fc-791e-b84ad64fab14 Address:afa0751b-4c32-c2fc-791e-b84ad64fab14} {Suffrage:Voter ID:server-38e08373-3b93-60c8-f46e-04d3969c9a8b Address:38e08373-3b93-60c8-f46e-04d3969c9a8b}]" -testLoggerAdapter verbose: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.118Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-12a9c79e-5a12-f014-115b-422de1be8ed9 Address:12a9c79e-5a12-f014-115b-422de1be8ed9} {Suffrage:Voter ID:server-5939a590-b9d0-5f82-4521-a2c243521e3e Address:5939a590-b9d0-5f82-4521-a2c243521e3e} {Suffrage:Voter ID:server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b Address:6eedb621-0cfe-58c0-f11c-90d9e3d9e56b} {Suffrage:Voter ID:server-6577524e-5102-c512-ed2b-ee8108ec66fa Address:6577524e-5102-c512-ed2b-ee8108ec66fa} {Suffrage:Voter ID:server-b861555d-211f-cb23-d74e-592c82f26832 Address:b861555d-211f-cb23-d74e-592c82f26832} {Suffrage:Voter ID:server-afa0751b-4c32-c2fc-791e-b84ad64fab14 Address:afa0751b-4c32-c2fc-791e-b84ad64fab14} {Suffrage:Voter ID:server-38e08373-3b93-60c8-f46e-04d3969c9a8b Address:38e08373-3b93-60c8-f46e-04d3969c9a8b}]" - testing.go:149: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.118Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-12a9c79e-5a12-f014-115b-422de1be8ed9 Address:12a9c79e-5a12-f014-115b-422de1be8ed9} {Suffrage:Voter ID:server-5939a590-b9d0-5f82-4521-a2c243521e3e Address:5939a590-b9d0-5f82-4521-a2c243521e3e} {Suffrage:Voter ID:server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b Address:6eedb621-0cfe-58c0-f11c-90d9e3d9e56b} {Suffrage:Voter ID:server-6577524e-5102-c512-ed2b-ee8108ec66fa Address:6577524e-5102-c512-ed2b-ee8108ec66fa} {Suffrage:Voter ID:server-b861555d-211f-cb23-d74e-592c82f26832 Address:b861555d-211f-cb23-d74e-592c82f26832} {Suffrage:Voter ID:server-afa0751b-4c32-c2fc-791e-b84ad64fab14 Address:afa0751b-4c32-c2fc-791e-b84ad64fab14} {Suffrage:Voter ID:server-38e08373-3b93-60c8-f46e-04d3969c9a8b Address:38e08373-3b93-60c8-f46e-04d3969c9a8b}]" -testLoggerAdapter verbose: server-6577524e-5102-c512-ed2b-ee8108ec66fa: 2024-09-03T21:34:36.118Z [INFO] server-6577524e-5102-c512-ed2b-ee8108ec66fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-12a9c79e-5a12-f014-115b-422de1be8ed9 Address:12a9c79e-5a12-f014-115b-422de1be8ed9} {Suffrage:Voter ID:server-5939a590-b9d0-5f82-4521-a2c243521e3e Address:5939a590-b9d0-5f82-4521-a2c243521e3e} {Suffrage:Voter ID:server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b Address:6eedb621-0cfe-58c0-f11c-90d9e3d9e56b} {Suffrage:Voter ID:server-6577524e-5102-c512-ed2b-ee8108ec66fa Address:6577524e-5102-c512-ed2b-ee8108ec66fa} {Suffrage:Voter ID:server-b861555d-211f-cb23-d74e-592c82f26832 Address:b861555d-211f-cb23-d74e-592c82f26832} {Suffrage:Voter ID:server-afa0751b-4c32-c2fc-791e-b84ad64fab14 Address:afa0751b-4c32-c2fc-791e-b84ad64fab14} {Suffrage:Voter ID:server-38e08373-3b93-60c8-f46e-04d3969c9a8b Address:38e08373-3b93-60c8-f46e-04d3969c9a8b}]" - testing.go:149: server-6577524e-5102-c512-ed2b-ee8108ec66fa: 2024-09-03T21:34:36.118Z [INFO] server-6577524e-5102-c512-ed2b-ee8108ec66fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-12a9c79e-5a12-f014-115b-422de1be8ed9 Address:12a9c79e-5a12-f014-115b-422de1be8ed9} {Suffrage:Voter ID:server-5939a590-b9d0-5f82-4521-a2c243521e3e Address:5939a590-b9d0-5f82-4521-a2c243521e3e} {Suffrage:Voter ID:server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b Address:6eedb621-0cfe-58c0-f11c-90d9e3d9e56b} {Suffrage:Voter ID:server-6577524e-5102-c512-ed2b-ee8108ec66fa Address:6577524e-5102-c512-ed2b-ee8108ec66fa} {Suffrage:Voter ID:server-b861555d-211f-cb23-d74e-592c82f26832 Address:b861555d-211f-cb23-d74e-592c82f26832} {Suffrage:Voter ID:server-afa0751b-4c32-c2fc-791e-b84ad64fab14 Address:afa0751b-4c32-c2fc-791e-b84ad64fab14} {Suffrage:Voter ID:server-38e08373-3b93-60c8-f46e-04d3969c9a8b Address:38e08373-3b93-60c8-f46e-04d3969c9a8b}]" -testLoggerAdapter verbose: server-b861555d-211f-cb23-d74e-592c82f26832: 2024-09-03T21:34:36.118Z [INFO] server-b861555d-211f-cb23-d74e-592c82f26832: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-12a9c79e-5a12-f014-115b-422de1be8ed9 Address:12a9c79e-5a12-f014-115b-422de1be8ed9} {Suffrage:Voter ID:server-5939a590-b9d0-5f82-4521-a2c243521e3e Address:5939a590-b9d0-5f82-4521-a2c243521e3e} {Suffrage:Voter ID:server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b Address:6eedb621-0cfe-58c0-f11c-90d9e3d9e56b} {Suffrage:Voter ID:server-6577524e-5102-c512-ed2b-ee8108ec66fa Address:6577524e-5102-c512-ed2b-ee8108ec66fa} {Suffrage:Voter ID:server-b861555d-211f-cb23-d74e-592c82f26832 Address:b861555d-211f-cb23-d74e-592c82f26832} {Suffrage:Voter ID:server-afa0751b-4c32-c2fc-791e-b84ad64fab14 Address:afa0751b-4c32-c2fc-791e-b84ad64fab14} {Suffrage:Voter ID:server-38e08373-3b93-60c8-f46e-04d3969c9a8b Address:38e08373-3b93-60c8-f46e-04d3969c9a8b}]" - testing.go:149: server-b861555d-211f-cb23-d74e-592c82f26832: 2024-09-03T21:34:36.118Z [INFO] server-b861555d-211f-cb23-d74e-592c82f26832: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-12a9c79e-5a12-f014-115b-422de1be8ed9 Address:12a9c79e-5a12-f014-115b-422de1be8ed9} {Suffrage:Voter ID:server-5939a590-b9d0-5f82-4521-a2c243521e3e Address:5939a590-b9d0-5f82-4521-a2c243521e3e} {Suffrage:Voter ID:server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b Address:6eedb621-0cfe-58c0-f11c-90d9e3d9e56b} {Suffrage:Voter ID:server-6577524e-5102-c512-ed2b-ee8108ec66fa Address:6577524e-5102-c512-ed2b-ee8108ec66fa} {Suffrage:Voter ID:server-b861555d-211f-cb23-d74e-592c82f26832 Address:b861555d-211f-cb23-d74e-592c82f26832} {Suffrage:Voter ID:server-afa0751b-4c32-c2fc-791e-b84ad64fab14 Address:afa0751b-4c32-c2fc-791e-b84ad64fab14} {Suffrage:Voter ID:server-38e08373-3b93-60c8-f46e-04d3969c9a8b Address:38e08373-3b93-60c8-f46e-04d3969c9a8b}]" -testLoggerAdapter verbose: server-afa0751b-4c32-c2fc-791e-b84ad64fab14: 2024-09-03T21:34:36.118Z [INFO] server-afa0751b-4c32-c2fc-791e-b84ad64fab14: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-12a9c79e-5a12-f014-115b-422de1be8ed9 Address:12a9c79e-5a12-f014-115b-422de1be8ed9} {Suffrage:Voter ID:server-5939a590-b9d0-5f82-4521-a2c243521e3e Address:5939a590-b9d0-5f82-4521-a2c243521e3e} {Suffrage:Voter ID:server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b Address:6eedb621-0cfe-58c0-f11c-90d9e3d9e56b} {Suffrage:Voter ID:server-6577524e-5102-c512-ed2b-ee8108ec66fa Address:6577524e-5102-c512-ed2b-ee8108ec66fa} {Suffrage:Voter ID:server-b861555d-211f-cb23-d74e-592c82f26832 Address:b861555d-211f-cb23-d74e-592c82f26832} {Suffrage:Voter ID:server-afa0751b-4c32-c2fc-791e-b84ad64fab14 Address:afa0751b-4c32-c2fc-791e-b84ad64fab14} {Suffrage:Voter ID:server-38e08373-3b93-60c8-f46e-04d3969c9a8b Address:38e08373-3b93-60c8-f46e-04d3969c9a8b}]" - testing.go:149: server-afa0751b-4c32-c2fc-791e-b84ad64fab14: 2024-09-03T21:34:36.118Z [INFO] server-afa0751b-4c32-c2fc-791e-b84ad64fab14: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-12a9c79e-5a12-f014-115b-422de1be8ed9 Address:12a9c79e-5a12-f014-115b-422de1be8ed9} {Suffrage:Voter ID:server-5939a590-b9d0-5f82-4521-a2c243521e3e Address:5939a590-b9d0-5f82-4521-a2c243521e3e} {Suffrage:Voter ID:server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b Address:6eedb621-0cfe-58c0-f11c-90d9e3d9e56b} {Suffrage:Voter ID:server-6577524e-5102-c512-ed2b-ee8108ec66fa Address:6577524e-5102-c512-ed2b-ee8108ec66fa} {Suffrage:Voter ID:server-b861555d-211f-cb23-d74e-592c82f26832 Address:b861555d-211f-cb23-d74e-592c82f26832} {Suffrage:Voter ID:server-afa0751b-4c32-c2fc-791e-b84ad64fab14 Address:afa0751b-4c32-c2fc-791e-b84ad64fab14} {Suffrage:Voter ID:server-38e08373-3b93-60c8-f46e-04d3969c9a8b Address:38e08373-3b93-60c8-f46e-04d3969c9a8b}]" -testLoggerAdapter verbose: server-38e08373-3b93-60c8-f46e-04d3969c9a8b: 2024-09-03T21:34:36.119Z [INFO] server-38e08373-3b93-60c8-f46e-04d3969c9a8b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-12a9c79e-5a12-f014-115b-422de1be8ed9 Address:12a9c79e-5a12-f014-115b-422de1be8ed9} {Suffrage:Voter ID:server-5939a590-b9d0-5f82-4521-a2c243521e3e Address:5939a590-b9d0-5f82-4521-a2c243521e3e} {Suffrage:Voter ID:server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b Address:6eedb621-0cfe-58c0-f11c-90d9e3d9e56b} {Suffrage:Voter ID:server-6577524e-5102-c512-ed2b-ee8108ec66fa Address:6577524e-5102-c512-ed2b-ee8108ec66fa} {Suffrage:Voter ID:server-b861555d-211f-cb23-d74e-592c82f26832 Address:b861555d-211f-cb23-d74e-592c82f26832} {Suffrage:Voter ID:server-afa0751b-4c32-c2fc-791e-b84ad64fab14 Address:afa0751b-4c32-c2fc-791e-b84ad64fab14} {Suffrage:Voter ID:server-38e08373-3b93-60c8-f46e-04d3969c9a8b Address:38e08373-3b93-60c8-f46e-04d3969c9a8b}]" - testing.go:149: server-38e08373-3b93-60c8-f46e-04d3969c9a8b: 2024-09-03T21:34:36.119Z [INFO] server-38e08373-3b93-60c8-f46e-04d3969c9a8b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-12a9c79e-5a12-f014-115b-422de1be8ed9 Address:12a9c79e-5a12-f014-115b-422de1be8ed9} {Suffrage:Voter ID:server-5939a590-b9d0-5f82-4521-a2c243521e3e Address:5939a590-b9d0-5f82-4521-a2c243521e3e} {Suffrage:Voter ID:server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b Address:6eedb621-0cfe-58c0-f11c-90d9e3d9e56b} {Suffrage:Voter ID:server-6577524e-5102-c512-ed2b-ee8108ec66fa Address:6577524e-5102-c512-ed2b-ee8108ec66fa} {Suffrage:Voter ID:server-b861555d-211f-cb23-d74e-592c82f26832 Address:b861555d-211f-cb23-d74e-592c82f26832} {Suffrage:Voter ID:server-afa0751b-4c32-c2fc-791e-b84ad64fab14 Address:afa0751b-4c32-c2fc-791e-b84ad64fab14} {Suffrage:Voter ID:server-38e08373-3b93-60c8-f46e-04d3969c9a8b Address:38e08373-3b93-60c8-f46e-04d3969c9a8b}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:36.119Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:36.119Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.119Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: entering follower state: follower="Node at 12a9c79e-5a12-f014-115b-422de1be8ed9 [Follower]" leader= - testing.go:149: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.119Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: entering follower state: follower="Node at 12a9c79e-5a12-f014-115b-422de1be8ed9 [Follower]" leader= -testLoggerAdapter verbose: server-5939a590-b9d0-5f82-4521-a2c243521e3e: 2024-09-03T21:34:36.119Z [INFO] server-5939a590-b9d0-5f82-4521-a2c243521e3e: entering follower state: follower="Node at 5939a590-b9d0-5f82-4521-a2c243521e3e [Follower]" leader= - testing.go:149: server-5939a590-b9d0-5f82-4521-a2c243521e3e: 2024-09-03T21:34:36.119Z [INFO] server-5939a590-b9d0-5f82-4521-a2c243521e3e: entering follower state: follower="Node at 5939a590-b9d0-5f82-4521-a2c243521e3e [Follower]" leader= -testLoggerAdapter verbose: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.119Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: entering follower state: follower="Node at 6eedb621-0cfe-58c0-f11c-90d9e3d9e56b [Follower]" leader= - testing.go:149: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.119Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: entering follower state: follower="Node at 6eedb621-0cfe-58c0-f11c-90d9e3d9e56b [Follower]" leader= -testLoggerAdapter verbose: server-6577524e-5102-c512-ed2b-ee8108ec66fa: 2024-09-03T21:34:36.119Z [INFO] server-6577524e-5102-c512-ed2b-ee8108ec66fa: entering follower state: follower="Node at 6577524e-5102-c512-ed2b-ee8108ec66fa [Follower]" leader= - testing.go:149: server-6577524e-5102-c512-ed2b-ee8108ec66fa: 2024-09-03T21:34:36.119Z [INFO] server-6577524e-5102-c512-ed2b-ee8108ec66fa: entering follower state: follower="Node at 6577524e-5102-c512-ed2b-ee8108ec66fa [Follower]" leader= -testLoggerAdapter verbose: server-b861555d-211f-cb23-d74e-592c82f26832: 2024-09-03T21:34:36.119Z [INFO] server-b861555d-211f-cb23-d74e-592c82f26832: entering follower state: follower="Node at b861555d-211f-cb23-d74e-592c82f26832 [Follower]" leader= - testing.go:149: server-b861555d-211f-cb23-d74e-592c82f26832: 2024-09-03T21:34:36.119Z [INFO] server-b861555d-211f-cb23-d74e-592c82f26832: entering follower state: follower="Node at b861555d-211f-cb23-d74e-592c82f26832 [Follower]" leader= -testLoggerAdapter verbose: server-afa0751b-4c32-c2fc-791e-b84ad64fab14: 2024-09-03T21:34:36.119Z [INFO] server-afa0751b-4c32-c2fc-791e-b84ad64fab14: entering follower state: follower="Node at afa0751b-4c32-c2fc-791e-b84ad64fab14 [Follower]" leader= - testing.go:149: server-afa0751b-4c32-c2fc-791e-b84ad64fab14: 2024-09-03T21:34:36.119Z [INFO] server-afa0751b-4c32-c2fc-791e-b84ad64fab14: entering follower state: follower="Node at afa0751b-4c32-c2fc-791e-b84ad64fab14 [Follower]" leader= -testLoggerAdapter verbose: server-38e08373-3b93-60c8-f46e-04d3969c9a8b: 2024-09-03T21:34:36.120Z [INFO] server-38e08373-3b93-60c8-f46e-04d3969c9a8b: entering follower state: follower="Node at 38e08373-3b93-60c8-f46e-04d3969c9a8b [Follower]" leader= - testing.go:149: server-38e08373-3b93-60c8-f46e-04d3969c9a8b: 2024-09-03T21:34:36.120Z [INFO] server-38e08373-3b93-60c8-f46e-04d3969c9a8b: entering follower state: follower="Node at 38e08373-3b93-60c8-f46e-04d3969c9a8b [Follower]" leader= -testLoggerAdapter verbose: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.172Z [WARN] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.172Z [WARN] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.172Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: entering candidate state: node="Node at 6eedb621-0cfe-58c0-f11c-90d9e3d9e56b [Candidate]" term=2 - testing.go:149: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.172Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: entering candidate state: node="Node at 6eedb621-0cfe-58c0-f11c-90d9e3d9e56b [Candidate]" term=2 -testLoggerAdapter verbose: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.172Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: election won: tally=4 - testing.go:149: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.172Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: election won: tally=4 -testLoggerAdapter verbose: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.172Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: entering leader state: leader="Node at 6eedb621-0cfe-58c0-f11c-90d9e3d9e56b [Leader]" - testing.go:149: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.172Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: entering leader state: leader="Node at 6eedb621-0cfe-58c0-f11c-90d9e3d9e56b [Leader]" -testLoggerAdapter verbose: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.172Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: added peer, starting replication: peer=server-12a9c79e-5a12-f014-115b-422de1be8ed9 - testing.go:149: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.172Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: added peer, starting replication: peer=server-12a9c79e-5a12-f014-115b-422de1be8ed9 -testLoggerAdapter verbose: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.172Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: added peer, starting replication: peer=server-5939a590-b9d0-5f82-4521-a2c243521e3e - testing.go:149: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.172Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: added peer, starting replication: peer=server-5939a590-b9d0-5f82-4521-a2c243521e3e -testLoggerAdapter verbose: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.172Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: added peer, starting replication: peer=server-6577524e-5102-c512-ed2b-ee8108ec66fa - testing.go:149: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.172Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: added peer, starting replication: peer=server-6577524e-5102-c512-ed2b-ee8108ec66fa -testLoggerAdapter verbose: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.172Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: added peer, starting replication: peer=server-b861555d-211f-cb23-d74e-592c82f26832 - testing.go:149: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.172Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: added peer, starting replication: peer=server-b861555d-211f-cb23-d74e-592c82f26832 -testLoggerAdapter verbose: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.172Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: added peer, starting replication: peer=server-afa0751b-4c32-c2fc-791e-b84ad64fab14 - testing.go:149: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.172Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: added peer, starting replication: peer=server-afa0751b-4c32-c2fc-791e-b84ad64fab14 -testLoggerAdapter verbose: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.172Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: pipelining replication: peer="{Voter server-5939a590-b9d0-5f82-4521-a2c243521e3e 5939a590-b9d0-5f82-4521-a2c243521e3e}" - testing.go:149: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.172Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: pipelining replication: peer="{Voter server-5939a590-b9d0-5f82-4521-a2c243521e3e 5939a590-b9d0-5f82-4521-a2c243521e3e}" -testLoggerAdapter verbose: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.172Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: added peer, starting replication: peer=server-38e08373-3b93-60c8-f46e-04d3969c9a8b - testing.go:149: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.172Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: added peer, starting replication: peer=server-38e08373-3b93-60c8-f46e-04d3969c9a8b -testLoggerAdapter verbose: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.172Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: pipelining replication: peer="{Voter server-6577524e-5102-c512-ed2b-ee8108ec66fa 6577524e-5102-c512-ed2b-ee8108ec66fa}" - testing.go:149: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.172Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: pipelining replication: peer="{Voter server-6577524e-5102-c512-ed2b-ee8108ec66fa 6577524e-5102-c512-ed2b-ee8108ec66fa}" -testLoggerAdapter verbose: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.172Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: pipelining replication: peer="{Voter server-afa0751b-4c32-c2fc-791e-b84ad64fab14 afa0751b-4c32-c2fc-791e-b84ad64fab14}" - testing.go:149: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.172Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: pipelining replication: peer="{Voter server-afa0751b-4c32-c2fc-791e-b84ad64fab14 afa0751b-4c32-c2fc-791e-b84ad64fab14}" -testLoggerAdapter verbose: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.172Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: pipelining replication: peer="{Voter server-12a9c79e-5a12-f014-115b-422de1be8ed9 12a9c79e-5a12-f014-115b-422de1be8ed9}" - testing.go:149: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.172Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: pipelining replication: peer="{Voter server-12a9c79e-5a12-f014-115b-422de1be8ed9 12a9c79e-5a12-f014-115b-422de1be8ed9}" -testLoggerAdapter verbose: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.172Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: pipelining replication: peer="{Voter server-b861555d-211f-cb23-d74e-592c82f26832 b861555d-211f-cb23-d74e-592c82f26832}" - testing.go:149: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.172Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: pipelining replication: peer="{Voter server-b861555d-211f-cb23-d74e-592c82f26832 b861555d-211f-cb23-d74e-592c82f26832}" -testLoggerAdapter verbose: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.173Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: pipelining replication: peer="{Voter server-38e08373-3b93-60c8-f46e-04d3969c9a8b 38e08373-3b93-60c8-f46e-04d3969c9a8b}" - testing.go:149: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.173Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: pipelining replication: peer="{Voter server-38e08373-3b93-60c8-f46e-04d3969c9a8b 38e08373-3b93-60c8-f46e-04d3969c9a8b}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:36.277Z [INFO] stable state for Leader reached at 2024-09-03 21:34:36.17281209 +0000 UTC m=+17.219469221 (1 nodes), 53.577166ms from start of poll, 54.845315ms from cluster start. Timeout at 2024-09-03 21:34:36.277837812 +0000 UTC m=+17.324494958, 105.025737ms after stability - testing.go:149: cluster: 2024-09-03T21:34:36.277Z [INFO] stable state for Leader reached at 2024-09-03 21:34:36.17281209 +0000 UTC m=+17.219469221 (1 nodes), 53.577166ms from start of poll, 54.845315ms from cluster start. Timeout at 2024-09-03 21:34:36.277837812 +0000 UTC m=+17.324494958, 105.025737ms after stability -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:36.278Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-09-03T21:34:36.278Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:36.383Z [INFO] stable state for Follower reached at 2024-09-03 21:34:36.278047645 +0000 UTC m=+17.324704785 (6 nodes), 4.434µs from start of poll, 160.080879ms from cluster start. Timeout at 2024-09-03 21:34:36.3831138 +0000 UTC m=+17.429770935, 105.06615ms after stability - testing.go:149: cluster: 2024-09-03T21:34:36.383Z [INFO] stable state for Follower reached at 2024-09-03 21:34:36.278047645 +0000 UTC m=+17.324704785 (6 nodes), 4.434µs from start of poll, 160.080879ms from cluster start. Timeout at 2024-09-03 21:34:36.3831138 +0000 UTC m=+17.429770935, 105.06615ms after stability -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:36.383Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:36.383Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:36.488Z [INFO] stable state for Leader reached at 2024-09-03 21:34:36.383261269 +0000 UTC m=+17.429918413 (1 nodes), 1.186µs from start of poll, 265.294507ms from cluster start. Timeout at 2024-09-03 21:34:36.488360893 +0000 UTC m=+17.535018032, 105.099619ms after stability - testing.go:149: cluster: 2024-09-03T21:34:36.488Z [INFO] stable state for Leader reached at 2024-09-03 21:34:36.383261269 +0000 UTC m=+17.429918413 (1 nodes), 1.186µs from start of poll, 265.294507ms from cluster start. Timeout at 2024-09-03 21:34:36.488360893 +0000 UTC m=+17.535018032, 105.099619ms after stability -testLoggerAdapter verbose: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.488Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: entering candidate state: node="Node at 12a9c79e-5a12-f014-115b-422de1be8ed9 [Candidate]" term=3 - testing.go:149: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.488Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: entering candidate state: node="Node at 12a9c79e-5a12-f014-115b-422de1be8ed9 [Candidate]" term=3 -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:36.488Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:36.488Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.488Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: entering follower state: follower="Node at 6eedb621-0cfe-58c0-f11c-90d9e3d9e56b [Follower]" leader= - testing.go:149: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.488Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: entering follower state: follower="Node at 6eedb621-0cfe-58c0-f11c-90d9e3d9e56b [Follower]" leader= -testLoggerAdapter verbose: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.488Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: aborting pipeline replication: peer="{Voter server-12a9c79e-5a12-f014-115b-422de1be8ed9 12a9c79e-5a12-f014-115b-422de1be8ed9}" - testing.go:149: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.488Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: aborting pipeline replication: peer="{Voter server-12a9c79e-5a12-f014-115b-422de1be8ed9 12a9c79e-5a12-f014-115b-422de1be8ed9}" -testLoggerAdapter verbose: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.488Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: election won: tally=4 - testing.go:149: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.488Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: election won: tally=4 -testLoggerAdapter verbose: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.488Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: entering leader state: leader="Node at 12a9c79e-5a12-f014-115b-422de1be8ed9 [Leader]" - testing.go:149: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.488Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: entering leader state: leader="Node at 12a9c79e-5a12-f014-115b-422de1be8ed9 [Leader]" -testLoggerAdapter verbose: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.488Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: added peer, starting replication: peer=server-5939a590-b9d0-5f82-4521-a2c243521e3e - testing.go:149: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.488Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: added peer, starting replication: peer=server-5939a590-b9d0-5f82-4521-a2c243521e3e -testLoggerAdapter verbose: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.488Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: added peer, starting replication: peer=server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b - testing.go:149: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.488Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: added peer, starting replication: peer=server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b -testLoggerAdapter verbose: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.488Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: added peer, starting replication: peer=server-6577524e-5102-c512-ed2b-ee8108ec66fa - testing.go:149: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.488Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: added peer, starting replication: peer=server-6577524e-5102-c512-ed2b-ee8108ec66fa -testLoggerAdapter verbose: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.488Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: added peer, starting replication: peer=server-b861555d-211f-cb23-d74e-592c82f26832 - testing.go:149: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.488Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: added peer, starting replication: peer=server-b861555d-211f-cb23-d74e-592c82f26832 -testLoggerAdapter verbose: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.489Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: added peer, starting replication: peer=server-afa0751b-4c32-c2fc-791e-b84ad64fab14 - testing.go:149: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.489Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: added peer, starting replication: peer=server-afa0751b-4c32-c2fc-791e-b84ad64fab14 -testLoggerAdapter verbose: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.489Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: added peer, starting replication: peer=server-38e08373-3b93-60c8-f46e-04d3969c9a8b - testing.go:149: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.489Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: added peer, starting replication: peer=server-38e08373-3b93-60c8-f46e-04d3969c9a8b -testLoggerAdapter verbose: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.489Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: pipelining replication: peer="{Voter server-38e08373-3b93-60c8-f46e-04d3969c9a8b 38e08373-3b93-60c8-f46e-04d3969c9a8b}" - testing.go:149: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.489Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: pipelining replication: peer="{Voter server-38e08373-3b93-60c8-f46e-04d3969c9a8b 38e08373-3b93-60c8-f46e-04d3969c9a8b}" -testLoggerAdapter verbose: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.489Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: aborting pipeline replication: peer="{Voter server-5939a590-b9d0-5f82-4521-a2c243521e3e 5939a590-b9d0-5f82-4521-a2c243521e3e}" - testing.go:149: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.489Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: aborting pipeline replication: peer="{Voter server-5939a590-b9d0-5f82-4521-a2c243521e3e 5939a590-b9d0-5f82-4521-a2c243521e3e}" -testLoggerAdapter verbose: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.489Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: aborting pipeline replication: peer="{Voter server-6577524e-5102-c512-ed2b-ee8108ec66fa 6577524e-5102-c512-ed2b-ee8108ec66fa}" - testing.go:149: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.489Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: aborting pipeline replication: peer="{Voter server-6577524e-5102-c512-ed2b-ee8108ec66fa 6577524e-5102-c512-ed2b-ee8108ec66fa}" -testLoggerAdapter verbose: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.489Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: aborting pipeline replication: peer="{Voter server-b861555d-211f-cb23-d74e-592c82f26832 b861555d-211f-cb23-d74e-592c82f26832}" - testing.go:149: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.489Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: aborting pipeline replication: peer="{Voter server-b861555d-211f-cb23-d74e-592c82f26832 b861555d-211f-cb23-d74e-592c82f26832}" -testLoggerAdapter verbose: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.489Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: aborting pipeline replication: peer="{Voter server-afa0751b-4c32-c2fc-791e-b84ad64fab14 afa0751b-4c32-c2fc-791e-b84ad64fab14}" - testing.go:149: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.489Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: aborting pipeline replication: peer="{Voter server-afa0751b-4c32-c2fc-791e-b84ad64fab14 afa0751b-4c32-c2fc-791e-b84ad64fab14}" -testLoggerAdapter verbose: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.489Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: aborting pipeline replication: peer="{Voter server-38e08373-3b93-60c8-f46e-04d3969c9a8b 38e08373-3b93-60c8-f46e-04d3969c9a8b}" - testing.go:149: server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: 2024-09-03T21:34:36.489Z [INFO] server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b: aborting pipeline replication: peer="{Voter server-38e08373-3b93-60c8-f46e-04d3969c9a8b 38e08373-3b93-60c8-f46e-04d3969c9a8b}" -testLoggerAdapter verbose: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.489Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: pipelining replication: peer="{Voter server-5939a590-b9d0-5f82-4521-a2c243521e3e 5939a590-b9d0-5f82-4521-a2c243521e3e}" - testing.go:149: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.489Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: pipelining replication: peer="{Voter server-5939a590-b9d0-5f82-4521-a2c243521e3e 5939a590-b9d0-5f82-4521-a2c243521e3e}" -testLoggerAdapter verbose: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.489Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: pipelining replication: peer="{Voter server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b 6eedb621-0cfe-58c0-f11c-90d9e3d9e56b}" - testing.go:149: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.489Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: pipelining replication: peer="{Voter server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b 6eedb621-0cfe-58c0-f11c-90d9e3d9e56b}" -testLoggerAdapter verbose: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.489Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: pipelining replication: peer="{Voter server-6577524e-5102-c512-ed2b-ee8108ec66fa 6577524e-5102-c512-ed2b-ee8108ec66fa}" - testing.go:149: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.489Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: pipelining replication: peer="{Voter server-6577524e-5102-c512-ed2b-ee8108ec66fa 6577524e-5102-c512-ed2b-ee8108ec66fa}" -testLoggerAdapter verbose: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.489Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: pipelining replication: peer="{Voter server-b861555d-211f-cb23-d74e-592c82f26832 b861555d-211f-cb23-d74e-592c82f26832}" - testing.go:149: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.489Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: pipelining replication: peer="{Voter server-b861555d-211f-cb23-d74e-592c82f26832 b861555d-211f-cb23-d74e-592c82f26832}" -testLoggerAdapter verbose: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.489Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: pipelining replication: peer="{Voter server-afa0751b-4c32-c2fc-791e-b84ad64fab14 afa0751b-4c32-c2fc-791e-b84ad64fab14}" - testing.go:149: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.489Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: pipelining replication: peer="{Voter server-afa0751b-4c32-c2fc-791e-b84ad64fab14 afa0751b-4c32-c2fc-791e-b84ad64fab14}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:36.594Z [INFO] stable state for Leader reached at 2024-09-03 21:34:36.489210868 +0000 UTC m=+17.535867998 (1 nodes), 543.435µs from start of poll, 371.244092ms from cluster start. Timeout at 2024-09-03 21:34:36.594286508 +0000 UTC m=+17.640943645, 105.075647ms after stability - testing.go:149: cluster: 2024-09-03T21:34:36.594Z [INFO] stable state for Leader reached at 2024-09-03 21:34:36.489210868 +0000 UTC m=+17.535867998 (1 nodes), 543.435µs from start of poll, 371.244092ms from cluster start. Timeout at 2024-09-03 21:34:36.594286508 +0000 UTC m=+17.640943645, 105.075647ms after stability -testLoggerAdapter verbose: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.594Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: aborting pipeline replication: peer="{Voter server-6577524e-5102-c512-ed2b-ee8108ec66fa 6577524e-5102-c512-ed2b-ee8108ec66fa}" - testing.go:149: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.594Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: aborting pipeline replication: peer="{Voter server-6577524e-5102-c512-ed2b-ee8108ec66fa 6577524e-5102-c512-ed2b-ee8108ec66fa}" -testLoggerAdapter verbose: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.594Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: aborting pipeline replication: peer="{Voter server-b861555d-211f-cb23-d74e-592c82f26832 b861555d-211f-cb23-d74e-592c82f26832}" - testing.go:149: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.594Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: aborting pipeline replication: peer="{Voter server-b861555d-211f-cb23-d74e-592c82f26832 b861555d-211f-cb23-d74e-592c82f26832}" -testLoggerAdapter verbose: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.594Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: aborting pipeline replication: peer="{Voter server-afa0751b-4c32-c2fc-791e-b84ad64fab14 afa0751b-4c32-c2fc-791e-b84ad64fab14}" - testing.go:149: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.594Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: aborting pipeline replication: peer="{Voter server-afa0751b-4c32-c2fc-791e-b84ad64fab14 afa0751b-4c32-c2fc-791e-b84ad64fab14}" -testLoggerAdapter verbose: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.594Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: aborting pipeline replication: peer="{Voter server-38e08373-3b93-60c8-f46e-04d3969c9a8b 38e08373-3b93-60c8-f46e-04d3969c9a8b}" - testing.go:149: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.594Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: aborting pipeline replication: peer="{Voter server-38e08373-3b93-60c8-f46e-04d3969c9a8b 38e08373-3b93-60c8-f46e-04d3969c9a8b}" -testLoggerAdapter verbose: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.594Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: aborting pipeline replication: peer="{Voter server-5939a590-b9d0-5f82-4521-a2c243521e3e 5939a590-b9d0-5f82-4521-a2c243521e3e}" - testing.go:149: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.594Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: aborting pipeline replication: peer="{Voter server-5939a590-b9d0-5f82-4521-a2c243521e3e 5939a590-b9d0-5f82-4521-a2c243521e3e}" -testLoggerAdapter verbose: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.594Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: aborting pipeline replication: peer="{Voter server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b 6eedb621-0cfe-58c0-f11c-90d9e3d9e56b}" - testing.go:149: server-12a9c79e-5a12-f014-115b-422de1be8ed9: 2024-09-03T21:34:36.594Z [INFO] server-12a9c79e-5a12-f014-115b-422de1be8ed9: aborting pipeline replication: peer="{Voter server-6eedb621-0cfe-58c0-f11c-90d9e3d9e56b 6eedb621-0cfe-58c0-f11c-90d9e3d9e56b}" ---- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.48s) +testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.169Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd Address:16c4d59e-8dcd-06b6-dc57-8e1b5667abdd} {Suffrage:Voter ID:server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 Address:d2db4dc6-ece6-d891-330a-3a459dfa2bd5} {Suffrage:Voter ID:server-06e0cc33-930b-b8b4-1164-30f2237882ba Address:06e0cc33-930b-b8b4-1164-30f2237882ba} {Suffrage:Voter ID:server-130b2734-1a0a-f1c3-6a90-3fd880c1748a Address:130b2734-1a0a-f1c3-6a90-3fd880c1748a} {Suffrage:Voter ID:server-fcf65dbc-909a-19b9-2058-4b94c479035b Address:fcf65dbc-909a-19b9-2058-4b94c479035b} {Suffrage:Voter ID:server-00a5042f-6283-5ea0-b291-ddca3bab9553 Address:00a5042f-6283-5ea0-b291-ddca3bab9553} {Suffrage:Voter ID:server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd Address:74a2e60e-7ba2-6dfb-22b7-2d76da0718bd}]" + testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.169Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd Address:16c4d59e-8dcd-06b6-dc57-8e1b5667abdd} {Suffrage:Voter ID:server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 Address:d2db4dc6-ece6-d891-330a-3a459dfa2bd5} {Suffrage:Voter ID:server-06e0cc33-930b-b8b4-1164-30f2237882ba Address:06e0cc33-930b-b8b4-1164-30f2237882ba} {Suffrage:Voter ID:server-130b2734-1a0a-f1c3-6a90-3fd880c1748a Address:130b2734-1a0a-f1c3-6a90-3fd880c1748a} {Suffrage:Voter ID:server-fcf65dbc-909a-19b9-2058-4b94c479035b Address:fcf65dbc-909a-19b9-2058-4b94c479035b} {Suffrage:Voter ID:server-00a5042f-6283-5ea0-b291-ddca3bab9553 Address:00a5042f-6283-5ea0-b291-ddca3bab9553} {Suffrage:Voter ID:server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd Address:74a2e60e-7ba2-6dfb-22b7-2d76da0718bd}]" +testLoggerAdapter verbose: server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5: 2023-08-02T15:13:05.170Z [INFO] server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd Address:16c4d59e-8dcd-06b6-dc57-8e1b5667abdd} {Suffrage:Voter ID:server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 Address:d2db4dc6-ece6-d891-330a-3a459dfa2bd5} {Suffrage:Voter ID:server-06e0cc33-930b-b8b4-1164-30f2237882ba Address:06e0cc33-930b-b8b4-1164-30f2237882ba} {Suffrage:Voter ID:server-130b2734-1a0a-f1c3-6a90-3fd880c1748a Address:130b2734-1a0a-f1c3-6a90-3fd880c1748a} {Suffrage:Voter ID:server-fcf65dbc-909a-19b9-2058-4b94c479035b Address:fcf65dbc-909a-19b9-2058-4b94c479035b} {Suffrage:Voter ID:server-00a5042f-6283-5ea0-b291-ddca3bab9553 Address:00a5042f-6283-5ea0-b291-ddca3bab9553} {Suffrage:Voter ID:server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd Address:74a2e60e-7ba2-6dfb-22b7-2d76da0718bd}]" + testing.go:149: server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5: 2023-08-02T15:13:05.170Z [INFO] server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd Address:16c4d59e-8dcd-06b6-dc57-8e1b5667abdd} {Suffrage:Voter ID:server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 Address:d2db4dc6-ece6-d891-330a-3a459dfa2bd5} {Suffrage:Voter ID:server-06e0cc33-930b-b8b4-1164-30f2237882ba Address:06e0cc33-930b-b8b4-1164-30f2237882ba} {Suffrage:Voter ID:server-130b2734-1a0a-f1c3-6a90-3fd880c1748a Address:130b2734-1a0a-f1c3-6a90-3fd880c1748a} {Suffrage:Voter ID:server-fcf65dbc-909a-19b9-2058-4b94c479035b Address:fcf65dbc-909a-19b9-2058-4b94c479035b} {Suffrage:Voter ID:server-00a5042f-6283-5ea0-b291-ddca3bab9553 Address:00a5042f-6283-5ea0-b291-ddca3bab9553} {Suffrage:Voter ID:server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd Address:74a2e60e-7ba2-6dfb-22b7-2d76da0718bd}]" +testLoggerAdapter verbose: server-06e0cc33-930b-b8b4-1164-30f2237882ba: 2023-08-02T15:13:05.170Z [INFO] server-06e0cc33-930b-b8b4-1164-30f2237882ba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd Address:16c4d59e-8dcd-06b6-dc57-8e1b5667abdd} {Suffrage:Voter ID:server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 Address:d2db4dc6-ece6-d891-330a-3a459dfa2bd5} {Suffrage:Voter ID:server-06e0cc33-930b-b8b4-1164-30f2237882ba Address:06e0cc33-930b-b8b4-1164-30f2237882ba} {Suffrage:Voter ID:server-130b2734-1a0a-f1c3-6a90-3fd880c1748a Address:130b2734-1a0a-f1c3-6a90-3fd880c1748a} {Suffrage:Voter ID:server-fcf65dbc-909a-19b9-2058-4b94c479035b Address:fcf65dbc-909a-19b9-2058-4b94c479035b} {Suffrage:Voter ID:server-00a5042f-6283-5ea0-b291-ddca3bab9553 Address:00a5042f-6283-5ea0-b291-ddca3bab9553} {Suffrage:Voter ID:server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd Address:74a2e60e-7ba2-6dfb-22b7-2d76da0718bd}]" + testing.go:149: server-06e0cc33-930b-b8b4-1164-30f2237882ba: 2023-08-02T15:13:05.170Z [INFO] server-06e0cc33-930b-b8b4-1164-30f2237882ba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd Address:16c4d59e-8dcd-06b6-dc57-8e1b5667abdd} {Suffrage:Voter ID:server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 Address:d2db4dc6-ece6-d891-330a-3a459dfa2bd5} {Suffrage:Voter ID:server-06e0cc33-930b-b8b4-1164-30f2237882ba Address:06e0cc33-930b-b8b4-1164-30f2237882ba} {Suffrage:Voter ID:server-130b2734-1a0a-f1c3-6a90-3fd880c1748a Address:130b2734-1a0a-f1c3-6a90-3fd880c1748a} {Suffrage:Voter ID:server-fcf65dbc-909a-19b9-2058-4b94c479035b Address:fcf65dbc-909a-19b9-2058-4b94c479035b} {Suffrage:Voter ID:server-00a5042f-6283-5ea0-b291-ddca3bab9553 Address:00a5042f-6283-5ea0-b291-ddca3bab9553} {Suffrage:Voter ID:server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd Address:74a2e60e-7ba2-6dfb-22b7-2d76da0718bd}]" +testLoggerAdapter verbose: server-130b2734-1a0a-f1c3-6a90-3fd880c1748a: 2023-08-02T15:13:05.170Z [INFO] server-130b2734-1a0a-f1c3-6a90-3fd880c1748a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd Address:16c4d59e-8dcd-06b6-dc57-8e1b5667abdd} {Suffrage:Voter ID:server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 Address:d2db4dc6-ece6-d891-330a-3a459dfa2bd5} {Suffrage:Voter ID:server-06e0cc33-930b-b8b4-1164-30f2237882ba Address:06e0cc33-930b-b8b4-1164-30f2237882ba} {Suffrage:Voter ID:server-130b2734-1a0a-f1c3-6a90-3fd880c1748a Address:130b2734-1a0a-f1c3-6a90-3fd880c1748a} {Suffrage:Voter ID:server-fcf65dbc-909a-19b9-2058-4b94c479035b Address:fcf65dbc-909a-19b9-2058-4b94c479035b} {Suffrage:Voter ID:server-00a5042f-6283-5ea0-b291-ddca3bab9553 Address:00a5042f-6283-5ea0-b291-ddca3bab9553} {Suffrage:Voter ID:server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd Address:74a2e60e-7ba2-6dfb-22b7-2d76da0718bd}]" + testing.go:149: server-130b2734-1a0a-f1c3-6a90-3fd880c1748a: 2023-08-02T15:13:05.170Z [INFO] server-130b2734-1a0a-f1c3-6a90-3fd880c1748a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd Address:16c4d59e-8dcd-06b6-dc57-8e1b5667abdd} {Suffrage:Voter ID:server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 Address:d2db4dc6-ece6-d891-330a-3a459dfa2bd5} {Suffrage:Voter ID:server-06e0cc33-930b-b8b4-1164-30f2237882ba Address:06e0cc33-930b-b8b4-1164-30f2237882ba} {Suffrage:Voter ID:server-130b2734-1a0a-f1c3-6a90-3fd880c1748a Address:130b2734-1a0a-f1c3-6a90-3fd880c1748a} {Suffrage:Voter ID:server-fcf65dbc-909a-19b9-2058-4b94c479035b Address:fcf65dbc-909a-19b9-2058-4b94c479035b} {Suffrage:Voter ID:server-00a5042f-6283-5ea0-b291-ddca3bab9553 Address:00a5042f-6283-5ea0-b291-ddca3bab9553} {Suffrage:Voter ID:server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd Address:74a2e60e-7ba2-6dfb-22b7-2d76da0718bd}]" +testLoggerAdapter verbose: server-fcf65dbc-909a-19b9-2058-4b94c479035b: 2023-08-02T15:13:05.170Z [INFO] server-fcf65dbc-909a-19b9-2058-4b94c479035b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd Address:16c4d59e-8dcd-06b6-dc57-8e1b5667abdd} {Suffrage:Voter ID:server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 Address:d2db4dc6-ece6-d891-330a-3a459dfa2bd5} {Suffrage:Voter ID:server-06e0cc33-930b-b8b4-1164-30f2237882ba Address:06e0cc33-930b-b8b4-1164-30f2237882ba} {Suffrage:Voter ID:server-130b2734-1a0a-f1c3-6a90-3fd880c1748a Address:130b2734-1a0a-f1c3-6a90-3fd880c1748a} {Suffrage:Voter ID:server-fcf65dbc-909a-19b9-2058-4b94c479035b Address:fcf65dbc-909a-19b9-2058-4b94c479035b} {Suffrage:Voter ID:server-00a5042f-6283-5ea0-b291-ddca3bab9553 Address:00a5042f-6283-5ea0-b291-ddca3bab9553} {Suffrage:Voter ID:server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd Address:74a2e60e-7ba2-6dfb-22b7-2d76da0718bd}]" + testing.go:149: server-fcf65dbc-909a-19b9-2058-4b94c479035b: 2023-08-02T15:13:05.170Z [INFO] server-fcf65dbc-909a-19b9-2058-4b94c479035b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd Address:16c4d59e-8dcd-06b6-dc57-8e1b5667abdd} {Suffrage:Voter ID:server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 Address:d2db4dc6-ece6-d891-330a-3a459dfa2bd5} {Suffrage:Voter ID:server-06e0cc33-930b-b8b4-1164-30f2237882ba Address:06e0cc33-930b-b8b4-1164-30f2237882ba} {Suffrage:Voter ID:server-130b2734-1a0a-f1c3-6a90-3fd880c1748a Address:130b2734-1a0a-f1c3-6a90-3fd880c1748a} {Suffrage:Voter ID:server-fcf65dbc-909a-19b9-2058-4b94c479035b Address:fcf65dbc-909a-19b9-2058-4b94c479035b} {Suffrage:Voter ID:server-00a5042f-6283-5ea0-b291-ddca3bab9553 Address:00a5042f-6283-5ea0-b291-ddca3bab9553} {Suffrage:Voter ID:server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd Address:74a2e60e-7ba2-6dfb-22b7-2d76da0718bd}]" +testLoggerAdapter verbose: server-00a5042f-6283-5ea0-b291-ddca3bab9553: 2023-08-02T15:13:05.170Z [INFO] server-00a5042f-6283-5ea0-b291-ddca3bab9553: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd Address:16c4d59e-8dcd-06b6-dc57-8e1b5667abdd} {Suffrage:Voter ID:server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 Address:d2db4dc6-ece6-d891-330a-3a459dfa2bd5} {Suffrage:Voter ID:server-06e0cc33-930b-b8b4-1164-30f2237882ba Address:06e0cc33-930b-b8b4-1164-30f2237882ba} {Suffrage:Voter ID:server-130b2734-1a0a-f1c3-6a90-3fd880c1748a Address:130b2734-1a0a-f1c3-6a90-3fd880c1748a} {Suffrage:Voter ID:server-fcf65dbc-909a-19b9-2058-4b94c479035b Address:fcf65dbc-909a-19b9-2058-4b94c479035b} {Suffrage:Voter ID:server-00a5042f-6283-5ea0-b291-ddca3bab9553 Address:00a5042f-6283-5ea0-b291-ddca3bab9553} {Suffrage:Voter ID:server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd Address:74a2e60e-7ba2-6dfb-22b7-2d76da0718bd}]" + testing.go:149: server-00a5042f-6283-5ea0-b291-ddca3bab9553: 2023-08-02T15:13:05.170Z [INFO] server-00a5042f-6283-5ea0-b291-ddca3bab9553: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd Address:16c4d59e-8dcd-06b6-dc57-8e1b5667abdd} {Suffrage:Voter ID:server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 Address:d2db4dc6-ece6-d891-330a-3a459dfa2bd5} {Suffrage:Voter ID:server-06e0cc33-930b-b8b4-1164-30f2237882ba Address:06e0cc33-930b-b8b4-1164-30f2237882ba} {Suffrage:Voter ID:server-130b2734-1a0a-f1c3-6a90-3fd880c1748a Address:130b2734-1a0a-f1c3-6a90-3fd880c1748a} {Suffrage:Voter ID:server-fcf65dbc-909a-19b9-2058-4b94c479035b Address:fcf65dbc-909a-19b9-2058-4b94c479035b} {Suffrage:Voter ID:server-00a5042f-6283-5ea0-b291-ddca3bab9553 Address:00a5042f-6283-5ea0-b291-ddca3bab9553} {Suffrage:Voter ID:server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd Address:74a2e60e-7ba2-6dfb-22b7-2d76da0718bd}]" +testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.170Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd Address:16c4d59e-8dcd-06b6-dc57-8e1b5667abdd} {Suffrage:Voter ID:server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 Address:d2db4dc6-ece6-d891-330a-3a459dfa2bd5} {Suffrage:Voter ID:server-06e0cc33-930b-b8b4-1164-30f2237882ba Address:06e0cc33-930b-b8b4-1164-30f2237882ba} {Suffrage:Voter ID:server-130b2734-1a0a-f1c3-6a90-3fd880c1748a Address:130b2734-1a0a-f1c3-6a90-3fd880c1748a} {Suffrage:Voter ID:server-fcf65dbc-909a-19b9-2058-4b94c479035b Address:fcf65dbc-909a-19b9-2058-4b94c479035b} {Suffrage:Voter ID:server-00a5042f-6283-5ea0-b291-ddca3bab9553 Address:00a5042f-6283-5ea0-b291-ddca3bab9553} {Suffrage:Voter ID:server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd Address:74a2e60e-7ba2-6dfb-22b7-2d76da0718bd}]" + testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.170Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd Address:16c4d59e-8dcd-06b6-dc57-8e1b5667abdd} {Suffrage:Voter ID:server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 Address:d2db4dc6-ece6-d891-330a-3a459dfa2bd5} {Suffrage:Voter ID:server-06e0cc33-930b-b8b4-1164-30f2237882ba Address:06e0cc33-930b-b8b4-1164-30f2237882ba} {Suffrage:Voter ID:server-130b2734-1a0a-f1c3-6a90-3fd880c1748a Address:130b2734-1a0a-f1c3-6a90-3fd880c1748a} {Suffrage:Voter ID:server-fcf65dbc-909a-19b9-2058-4b94c479035b Address:fcf65dbc-909a-19b9-2058-4b94c479035b} {Suffrage:Voter ID:server-00a5042f-6283-5ea0-b291-ddca3bab9553 Address:00a5042f-6283-5ea0-b291-ddca3bab9553} {Suffrage:Voter ID:server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd Address:74a2e60e-7ba2-6dfb-22b7-2d76da0718bd}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:05.171Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:13:05.171Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.171Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: entering follower state: follower="Node at 16c4d59e-8dcd-06b6-dc57-8e1b5667abdd [Follower]" leader= + testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.171Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: entering follower state: follower="Node at 16c4d59e-8dcd-06b6-dc57-8e1b5667abdd [Follower]" leader= +testLoggerAdapter verbose: server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5: 2023-08-02T15:13:05.171Z [INFO] server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5: entering follower state: follower="Node at d2db4dc6-ece6-d891-330a-3a459dfa2bd5 [Follower]" leader= + testing.go:149: server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5: 2023-08-02T15:13:05.171Z [INFO] server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5: entering follower state: follower="Node at d2db4dc6-ece6-d891-330a-3a459dfa2bd5 [Follower]" leader= +testLoggerAdapter verbose: server-06e0cc33-930b-b8b4-1164-30f2237882ba: 2023-08-02T15:13:05.171Z [INFO] server-06e0cc33-930b-b8b4-1164-30f2237882ba: entering follower state: follower="Node at 06e0cc33-930b-b8b4-1164-30f2237882ba [Follower]" leader= + testing.go:149: server-06e0cc33-930b-b8b4-1164-30f2237882ba: 2023-08-02T15:13:05.171Z [INFO] server-06e0cc33-930b-b8b4-1164-30f2237882ba: entering follower state: follower="Node at 06e0cc33-930b-b8b4-1164-30f2237882ba [Follower]" leader= +testLoggerAdapter verbose: server-130b2734-1a0a-f1c3-6a90-3fd880c1748a: 2023-08-02T15:13:05.171Z [INFO] server-130b2734-1a0a-f1c3-6a90-3fd880c1748a: entering follower state: follower="Node at 130b2734-1a0a-f1c3-6a90-3fd880c1748a [Follower]" leader= + testing.go:149: server-130b2734-1a0a-f1c3-6a90-3fd880c1748a: 2023-08-02T15:13:05.171Z [INFO] server-130b2734-1a0a-f1c3-6a90-3fd880c1748a: entering follower state: follower="Node at 130b2734-1a0a-f1c3-6a90-3fd880c1748a [Follower]" leader= +testLoggerAdapter verbose: server-fcf65dbc-909a-19b9-2058-4b94c479035b: 2023-08-02T15:13:05.171Z [INFO] server-fcf65dbc-909a-19b9-2058-4b94c479035b: entering follower state: follower="Node at fcf65dbc-909a-19b9-2058-4b94c479035b [Follower]" leader= + testing.go:149: server-fcf65dbc-909a-19b9-2058-4b94c479035b: 2023-08-02T15:13:05.171Z [INFO] server-fcf65dbc-909a-19b9-2058-4b94c479035b: entering follower state: follower="Node at fcf65dbc-909a-19b9-2058-4b94c479035b [Follower]" leader= +testLoggerAdapter verbose: server-00a5042f-6283-5ea0-b291-ddca3bab9553: 2023-08-02T15:13:05.171Z [INFO] server-00a5042f-6283-5ea0-b291-ddca3bab9553: entering follower state: follower="Node at 00a5042f-6283-5ea0-b291-ddca3bab9553 [Follower]" leader= + testing.go:149: server-00a5042f-6283-5ea0-b291-ddca3bab9553: 2023-08-02T15:13:05.171Z [INFO] server-00a5042f-6283-5ea0-b291-ddca3bab9553: entering follower state: follower="Node at 00a5042f-6283-5ea0-b291-ddca3bab9553 [Follower]" leader= +testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.171Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: entering follower state: follower="Node at 74a2e60e-7ba2-6dfb-22b7-2d76da0718bd [Follower]" leader= + testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.171Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: entering follower state: follower="Node at 74a2e60e-7ba2-6dfb-22b7-2d76da0718bd [Follower]" leader= +testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.222Z [WARN] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.222Z [WARN] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.222Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: entering candidate state: node="Node at 74a2e60e-7ba2-6dfb-22b7-2d76da0718bd [Candidate]" term=2 + testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.222Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: entering candidate state: node="Node at 74a2e60e-7ba2-6dfb-22b7-2d76da0718bd [Candidate]" term=2 +testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.222Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: election won: tally=4 + testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.222Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: election won: tally=4 +testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.222Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: entering leader state: leader="Node at 74a2e60e-7ba2-6dfb-22b7-2d76da0718bd [Leader]" + testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.222Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: entering leader state: leader="Node at 74a2e60e-7ba2-6dfb-22b7-2d76da0718bd [Leader]" +testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.222Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: added peer, starting replication: peer=server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd + testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.222Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: added peer, starting replication: peer=server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd +testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.222Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: added peer, starting replication: peer=server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 + testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.222Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: added peer, starting replication: peer=server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 +testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.230Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: pipelining replication: peer="{Voter server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd 16c4d59e-8dcd-06b6-dc57-8e1b5667abdd}" + testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.230Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: pipelining replication: peer="{Voter server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd 16c4d59e-8dcd-06b6-dc57-8e1b5667abdd}" +testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.230Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: added peer, starting replication: peer=server-06e0cc33-930b-b8b4-1164-30f2237882ba + testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.230Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: added peer, starting replication: peer=server-06e0cc33-930b-b8b4-1164-30f2237882ba +testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.235Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: added peer, starting replication: peer=server-130b2734-1a0a-f1c3-6a90-3fd880c1748a + testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.235Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: added peer, starting replication: peer=server-130b2734-1a0a-f1c3-6a90-3fd880c1748a +testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.235Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: added peer, starting replication: peer=server-fcf65dbc-909a-19b9-2058-4b94c479035b + testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.235Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: added peer, starting replication: peer=server-fcf65dbc-909a-19b9-2058-4b94c479035b +testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.235Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: added peer, starting replication: peer=server-00a5042f-6283-5ea0-b291-ddca3bab9553 + testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.235Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: added peer, starting replication: peer=server-00a5042f-6283-5ea0-b291-ddca3bab9553 +testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.230Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: pipelining replication: peer="{Voter server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 d2db4dc6-ece6-d891-330a-3a459dfa2bd5}" + testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.230Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: pipelining replication: peer="{Voter server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 d2db4dc6-ece6-d891-330a-3a459dfa2bd5}" +testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.235Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: pipelining replication: peer="{Voter server-fcf65dbc-909a-19b9-2058-4b94c479035b fcf65dbc-909a-19b9-2058-4b94c479035b}" + testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.235Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: pipelining replication: peer="{Voter server-fcf65dbc-909a-19b9-2058-4b94c479035b fcf65dbc-909a-19b9-2058-4b94c479035b}" +testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.236Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: pipelining replication: peer="{Voter server-130b2734-1a0a-f1c3-6a90-3fd880c1748a 130b2734-1a0a-f1c3-6a90-3fd880c1748a}" + testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.236Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: pipelining replication: peer="{Voter server-130b2734-1a0a-f1c3-6a90-3fd880c1748a 130b2734-1a0a-f1c3-6a90-3fd880c1748a}" +testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.236Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: pipelining replication: peer="{Voter server-00a5042f-6283-5ea0-b291-ddca3bab9553 00a5042f-6283-5ea0-b291-ddca3bab9553}" + testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.236Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: pipelining replication: peer="{Voter server-00a5042f-6283-5ea0-b291-ddca3bab9553 00a5042f-6283-5ea0-b291-ddca3bab9553}" +testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.236Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: pipelining replication: peer="{Voter server-06e0cc33-930b-b8b4-1164-30f2237882ba 06e0cc33-930b-b8b4-1164-30f2237882ba}" + testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.236Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: pipelining replication: peer="{Voter server-06e0cc33-930b-b8b4-1164-30f2237882ba 06e0cc33-930b-b8b4-1164-30f2237882ba}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:05.338Z [INFO] stable state for Leader reached at 2023-08-02 15:13:05.23103949 +0000 UTC m=+17.326975817 (1 nodes), 59.949266ms from start of poll, 61.29534ms from cluster start. Timeout at 2023-08-02 15:13:05.33855993 +0000 UTC m=+17.434496264, 107.520447ms after stability + testing.go:149: cluster: 2023-08-02T15:13:05.338Z [INFO] stable state for Leader reached at 2023-08-02 15:13:05.23103949 +0000 UTC m=+17.326975817 (1 nodes), 59.949266ms from start of poll, 61.29534ms from cluster start. Timeout at 2023-08-02 15:13:05.33855993 +0000 UTC m=+17.434496264, 107.520447ms after stability +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:05.338Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-08-02T15:13:05.338Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:05.443Z [INFO] stable state for Follower reached at 2023-08-02 15:13:05.338740762 +0000 UTC m=+17.434677088 (6 nodes), 3.116µs from start of poll, 168.996611ms from cluster start. Timeout at 2023-08-02 15:13:05.443830348 +0000 UTC m=+17.539766675, 105.089587ms after stability + testing.go:149: cluster: 2023-08-02T15:13:05.443Z [INFO] stable state for Follower reached at 2023-08-02 15:13:05.338740762 +0000 UTC m=+17.434677088 (6 nodes), 3.116µs from start of poll, 168.996611ms from cluster start. Timeout at 2023-08-02 15:13:05.443830348 +0000 UTC m=+17.539766675, 105.089587ms after stability +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:05.443Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:13:05.443Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:05.549Z [INFO] stable state for Leader reached at 2023-08-02 15:13:05.444018076 +0000 UTC m=+17.539954403 (1 nodes), 2.396µs from start of poll, 274.273926ms from cluster start. Timeout at 2023-08-02 15:13:05.549035874 +0000 UTC m=+17.644972204, 105.017801ms after stability + testing.go:149: cluster: 2023-08-02T15:13:05.549Z [INFO] stable state for Leader reached at 2023-08-02 15:13:05.444018076 +0000 UTC m=+17.539954403 (1 nodes), 2.396µs from start of poll, 274.273926ms from cluster start. Timeout at 2023-08-02 15:13:05.549035874 +0000 UTC m=+17.644972204, 105.017801ms after stability +testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: entering candidate state: node="Node at 16c4d59e-8dcd-06b6-dc57-8e1b5667abdd [Candidate]" term=3 + testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: entering candidate state: node="Node at 16c4d59e-8dcd-06b6-dc57-8e1b5667abdd [Candidate]" term=3 +testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.549Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: entering follower state: follower="Node at 74a2e60e-7ba2-6dfb-22b7-2d76da0718bd [Follower]" leader= + testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.549Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: entering follower state: follower="Node at 74a2e60e-7ba2-6dfb-22b7-2d76da0718bd [Follower]" leader= +testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.549Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: aborting pipeline replication: peer="{Voter server-00a5042f-6283-5ea0-b291-ddca3bab9553 00a5042f-6283-5ea0-b291-ddca3bab9553}" + testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.549Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: aborting pipeline replication: peer="{Voter server-00a5042f-6283-5ea0-b291-ddca3bab9553 00a5042f-6283-5ea0-b291-ddca3bab9553}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:05.549Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:13:05.549Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.549Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: aborting pipeline replication: peer="{Voter server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd 16c4d59e-8dcd-06b6-dc57-8e1b5667abdd}" + testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.549Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: aborting pipeline replication: peer="{Voter server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd 16c4d59e-8dcd-06b6-dc57-8e1b5667abdd}" +testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.549Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: aborting pipeline replication: peer="{Voter server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 d2db4dc6-ece6-d891-330a-3a459dfa2bd5}" + testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.549Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: aborting pipeline replication: peer="{Voter server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 d2db4dc6-ece6-d891-330a-3a459dfa2bd5}" +testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.549Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: aborting pipeline replication: peer="{Voter server-06e0cc33-930b-b8b4-1164-30f2237882ba 06e0cc33-930b-b8b4-1164-30f2237882ba}" + testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.549Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: aborting pipeline replication: peer="{Voter server-06e0cc33-930b-b8b4-1164-30f2237882ba 06e0cc33-930b-b8b4-1164-30f2237882ba}" +testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: election won: tally=4 +testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.549Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: aborting pipeline replication: peer="{Voter server-130b2734-1a0a-f1c3-6a90-3fd880c1748a 130b2734-1a0a-f1c3-6a90-3fd880c1748a}" + testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.549Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: aborting pipeline replication: peer="{Voter server-130b2734-1a0a-f1c3-6a90-3fd880c1748a 130b2734-1a0a-f1c3-6a90-3fd880c1748a}" +testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.549Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: aborting pipeline replication: peer="{Voter server-fcf65dbc-909a-19b9-2058-4b94c479035b fcf65dbc-909a-19b9-2058-4b94c479035b}" + testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.549Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: aborting pipeline replication: peer="{Voter server-fcf65dbc-909a-19b9-2058-4b94c479035b fcf65dbc-909a-19b9-2058-4b94c479035b}" + testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: election won: tally=4 +testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: entering leader state: leader="Node at 16c4d59e-8dcd-06b6-dc57-8e1b5667abdd [Leader]" + testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: entering leader state: leader="Node at 16c4d59e-8dcd-06b6-dc57-8e1b5667abdd [Leader]" +testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: added peer, starting replication: peer=server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 + testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: added peer, starting replication: peer=server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 +testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: added peer, starting replication: peer=server-06e0cc33-930b-b8b4-1164-30f2237882ba + testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: added peer, starting replication: peer=server-06e0cc33-930b-b8b4-1164-30f2237882ba +testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: added peer, starting replication: peer=server-130b2734-1a0a-f1c3-6a90-3fd880c1748a + testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: added peer, starting replication: peer=server-130b2734-1a0a-f1c3-6a90-3fd880c1748a +testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: added peer, starting replication: peer=server-fcf65dbc-909a-19b9-2058-4b94c479035b + testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: added peer, starting replication: peer=server-fcf65dbc-909a-19b9-2058-4b94c479035b +testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: added peer, starting replication: peer=server-00a5042f-6283-5ea0-b291-ddca3bab9553 + testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: added peer, starting replication: peer=server-00a5042f-6283-5ea0-b291-ddca3bab9553 +testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: added peer, starting replication: peer=server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd + testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: added peer, starting replication: peer=server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd +testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.550Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: pipelining replication: peer="{Voter server-fcf65dbc-909a-19b9-2058-4b94c479035b fcf65dbc-909a-19b9-2058-4b94c479035b}" + testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.550Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: pipelining replication: peer="{Voter server-fcf65dbc-909a-19b9-2058-4b94c479035b fcf65dbc-909a-19b9-2058-4b94c479035b}" +testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.550Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: pipelining replication: peer="{Voter server-00a5042f-6283-5ea0-b291-ddca3bab9553 00a5042f-6283-5ea0-b291-ddca3bab9553}" + testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.550Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: pipelining replication: peer="{Voter server-00a5042f-6283-5ea0-b291-ddca3bab9553 00a5042f-6283-5ea0-b291-ddca3bab9553}" +testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.550Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: pipelining replication: peer="{Voter server-130b2734-1a0a-f1c3-6a90-3fd880c1748a 130b2734-1a0a-f1c3-6a90-3fd880c1748a}" + testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.550Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: pipelining replication: peer="{Voter server-130b2734-1a0a-f1c3-6a90-3fd880c1748a 130b2734-1a0a-f1c3-6a90-3fd880c1748a}" +testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.550Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: pipelining replication: peer="{Voter server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd 74a2e60e-7ba2-6dfb-22b7-2d76da0718bd}" + testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.550Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: pipelining replication: peer="{Voter server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd 74a2e60e-7ba2-6dfb-22b7-2d76da0718bd}" +testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: pipelining replication: peer="{Voter server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 d2db4dc6-ece6-d891-330a-3a459dfa2bd5}" + testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: pipelining replication: peer="{Voter server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 d2db4dc6-ece6-d891-330a-3a459dfa2bd5}" +testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: pipelining replication: peer="{Voter server-06e0cc33-930b-b8b4-1164-30f2237882ba 06e0cc33-930b-b8b4-1164-30f2237882ba}" + testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: pipelining replication: peer="{Voter server-06e0cc33-930b-b8b4-1164-30f2237882ba 06e0cc33-930b-b8b4-1164-30f2237882ba}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:05.658Z [INFO] stable state for Leader reached at 2023-08-02 15:13:05.549842462 +0000 UTC m=+17.645778805 (1 nodes), 370.663µs from start of poll, 380.098328ms from cluster start. Timeout at 2023-08-02 15:13:05.658567131 +0000 UTC m=+17.754503458, 108.724653ms after stability + testing.go:149: cluster: 2023-08-02T15:13:05.658Z [INFO] stable state for Leader reached at 2023-08-02 15:13:05.549842462 +0000 UTC m=+17.645778805 (1 nodes), 370.663µs from start of poll, 380.098328ms from cluster start. Timeout at 2023-08-02 15:13:05.658567131 +0000 UTC m=+17.754503458, 108.724653ms after stability +testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.658Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: aborting pipeline replication: peer="{Voter server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd 74a2e60e-7ba2-6dfb-22b7-2d76da0718bd}" + testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.658Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: aborting pipeline replication: peer="{Voter server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd 74a2e60e-7ba2-6dfb-22b7-2d76da0718bd}" +testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.658Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: aborting pipeline replication: peer="{Voter server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 d2db4dc6-ece6-d891-330a-3a459dfa2bd5}" + testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.658Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: aborting pipeline replication: peer="{Voter server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 d2db4dc6-ece6-d891-330a-3a459dfa2bd5}" +testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.658Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: aborting pipeline replication: peer="{Voter server-00a5042f-6283-5ea0-b291-ddca3bab9553 00a5042f-6283-5ea0-b291-ddca3bab9553}" + testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.658Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: aborting pipeline replication: peer="{Voter server-00a5042f-6283-5ea0-b291-ddca3bab9553 00a5042f-6283-5ea0-b291-ddca3bab9553}" +testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.658Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: aborting pipeline replication: peer="{Voter server-06e0cc33-930b-b8b4-1164-30f2237882ba 06e0cc33-930b-b8b4-1164-30f2237882ba}" + testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.658Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: aborting pipeline replication: peer="{Voter server-06e0cc33-930b-b8b4-1164-30f2237882ba 06e0cc33-930b-b8b4-1164-30f2237882ba}" +testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.658Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: aborting pipeline replication: peer="{Voter server-130b2734-1a0a-f1c3-6a90-3fd880c1748a 130b2734-1a0a-f1c3-6a90-3fd880c1748a}" + testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.658Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: aborting pipeline replication: peer="{Voter server-130b2734-1a0a-f1c3-6a90-3fd880c1748a 130b2734-1a0a-f1c3-6a90-3fd880c1748a}" +testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.658Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: aborting pipeline replication: peer="{Voter server-fcf65dbc-909a-19b9-2058-4b94c479035b fcf65dbc-909a-19b9-2058-4b94c479035b}" + testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.658Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: aborting pipeline replication: peer="{Voter server-fcf65dbc-909a-19b9-2058-4b94c479035b fcf65dbc-909a-19b9-2058-4b94c479035b}" +testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.709Z [ERROR] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: failed to heartbeat to: peer=06e0cc33-930b-b8b4-1164-30f2237882ba error="command timed out" + testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.709Z [ERROR] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: failed to heartbeat to: peer=06e0cc33-930b-b8b4-1164-30f2237882ba error="command timed out" +--- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.54s) === RUN TestRaft_LeadershipTransferToInvalidID -testLoggerAdapter verbose: server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: 2024-09-03T21:34:36.596Z [INFO] server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e49e2d89-fe7a-0f00-423d-1ee6ba163754 Address:e49e2d89-fe7a-0f00-423d-1ee6ba163754} {Suffrage:Voter ID:server-1b8f777c-c181-addb-b7ae-e2bb740cf455 Address:1b8f777c-c181-addb-b7ae-e2bb740cf455} {Suffrage:Voter ID:server-45314bee-ed50-b29d-8870-1fd2808bf207 Address:45314bee-ed50-b29d-8870-1fd2808bf207}]" - testing.go:149: server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: 2024-09-03T21:34:36.596Z [INFO] server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e49e2d89-fe7a-0f00-423d-1ee6ba163754 Address:e49e2d89-fe7a-0f00-423d-1ee6ba163754} {Suffrage:Voter ID:server-1b8f777c-c181-addb-b7ae-e2bb740cf455 Address:1b8f777c-c181-addb-b7ae-e2bb740cf455} {Suffrage:Voter ID:server-45314bee-ed50-b29d-8870-1fd2808bf207 Address:45314bee-ed50-b29d-8870-1fd2808bf207}]" -testLoggerAdapter verbose: server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: 2024-09-03T21:34:36.596Z [INFO] server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: entering follower state: follower="Node at e49e2d89-fe7a-0f00-423d-1ee6ba163754 [Follower]" leader= - testing.go:149: server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: 2024-09-03T21:34:36.596Z [INFO] server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: entering follower state: follower="Node at e49e2d89-fe7a-0f00-423d-1ee6ba163754 [Follower]" leader= -testLoggerAdapter verbose: server-1b8f777c-c181-addb-b7ae-e2bb740cf455: 2024-09-03T21:34:36.596Z [INFO] server-1b8f777c-c181-addb-b7ae-e2bb740cf455: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e49e2d89-fe7a-0f00-423d-1ee6ba163754 Address:e49e2d89-fe7a-0f00-423d-1ee6ba163754} {Suffrage:Voter ID:server-1b8f777c-c181-addb-b7ae-e2bb740cf455 Address:1b8f777c-c181-addb-b7ae-e2bb740cf455} {Suffrage:Voter ID:server-45314bee-ed50-b29d-8870-1fd2808bf207 Address:45314bee-ed50-b29d-8870-1fd2808bf207}]" - testing.go:149: server-1b8f777c-c181-addb-b7ae-e2bb740cf455: 2024-09-03T21:34:36.596Z [INFO] server-1b8f777c-c181-addb-b7ae-e2bb740cf455: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e49e2d89-fe7a-0f00-423d-1ee6ba163754 Address:e49e2d89-fe7a-0f00-423d-1ee6ba163754} {Suffrage:Voter ID:server-1b8f777c-c181-addb-b7ae-e2bb740cf455 Address:1b8f777c-c181-addb-b7ae-e2bb740cf455} {Suffrage:Voter ID:server-45314bee-ed50-b29d-8870-1fd2808bf207 Address:45314bee-ed50-b29d-8870-1fd2808bf207}]" -testLoggerAdapter verbose: server-45314bee-ed50-b29d-8870-1fd2808bf207: 2024-09-03T21:34:36.596Z [INFO] server-45314bee-ed50-b29d-8870-1fd2808bf207: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e49e2d89-fe7a-0f00-423d-1ee6ba163754 Address:e49e2d89-fe7a-0f00-423d-1ee6ba163754} {Suffrage:Voter ID:server-1b8f777c-c181-addb-b7ae-e2bb740cf455 Address:1b8f777c-c181-addb-b7ae-e2bb740cf455} {Suffrage:Voter ID:server-45314bee-ed50-b29d-8870-1fd2808bf207 Address:45314bee-ed50-b29d-8870-1fd2808bf207}]" - testing.go:149: server-45314bee-ed50-b29d-8870-1fd2808bf207: 2024-09-03T21:34:36.596Z [INFO] server-45314bee-ed50-b29d-8870-1fd2808bf207: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e49e2d89-fe7a-0f00-423d-1ee6ba163754 Address:e49e2d89-fe7a-0f00-423d-1ee6ba163754} {Suffrage:Voter ID:server-1b8f777c-c181-addb-b7ae-e2bb740cf455 Address:1b8f777c-c181-addb-b7ae-e2bb740cf455} {Suffrage:Voter ID:server-45314bee-ed50-b29d-8870-1fd2808bf207 Address:45314bee-ed50-b29d-8870-1fd2808bf207}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:36.596Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:36.596Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-1b8f777c-c181-addb-b7ae-e2bb740cf455: 2024-09-03T21:34:36.596Z [INFO] server-1b8f777c-c181-addb-b7ae-e2bb740cf455: entering follower state: follower="Node at 1b8f777c-c181-addb-b7ae-e2bb740cf455 [Follower]" leader= - testing.go:149: server-1b8f777c-c181-addb-b7ae-e2bb740cf455: 2024-09-03T21:34:36.596Z [INFO] server-1b8f777c-c181-addb-b7ae-e2bb740cf455: entering follower state: follower="Node at 1b8f777c-c181-addb-b7ae-e2bb740cf455 [Follower]" leader= -testLoggerAdapter verbose: server-45314bee-ed50-b29d-8870-1fd2808bf207: 2024-09-03T21:34:36.596Z [INFO] server-45314bee-ed50-b29d-8870-1fd2808bf207: entering follower state: follower="Node at 45314bee-ed50-b29d-8870-1fd2808bf207 [Follower]" leader= - testing.go:149: server-45314bee-ed50-b29d-8870-1fd2808bf207: 2024-09-03T21:34:36.596Z [INFO] server-45314bee-ed50-b29d-8870-1fd2808bf207: entering follower state: follower="Node at 45314bee-ed50-b29d-8870-1fd2808bf207 [Follower]" leader= -testLoggerAdapter verbose: server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: 2024-09-03T21:34:36.650Z [WARN] server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: 2024-09-03T21:34:36.650Z [WARN] server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: 2024-09-03T21:34:36.650Z [INFO] server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: entering candidate state: node="Node at e49e2d89-fe7a-0f00-423d-1ee6ba163754 [Candidate]" term=2 - testing.go:149: server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: 2024-09-03T21:34:36.650Z [INFO] server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: entering candidate state: node="Node at e49e2d89-fe7a-0f00-423d-1ee6ba163754 [Candidate]" term=2 -testLoggerAdapter verbose: server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: 2024-09-03T21:34:36.650Z [INFO] server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: election won: tally=2 - testing.go:149: server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: 2024-09-03T21:34:36.650Z [INFO] server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: election won: tally=2 -testLoggerAdapter verbose: server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: 2024-09-03T21:34:36.650Z [INFO] server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: entering leader state: leader="Node at e49e2d89-fe7a-0f00-423d-1ee6ba163754 [Leader]" - testing.go:149: server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: 2024-09-03T21:34:36.650Z [INFO] server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: entering leader state: leader="Node at e49e2d89-fe7a-0f00-423d-1ee6ba163754 [Leader]" -testLoggerAdapter verbose: server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: 2024-09-03T21:34:36.650Z [INFO] server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: added peer, starting replication: peer=server-1b8f777c-c181-addb-b7ae-e2bb740cf455 - testing.go:149: server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: 2024-09-03T21:34:36.650Z [INFO] server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: added peer, starting replication: peer=server-1b8f777c-c181-addb-b7ae-e2bb740cf455 -testLoggerAdapter verbose: server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: 2024-09-03T21:34:36.650Z [INFO] server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: added peer, starting replication: peer=server-45314bee-ed50-b29d-8870-1fd2808bf207 - testing.go:149: server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: 2024-09-03T21:34:36.650Z [INFO] server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: added peer, starting replication: peer=server-45314bee-ed50-b29d-8870-1fd2808bf207 -testLoggerAdapter verbose: server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: 2024-09-03T21:34:36.650Z [INFO] server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: pipelining replication: peer="{Voter server-45314bee-ed50-b29d-8870-1fd2808bf207 45314bee-ed50-b29d-8870-1fd2808bf207}" - testing.go:149: server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: 2024-09-03T21:34:36.650Z [INFO] server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: pipelining replication: peer="{Voter server-45314bee-ed50-b29d-8870-1fd2808bf207 45314bee-ed50-b29d-8870-1fd2808bf207}" -testLoggerAdapter verbose: server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: 2024-09-03T21:34:36.650Z [INFO] server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: pipelining replication: peer="{Voter server-1b8f777c-c181-addb-b7ae-e2bb740cf455 1b8f777c-c181-addb-b7ae-e2bb740cf455}" - testing.go:149: server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: 2024-09-03T21:34:36.650Z [INFO] server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: pipelining replication: peer="{Voter server-1b8f777c-c181-addb-b7ae-e2bb740cf455 1b8f777c-c181-addb-b7ae-e2bb740cf455}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:36.755Z [INFO] stable state for Leader reached at 2024-09-03 21:34:36.650496063 +0000 UTC m=+17.697153200 (1 nodes), 53.573636ms from start of poll, 54.518561ms from cluster start. Timeout at 2024-09-03 21:34:36.755558978 +0000 UTC m=+17.802216114, 105.062914ms after stability - testing.go:149: cluster: 2024-09-03T21:34:36.755Z [INFO] stable state for Leader reached at 2024-09-03 21:34:36.650496063 +0000 UTC m=+17.697153200 (1 nodes), 53.573636ms from start of poll, 54.518561ms from cluster start. Timeout at 2024-09-03 21:34:36.755558978 +0000 UTC m=+17.802216114, 105.062914ms after stability -testLoggerAdapter verbose: server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: 2024-09-03T21:34:36.755Z [INFO] server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: aborting pipeline replication: peer="{Voter server-45314bee-ed50-b29d-8870-1fd2808bf207 45314bee-ed50-b29d-8870-1fd2808bf207}" - testing.go:149: server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: 2024-09-03T21:34:36.755Z [INFO] server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: aborting pipeline replication: peer="{Voter server-45314bee-ed50-b29d-8870-1fd2808bf207 45314bee-ed50-b29d-8870-1fd2808bf207}" -testLoggerAdapter verbose: server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: 2024-09-03T21:34:36.755Z [INFO] server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: aborting pipeline replication: peer="{Voter server-1b8f777c-c181-addb-b7ae-e2bb740cf455 1b8f777c-c181-addb-b7ae-e2bb740cf455}" - testing.go:149: server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: 2024-09-03T21:34:36.755Z [INFO] server-e49e2d89-fe7a-0f00-423d-1ee6ba163754: aborting pipeline replication: peer="{Voter server-1b8f777c-c181-addb-b7ae-e2bb740cf455 1b8f777c-c181-addb-b7ae-e2bb740cf455}" ---- PASS: TestRaft_LeadershipTransferToInvalidID (0.16s) +testLoggerAdapter verbose: server-7b75885a-58df-7895-aca2-8b53f27dcc4d: 2023-08-02T15:13:05.710Z [INFO] server-7b75885a-58df-7895-aca2-8b53f27dcc4d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b75885a-58df-7895-aca2-8b53f27dcc4d Address:7b75885a-58df-7895-aca2-8b53f27dcc4d} {Suffrage:Voter ID:server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93 Address:ab6bbf92-dd6c-152a-570b-7bdb8d8def93} {Suffrage:Voter ID:server-c540ccc5-51a2-2b80-4840-6b9cf369567b Address:c540ccc5-51a2-2b80-4840-6b9cf369567b}]" + testing.go:149: server-7b75885a-58df-7895-aca2-8b53f27dcc4d: 2023-08-02T15:13:05.710Z [INFO] server-7b75885a-58df-7895-aca2-8b53f27dcc4d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b75885a-58df-7895-aca2-8b53f27dcc4d Address:7b75885a-58df-7895-aca2-8b53f27dcc4d} {Suffrage:Voter ID:server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93 Address:ab6bbf92-dd6c-152a-570b-7bdb8d8def93} {Suffrage:Voter ID:server-c540ccc5-51a2-2b80-4840-6b9cf369567b Address:c540ccc5-51a2-2b80-4840-6b9cf369567b}]" +testLoggerAdapter verbose: server-7b75885a-58df-7895-aca2-8b53f27dcc4d: 2023-08-02T15:13:05.711Z [INFO] server-7b75885a-58df-7895-aca2-8b53f27dcc4d: entering follower state: follower="Node at 7b75885a-58df-7895-aca2-8b53f27dcc4d [Follower]" leader= + testing.go:149: server-7b75885a-58df-7895-aca2-8b53f27dcc4d: 2023-08-02T15:13:05.711Z [INFO] server-7b75885a-58df-7895-aca2-8b53f27dcc4d: entering follower state: follower="Node at 7b75885a-58df-7895-aca2-8b53f27dcc4d [Follower]" leader= +testLoggerAdapter verbose: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.711Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b75885a-58df-7895-aca2-8b53f27dcc4d Address:7b75885a-58df-7895-aca2-8b53f27dcc4d} {Suffrage:Voter ID:server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93 Address:ab6bbf92-dd6c-152a-570b-7bdb8d8def93} {Suffrage:Voter ID:server-c540ccc5-51a2-2b80-4840-6b9cf369567b Address:c540ccc5-51a2-2b80-4840-6b9cf369567b}]" + testing.go:149: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.711Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b75885a-58df-7895-aca2-8b53f27dcc4d Address:7b75885a-58df-7895-aca2-8b53f27dcc4d} {Suffrage:Voter ID:server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93 Address:ab6bbf92-dd6c-152a-570b-7bdb8d8def93} {Suffrage:Voter ID:server-c540ccc5-51a2-2b80-4840-6b9cf369567b Address:c540ccc5-51a2-2b80-4840-6b9cf369567b}]" +testLoggerAdapter verbose: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.711Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: entering follower state: follower="Node at ab6bbf92-dd6c-152a-570b-7bdb8d8def93 [Follower]" leader= + testing.go:149: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.711Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: entering follower state: follower="Node at ab6bbf92-dd6c-152a-570b-7bdb8d8def93 [Follower]" leader= +testLoggerAdapter verbose: server-c540ccc5-51a2-2b80-4840-6b9cf369567b: 2023-08-02T15:13:05.711Z [INFO] server-c540ccc5-51a2-2b80-4840-6b9cf369567b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b75885a-58df-7895-aca2-8b53f27dcc4d Address:7b75885a-58df-7895-aca2-8b53f27dcc4d} {Suffrage:Voter ID:server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93 Address:ab6bbf92-dd6c-152a-570b-7bdb8d8def93} {Suffrage:Voter ID:server-c540ccc5-51a2-2b80-4840-6b9cf369567b Address:c540ccc5-51a2-2b80-4840-6b9cf369567b}]" + testing.go:149: server-c540ccc5-51a2-2b80-4840-6b9cf369567b: 2023-08-02T15:13:05.711Z [INFO] server-c540ccc5-51a2-2b80-4840-6b9cf369567b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b75885a-58df-7895-aca2-8b53f27dcc4d Address:7b75885a-58df-7895-aca2-8b53f27dcc4d} {Suffrage:Voter ID:server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93 Address:ab6bbf92-dd6c-152a-570b-7bdb8d8def93} {Suffrage:Voter ID:server-c540ccc5-51a2-2b80-4840-6b9cf369567b Address:c540ccc5-51a2-2b80-4840-6b9cf369567b}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:05.711Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:13:05.711Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-c540ccc5-51a2-2b80-4840-6b9cf369567b: 2023-08-02T15:13:05.711Z [INFO] server-c540ccc5-51a2-2b80-4840-6b9cf369567b: entering follower state: follower="Node at c540ccc5-51a2-2b80-4840-6b9cf369567b [Follower]" leader= + testing.go:149: server-c540ccc5-51a2-2b80-4840-6b9cf369567b: 2023-08-02T15:13:05.711Z [INFO] server-c540ccc5-51a2-2b80-4840-6b9cf369567b: entering follower state: follower="Node at c540ccc5-51a2-2b80-4840-6b9cf369567b [Follower]" leader= +testLoggerAdapter verbose: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.769Z [WARN] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.769Z [WARN] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.769Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: entering candidate state: node="Node at ab6bbf92-dd6c-152a-570b-7bdb8d8def93 [Candidate]" term=2 + testing.go:149: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.769Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: entering candidate state: node="Node at ab6bbf92-dd6c-152a-570b-7bdb8d8def93 [Candidate]" term=2 +testLoggerAdapter verbose: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.770Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: election won: tally=2 + testing.go:149: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.770Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: election won: tally=2 +testLoggerAdapter verbose: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.770Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: entering leader state: leader="Node at ab6bbf92-dd6c-152a-570b-7bdb8d8def93 [Leader]" + testing.go:149: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.770Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: entering leader state: leader="Node at ab6bbf92-dd6c-152a-570b-7bdb8d8def93 [Leader]" +testLoggerAdapter verbose: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.770Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: added peer, starting replication: peer=server-7b75885a-58df-7895-aca2-8b53f27dcc4d + testing.go:149: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.770Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: added peer, starting replication: peer=server-7b75885a-58df-7895-aca2-8b53f27dcc4d +testLoggerAdapter verbose: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.770Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: added peer, starting replication: peer=server-c540ccc5-51a2-2b80-4840-6b9cf369567b + testing.go:149: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.770Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: added peer, starting replication: peer=server-c540ccc5-51a2-2b80-4840-6b9cf369567b +testLoggerAdapter verbose: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.770Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: pipelining replication: peer="{Voter server-c540ccc5-51a2-2b80-4840-6b9cf369567b c540ccc5-51a2-2b80-4840-6b9cf369567b}" + testing.go:149: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.770Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: pipelining replication: peer="{Voter server-c540ccc5-51a2-2b80-4840-6b9cf369567b c540ccc5-51a2-2b80-4840-6b9cf369567b}" +testLoggerAdapter verbose: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.770Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: pipelining replication: peer="{Voter server-7b75885a-58df-7895-aca2-8b53f27dcc4d 7b75885a-58df-7895-aca2-8b53f27dcc4d}" + testing.go:149: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.770Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: pipelining replication: peer="{Voter server-7b75885a-58df-7895-aca2-8b53f27dcc4d 7b75885a-58df-7895-aca2-8b53f27dcc4d}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:05.875Z [INFO] stable state for Leader reached at 2023-08-02 15:13:05.77032997 +0000 UTC m=+17.866266298 (1 nodes), 58.695271ms from start of poll, 59.523058ms from cluster start. Timeout at 2023-08-02 15:13:05.875657213 +0000 UTC m=+17.971593547, 105.327249ms after stability + testing.go:149: cluster: 2023-08-02T15:13:05.875Z [INFO] stable state for Leader reached at 2023-08-02 15:13:05.77032997 +0000 UTC m=+17.866266298 (1 nodes), 58.695271ms from start of poll, 59.523058ms from cluster start. Timeout at 2023-08-02 15:13:05.875657213 +0000 UTC m=+17.971593547, 105.327249ms after stability +testLoggerAdapter verbose: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.875Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: aborting pipeline replication: peer="{Voter server-c540ccc5-51a2-2b80-4840-6b9cf369567b c540ccc5-51a2-2b80-4840-6b9cf369567b}" + testing.go:149: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.875Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: aborting pipeline replication: peer="{Voter server-c540ccc5-51a2-2b80-4840-6b9cf369567b c540ccc5-51a2-2b80-4840-6b9cf369567b}" +testLoggerAdapter verbose: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.875Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: aborting pipeline replication: peer="{Voter server-7b75885a-58df-7895-aca2-8b53f27dcc4d 7b75885a-58df-7895-aca2-8b53f27dcc4d}" + testing.go:149: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.875Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: aborting pipeline replication: peer="{Voter server-7b75885a-58df-7895-aca2-8b53f27dcc4d 7b75885a-58df-7895-aca2-8b53f27dcc4d}" +--- PASS: TestRaft_LeadershipTransferToInvalidID (0.17s) === RUN TestRaft_LeadershipTransferToInvalidAddress -testLoggerAdapter verbose: server-cda57a40-a952-f20f-9744-9f1649baac06: 2024-09-03T21:34:36.756Z [INFO] server-cda57a40-a952-f20f-9744-9f1649baac06: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cda57a40-a952-f20f-9744-9f1649baac06 Address:cda57a40-a952-f20f-9744-9f1649baac06} {Suffrage:Voter ID:server-7efed191-ae15-da60-e5fc-c715213eeef5 Address:7efed191-ae15-da60-e5fc-c715213eeef5} {Suffrage:Voter ID:server-e4755f27-e738-fe52-86dd-ff8af0d55560 Address:e4755f27-e738-fe52-86dd-ff8af0d55560}]" - testing.go:149: server-cda57a40-a952-f20f-9744-9f1649baac06: 2024-09-03T21:34:36.756Z [INFO] server-cda57a40-a952-f20f-9744-9f1649baac06: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cda57a40-a952-f20f-9744-9f1649baac06 Address:cda57a40-a952-f20f-9744-9f1649baac06} {Suffrage:Voter ID:server-7efed191-ae15-da60-e5fc-c715213eeef5 Address:7efed191-ae15-da60-e5fc-c715213eeef5} {Suffrage:Voter ID:server-e4755f27-e738-fe52-86dd-ff8af0d55560 Address:e4755f27-e738-fe52-86dd-ff8af0d55560}]" -testLoggerAdapter verbose: server-7efed191-ae15-da60-e5fc-c715213eeef5: 2024-09-03T21:34:36.756Z [INFO] server-7efed191-ae15-da60-e5fc-c715213eeef5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cda57a40-a952-f20f-9744-9f1649baac06 Address:cda57a40-a952-f20f-9744-9f1649baac06} {Suffrage:Voter ID:server-7efed191-ae15-da60-e5fc-c715213eeef5 Address:7efed191-ae15-da60-e5fc-c715213eeef5} {Suffrage:Voter ID:server-e4755f27-e738-fe52-86dd-ff8af0d55560 Address:e4755f27-e738-fe52-86dd-ff8af0d55560}]" - testing.go:149: server-7efed191-ae15-da60-e5fc-c715213eeef5: 2024-09-03T21:34:36.756Z [INFO] server-7efed191-ae15-da60-e5fc-c715213eeef5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cda57a40-a952-f20f-9744-9f1649baac06 Address:cda57a40-a952-f20f-9744-9f1649baac06} {Suffrage:Voter ID:server-7efed191-ae15-da60-e5fc-c715213eeef5 Address:7efed191-ae15-da60-e5fc-c715213eeef5} {Suffrage:Voter ID:server-e4755f27-e738-fe52-86dd-ff8af0d55560 Address:e4755f27-e738-fe52-86dd-ff8af0d55560}]" -testLoggerAdapter verbose: server-e4755f27-e738-fe52-86dd-ff8af0d55560: 2024-09-03T21:34:36.756Z [INFO] server-e4755f27-e738-fe52-86dd-ff8af0d55560: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cda57a40-a952-f20f-9744-9f1649baac06 Address:cda57a40-a952-f20f-9744-9f1649baac06} {Suffrage:Voter ID:server-7efed191-ae15-da60-e5fc-c715213eeef5 Address:7efed191-ae15-da60-e5fc-c715213eeef5} {Suffrage:Voter ID:server-e4755f27-e738-fe52-86dd-ff8af0d55560 Address:e4755f27-e738-fe52-86dd-ff8af0d55560}]" - testing.go:149: server-e4755f27-e738-fe52-86dd-ff8af0d55560: 2024-09-03T21:34:36.756Z [INFO] server-e4755f27-e738-fe52-86dd-ff8af0d55560: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cda57a40-a952-f20f-9744-9f1649baac06 Address:cda57a40-a952-f20f-9744-9f1649baac06} {Suffrage:Voter ID:server-7efed191-ae15-da60-e5fc-c715213eeef5 Address:7efed191-ae15-da60-e5fc-c715213eeef5} {Suffrage:Voter ID:server-e4755f27-e738-fe52-86dd-ff8af0d55560 Address:e4755f27-e738-fe52-86dd-ff8af0d55560}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:36.756Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-09-03T21:34:36.756Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-cda57a40-a952-f20f-9744-9f1649baac06: 2024-09-03T21:34:36.756Z [INFO] server-cda57a40-a952-f20f-9744-9f1649baac06: entering follower state: follower="Node at cda57a40-a952-f20f-9744-9f1649baac06 [Follower]" leader= - testing.go:149: server-cda57a40-a952-f20f-9744-9f1649baac06: 2024-09-03T21:34:36.756Z [INFO] server-cda57a40-a952-f20f-9744-9f1649baac06: entering follower state: follower="Node at cda57a40-a952-f20f-9744-9f1649baac06 [Follower]" leader= -testLoggerAdapter verbose: server-7efed191-ae15-da60-e5fc-c715213eeef5: 2024-09-03T21:34:36.756Z [INFO] server-7efed191-ae15-da60-e5fc-c715213eeef5: entering follower state: follower="Node at 7efed191-ae15-da60-e5fc-c715213eeef5 [Follower]" leader= - testing.go:149: server-7efed191-ae15-da60-e5fc-c715213eeef5: 2024-09-03T21:34:36.756Z [INFO] server-7efed191-ae15-da60-e5fc-c715213eeef5: entering follower state: follower="Node at 7efed191-ae15-da60-e5fc-c715213eeef5 [Follower]" leader= -testLoggerAdapter verbose: server-e4755f27-e738-fe52-86dd-ff8af0d55560: 2024-09-03T21:34:36.756Z [INFO] server-e4755f27-e738-fe52-86dd-ff8af0d55560: entering follower state: follower="Node at e4755f27-e738-fe52-86dd-ff8af0d55560 [Follower]" leader= - testing.go:149: server-e4755f27-e738-fe52-86dd-ff8af0d55560: 2024-09-03T21:34:36.756Z [INFO] server-e4755f27-e738-fe52-86dd-ff8af0d55560: entering follower state: follower="Node at e4755f27-e738-fe52-86dd-ff8af0d55560 [Follower]" leader= -testLoggerAdapter verbose: server-e4755f27-e738-fe52-86dd-ff8af0d55560: 2024-09-03T21:34:36.828Z [WARN] server-e4755f27-e738-fe52-86dd-ff8af0d55560: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-e4755f27-e738-fe52-86dd-ff8af0d55560: 2024-09-03T21:34:36.828Z [WARN] server-e4755f27-e738-fe52-86dd-ff8af0d55560: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-e4755f27-e738-fe52-86dd-ff8af0d55560: 2024-09-03T21:34:36.829Z [INFO] server-e4755f27-e738-fe52-86dd-ff8af0d55560: entering candidate state: node="Node at e4755f27-e738-fe52-86dd-ff8af0d55560 [Candidate]" term=2 - testing.go:149: server-e4755f27-e738-fe52-86dd-ff8af0d55560: 2024-09-03T21:34:36.829Z [INFO] server-e4755f27-e738-fe52-86dd-ff8af0d55560: entering candidate state: node="Node at e4755f27-e738-fe52-86dd-ff8af0d55560 [Candidate]" term=2 -testLoggerAdapter verbose: server-e4755f27-e738-fe52-86dd-ff8af0d55560: 2024-09-03T21:34:36.829Z [INFO] server-e4755f27-e738-fe52-86dd-ff8af0d55560: election won: tally=2 - testing.go:149: server-e4755f27-e738-fe52-86dd-ff8af0d55560: 2024-09-03T21:34:36.829Z [INFO] server-e4755f27-e738-fe52-86dd-ff8af0d55560: election won: tally=2 -testLoggerAdapter verbose: server-e4755f27-e738-fe52-86dd-ff8af0d55560: 2024-09-03T21:34:36.829Z [INFO] server-e4755f27-e738-fe52-86dd-ff8af0d55560: entering leader state: leader="Node at e4755f27-e738-fe52-86dd-ff8af0d55560 [Leader]" - testing.go:149: server-e4755f27-e738-fe52-86dd-ff8af0d55560: 2024-09-03T21:34:36.829Z [INFO] server-e4755f27-e738-fe52-86dd-ff8af0d55560: entering leader state: leader="Node at e4755f27-e738-fe52-86dd-ff8af0d55560 [Leader]" -testLoggerAdapter verbose: server-e4755f27-e738-fe52-86dd-ff8af0d55560: 2024-09-03T21:34:36.829Z [INFO] server-e4755f27-e738-fe52-86dd-ff8af0d55560: added peer, starting replication: peer=server-cda57a40-a952-f20f-9744-9f1649baac06 - testing.go:149: server-e4755f27-e738-fe52-86dd-ff8af0d55560: 2024-09-03T21:34:36.829Z [INFO] server-e4755f27-e738-fe52-86dd-ff8af0d55560: added peer, starting replication: peer=server-cda57a40-a952-f20f-9744-9f1649baac06 -testLoggerAdapter verbose: server-e4755f27-e738-fe52-86dd-ff8af0d55560: 2024-09-03T21:34:36.829Z [INFO] server-e4755f27-e738-fe52-86dd-ff8af0d55560: added peer, starting replication: peer=server-7efed191-ae15-da60-e5fc-c715213eeef5 - testing.go:149: server-e4755f27-e738-fe52-86dd-ff8af0d55560: 2024-09-03T21:34:36.829Z [INFO] server-e4755f27-e738-fe52-86dd-ff8af0d55560: added peer, starting replication: peer=server-7efed191-ae15-da60-e5fc-c715213eeef5 -testLoggerAdapter verbose: server-e4755f27-e738-fe52-86dd-ff8af0d55560: 2024-09-03T21:34:36.829Z [INFO] server-e4755f27-e738-fe52-86dd-ff8af0d55560: pipelining replication: peer="{Voter server-7efed191-ae15-da60-e5fc-c715213eeef5 7efed191-ae15-da60-e5fc-c715213eeef5}" - testing.go:149: server-e4755f27-e738-fe52-86dd-ff8af0d55560: 2024-09-03T21:34:36.829Z [INFO] server-e4755f27-e738-fe52-86dd-ff8af0d55560: pipelining replication: peer="{Voter server-7efed191-ae15-da60-e5fc-c715213eeef5 7efed191-ae15-da60-e5fc-c715213eeef5}" -testLoggerAdapter verbose: server-e4755f27-e738-fe52-86dd-ff8af0d55560: 2024-09-03T21:34:36.829Z [INFO] server-e4755f27-e738-fe52-86dd-ff8af0d55560: pipelining replication: peer="{Voter server-cda57a40-a952-f20f-9744-9f1649baac06 cda57a40-a952-f20f-9744-9f1649baac06}" - testing.go:149: server-e4755f27-e738-fe52-86dd-ff8af0d55560: 2024-09-03T21:34:36.829Z [INFO] server-e4755f27-e738-fe52-86dd-ff8af0d55560: pipelining replication: peer="{Voter server-cda57a40-a952-f20f-9744-9f1649baac06 cda57a40-a952-f20f-9744-9f1649baac06}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:36.934Z [INFO] stable state for Follower reached at 2024-09-03 21:34:36.82955359 +0000 UTC m=+17.876210726 (2 nodes), 72.723878ms from start of poll, 73.110923ms from cluster start. Timeout at 2024-09-03 21:34:36.934635826 +0000 UTC m=+17.981292961, 105.082235ms after stability - testing.go:149: cluster: 2024-09-03T21:34:36.934Z [INFO] stable state for Follower reached at 2024-09-03 21:34:36.82955359 +0000 UTC m=+17.876210726 (2 nodes), 72.723878ms from start of poll, 73.110923ms from cluster start. Timeout at 2024-09-03 21:34:36.934635826 +0000 UTC m=+17.981292961, 105.082235ms after stability -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:36.934Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:36.934Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:37.039Z [INFO] stable state for Leader reached at 2024-09-03 21:34:36.934819239 +0000 UTC m=+17.981476373 (1 nodes), 851ns from start of poll, 178.37657ms from cluster start. Timeout at 2024-09-03 21:34:37.039904784 +0000 UTC m=+18.086561927, 105.085554ms after stability - testing.go:149: cluster: 2024-09-03T21:34:37.039Z [INFO] stable state for Leader reached at 2024-09-03 21:34:36.934819239 +0000 UTC m=+17.981476373 (1 nodes), 851ns from start of poll, 178.37657ms from cluster start. Timeout at 2024-09-03 21:34:37.039904784 +0000 UTC m=+18.086561927, 105.085554ms after stability -testLoggerAdapter verbose: server-e4755f27-e738-fe52-86dd-ff8af0d55560: 2024-09-03T21:34:37.040Z [INFO] server-e4755f27-e738-fe52-86dd-ff8af0d55560: aborting pipeline replication: peer="{Voter server-7efed191-ae15-da60-e5fc-c715213eeef5 7efed191-ae15-da60-e5fc-c715213eeef5}" - testing.go:149: server-e4755f27-e738-fe52-86dd-ff8af0d55560: 2024-09-03T21:34:37.040Z [INFO] server-e4755f27-e738-fe52-86dd-ff8af0d55560: aborting pipeline replication: peer="{Voter server-7efed191-ae15-da60-e5fc-c715213eeef5 7efed191-ae15-da60-e5fc-c715213eeef5}" -testLoggerAdapter verbose: server-e4755f27-e738-fe52-86dd-ff8af0d55560: 2024-09-03T21:34:37.040Z [INFO] server-e4755f27-e738-fe52-86dd-ff8af0d55560: aborting pipeline replication: peer="{Voter server-cda57a40-a952-f20f-9744-9f1649baac06 cda57a40-a952-f20f-9744-9f1649baac06}" - testing.go:149: server-e4755f27-e738-fe52-86dd-ff8af0d55560: 2024-09-03T21:34:37.040Z [INFO] server-e4755f27-e738-fe52-86dd-ff8af0d55560: aborting pipeline replication: peer="{Voter server-cda57a40-a952-f20f-9744-9f1649baac06 cda57a40-a952-f20f-9744-9f1649baac06}" +testLoggerAdapter verbose: server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb: 2023-08-02T15:13:05.879Z [INFO] server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb Address:8310b25d-94b0-b733-7d80-ee5e4ca6f0bb} {Suffrage:Voter ID:server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f Address:ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f} {Suffrage:Voter ID:server-9ace28e7-af59-7e83-274a-696f16352763 Address:9ace28e7-af59-7e83-274a-696f16352763}]" + testing.go:149: server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb: 2023-08-02T15:13:05.879Z [INFO] server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb Address:8310b25d-94b0-b733-7d80-ee5e4ca6f0bb} {Suffrage:Voter ID:server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f Address:ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f} {Suffrage:Voter ID:server-9ace28e7-af59-7e83-274a-696f16352763 Address:9ace28e7-af59-7e83-274a-696f16352763}]" +testLoggerAdapter verbose: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.879Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb Address:8310b25d-94b0-b733-7d80-ee5e4ca6f0bb} {Suffrage:Voter ID:server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f Address:ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f} {Suffrage:Voter ID:server-9ace28e7-af59-7e83-274a-696f16352763 Address:9ace28e7-af59-7e83-274a-696f16352763}]" + testing.go:149: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.879Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb Address:8310b25d-94b0-b733-7d80-ee5e4ca6f0bb} {Suffrage:Voter ID:server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f Address:ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f} {Suffrage:Voter ID:server-9ace28e7-af59-7e83-274a-696f16352763 Address:9ace28e7-af59-7e83-274a-696f16352763}]" +testLoggerAdapter verbose: server-9ace28e7-af59-7e83-274a-696f16352763: 2023-08-02T15:13:05.880Z [INFO] server-9ace28e7-af59-7e83-274a-696f16352763: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb Address:8310b25d-94b0-b733-7d80-ee5e4ca6f0bb} {Suffrage:Voter ID:server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f Address:ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f} {Suffrage:Voter ID:server-9ace28e7-af59-7e83-274a-696f16352763 Address:9ace28e7-af59-7e83-274a-696f16352763}]" + testing.go:149: server-9ace28e7-af59-7e83-274a-696f16352763: 2023-08-02T15:13:05.880Z [INFO] server-9ace28e7-af59-7e83-274a-696f16352763: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb Address:8310b25d-94b0-b733-7d80-ee5e4ca6f0bb} {Suffrage:Voter ID:server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f Address:ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f} {Suffrage:Voter ID:server-9ace28e7-af59-7e83-274a-696f16352763 Address:9ace28e7-af59-7e83-274a-696f16352763}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:05.880Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-08-02T15:13:05.880Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb: 2023-08-02T15:13:05.880Z [INFO] server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb: entering follower state: follower="Node at 8310b25d-94b0-b733-7d80-ee5e4ca6f0bb [Follower]" leader= + testing.go:149: server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb: 2023-08-02T15:13:05.880Z [INFO] server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb: entering follower state: follower="Node at 8310b25d-94b0-b733-7d80-ee5e4ca6f0bb [Follower]" leader= +testLoggerAdapter verbose: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.880Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: entering follower state: follower="Node at ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f [Follower]" leader= + testing.go:149: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.880Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: entering follower state: follower="Node at ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f [Follower]" leader= +testLoggerAdapter verbose: server-9ace28e7-af59-7e83-274a-696f16352763: 2023-08-02T15:13:05.880Z [INFO] server-9ace28e7-af59-7e83-274a-696f16352763: entering follower state: follower="Node at 9ace28e7-af59-7e83-274a-696f16352763 [Follower]" leader= + testing.go:149: server-9ace28e7-af59-7e83-274a-696f16352763: 2023-08-02T15:13:05.880Z [INFO] server-9ace28e7-af59-7e83-274a-696f16352763: entering follower state: follower="Node at 9ace28e7-af59-7e83-274a-696f16352763 [Follower]" leader= +testLoggerAdapter verbose: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.941Z [WARN] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.941Z [WARN] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.942Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: entering candidate state: node="Node at ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f [Candidate]" term=2 + testing.go:149: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.942Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: entering candidate state: node="Node at ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f [Candidate]" term=2 +testLoggerAdapter verbose: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.942Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: election won: tally=2 + testing.go:149: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.942Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: election won: tally=2 +testLoggerAdapter verbose: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.942Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: entering leader state: leader="Node at ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f [Leader]" + testing.go:149: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.942Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: entering leader state: leader="Node at ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f [Leader]" +testLoggerAdapter verbose: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.942Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: added peer, starting replication: peer=server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb + testing.go:149: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.942Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: added peer, starting replication: peer=server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb +testLoggerAdapter verbose: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.942Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: added peer, starting replication: peer=server-9ace28e7-af59-7e83-274a-696f16352763 + testing.go:149: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.942Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: added peer, starting replication: peer=server-9ace28e7-af59-7e83-274a-696f16352763 +testLoggerAdapter verbose: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.942Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: pipelining replication: peer="{Voter server-9ace28e7-af59-7e83-274a-696f16352763 9ace28e7-af59-7e83-274a-696f16352763}" + testing.go:149: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.942Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: pipelining replication: peer="{Voter server-9ace28e7-af59-7e83-274a-696f16352763 9ace28e7-af59-7e83-274a-696f16352763}" +testLoggerAdapter verbose: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.942Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: pipelining replication: peer="{Voter server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb 8310b25d-94b0-b733-7d80-ee5e4ca6f0bb}" + testing.go:149: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.942Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: pipelining replication: peer="{Voter server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb 8310b25d-94b0-b733-7d80-ee5e4ca6f0bb}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:06.049Z [INFO] stable state for Follower reached at 2023-08-02 15:13:05.942635779 +0000 UTC m=+18.038572113 (2 nodes), 62.427552ms from start of poll, 63.226792ms from cluster start. Timeout at 2023-08-02 15:13:06.049021964 +0000 UTC m=+18.144958291, 106.386178ms after stability + testing.go:149: cluster: 2023-08-02T15:13:06.049Z [INFO] stable state for Follower reached at 2023-08-02 15:13:05.942635779 +0000 UTC m=+18.038572113 (2 nodes), 62.427552ms from start of poll, 63.226792ms from cluster start. Timeout at 2023-08-02 15:13:06.049021964 +0000 UTC m=+18.144958291, 106.386178ms after stability +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:06.049Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:13:06.049Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:06.154Z [INFO] stable state for Leader reached at 2023-08-02 15:13:06.049177891 +0000 UTC m=+18.145114217 (1 nodes), 2.057µs from start of poll, 169.768896ms from cluster start. Timeout at 2023-08-02 15:13:06.154393648 +0000 UTC m=+18.250329981, 105.215764ms after stability + testing.go:149: cluster: 2023-08-02T15:13:06.154Z [INFO] stable state for Leader reached at 2023-08-02 15:13:06.049177891 +0000 UTC m=+18.145114217 (1 nodes), 2.057µs from start of poll, 169.768896ms from cluster start. Timeout at 2023-08-02 15:13:06.154393648 +0000 UTC m=+18.250329981, 105.215764ms after stability +testLoggerAdapter verbose: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:06.154Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: aborting pipeline replication: peer="{Voter server-9ace28e7-af59-7e83-274a-696f16352763 9ace28e7-af59-7e83-274a-696f16352763}" + testing.go:149: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:06.154Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: aborting pipeline replication: peer="{Voter server-9ace28e7-af59-7e83-274a-696f16352763 9ace28e7-af59-7e83-274a-696f16352763}" +testLoggerAdapter verbose: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:06.154Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: aborting pipeline replication: peer="{Voter server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb 8310b25d-94b0-b733-7d80-ee5e4ca6f0bb}" + testing.go:149: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:06.154Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: aborting pipeline replication: peer="{Voter server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb 8310b25d-94b0-b733-7d80-ee5e4ca6f0bb}" --- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.28s) === RUN TestRaft_LeadershipTransferToBehindServer -testLoggerAdapter verbose: server-34f94242-c236-7328-fe14-1c69eba03247: 2024-09-03T21:34:37.040Z [INFO] server-34f94242-c236-7328-fe14-1c69eba03247: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-34f94242-c236-7328-fe14-1c69eba03247 Address:34f94242-c236-7328-fe14-1c69eba03247} {Suffrage:Voter ID:server-365d0a07-ea1a-4df9-edf3-443c2010331f Address:365d0a07-ea1a-4df9-edf3-443c2010331f} {Suffrage:Voter ID:server-6f025d68-e340-6b0e-235f-65d8c1b10663 Address:6f025d68-e340-6b0e-235f-65d8c1b10663}]" - testing.go:149: server-34f94242-c236-7328-fe14-1c69eba03247: 2024-09-03T21:34:37.040Z [INFO] server-34f94242-c236-7328-fe14-1c69eba03247: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-34f94242-c236-7328-fe14-1c69eba03247 Address:34f94242-c236-7328-fe14-1c69eba03247} {Suffrage:Voter ID:server-365d0a07-ea1a-4df9-edf3-443c2010331f Address:365d0a07-ea1a-4df9-edf3-443c2010331f} {Suffrage:Voter ID:server-6f025d68-e340-6b0e-235f-65d8c1b10663 Address:6f025d68-e340-6b0e-235f-65d8c1b10663}]" -testLoggerAdapter verbose: server-365d0a07-ea1a-4df9-edf3-443c2010331f: 2024-09-03T21:34:37.041Z [INFO] server-365d0a07-ea1a-4df9-edf3-443c2010331f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-34f94242-c236-7328-fe14-1c69eba03247 Address:34f94242-c236-7328-fe14-1c69eba03247} {Suffrage:Voter ID:server-365d0a07-ea1a-4df9-edf3-443c2010331f Address:365d0a07-ea1a-4df9-edf3-443c2010331f} {Suffrage:Voter ID:server-6f025d68-e340-6b0e-235f-65d8c1b10663 Address:6f025d68-e340-6b0e-235f-65d8c1b10663}]" - testing.go:149: server-365d0a07-ea1a-4df9-edf3-443c2010331f: 2024-09-03T21:34:37.041Z [INFO] server-365d0a07-ea1a-4df9-edf3-443c2010331f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-34f94242-c236-7328-fe14-1c69eba03247 Address:34f94242-c236-7328-fe14-1c69eba03247} {Suffrage:Voter ID:server-365d0a07-ea1a-4df9-edf3-443c2010331f Address:365d0a07-ea1a-4df9-edf3-443c2010331f} {Suffrage:Voter ID:server-6f025d68-e340-6b0e-235f-65d8c1b10663 Address:6f025d68-e340-6b0e-235f-65d8c1b10663}]" -testLoggerAdapter verbose: server-6f025d68-e340-6b0e-235f-65d8c1b10663: 2024-09-03T21:34:37.041Z [INFO] server-6f025d68-e340-6b0e-235f-65d8c1b10663: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-34f94242-c236-7328-fe14-1c69eba03247 Address:34f94242-c236-7328-fe14-1c69eba03247} {Suffrage:Voter ID:server-365d0a07-ea1a-4df9-edf3-443c2010331f Address:365d0a07-ea1a-4df9-edf3-443c2010331f} {Suffrage:Voter ID:server-6f025d68-e340-6b0e-235f-65d8c1b10663 Address:6f025d68-e340-6b0e-235f-65d8c1b10663}]" - testing.go:149: server-6f025d68-e340-6b0e-235f-65d8c1b10663: 2024-09-03T21:34:37.041Z [INFO] server-6f025d68-e340-6b0e-235f-65d8c1b10663: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-34f94242-c236-7328-fe14-1c69eba03247 Address:34f94242-c236-7328-fe14-1c69eba03247} {Suffrage:Voter ID:server-365d0a07-ea1a-4df9-edf3-443c2010331f Address:365d0a07-ea1a-4df9-edf3-443c2010331f} {Suffrage:Voter ID:server-6f025d68-e340-6b0e-235f-65d8c1b10663 Address:6f025d68-e340-6b0e-235f-65d8c1b10663}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:37.041Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:37.041Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-34f94242-c236-7328-fe14-1c69eba03247: 2024-09-03T21:34:37.041Z [INFO] server-34f94242-c236-7328-fe14-1c69eba03247: entering follower state: follower="Node at 34f94242-c236-7328-fe14-1c69eba03247 [Follower]" leader= - testing.go:149: server-34f94242-c236-7328-fe14-1c69eba03247: 2024-09-03T21:34:37.041Z [INFO] server-34f94242-c236-7328-fe14-1c69eba03247: entering follower state: follower="Node at 34f94242-c236-7328-fe14-1c69eba03247 [Follower]" leader= -testLoggerAdapter verbose: server-365d0a07-ea1a-4df9-edf3-443c2010331f: 2024-09-03T21:34:37.041Z [INFO] server-365d0a07-ea1a-4df9-edf3-443c2010331f: entering follower state: follower="Node at 365d0a07-ea1a-4df9-edf3-443c2010331f [Follower]" leader= - testing.go:149: server-365d0a07-ea1a-4df9-edf3-443c2010331f: 2024-09-03T21:34:37.041Z [INFO] server-365d0a07-ea1a-4df9-edf3-443c2010331f: entering follower state: follower="Node at 365d0a07-ea1a-4df9-edf3-443c2010331f [Follower]" leader= -testLoggerAdapter verbose: server-6f025d68-e340-6b0e-235f-65d8c1b10663: 2024-09-03T21:34:37.041Z [INFO] server-6f025d68-e340-6b0e-235f-65d8c1b10663: entering follower state: follower="Node at 6f025d68-e340-6b0e-235f-65d8c1b10663 [Follower]" leader= - testing.go:149: server-6f025d68-e340-6b0e-235f-65d8c1b10663: 2024-09-03T21:34:37.041Z [INFO] server-6f025d68-e340-6b0e-235f-65d8c1b10663: entering follower state: follower="Node at 6f025d68-e340-6b0e-235f-65d8c1b10663 [Follower]" leader= -testLoggerAdapter verbose: server-365d0a07-ea1a-4df9-edf3-443c2010331f: 2024-09-03T21:34:37.091Z [WARN] server-365d0a07-ea1a-4df9-edf3-443c2010331f: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-365d0a07-ea1a-4df9-edf3-443c2010331f: 2024-09-03T21:34:37.091Z [WARN] server-365d0a07-ea1a-4df9-edf3-443c2010331f: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-365d0a07-ea1a-4df9-edf3-443c2010331f: 2024-09-03T21:34:37.091Z [INFO] server-365d0a07-ea1a-4df9-edf3-443c2010331f: entering candidate state: node="Node at 365d0a07-ea1a-4df9-edf3-443c2010331f [Candidate]" term=2 - testing.go:149: server-365d0a07-ea1a-4df9-edf3-443c2010331f: 2024-09-03T21:34:37.091Z [INFO] server-365d0a07-ea1a-4df9-edf3-443c2010331f: entering candidate state: node="Node at 365d0a07-ea1a-4df9-edf3-443c2010331f [Candidate]" term=2 -testLoggerAdapter verbose: server-365d0a07-ea1a-4df9-edf3-443c2010331f: 2024-09-03T21:34:37.091Z [INFO] server-365d0a07-ea1a-4df9-edf3-443c2010331f: election won: tally=2 - testing.go:149: server-365d0a07-ea1a-4df9-edf3-443c2010331f: 2024-09-03T21:34:37.091Z [INFO] server-365d0a07-ea1a-4df9-edf3-443c2010331f: election won: tally=2 -testLoggerAdapter verbose: server-365d0a07-ea1a-4df9-edf3-443c2010331f: 2024-09-03T21:34:37.091Z [INFO] server-365d0a07-ea1a-4df9-edf3-443c2010331f: entering leader state: leader="Node at 365d0a07-ea1a-4df9-edf3-443c2010331f [Leader]" - testing.go:149: server-365d0a07-ea1a-4df9-edf3-443c2010331f: 2024-09-03T21:34:37.091Z [INFO] server-365d0a07-ea1a-4df9-edf3-443c2010331f: entering leader state: leader="Node at 365d0a07-ea1a-4df9-edf3-443c2010331f [Leader]" -testLoggerAdapter verbose: server-365d0a07-ea1a-4df9-edf3-443c2010331f: 2024-09-03T21:34:37.091Z [INFO] server-365d0a07-ea1a-4df9-edf3-443c2010331f: added peer, starting replication: peer=server-34f94242-c236-7328-fe14-1c69eba03247 - testing.go:149: server-365d0a07-ea1a-4df9-edf3-443c2010331f: 2024-09-03T21:34:37.091Z [INFO] server-365d0a07-ea1a-4df9-edf3-443c2010331f: added peer, starting replication: peer=server-34f94242-c236-7328-fe14-1c69eba03247 -testLoggerAdapter verbose: server-365d0a07-ea1a-4df9-edf3-443c2010331f: 2024-09-03T21:34:37.091Z [INFO] server-365d0a07-ea1a-4df9-edf3-443c2010331f: added peer, starting replication: peer=server-6f025d68-e340-6b0e-235f-65d8c1b10663 - testing.go:149: server-365d0a07-ea1a-4df9-edf3-443c2010331f: 2024-09-03T21:34:37.091Z [INFO] server-365d0a07-ea1a-4df9-edf3-443c2010331f: added peer, starting replication: peer=server-6f025d68-e340-6b0e-235f-65d8c1b10663 -testLoggerAdapter verbose: server-365d0a07-ea1a-4df9-edf3-443c2010331f: 2024-09-03T21:34:37.092Z [INFO] server-365d0a07-ea1a-4df9-edf3-443c2010331f: pipelining replication: peer="{Voter server-6f025d68-e340-6b0e-235f-65d8c1b10663 6f025d68-e340-6b0e-235f-65d8c1b10663}" - testing.go:149: server-365d0a07-ea1a-4df9-edf3-443c2010331f: 2024-09-03T21:34:37.092Z [INFO] server-365d0a07-ea1a-4df9-edf3-443c2010331f: pipelining replication: peer="{Voter server-6f025d68-e340-6b0e-235f-65d8c1b10663 6f025d68-e340-6b0e-235f-65d8c1b10663}" -testLoggerAdapter verbose: server-365d0a07-ea1a-4df9-edf3-443c2010331f: 2024-09-03T21:34:37.092Z [INFO] server-365d0a07-ea1a-4df9-edf3-443c2010331f: pipelining replication: peer="{Voter server-34f94242-c236-7328-fe14-1c69eba03247 34f94242-c236-7328-fe14-1c69eba03247}" - testing.go:149: server-365d0a07-ea1a-4df9-edf3-443c2010331f: 2024-09-03T21:34:37.092Z [INFO] server-365d0a07-ea1a-4df9-edf3-443c2010331f: pipelining replication: peer="{Voter server-34f94242-c236-7328-fe14-1c69eba03247 34f94242-c236-7328-fe14-1c69eba03247}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:37.197Z [INFO] stable state for Leader reached at 2024-09-03 21:34:37.091909127 +0000 UTC m=+18.138566273 (1 nodes), 50.639583ms from start of poll, 51.04217ms from cluster start. Timeout at 2024-09-03 21:34:37.197004461 +0000 UTC m=+18.243661597, 105.095324ms after stability - testing.go:149: cluster: 2024-09-03T21:34:37.197Z [INFO] stable state for Leader reached at 2024-09-03 21:34:37.091909127 +0000 UTC m=+18.138566273 (1 nodes), 50.639583ms from start of poll, 51.04217ms from cluster start. Timeout at 2024-09-03 21:34:37.197004461 +0000 UTC m=+18.243661597, 105.095324ms after stability -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:37.197Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-09-03T21:34:37.197Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:37.302Z [INFO] stable state for Follower reached at 2024-09-03 21:34:37.1971607 +0000 UTC m=+18.243817834 (2 nodes), 1.453µs from start of poll, 156.293731ms from cluster start. Timeout at 2024-09-03 21:34:37.302260081 +0000 UTC m=+18.348917211, 105.099377ms after stability - testing.go:149: cluster: 2024-09-03T21:34:37.302Z [INFO] stable state for Follower reached at 2024-09-03 21:34:37.1971607 +0000 UTC m=+18.243817834 (2 nodes), 1.453µs from start of poll, 156.293731ms from cluster start. Timeout at 2024-09-03 21:34:37.302260081 +0000 UTC m=+18.348917211, 105.099377ms after stability -testLoggerAdapter verbose: server-34f94242-c236-7328-fe14-1c69eba03247: 2024-09-03T21:34:37.326Z [INFO] server-34f94242-c236-7328-fe14-1c69eba03247: entering candidate state: node="Node at 34f94242-c236-7328-fe14-1c69eba03247 [Candidate]" term=3 - testing.go:149: server-34f94242-c236-7328-fe14-1c69eba03247: 2024-09-03T21:34:37.326Z [INFO] server-34f94242-c236-7328-fe14-1c69eba03247: entering candidate state: node="Node at 34f94242-c236-7328-fe14-1c69eba03247 [Candidate]" term=3 -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:37.326Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-34f94242-c236-7328-fe14-1c69eba03247: 2024-09-03T21:34:37.326Z [INFO] server-34f94242-c236-7328-fe14-1c69eba03247: election won: tally=2 - testing.go:149: cluster: 2024-09-03T21:34:37.326Z [INFO] starting stability test: raft-state=Leader - testing.go:149: server-34f94242-c236-7328-fe14-1c69eba03247: 2024-09-03T21:34:37.326Z [INFO] server-34f94242-c236-7328-fe14-1c69eba03247: election won: tally=2 -testLoggerAdapter verbose: server-34f94242-c236-7328-fe14-1c69eba03247: 2024-09-03T21:34:37.326Z [INFO] server-34f94242-c236-7328-fe14-1c69eba03247: entering leader state: leader="Node at 34f94242-c236-7328-fe14-1c69eba03247 [Leader]" - testing.go:149: server-34f94242-c236-7328-fe14-1c69eba03247: 2024-09-03T21:34:37.326Z [INFO] server-34f94242-c236-7328-fe14-1c69eba03247: entering leader state: leader="Node at 34f94242-c236-7328-fe14-1c69eba03247 [Leader]" -testLoggerAdapter verbose: server-365d0a07-ea1a-4df9-edf3-443c2010331f: 2024-09-03T21:34:37.326Z [INFO] server-365d0a07-ea1a-4df9-edf3-443c2010331f: entering follower state: follower="Node at 365d0a07-ea1a-4df9-edf3-443c2010331f [Follower]" leader= -testLoggerAdapter verbose: server-34f94242-c236-7328-fe14-1c69eba03247: 2024-09-03T21:34:37.326Z [INFO] server-34f94242-c236-7328-fe14-1c69eba03247: added peer, starting replication: peer=server-365d0a07-ea1a-4df9-edf3-443c2010331f - testing.go:149: server-34f94242-c236-7328-fe14-1c69eba03247: 2024-09-03T21:34:37.326Z [INFO] server-34f94242-c236-7328-fe14-1c69eba03247: added peer, starting replication: peer=server-365d0a07-ea1a-4df9-edf3-443c2010331f -testLoggerAdapter verbose: server-34f94242-c236-7328-fe14-1c69eba03247: 2024-09-03T21:34:37.326Z [INFO] server-34f94242-c236-7328-fe14-1c69eba03247: added peer, starting replication: peer=server-6f025d68-e340-6b0e-235f-65d8c1b10663 - testing.go:149: server-34f94242-c236-7328-fe14-1c69eba03247: 2024-09-03T21:34:37.326Z [INFO] server-34f94242-c236-7328-fe14-1c69eba03247: added peer, starting replication: peer=server-6f025d68-e340-6b0e-235f-65d8c1b10663 -testLoggerAdapter verbose: server-34f94242-c236-7328-fe14-1c69eba03247: 2024-09-03T21:34:37.327Z [INFO] server-34f94242-c236-7328-fe14-1c69eba03247: pipelining replication: peer="{Voter server-6f025d68-e340-6b0e-235f-65d8c1b10663 6f025d68-e340-6b0e-235f-65d8c1b10663}" - testing.go:149: server-34f94242-c236-7328-fe14-1c69eba03247: 2024-09-03T21:34:37.327Z [INFO] server-34f94242-c236-7328-fe14-1c69eba03247: pipelining replication: peer="{Voter server-6f025d68-e340-6b0e-235f-65d8c1b10663 6f025d68-e340-6b0e-235f-65d8c1b10663}" - testing.go:149: server-365d0a07-ea1a-4df9-edf3-443c2010331f: 2024-09-03T21:34:37.326Z [INFO] server-365d0a07-ea1a-4df9-edf3-443c2010331f: entering follower state: follower="Node at 365d0a07-ea1a-4df9-edf3-443c2010331f [Follower]" leader= -testLoggerAdapter verbose: server-34f94242-c236-7328-fe14-1c69eba03247: 2024-09-03T21:34:37.327Z [INFO] server-34f94242-c236-7328-fe14-1c69eba03247: pipelining replication: peer="{Voter server-365d0a07-ea1a-4df9-edf3-443c2010331f 365d0a07-ea1a-4df9-edf3-443c2010331f}" - testing.go:149: server-34f94242-c236-7328-fe14-1c69eba03247: 2024-09-03T21:34:37.327Z [INFO] server-34f94242-c236-7328-fe14-1c69eba03247: pipelining replication: peer="{Voter server-365d0a07-ea1a-4df9-edf3-443c2010331f 365d0a07-ea1a-4df9-edf3-443c2010331f}" -testLoggerAdapter verbose: server-365d0a07-ea1a-4df9-edf3-443c2010331f: 2024-09-03T21:34:37.327Z [INFO] server-365d0a07-ea1a-4df9-edf3-443c2010331f: aborting pipeline replication: peer="{Voter server-34f94242-c236-7328-fe14-1c69eba03247 34f94242-c236-7328-fe14-1c69eba03247}" - testing.go:149: server-365d0a07-ea1a-4df9-edf3-443c2010331f: 2024-09-03T21:34:37.327Z [INFO] server-365d0a07-ea1a-4df9-edf3-443c2010331f: aborting pipeline replication: peer="{Voter server-34f94242-c236-7328-fe14-1c69eba03247 34f94242-c236-7328-fe14-1c69eba03247}" -testLoggerAdapter verbose: server-365d0a07-ea1a-4df9-edf3-443c2010331f: 2024-09-03T21:34:37.327Z [INFO] server-365d0a07-ea1a-4df9-edf3-443c2010331f: aborting pipeline replication: peer="{Voter server-6f025d68-e340-6b0e-235f-65d8c1b10663 6f025d68-e340-6b0e-235f-65d8c1b10663}" - testing.go:149: server-365d0a07-ea1a-4df9-edf3-443c2010331f: 2024-09-03T21:34:37.327Z [INFO] server-365d0a07-ea1a-4df9-edf3-443c2010331f: aborting pipeline replication: peer="{Voter server-6f025d68-e340-6b0e-235f-65d8c1b10663 6f025d68-e340-6b0e-235f-65d8c1b10663}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:37.432Z [INFO] stable state for Leader reached at 2024-09-03 21:34:37.327621255 +0000 UTC m=+18.374278411 (1 nodes), 1.350343ms from start of poll, 286.754308ms from cluster start. Timeout at 2024-09-03 21:34:37.432682581 +0000 UTC m=+18.479339715, 105.061304ms after stability - testing.go:149: cluster: 2024-09-03T21:34:37.432Z [INFO] stable state for Leader reached at 2024-09-03 21:34:37.327621255 +0000 UTC m=+18.374278411 (1 nodes), 1.350343ms from start of poll, 286.754308ms from cluster start. Timeout at 2024-09-03 21:34:37.432682581 +0000 UTC m=+18.479339715, 105.061304ms after stability -testLoggerAdapter verbose: server-34f94242-c236-7328-fe14-1c69eba03247: 2024-09-03T21:34:37.432Z [INFO] server-34f94242-c236-7328-fe14-1c69eba03247: aborting pipeline replication: peer="{Voter server-6f025d68-e340-6b0e-235f-65d8c1b10663 6f025d68-e340-6b0e-235f-65d8c1b10663}" - testing.go:149: server-34f94242-c236-7328-fe14-1c69eba03247: 2024-09-03T21:34:37.432Z [INFO] server-34f94242-c236-7328-fe14-1c69eba03247: aborting pipeline replication: peer="{Voter server-6f025d68-e340-6b0e-235f-65d8c1b10663 6f025d68-e340-6b0e-235f-65d8c1b10663}" -testLoggerAdapter verbose: server-34f94242-c236-7328-fe14-1c69eba03247: 2024-09-03T21:34:37.432Z [INFO] server-34f94242-c236-7328-fe14-1c69eba03247: aborting pipeline replication: peer="{Voter server-365d0a07-ea1a-4df9-edf3-443c2010331f 365d0a07-ea1a-4df9-edf3-443c2010331f}" - testing.go:149: server-34f94242-c236-7328-fe14-1c69eba03247: 2024-09-03T21:34:37.432Z [INFO] server-34f94242-c236-7328-fe14-1c69eba03247: aborting pipeline replication: peer="{Voter server-365d0a07-ea1a-4df9-edf3-443c2010331f 365d0a07-ea1a-4df9-edf3-443c2010331f}" ---- PASS: TestRaft_LeadershipTransferToBehindServer (0.39s) +testLoggerAdapter verbose: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.155Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-361c6eda-3ec7-9fbe-2143-3add7f61b589 Address:361c6eda-3ec7-9fbe-2143-3add7f61b589} {Suffrage:Voter ID:server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad Address:692b65e7-a6eb-02c9-8cd2-b051259fa6ad} {Suffrage:Voter ID:server-16957a74-3d2b-9d6f-b9e4-51212393e041 Address:16957a74-3d2b-9d6f-b9e4-51212393e041}]" + testing.go:149: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.155Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-361c6eda-3ec7-9fbe-2143-3add7f61b589 Address:361c6eda-3ec7-9fbe-2143-3add7f61b589} {Suffrage:Voter ID:server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad Address:692b65e7-a6eb-02c9-8cd2-b051259fa6ad} {Suffrage:Voter ID:server-16957a74-3d2b-9d6f-b9e4-51212393e041 Address:16957a74-3d2b-9d6f-b9e4-51212393e041}]" +testLoggerAdapter verbose: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.155Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-361c6eda-3ec7-9fbe-2143-3add7f61b589 Address:361c6eda-3ec7-9fbe-2143-3add7f61b589} {Suffrage:Voter ID:server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad Address:692b65e7-a6eb-02c9-8cd2-b051259fa6ad} {Suffrage:Voter ID:server-16957a74-3d2b-9d6f-b9e4-51212393e041 Address:16957a74-3d2b-9d6f-b9e4-51212393e041}]" + testing.go:149: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.155Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-361c6eda-3ec7-9fbe-2143-3add7f61b589 Address:361c6eda-3ec7-9fbe-2143-3add7f61b589} {Suffrage:Voter ID:server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad Address:692b65e7-a6eb-02c9-8cd2-b051259fa6ad} {Suffrage:Voter ID:server-16957a74-3d2b-9d6f-b9e4-51212393e041 Address:16957a74-3d2b-9d6f-b9e4-51212393e041}]" +testLoggerAdapter verbose: server-16957a74-3d2b-9d6f-b9e4-51212393e041: 2023-08-02T15:13:06.155Z [INFO] server-16957a74-3d2b-9d6f-b9e4-51212393e041: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-361c6eda-3ec7-9fbe-2143-3add7f61b589 Address:361c6eda-3ec7-9fbe-2143-3add7f61b589} {Suffrage:Voter ID:server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad Address:692b65e7-a6eb-02c9-8cd2-b051259fa6ad} {Suffrage:Voter ID:server-16957a74-3d2b-9d6f-b9e4-51212393e041 Address:16957a74-3d2b-9d6f-b9e4-51212393e041}]" + testing.go:149: server-16957a74-3d2b-9d6f-b9e4-51212393e041: 2023-08-02T15:13:06.155Z [INFO] server-16957a74-3d2b-9d6f-b9e4-51212393e041: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-361c6eda-3ec7-9fbe-2143-3add7f61b589 Address:361c6eda-3ec7-9fbe-2143-3add7f61b589} {Suffrage:Voter ID:server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad Address:692b65e7-a6eb-02c9-8cd2-b051259fa6ad} {Suffrage:Voter ID:server-16957a74-3d2b-9d6f-b9e4-51212393e041 Address:16957a74-3d2b-9d6f-b9e4-51212393e041}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:06.155Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:13:06.155Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.155Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: entering follower state: follower="Node at 361c6eda-3ec7-9fbe-2143-3add7f61b589 [Follower]" leader= + testing.go:149: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.155Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: entering follower state: follower="Node at 361c6eda-3ec7-9fbe-2143-3add7f61b589 [Follower]" leader= +testLoggerAdapter verbose: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.155Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: entering follower state: follower="Node at 692b65e7-a6eb-02c9-8cd2-b051259fa6ad [Follower]" leader= + testing.go:149: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.155Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: entering follower state: follower="Node at 692b65e7-a6eb-02c9-8cd2-b051259fa6ad [Follower]" leader= +testLoggerAdapter verbose: server-16957a74-3d2b-9d6f-b9e4-51212393e041: 2023-08-02T15:13:06.156Z [INFO] server-16957a74-3d2b-9d6f-b9e4-51212393e041: entering follower state: follower="Node at 16957a74-3d2b-9d6f-b9e4-51212393e041 [Follower]" leader= + testing.go:149: server-16957a74-3d2b-9d6f-b9e4-51212393e041: 2023-08-02T15:13:06.156Z [INFO] server-16957a74-3d2b-9d6f-b9e4-51212393e041: entering follower state: follower="Node at 16957a74-3d2b-9d6f-b9e4-51212393e041 [Follower]" leader= +testLoggerAdapter verbose: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.216Z [WARN] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.216Z [WARN] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.216Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: entering candidate state: node="Node at 361c6eda-3ec7-9fbe-2143-3add7f61b589 [Candidate]" term=2 + testing.go:149: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.216Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: entering candidate state: node="Node at 361c6eda-3ec7-9fbe-2143-3add7f61b589 [Candidate]" term=2 +testLoggerAdapter verbose: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.216Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: election won: tally=2 + testing.go:149: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.216Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: election won: tally=2 +testLoggerAdapter verbose: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.216Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: entering leader state: leader="Node at 361c6eda-3ec7-9fbe-2143-3add7f61b589 [Leader]" + testing.go:149: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.216Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: entering leader state: leader="Node at 361c6eda-3ec7-9fbe-2143-3add7f61b589 [Leader]" +testLoggerAdapter verbose: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.216Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: added peer, starting replication: peer=server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad + testing.go:149: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.216Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: added peer, starting replication: peer=server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad +testLoggerAdapter verbose: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.216Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: added peer, starting replication: peer=server-16957a74-3d2b-9d6f-b9e4-51212393e041 + testing.go:149: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.216Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: added peer, starting replication: peer=server-16957a74-3d2b-9d6f-b9e4-51212393e041 +testLoggerAdapter verbose: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.216Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: pipelining replication: peer="{Voter server-16957a74-3d2b-9d6f-b9e4-51212393e041 16957a74-3d2b-9d6f-b9e4-51212393e041}" + testing.go:149: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.216Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: pipelining replication: peer="{Voter server-16957a74-3d2b-9d6f-b9e4-51212393e041 16957a74-3d2b-9d6f-b9e4-51212393e041}" +testLoggerAdapter verbose: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.216Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: pipelining replication: peer="{Voter server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad 692b65e7-a6eb-02c9-8cd2-b051259fa6ad}" + testing.go:149: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.216Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: pipelining replication: peer="{Voter server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad 692b65e7-a6eb-02c9-8cd2-b051259fa6ad}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:06.321Z [INFO] stable state for Leader reached at 2023-08-02 15:13:06.216899543 +0000 UTC m=+18.312835877 (1 nodes), 60.997433ms from start of poll, 61.453724ms from cluster start. Timeout at 2023-08-02 15:13:06.32192285 +0000 UTC m=+18.417859176, 105.023299ms after stability + testing.go:149: cluster: 2023-08-02T15:13:06.321Z [INFO] stable state for Leader reached at 2023-08-02 15:13:06.216899543 +0000 UTC m=+18.312835877 (1 nodes), 60.997433ms from start of poll, 61.453724ms from cluster start. Timeout at 2023-08-02 15:13:06.32192285 +0000 UTC m=+18.417859176, 105.023299ms after stability +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:06.322Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-08-02T15:13:06.322Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:06.428Z [INFO] stable state for Follower reached at 2023-08-02 15:13:06.322102992 +0000 UTC m=+18.418039318 (2 nodes), 2.351µs from start of poll, 166.657165ms from cluster start. Timeout at 2023-08-02 15:13:06.428071596 +0000 UTC m=+18.524007939, 105.968621ms after stability + testing.go:149: cluster: 2023-08-02T15:13:06.428Z [INFO] stable state for Follower reached at 2023-08-02 15:13:06.322102992 +0000 UTC m=+18.418039318 (2 nodes), 2.351µs from start of poll, 166.657165ms from cluster start. Timeout at 2023-08-02 15:13:06.428071596 +0000 UTC m=+18.524007939, 105.968621ms after stability +testLoggerAdapter verbose: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.465Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: entering candidate state: node="Node at 692b65e7-a6eb-02c9-8cd2-b051259fa6ad [Candidate]" term=3 + testing.go:149: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.465Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: entering candidate state: node="Node at 692b65e7-a6eb-02c9-8cd2-b051259fa6ad [Candidate]" term=3 +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:06.465Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.465Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: election won: tally=2 + testing.go:149: cluster: 2023-08-02T15:13:06.465Z [INFO] starting stability test: raft-state=Leader + testing.go:149: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.465Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: election won: tally=2 +testLoggerAdapter verbose: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.465Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: entering follower state: follower="Node at 361c6eda-3ec7-9fbe-2143-3add7f61b589 [Follower]" leader= + testing.go:149: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.465Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: entering follower state: follower="Node at 361c6eda-3ec7-9fbe-2143-3add7f61b589 [Follower]" leader= +testLoggerAdapter verbose: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.465Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: entering leader state: leader="Node at 692b65e7-a6eb-02c9-8cd2-b051259fa6ad [Leader]" +testLoggerAdapter verbose: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.465Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: aborting pipeline replication: peer="{Voter server-16957a74-3d2b-9d6f-b9e4-51212393e041 16957a74-3d2b-9d6f-b9e4-51212393e041}" + testing.go:149: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.465Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: aborting pipeline replication: peer="{Voter server-16957a74-3d2b-9d6f-b9e4-51212393e041 16957a74-3d2b-9d6f-b9e4-51212393e041}" +testLoggerAdapter verbose: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.465Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: aborting pipeline replication: peer="{Voter server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad 692b65e7-a6eb-02c9-8cd2-b051259fa6ad}" + testing.go:149: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.465Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: entering leader state: leader="Node at 692b65e7-a6eb-02c9-8cd2-b051259fa6ad [Leader]" +testLoggerAdapter verbose: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.465Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: added peer, starting replication: peer=server-361c6eda-3ec7-9fbe-2143-3add7f61b589 + testing.go:149: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.465Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: added peer, starting replication: peer=server-361c6eda-3ec7-9fbe-2143-3add7f61b589 +testLoggerAdapter verbose: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.465Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: added peer, starting replication: peer=server-16957a74-3d2b-9d6f-b9e4-51212393e041 + testing.go:149: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.465Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: added peer, starting replication: peer=server-16957a74-3d2b-9d6f-b9e4-51212393e041 +testLoggerAdapter verbose: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.466Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: pipelining replication: peer="{Voter server-361c6eda-3ec7-9fbe-2143-3add7f61b589 361c6eda-3ec7-9fbe-2143-3add7f61b589}" + testing.go:149: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.465Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: aborting pipeline replication: peer="{Voter server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad 692b65e7-a6eb-02c9-8cd2-b051259fa6ad}" + testing.go:149: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.466Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: pipelining replication: peer="{Voter server-361c6eda-3ec7-9fbe-2143-3add7f61b589 361c6eda-3ec7-9fbe-2143-3add7f61b589}" +testLoggerAdapter verbose: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.466Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: pipelining replication: peer="{Voter server-16957a74-3d2b-9d6f-b9e4-51212393e041 16957a74-3d2b-9d6f-b9e4-51212393e041}" + testing.go:149: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.466Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: pipelining replication: peer="{Voter server-16957a74-3d2b-9d6f-b9e4-51212393e041 16957a74-3d2b-9d6f-b9e4-51212393e041}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:06.571Z [INFO] stable state for Leader reached at 2023-08-02 15:13:06.465711788 +0000 UTC m=+18.561648121 (1 nodes), 22.751µs from start of poll, 310.265968ms from cluster start. Timeout at 2023-08-02 15:13:06.571569975 +0000 UTC m=+18.667506377, 105.858256ms after stability + testing.go:149: cluster: 2023-08-02T15:13:06.571Z [INFO] stable state for Leader reached at 2023-08-02 15:13:06.465711788 +0000 UTC m=+18.561648121 (1 nodes), 22.751µs from start of poll, 310.265968ms from cluster start. Timeout at 2023-08-02 15:13:06.571569975 +0000 UTC m=+18.667506377, 105.858256ms after stability +testLoggerAdapter verbose: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.571Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: aborting pipeline replication: peer="{Voter server-16957a74-3d2b-9d6f-b9e4-51212393e041 16957a74-3d2b-9d6f-b9e4-51212393e041}" + testing.go:149: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.571Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: aborting pipeline replication: peer="{Voter server-16957a74-3d2b-9d6f-b9e4-51212393e041 16957a74-3d2b-9d6f-b9e4-51212393e041}" +testLoggerAdapter verbose: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.571Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: aborting pipeline replication: peer="{Voter server-361c6eda-3ec7-9fbe-2143-3add7f61b589 361c6eda-3ec7-9fbe-2143-3add7f61b589}" + testing.go:149: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.571Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: aborting pipeline replication: peer="{Voter server-361c6eda-3ec7-9fbe-2143-3add7f61b589 361c6eda-3ec7-9fbe-2143-3add7f61b589}" +--- PASS: TestRaft_LeadershipTransferToBehindServer (0.42s) === RUN TestRaft_LeadershipTransferToItself -testLoggerAdapter verbose: server-d6343d6b-5c43-56e0-03fa-46254d0ae20a: 2024-09-03T21:34:37.433Z [INFO] server-d6343d6b-5c43-56e0-03fa-46254d0ae20a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d6343d6b-5c43-56e0-03fa-46254d0ae20a Address:d6343d6b-5c43-56e0-03fa-46254d0ae20a} {Suffrage:Voter ID:server-111cbf3a-75ab-ce30-7b6c-84e8b82fc8f5 Address:111cbf3a-75ab-ce30-7b6c-84e8b82fc8f5} {Suffrage:Voter ID:server-e89cc6c4-3105-eaa7-326a-865845b720e4 Address:e89cc6c4-3105-eaa7-326a-865845b720e4}]" - testing.go:149: server-d6343d6b-5c43-56e0-03fa-46254d0ae20a: 2024-09-03T21:34:37.433Z [INFO] server-d6343d6b-5c43-56e0-03fa-46254d0ae20a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d6343d6b-5c43-56e0-03fa-46254d0ae20a Address:d6343d6b-5c43-56e0-03fa-46254d0ae20a} {Suffrage:Voter ID:server-111cbf3a-75ab-ce30-7b6c-84e8b82fc8f5 Address:111cbf3a-75ab-ce30-7b6c-84e8b82fc8f5} {Suffrage:Voter ID:server-e89cc6c4-3105-eaa7-326a-865845b720e4 Address:e89cc6c4-3105-eaa7-326a-865845b720e4}]" -testLoggerAdapter verbose: server-d6343d6b-5c43-56e0-03fa-46254d0ae20a: 2024-09-03T21:34:37.434Z [INFO] server-d6343d6b-5c43-56e0-03fa-46254d0ae20a: entering follower state: follower="Node at d6343d6b-5c43-56e0-03fa-46254d0ae20a [Follower]" leader= - testing.go:149: server-d6343d6b-5c43-56e0-03fa-46254d0ae20a: 2024-09-03T21:34:37.434Z [INFO] server-d6343d6b-5c43-56e0-03fa-46254d0ae20a: entering follower state: follower="Node at d6343d6b-5c43-56e0-03fa-46254d0ae20a [Follower]" leader= -testLoggerAdapter verbose: server-111cbf3a-75ab-ce30-7b6c-84e8b82fc8f5: 2024-09-03T21:34:37.434Z [INFO] server-111cbf3a-75ab-ce30-7b6c-84e8b82fc8f5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d6343d6b-5c43-56e0-03fa-46254d0ae20a Address:d6343d6b-5c43-56e0-03fa-46254d0ae20a} {Suffrage:Voter ID:server-111cbf3a-75ab-ce30-7b6c-84e8b82fc8f5 Address:111cbf3a-75ab-ce30-7b6c-84e8b82fc8f5} {Suffrage:Voter ID:server-e89cc6c4-3105-eaa7-326a-865845b720e4 Address:e89cc6c4-3105-eaa7-326a-865845b720e4}]" - testing.go:149: server-111cbf3a-75ab-ce30-7b6c-84e8b82fc8f5: 2024-09-03T21:34:37.434Z [INFO] server-111cbf3a-75ab-ce30-7b6c-84e8b82fc8f5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d6343d6b-5c43-56e0-03fa-46254d0ae20a Address:d6343d6b-5c43-56e0-03fa-46254d0ae20a} {Suffrage:Voter ID:server-111cbf3a-75ab-ce30-7b6c-84e8b82fc8f5 Address:111cbf3a-75ab-ce30-7b6c-84e8b82fc8f5} {Suffrage:Voter ID:server-e89cc6c4-3105-eaa7-326a-865845b720e4 Address:e89cc6c4-3105-eaa7-326a-865845b720e4}]" -testLoggerAdapter verbose: server-111cbf3a-75ab-ce30-7b6c-84e8b82fc8f5: 2024-09-03T21:34:37.434Z [INFO] server-111cbf3a-75ab-ce30-7b6c-84e8b82fc8f5: entering follower state: follower="Node at 111cbf3a-75ab-ce30-7b6c-84e8b82fc8f5 [Follower]" leader= - testing.go:149: server-111cbf3a-75ab-ce30-7b6c-84e8b82fc8f5: 2024-09-03T21:34:37.434Z [INFO] server-111cbf3a-75ab-ce30-7b6c-84e8b82fc8f5: entering follower state: follower="Node at 111cbf3a-75ab-ce30-7b6c-84e8b82fc8f5 [Follower]" leader= -testLoggerAdapter verbose: server-e89cc6c4-3105-eaa7-326a-865845b720e4: 2024-09-03T21:34:37.434Z [INFO] server-e89cc6c4-3105-eaa7-326a-865845b720e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d6343d6b-5c43-56e0-03fa-46254d0ae20a Address:d6343d6b-5c43-56e0-03fa-46254d0ae20a} {Suffrage:Voter ID:server-111cbf3a-75ab-ce30-7b6c-84e8b82fc8f5 Address:111cbf3a-75ab-ce30-7b6c-84e8b82fc8f5} {Suffrage:Voter ID:server-e89cc6c4-3105-eaa7-326a-865845b720e4 Address:e89cc6c4-3105-eaa7-326a-865845b720e4}]" - testing.go:149: server-e89cc6c4-3105-eaa7-326a-865845b720e4: 2024-09-03T21:34:37.434Z [INFO] server-e89cc6c4-3105-eaa7-326a-865845b720e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d6343d6b-5c43-56e0-03fa-46254d0ae20a Address:d6343d6b-5c43-56e0-03fa-46254d0ae20a} {Suffrage:Voter ID:server-111cbf3a-75ab-ce30-7b6c-84e8b82fc8f5 Address:111cbf3a-75ab-ce30-7b6c-84e8b82fc8f5} {Suffrage:Voter ID:server-e89cc6c4-3105-eaa7-326a-865845b720e4 Address:e89cc6c4-3105-eaa7-326a-865845b720e4}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:37.434Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:37.434Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-e89cc6c4-3105-eaa7-326a-865845b720e4: 2024-09-03T21:34:37.434Z [INFO] server-e89cc6c4-3105-eaa7-326a-865845b720e4: entering follower state: follower="Node at e89cc6c4-3105-eaa7-326a-865845b720e4 [Follower]" leader= - testing.go:149: server-e89cc6c4-3105-eaa7-326a-865845b720e4: 2024-09-03T21:34:37.434Z [INFO] server-e89cc6c4-3105-eaa7-326a-865845b720e4: entering follower state: follower="Node at e89cc6c4-3105-eaa7-326a-865845b720e4 [Follower]" leader= -testLoggerAdapter verbose: server-e89cc6c4-3105-eaa7-326a-865845b720e4: 2024-09-03T21:34:37.500Z [WARN] server-e89cc6c4-3105-eaa7-326a-865845b720e4: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-e89cc6c4-3105-eaa7-326a-865845b720e4: 2024-09-03T21:34:37.500Z [WARN] server-e89cc6c4-3105-eaa7-326a-865845b720e4: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-e89cc6c4-3105-eaa7-326a-865845b720e4: 2024-09-03T21:34:37.500Z [INFO] server-e89cc6c4-3105-eaa7-326a-865845b720e4: entering candidate state: node="Node at e89cc6c4-3105-eaa7-326a-865845b720e4 [Candidate]" term=2 - testing.go:149: server-e89cc6c4-3105-eaa7-326a-865845b720e4: 2024-09-03T21:34:37.500Z [INFO] server-e89cc6c4-3105-eaa7-326a-865845b720e4: entering candidate state: node="Node at e89cc6c4-3105-eaa7-326a-865845b720e4 [Candidate]" term=2 -testLoggerAdapter verbose: server-e89cc6c4-3105-eaa7-326a-865845b720e4: 2024-09-03T21:34:37.500Z [INFO] server-e89cc6c4-3105-eaa7-326a-865845b720e4: election won: tally=2 - testing.go:149: server-e89cc6c4-3105-eaa7-326a-865845b720e4: 2024-09-03T21:34:37.500Z [INFO] server-e89cc6c4-3105-eaa7-326a-865845b720e4: election won: tally=2 -testLoggerAdapter verbose: server-e89cc6c4-3105-eaa7-326a-865845b720e4: 2024-09-03T21:34:37.500Z [INFO] server-e89cc6c4-3105-eaa7-326a-865845b720e4: entering leader state: leader="Node at e89cc6c4-3105-eaa7-326a-865845b720e4 [Leader]" - testing.go:149: server-e89cc6c4-3105-eaa7-326a-865845b720e4: 2024-09-03T21:34:37.500Z [INFO] server-e89cc6c4-3105-eaa7-326a-865845b720e4: entering leader state: leader="Node at e89cc6c4-3105-eaa7-326a-865845b720e4 [Leader]" -testLoggerAdapter verbose: server-e89cc6c4-3105-eaa7-326a-865845b720e4: 2024-09-03T21:34:37.500Z [INFO] server-e89cc6c4-3105-eaa7-326a-865845b720e4: added peer, starting replication: peer=server-d6343d6b-5c43-56e0-03fa-46254d0ae20a - testing.go:149: server-e89cc6c4-3105-eaa7-326a-865845b720e4: 2024-09-03T21:34:37.500Z [INFO] server-e89cc6c4-3105-eaa7-326a-865845b720e4: added peer, starting replication: peer=server-d6343d6b-5c43-56e0-03fa-46254d0ae20a -testLoggerAdapter verbose: server-e89cc6c4-3105-eaa7-326a-865845b720e4: 2024-09-03T21:34:37.500Z [INFO] server-e89cc6c4-3105-eaa7-326a-865845b720e4: added peer, starting replication: peer=server-111cbf3a-75ab-ce30-7b6c-84e8b82fc8f5 - testing.go:149: server-e89cc6c4-3105-eaa7-326a-865845b720e4: 2024-09-03T21:34:37.500Z [INFO] server-e89cc6c4-3105-eaa7-326a-865845b720e4: added peer, starting replication: peer=server-111cbf3a-75ab-ce30-7b6c-84e8b82fc8f5 -testLoggerAdapter verbose: server-e89cc6c4-3105-eaa7-326a-865845b720e4: 2024-09-03T21:34:37.501Z [INFO] server-e89cc6c4-3105-eaa7-326a-865845b720e4: pipelining replication: peer="{Voter server-d6343d6b-5c43-56e0-03fa-46254d0ae20a d6343d6b-5c43-56e0-03fa-46254d0ae20a}" - testing.go:149: server-e89cc6c4-3105-eaa7-326a-865845b720e4: 2024-09-03T21:34:37.501Z [INFO] server-e89cc6c4-3105-eaa7-326a-865845b720e4: pipelining replication: peer="{Voter server-d6343d6b-5c43-56e0-03fa-46254d0ae20a d6343d6b-5c43-56e0-03fa-46254d0ae20a}" -testLoggerAdapter verbose: server-e89cc6c4-3105-eaa7-326a-865845b720e4: 2024-09-03T21:34:37.501Z [INFO] server-e89cc6c4-3105-eaa7-326a-865845b720e4: pipelining replication: peer="{Voter server-111cbf3a-75ab-ce30-7b6c-84e8b82fc8f5 111cbf3a-75ab-ce30-7b6c-84e8b82fc8f5}" - testing.go:149: server-e89cc6c4-3105-eaa7-326a-865845b720e4: 2024-09-03T21:34:37.501Z [INFO] server-e89cc6c4-3105-eaa7-326a-865845b720e4: pipelining replication: peer="{Voter server-111cbf3a-75ab-ce30-7b6c-84e8b82fc8f5 111cbf3a-75ab-ce30-7b6c-84e8b82fc8f5}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:37.606Z [INFO] stable state for Leader reached at 2024-09-03 21:34:37.500965632 +0000 UTC m=+18.547622768 (1 nodes), 66.653782ms from start of poll, 67.204834ms from cluster start. Timeout at 2024-09-03 21:34:37.606041576 +0000 UTC m=+18.652698707, 105.075939ms after stability - testing.go:149: cluster: 2024-09-03T21:34:37.606Z [INFO] stable state for Leader reached at 2024-09-03 21:34:37.500965632 +0000 UTC m=+18.547622768 (1 nodes), 66.653782ms from start of poll, 67.204834ms from cluster start. Timeout at 2024-09-03 21:34:37.606041576 +0000 UTC m=+18.652698707, 105.075939ms after stability -testLoggerAdapter verbose: server-e89cc6c4-3105-eaa7-326a-865845b720e4: 2024-09-03T21:34:37.606Z [INFO] server-e89cc6c4-3105-eaa7-326a-865845b720e4: cannot transfer leadership to itself - testing.go:149: server-e89cc6c4-3105-eaa7-326a-865845b720e4: 2024-09-03T21:34:37.606Z [INFO] server-e89cc6c4-3105-eaa7-326a-865845b720e4: cannot transfer leadership to itself -testLoggerAdapter verbose: server-e89cc6c4-3105-eaa7-326a-865845b720e4: 2024-09-03T21:34:37.606Z [INFO] server-e89cc6c4-3105-eaa7-326a-865845b720e4: aborting pipeline replication: peer="{Voter server-111cbf3a-75ab-ce30-7b6c-84e8b82fc8f5 111cbf3a-75ab-ce30-7b6c-84e8b82fc8f5}" - testing.go:149: server-e89cc6c4-3105-eaa7-326a-865845b720e4: 2024-09-03T21:34:37.606Z [INFO] server-e89cc6c4-3105-eaa7-326a-865845b720e4: aborting pipeline replication: peer="{Voter server-111cbf3a-75ab-ce30-7b6c-84e8b82fc8f5 111cbf3a-75ab-ce30-7b6c-84e8b82fc8f5}" -testLoggerAdapter verbose: server-e89cc6c4-3105-eaa7-326a-865845b720e4: 2024-09-03T21:34:37.606Z [INFO] server-e89cc6c4-3105-eaa7-326a-865845b720e4: aborting pipeline replication: peer="{Voter server-d6343d6b-5c43-56e0-03fa-46254d0ae20a d6343d6b-5c43-56e0-03fa-46254d0ae20a}" - testing.go:149: server-e89cc6c4-3105-eaa7-326a-865845b720e4: 2024-09-03T21:34:37.606Z [INFO] server-e89cc6c4-3105-eaa7-326a-865845b720e4: aborting pipeline replication: peer="{Voter server-d6343d6b-5c43-56e0-03fa-46254d0ae20a d6343d6b-5c43-56e0-03fa-46254d0ae20a}" +testLoggerAdapter verbose: server-0137d5fb-72ab-e229-35fa-8443f56e385d: 2023-08-02T15:13:06.572Z [INFO] server-0137d5fb-72ab-e229-35fa-8443f56e385d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0137d5fb-72ab-e229-35fa-8443f56e385d Address:0137d5fb-72ab-e229-35fa-8443f56e385d} {Suffrage:Voter ID:server-cb6b69c4-336f-faaa-53d5-815fb5c02720 Address:cb6b69c4-336f-faaa-53d5-815fb5c02720} {Suffrage:Voter ID:server-40bdc3a2-eba6-6266-135a-e92d125923bc Address:40bdc3a2-eba6-6266-135a-e92d125923bc}]" + testing.go:149: server-0137d5fb-72ab-e229-35fa-8443f56e385d: 2023-08-02T15:13:06.572Z [INFO] server-0137d5fb-72ab-e229-35fa-8443f56e385d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0137d5fb-72ab-e229-35fa-8443f56e385d Address:0137d5fb-72ab-e229-35fa-8443f56e385d} {Suffrage:Voter ID:server-cb6b69c4-336f-faaa-53d5-815fb5c02720 Address:cb6b69c4-336f-faaa-53d5-815fb5c02720} {Suffrage:Voter ID:server-40bdc3a2-eba6-6266-135a-e92d125923bc Address:40bdc3a2-eba6-6266-135a-e92d125923bc}]" +testLoggerAdapter verbose: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.573Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0137d5fb-72ab-e229-35fa-8443f56e385d Address:0137d5fb-72ab-e229-35fa-8443f56e385d} {Suffrage:Voter ID:server-cb6b69c4-336f-faaa-53d5-815fb5c02720 Address:cb6b69c4-336f-faaa-53d5-815fb5c02720} {Suffrage:Voter ID:server-40bdc3a2-eba6-6266-135a-e92d125923bc Address:40bdc3a2-eba6-6266-135a-e92d125923bc}]" +testLoggerAdapter verbose: server-0137d5fb-72ab-e229-35fa-8443f56e385d: 2023-08-02T15:13:06.573Z [INFO] server-0137d5fb-72ab-e229-35fa-8443f56e385d: entering follower state: follower="Node at 0137d5fb-72ab-e229-35fa-8443f56e385d [Follower]" leader= + testing.go:149: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.573Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0137d5fb-72ab-e229-35fa-8443f56e385d Address:0137d5fb-72ab-e229-35fa-8443f56e385d} {Suffrage:Voter ID:server-cb6b69c4-336f-faaa-53d5-815fb5c02720 Address:cb6b69c4-336f-faaa-53d5-815fb5c02720} {Suffrage:Voter ID:server-40bdc3a2-eba6-6266-135a-e92d125923bc Address:40bdc3a2-eba6-6266-135a-e92d125923bc}]" + testing.go:149: server-0137d5fb-72ab-e229-35fa-8443f56e385d: 2023-08-02T15:13:06.573Z [INFO] server-0137d5fb-72ab-e229-35fa-8443f56e385d: entering follower state: follower="Node at 0137d5fb-72ab-e229-35fa-8443f56e385d [Follower]" leader= +testLoggerAdapter verbose: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.573Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: entering follower state: follower="Node at cb6b69c4-336f-faaa-53d5-815fb5c02720 [Follower]" leader= + testing.go:149: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.573Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: entering follower state: follower="Node at cb6b69c4-336f-faaa-53d5-815fb5c02720 [Follower]" leader= +testLoggerAdapter verbose: server-40bdc3a2-eba6-6266-135a-e92d125923bc: 2023-08-02T15:13:06.573Z [INFO] server-40bdc3a2-eba6-6266-135a-e92d125923bc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0137d5fb-72ab-e229-35fa-8443f56e385d Address:0137d5fb-72ab-e229-35fa-8443f56e385d} {Suffrage:Voter ID:server-cb6b69c4-336f-faaa-53d5-815fb5c02720 Address:cb6b69c4-336f-faaa-53d5-815fb5c02720} {Suffrage:Voter ID:server-40bdc3a2-eba6-6266-135a-e92d125923bc Address:40bdc3a2-eba6-6266-135a-e92d125923bc}]" + testing.go:149: server-40bdc3a2-eba6-6266-135a-e92d125923bc: 2023-08-02T15:13:06.573Z [INFO] server-40bdc3a2-eba6-6266-135a-e92d125923bc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0137d5fb-72ab-e229-35fa-8443f56e385d Address:0137d5fb-72ab-e229-35fa-8443f56e385d} {Suffrage:Voter ID:server-cb6b69c4-336f-faaa-53d5-815fb5c02720 Address:cb6b69c4-336f-faaa-53d5-815fb5c02720} {Suffrage:Voter ID:server-40bdc3a2-eba6-6266-135a-e92d125923bc Address:40bdc3a2-eba6-6266-135a-e92d125923bc}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:06.573Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-40bdc3a2-eba6-6266-135a-e92d125923bc: 2023-08-02T15:13:06.573Z [INFO] server-40bdc3a2-eba6-6266-135a-e92d125923bc: entering follower state: follower="Node at 40bdc3a2-eba6-6266-135a-e92d125923bc [Follower]" leader= + testing.go:149: cluster: 2023-08-02T15:13:06.573Z [INFO] starting stability test: raft-state=Leader + testing.go:149: server-40bdc3a2-eba6-6266-135a-e92d125923bc: 2023-08-02T15:13:06.573Z [INFO] server-40bdc3a2-eba6-6266-135a-e92d125923bc: entering follower state: follower="Node at 40bdc3a2-eba6-6266-135a-e92d125923bc [Follower]" leader= +testLoggerAdapter verbose: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.634Z [WARN] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.634Z [WARN] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.634Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: entering candidate state: node="Node at cb6b69c4-336f-faaa-53d5-815fb5c02720 [Candidate]" term=2 + testing.go:149: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.634Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: entering candidate state: node="Node at cb6b69c4-336f-faaa-53d5-815fb5c02720 [Candidate]" term=2 +testLoggerAdapter verbose: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.634Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: election won: tally=2 + testing.go:149: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.634Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: election won: tally=2 +testLoggerAdapter verbose: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.634Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: entering leader state: leader="Node at cb6b69c4-336f-faaa-53d5-815fb5c02720 [Leader]" + testing.go:149: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.634Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: entering leader state: leader="Node at cb6b69c4-336f-faaa-53d5-815fb5c02720 [Leader]" +testLoggerAdapter verbose: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.634Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: added peer, starting replication: peer=server-0137d5fb-72ab-e229-35fa-8443f56e385d + testing.go:149: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.634Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: added peer, starting replication: peer=server-0137d5fb-72ab-e229-35fa-8443f56e385d +testLoggerAdapter verbose: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.634Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: added peer, starting replication: peer=server-40bdc3a2-eba6-6266-135a-e92d125923bc + testing.go:149: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.634Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: added peer, starting replication: peer=server-40bdc3a2-eba6-6266-135a-e92d125923bc +testLoggerAdapter verbose: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.634Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: pipelining replication: peer="{Voter server-40bdc3a2-eba6-6266-135a-e92d125923bc 40bdc3a2-eba6-6266-135a-e92d125923bc}" + testing.go:149: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.634Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: pipelining replication: peer="{Voter server-40bdc3a2-eba6-6266-135a-e92d125923bc 40bdc3a2-eba6-6266-135a-e92d125923bc}" +testLoggerAdapter verbose: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.634Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: pipelining replication: peer="{Voter server-0137d5fb-72ab-e229-35fa-8443f56e385d 0137d5fb-72ab-e229-35fa-8443f56e385d}" + testing.go:149: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.634Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: pipelining replication: peer="{Voter server-0137d5fb-72ab-e229-35fa-8443f56e385d 0137d5fb-72ab-e229-35fa-8443f56e385d}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:06.739Z [INFO] stable state for Leader reached at 2023-08-02 15:13:06.634561329 +0000 UTC m=+18.730497671 (1 nodes), 61.129914ms from start of poll, 61.754533ms from cluster start. Timeout at 2023-08-02 15:13:06.739569554 +0000 UTC m=+18.835505895, 105.008224ms after stability + testing.go:149: cluster: 2023-08-02T15:13:06.739Z [INFO] stable state for Leader reached at 2023-08-02 15:13:06.634561329 +0000 UTC m=+18.730497671 (1 nodes), 61.129914ms from start of poll, 61.754533ms from cluster start. Timeout at 2023-08-02 15:13:06.739569554 +0000 UTC m=+18.835505895, 105.008224ms after stability +testLoggerAdapter verbose: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.739Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: cannot transfer leadership to itself + testing.go:149: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.739Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: cannot transfer leadership to itself +testLoggerAdapter verbose: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.739Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: aborting pipeline replication: peer="{Voter server-40bdc3a2-eba6-6266-135a-e92d125923bc 40bdc3a2-eba6-6266-135a-e92d125923bc}" + testing.go:149: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.739Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: aborting pipeline replication: peer="{Voter server-40bdc3a2-eba6-6266-135a-e92d125923bc 40bdc3a2-eba6-6266-135a-e92d125923bc}" +testLoggerAdapter verbose: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.739Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: aborting pipeline replication: peer="{Voter server-0137d5fb-72ab-e229-35fa-8443f56e385d 0137d5fb-72ab-e229-35fa-8443f56e385d}" + testing.go:149: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.739Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: aborting pipeline replication: peer="{Voter server-0137d5fb-72ab-e229-35fa-8443f56e385d 0137d5fb-72ab-e229-35fa-8443f56e385d}" --- PASS: TestRaft_LeadershipTransferToItself (0.17s) === RUN TestRaft_LeadershipTransferLeaderRejectsClientRequests raft_test.go:2064: Disabled unreliable test --- SKIP: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.00s) === RUN TestRaft_LeadershipTransferLeaderReplicationTimeout -testLoggerAdapter verbose: server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: 2024-09-03T21:34:37.607Z [INFO] server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9008c2f0-d0ba-9c75-5721-eb1473aaf796 Address:9008c2f0-d0ba-9c75-5721-eb1473aaf796} {Suffrage:Voter ID:server-40a77dd1-a78c-6a61-d236-b6e7bb6fa3db Address:40a77dd1-a78c-6a61-d236-b6e7bb6fa3db} {Suffrage:Voter ID:server-2461d6a1-feda-723f-eb74-6ec87145f6c9 Address:2461d6a1-feda-723f-eb74-6ec87145f6c9}]" - testing.go:149: server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: 2024-09-03T21:34:37.607Z [INFO] server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9008c2f0-d0ba-9c75-5721-eb1473aaf796 Address:9008c2f0-d0ba-9c75-5721-eb1473aaf796} {Suffrage:Voter ID:server-40a77dd1-a78c-6a61-d236-b6e7bb6fa3db Address:40a77dd1-a78c-6a61-d236-b6e7bb6fa3db} {Suffrage:Voter ID:server-2461d6a1-feda-723f-eb74-6ec87145f6c9 Address:2461d6a1-feda-723f-eb74-6ec87145f6c9}]" -testLoggerAdapter verbose: server-40a77dd1-a78c-6a61-d236-b6e7bb6fa3db: 2024-09-03T21:34:37.607Z [INFO] server-40a77dd1-a78c-6a61-d236-b6e7bb6fa3db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9008c2f0-d0ba-9c75-5721-eb1473aaf796 Address:9008c2f0-d0ba-9c75-5721-eb1473aaf796} {Suffrage:Voter ID:server-40a77dd1-a78c-6a61-d236-b6e7bb6fa3db Address:40a77dd1-a78c-6a61-d236-b6e7bb6fa3db} {Suffrage:Voter ID:server-2461d6a1-feda-723f-eb74-6ec87145f6c9 Address:2461d6a1-feda-723f-eb74-6ec87145f6c9}]" - testing.go:149: server-40a77dd1-a78c-6a61-d236-b6e7bb6fa3db: 2024-09-03T21:34:37.607Z [INFO] server-40a77dd1-a78c-6a61-d236-b6e7bb6fa3db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9008c2f0-d0ba-9c75-5721-eb1473aaf796 Address:9008c2f0-d0ba-9c75-5721-eb1473aaf796} {Suffrage:Voter ID:server-40a77dd1-a78c-6a61-d236-b6e7bb6fa3db Address:40a77dd1-a78c-6a61-d236-b6e7bb6fa3db} {Suffrage:Voter ID:server-2461d6a1-feda-723f-eb74-6ec87145f6c9 Address:2461d6a1-feda-723f-eb74-6ec87145f6c9}]" -testLoggerAdapter verbose: server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: 2024-09-03T21:34:37.607Z [INFO] server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: entering follower state: follower="Node at 9008c2f0-d0ba-9c75-5721-eb1473aaf796 [Follower]" leader= - testing.go:149: server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: 2024-09-03T21:34:37.607Z [INFO] server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: entering follower state: follower="Node at 9008c2f0-d0ba-9c75-5721-eb1473aaf796 [Follower]" leader= -testLoggerAdapter verbose: server-40a77dd1-a78c-6a61-d236-b6e7bb6fa3db: 2024-09-03T21:34:37.607Z [INFO] server-40a77dd1-a78c-6a61-d236-b6e7bb6fa3db: entering follower state: follower="Node at 40a77dd1-a78c-6a61-d236-b6e7bb6fa3db [Follower]" leader= -testLoggerAdapter verbose: server-2461d6a1-feda-723f-eb74-6ec87145f6c9: 2024-09-03T21:34:37.607Z [INFO] server-2461d6a1-feda-723f-eb74-6ec87145f6c9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9008c2f0-d0ba-9c75-5721-eb1473aaf796 Address:9008c2f0-d0ba-9c75-5721-eb1473aaf796} {Suffrage:Voter ID:server-40a77dd1-a78c-6a61-d236-b6e7bb6fa3db Address:40a77dd1-a78c-6a61-d236-b6e7bb6fa3db} {Suffrage:Voter ID:server-2461d6a1-feda-723f-eb74-6ec87145f6c9 Address:2461d6a1-feda-723f-eb74-6ec87145f6c9}]" - testing.go:149: server-40a77dd1-a78c-6a61-d236-b6e7bb6fa3db: 2024-09-03T21:34:37.607Z [INFO] server-40a77dd1-a78c-6a61-d236-b6e7bb6fa3db: entering follower state: follower="Node at 40a77dd1-a78c-6a61-d236-b6e7bb6fa3db [Follower]" leader= - testing.go:149: server-2461d6a1-feda-723f-eb74-6ec87145f6c9: 2024-09-03T21:34:37.607Z [INFO] server-2461d6a1-feda-723f-eb74-6ec87145f6c9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9008c2f0-d0ba-9c75-5721-eb1473aaf796 Address:9008c2f0-d0ba-9c75-5721-eb1473aaf796} {Suffrage:Voter ID:server-40a77dd1-a78c-6a61-d236-b6e7bb6fa3db Address:40a77dd1-a78c-6a61-d236-b6e7bb6fa3db} {Suffrage:Voter ID:server-2461d6a1-feda-723f-eb74-6ec87145f6c9 Address:2461d6a1-feda-723f-eb74-6ec87145f6c9}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:37.607Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:37.607Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-2461d6a1-feda-723f-eb74-6ec87145f6c9: 2024-09-03T21:34:37.607Z [INFO] server-2461d6a1-feda-723f-eb74-6ec87145f6c9: entering follower state: follower="Node at 2461d6a1-feda-723f-eb74-6ec87145f6c9 [Follower]" leader= - testing.go:149: server-2461d6a1-feda-723f-eb74-6ec87145f6c9: 2024-09-03T21:34:37.607Z [INFO] server-2461d6a1-feda-723f-eb74-6ec87145f6c9: entering follower state: follower="Node at 2461d6a1-feda-723f-eb74-6ec87145f6c9 [Follower]" leader= -testLoggerAdapter verbose: server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: 2024-09-03T21:34:37.663Z [WARN] server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: 2024-09-03T21:34:37.663Z [WARN] server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: 2024-09-03T21:34:37.663Z [INFO] server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: entering candidate state: node="Node at 9008c2f0-d0ba-9c75-5721-eb1473aaf796 [Candidate]" term=2 - testing.go:149: server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: 2024-09-03T21:34:37.663Z [INFO] server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: entering candidate state: node="Node at 9008c2f0-d0ba-9c75-5721-eb1473aaf796 [Candidate]" term=2 -testLoggerAdapter verbose: server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: 2024-09-03T21:34:37.664Z [INFO] server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: election won: tally=2 - testing.go:149: server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: 2024-09-03T21:34:37.664Z [INFO] server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: election won: tally=2 -testLoggerAdapter verbose: server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: 2024-09-03T21:34:37.664Z [INFO] server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: entering leader state: leader="Node at 9008c2f0-d0ba-9c75-5721-eb1473aaf796 [Leader]" - testing.go:149: server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: 2024-09-03T21:34:37.664Z [INFO] server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: entering leader state: leader="Node at 9008c2f0-d0ba-9c75-5721-eb1473aaf796 [Leader]" -testLoggerAdapter verbose: server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: 2024-09-03T21:34:37.664Z [INFO] server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: added peer, starting replication: peer=server-40a77dd1-a78c-6a61-d236-b6e7bb6fa3db - testing.go:149: server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: 2024-09-03T21:34:37.664Z [INFO] server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: added peer, starting replication: peer=server-40a77dd1-a78c-6a61-d236-b6e7bb6fa3db -testLoggerAdapter verbose: server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: 2024-09-03T21:34:37.664Z [INFO] server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: added peer, starting replication: peer=server-2461d6a1-feda-723f-eb74-6ec87145f6c9 - testing.go:149: server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: 2024-09-03T21:34:37.664Z [INFO] server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: added peer, starting replication: peer=server-2461d6a1-feda-723f-eb74-6ec87145f6c9 -testLoggerAdapter verbose: server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: 2024-09-03T21:34:37.664Z [INFO] server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: pipelining replication: peer="{Voter server-2461d6a1-feda-723f-eb74-6ec87145f6c9 2461d6a1-feda-723f-eb74-6ec87145f6c9}" - testing.go:149: server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: 2024-09-03T21:34:37.664Z [INFO] server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: pipelining replication: peer="{Voter server-2461d6a1-feda-723f-eb74-6ec87145f6c9 2461d6a1-feda-723f-eb74-6ec87145f6c9}" -testLoggerAdapter verbose: server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: 2024-09-03T21:34:37.664Z [INFO] server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: pipelining replication: peer="{Voter server-40a77dd1-a78c-6a61-d236-b6e7bb6fa3db 40a77dd1-a78c-6a61-d236-b6e7bb6fa3db}" - testing.go:149: server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: 2024-09-03T21:34:37.664Z [INFO] server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: pipelining replication: peer="{Voter server-40a77dd1-a78c-6a61-d236-b6e7bb6fa3db 40a77dd1-a78c-6a61-d236-b6e7bb6fa3db}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:37.769Z [INFO] stable state for Leader reached at 2024-09-03 21:34:37.664100899 +0000 UTC m=+18.710758038 (1 nodes), 56.432507ms from start of poll, 56.9406ms from cluster start. Timeout at 2024-09-03 21:34:37.76912111 +0000 UTC m=+18.815778249, 105.020211ms after stability - testing.go:149: cluster: 2024-09-03T21:34:37.769Z [INFO] stable state for Leader reached at 2024-09-03 21:34:37.664100899 +0000 UTC m=+18.710758038 (1 nodes), 56.432507ms from start of poll, 56.9406ms from cluster start. Timeout at 2024-09-03 21:34:37.76912111 +0000 UTC m=+18.815778249, 105.020211ms after stability -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:37.769Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-09-03T21:34:37.769Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:37.874Z [INFO] stable state for Follower reached at 2024-09-03 21:34:37.769269584 +0000 UTC m=+18.815926723 (2 nodes), 1.401µs from start of poll, 162.109285ms from cluster start. Timeout at 2024-09-03 21:34:37.874325954 +0000 UTC m=+18.920983085, 105.056362ms after stability - testing.go:149: cluster: 2024-09-03T21:34:37.874Z [INFO] stable state for Follower reached at 2024-09-03 21:34:37.769269584 +0000 UTC m=+18.815926723 (2 nodes), 1.401µs from start of poll, 162.109285ms from cluster start. Timeout at 2024-09-03 21:34:37.874325954 +0000 UTC m=+18.920983085, 105.056362ms after stability -testLoggerAdapter verbose: server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: 2024-09-03T21:34:38.152Z [INFO] server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: aborting pipeline replication: peer="{Voter server-2461d6a1-feda-723f-eb74-6ec87145f6c9 2461d6a1-feda-723f-eb74-6ec87145f6c9}" - testing.go:149: server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: 2024-09-03T21:34:38.152Z [INFO] server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: aborting pipeline replication: peer="{Voter server-2461d6a1-feda-723f-eb74-6ec87145f6c9 2461d6a1-feda-723f-eb74-6ec87145f6c9}" -testLoggerAdapter verbose: server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: 2024-09-03T21:34:38.152Z [INFO] server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: aborting pipeline replication: peer="{Voter server-40a77dd1-a78c-6a61-d236-b6e7bb6fa3db 40a77dd1-a78c-6a61-d236-b6e7bb6fa3db}" - testing.go:149: server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: 2024-09-03T21:34:38.152Z [INFO] server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: aborting pipeline replication: peer="{Voter server-40a77dd1-a78c-6a61-d236-b6e7bb6fa3db 40a77dd1-a78c-6a61-d236-b6e7bb6fa3db}" -testLoggerAdapter verbose: server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: 2024-09-03T21:34:38.202Z [ERROR] server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: failed to heartbeat to: peer=40a77dd1-a78c-6a61-d236-b6e7bb6fa3db error="command timed out" - testing.go:149: server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: 2024-09-03T21:34:38.202Z [ERROR] server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: failed to heartbeat to: peer=40a77dd1-a78c-6a61-d236-b6e7bb6fa3db error="command timed out" -testLoggerAdapter verbose: server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: 2024-09-03T21:34:38.202Z [ERROR] server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: failed to heartbeat to: peer=2461d6a1-feda-723f-eb74-6ec87145f6c9 error="command timed out" - testing.go:149: server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: 2024-09-03T21:34:38.202Z [ERROR] server-9008c2f0-d0ba-9c75-5721-eb1473aaf796: failed to heartbeat to: peer=2461d6a1-feda-723f-eb74-6ec87145f6c9 error="command timed out" ---- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (0.60s) +testLoggerAdapter verbose: server-1e7dbaaa-cfaf-c459-0903-55ccc957901e: 2023-08-02T15:13:06.741Z [INFO] server-1e7dbaaa-cfaf-c459-0903-55ccc957901e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e7dbaaa-cfaf-c459-0903-55ccc957901e Address:1e7dbaaa-cfaf-c459-0903-55ccc957901e} {Suffrage:Voter ID:server-ad663a07-48f4-b5af-a68c-21294e0da3c6 Address:ad663a07-48f4-b5af-a68c-21294e0da3c6} {Suffrage:Voter ID:server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 Address:5acdc2a5-4065-9b1f-727c-b07fb59d6ef3}]" + testing.go:149: server-1e7dbaaa-cfaf-c459-0903-55ccc957901e: 2023-08-02T15:13:06.741Z [INFO] server-1e7dbaaa-cfaf-c459-0903-55ccc957901e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e7dbaaa-cfaf-c459-0903-55ccc957901e Address:1e7dbaaa-cfaf-c459-0903-55ccc957901e} {Suffrage:Voter ID:server-ad663a07-48f4-b5af-a68c-21294e0da3c6 Address:ad663a07-48f4-b5af-a68c-21294e0da3c6} {Suffrage:Voter ID:server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 Address:5acdc2a5-4065-9b1f-727c-b07fb59d6ef3}]" +testLoggerAdapter verbose: server-1e7dbaaa-cfaf-c459-0903-55ccc957901e: 2023-08-02T15:13:06.741Z [INFO] server-1e7dbaaa-cfaf-c459-0903-55ccc957901e: entering follower state: follower="Node at 1e7dbaaa-cfaf-c459-0903-55ccc957901e [Follower]" leader= +testLoggerAdapter verbose: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.741Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e7dbaaa-cfaf-c459-0903-55ccc957901e Address:1e7dbaaa-cfaf-c459-0903-55ccc957901e} {Suffrage:Voter ID:server-ad663a07-48f4-b5af-a68c-21294e0da3c6 Address:ad663a07-48f4-b5af-a68c-21294e0da3c6} {Suffrage:Voter ID:server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 Address:5acdc2a5-4065-9b1f-727c-b07fb59d6ef3}]" + testing.go:149: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.741Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e7dbaaa-cfaf-c459-0903-55ccc957901e Address:1e7dbaaa-cfaf-c459-0903-55ccc957901e} {Suffrage:Voter ID:server-ad663a07-48f4-b5af-a68c-21294e0da3c6 Address:ad663a07-48f4-b5af-a68c-21294e0da3c6} {Suffrage:Voter ID:server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 Address:5acdc2a5-4065-9b1f-727c-b07fb59d6ef3}]" + testing.go:149: server-1e7dbaaa-cfaf-c459-0903-55ccc957901e: 2023-08-02T15:13:06.741Z [INFO] server-1e7dbaaa-cfaf-c459-0903-55ccc957901e: entering follower state: follower="Node at 1e7dbaaa-cfaf-c459-0903-55ccc957901e [Follower]" leader= +testLoggerAdapter verbose: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.743Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: entering follower state: follower="Node at ad663a07-48f4-b5af-a68c-21294e0da3c6 [Follower]" leader= + testing.go:149: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.743Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: entering follower state: follower="Node at ad663a07-48f4-b5af-a68c-21294e0da3c6 [Follower]" leader= +testLoggerAdapter verbose: server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: 2023-08-02T15:13:06.743Z [INFO] server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e7dbaaa-cfaf-c459-0903-55ccc957901e Address:1e7dbaaa-cfaf-c459-0903-55ccc957901e} {Suffrage:Voter ID:server-ad663a07-48f4-b5af-a68c-21294e0da3c6 Address:ad663a07-48f4-b5af-a68c-21294e0da3c6} {Suffrage:Voter ID:server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 Address:5acdc2a5-4065-9b1f-727c-b07fb59d6ef3}]" + testing.go:149: server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: 2023-08-02T15:13:06.743Z [INFO] server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e7dbaaa-cfaf-c459-0903-55ccc957901e Address:1e7dbaaa-cfaf-c459-0903-55ccc957901e} {Suffrage:Voter ID:server-ad663a07-48f4-b5af-a68c-21294e0da3c6 Address:ad663a07-48f4-b5af-a68c-21294e0da3c6} {Suffrage:Voter ID:server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 Address:5acdc2a5-4065-9b1f-727c-b07fb59d6ef3}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:06.744Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:13:06.744Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: 2023-08-02T15:13:06.744Z [INFO] server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: entering follower state: follower="Node at 5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 [Follower]" leader= + testing.go:149: server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: 2023-08-02T15:13:06.744Z [INFO] server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: entering follower state: follower="Node at 5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 [Follower]" leader= +testLoggerAdapter verbose: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.810Z [WARN] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.810Z [WARN] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.810Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: entering candidate state: node="Node at ad663a07-48f4-b5af-a68c-21294e0da3c6 [Candidate]" term=2 + testing.go:149: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.810Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: entering candidate state: node="Node at ad663a07-48f4-b5af-a68c-21294e0da3c6 [Candidate]" term=2 +testLoggerAdapter verbose: server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: 2023-08-02T15:13:06.810Z [WARN] server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: 2023-08-02T15:13:06.810Z [WARN] server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: 2023-08-02T15:13:06.810Z [INFO] server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: entering candidate state: node="Node at 5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 [Candidate]" term=2 +testLoggerAdapter verbose: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.810Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: election won: tally=2 + testing.go:149: server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: 2023-08-02T15:13:06.810Z [INFO] server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: entering candidate state: node="Node at 5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 [Candidate]" term=2 + testing.go:149: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.810Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: election won: tally=2 +testLoggerAdapter verbose: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.810Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: entering leader state: leader="Node at ad663a07-48f4-b5af-a68c-21294e0da3c6 [Leader]" + testing.go:149: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.810Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: entering leader state: leader="Node at ad663a07-48f4-b5af-a68c-21294e0da3c6 [Leader]" +testLoggerAdapter verbose: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.810Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: added peer, starting replication: peer=server-1e7dbaaa-cfaf-c459-0903-55ccc957901e +testLoggerAdapter verbose: server-1e7dbaaa-cfaf-c459-0903-55ccc957901e: 2023-08-02T15:13:06.810Z [INFO] server-1e7dbaaa-cfaf-c459-0903-55ccc957901e: duplicate requestVote for same term: term=2 + testing.go:149: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.810Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: added peer, starting replication: peer=server-1e7dbaaa-cfaf-c459-0903-55ccc957901e + testing.go:149: server-1e7dbaaa-cfaf-c459-0903-55ccc957901e: 2023-08-02T15:13:06.810Z [INFO] server-1e7dbaaa-cfaf-c459-0903-55ccc957901e: duplicate requestVote for same term: term=2 +testLoggerAdapter verbose: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.810Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: added peer, starting replication: peer=server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 + testing.go:149: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.810Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: added peer, starting replication: peer=server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 +testLoggerAdapter verbose: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.810Z [WARN] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: rejecting vote request since we have a leader: from=5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 leader=ad663a07-48f4-b5af-a68c-21294e0da3c6 + testing.go:149: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.810Z [WARN] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: rejecting vote request since we have a leader: from=5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 leader=ad663a07-48f4-b5af-a68c-21294e0da3c6 +testLoggerAdapter verbose: server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: 2023-08-02T15:13:06.810Z [INFO] server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: entering follower state: follower="Node at 5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 [Follower]" leader=ad663a07-48f4-b5af-a68c-21294e0da3c6 + testing.go:149: server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: 2023-08-02T15:13:06.810Z [INFO] server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: entering follower state: follower="Node at 5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 [Follower]" leader=ad663a07-48f4-b5af-a68c-21294e0da3c6 +testLoggerAdapter verbose: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.811Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: pipelining replication: peer="{Voter server-1e7dbaaa-cfaf-c459-0903-55ccc957901e 1e7dbaaa-cfaf-c459-0903-55ccc957901e}" + testing.go:149: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.811Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: pipelining replication: peer="{Voter server-1e7dbaaa-cfaf-c459-0903-55ccc957901e 1e7dbaaa-cfaf-c459-0903-55ccc957901e}" +testLoggerAdapter verbose: server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: 2023-08-02T15:13:06.811Z [INFO] server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: duplicate requestVote for same term: term=2 + testing.go:149: server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: 2023-08-02T15:13:06.811Z [INFO] server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: duplicate requestVote for same term: term=2 +testLoggerAdapter verbose: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.811Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: pipelining replication: peer="{Voter server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 5acdc2a5-4065-9b1f-727c-b07fb59d6ef3}" + testing.go:149: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.811Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: pipelining replication: peer="{Voter server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 5acdc2a5-4065-9b1f-727c-b07fb59d6ef3}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:06.916Z [INFO] stable state for Leader reached at 2023-08-02 15:13:06.811106675 +0000 UTC m=+18.907043026 (1 nodes), 66.943868ms from start of poll, 70.204546ms from cluster start. Timeout at 2023-08-02 15:13:06.916675089 +0000 UTC m=+19.012611425, 105.568399ms after stability + testing.go:149: cluster: 2023-08-02T15:13:06.916Z [INFO] stable state for Leader reached at 2023-08-02 15:13:06.811106675 +0000 UTC m=+18.907043026 (1 nodes), 66.943868ms from start of poll, 70.204546ms from cluster start. Timeout at 2023-08-02 15:13:06.916675089 +0000 UTC m=+19.012611425, 105.568399ms after stability +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:06.916Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-08-02T15:13:06.916Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:07.022Z [INFO] stable state for Follower reached at 2023-08-02 15:13:06.916843278 +0000 UTC m=+19.012779615 (2 nodes), 1.377µs from start of poll, 175.941135ms from cluster start. Timeout at 2023-08-02 15:13:07.022916744 +0000 UTC m=+19.118853083, 106.073468ms after stability + testing.go:149: cluster: 2023-08-02T15:13:07.022Z [INFO] stable state for Follower reached at 2023-08-02 15:13:06.916843278 +0000 UTC m=+19.012779615 (2 nodes), 1.377µs from start of poll, 175.941135ms from cluster start. Timeout at 2023-08-02 15:13:07.022916744 +0000 UTC m=+19.118853083, 106.073468ms after stability +testLoggerAdapter verbose: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:07.389Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: aborting pipeline replication: peer="{Voter server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 5acdc2a5-4065-9b1f-727c-b07fb59d6ef3}" + testing.go:149: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:07.389Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: aborting pipeline replication: peer="{Voter server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 5acdc2a5-4065-9b1f-727c-b07fb59d6ef3}" +testLoggerAdapter verbose: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:07.389Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: aborting pipeline replication: peer="{Voter server-1e7dbaaa-cfaf-c459-0903-55ccc957901e 1e7dbaaa-cfaf-c459-0903-55ccc957901e}" + testing.go:149: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:07.389Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: aborting pipeline replication: peer="{Voter server-1e7dbaaa-cfaf-c459-0903-55ccc957901e 1e7dbaaa-cfaf-c459-0903-55ccc957901e}" +--- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (0.65s) === RUN TestRaft_LeadershipTransferStopRightAway --- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s) === RUN TestRaft_GetConfigurationNoBootstrap -testLoggerAdapter verbose: server-02e21a09-4480-c584-c888-edfc263fbb81: 2024-09-03T21:34:38.203Z [INFO] server-02e21a09-4480-c584-c888-edfc263fbb81: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-02e21a09-4480-c584-c888-edfc263fbb81 Address:02e21a09-4480-c584-c888-edfc263fbb81} {Suffrage:Voter ID:server-db6923a8-a236-985c-643d-3a0a01e82211 Address:db6923a8-a236-985c-643d-3a0a01e82211}]" - testing.go:149: server-02e21a09-4480-c584-c888-edfc263fbb81: 2024-09-03T21:34:38.203Z [INFO] server-02e21a09-4480-c584-c888-edfc263fbb81: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-02e21a09-4480-c584-c888-edfc263fbb81 Address:02e21a09-4480-c584-c888-edfc263fbb81} {Suffrage:Voter ID:server-db6923a8-a236-985c-643d-3a0a01e82211 Address:db6923a8-a236-985c-643d-3a0a01e82211}]" -testLoggerAdapter verbose: server-db6923a8-a236-985c-643d-3a0a01e82211: 2024-09-03T21:34:38.204Z [INFO] server-db6923a8-a236-985c-643d-3a0a01e82211: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-02e21a09-4480-c584-c888-edfc263fbb81 Address:02e21a09-4480-c584-c888-edfc263fbb81} {Suffrage:Voter ID:server-db6923a8-a236-985c-643d-3a0a01e82211 Address:db6923a8-a236-985c-643d-3a0a01e82211}]" - testing.go:149: server-db6923a8-a236-985c-643d-3a0a01e82211: 2024-09-03T21:34:38.204Z [INFO] server-db6923a8-a236-985c-643d-3a0a01e82211: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-02e21a09-4480-c584-c888-edfc263fbb81 Address:02e21a09-4480-c584-c888-edfc263fbb81} {Suffrage:Voter ID:server-db6923a8-a236-985c-643d-3a0a01e82211 Address:db6923a8-a236-985c-643d-3a0a01e82211}]" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:38.204Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2024-09-03T21:34:38.204Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-02e21a09-4480-c584-c888-edfc263fbb81: 2024-09-03T21:34:38.204Z [INFO] server-02e21a09-4480-c584-c888-edfc263fbb81: entering follower state: follower="Node at 02e21a09-4480-c584-c888-edfc263fbb81 [Follower]" leader= - testing.go:149: server-02e21a09-4480-c584-c888-edfc263fbb81: 2024-09-03T21:34:38.204Z [INFO] server-02e21a09-4480-c584-c888-edfc263fbb81: entering follower state: follower="Node at 02e21a09-4480-c584-c888-edfc263fbb81 [Follower]" leader= -testLoggerAdapter verbose: server-db6923a8-a236-985c-643d-3a0a01e82211: 2024-09-03T21:34:38.204Z [INFO] server-db6923a8-a236-985c-643d-3a0a01e82211: entering follower state: follower="Node at db6923a8-a236-985c-643d-3a0a01e82211 [Follower]" leader= - testing.go:149: server-db6923a8-a236-985c-643d-3a0a01e82211: 2024-09-03T21:34:38.204Z [INFO] server-db6923a8-a236-985c-643d-3a0a01e82211: entering follower state: follower="Node at db6923a8-a236-985c-643d-3a0a01e82211 [Follower]" leader= -testLoggerAdapter verbose: server-db6923a8-a236-985c-643d-3a0a01e82211: 2024-09-03T21:34:38.275Z [WARN] server-db6923a8-a236-985c-643d-3a0a01e82211: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-db6923a8-a236-985c-643d-3a0a01e82211: 2024-09-03T21:34:38.275Z [WARN] server-db6923a8-a236-985c-643d-3a0a01e82211: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-db6923a8-a236-985c-643d-3a0a01e82211: 2024-09-03T21:34:38.275Z [INFO] server-db6923a8-a236-985c-643d-3a0a01e82211: entering candidate state: node="Node at db6923a8-a236-985c-643d-3a0a01e82211 [Candidate]" term=2 - testing.go:149: server-db6923a8-a236-985c-643d-3a0a01e82211: 2024-09-03T21:34:38.275Z [INFO] server-db6923a8-a236-985c-643d-3a0a01e82211: entering candidate state: node="Node at db6923a8-a236-985c-643d-3a0a01e82211 [Candidate]" term=2 -testLoggerAdapter verbose: server-db6923a8-a236-985c-643d-3a0a01e82211: 2024-09-03T21:34:38.275Z [INFO] server-db6923a8-a236-985c-643d-3a0a01e82211: election won: tally=2 - testing.go:149: server-db6923a8-a236-985c-643d-3a0a01e82211: 2024-09-03T21:34:38.275Z [INFO] server-db6923a8-a236-985c-643d-3a0a01e82211: election won: tally=2 -testLoggerAdapter verbose: server-db6923a8-a236-985c-643d-3a0a01e82211: 2024-09-03T21:34:38.275Z [INFO] server-db6923a8-a236-985c-643d-3a0a01e82211: entering leader state: leader="Node at db6923a8-a236-985c-643d-3a0a01e82211 [Leader]" - testing.go:149: server-db6923a8-a236-985c-643d-3a0a01e82211: 2024-09-03T21:34:38.275Z [INFO] server-db6923a8-a236-985c-643d-3a0a01e82211: entering leader state: leader="Node at db6923a8-a236-985c-643d-3a0a01e82211 [Leader]" -testLoggerAdapter verbose: server-db6923a8-a236-985c-643d-3a0a01e82211: 2024-09-03T21:34:38.275Z [INFO] server-db6923a8-a236-985c-643d-3a0a01e82211: added peer, starting replication: peer=server-02e21a09-4480-c584-c888-edfc263fbb81 - testing.go:149: server-db6923a8-a236-985c-643d-3a0a01e82211: 2024-09-03T21:34:38.275Z [INFO] server-db6923a8-a236-985c-643d-3a0a01e82211: added peer, starting replication: peer=server-02e21a09-4480-c584-c888-edfc263fbb81 -testLoggerAdapter verbose: server-db6923a8-a236-985c-643d-3a0a01e82211: 2024-09-03T21:34:38.276Z [INFO] server-db6923a8-a236-985c-643d-3a0a01e82211: pipelining replication: peer="{Voter server-02e21a09-4480-c584-c888-edfc263fbb81 02e21a09-4480-c584-c888-edfc263fbb81}" - testing.go:149: server-db6923a8-a236-985c-643d-3a0a01e82211: 2024-09-03T21:34:38.276Z [INFO] server-db6923a8-a236-985c-643d-3a0a01e82211: pipelining replication: peer="{Voter server-02e21a09-4480-c584-c888-edfc263fbb81 02e21a09-4480-c584-c888-edfc263fbb81}" -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:38.381Z [INFO] stable state for Follower reached at 2024-09-03 21:34:38.27600272 +0000 UTC m=+19.322659886 (1 nodes), 71.767982ms from start of poll, 72.203058ms from cluster start. Timeout at 2024-09-03 21:34:38.381115667 +0000 UTC m=+19.427772800, 105.112914ms after stability - testing.go:149: cluster: 2024-09-03T21:34:38.381Z [INFO] stable state for Follower reached at 2024-09-03 21:34:38.27600272 +0000 UTC m=+19.322659886 (1 nodes), 71.767982ms from start of poll, 72.203058ms from cluster start. Timeout at 2024-09-03 21:34:38.381115667 +0000 UTC m=+19.427772800, 105.112914ms after stability -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:38.381Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2024-09-03T21:34:38.381Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2024-09-03T21:34:38.486Z [INFO] stable state for Leader reached at 2024-09-03 21:34:38.381282007 +0000 UTC m=+19.427939147 (1 nodes), 1.592µs from start of poll, 177.482319ms from cluster start. Timeout at 2024-09-03 21:34:38.48637387 +0000 UTC m=+19.533031012, 105.091865ms after stability - testing.go:149: cluster: 2024-09-03T21:34:38.486Z [INFO] stable state for Leader reached at 2024-09-03 21:34:38.381282007 +0000 UTC m=+19.427939147 (1 nodes), 1.592µs from start of poll, 177.482319ms from cluster start. Timeout at 2024-09-03 21:34:38.48637387 +0000 UTC m=+19.533031012, 105.091865ms after stability -testLoggerAdapter verbose: server-db6923a8-a236-985c-643d-3a0a01e82211: 2024-09-03T21:34:38.496Z [INFO] server-db6923a8-a236-985c-643d-3a0a01e82211: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-02e21a09-4480-c584-c888-edfc263fbb81 Address:02e21a09-4480-c584-c888-edfc263fbb81} {Suffrage:Voter ID:server-db6923a8-a236-985c-643d-3a0a01e82211 Address:db6923a8-a236-985c-643d-3a0a01e82211}]" - testing.go:149: server-db6923a8-a236-985c-643d-3a0a01e82211: 2024-09-03T21:34:38.496Z [INFO] server-db6923a8-a236-985c-643d-3a0a01e82211: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-02e21a09-4480-c584-c888-edfc263fbb81 Address:02e21a09-4480-c584-c888-edfc263fbb81} {Suffrage:Voter ID:server-db6923a8-a236-985c-643d-3a0a01e82211 Address:db6923a8-a236-985c-643d-3a0a01e82211}]" -testLoggerAdapter verbose: server-db6923a8-a236-985c-643d-3a0a01e82211: 2024-09-03T21:34:38.497Z [INFO] server-db6923a8-a236-985c-643d-3a0a01e82211: aborting pipeline replication: peer="{Voter server-02e21a09-4480-c584-c888-edfc263fbb81 02e21a09-4480-c584-c888-edfc263fbb81}" - testing.go:149: server-db6923a8-a236-985c-643d-3a0a01e82211: 2024-09-03T21:34:38.497Z [INFO] server-db6923a8-a236-985c-643d-3a0a01e82211: aborting pipeline replication: peer="{Voter server-02e21a09-4480-c584-c888-edfc263fbb81 02e21a09-4480-c584-c888-edfc263fbb81}" +testLoggerAdapter verbose: server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f: 2023-08-02T15:13:07.390Z [INFO] server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f Address:2eff0449-18f1-1803-8b89-a0cb3d2ed45f} {Suffrage:Voter ID:server-0c0b1d30-08a0-862e-86f5-15c95132833d Address:0c0b1d30-08a0-862e-86f5-15c95132833d}]" + testing.go:149: server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f: 2023-08-02T15:13:07.390Z [INFO] server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f Address:2eff0449-18f1-1803-8b89-a0cb3d2ed45f} {Suffrage:Voter ID:server-0c0b1d30-08a0-862e-86f5-15c95132833d Address:0c0b1d30-08a0-862e-86f5-15c95132833d}]" +testLoggerAdapter verbose: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.390Z [INFO] server-0c0b1d30-08a0-862e-86f5-15c95132833d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f Address:2eff0449-18f1-1803-8b89-a0cb3d2ed45f} {Suffrage:Voter ID:server-0c0b1d30-08a0-862e-86f5-15c95132833d Address:0c0b1d30-08a0-862e-86f5-15c95132833d}]" + testing.go:149: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.390Z [INFO] server-0c0b1d30-08a0-862e-86f5-15c95132833d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f Address:2eff0449-18f1-1803-8b89-a0cb3d2ed45f} {Suffrage:Voter ID:server-0c0b1d30-08a0-862e-86f5-15c95132833d Address:0c0b1d30-08a0-862e-86f5-15c95132833d}]" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:07.390Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2023-08-02T15:13:07.390Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f: 2023-08-02T15:13:07.390Z [INFO] server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f: entering follower state: follower="Node at 2eff0449-18f1-1803-8b89-a0cb3d2ed45f [Follower]" leader= + testing.go:149: server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f: 2023-08-02T15:13:07.390Z [INFO] server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f: entering follower state: follower="Node at 2eff0449-18f1-1803-8b89-a0cb3d2ed45f [Follower]" leader= +testLoggerAdapter verbose: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.390Z [INFO] server-0c0b1d30-08a0-862e-86f5-15c95132833d: entering follower state: follower="Node at 0c0b1d30-08a0-862e-86f5-15c95132833d [Follower]" leader= + testing.go:149: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.390Z [INFO] server-0c0b1d30-08a0-862e-86f5-15c95132833d: entering follower state: follower="Node at 0c0b1d30-08a0-862e-86f5-15c95132833d [Follower]" leader= +testLoggerAdapter verbose: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.459Z [WARN] server-0c0b1d30-08a0-862e-86f5-15c95132833d: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.459Z [WARN] server-0c0b1d30-08a0-862e-86f5-15c95132833d: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.460Z [INFO] server-0c0b1d30-08a0-862e-86f5-15c95132833d: entering candidate state: node="Node at 0c0b1d30-08a0-862e-86f5-15c95132833d [Candidate]" term=2 + testing.go:149: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.460Z [INFO] server-0c0b1d30-08a0-862e-86f5-15c95132833d: entering candidate state: node="Node at 0c0b1d30-08a0-862e-86f5-15c95132833d [Candidate]" term=2 +testLoggerAdapter verbose: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.463Z [INFO] server-0c0b1d30-08a0-862e-86f5-15c95132833d: election won: tally=2 + testing.go:149: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.463Z [INFO] server-0c0b1d30-08a0-862e-86f5-15c95132833d: election won: tally=2 +testLoggerAdapter verbose: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.463Z [INFO] server-0c0b1d30-08a0-862e-86f5-15c95132833d: entering leader state: leader="Node at 0c0b1d30-08a0-862e-86f5-15c95132833d [Leader]" + testing.go:149: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.463Z [INFO] server-0c0b1d30-08a0-862e-86f5-15c95132833d: entering leader state: leader="Node at 0c0b1d30-08a0-862e-86f5-15c95132833d [Leader]" +testLoggerAdapter verbose: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.463Z [INFO] server-0c0b1d30-08a0-862e-86f5-15c95132833d: added peer, starting replication: peer=server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f + testing.go:149: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.463Z [INFO] server-0c0b1d30-08a0-862e-86f5-15c95132833d: added peer, starting replication: peer=server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f +testLoggerAdapter verbose: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.463Z [INFO] server-0c0b1d30-08a0-862e-86f5-15c95132833d: pipelining replication: peer="{Voter server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f 2eff0449-18f1-1803-8b89-a0cb3d2ed45f}" + testing.go:149: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.463Z [INFO] server-0c0b1d30-08a0-862e-86f5-15c95132833d: pipelining replication: peer="{Voter server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f 2eff0449-18f1-1803-8b89-a0cb3d2ed45f}" +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:07.568Z [INFO] stable state for Follower reached at 2023-08-02 15:13:07.46341475 +0000 UTC m=+19.559351103 (1 nodes), 72.519126ms from start of poll, 72.882111ms from cluster start. Timeout at 2023-08-02 15:13:07.568468396 +0000 UTC m=+19.664404746, 105.053643ms after stability + testing.go:149: cluster: 2023-08-02T15:13:07.568Z [INFO] stable state for Follower reached at 2023-08-02 15:13:07.46341475 +0000 UTC m=+19.559351103 (1 nodes), 72.519126ms from start of poll, 72.882111ms from cluster start. Timeout at 2023-08-02 15:13:07.568468396 +0000 UTC m=+19.664404746, 105.053643ms after stability +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:07.568Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2023-08-02T15:13:07.568Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2023-08-02T15:13:07.673Z [INFO] stable state for Leader reached at 2023-08-02 15:13:07.568662813 +0000 UTC m=+19.664599149 (1 nodes), 2.17µs from start of poll, 178.130157ms from cluster start. Timeout at 2023-08-02 15:13:07.67379294 +0000 UTC m=+19.769729295, 105.130146ms after stability + testing.go:149: cluster: 2023-08-02T15:13:07.673Z [INFO] stable state for Leader reached at 2023-08-02 15:13:07.568662813 +0000 UTC m=+19.664599149 (1 nodes), 2.17µs from start of poll, 178.130157ms from cluster start. Timeout at 2023-08-02 15:13:07.67379294 +0000 UTC m=+19.769729295, 105.130146ms after stability +testLoggerAdapter verbose: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.684Z [INFO] server-0c0b1d30-08a0-862e-86f5-15c95132833d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f Address:2eff0449-18f1-1803-8b89-a0cb3d2ed45f} {Suffrage:Voter ID:server-0c0b1d30-08a0-862e-86f5-15c95132833d Address:0c0b1d30-08a0-862e-86f5-15c95132833d}]" + testing.go:149: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.684Z [INFO] server-0c0b1d30-08a0-862e-86f5-15c95132833d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f Address:2eff0449-18f1-1803-8b89-a0cb3d2ed45f} {Suffrage:Voter ID:server-0c0b1d30-08a0-862e-86f5-15c95132833d Address:0c0b1d30-08a0-862e-86f5-15c95132833d}]" +testLoggerAdapter verbose: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.684Z [INFO] server-0c0b1d30-08a0-862e-86f5-15c95132833d: aborting pipeline replication: peer="{Voter server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f 2eff0449-18f1-1803-8b89-a0cb3d2ed45f}" + testing.go:149: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.684Z [INFO] server-0c0b1d30-08a0-862e-86f5-15c95132833d: aborting pipeline replication: peer="{Voter server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f 2eff0449-18f1-1803-8b89-a0cb3d2ed45f}" --- PASS: TestRaft_GetConfigurationNoBootstrap (0.29s) === RUN TestTCPTransport_BadAddr --- PASS: TestTCPTransport_BadAddr (0.00s) @@ -3925,7 +4018,7 @@ === RUN TestBackoff --- PASS: TestBackoff (0.00s) PASS -ok github.com/hashicorp/raft 19.561s +ok github.com/hashicorp/raft 19.806s ? github.com/hashicorp/raft/bench [no test files] create-stamp debian/debhelper-build-stamp fakeroot debian/rules binary @@ -3957,12 +4050,14 @@ dpkg-buildpackage: info: binary-only upload (no source included) dpkg-genchanges: info: including full source code in upload I: copying local configuration +I: user script /srv/workspace/pbuilder/2531752/tmp/hooks/B01_cleanup starting +I: user script /srv/workspace/pbuilder/2531752/tmp/hooks/B01_cleanup finished I: unmounting dev/ptmx filesystem I: unmounting dev/pts filesystem I: unmounting dev/shm filesystem I: unmounting proc filesystem I: unmounting sys filesystem I: cleaning the build env -I: removing directory /srv/workspace/pbuilder/1777596 and its subdirectories -I: Current time: Tue Sep 3 09:34:44 -12 2024 -I: pbuilder-time-stamp: 1725399284 +I: removing directory /srv/workspace/pbuilder/2531752 and its subdirectories +I: Current time: Thu Aug 3 05:13:16 +14 2023 +I: pbuilder-time-stamp: 1690989196