Diff of the two buildlogs: -- --- b1/build.log 2021-07-10 01:05:23.894821358 +0000 +++ b2/build.log 2021-07-10 01:07:28.522147283 +0000 @@ -1,6 +1,6 @@ I: pbuilder: network access will be disabled during build -I: Current time: Fri Jul 9 13:03:34 -12 2021 -I: pbuilder-time-stamp: 1625879014 +I: Current time: Fri Aug 12 21:28:26 +14 2022 +I: pbuilder-time-stamp: 1660289306 I: Building the build Environment I: extracting base tarball [/var/cache/pbuilder/bullseye-reproducible-base.tgz] I: copying local configuration @@ -16,8 +16,8 @@ I: copying [./golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz] I: Extracting source gpgv: unknown type of key resource 'trustedkeys.kbx' -gpgv: keyblock resource '/tmp/dpkg-verify-sig.vJtVmMUa/trustedkeys.kbx': General error -gpgv: Signature made Sat Jan 18 19:24:28 2020 -12 +gpgv: keyblock resource '/tmp/dpkg-verify-sig.WJHNHUpY/trustedkeys.kbx': General error +gpgv: Signature made Sun Jan 19 21:24:28 2020 +14 gpgv: using RSA key 50BC7CF939D20C272A6B065652B6BBD953968D1B gpgv: Can't check signature: No public key dpkg-source: warning: failed to verify signature on ./golang-github-hashicorp-raft_1.1.2-1.dsc @@ -32,135 +32,169 @@ dpkg-source: info: applying t-skip--TestRaft_SnapshotRestore_PeerChange.patch I: using fakeroot in build. I: Installing the build-deps -I: user script /srv/workspace/pbuilder/15724/tmp/hooks/D02_print_environment starting +I: user script /srv/workspace/pbuilder/22862/tmp/hooks/D01_modify_environment starting +debug: Running on codethink11-arm64. +I: Changing host+domainname to test build reproducibility +I: Adding a custom variable just for the fun of it... +I: Changing /bin/sh to bash +Removing 'diversion of /bin/sh to /bin/sh.distrib by dash' +Adding 'diversion of /bin/sh to /bin/sh.distrib by bash' +Removing 'diversion of /usr/share/man/man1/sh.1.gz to /usr/share/man/man1/sh.distrib.1.gz by dash' +Adding 'diversion of /usr/share/man/man1/sh.1.gz to /usr/share/man/man1/sh.distrib.1.gz by bash' +I: Setting pbuilder2's login shell to /bin/bash +I: Setting pbuilder2's GECOS to second user,second room,second work-phone,second home-phone,second other +I: user script /srv/workspace/pbuilder/22862/tmp/hooks/D01_modify_environment finished +I: user script /srv/workspace/pbuilder/22862/tmp/hooks/D02_print_environment starting I: set - BUILDDIR='/build' - BUILDUSERGECOS='first user,first room,first work-phone,first home-phone,first other' - BUILDUSERNAME='pbuilder1' - BUILD_ARCH='arm64' - DEBIAN_FRONTEND='noninteractive' + BASH=/bin/sh + BASHOPTS=checkwinsize:cmdhist:complete_fullquote:extquote:force_fignore:globasciiranges:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath + BASH_ALIASES=() + BASH_ARGC=() + BASH_ARGV=() + BASH_CMDS=() + BASH_LINENO=([0]="12" [1]="0") + BASH_SOURCE=([0]="/tmp/hooks/D02_print_environment" [1]="/tmp/hooks/D02_print_environment") + BASH_VERSINFO=([0]="5" [1]="1" [2]="4" [3]="1" [4]="release" [5]="aarch64-unknown-linux-gnu") + BASH_VERSION='5.1.4(1)-release' + BUILDDIR=/build + BUILDUSERGECOS='second user,second room,second work-phone,second home-phone,second other' + BUILDUSERNAME=pbuilder2 + BUILD_ARCH=arm64 + DEBIAN_FRONTEND=noninteractive DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all,-fixfilepath parallel=8' - DISTRIBUTION='' - HOME='/var/lib/jenkins' - HOST_ARCH='arm64' + DIRSTACK=() + DISTRIBUTION= + EUID=0 + FUNCNAME=([0]="Echo" [1]="main") + GROUPS=() + HOME=/var/lib/jenkins + HOSTNAME=i-capture-the-hostname + HOSTTYPE=aarch64 + HOST_ARCH=arm64 IFS=' ' - LANG='C' - LANGUAGE='en_US:en' - LC_ALL='C' - MAIL='/var/mail/root' - OPTIND='1' - PATH='/usr/sbin:/usr/bin:/sbin:/bin:/usr/games' - PBCURRENTCOMMANDLINEOPERATION='build' - PBUILDER_OPERATION='build' - PBUILDER_PKGDATADIR='/usr/share/pbuilder' - PBUILDER_PKGLIBDIR='/usr/lib/pbuilder' - PBUILDER_SYSCONFDIR='/etc' - PPID='15724' - PS1='# ' - PS2='> ' + LANG=C + LANGUAGE=nl_BE:nl + LC_ALL=C + MACHTYPE=aarch64-unknown-linux-gnu + MAIL=/var/mail/root + OPTERR=1 + OPTIND=1 + OSTYPE=linux-gnu + PATH=/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path + PBCURRENTCOMMANDLINEOPERATION=build + PBUILDER_OPERATION=build + PBUILDER_PKGDATADIR=/usr/share/pbuilder + PBUILDER_PKGLIBDIR=/usr/lib/pbuilder + PBUILDER_SYSCONFDIR=/etc + PIPESTATUS=([0]="0") + POSIXLY_CORRECT=y + PPID=22862 PS4='+ ' - PWD='/' - SHELL='/bin/bash' - SHLVL='2' - SUDO_COMMAND='/usr/bin/timeout -k 18.1h 18h /usr/bin/ionice -c 3 /usr/bin/nice /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/tmp.tE7qwSSjNo/pbuilderrc_G4FL --hookdir /etc/pbuilder/first-build-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/bullseye-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/tmp.tE7qwSSjNo/b1 --logfile b1/build.log golang-github-hashicorp-raft_1.1.2-1.dsc' - SUDO_GID='117' - SUDO_UID='110' - SUDO_USER='jenkins' - TERM='unknown' - TZ='/usr/share/zoneinfo/Etc/GMT+12' - USER='root' - USERNAME='root' - _='/usr/bin/systemd-run' - http_proxy='http://192.168.101.16:3128' + PWD=/ + SHELL=/bin/bash + SHELLOPTS=braceexpand:errexit:hashall:interactive-comments:posix + SHLVL=3 + SUDO_COMMAND='/usr/bin/timeout -k 24.1h 24h /usr/bin/ionice -c 3 /usr/bin/nice -n 11 /usr/bin/unshare --uts -- /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/tmp.tE7qwSSjNo/pbuilderrc_Dqg7 --hookdir /etc/pbuilder/rebuild-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/bullseye-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/tmp.tE7qwSSjNo/b2 --logfile b2/build.log --extrapackages usrmerge golang-github-hashicorp-raft_1.1.2-1.dsc' + SUDO_GID=117 + SUDO_UID=110 + SUDO_USER=jenkins + TERM=unknown + TZ=/usr/share/zoneinfo/Etc/GMT-14 + UID=0 + USER=root + USERNAME=root + _='I: set' + http_proxy=http://192.168.101.16:3128 I: uname -a - Linux codethink16-arm64 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:18:37 UTC 2021 aarch64 GNU/Linux + Linux i-capture-the-hostname 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:18:37 UTC 2021 aarch64 GNU/Linux I: ls -l /bin total 5252 - -rwxr-xr-x 1 root root 1282512 Jun 21 14:26 bash - -rwxr-xr-x 3 root root 34808 Jul 20 2020 bunzip2 - -rwxr-xr-x 3 root root 34808 Jul 20 2020 bzcat - lrwxrwxrwx 1 root root 6 Jul 20 2020 bzcmp -> bzdiff - -rwxr-xr-x 1 root root 2225 Jul 20 2020 bzdiff - lrwxrwxrwx 1 root root 6 Jul 20 2020 bzegrep -> bzgrep - -rwxr-xr-x 1 root root 4877 Sep 4 2019 bzexe - lrwxrwxrwx 1 root root 6 Jul 20 2020 bzfgrep -> bzgrep - -rwxr-xr-x 1 root root 3775 Jul 20 2020 bzgrep - -rwxr-xr-x 3 root root 34808 Jul 20 2020 bzip2 - -rwxr-xr-x 1 root root 14264 Jul 20 2020 bzip2recover - lrwxrwxrwx 1 root root 6 Jul 20 2020 bzless -> bzmore - -rwxr-xr-x 1 root root 1297 Jul 20 2020 bzmore - -rwxr-xr-x 1 root root 39832 Sep 22 2020 cat - -rwxr-xr-x 1 root root 64512 Sep 22 2020 chgrp - -rwxr-xr-x 1 root root 60368 Sep 22 2020 chmod - -rwxr-xr-x 1 root root 64528 Sep 22 2020 chown - -rwxr-xr-x 1 root root 138896 Sep 22 2020 cp - -rwxr-xr-x 1 root root 129544 Dec 10 2020 dash - -rwxr-xr-x 1 root root 101384 Sep 22 2020 date - -rwxr-xr-x 1 root root 80984 Sep 22 2020 dd - -rwxr-xr-x 1 root root 89824 Sep 22 2020 df - -rwxr-xr-x 1 root root 143088 Sep 22 2020 dir - -rwxr-xr-x 1 root root 76152 Feb 7 02:38 dmesg - lrwxrwxrwx 1 root root 8 Nov 6 2019 dnsdomainname -> hostname - lrwxrwxrwx 1 root root 8 Nov 6 2019 domainname -> hostname - -rwxr-xr-x 1 root root 35632 Sep 22 2020 echo - -rwxr-xr-x 1 root root 28 Nov 9 2020 egrep - -rwxr-xr-x 1 root root 31512 Sep 22 2020 false - -rwxr-xr-x 1 root root 28 Nov 9 2020 fgrep - -rwxr-xr-x 1 root root 64856 Feb 7 02:38 findmnt - -rwsr-xr-x 1 root root 34824 Feb 26 04:12 fusermount - -rwxr-xr-x 1 root root 178400 Nov 9 2020 grep - -rwxr-xr-x 2 root root 2346 Mar 2 11:30 gunzip - -rwxr-xr-x 1 root root 6376 Mar 2 11:30 gzexe - -rwxr-xr-x 1 root root 93744 Mar 2 11:30 gzip - -rwxr-xr-x 1 root root 18440 Nov 6 2019 hostname - -rwxr-xr-x 1 root root 68720 Sep 22 2020 ln - -rwxr-xr-x 1 root root 52720 Feb 7 2020 login - -rwxr-xr-x 1 root root 143088 Sep 22 2020 ls - -rwxr-xr-x 1 root root 161960 Feb 7 02:38 lsblk - -rwxr-xr-x 1 root root 85200 Sep 22 2020 mkdir - -rwxr-xr-x 1 root root 68744 Sep 22 2020 mknod - -rwxr-xr-x 1 root root 43976 Sep 22 2020 mktemp - -rwxr-xr-x 1 root root 51368 Feb 7 02:38 more - -rwsr-xr-x 1 root root 51360 Feb 7 02:38 mount - -rwxr-xr-x 1 root root 14496 Feb 7 02:38 mountpoint - -rwxr-xr-x 1 root root 134808 Sep 22 2020 mv - lrwxrwxrwx 1 root root 8 Nov 6 2019 nisdomainname -> hostname - lrwxrwxrwx 1 root root 14 Apr 18 03:38 pidof -> /sbin/killall5 - -rwxr-xr-x 1 root root 35720 Sep 22 2020 pwd - lrwxrwxrwx 1 root root 4 Jun 21 14:26 rbash -> bash - -rwxr-xr-x 1 root root 43872 Sep 22 2020 readlink - -rwxr-xr-x 1 root root 68592 Sep 22 2020 rm - -rwxr-xr-x 1 root root 43880 Sep 22 2020 rmdir - -rwxr-xr-x 1 root root 19208 Sep 27 2020 run-parts - -rwxr-xr-x 1 root root 114016 Dec 22 2018 sed - lrwxrwxrwx 1 root root 4 Jul 7 21:24 sh -> dash - -rwxr-xr-x 1 root root 35656 Sep 22 2020 sleep - -rwxr-xr-x 1 root root 72640 Sep 22 2020 stty - -rwsr-xr-x 1 root root 67776 Feb 7 02:38 su - -rwxr-xr-x 1 root root 35672 Sep 22 2020 sync - -rwxr-xr-x 1 root root 535768 Feb 16 21:55 tar - -rwxr-xr-x 1 root root 10568 Sep 27 2020 tempfile - -rwxr-xr-x 1 root root 89120 Sep 22 2020 touch - -rwxr-xr-x 1 root root 31512 Sep 22 2020 true - -rwxr-xr-x 1 root root 14264 Feb 26 04:12 ulockmgr_server - -rwsr-xr-x 1 root root 30880 Feb 7 02:38 umount - -rwxr-xr-x 1 root root 35640 Sep 22 2020 uname - -rwxr-xr-x 2 root root 2346 Mar 2 11:30 uncompress - -rwxr-xr-x 1 root root 143088 Sep 22 2020 vdir - -rwxr-xr-x 1 root root 59584 Feb 7 02:38 wdctl - lrwxrwxrwx 1 root root 8 Nov 6 2019 ypdomainname -> hostname - -rwxr-xr-x 1 root root 1984 Mar 2 11:30 zcat - -rwxr-xr-x 1 root root 1678 Mar 2 11:30 zcmp - -rwxr-xr-x 1 root root 5880 Mar 2 11:30 zdiff - -rwxr-xr-x 1 root root 29 Mar 2 11:30 zegrep - -rwxr-xr-x 1 root root 29 Mar 2 11:30 zfgrep - -rwxr-xr-x 1 root root 2081 Mar 2 11:30 zforce - -rwxr-xr-x 1 root root 7585 Mar 2 11:30 zgrep - -rwxr-xr-x 1 root root 2206 Mar 2 11:30 zless - -rwxr-xr-x 1 root root 1842 Mar 2 11:30 zmore - -rwxr-xr-x 1 root root 4553 Mar 2 11:30 znew -I: user script /srv/workspace/pbuilder/15724/tmp/hooks/D02_print_environment finished + -rwxr-xr-x 1 root root 1282512 Jun 22 2021 bash + -rwxr-xr-x 3 root root 34808 Jul 21 2020 bunzip2 + -rwxr-xr-x 3 root root 34808 Jul 21 2020 bzcat + lrwxrwxrwx 1 root root 6 Jul 21 2020 bzcmp -> bzdiff + -rwxr-xr-x 1 root root 2225 Jul 21 2020 bzdiff + lrwxrwxrwx 1 root root 6 Jul 21 2020 bzegrep -> bzgrep + -rwxr-xr-x 1 root root 4877 Sep 5 2019 bzexe + lrwxrwxrwx 1 root root 6 Jul 21 2020 bzfgrep -> bzgrep + -rwxr-xr-x 1 root root 3775 Jul 21 2020 bzgrep + -rwxr-xr-x 3 root root 34808 Jul 21 2020 bzip2 + -rwxr-xr-x 1 root root 14264 Jul 21 2020 bzip2recover + lrwxrwxrwx 1 root root 6 Jul 21 2020 bzless -> bzmore + -rwxr-xr-x 1 root root 1297 Jul 21 2020 bzmore + -rwxr-xr-x 1 root root 39832 Sep 23 2020 cat + -rwxr-xr-x 1 root root 64512 Sep 23 2020 chgrp + -rwxr-xr-x 1 root root 60368 Sep 23 2020 chmod + -rwxr-xr-x 1 root root 64528 Sep 23 2020 chown + -rwxr-xr-x 1 root root 138896 Sep 23 2020 cp + -rwxr-xr-x 1 root root 129544 Dec 11 2020 dash + -rwxr-xr-x 1 root root 101384 Sep 23 2020 date + -rwxr-xr-x 1 root root 80984 Sep 23 2020 dd + -rwxr-xr-x 1 root root 89824 Sep 23 2020 df + -rwxr-xr-x 1 root root 143088 Sep 23 2020 dir + -rwxr-xr-x 1 root root 76152 Feb 8 2021 dmesg + lrwxrwxrwx 1 root root 8 Nov 8 2019 dnsdomainname -> hostname + lrwxrwxrwx 1 root root 8 Nov 8 2019 domainname -> hostname + -rwxr-xr-x 1 root root 35632 Sep 23 2020 echo + -rwxr-xr-x 1 root root 28 Nov 10 2020 egrep + -rwxr-xr-x 1 root root 31512 Sep 23 2020 false + -rwxr-xr-x 1 root root 28 Nov 10 2020 fgrep + -rwxr-xr-x 1 root root 64856 Feb 8 2021 findmnt + -rwsr-xr-x 1 root root 34824 Feb 27 2021 fusermount + -rwxr-xr-x 1 root root 178400 Nov 10 2020 grep + -rwxr-xr-x 2 root root 2346 Mar 3 2021 gunzip + -rwxr-xr-x 1 root root 6376 Mar 3 2021 gzexe + -rwxr-xr-x 1 root root 93744 Mar 3 2021 gzip + -rwxr-xr-x 1 root root 18440 Nov 8 2019 hostname + -rwxr-xr-x 1 root root 68720 Sep 23 2020 ln + -rwxr-xr-x 1 root root 52720 Feb 8 2020 login + -rwxr-xr-x 1 root root 143088 Sep 23 2020 ls + -rwxr-xr-x 1 root root 161960 Feb 8 2021 lsblk + -rwxr-xr-x 1 root root 85200 Sep 23 2020 mkdir + -rwxr-xr-x 1 root root 68744 Sep 23 2020 mknod + -rwxr-xr-x 1 root root 43976 Sep 23 2020 mktemp + -rwxr-xr-x 1 root root 51368 Feb 8 2021 more + -rwsr-xr-x 1 root root 51360 Feb 8 2021 mount + -rwxr-xr-x 1 root root 14496 Feb 8 2021 mountpoint + -rwxr-xr-x 1 root root 134808 Sep 23 2020 mv + lrwxrwxrwx 1 root root 8 Nov 8 2019 nisdomainname -> hostname + lrwxrwxrwx 1 root root 14 Apr 19 2021 pidof -> /sbin/killall5 + -rwxr-xr-x 1 root root 35720 Sep 23 2020 pwd + lrwxrwxrwx 1 root root 4 Jun 22 2021 rbash -> bash + -rwxr-xr-x 1 root root 43872 Sep 23 2020 readlink + -rwxr-xr-x 1 root root 68592 Sep 23 2020 rm + -rwxr-xr-x 1 root root 43880 Sep 23 2020 rmdir + -rwxr-xr-x 1 root root 19208 Sep 28 2020 run-parts + -rwxr-xr-x 1 root root 114016 Dec 23 2018 sed + lrwxrwxrwx 1 root root 4 Aug 12 21:28 sh -> bash + lrwxrwxrwx 1 root root 4 Aug 11 05:47 sh.distrib -> dash + -rwxr-xr-x 1 root root 35656 Sep 23 2020 sleep + -rwxr-xr-x 1 root root 72640 Sep 23 2020 stty + -rwsr-xr-x 1 root root 67776 Feb 8 2021 su + -rwxr-xr-x 1 root root 35672 Sep 23 2020 sync + -rwxr-xr-x 1 root root 535768 Feb 17 2021 tar + -rwxr-xr-x 1 root root 10568 Sep 28 2020 tempfile + -rwxr-xr-x 1 root root 89120 Sep 23 2020 touch + -rwxr-xr-x 1 root root 31512 Sep 23 2020 true + -rwxr-xr-x 1 root root 14264 Feb 27 2021 ulockmgr_server + -rwsr-xr-x 1 root root 30880 Feb 8 2021 umount + -rwxr-xr-x 1 root root 35640 Sep 23 2020 uname + -rwxr-xr-x 2 root root 2346 Mar 3 2021 uncompress + -rwxr-xr-x 1 root root 143088 Sep 23 2020 vdir + -rwxr-xr-x 1 root root 59584 Feb 8 2021 wdctl + lrwxrwxrwx 1 root root 8 Nov 8 2019 ypdomainname -> hostname + -rwxr-xr-x 1 root root 1984 Mar 3 2021 zcat + -rwxr-xr-x 1 root root 1678 Mar 3 2021 zcmp + -rwxr-xr-x 1 root root 5880 Mar 3 2021 zdiff + -rwxr-xr-x 1 root root 29 Mar 3 2021 zegrep + -rwxr-xr-x 1 root root 29 Mar 3 2021 zfgrep + -rwxr-xr-x 1 root root 2081 Mar 3 2021 zforce + -rwxr-xr-x 1 root root 7585 Mar 3 2021 zgrep + -rwxr-xr-x 1 root root 2206 Mar 3 2021 zless + -rwxr-xr-x 1 root root 1842 Mar 3 2021 zmore + -rwxr-xr-x 1 root root 4553 Mar 3 2021 znew +I: user script /srv/workspace/pbuilder/22862/tmp/hooks/D02_print_environment finished -> Attempting to satisfy build-dependencies -> Creating pbuilder-satisfydepends-dummy package Package: pbuilder-satisfydepends-dummy @@ -319,7 +353,7 @@ Get: 111 http://deb.debian.org/debian bullseye/main arm64 golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-1 [32.1 kB] Get: 112 http://deb.debian.org/debian bullseye/main arm64 golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB] Get: 113 http://deb.debian.org/debian bullseye/main arm64 golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB] -Fetched 91.1 MB in 3s (36.4 MB/s) +Fetched 91.1 MB in 2s (49.6 MB/s) debconf: delaying package configuration, since apt-utils is not installed Selecting previously unselected package bsdextrautils. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 19646 files and directories currently installed.) @@ -790,9 +824,42 @@ Building dependency tree... Reading state information... fakeroot is already the newest version (1.25.3-1.1). -0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. +The following additional packages will be installed: + libfile-find-rule-perl libnumber-compare-perl libtext-glob-perl +The following NEW packages will be installed: + libfile-find-rule-perl libnumber-compare-perl libtext-glob-perl usrmerge +0 upgraded, 4 newly installed, 0 to remove and 0 not upgraded. +Need to get 59.5 kB of archives. +After this operation, 157 kB of additional disk space will be used. +Get:1 http://deb.debian.org/debian bullseye/main arm64 libnumber-compare-perl all 0.03-1.1 [6956 B] +Get:2 http://deb.debian.org/debian bullseye/main arm64 libtext-glob-perl all 0.11-1 [8888 B] +Get:3 http://deb.debian.org/debian bullseye/main arm64 libfile-find-rule-perl all 0.34-1 [30.6 kB] +Get:4 http://deb.debian.org/debian bullseye/main arm64 usrmerge all 25 [13.0 kB] +debconf: delaying package configuration, since apt-utils is not installed +Fetched 59.5 kB in 3s (17.3 kB/s) +Selecting previously unselected package libnumber-compare-perl. +(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 39811 files and directories currently installed.) +Preparing to unpack .../libnumber-compare-perl_0.03-1.1_all.deb ... +Unpacking libnumber-compare-perl (0.03-1.1) ... +Selecting previously unselected package libtext-glob-perl. +Preparing to unpack .../libtext-glob-perl_0.11-1_all.deb ... +Unpacking libtext-glob-perl (0.11-1) ... +Selecting previously unselected package libfile-find-rule-perl. +Preparing to unpack .../libfile-find-rule-perl_0.34-1_all.deb ... +Unpacking libfile-find-rule-perl (0.34-1) ... +Selecting previously unselected package usrmerge. +Preparing to unpack .../archives/usrmerge_25_all.deb ... +Unpacking usrmerge (25) ... +Setting up libtext-glob-perl (0.11-1) ... +Setting up libnumber-compare-perl (0.03-1.1) ... +Setting up libfile-find-rule-perl (0.34-1) ... +Setting up usrmerge (25) ... +The system has been successfully converted. +Processing triggers for man-db (2.9.4-2) ... +Not building database; man-db/auto-update is not 'true'. I: Building the package -I: Running cd /build/golang-github-hashicorp-raft-1.1.2/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-genchanges -S > ../golang-github-hashicorp-raft_1.1.2-1_source.changes +hostname: Temporary failure in name resolution +I: Running cd /build/golang-github-hashicorp-raft-1.1.2/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-genchanges -S > ../golang-github-hashicorp-raft_1.1.2-1_source.changes dpkg-buildpackage: info: source package golang-github-hashicorp-raft dpkg-buildpackage: info: source version 1.1.2-1 dpkg-buildpackage: info: source distribution unstable @@ -812,26 +879,26 @@ dh_auto_build -O--buildsystem=golang cd obj-aarch64-linux-gnu && go install -trimpath -v -p 8 github.com/hashicorp/raft github.com/hashicorp/raft/bench internal/cpu +internal/unsafeheader internal/race runtime/internal/sys -internal/unsafeheader -runtime/internal/atomic unicode/utf8 -sync/atomic +runtime/internal/atomic unicode container/list +sync/atomic crypto/internal/subtle runtime/internal/math -internal/bytealg crypto/subtle math/bits -internal/testlog +internal/bytealg encoding +internal/testlog unicode/utf16 internal/nettrace +runtime/cgo vendor/golang.org/x/crypto/cryptobyte/asn1 vendor/golang.org/x/crypto/internal/subtle -runtime/cgo golang.org/x/sys/internal/unsafeheader math runtime @@ -841,28 +908,28 @@ math/rand errors sort -internal/oserror io +internal/oserror +strconv github.com/hashicorp/golang-lru/simplelru vendor/golang.org/x/net/dns/dnsmessage -strconv syscall bytes -hash strings +hash crypto/internal/randutil hash/crc32 -crypto/hmac hash/crc64 +crypto/hmac vendor/golang.org/x/crypto/hkdf crypto -crypto/rc4 reflect +crypto/rc4 vendor/golang.org/x/text/transform github.com/hashicorp/go-immutable-radix +path bufio regexp/syntax -path html internal/syscall/execenv internal/syscall/unix @@ -870,66 +937,66 @@ regexp context internal/poll -encoding/binary internal/fmtsort +encoding/binary os -crypto/sha512 crypto/md5 -crypto/cipher -crypto/sha1 encoding/base64 +crypto/sha1 crypto/ed25519/internal/edwards25519 +crypto/sha512 crypto/sha256 +crypto/cipher encoding/pem vendor/golang.org/x/crypto/poly1305 golang.org/x/sys/unix crypto/des crypto/aes vendor/golang.org/x/crypto/chacha20 -path/filepath fmt -net +path/filepath os/signal runtime/debug +net vendor/golang.org/x/crypto/chacha20poly1305 io/ioutil +encoding/json log encoding/hex -encoding/json compress/flate -net/url math/big +net/url vendor/golang.org/x/crypto/curve25519 vendor/golang.org/x/text/unicode/bidi -vendor/golang.org/x/net/http2/hpack vendor/golang.org/x/text/unicode/norm +vendor/golang.org/x/net/http2/hpack compress/gzip mime -vendor/golang.org/x/text/secure/bidirule mime/quotedprintable +vendor/golang.org/x/text/secure/bidirule net/http/internal github.com/mattn/go-isatty encoding/gob -github.com/mattn/go-colorable go/token -text/template/parse +github.com/mattn/go-colorable github.com/fatih/color +text/template/parse flag runtime/trace github.com/hashicorp/go-hclog -crypto/rand +vendor/golang.org/x/net/idna +testing crypto/elliptic +crypto/rand encoding/asn1 crypto/dsa -vendor/golang.org/x/net/idna crypto/ed25519 -crypto/rsa -testing text/template -crypto/x509/pkix +crypto/rsa vendor/golang.org/x/crypto/cryptobyte -crypto/ecdsa +crypto/x509/pkix html/template +crypto/ecdsa vendor/golang.org/x/net/http/httpproxy net/textproto crypto/x509 @@ -986,32 +1053,32 @@ === RUN TestFileSnapshotSinkImpl --- PASS: TestFileSnapshotSinkImpl (0.00s) === RUN TestFileSS_CreateSnapshotMissingParentDir - testing.go:153: 2021-07-10T01:04:54.617Z [INFO] creating new snapshot: path=/tmp/raft504193843/raft905351414/snapshots/3-10-1625879094617.tmp + testing.go:153: 2022-08-12T07:29:59.049Z [INFO] creating new snapshot: path=/tmp/raft307686903/raft180908778/snapshots/3-10-1660289399049.tmp --- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.00s) === RUN TestFileSS_CreateSnapshot - testing.go:153: 2021-07-10T01:04:54.619Z [INFO] creating new snapshot: path=/tmp/raft606177757/snapshots/3-10-1625879094619.tmp - testing.go:153: 2021-07-10T01:04:54.619Z [WARN] found temporary snapshot: name=3-10-1625879094619.tmp + testing.go:153: 2022-08-12T07:29:59.050Z [INFO] creating new snapshot: path=/tmp/raft745514817/snapshots/3-10-1660289399050.tmp + testing.go:153: 2022-08-12T07:29:59.051Z [WARN] found temporary snapshot: name=3-10-1660289399050.tmp --- PASS: TestFileSS_CreateSnapshot (0.00s) === RUN TestFileSS_CancelSnapshot - testing.go:153: 2021-07-10T01:04:54.620Z [INFO] creating new snapshot: path=/tmp/raft392395160/snapshots/3-10-1625879094620.tmp + testing.go:153: 2022-08-12T07:29:59.051Z [INFO] creating new snapshot: path=/tmp/raft552241580/snapshots/3-10-1660289399051.tmp --- PASS: TestFileSS_CancelSnapshot (0.00s) === RUN TestFileSS_Retention - testing.go:153: 2021-07-10T01:04:54.621Z [INFO] creating new snapshot: path=/tmp/raft545531927/snapshots/3-10-1625879094621.tmp - testing.go:153: 2021-07-10T01:04:54.621Z [INFO] creating new snapshot: path=/tmp/raft545531927/snapshots/3-11-1625879094621.tmp - testing.go:153: 2021-07-10T01:04:54.622Z [INFO] creating new snapshot: path=/tmp/raft545531927/snapshots/3-12-1625879094622.tmp - testing.go:153: 2021-07-10T01:04:54.622Z [INFO] reaping snapshot: path=/tmp/raft545531927/snapshots/3-10-1625879094621 - testing.go:153: 2021-07-10T01:04:54.622Z [INFO] creating new snapshot: path=/tmp/raft545531927/snapshots/3-13-1625879094622.tmp - testing.go:153: 2021-07-10T01:04:54.623Z [INFO] reaping snapshot: path=/tmp/raft545531927/snapshots/3-11-1625879094621 - testing.go:153: 2021-07-10T01:04:54.623Z [INFO] creating new snapshot: path=/tmp/raft545531927/snapshots/3-14-1625879094623.tmp - testing.go:153: 2021-07-10T01:04:54.624Z [INFO] reaping snapshot: path=/tmp/raft545531927/snapshots/3-12-1625879094622 + testing.go:153: 2022-08-12T07:29:59.052Z [INFO] creating new snapshot: path=/tmp/raft813263131/snapshots/3-10-1660289399052.tmp + testing.go:153: 2022-08-12T07:29:59.053Z [INFO] creating new snapshot: path=/tmp/raft813263131/snapshots/3-11-1660289399053.tmp + testing.go:153: 2022-08-12T07:29:59.053Z [INFO] creating new snapshot: path=/tmp/raft813263131/snapshots/3-12-1660289399053.tmp + testing.go:153: 2022-08-12T07:29:59.054Z [INFO] reaping snapshot: path=/tmp/raft813263131/snapshots/3-10-1660289399052 + testing.go:153: 2022-08-12T07:29:59.054Z [INFO] creating new snapshot: path=/tmp/raft813263131/snapshots/3-13-1660289399054.tmp + testing.go:153: 2022-08-12T07:29:59.054Z [INFO] reaping snapshot: path=/tmp/raft813263131/snapshots/3-11-1660289399053 + testing.go:153: 2022-08-12T07:29:59.055Z [INFO] creating new snapshot: path=/tmp/raft813263131/snapshots/3-14-1660289399055.tmp + testing.go:153: 2022-08-12T07:29:59.055Z [INFO] reaping snapshot: path=/tmp/raft813263131/snapshots/3-12-1660289399053 --- PASS: TestFileSS_Retention (0.00s) === RUN TestFileSS_BadPerm --- PASS: TestFileSS_BadPerm (0.00s) === RUN TestFileSS_MissingParentDir --- PASS: TestFileSS_MissingParentDir (0.00s) === RUN TestFileSS_Ordering - testing.go:153: 2021-07-10T01:04:54.625Z [INFO] creating new snapshot: path=/tmp/raft138940254/snapshots/5-130350-1625879094625.tmp - testing.go:153: 2021-07-10T01:04:54.626Z [INFO] creating new snapshot: path=/tmp/raft138940254/snapshots/36-204917-1625879094626.tmp + testing.go:153: 2022-08-12T07:29:59.057Z [INFO] creating new snapshot: path=/tmp/raft793732562/snapshots/5-130350-1660289399057.tmp + testing.go:153: 2022-08-12T07:29:59.057Z [INFO] creating new snapshot: path=/tmp/raft793732562/snapshots/36-204917-1660289399057.tmp --- PASS: TestFileSS_Ordering (0.00s) === RUN TestDeferFutureSuccess --- PASS: TestDeferFutureSuccess (0.00s) @@ -1059,27 +1126,27 @@ === RUN TestNetworkTransport_PooledConn --- PASS: TestNetworkTransport_PooledConn (0.00s) === RUN TestNetworkTransport_ListenBackoff - net_transport_test.go:718: countingWriter: 2021-07-10T01:04:54.651Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2022-08-12T07:29:59.082Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2021-07-10T01:04:54.656Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2022-08-12T07:29:59.087Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2021-07-10T01:04:54.666Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2022-08-12T07:29:59.097Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2021-07-10T01:04:54.687Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2022-08-12T07:29:59.118Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2021-07-10T01:04:54.727Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2022-08-12T07:29:59.158Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2021-07-10T01:04:54.807Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2022-08-12T07:29:59.238Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2021-07-10T01:04:54.968Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2022-08-12T07:29:59.399Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2021-07-10T01:04:55.288Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2022-08-12T07:29:59.719Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2021-07-10T01:04:55.928Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2022-08-12T07:30:00.359Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2021-07-10T01:04:56.928Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2022-08-12T07:30:01.360Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:718: countingWriter: 2021-07-10T01:04:57.929Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:718: countingWriter: 2022-08-12T07:30:02.360Z [ERROR] test: failed to accept connection: error="intentional error in test" --- PASS: TestNetworkTransport_ListenBackoff (4.00s) === RUN TestPeersJSON_BadConfiguration @@ -1089,2821 +1156,2885 @@ === RUN TestPeersJSON_ReadConfigJSON --- PASS: TestPeersJSON_ReadConfigJSON (0.00s) === RUN TestRaft_StartStop -testLoggerAdapter verbose: server-bb99c6b6-04d1-6795-0247-6e3c6bfa105b: 2021-07-10T01:04:58.653Z [INFO] server-bb99c6b6-04d1-6795-0247-6e3c6bfa105b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bb99c6b6-04d1-6795-0247-6e3c6bfa105b Address:bb99c6b6-04d1-6795-0247-6e3c6bfa105b}]" - testing.go:149: server-bb99c6b6-04d1-6795-0247-6e3c6bfa105b: 2021-07-10T01:04:58.653Z [INFO] server-bb99c6b6-04d1-6795-0247-6e3c6bfa105b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bb99c6b6-04d1-6795-0247-6e3c6bfa105b Address:bb99c6b6-04d1-6795-0247-6e3c6bfa105b}]" +testLoggerAdapter verbose: server-4c865689-a8e0-b161-64a0-d040ce325a9a: 2022-08-12T07:30:03.084Z [INFO] server-4c865689-a8e0-b161-64a0-d040ce325a9a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4c865689-a8e0-b161-64a0-d040ce325a9a Address:4c865689-a8e0-b161-64a0-d040ce325a9a}]" + testing.go:149: server-4c865689-a8e0-b161-64a0-d040ce325a9a: 2022-08-12T07:30:03.084Z [INFO] server-4c865689-a8e0-b161-64a0-d040ce325a9a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4c865689-a8e0-b161-64a0-d040ce325a9a Address:4c865689-a8e0-b161-64a0-d040ce325a9a}]" --- PASS: TestRaft_StartStop (0.00s) === RUN TestRaft_AfterShutdown -testLoggerAdapter verbose: server-25a1494d-f4ca-4ea3-c55d-acac012f35b4: 2021-07-10T01:04:58.655Z [INFO] server-25a1494d-f4ca-4ea3-c55d-acac012f35b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25a1494d-f4ca-4ea3-c55d-acac012f35b4 Address:25a1494d-f4ca-4ea3-c55d-acac012f35b4}]" - testing.go:149: server-25a1494d-f4ca-4ea3-c55d-acac012f35b4: 2021-07-10T01:04:58.655Z [INFO] server-25a1494d-f4ca-4ea3-c55d-acac012f35b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25a1494d-f4ca-4ea3-c55d-acac012f35b4 Address:25a1494d-f4ca-4ea3-c55d-acac012f35b4}]" +testLoggerAdapter verbose: server-f824749a-4be5-4872-e1f6-0ce96e4b5137: 2022-08-12T07:30:03.085Z [INFO] server-f824749a-4be5-4872-e1f6-0ce96e4b5137: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f824749a-4be5-4872-e1f6-0ce96e4b5137 Address:f824749a-4be5-4872-e1f6-0ce96e4b5137}]" + testing.go:149: server-f824749a-4be5-4872-e1f6-0ce96e4b5137: 2022-08-12T07:30:03.085Z [INFO] server-f824749a-4be5-4872-e1f6-0ce96e4b5137: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f824749a-4be5-4872-e1f6-0ce96e4b5137 Address:f824749a-4be5-4872-e1f6-0ce96e4b5137}]" --- PASS: TestRaft_AfterShutdown (0.00s) === RUN TestRaft_LiveBootstrap -testLoggerAdapter verbose: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.656Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: initial configuration: index=0 servers=[] - testing.go:149: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.656Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: server-ca294112-77c7-33bf-2167-2c51be1c818f: 2021-07-10T01:04:58.656Z [INFO] server-ca294112-77c7-33bf-2167-2c51be1c818f: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.656Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: entering follower state: follower="Node at 3bc7bf7d-e3f3-b648-cf09-5af67d9cf607 [Follower]" leader= - testing.go:149: server-ca294112-77c7-33bf-2167-2c51be1c818f: 2021-07-10T01:04:58.656Z [INFO] server-ca294112-77c7-33bf-2167-2c51be1c818f: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5: 2021-07-10T01:04:58.657Z [INFO] server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5: initial configuration: index=0 servers=[] - testing.go:149: server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5: 2021-07-10T01:04:58.657Z [INFO] server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5: initial configuration: index=0 servers=[] - testing.go:149: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.656Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: entering follower state: follower="Node at 3bc7bf7d-e3f3-b648-cf09-5af67d9cf607 [Follower]" leader= -testLoggerAdapter verbose: cluster: 2021-07-10T01:04:58.657Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-07-10T01:04:58.657Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-ca294112-77c7-33bf-2167-2c51be1c818f: 2021-07-10T01:04:58.660Z [INFO] server-ca294112-77c7-33bf-2167-2c51be1c818f: entering follower state: follower="Node at ca294112-77c7-33bf-2167-2c51be1c818f [Follower]" leader= - testing.go:149: server-ca294112-77c7-33bf-2167-2c51be1c818f: 2021-07-10T01:04:58.660Z [INFO] server-ca294112-77c7-33bf-2167-2c51be1c818f: entering follower state: follower="Node at ca294112-77c7-33bf-2167-2c51be1c818f [Follower]" leader= -testLoggerAdapter verbose: server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5: 2021-07-10T01:04:58.660Z [INFO] server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5: entering follower state: follower="Node at 8b7f9582-6d45-2c61-320b-9123cdf5dfa5 [Follower]" leader= - testing.go:149: server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5: 2021-07-10T01:04:58.660Z [INFO] server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5: entering follower state: follower="Node at 8b7f9582-6d45-2c61-320b-9123cdf5dfa5 [Follower]" leader= -testLoggerAdapter verbose: server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5: 2021-07-10T01:04:58.733Z [WARN] server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5: no known peers, aborting election - testing.go:149: server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5: 2021-07-10T01:04:58.733Z [WARN] server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5: no known peers, aborting election -testLoggerAdapter verbose: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.737Z [WARN] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.737Z [WARN] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.737Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: entering candidate state: node="Node at 3bc7bf7d-e3f3-b648-cf09-5af67d9cf607 [Candidate]" term=2 - testing.go:149: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.737Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: entering candidate state: node="Node at 3bc7bf7d-e3f3-b648-cf09-5af67d9cf607 [Candidate]" term=2 -testLoggerAdapter verbose: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.737Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: election won: tally=2 - testing.go:149: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.737Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: election won: tally=2 -testLoggerAdapter verbose: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.737Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: entering leader state: leader="Node at 3bc7bf7d-e3f3-b648-cf09-5af67d9cf607 [Leader]" - testing.go:149: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.737Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: entering leader state: leader="Node at 3bc7bf7d-e3f3-b648-cf09-5af67d9cf607 [Leader]" -testLoggerAdapter verbose: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.737Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: added peer, starting replication: peer=server-ca294112-77c7-33bf-2167-2c51be1c818f - testing.go:149: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.737Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: added peer, starting replication: peer=server-ca294112-77c7-33bf-2167-2c51be1c818f -testLoggerAdapter verbose: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.738Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: added peer, starting replication: peer=server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5 - testing.go:149: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.738Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: added peer, starting replication: peer=server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5 -testLoggerAdapter verbose: server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5: 2021-07-10T01:04:58.738Z [WARN] server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5: failed to get previous log: previous-index=1 last-index=0 error="log not found" - testing.go:149: server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5: 2021-07-10T01:04:58.738Z [WARN] server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5: failed to get previous log: previous-index=1 last-index=0 error="log not found" -testLoggerAdapter verbose: server-ca294112-77c7-33bf-2167-2c51be1c818f: 2021-07-10T01:04:58.738Z [WARN] server-ca294112-77c7-33bf-2167-2c51be1c818f: failed to get previous log: previous-index=1 last-index=0 error="log not found" - testing.go:149: server-ca294112-77c7-33bf-2167-2c51be1c818f: 2021-07-10T01:04:58.738Z [WARN] server-ca294112-77c7-33bf-2167-2c51be1c818f: failed to get previous log: previous-index=1 last-index=0 error="log not found" -testLoggerAdapter verbose: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.738Z [WARN] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: appendEntries rejected, sending older logs: peer="{Voter server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5 8b7f9582-6d45-2c61-320b-9123cdf5dfa5}" next=1 - testing.go:149: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.738Z [WARN] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: appendEntries rejected, sending older logs: peer="{Voter server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5 8b7f9582-6d45-2c61-320b-9123cdf5dfa5}" next=1 -testLoggerAdapter verbose: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.738Z [WARN] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: appendEntries rejected, sending older logs: peer="{Voter server-ca294112-77c7-33bf-2167-2c51be1c818f ca294112-77c7-33bf-2167-2c51be1c818f}" next=1 - testing.go:149: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.738Z [WARN] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: appendEntries rejected, sending older logs: peer="{Voter server-ca294112-77c7-33bf-2167-2c51be1c818f ca294112-77c7-33bf-2167-2c51be1c818f}" next=1 -testLoggerAdapter verbose: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.738Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: pipelining replication: peer="{Voter server-ca294112-77c7-33bf-2167-2c51be1c818f ca294112-77c7-33bf-2167-2c51be1c818f}" - testing.go:149: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.738Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: pipelining replication: peer="{Voter server-ca294112-77c7-33bf-2167-2c51be1c818f ca294112-77c7-33bf-2167-2c51be1c818f}" -testLoggerAdapter verbose: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.738Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: pipelining replication: peer="{Voter server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5 8b7f9582-6d45-2c61-320b-9123cdf5dfa5}" - testing.go:149: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.738Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: pipelining replication: peer="{Voter server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5 8b7f9582-6d45-2c61-320b-9123cdf5dfa5}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:04:58.843Z [INFO] stable state for Follower reached at 2021-07-10 01:04:58.738227512 +0000 UTC m=+4.126385522 (2 nodes), 80.466228ms from start of poll, 81.749342ms from cluster start. Timeout at 2021-07-10 01:04:58.84329306 +0000 UTC m=+4.231451030, 105.065508ms after stability - testing.go:149: cluster: 2021-07-10T01:04:58.843Z [INFO] stable state for Follower reached at 2021-07-10 01:04:58.738227512 +0000 UTC m=+4.126385522 (2 nodes), 80.466228ms from start of poll, 81.749342ms from cluster start. Timeout at 2021-07-10 01:04:58.84329306 +0000 UTC m=+4.231451030, 105.065508ms after stability -testLoggerAdapter verbose: cluster: 2021-07-10T01:04:58.843Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:04:58.843Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-07-10T01:04:58.948Z [INFO] stable state for Leader reached at 2021-07-10 01:04:58.843585883 +0000 UTC m=+4.231743853 (1 nodes), 2.98µs from start of poll, 187.107673ms from cluster start. Timeout at 2021-07-10 01:04:58.948650731 +0000 UTC m=+4.336808701, 105.064848ms after stability - testing.go:149: cluster: 2021-07-10T01:04:58.948Z [INFO] stable state for Leader reached at 2021-07-10 01:04:58.843585883 +0000 UTC m=+4.231743853 (1 nodes), 2.98µs from start of poll, 187.107673ms from cluster start. Timeout at 2021-07-10 01:04:58.948650731 +0000 UTC m=+4.336808701, 105.064848ms after stability -testLoggerAdapter verbose: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.959Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: aborting pipeline replication: peer="{Voter server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5 8b7f9582-6d45-2c61-320b-9123cdf5dfa5}" - testing.go:149: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.959Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: aborting pipeline replication: peer="{Voter server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5 8b7f9582-6d45-2c61-320b-9123cdf5dfa5}" -testLoggerAdapter verbose: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.959Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: aborting pipeline replication: peer="{Voter server-ca294112-77c7-33bf-2167-2c51be1c818f ca294112-77c7-33bf-2167-2c51be1c818f}" - testing.go:149: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.959Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: aborting pipeline replication: peer="{Voter server-ca294112-77c7-33bf-2167-2c51be1c818f ca294112-77c7-33bf-2167-2c51be1c818f}" +testLoggerAdapter verbose: server-781eb135-4af0-967e-7c8c-92643ed79b14: 2022-08-12T07:30:03.089Z [INFO] server-781eb135-4af0-967e-7c8c-92643ed79b14: initial configuration: index=0 servers=[] + testing.go:149: server-781eb135-4af0-967e-7c8c-92643ed79b14: 2022-08-12T07:30:03.089Z [INFO] server-781eb135-4af0-967e-7c8c-92643ed79b14: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: server-df846254-9e57-05da-5a4e-fa5dfb22883b: 2022-08-12T07:30:03.089Z [INFO] server-df846254-9e57-05da-5a4e-fa5dfb22883b: initial configuration: index=0 servers=[] + testing.go:149: server-df846254-9e57-05da-5a4e-fa5dfb22883b: 2022-08-12T07:30:03.089Z [INFO] server-df846254-9e57-05da-5a4e-fa5dfb22883b: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: server-781eb135-4af0-967e-7c8c-92643ed79b14: 2022-08-12T07:30:03.089Z [INFO] server-781eb135-4af0-967e-7c8c-92643ed79b14: entering follower state: follower="Node at 781eb135-4af0-967e-7c8c-92643ed79b14 [Follower]" leader= + testing.go:149: server-781eb135-4af0-967e-7c8c-92643ed79b14: 2022-08-12T07:30:03.089Z [INFO] server-781eb135-4af0-967e-7c8c-92643ed79b14: entering follower state: follower="Node at 781eb135-4af0-967e-7c8c-92643ed79b14 [Follower]" leader= +testLoggerAdapter verbose: server-4567c6b0-2774-879b-ac9e-d8719989c622: 2022-08-12T07:30:03.089Z [INFO] server-4567c6b0-2774-879b-ac9e-d8719989c622: initial configuration: index=0 servers=[] + testing.go:149: server-4567c6b0-2774-879b-ac9e-d8719989c622: 2022-08-12T07:30:03.089Z [INFO] server-4567c6b0-2774-879b-ac9e-d8719989c622: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: server-df846254-9e57-05da-5a4e-fa5dfb22883b: 2022-08-12T07:30:03.089Z [INFO] server-df846254-9e57-05da-5a4e-fa5dfb22883b: entering follower state: follower="Node at df846254-9e57-05da-5a4e-fa5dfb22883b [Follower]" leader= + testing.go:149: server-df846254-9e57-05da-5a4e-fa5dfb22883b: 2022-08-12T07:30:03.089Z [INFO] server-df846254-9e57-05da-5a4e-fa5dfb22883b: entering follower state: follower="Node at df846254-9e57-05da-5a4e-fa5dfb22883b [Follower]" leader= +testLoggerAdapter verbose: server-4567c6b0-2774-879b-ac9e-d8719989c622: 2022-08-12T07:30:03.089Z [INFO] server-4567c6b0-2774-879b-ac9e-d8719989c622: entering follower state: follower="Node at 4567c6b0-2774-879b-ac9e-d8719989c622 [Follower]" leader= + testing.go:149: server-4567c6b0-2774-879b-ac9e-d8719989c622: 2022-08-12T07:30:03.089Z [INFO] server-4567c6b0-2774-879b-ac9e-d8719989c622: entering follower state: follower="Node at 4567c6b0-2774-879b-ac9e-d8719989c622 [Follower]" leader= +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:03.090Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2022-08-12T07:30:03.090Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-4567c6b0-2774-879b-ac9e-d8719989c622: 2022-08-12T07:30:03.141Z [WARN] server-4567c6b0-2774-879b-ac9e-d8719989c622: no known peers, aborting election + testing.go:149: server-4567c6b0-2774-879b-ac9e-d8719989c622: 2022-08-12T07:30:03.141Z [WARN] server-4567c6b0-2774-879b-ac9e-d8719989c622: no known peers, aborting election +testLoggerAdapter verbose: server-df846254-9e57-05da-5a4e-fa5dfb22883b: 2022-08-12T07:30:03.161Z [WARN] server-df846254-9e57-05da-5a4e-fa5dfb22883b: no known peers, aborting election + testing.go:149: server-df846254-9e57-05da-5a4e-fa5dfb22883b: 2022-08-12T07:30:03.161Z [WARN] server-df846254-9e57-05da-5a4e-fa5dfb22883b: no known peers, aborting election +testLoggerAdapter verbose: server-781eb135-4af0-967e-7c8c-92643ed79b14: 2022-08-12T07:30:03.165Z [WARN] server-781eb135-4af0-967e-7c8c-92643ed79b14: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-781eb135-4af0-967e-7c8c-92643ed79b14: 2022-08-12T07:30:03.165Z [WARN] server-781eb135-4af0-967e-7c8c-92643ed79b14: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-781eb135-4af0-967e-7c8c-92643ed79b14: 2022-08-12T07:30:03.165Z [INFO] server-781eb135-4af0-967e-7c8c-92643ed79b14: entering candidate state: node="Node at 781eb135-4af0-967e-7c8c-92643ed79b14 [Candidate]" term=2 + testing.go:149: server-781eb135-4af0-967e-7c8c-92643ed79b14: 2022-08-12T07:30:03.165Z [INFO] server-781eb135-4af0-967e-7c8c-92643ed79b14: entering candidate state: node="Node at 781eb135-4af0-967e-7c8c-92643ed79b14 [Candidate]" term=2 +testLoggerAdapter verbose: server-781eb135-4af0-967e-7c8c-92643ed79b14: 2022-08-12T07:30:03.166Z [INFO] server-781eb135-4af0-967e-7c8c-92643ed79b14: election won: tally=2 + testing.go:149: server-781eb135-4af0-967e-7c8c-92643ed79b14: 2022-08-12T07:30:03.166Z [INFO] server-781eb135-4af0-967e-7c8c-92643ed79b14: election won: tally=2 +testLoggerAdapter verbose: server-781eb135-4af0-967e-7c8c-92643ed79b14: 2022-08-12T07:30:03.166Z [INFO] server-781eb135-4af0-967e-7c8c-92643ed79b14: entering leader state: leader="Node at 781eb135-4af0-967e-7c8c-92643ed79b14 [Leader]" + testing.go:149: server-781eb135-4af0-967e-7c8c-92643ed79b14: 2022-08-12T07:30:03.166Z [INFO] server-781eb135-4af0-967e-7c8c-92643ed79b14: entering leader state: leader="Node at 781eb135-4af0-967e-7c8c-92643ed79b14 [Leader]" +testLoggerAdapter verbose: server-781eb135-4af0-967e-7c8c-92643ed79b14: 2022-08-12T07:30:03.166Z [INFO] server-781eb135-4af0-967e-7c8c-92643ed79b14: added peer, starting replication: peer=server-df846254-9e57-05da-5a4e-fa5dfb22883b + testing.go:149: server-781eb135-4af0-967e-7c8c-92643ed79b14: 2022-08-12T07:30:03.166Z [INFO] server-781eb135-4af0-967e-7c8c-92643ed79b14: added peer, starting replication: peer=server-df846254-9e57-05da-5a4e-fa5dfb22883b +testLoggerAdapter verbose: server-781eb135-4af0-967e-7c8c-92643ed79b14: 2022-08-12T07:30:03.166Z [INFO] server-781eb135-4af0-967e-7c8c-92643ed79b14: added peer, starting replication: peer=server-4567c6b0-2774-879b-ac9e-d8719989c622 + testing.go:149: server-781eb135-4af0-967e-7c8c-92643ed79b14: 2022-08-12T07:30:03.166Z [INFO] server-781eb135-4af0-967e-7c8c-92643ed79b14: added peer, starting replication: peer=server-4567c6b0-2774-879b-ac9e-d8719989c622 +testLoggerAdapter verbose: server-df846254-9e57-05da-5a4e-fa5dfb22883b: 2022-08-12T07:30:03.166Z [WARN] server-df846254-9e57-05da-5a4e-fa5dfb22883b: failed to get previous log: previous-index=1 last-index=0 error="log not found" + testing.go:149: server-df846254-9e57-05da-5a4e-fa5dfb22883b: 2022-08-12T07:30:03.166Z [WARN] server-df846254-9e57-05da-5a4e-fa5dfb22883b: failed to get previous log: previous-index=1 last-index=0 error="log not found" +testLoggerAdapter verbose: server-4567c6b0-2774-879b-ac9e-d8719989c622: 2022-08-12T07:30:03.166Z [WARN] server-4567c6b0-2774-879b-ac9e-d8719989c622: failed to get previous log: previous-index=1 last-index=0 error="log not found" + testing.go:149: server-4567c6b0-2774-879b-ac9e-d8719989c622: 2022-08-12T07:30:03.166Z [WARN] server-4567c6b0-2774-879b-ac9e-d8719989c622: failed to get previous log: previous-index=1 last-index=0 error="log not found" +testLoggerAdapter verbose: server-781eb135-4af0-967e-7c8c-92643ed79b14: 2022-08-12T07:30:03.166Z [WARN] server-781eb135-4af0-967e-7c8c-92643ed79b14: appendEntries rejected, sending older logs: peer="{Voter server-df846254-9e57-05da-5a4e-fa5dfb22883b df846254-9e57-05da-5a4e-fa5dfb22883b}" next=1 + testing.go:149: server-781eb135-4af0-967e-7c8c-92643ed79b14: 2022-08-12T07:30:03.166Z [WARN] server-781eb135-4af0-967e-7c8c-92643ed79b14: appendEntries rejected, sending older logs: peer="{Voter server-df846254-9e57-05da-5a4e-fa5dfb22883b df846254-9e57-05da-5a4e-fa5dfb22883b}" next=1 +testLoggerAdapter verbose: server-781eb135-4af0-967e-7c8c-92643ed79b14: 2022-08-12T07:30:03.166Z [WARN] server-781eb135-4af0-967e-7c8c-92643ed79b14: appendEntries rejected, sending older logs: peer="{Voter server-4567c6b0-2774-879b-ac9e-d8719989c622 4567c6b0-2774-879b-ac9e-d8719989c622}" next=1 + testing.go:149: server-781eb135-4af0-967e-7c8c-92643ed79b14: 2022-08-12T07:30:03.166Z [WARN] server-781eb135-4af0-967e-7c8c-92643ed79b14: appendEntries rejected, sending older logs: peer="{Voter server-4567c6b0-2774-879b-ac9e-d8719989c622 4567c6b0-2774-879b-ac9e-d8719989c622}" next=1 +testLoggerAdapter verbose: server-781eb135-4af0-967e-7c8c-92643ed79b14: 2022-08-12T07:30:03.166Z [INFO] server-781eb135-4af0-967e-7c8c-92643ed79b14: pipelining replication: peer="{Voter server-4567c6b0-2774-879b-ac9e-d8719989c622 4567c6b0-2774-879b-ac9e-d8719989c622}" + testing.go:149: server-781eb135-4af0-967e-7c8c-92643ed79b14: 2022-08-12T07:30:03.166Z [INFO] server-781eb135-4af0-967e-7c8c-92643ed79b14: pipelining replication: peer="{Voter server-4567c6b0-2774-879b-ac9e-d8719989c622 4567c6b0-2774-879b-ac9e-d8719989c622}" +testLoggerAdapter verbose: server-781eb135-4af0-967e-7c8c-92643ed79b14: 2022-08-12T07:30:03.166Z [INFO] server-781eb135-4af0-967e-7c8c-92643ed79b14: pipelining replication: peer="{Voter server-df846254-9e57-05da-5a4e-fa5dfb22883b df846254-9e57-05da-5a4e-fa5dfb22883b}" + testing.go:149: server-781eb135-4af0-967e-7c8c-92643ed79b14: 2022-08-12T07:30:03.166Z [INFO] server-781eb135-4af0-967e-7c8c-92643ed79b14: pipelining replication: peer="{Voter server-df846254-9e57-05da-5a4e-fa5dfb22883b df846254-9e57-05da-5a4e-fa5dfb22883b}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:03.271Z [INFO] stable state for Follower reached at 2022-08-12 07:30:03.1661892 +0000 UTC m=+4.122875701 (2 nodes), 76.05356ms from start of poll, 76.8023ms from cluster start. Timeout at 2022-08-12 07:30:03.27122746 +0000 UTC m=+4.227913941, 105.03824ms after stability + testing.go:149: cluster: 2022-08-12T07:30:03.271Z [INFO] stable state for Follower reached at 2022-08-12 07:30:03.1661892 +0000 UTC m=+4.122875701 (2 nodes), 76.05356ms from start of poll, 76.8023ms from cluster start. Timeout at 2022-08-12 07:30:03.27122746 +0000 UTC m=+4.227913941, 105.03824ms after stability +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:03.271Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:03.271Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:03.376Z [INFO] stable state for Leader reached at 2022-08-12 07:30:03.27154076 +0000 UTC m=+4.228227241 (1 nodes), 1.84µs from start of poll, 182.15384ms from cluster start. Timeout at 2022-08-12 07:30:03.3766243 +0000 UTC m=+4.333310761, 105.08352ms after stability + testing.go:149: cluster: 2022-08-12T07:30:03.376Z [INFO] stable state for Leader reached at 2022-08-12 07:30:03.27154076 +0000 UTC m=+4.228227241 (1 nodes), 1.84µs from start of poll, 182.15384ms from cluster start. Timeout at 2022-08-12 07:30:03.3766243 +0000 UTC m=+4.333310761, 105.08352ms after stability +testLoggerAdapter verbose: server-781eb135-4af0-967e-7c8c-92643ed79b14: 2022-08-12T07:30:03.387Z [INFO] server-781eb135-4af0-967e-7c8c-92643ed79b14: aborting pipeline replication: peer="{Voter server-4567c6b0-2774-879b-ac9e-d8719989c622 4567c6b0-2774-879b-ac9e-d8719989c622}" + testing.go:149: server-781eb135-4af0-967e-7c8c-92643ed79b14: 2022-08-12T07:30:03.387Z [INFO] server-781eb135-4af0-967e-7c8c-92643ed79b14: aborting pipeline replication: peer="{Voter server-4567c6b0-2774-879b-ac9e-d8719989c622 4567c6b0-2774-879b-ac9e-d8719989c622}" +testLoggerAdapter verbose: server-781eb135-4af0-967e-7c8c-92643ed79b14: 2022-08-12T07:30:03.387Z [INFO] server-781eb135-4af0-967e-7c8c-92643ed79b14: aborting pipeline replication: peer="{Voter server-df846254-9e57-05da-5a4e-fa5dfb22883b df846254-9e57-05da-5a4e-fa5dfb22883b}" + testing.go:149: server-781eb135-4af0-967e-7c8c-92643ed79b14: 2022-08-12T07:30:03.387Z [INFO] server-781eb135-4af0-967e-7c8c-92643ed79b14: aborting pipeline replication: peer="{Voter server-df846254-9e57-05da-5a4e-fa5dfb22883b df846254-9e57-05da-5a4e-fa5dfb22883b}" --- PASS: TestRaft_LiveBootstrap (0.30s) === RUN TestRaft_RecoverCluster_NoState -testLoggerAdapter verbose: server-ebd8bc83-dd72-d521-8a83-974dfd53fff6: 2021-07-10T01:04:58.960Z [INFO] server-ebd8bc83-dd72-d521-8a83-974dfd53fff6: initial configuration: index=0 servers=[] - testing.go:149: server-ebd8bc83-dd72-d521-8a83-974dfd53fff6: 2021-07-10T01:04:58.960Z [INFO] server-ebd8bc83-dd72-d521-8a83-974dfd53fff6: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: server-ebd8bc83-dd72-d521-8a83-974dfd53fff6: 2021-07-10T01:04:58.960Z [INFO] server-ebd8bc83-dd72-d521-8a83-974dfd53fff6: entering follower state: follower="Node at ebd8bc83-dd72-d521-8a83-974dfd53fff6 [Follower]" leader= - testing.go:149: server-ebd8bc83-dd72-d521-8a83-974dfd53fff6: 2021-07-10T01:04:58.960Z [INFO] server-ebd8bc83-dd72-d521-8a83-974dfd53fff6: entering follower state: follower="Node at ebd8bc83-dd72-d521-8a83-974dfd53fff6 [Follower]" leader= +testLoggerAdapter verbose: server-834f2a40-e2ce-820e-f746-dcb1bf23ebc6: 2022-08-12T07:30:03.388Z [INFO] server-834f2a40-e2ce-820e-f746-dcb1bf23ebc6: initial configuration: index=0 servers=[] + testing.go:149: server-834f2a40-e2ce-820e-f746-dcb1bf23ebc6: 2022-08-12T07:30:03.388Z [INFO] server-834f2a40-e2ce-820e-f746-dcb1bf23ebc6: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: server-834f2a40-e2ce-820e-f746-dcb1bf23ebc6: 2022-08-12T07:30:03.388Z [INFO] server-834f2a40-e2ce-820e-f746-dcb1bf23ebc6: entering follower state: follower="Node at 834f2a40-e2ce-820e-f746-dcb1bf23ebc6 [Follower]" leader= + testing.go:149: server-834f2a40-e2ce-820e-f746-dcb1bf23ebc6: 2022-08-12T07:30:03.388Z [INFO] server-834f2a40-e2ce-820e-f746-dcb1bf23ebc6: entering follower state: follower="Node at 834f2a40-e2ce-820e-f746-dcb1bf23ebc6 [Follower]" leader= --- PASS: TestRaft_RecoverCluster_NoState (0.00s) === RUN TestRaft_RecoverCluster raft_test.go:119: Disabled unreliable test --- SKIP: TestRaft_RecoverCluster (0.00s) === RUN TestRaft_HasExistingState -testLoggerAdapter verbose: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:58.961Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4510cfdf-e38a-e76d-5524-8f0140a945fc Address:4510cfdf-e38a-e76d-5524-8f0140a945fc} {Suffrage:Voter ID:server-b07c6965-7364-ac4c-74cc-5d6997bcc545 Address:b07c6965-7364-ac4c-74cc-5d6997bcc545}]" - testing.go:149: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:58.961Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4510cfdf-e38a-e76d-5524-8f0140a945fc Address:4510cfdf-e38a-e76d-5524-8f0140a945fc} {Suffrage:Voter ID:server-b07c6965-7364-ac4c-74cc-5d6997bcc545 Address:b07c6965-7364-ac4c-74cc-5d6997bcc545}]" -testLoggerAdapter verbose: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:58.962Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: entering follower state: follower="Node at 4510cfdf-e38a-e76d-5524-8f0140a945fc [Follower]" leader= - testing.go:149: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:58.962Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: entering follower state: follower="Node at 4510cfdf-e38a-e76d-5524-8f0140a945fc [Follower]" leader= -testLoggerAdapter verbose: server-b07c6965-7364-ac4c-74cc-5d6997bcc545: 2021-07-10T01:04:58.962Z [INFO] server-b07c6965-7364-ac4c-74cc-5d6997bcc545: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4510cfdf-e38a-e76d-5524-8f0140a945fc Address:4510cfdf-e38a-e76d-5524-8f0140a945fc} {Suffrage:Voter ID:server-b07c6965-7364-ac4c-74cc-5d6997bcc545 Address:b07c6965-7364-ac4c-74cc-5d6997bcc545}]" - testing.go:149: server-b07c6965-7364-ac4c-74cc-5d6997bcc545: 2021-07-10T01:04:58.962Z [INFO] server-b07c6965-7364-ac4c-74cc-5d6997bcc545: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4510cfdf-e38a-e76d-5524-8f0140a945fc Address:4510cfdf-e38a-e76d-5524-8f0140a945fc} {Suffrage:Voter ID:server-b07c6965-7364-ac4c-74cc-5d6997bcc545 Address:b07c6965-7364-ac4c-74cc-5d6997bcc545}]" -testLoggerAdapter verbose: server-b07c6965-7364-ac4c-74cc-5d6997bcc545: 2021-07-10T01:04:58.962Z [INFO] server-b07c6965-7364-ac4c-74cc-5d6997bcc545: entering follower state: follower="Node at b07c6965-7364-ac4c-74cc-5d6997bcc545 [Follower]" leader= - testing.go:149: server-b07c6965-7364-ac4c-74cc-5d6997bcc545: 2021-07-10T01:04:58.962Z [INFO] server-b07c6965-7364-ac4c-74cc-5d6997bcc545: entering follower state: follower="Node at b07c6965-7364-ac4c-74cc-5d6997bcc545 [Follower]" leader= -testLoggerAdapter verbose: server-b1b024f2-7c48-1049-2a38-bdbbc303259e: 2021-07-10T01:04:58.962Z [INFO] server-b1b024f2-7c48-1049-2a38-bdbbc303259e: initial configuration: index=0 servers=[] - testing.go:149: server-b1b024f2-7c48-1049-2a38-bdbbc303259e: 2021-07-10T01:04:58.962Z [INFO] server-b1b024f2-7c48-1049-2a38-bdbbc303259e: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: server-b1b024f2-7c48-1049-2a38-bdbbc303259e: 2021-07-10T01:04:58.962Z [INFO] server-b1b024f2-7c48-1049-2a38-bdbbc303259e: entering follower state: follower="Node at b1b024f2-7c48-1049-2a38-bdbbc303259e [Follower]" leader= - testing.go:149: server-b1b024f2-7c48-1049-2a38-bdbbc303259e: 2021-07-10T01:04:58.962Z [INFO] server-b1b024f2-7c48-1049-2a38-bdbbc303259e: entering follower state: follower="Node at b1b024f2-7c48-1049-2a38-bdbbc303259e [Follower]" leader= -testLoggerAdapter verbose: cluster: 2021-07-10T01:04:58.962Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:04:58.962Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-b1b024f2-7c48-1049-2a38-bdbbc303259e: 2021-07-10T01:04:59.022Z [WARN] server-b1b024f2-7c48-1049-2a38-bdbbc303259e: no known peers, aborting election - testing.go:149: server-b1b024f2-7c48-1049-2a38-bdbbc303259e: 2021-07-10T01:04:59.022Z [WARN] server-b1b024f2-7c48-1049-2a38-bdbbc303259e: no known peers, aborting election -testLoggerAdapter verbose: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.023Z [WARN] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.023Z [WARN] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.023Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: entering candidate state: node="Node at 4510cfdf-e38a-e76d-5524-8f0140a945fc [Candidate]" term=2 - testing.go:149: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.023Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: entering candidate state: node="Node at 4510cfdf-e38a-e76d-5524-8f0140a945fc [Candidate]" term=2 -testLoggerAdapter verbose: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.023Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: election won: tally=2 - testing.go:149: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.023Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: election won: tally=2 -testLoggerAdapter verbose: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.023Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: entering leader state: leader="Node at 4510cfdf-e38a-e76d-5524-8f0140a945fc [Leader]" - testing.go:149: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.023Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: entering leader state: leader="Node at 4510cfdf-e38a-e76d-5524-8f0140a945fc [Leader]" -testLoggerAdapter verbose: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.023Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: added peer, starting replication: peer=server-b07c6965-7364-ac4c-74cc-5d6997bcc545 - testing.go:149: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.023Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: added peer, starting replication: peer=server-b07c6965-7364-ac4c-74cc-5d6997bcc545 -testLoggerAdapter verbose: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.023Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: pipelining replication: peer="{Voter server-b07c6965-7364-ac4c-74cc-5d6997bcc545 b07c6965-7364-ac4c-74cc-5d6997bcc545}" - testing.go:149: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.023Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: pipelining replication: peer="{Voter server-b07c6965-7364-ac4c-74cc-5d6997bcc545 b07c6965-7364-ac4c-74cc-5d6997bcc545}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:04:59.128Z [INFO] stable state for Leader reached at 2021-07-10 01:04:59.02356428 +0000 UTC m=+4.411722250 (1 nodes), 60.581478ms from start of poll, 61.968253ms from cluster start. Timeout at 2021-07-10 01:04:59.128642848 +0000 UTC m=+4.516800818, 105.078568ms after stability - testing.go:149: cluster: 2021-07-10T01:04:59.128Z [INFO] stable state for Leader reached at 2021-07-10 01:04:59.02356428 +0000 UTC m=+4.411722250 (1 nodes), 60.581478ms from start of poll, 61.968253ms from cluster start. Timeout at 2021-07-10 01:04:59.128642848 +0000 UTC m=+4.516800818, 105.078568ms after stability -testLoggerAdapter verbose: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.128Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: updating configuration: command=AddStaging server-id=server-b1b024f2-7c48-1049-2a38-bdbbc303259e server-addr=b1b024f2-7c48-1049-2a38-bdbbc303259e servers="[{Suffrage:Voter ID:server-4510cfdf-e38a-e76d-5524-8f0140a945fc Address:4510cfdf-e38a-e76d-5524-8f0140a945fc} {Suffrage:Voter ID:server-b07c6965-7364-ac4c-74cc-5d6997bcc545 Address:b07c6965-7364-ac4c-74cc-5d6997bcc545} {Suffrage:Voter ID:server-b1b024f2-7c48-1049-2a38-bdbbc303259e Address:b1b024f2-7c48-1049-2a38-bdbbc303259e}]" - testing.go:149: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.128Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: updating configuration: command=AddStaging server-id=server-b1b024f2-7c48-1049-2a38-bdbbc303259e server-addr=b1b024f2-7c48-1049-2a38-bdbbc303259e servers="[{Suffrage:Voter ID:server-4510cfdf-e38a-e76d-5524-8f0140a945fc Address:4510cfdf-e38a-e76d-5524-8f0140a945fc} {Suffrage:Voter ID:server-b07c6965-7364-ac4c-74cc-5d6997bcc545 Address:b07c6965-7364-ac4c-74cc-5d6997bcc545} {Suffrage:Voter ID:server-b1b024f2-7c48-1049-2a38-bdbbc303259e Address:b1b024f2-7c48-1049-2a38-bdbbc303259e}]" -testLoggerAdapter verbose: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.129Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: added peer, starting replication: peer=server-b1b024f2-7c48-1049-2a38-bdbbc303259e - testing.go:149: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.129Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: added peer, starting replication: peer=server-b1b024f2-7c48-1049-2a38-bdbbc303259e -testLoggerAdapter verbose: server-b1b024f2-7c48-1049-2a38-bdbbc303259e: 2021-07-10T01:04:59.129Z [WARN] server-b1b024f2-7c48-1049-2a38-bdbbc303259e: failed to get previous log: previous-index=3 last-index=0 error="log not found" - testing.go:149: server-b1b024f2-7c48-1049-2a38-bdbbc303259e: 2021-07-10T01:04:59.129Z [WARN] server-b1b024f2-7c48-1049-2a38-bdbbc303259e: failed to get previous log: previous-index=3 last-index=0 error="log not found" -testLoggerAdapter verbose: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.129Z [WARN] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: appendEntries rejected, sending older logs: peer="{Voter server-b1b024f2-7c48-1049-2a38-bdbbc303259e b1b024f2-7c48-1049-2a38-bdbbc303259e}" next=1 - testing.go:149: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.129Z [WARN] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: appendEntries rejected, sending older logs: peer="{Voter server-b1b024f2-7c48-1049-2a38-bdbbc303259e b1b024f2-7c48-1049-2a38-bdbbc303259e}" next=1 -testLoggerAdapter verbose: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.129Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: pipelining replication: peer="{Voter server-b1b024f2-7c48-1049-2a38-bdbbc303259e b1b024f2-7c48-1049-2a38-bdbbc303259e}" - testing.go:149: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.129Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: pipelining replication: peer="{Voter server-b1b024f2-7c48-1049-2a38-bdbbc303259e b1b024f2-7c48-1049-2a38-bdbbc303259e}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:04:59.134Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:04:59.134Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-07-10T01:04:59.239Z [INFO] stable state for Leader reached at 2021-07-10 01:04:59.13453515 +0000 UTC m=+4.522693120 (1 nodes), 2.04µs from start of poll, 172.939123ms from cluster start. Timeout at 2021-07-10 01:04:59.239607078 +0000 UTC m=+4.627765048, 105.071928ms after stability - testing.go:149: cluster: 2021-07-10T01:04:59.239Z [INFO] stable state for Leader reached at 2021-07-10 01:04:59.13453515 +0000 UTC m=+4.522693120 (1 nodes), 2.04µs from start of poll, 172.939123ms from cluster start. Timeout at 2021-07-10 01:04:59.239607078 +0000 UTC m=+4.627765048, 105.071928ms after stability -testLoggerAdapter verbose: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.239Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: aborting pipeline replication: peer="{Voter server-b1b024f2-7c48-1049-2a38-bdbbc303259e b1b024f2-7c48-1049-2a38-bdbbc303259e}" - testing.go:149: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.239Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: aborting pipeline replication: peer="{Voter server-b1b024f2-7c48-1049-2a38-bdbbc303259e b1b024f2-7c48-1049-2a38-bdbbc303259e}" -testLoggerAdapter verbose: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.239Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: aborting pipeline replication: peer="{Voter server-b07c6965-7364-ac4c-74cc-5d6997bcc545 b07c6965-7364-ac4c-74cc-5d6997bcc545}" - testing.go:149: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.239Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: aborting pipeline replication: peer="{Voter server-b07c6965-7364-ac4c-74cc-5d6997bcc545 b07c6965-7364-ac4c-74cc-5d6997bcc545}" ---- PASS: TestRaft_HasExistingState (0.28s) +testLoggerAdapter verbose: server-63689627-ec61-cd07-0a5f-ca4abe59e7c8: 2022-08-12T07:30:03.389Z [INFO] server-63689627-ec61-cd07-0a5f-ca4abe59e7c8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63689627-ec61-cd07-0a5f-ca4abe59e7c8 Address:63689627-ec61-cd07-0a5f-ca4abe59e7c8} {Suffrage:Voter ID:server-5c6076e5-2c95-54f6-598c-1a9e5885812f Address:5c6076e5-2c95-54f6-598c-1a9e5885812f}]" + testing.go:149: server-63689627-ec61-cd07-0a5f-ca4abe59e7c8: 2022-08-12T07:30:03.389Z [INFO] server-63689627-ec61-cd07-0a5f-ca4abe59e7c8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63689627-ec61-cd07-0a5f-ca4abe59e7c8 Address:63689627-ec61-cd07-0a5f-ca4abe59e7c8} {Suffrage:Voter ID:server-5c6076e5-2c95-54f6-598c-1a9e5885812f Address:5c6076e5-2c95-54f6-598c-1a9e5885812f}]" +testLoggerAdapter verbose: server-63689627-ec61-cd07-0a5f-ca4abe59e7c8: 2022-08-12T07:30:03.390Z [INFO] server-63689627-ec61-cd07-0a5f-ca4abe59e7c8: entering follower state: follower="Node at 63689627-ec61-cd07-0a5f-ca4abe59e7c8 [Follower]" leader= + testing.go:149: server-63689627-ec61-cd07-0a5f-ca4abe59e7c8: 2022-08-12T07:30:03.390Z [INFO] server-63689627-ec61-cd07-0a5f-ca4abe59e7c8: entering follower state: follower="Node at 63689627-ec61-cd07-0a5f-ca4abe59e7c8 [Follower]" leader= +testLoggerAdapter verbose: server-5c6076e5-2c95-54f6-598c-1a9e5885812f: 2022-08-12T07:30:03.390Z [INFO] server-5c6076e5-2c95-54f6-598c-1a9e5885812f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63689627-ec61-cd07-0a5f-ca4abe59e7c8 Address:63689627-ec61-cd07-0a5f-ca4abe59e7c8} {Suffrage:Voter ID:server-5c6076e5-2c95-54f6-598c-1a9e5885812f Address:5c6076e5-2c95-54f6-598c-1a9e5885812f}]" + testing.go:149: server-5c6076e5-2c95-54f6-598c-1a9e5885812f: 2022-08-12T07:30:03.390Z [INFO] server-5c6076e5-2c95-54f6-598c-1a9e5885812f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63689627-ec61-cd07-0a5f-ca4abe59e7c8 Address:63689627-ec61-cd07-0a5f-ca4abe59e7c8} {Suffrage:Voter ID:server-5c6076e5-2c95-54f6-598c-1a9e5885812f Address:5c6076e5-2c95-54f6-598c-1a9e5885812f}]" +testLoggerAdapter verbose: server-5c6076e5-2c95-54f6-598c-1a9e5885812f: 2022-08-12T07:30:03.390Z [INFO] server-5c6076e5-2c95-54f6-598c-1a9e5885812f: entering follower state: follower="Node at 5c6076e5-2c95-54f6-598c-1a9e5885812f [Follower]" leader= + testing.go:149: server-5c6076e5-2c95-54f6-598c-1a9e5885812f: 2022-08-12T07:30:03.390Z [INFO] server-5c6076e5-2c95-54f6-598c-1a9e5885812f: entering follower state: follower="Node at 5c6076e5-2c95-54f6-598c-1a9e5885812f [Follower]" leader= +testLoggerAdapter verbose: server-e0c8369c-6270-0120-566a-9ecf3014fd6c: 2022-08-12T07:30:03.390Z [INFO] server-e0c8369c-6270-0120-566a-9ecf3014fd6c: initial configuration: index=0 servers=[] + testing.go:149: server-e0c8369c-6270-0120-566a-9ecf3014fd6c: 2022-08-12T07:30:03.390Z [INFO] server-e0c8369c-6270-0120-566a-9ecf3014fd6c: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: server-e0c8369c-6270-0120-566a-9ecf3014fd6c: 2022-08-12T07:30:03.390Z [INFO] server-e0c8369c-6270-0120-566a-9ecf3014fd6c: entering follower state: follower="Node at e0c8369c-6270-0120-566a-9ecf3014fd6c [Follower]" leader= +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:03.390Z [INFO] starting stability test: raft-state=Leader + testing.go:149: server-e0c8369c-6270-0120-566a-9ecf3014fd6c: 2022-08-12T07:30:03.390Z [INFO] server-e0c8369c-6270-0120-566a-9ecf3014fd6c: entering follower state: follower="Node at e0c8369c-6270-0120-566a-9ecf3014fd6c [Follower]" leader= + testing.go:149: cluster: 2022-08-12T07:30:03.390Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-5c6076e5-2c95-54f6-598c-1a9e5885812f: 2022-08-12T07:30:03.442Z [WARN] server-5c6076e5-2c95-54f6-598c-1a9e5885812f: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-5c6076e5-2c95-54f6-598c-1a9e5885812f: 2022-08-12T07:30:03.442Z [WARN] server-5c6076e5-2c95-54f6-598c-1a9e5885812f: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-5c6076e5-2c95-54f6-598c-1a9e5885812f: 2022-08-12T07:30:03.443Z [INFO] server-5c6076e5-2c95-54f6-598c-1a9e5885812f: entering candidate state: node="Node at 5c6076e5-2c95-54f6-598c-1a9e5885812f [Candidate]" term=2 + testing.go:149: server-5c6076e5-2c95-54f6-598c-1a9e5885812f: 2022-08-12T07:30:03.443Z [INFO] server-5c6076e5-2c95-54f6-598c-1a9e5885812f: entering candidate state: node="Node at 5c6076e5-2c95-54f6-598c-1a9e5885812f [Candidate]" term=2 +testLoggerAdapter verbose: server-5c6076e5-2c95-54f6-598c-1a9e5885812f: 2022-08-12T07:30:03.443Z [INFO] server-5c6076e5-2c95-54f6-598c-1a9e5885812f: election won: tally=2 + testing.go:149: server-5c6076e5-2c95-54f6-598c-1a9e5885812f: 2022-08-12T07:30:03.443Z [INFO] server-5c6076e5-2c95-54f6-598c-1a9e5885812f: election won: tally=2 +testLoggerAdapter verbose: server-5c6076e5-2c95-54f6-598c-1a9e5885812f: 2022-08-12T07:30:03.443Z [INFO] server-5c6076e5-2c95-54f6-598c-1a9e5885812f: entering leader state: leader="Node at 5c6076e5-2c95-54f6-598c-1a9e5885812f [Leader]" + testing.go:149: server-5c6076e5-2c95-54f6-598c-1a9e5885812f: 2022-08-12T07:30:03.443Z [INFO] server-5c6076e5-2c95-54f6-598c-1a9e5885812f: entering leader state: leader="Node at 5c6076e5-2c95-54f6-598c-1a9e5885812f [Leader]" +testLoggerAdapter verbose: server-5c6076e5-2c95-54f6-598c-1a9e5885812f: 2022-08-12T07:30:03.443Z [INFO] server-5c6076e5-2c95-54f6-598c-1a9e5885812f: added peer, starting replication: peer=server-63689627-ec61-cd07-0a5f-ca4abe59e7c8 + testing.go:149: server-5c6076e5-2c95-54f6-598c-1a9e5885812f: 2022-08-12T07:30:03.443Z [INFO] server-5c6076e5-2c95-54f6-598c-1a9e5885812f: added peer, starting replication: peer=server-63689627-ec61-cd07-0a5f-ca4abe59e7c8 +testLoggerAdapter verbose: server-5c6076e5-2c95-54f6-598c-1a9e5885812f: 2022-08-12T07:30:03.443Z [INFO] server-5c6076e5-2c95-54f6-598c-1a9e5885812f: pipelining replication: peer="{Voter server-63689627-ec61-cd07-0a5f-ca4abe59e7c8 63689627-ec61-cd07-0a5f-ca4abe59e7c8}" + testing.go:149: server-5c6076e5-2c95-54f6-598c-1a9e5885812f: 2022-08-12T07:30:03.443Z [INFO] server-5c6076e5-2c95-54f6-598c-1a9e5885812f: pipelining replication: peer="{Voter server-63689627-ec61-cd07-0a5f-ca4abe59e7c8 63689627-ec61-cd07-0a5f-ca4abe59e7c8}" +testLoggerAdapter verbose: server-e0c8369c-6270-0120-566a-9ecf3014fd6c: 2022-08-12T07:30:03.467Z [WARN] server-e0c8369c-6270-0120-566a-9ecf3014fd6c: no known peers, aborting election + testing.go:149: server-e0c8369c-6270-0120-566a-9ecf3014fd6c: 2022-08-12T07:30:03.467Z [WARN] server-e0c8369c-6270-0120-566a-9ecf3014fd6c: no known peers, aborting election +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:03.548Z [INFO] stable state for Leader reached at 2022-08-12 07:30:03.4435445 +0000 UTC m=+4.400230981 (1 nodes), 52.73052ms from start of poll, 53.8977ms from cluster start. Timeout at 2022-08-12 07:30:03.54862656 +0000 UTC m=+4.505313041, 105.08206ms after stability + testing.go:149: cluster: 2022-08-12T07:30:03.548Z [INFO] stable state for Leader reached at 2022-08-12 07:30:03.4435445 +0000 UTC m=+4.400230981 (1 nodes), 52.73052ms from start of poll, 53.8977ms from cluster start. Timeout at 2022-08-12 07:30:03.54862656 +0000 UTC m=+4.505313041, 105.08206ms after stability +testLoggerAdapter verbose: server-5c6076e5-2c95-54f6-598c-1a9e5885812f: 2022-08-12T07:30:03.548Z [INFO] server-5c6076e5-2c95-54f6-598c-1a9e5885812f: updating configuration: command=AddStaging server-id=server-e0c8369c-6270-0120-566a-9ecf3014fd6c server-addr=e0c8369c-6270-0120-566a-9ecf3014fd6c servers="[{Suffrage:Voter ID:server-63689627-ec61-cd07-0a5f-ca4abe59e7c8 Address:63689627-ec61-cd07-0a5f-ca4abe59e7c8} {Suffrage:Voter ID:server-5c6076e5-2c95-54f6-598c-1a9e5885812f Address:5c6076e5-2c95-54f6-598c-1a9e5885812f} {Suffrage:Voter ID:server-e0c8369c-6270-0120-566a-9ecf3014fd6c Address:e0c8369c-6270-0120-566a-9ecf3014fd6c}]" + testing.go:149: server-5c6076e5-2c95-54f6-598c-1a9e5885812f: 2022-08-12T07:30:03.548Z [INFO] server-5c6076e5-2c95-54f6-598c-1a9e5885812f: updating configuration: command=AddStaging server-id=server-e0c8369c-6270-0120-566a-9ecf3014fd6c server-addr=e0c8369c-6270-0120-566a-9ecf3014fd6c servers="[{Suffrage:Voter ID:server-63689627-ec61-cd07-0a5f-ca4abe59e7c8 Address:63689627-ec61-cd07-0a5f-ca4abe59e7c8} {Suffrage:Voter ID:server-5c6076e5-2c95-54f6-598c-1a9e5885812f Address:5c6076e5-2c95-54f6-598c-1a9e5885812f} {Suffrage:Voter ID:server-e0c8369c-6270-0120-566a-9ecf3014fd6c Address:e0c8369c-6270-0120-566a-9ecf3014fd6c}]" +testLoggerAdapter verbose: server-5c6076e5-2c95-54f6-598c-1a9e5885812f: 2022-08-12T07:30:03.549Z [INFO] server-5c6076e5-2c95-54f6-598c-1a9e5885812f: added peer, starting replication: peer=server-e0c8369c-6270-0120-566a-9ecf3014fd6c + testing.go:149: server-5c6076e5-2c95-54f6-598c-1a9e5885812f: 2022-08-12T07:30:03.549Z [INFO] server-5c6076e5-2c95-54f6-598c-1a9e5885812f: added peer, starting replication: peer=server-e0c8369c-6270-0120-566a-9ecf3014fd6c +testLoggerAdapter verbose: server-e0c8369c-6270-0120-566a-9ecf3014fd6c: 2022-08-12T07:30:03.549Z [WARN] server-e0c8369c-6270-0120-566a-9ecf3014fd6c: failed to get previous log: previous-index=3 last-index=0 error="log not found" + testing.go:149: server-e0c8369c-6270-0120-566a-9ecf3014fd6c: 2022-08-12T07:30:03.549Z [WARN] server-e0c8369c-6270-0120-566a-9ecf3014fd6c: failed to get previous log: previous-index=3 last-index=0 error="log not found" +testLoggerAdapter verbose: server-5c6076e5-2c95-54f6-598c-1a9e5885812f: 2022-08-12T07:30:03.549Z [WARN] server-5c6076e5-2c95-54f6-598c-1a9e5885812f: appendEntries rejected, sending older logs: peer="{Voter server-e0c8369c-6270-0120-566a-9ecf3014fd6c e0c8369c-6270-0120-566a-9ecf3014fd6c}" next=1 + testing.go:149: server-5c6076e5-2c95-54f6-598c-1a9e5885812f: 2022-08-12T07:30:03.549Z [WARN] server-5c6076e5-2c95-54f6-598c-1a9e5885812f: appendEntries rejected, sending older logs: peer="{Voter server-e0c8369c-6270-0120-566a-9ecf3014fd6c e0c8369c-6270-0120-566a-9ecf3014fd6c}" next=1 +testLoggerAdapter verbose: server-5c6076e5-2c95-54f6-598c-1a9e5885812f: 2022-08-12T07:30:03.549Z [INFO] server-5c6076e5-2c95-54f6-598c-1a9e5885812f: pipelining replication: peer="{Voter server-e0c8369c-6270-0120-566a-9ecf3014fd6c e0c8369c-6270-0120-566a-9ecf3014fd6c}" + testing.go:149: server-5c6076e5-2c95-54f6-598c-1a9e5885812f: 2022-08-12T07:30:03.549Z [INFO] server-5c6076e5-2c95-54f6-598c-1a9e5885812f: pipelining replication: peer="{Voter server-e0c8369c-6270-0120-566a-9ecf3014fd6c e0c8369c-6270-0120-566a-9ecf3014fd6c}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:03.554Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:03.554Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:03.659Z [INFO] stable state for Leader reached at 2022-08-12 07:30:03.55459532 +0000 UTC m=+4.511281801 (1 nodes), 1.48µs from start of poll, 164.94852ms from cluster start. Timeout at 2022-08-12 07:30:03.65962516 +0000 UTC m=+4.616311641, 105.02984ms after stability + testing.go:149: cluster: 2022-08-12T07:30:03.659Z [INFO] stable state for Leader reached at 2022-08-12 07:30:03.55459532 +0000 UTC m=+4.511281801 (1 nodes), 1.48µs from start of poll, 164.94852ms from cluster start. Timeout at 2022-08-12 07:30:03.65962516 +0000 UTC m=+4.616311641, 105.02984ms after stability +testLoggerAdapter verbose: server-5c6076e5-2c95-54f6-598c-1a9e5885812f: 2022-08-12T07:30:03.659Z [INFO] server-5c6076e5-2c95-54f6-598c-1a9e5885812f: aborting pipeline replication: peer="{Voter server-e0c8369c-6270-0120-566a-9ecf3014fd6c e0c8369c-6270-0120-566a-9ecf3014fd6c}" + testing.go:149: server-5c6076e5-2c95-54f6-598c-1a9e5885812f: 2022-08-12T07:30:03.659Z [INFO] server-5c6076e5-2c95-54f6-598c-1a9e5885812f: aborting pipeline replication: peer="{Voter server-e0c8369c-6270-0120-566a-9ecf3014fd6c e0c8369c-6270-0120-566a-9ecf3014fd6c}" +testLoggerAdapter verbose: server-5c6076e5-2c95-54f6-598c-1a9e5885812f: 2022-08-12T07:30:03.659Z [INFO] server-5c6076e5-2c95-54f6-598c-1a9e5885812f: aborting pipeline replication: peer="{Voter server-63689627-ec61-cd07-0a5f-ca4abe59e7c8 63689627-ec61-cd07-0a5f-ca4abe59e7c8}" + testing.go:149: server-5c6076e5-2c95-54f6-598c-1a9e5885812f: 2022-08-12T07:30:03.659Z [INFO] server-5c6076e5-2c95-54f6-598c-1a9e5885812f: aborting pipeline replication: peer="{Voter server-63689627-ec61-cd07-0a5f-ca4abe59e7c8 63689627-ec61-cd07-0a5f-ca4abe59e7c8}" +--- PASS: TestRaft_HasExistingState (0.27s) === RUN TestRaft_SingleNode -testLoggerAdapter verbose: server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: 2021-07-10T01:04:59.241Z [INFO] server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f Address:47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f}]" - testing.go:149: server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: 2021-07-10T01:04:59.241Z [INFO] server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f Address:47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f}]" -testLoggerAdapter verbose: server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: 2021-07-10T01:04:59.241Z [INFO] server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: entering follower state: follower="Node at 47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f [Follower]" leader= - testing.go:149: server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: 2021-07-10T01:04:59.241Z [INFO] server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: entering follower state: follower="Node at 47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f [Follower]" leader= -testLoggerAdapter verbose: server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: 2021-07-10T01:04:59.302Z [WARN] server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: 2021-07-10T01:04:59.302Z [WARN] server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: 2021-07-10T01:04:59.302Z [INFO] server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: entering candidate state: node="Node at 47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f [Candidate]" term=2 - testing.go:149: server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: 2021-07-10T01:04:59.302Z [INFO] server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: entering candidate state: node="Node at 47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f [Candidate]" term=2 -testLoggerAdapter verbose: server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: 2021-07-10T01:04:59.302Z [INFO] server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: election won: tally=1 - testing.go:149: server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: 2021-07-10T01:04:59.302Z [INFO] server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: election won: tally=1 -testLoggerAdapter verbose: server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: 2021-07-10T01:04:59.303Z [INFO] server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: entering leader state: leader="Node at 47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f [Leader]" - testing.go:149: server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: 2021-07-10T01:04:59.303Z [INFO] server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: entering leader state: leader="Node at 47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f [Leader]" ---- PASS: TestRaft_SingleNode (0.06s) +testLoggerAdapter verbose: server-75c54fa0-8ff6-d555-4cf1-dab7a83cb86b: 2022-08-12T07:30:03.661Z [INFO] server-75c54fa0-8ff6-d555-4cf1-dab7a83cb86b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-75c54fa0-8ff6-d555-4cf1-dab7a83cb86b Address:75c54fa0-8ff6-d555-4cf1-dab7a83cb86b}]" + testing.go:149: server-75c54fa0-8ff6-d555-4cf1-dab7a83cb86b: 2022-08-12T07:30:03.661Z [INFO] server-75c54fa0-8ff6-d555-4cf1-dab7a83cb86b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-75c54fa0-8ff6-d555-4cf1-dab7a83cb86b Address:75c54fa0-8ff6-d555-4cf1-dab7a83cb86b}]" +testLoggerAdapter verbose: server-75c54fa0-8ff6-d555-4cf1-dab7a83cb86b: 2022-08-12T07:30:03.661Z [INFO] server-75c54fa0-8ff6-d555-4cf1-dab7a83cb86b: entering follower state: follower="Node at 75c54fa0-8ff6-d555-4cf1-dab7a83cb86b [Follower]" leader= + testing.go:149: server-75c54fa0-8ff6-d555-4cf1-dab7a83cb86b: 2022-08-12T07:30:03.661Z [INFO] server-75c54fa0-8ff6-d555-4cf1-dab7a83cb86b: entering follower state: follower="Node at 75c54fa0-8ff6-d555-4cf1-dab7a83cb86b [Follower]" leader= +testLoggerAdapter verbose: server-75c54fa0-8ff6-d555-4cf1-dab7a83cb86b: 2022-08-12T07:30:03.757Z [WARN] server-75c54fa0-8ff6-d555-4cf1-dab7a83cb86b: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-75c54fa0-8ff6-d555-4cf1-dab7a83cb86b: 2022-08-12T07:30:03.757Z [WARN] server-75c54fa0-8ff6-d555-4cf1-dab7a83cb86b: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-75c54fa0-8ff6-d555-4cf1-dab7a83cb86b: 2022-08-12T07:30:03.757Z [INFO] server-75c54fa0-8ff6-d555-4cf1-dab7a83cb86b: entering candidate state: node="Node at 75c54fa0-8ff6-d555-4cf1-dab7a83cb86b [Candidate]" term=2 + testing.go:149: server-75c54fa0-8ff6-d555-4cf1-dab7a83cb86b: 2022-08-12T07:30:03.757Z [INFO] server-75c54fa0-8ff6-d555-4cf1-dab7a83cb86b: entering candidate state: node="Node at 75c54fa0-8ff6-d555-4cf1-dab7a83cb86b [Candidate]" term=2 +testLoggerAdapter verbose: server-75c54fa0-8ff6-d555-4cf1-dab7a83cb86b: 2022-08-12T07:30:03.757Z [INFO] server-75c54fa0-8ff6-d555-4cf1-dab7a83cb86b: election won: tally=1 + testing.go:149: server-75c54fa0-8ff6-d555-4cf1-dab7a83cb86b: 2022-08-12T07:30:03.757Z [INFO] server-75c54fa0-8ff6-d555-4cf1-dab7a83cb86b: election won: tally=1 +testLoggerAdapter verbose: server-75c54fa0-8ff6-d555-4cf1-dab7a83cb86b: 2022-08-12T07:30:03.757Z [INFO] server-75c54fa0-8ff6-d555-4cf1-dab7a83cb86b: entering leader state: leader="Node at 75c54fa0-8ff6-d555-4cf1-dab7a83cb86b [Leader]" + testing.go:149: server-75c54fa0-8ff6-d555-4cf1-dab7a83cb86b: 2022-08-12T07:30:03.757Z [INFO] server-75c54fa0-8ff6-d555-4cf1-dab7a83cb86b: entering leader state: leader="Node at 75c54fa0-8ff6-d555-4cf1-dab7a83cb86b [Leader]" +--- PASS: TestRaft_SingleNode (0.10s) === RUN TestRaft_TripleNode -testLoggerAdapter verbose: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.304Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cdd76ee3-70b9-f7fa-aeb0-3524550db012 Address:cdd76ee3-70b9-f7fa-aeb0-3524550db012} {Suffrage:Voter ID:server-ece067ae-beeb-1de5-b057-f7feaf981b24 Address:ece067ae-beeb-1de5-b057-f7feaf981b24} {Suffrage:Voter ID:server-d59a0575-eef2-c732-f077-0e80a179a16a Address:d59a0575-eef2-c732-f077-0e80a179a16a}]" - testing.go:149: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.304Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cdd76ee3-70b9-f7fa-aeb0-3524550db012 Address:cdd76ee3-70b9-f7fa-aeb0-3524550db012} {Suffrage:Voter ID:server-ece067ae-beeb-1de5-b057-f7feaf981b24 Address:ece067ae-beeb-1de5-b057-f7feaf981b24} {Suffrage:Voter ID:server-d59a0575-eef2-c732-f077-0e80a179a16a Address:d59a0575-eef2-c732-f077-0e80a179a16a}]" -testLoggerAdapter verbose: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.304Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: entering follower state: follower="Node at cdd76ee3-70b9-f7fa-aeb0-3524550db012 [Follower]" leader= - testing.go:149: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.304Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: entering follower state: follower="Node at cdd76ee3-70b9-f7fa-aeb0-3524550db012 [Follower]" leader= -testLoggerAdapter verbose: server-ece067ae-beeb-1de5-b057-f7feaf981b24: 2021-07-10T01:04:59.304Z [INFO] server-ece067ae-beeb-1de5-b057-f7feaf981b24: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cdd76ee3-70b9-f7fa-aeb0-3524550db012 Address:cdd76ee3-70b9-f7fa-aeb0-3524550db012} {Suffrage:Voter ID:server-ece067ae-beeb-1de5-b057-f7feaf981b24 Address:ece067ae-beeb-1de5-b057-f7feaf981b24} {Suffrage:Voter ID:server-d59a0575-eef2-c732-f077-0e80a179a16a Address:d59a0575-eef2-c732-f077-0e80a179a16a}]" - testing.go:149: server-ece067ae-beeb-1de5-b057-f7feaf981b24: 2021-07-10T01:04:59.304Z [INFO] server-ece067ae-beeb-1de5-b057-f7feaf981b24: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cdd76ee3-70b9-f7fa-aeb0-3524550db012 Address:cdd76ee3-70b9-f7fa-aeb0-3524550db012} {Suffrage:Voter ID:server-ece067ae-beeb-1de5-b057-f7feaf981b24 Address:ece067ae-beeb-1de5-b057-f7feaf981b24} {Suffrage:Voter ID:server-d59a0575-eef2-c732-f077-0e80a179a16a Address:d59a0575-eef2-c732-f077-0e80a179a16a}]" -testLoggerAdapter verbose: server-ece067ae-beeb-1de5-b057-f7feaf981b24: 2021-07-10T01:04:59.304Z [INFO] server-ece067ae-beeb-1de5-b057-f7feaf981b24: entering follower state: follower="Node at ece067ae-beeb-1de5-b057-f7feaf981b24 [Follower]" leader= - testing.go:149: server-ece067ae-beeb-1de5-b057-f7feaf981b24: 2021-07-10T01:04:59.304Z [INFO] server-ece067ae-beeb-1de5-b057-f7feaf981b24: entering follower state: follower="Node at ece067ae-beeb-1de5-b057-f7feaf981b24 [Follower]" leader= -testLoggerAdapter verbose: server-d59a0575-eef2-c732-f077-0e80a179a16a: 2021-07-10T01:04:59.304Z [INFO] server-d59a0575-eef2-c732-f077-0e80a179a16a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cdd76ee3-70b9-f7fa-aeb0-3524550db012 Address:cdd76ee3-70b9-f7fa-aeb0-3524550db012} {Suffrage:Voter ID:server-ece067ae-beeb-1de5-b057-f7feaf981b24 Address:ece067ae-beeb-1de5-b057-f7feaf981b24} {Suffrage:Voter ID:server-d59a0575-eef2-c732-f077-0e80a179a16a Address:d59a0575-eef2-c732-f077-0e80a179a16a}]" - testing.go:149: server-d59a0575-eef2-c732-f077-0e80a179a16a: 2021-07-10T01:04:59.304Z [INFO] server-d59a0575-eef2-c732-f077-0e80a179a16a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cdd76ee3-70b9-f7fa-aeb0-3524550db012 Address:cdd76ee3-70b9-f7fa-aeb0-3524550db012} {Suffrage:Voter ID:server-ece067ae-beeb-1de5-b057-f7feaf981b24 Address:ece067ae-beeb-1de5-b057-f7feaf981b24} {Suffrage:Voter ID:server-d59a0575-eef2-c732-f077-0e80a179a16a Address:d59a0575-eef2-c732-f077-0e80a179a16a}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:04:59.305Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-07-10T01:04:59.305Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-d59a0575-eef2-c732-f077-0e80a179a16a: 2021-07-10T01:04:59.305Z [INFO] server-d59a0575-eef2-c732-f077-0e80a179a16a: entering follower state: follower="Node at d59a0575-eef2-c732-f077-0e80a179a16a [Follower]" leader= - testing.go:149: server-d59a0575-eef2-c732-f077-0e80a179a16a: 2021-07-10T01:04:59.305Z [INFO] server-d59a0575-eef2-c732-f077-0e80a179a16a: entering follower state: follower="Node at d59a0575-eef2-c732-f077-0e80a179a16a [Follower]" leader= -testLoggerAdapter verbose: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.360Z [WARN] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.360Z [WARN] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.360Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: entering candidate state: node="Node at cdd76ee3-70b9-f7fa-aeb0-3524550db012 [Candidate]" term=2 - testing.go:149: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.360Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: entering candidate state: node="Node at cdd76ee3-70b9-f7fa-aeb0-3524550db012 [Candidate]" term=2 -testLoggerAdapter verbose: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.360Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: election won: tally=2 - testing.go:149: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.360Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: election won: tally=2 -testLoggerAdapter verbose: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.360Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: entering leader state: leader="Node at cdd76ee3-70b9-f7fa-aeb0-3524550db012 [Leader]" - testing.go:149: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.360Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: entering leader state: leader="Node at cdd76ee3-70b9-f7fa-aeb0-3524550db012 [Leader]" -testLoggerAdapter verbose: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.360Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: added peer, starting replication: peer=server-ece067ae-beeb-1de5-b057-f7feaf981b24 - testing.go:149: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.360Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: added peer, starting replication: peer=server-ece067ae-beeb-1de5-b057-f7feaf981b24 -testLoggerAdapter verbose: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.360Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: added peer, starting replication: peer=server-d59a0575-eef2-c732-f077-0e80a179a16a - testing.go:149: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.360Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: added peer, starting replication: peer=server-d59a0575-eef2-c732-f077-0e80a179a16a -testLoggerAdapter verbose: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.360Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: pipelining replication: peer="{Voter server-d59a0575-eef2-c732-f077-0e80a179a16a d59a0575-eef2-c732-f077-0e80a179a16a}" - testing.go:149: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.360Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: pipelining replication: peer="{Voter server-d59a0575-eef2-c732-f077-0e80a179a16a d59a0575-eef2-c732-f077-0e80a179a16a}" -testLoggerAdapter verbose: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.361Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: pipelining replication: peer="{Voter server-ece067ae-beeb-1de5-b057-f7feaf981b24 ece067ae-beeb-1de5-b057-f7feaf981b24}" - testing.go:149: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.361Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: pipelining replication: peer="{Voter server-ece067ae-beeb-1de5-b057-f7feaf981b24 ece067ae-beeb-1de5-b057-f7feaf981b24}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:04:59.465Z [INFO] stable state for Follower reached at 2021-07-10 01:04:59.360798115 +0000 UTC m=+4.748956085 (2 nodes), 55.687307ms from start of poll, 56.708777ms from cluster start. Timeout at 2021-07-10 01:04:59.465877843 +0000 UTC m=+4.854035813, 105.079728ms after stability - testing.go:149: cluster: 2021-07-10T01:04:59.465Z [INFO] stable state for Follower reached at 2021-07-10 01:04:59.360798115 +0000 UTC m=+4.748956085 (2 nodes), 55.687307ms from start of poll, 56.708777ms from cluster start. Timeout at 2021-07-10 01:04:59.465877843 +0000 UTC m=+4.854035813, 105.079728ms after stability -testLoggerAdapter verbose: cluster: 2021-07-10T01:04:59.466Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:04:59.466Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-07-10T01:04:59.571Z [INFO] stable state for Leader reached at 2021-07-10 01:04:59.466124766 +0000 UTC m=+4.854282736 (1 nodes), 2.16µs from start of poll, 162.035428ms from cluster start. Timeout at 2021-07-10 01:04:59.571186353 +0000 UTC m=+4.959344323, 105.061587ms after stability - testing.go:149: cluster: 2021-07-10T01:04:59.571Z [INFO] stable state for Leader reached at 2021-07-10 01:04:59.466124766 +0000 UTC m=+4.854282736 (1 nodes), 2.16µs from start of poll, 162.035428ms from cluster start. Timeout at 2021-07-10 01:04:59.571186353 +0000 UTC m=+4.959344323, 105.061587ms after stability -testLoggerAdapter verbose: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.581Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: aborting pipeline replication: peer="{Voter server-d59a0575-eef2-c732-f077-0e80a179a16a d59a0575-eef2-c732-f077-0e80a179a16a}" - testing.go:149: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.581Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: aborting pipeline replication: peer="{Voter server-d59a0575-eef2-c732-f077-0e80a179a16a d59a0575-eef2-c732-f077-0e80a179a16a}" -testLoggerAdapter verbose: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.581Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: aborting pipeline replication: peer="{Voter server-ece067ae-beeb-1de5-b057-f7feaf981b24 ece067ae-beeb-1de5-b057-f7feaf981b24}" - testing.go:149: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.581Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: aborting pipeline replication: peer="{Voter server-ece067ae-beeb-1de5-b057-f7feaf981b24 ece067ae-beeb-1de5-b057-f7feaf981b24}" ---- PASS: TestRaft_TripleNode (0.28s) +testLoggerAdapter verbose: server-857e5099-4508-2065-d19c-6a0cd5c182da: 2022-08-12T07:30:03.758Z [INFO] server-857e5099-4508-2065-d19c-6a0cd5c182da: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-857e5099-4508-2065-d19c-6a0cd5c182da Address:857e5099-4508-2065-d19c-6a0cd5c182da} {Suffrage:Voter ID:server-cf93d335-a5d0-79be-cfd9-a029c23a2b7c Address:cf93d335-a5d0-79be-cfd9-a029c23a2b7c} {Suffrage:Voter ID:server-61930dba-b946-f56a-fdc0-0699c2ef4f95 Address:61930dba-b946-f56a-fdc0-0699c2ef4f95}]" + testing.go:149: server-857e5099-4508-2065-d19c-6a0cd5c182da: 2022-08-12T07:30:03.758Z [INFO] server-857e5099-4508-2065-d19c-6a0cd5c182da: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-857e5099-4508-2065-d19c-6a0cd5c182da Address:857e5099-4508-2065-d19c-6a0cd5c182da} {Suffrage:Voter ID:server-cf93d335-a5d0-79be-cfd9-a029c23a2b7c Address:cf93d335-a5d0-79be-cfd9-a029c23a2b7c} {Suffrage:Voter ID:server-61930dba-b946-f56a-fdc0-0699c2ef4f95 Address:61930dba-b946-f56a-fdc0-0699c2ef4f95}]" +testLoggerAdapter verbose: server-857e5099-4508-2065-d19c-6a0cd5c182da: 2022-08-12T07:30:03.758Z [INFO] server-857e5099-4508-2065-d19c-6a0cd5c182da: entering follower state: follower="Node at 857e5099-4508-2065-d19c-6a0cd5c182da [Follower]" leader= + testing.go:149: server-857e5099-4508-2065-d19c-6a0cd5c182da: 2022-08-12T07:30:03.758Z [INFO] server-857e5099-4508-2065-d19c-6a0cd5c182da: entering follower state: follower="Node at 857e5099-4508-2065-d19c-6a0cd5c182da [Follower]" leader= +testLoggerAdapter verbose: server-cf93d335-a5d0-79be-cfd9-a029c23a2b7c: 2022-08-12T07:30:03.759Z [INFO] server-cf93d335-a5d0-79be-cfd9-a029c23a2b7c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-857e5099-4508-2065-d19c-6a0cd5c182da Address:857e5099-4508-2065-d19c-6a0cd5c182da} {Suffrage:Voter ID:server-cf93d335-a5d0-79be-cfd9-a029c23a2b7c Address:cf93d335-a5d0-79be-cfd9-a029c23a2b7c} {Suffrage:Voter ID:server-61930dba-b946-f56a-fdc0-0699c2ef4f95 Address:61930dba-b946-f56a-fdc0-0699c2ef4f95}]" + testing.go:149: server-cf93d335-a5d0-79be-cfd9-a029c23a2b7c: 2022-08-12T07:30:03.759Z [INFO] server-cf93d335-a5d0-79be-cfd9-a029c23a2b7c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-857e5099-4508-2065-d19c-6a0cd5c182da Address:857e5099-4508-2065-d19c-6a0cd5c182da} {Suffrage:Voter ID:server-cf93d335-a5d0-79be-cfd9-a029c23a2b7c Address:cf93d335-a5d0-79be-cfd9-a029c23a2b7c} {Suffrage:Voter ID:server-61930dba-b946-f56a-fdc0-0699c2ef4f95 Address:61930dba-b946-f56a-fdc0-0699c2ef4f95}]" +testLoggerAdapter verbose: server-cf93d335-a5d0-79be-cfd9-a029c23a2b7c: 2022-08-12T07:30:03.759Z [INFO] server-cf93d335-a5d0-79be-cfd9-a029c23a2b7c: entering follower state: follower="Node at cf93d335-a5d0-79be-cfd9-a029c23a2b7c [Follower]" leader= + testing.go:149: server-cf93d335-a5d0-79be-cfd9-a029c23a2b7c: 2022-08-12T07:30:03.759Z [INFO] server-cf93d335-a5d0-79be-cfd9-a029c23a2b7c: entering follower state: follower="Node at cf93d335-a5d0-79be-cfd9-a029c23a2b7c [Follower]" leader= +testLoggerAdapter verbose: server-61930dba-b946-f56a-fdc0-0699c2ef4f95: 2022-08-12T07:30:03.759Z [INFO] server-61930dba-b946-f56a-fdc0-0699c2ef4f95: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-857e5099-4508-2065-d19c-6a0cd5c182da Address:857e5099-4508-2065-d19c-6a0cd5c182da} {Suffrage:Voter ID:server-cf93d335-a5d0-79be-cfd9-a029c23a2b7c Address:cf93d335-a5d0-79be-cfd9-a029c23a2b7c} {Suffrage:Voter ID:server-61930dba-b946-f56a-fdc0-0699c2ef4f95 Address:61930dba-b946-f56a-fdc0-0699c2ef4f95}]" + testing.go:149: server-61930dba-b946-f56a-fdc0-0699c2ef4f95: 2022-08-12T07:30:03.759Z [INFO] server-61930dba-b946-f56a-fdc0-0699c2ef4f95: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-857e5099-4508-2065-d19c-6a0cd5c182da Address:857e5099-4508-2065-d19c-6a0cd5c182da} {Suffrage:Voter ID:server-cf93d335-a5d0-79be-cfd9-a029c23a2b7c Address:cf93d335-a5d0-79be-cfd9-a029c23a2b7c} {Suffrage:Voter ID:server-61930dba-b946-f56a-fdc0-0699c2ef4f95 Address:61930dba-b946-f56a-fdc0-0699c2ef4f95}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:03.759Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-61930dba-b946-f56a-fdc0-0699c2ef4f95: 2022-08-12T07:30:03.759Z [INFO] server-61930dba-b946-f56a-fdc0-0699c2ef4f95: entering follower state: follower="Node at 61930dba-b946-f56a-fdc0-0699c2ef4f95 [Follower]" leader= + testing.go:149: cluster: 2022-08-12T07:30:03.759Z [INFO] starting stability test: raft-state=Follower + testing.go:149: server-61930dba-b946-f56a-fdc0-0699c2ef4f95: 2022-08-12T07:30:03.759Z [INFO] server-61930dba-b946-f56a-fdc0-0699c2ef4f95: entering follower state: follower="Node at 61930dba-b946-f56a-fdc0-0699c2ef4f95 [Follower]" leader= +testLoggerAdapter verbose: server-61930dba-b946-f56a-fdc0-0699c2ef4f95: 2022-08-12T07:30:03.823Z [WARN] server-61930dba-b946-f56a-fdc0-0699c2ef4f95: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-61930dba-b946-f56a-fdc0-0699c2ef4f95: 2022-08-12T07:30:03.823Z [WARN] server-61930dba-b946-f56a-fdc0-0699c2ef4f95: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-61930dba-b946-f56a-fdc0-0699c2ef4f95: 2022-08-12T07:30:03.824Z [INFO] server-61930dba-b946-f56a-fdc0-0699c2ef4f95: entering candidate state: node="Node at 61930dba-b946-f56a-fdc0-0699c2ef4f95 [Candidate]" term=2 + testing.go:149: server-61930dba-b946-f56a-fdc0-0699c2ef4f95: 2022-08-12T07:30:03.824Z [INFO] server-61930dba-b946-f56a-fdc0-0699c2ef4f95: entering candidate state: node="Node at 61930dba-b946-f56a-fdc0-0699c2ef4f95 [Candidate]" term=2 +testLoggerAdapter verbose: server-61930dba-b946-f56a-fdc0-0699c2ef4f95: 2022-08-12T07:30:03.824Z [INFO] server-61930dba-b946-f56a-fdc0-0699c2ef4f95: election won: tally=2 + testing.go:149: server-61930dba-b946-f56a-fdc0-0699c2ef4f95: 2022-08-12T07:30:03.824Z [INFO] server-61930dba-b946-f56a-fdc0-0699c2ef4f95: election won: tally=2 +testLoggerAdapter verbose: server-61930dba-b946-f56a-fdc0-0699c2ef4f95: 2022-08-12T07:30:03.824Z [INFO] server-61930dba-b946-f56a-fdc0-0699c2ef4f95: entering leader state: leader="Node at 61930dba-b946-f56a-fdc0-0699c2ef4f95 [Leader]" + testing.go:149: server-61930dba-b946-f56a-fdc0-0699c2ef4f95: 2022-08-12T07:30:03.824Z [INFO] server-61930dba-b946-f56a-fdc0-0699c2ef4f95: entering leader state: leader="Node at 61930dba-b946-f56a-fdc0-0699c2ef4f95 [Leader]" +testLoggerAdapter verbose: server-61930dba-b946-f56a-fdc0-0699c2ef4f95: 2022-08-12T07:30:03.824Z [INFO] server-61930dba-b946-f56a-fdc0-0699c2ef4f95: added peer, starting replication: peer=server-857e5099-4508-2065-d19c-6a0cd5c182da + testing.go:149: server-61930dba-b946-f56a-fdc0-0699c2ef4f95: 2022-08-12T07:30:03.824Z [INFO] server-61930dba-b946-f56a-fdc0-0699c2ef4f95: added peer, starting replication: peer=server-857e5099-4508-2065-d19c-6a0cd5c182da +testLoggerAdapter verbose: server-61930dba-b946-f56a-fdc0-0699c2ef4f95: 2022-08-12T07:30:03.824Z [INFO] server-61930dba-b946-f56a-fdc0-0699c2ef4f95: added peer, starting replication: peer=server-cf93d335-a5d0-79be-cfd9-a029c23a2b7c + testing.go:149: server-61930dba-b946-f56a-fdc0-0699c2ef4f95: 2022-08-12T07:30:03.824Z [INFO] server-61930dba-b946-f56a-fdc0-0699c2ef4f95: added peer, starting replication: peer=server-cf93d335-a5d0-79be-cfd9-a029c23a2b7c +testLoggerAdapter verbose: server-61930dba-b946-f56a-fdc0-0699c2ef4f95: 2022-08-12T07:30:03.824Z [INFO] server-61930dba-b946-f56a-fdc0-0699c2ef4f95: pipelining replication: peer="{Voter server-857e5099-4508-2065-d19c-6a0cd5c182da 857e5099-4508-2065-d19c-6a0cd5c182da}" + testing.go:149: server-61930dba-b946-f56a-fdc0-0699c2ef4f95: 2022-08-12T07:30:03.824Z [INFO] server-61930dba-b946-f56a-fdc0-0699c2ef4f95: pipelining replication: peer="{Voter server-857e5099-4508-2065-d19c-6a0cd5c182da 857e5099-4508-2065-d19c-6a0cd5c182da}" +testLoggerAdapter verbose: server-61930dba-b946-f56a-fdc0-0699c2ef4f95: 2022-08-12T07:30:03.824Z [INFO] server-61930dba-b946-f56a-fdc0-0699c2ef4f95: pipelining replication: peer="{Voter server-cf93d335-a5d0-79be-cfd9-a029c23a2b7c cf93d335-a5d0-79be-cfd9-a029c23a2b7c}" + testing.go:149: server-61930dba-b946-f56a-fdc0-0699c2ef4f95: 2022-08-12T07:30:03.824Z [INFO] server-61930dba-b946-f56a-fdc0-0699c2ef4f95: pipelining replication: peer="{Voter server-cf93d335-a5d0-79be-cfd9-a029c23a2b7c cf93d335-a5d0-79be-cfd9-a029c23a2b7c}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:03.929Z [INFO] stable state for Follower reached at 2022-08-12 07:30:03.82449048 +0000 UTC m=+4.781176961 (2 nodes), 65.04146ms from start of poll, 66.04436ms from cluster start. Timeout at 2022-08-12 07:30:03.92961728 +0000 UTC m=+4.886303781, 105.12682ms after stability + testing.go:149: cluster: 2022-08-12T07:30:03.929Z [INFO] stable state for Follower reached at 2022-08-12 07:30:03.82449048 +0000 UTC m=+4.781176961 (2 nodes), 65.04146ms from start of poll, 66.04436ms from cluster start. Timeout at 2022-08-12 07:30:03.92961728 +0000 UTC m=+4.886303781, 105.12682ms after stability +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:03.929Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:03.929Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:04.035Z [INFO] stable state for Leader reached at 2022-08-12 07:30:03.9299837 +0000 UTC m=+4.886670161 (1 nodes), 2.56µs from start of poll, 171.53756ms from cluster start. Timeout at 2022-08-12 07:30:04.03506002 +0000 UTC m=+4.991746501, 105.07634ms after stability + testing.go:149: cluster: 2022-08-12T07:30:04.035Z [INFO] stable state for Leader reached at 2022-08-12 07:30:03.9299837 +0000 UTC m=+4.886670161 (1 nodes), 2.56µs from start of poll, 171.53756ms from cluster start. Timeout at 2022-08-12 07:30:04.03506002 +0000 UTC m=+4.991746501, 105.07634ms after stability +testLoggerAdapter verbose: server-61930dba-b946-f56a-fdc0-0699c2ef4f95: 2022-08-12T07:30:04.045Z [INFO] server-61930dba-b946-f56a-fdc0-0699c2ef4f95: aborting pipeline replication: peer="{Voter server-cf93d335-a5d0-79be-cfd9-a029c23a2b7c cf93d335-a5d0-79be-cfd9-a029c23a2b7c}" + testing.go:149: server-61930dba-b946-f56a-fdc0-0699c2ef4f95: 2022-08-12T07:30:04.045Z [INFO] server-61930dba-b946-f56a-fdc0-0699c2ef4f95: aborting pipeline replication: peer="{Voter server-cf93d335-a5d0-79be-cfd9-a029c23a2b7c cf93d335-a5d0-79be-cfd9-a029c23a2b7c}" +testLoggerAdapter verbose: server-61930dba-b946-f56a-fdc0-0699c2ef4f95: 2022-08-12T07:30:04.045Z [INFO] server-61930dba-b946-f56a-fdc0-0699c2ef4f95: aborting pipeline replication: peer="{Voter server-857e5099-4508-2065-d19c-6a0cd5c182da 857e5099-4508-2065-d19c-6a0cd5c182da}" + testing.go:149: server-61930dba-b946-f56a-fdc0-0699c2ef4f95: 2022-08-12T07:30:04.045Z [INFO] server-61930dba-b946-f56a-fdc0-0699c2ef4f95: aborting pipeline replication: peer="{Voter server-857e5099-4508-2065-d19c-6a0cd5c182da 857e5099-4508-2065-d19c-6a0cd5c182da}" +--- PASS: TestRaft_TripleNode (0.29s) === RUN TestRaft_LeaderFail -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.583Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-772b73f7-89b2-6b24-3174-ae6a9841a41c Address:772b73f7-89b2-6b24-3174-ae6a9841a41c} {Suffrage:Voter ID:server-dc8eae36-1d7d-8f36-117a-63bca2af9124 Address:dc8eae36-1d7d-8f36-117a-63bca2af9124} {Suffrage:Voter ID:server-7e5a1339-ba92-b6b9-afd4-c4de792836ac Address:7e5a1339-ba92-b6b9-afd4-c4de792836ac}]" - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.583Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-772b73f7-89b2-6b24-3174-ae6a9841a41c Address:772b73f7-89b2-6b24-3174-ae6a9841a41c} {Suffrage:Voter ID:server-dc8eae36-1d7d-8f36-117a-63bca2af9124 Address:dc8eae36-1d7d-8f36-117a-63bca2af9124} {Suffrage:Voter ID:server-7e5a1339-ba92-b6b9-afd4-c4de792836ac Address:7e5a1339-ba92-b6b9-afd4-c4de792836ac}]" -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.583Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: entering follower state: follower="Node at 772b73f7-89b2-6b24-3174-ae6a9841a41c [Follower]" leader= - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.583Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: entering follower state: follower="Node at 772b73f7-89b2-6b24-3174-ae6a9841a41c [Follower]" leader= -testLoggerAdapter verbose: server-dc8eae36-1d7d-8f36-117a-63bca2af9124: 2021-07-10T01:04:59.584Z [INFO] server-dc8eae36-1d7d-8f36-117a-63bca2af9124: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-772b73f7-89b2-6b24-3174-ae6a9841a41c Address:772b73f7-89b2-6b24-3174-ae6a9841a41c} {Suffrage:Voter ID:server-dc8eae36-1d7d-8f36-117a-63bca2af9124 Address:dc8eae36-1d7d-8f36-117a-63bca2af9124} {Suffrage:Voter ID:server-7e5a1339-ba92-b6b9-afd4-c4de792836ac Address:7e5a1339-ba92-b6b9-afd4-c4de792836ac}]" - testing.go:149: server-dc8eae36-1d7d-8f36-117a-63bca2af9124: 2021-07-10T01:04:59.584Z [INFO] server-dc8eae36-1d7d-8f36-117a-63bca2af9124: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-772b73f7-89b2-6b24-3174-ae6a9841a41c Address:772b73f7-89b2-6b24-3174-ae6a9841a41c} {Suffrage:Voter ID:server-dc8eae36-1d7d-8f36-117a-63bca2af9124 Address:dc8eae36-1d7d-8f36-117a-63bca2af9124} {Suffrage:Voter ID:server-7e5a1339-ba92-b6b9-afd4-c4de792836ac Address:7e5a1339-ba92-b6b9-afd4-c4de792836ac}]" -testLoggerAdapter verbose: server-dc8eae36-1d7d-8f36-117a-63bca2af9124: 2021-07-10T01:04:59.584Z [INFO] server-dc8eae36-1d7d-8f36-117a-63bca2af9124: entering follower state: follower="Node at dc8eae36-1d7d-8f36-117a-63bca2af9124 [Follower]" leader= - testing.go:149: server-dc8eae36-1d7d-8f36-117a-63bca2af9124: 2021-07-10T01:04:59.584Z [INFO] server-dc8eae36-1d7d-8f36-117a-63bca2af9124: entering follower state: follower="Node at dc8eae36-1d7d-8f36-117a-63bca2af9124 [Follower]" leader= -testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:04:59.584Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-772b73f7-89b2-6b24-3174-ae6a9841a41c Address:772b73f7-89b2-6b24-3174-ae6a9841a41c} {Suffrage:Voter ID:server-dc8eae36-1d7d-8f36-117a-63bca2af9124 Address:dc8eae36-1d7d-8f36-117a-63bca2af9124} {Suffrage:Voter ID:server-7e5a1339-ba92-b6b9-afd4-c4de792836ac Address:7e5a1339-ba92-b6b9-afd4-c4de792836ac}]" - testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:04:59.584Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-772b73f7-89b2-6b24-3174-ae6a9841a41c Address:772b73f7-89b2-6b24-3174-ae6a9841a41c} {Suffrage:Voter ID:server-dc8eae36-1d7d-8f36-117a-63bca2af9124 Address:dc8eae36-1d7d-8f36-117a-63bca2af9124} {Suffrage:Voter ID:server-7e5a1339-ba92-b6b9-afd4-c4de792836ac Address:7e5a1339-ba92-b6b9-afd4-c4de792836ac}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:04:59.584Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-07-10T01:04:59.584Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:04:59.584Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: entering follower state: follower="Node at 7e5a1339-ba92-b6b9-afd4-c4de792836ac [Follower]" leader= - testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:04:59.584Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: entering follower state: follower="Node at 7e5a1339-ba92-b6b9-afd4-c4de792836ac [Follower]" leader= -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.639Z [WARN] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.639Z [WARN] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.639Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: entering candidate state: node="Node at 772b73f7-89b2-6b24-3174-ae6a9841a41c [Candidate]" term=2 - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.639Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: entering candidate state: node="Node at 772b73f7-89b2-6b24-3174-ae6a9841a41c [Candidate]" term=2 -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.640Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: election won: tally=2 - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.640Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: election won: tally=2 -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.640Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: entering leader state: leader="Node at 772b73f7-89b2-6b24-3174-ae6a9841a41c [Leader]" - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.640Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: entering leader state: leader="Node at 772b73f7-89b2-6b24-3174-ae6a9841a41c [Leader]" -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.640Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: added peer, starting replication: peer=server-dc8eae36-1d7d-8f36-117a-63bca2af9124 - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.640Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: added peer, starting replication: peer=server-dc8eae36-1d7d-8f36-117a-63bca2af9124 -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.640Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: added peer, starting replication: peer=server-7e5a1339-ba92-b6b9-afd4-c4de792836ac - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.640Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: added peer, starting replication: peer=server-7e5a1339-ba92-b6b9-afd4-c4de792836ac -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.640Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: pipelining replication: peer="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.640Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: pipelining replication: peer="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.640Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: pipelining replication: peer="{Voter server-7e5a1339-ba92-b6b9-afd4-c4de792836ac 7e5a1339-ba92-b6b9-afd4-c4de792836ac}" - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.640Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: pipelining replication: peer="{Voter server-7e5a1339-ba92-b6b9-afd4-c4de792836ac 7e5a1339-ba92-b6b9-afd4-c4de792836ac}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:04:59.745Z [INFO] stable state for Follower reached at 2021-07-10 01:04:59.640240241 +0000 UTC m=+5.028398191 (2 nodes), 55.730567ms from start of poll, 56.806438ms from cluster start. Timeout at 2021-07-10 01:04:59.745307109 +0000 UTC m=+5.133465079, 105.066888ms after stability - testing.go:149: cluster: 2021-07-10T01:04:59.745Z [INFO] stable state for Follower reached at 2021-07-10 01:04:59.640240241 +0000 UTC m=+5.028398191 (2 nodes), 55.730567ms from start of poll, 56.806438ms from cluster start. Timeout at 2021-07-10 01:04:59.745307109 +0000 UTC m=+5.133465079, 105.066888ms after stability -testLoggerAdapter verbose: cluster: 2021-07-10T01:04:59.745Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:04:59.745Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-07-10T01:04:59.850Z [INFO] stable state for Leader reached at 2021-07-10 01:04:59.745520691 +0000 UTC m=+5.133678641 (1 nodes), 2.4µs from start of poll, 162.086888ms from cluster start. Timeout at 2021-07-10 01:04:59.850582219 +0000 UTC m=+5.238740189, 105.061548ms after stability - testing.go:149: cluster: 2021-07-10T01:04:59.850Z [INFO] stable state for Leader reached at 2021-07-10 01:04:59.745520691 +0000 UTC m=+5.133678641 (1 nodes), 2.4µs from start of poll, 162.086888ms from cluster start. Timeout at 2021-07-10 01:04:59.850582219 +0000 UTC m=+5.238740189, 105.061548ms after stability - raft_test.go:336: [INFO] Disconnecting Node at 772b73f7-89b2-6b24-3174-ae6a9841a41c [Leader] -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.865Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to pipeline appendEntries: peer="{Voter server-7e5a1339-ba92-b6b9-afd4-c4de792836ac 7e5a1339-ba92-b6b9-afd4-c4de792836ac}" error="append pipeline closed" - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.865Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to pipeline appendEntries: peer="{Voter server-7e5a1339-ba92-b6b9-afd4-c4de792836ac 7e5a1339-ba92-b6b9-afd4-c4de792836ac}" error="append pipeline closed" -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.865Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to heartbeat to: peer=dc8eae36-1d7d-8f36-117a-63bca2af9124 error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.865Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to heartbeat to: peer=dc8eae36-1d7d-8f36-117a-63bca2af9124 error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.865Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: aborting pipeline replication: peer="{Voter server-7e5a1339-ba92-b6b9-afd4-c4de792836ac 7e5a1339-ba92-b6b9-afd4-c4de792836ac}" - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.865Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: aborting pipeline replication: peer="{Voter server-7e5a1339-ba92-b6b9-afd4-c4de792836ac 7e5a1339-ba92-b6b9-afd4-c4de792836ac}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:04:59.866Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:04:59.866Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.867Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to pipeline appendEntries: peer="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" error="append pipeline closed" - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.867Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to pipeline appendEntries: peer="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" error="append pipeline closed" -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.867Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to heartbeat to: peer=7e5a1339-ba92-b6b9-afd4-c4de792836ac error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.867Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to heartbeat to: peer=7e5a1339-ba92-b6b9-afd4-c4de792836ac error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.867Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: aborting pipeline replication: peer="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.867Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: aborting pipeline replication: peer="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.873Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to appendEntries to: peer="{Voter server-7e5a1339-ba92-b6b9-afd4-c4de792836ac 7e5a1339-ba92-b6b9-afd4-c4de792836ac}" error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.873Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to appendEntries to: peer="{Voter server-7e5a1339-ba92-b6b9-afd4-c4de792836ac 7e5a1339-ba92-b6b9-afd4-c4de792836ac}" error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.875Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to appendEntries to: peer="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.875Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to appendEntries to: peer="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.884Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to heartbeat to: peer=dc8eae36-1d7d-8f36-117a-63bca2af9124 error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.884Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to heartbeat to: peer=dc8eae36-1d7d-8f36-117a-63bca2af9124 error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.886Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to heartbeat to: peer=7e5a1339-ba92-b6b9-afd4-c4de792836ac error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.886Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to heartbeat to: peer=7e5a1339-ba92-b6b9-afd4-c4de792836ac error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.891Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to appendEntries to: peer="{Voter server-7e5a1339-ba92-b6b9-afd4-c4de792836ac 7e5a1339-ba92-b6b9-afd4-c4de792836ac}" error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.891Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to appendEntries to: peer="{Voter server-7e5a1339-ba92-b6b9-afd4-c4de792836ac 7e5a1339-ba92-b6b9-afd4-c4de792836ac}" error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.891Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to appendEntries to: peer="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.891Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to appendEntries to: peer="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.902Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to heartbeat to: peer=dc8eae36-1d7d-8f36-117a-63bca2af9124 error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.902Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to heartbeat to: peer=dc8eae36-1d7d-8f36-117a-63bca2af9124 error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.902Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to heartbeat to: peer=7e5a1339-ba92-b6b9-afd4-c4de792836ac error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.902Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to heartbeat to: peer=7e5a1339-ba92-b6b9-afd4-c4de792836ac error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.907Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to appendEntries to: peer="{Voter server-7e5a1339-ba92-b6b9-afd4-c4de792836ac 7e5a1339-ba92-b6b9-afd4-c4de792836ac}" error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.907Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to appendEntries to: peer="{Voter server-7e5a1339-ba92-b6b9-afd4-c4de792836ac 7e5a1339-ba92-b6b9-afd4-c4de792836ac}" error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.907Z [WARN] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to contact: server-id=server-dc8eae36-1d7d-8f36-117a-63bca2af9124 time=50.049808ms - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.907Z [WARN] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to contact: server-id=server-dc8eae36-1d7d-8f36-117a-63bca2af9124 time=50.049808ms -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.908Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to appendEntries to: peer="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.908Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to appendEntries to: peer="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.917Z [WARN] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to contact: server-id=server-dc8eae36-1d7d-8f36-117a-63bca2af9124 time=60.271416ms - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.917Z [WARN] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to contact: server-id=server-dc8eae36-1d7d-8f36-117a-63bca2af9124 time=60.271416ms -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.917Z [WARN] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to contact: server-id=server-7e5a1339-ba92-b6b9-afd4-c4de792836ac time=59.134684ms - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.917Z [WARN] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to contact: server-id=server-7e5a1339-ba92-b6b9-afd4-c4de792836ac time=59.134684ms -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.918Z [WARN] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to contact quorum of nodes, stepping down - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.918Z [WARN] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to contact quorum of nodes, stepping down -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.918Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: entering follower state: follower="Node at 772b73f7-89b2-6b24-3174-ae6a9841a41c [Follower]" leader= - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.918Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: entering follower state: follower="Node at 772b73f7-89b2-6b24-3174-ae6a9841a41c [Follower]" leader= -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.928Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to heartbeat to: peer=7e5a1339-ba92-b6b9-afd4-c4de792836ac error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.928Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to heartbeat to: peer=7e5a1339-ba92-b6b9-afd4-c4de792836ac error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.931Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to heartbeat to: peer=dc8eae36-1d7d-8f36-117a-63bca2af9124 error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.931Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to heartbeat to: peer=dc8eae36-1d7d-8f36-117a-63bca2af9124 error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.933Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to appendEntries to: peer="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.933Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to appendEntries to: peer="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" -testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:04:59.935Z [WARN] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: heartbeat timeout reached, starting election: last-leader=772b73f7-89b2-6b24-3174-ae6a9841a41c - testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:04:59.935Z [WARN] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: heartbeat timeout reached, starting election: last-leader=772b73f7-89b2-6b24-3174-ae6a9841a41c -testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:04:59.935Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: entering candidate state: node="Node at 7e5a1339-ba92-b6b9-afd4-c4de792836ac [Candidate]" term=3 - testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:04:59.935Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: entering candidate state: node="Node at 7e5a1339-ba92-b6b9-afd4-c4de792836ac [Candidate]" term=3 -testLoggerAdapter verbose: server-dc8eae36-1d7d-8f36-117a-63bca2af9124: 2021-07-10T01:04:59.936Z [WARN] server-dc8eae36-1d7d-8f36-117a-63bca2af9124: rejecting vote request since we have a leader: from=7e5a1339-ba92-b6b9-afd4-c4de792836ac leader=772b73f7-89b2-6b24-3174-ae6a9841a41c - testing.go:149: server-dc8eae36-1d7d-8f36-117a-63bca2af9124: 2021-07-10T01:04:59.936Z [WARN] server-dc8eae36-1d7d-8f36-117a-63bca2af9124: rejecting vote request since we have a leader: from=7e5a1339-ba92-b6b9-afd4-c4de792836ac leader=772b73f7-89b2-6b24-3174-ae6a9841a41c -testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:04:59.935Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to make requestVote RPC: target="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" - testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:04:59.935Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to make requestVote RPC: target="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.937Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to appendEntries to: peer="{Voter server-7e5a1339-ba92-b6b9-afd4-c4de792836ac 7e5a1339-ba92-b6b9-afd4-c4de792836ac}" error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.937Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to appendEntries to: peer="{Voter server-7e5a1339-ba92-b6b9-afd4-c4de792836ac 7e5a1339-ba92-b6b9-afd4-c4de792836ac}" error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" -testLoggerAdapter verbose: server-dc8eae36-1d7d-8f36-117a-63bca2af9124: 2021-07-10T01:04:59.979Z [WARN] server-dc8eae36-1d7d-8f36-117a-63bca2af9124: heartbeat timeout reached, starting election: last-leader=772b73f7-89b2-6b24-3174-ae6a9841a41c - testing.go:149: server-dc8eae36-1d7d-8f36-117a-63bca2af9124: 2021-07-10T01:04:59.979Z [WARN] server-dc8eae36-1d7d-8f36-117a-63bca2af9124: heartbeat timeout reached, starting election: last-leader=772b73f7-89b2-6b24-3174-ae6a9841a41c -testLoggerAdapter verbose: server-dc8eae36-1d7d-8f36-117a-63bca2af9124: 2021-07-10T01:04:59.979Z [INFO] server-dc8eae36-1d7d-8f36-117a-63bca2af9124: entering candidate state: node="Node at dc8eae36-1d7d-8f36-117a-63bca2af9124 [Candidate]" term=3 - testing.go:149: server-dc8eae36-1d7d-8f36-117a-63bca2af9124: 2021-07-10T01:04:59.979Z [INFO] server-dc8eae36-1d7d-8f36-117a-63bca2af9124: entering candidate state: node="Node at dc8eae36-1d7d-8f36-117a-63bca2af9124 [Candidate]" term=3 -testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:04:59.980Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: duplicate requestVote for same term: term=3 -testLoggerAdapter verbose: server-dc8eae36-1d7d-8f36-117a-63bca2af9124: 2021-07-10T01:04:59.980Z [ERROR] server-dc8eae36-1d7d-8f36-117a-63bca2af9124: failed to make requestVote RPC: target="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" - testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:04:59.980Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: duplicate requestVote for same term: term=3 - testing.go:149: server-dc8eae36-1d7d-8f36-117a-63bca2af9124: 2021-07-10T01:04:59.980Z [ERROR] server-dc8eae36-1d7d-8f36-117a-63bca2af9124: failed to make requestVote RPC: target="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" -testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.002Z [WARN] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: Election timeout reached, restarting election - testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.002Z [WARN] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: Election timeout reached, restarting election -testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.003Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: entering candidate state: node="Node at 7e5a1339-ba92-b6b9-afd4-c4de792836ac [Candidate]" term=4 - testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.003Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: entering candidate state: node="Node at 7e5a1339-ba92-b6b9-afd4-c4de792836ac [Candidate]" term=4 -testLoggerAdapter verbose: server-dc8eae36-1d7d-8f36-117a-63bca2af9124: 2021-07-10T01:05:00.003Z [INFO] server-dc8eae36-1d7d-8f36-117a-63bca2af9124: entering follower state: follower="Node at dc8eae36-1d7d-8f36-117a-63bca2af9124 [Follower]" leader= - testing.go:149: server-dc8eae36-1d7d-8f36-117a-63bca2af9124: 2021-07-10T01:05:00.003Z [INFO] server-dc8eae36-1d7d-8f36-117a-63bca2af9124: entering follower state: follower="Node at dc8eae36-1d7d-8f36-117a-63bca2af9124 [Follower]" leader= -testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.003Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: election won: tally=2 - testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.003Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: election won: tally=2 -testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.003Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: entering leader state: leader="Node at 7e5a1339-ba92-b6b9-afd4-c4de792836ac [Leader]" - testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.003Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: entering leader state: leader="Node at 7e5a1339-ba92-b6b9-afd4-c4de792836ac [Leader]" -testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.003Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: added peer, starting replication: peer=server-772b73f7-89b2-6b24-3174-ae6a9841a41c - testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.003Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: added peer, starting replication: peer=server-772b73f7-89b2-6b24-3174-ae6a9841a41c -testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.003Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to make requestVote RPC: target="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" - testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.003Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to make requestVote RPC: target="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" -testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.003Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: added peer, starting replication: peer=server-dc8eae36-1d7d-8f36-117a-63bca2af9124 - testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.003Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: added peer, starting replication: peer=server-dc8eae36-1d7d-8f36-117a-63bca2af9124 -testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.003Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to appendEntries to: peer="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" - testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.003Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to appendEntries to: peer="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" -testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.004Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: pipelining replication: peer="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" - testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.004Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: pipelining replication: peer="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" -testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.010Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to heartbeat to: peer=772b73f7-89b2-6b24-3174-ae6a9841a41c error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" - testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.010Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to heartbeat to: peer=772b73f7-89b2-6b24-3174-ae6a9841a41c error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:05:00.013Z [WARN] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:05:00.013Z [WARN] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:05:00.013Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: entering candidate state: node="Node at 772b73f7-89b2-6b24-3174-ae6a9841a41c [Candidate]" term=3 - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:05:00.013Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: entering candidate state: node="Node at 772b73f7-89b2-6b24-3174-ae6a9841a41c [Candidate]" term=3 -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:05:00.013Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to make requestVote RPC: target="{Voter server-7e5a1339-ba92-b6b9-afd4-c4de792836ac 7e5a1339-ba92-b6b9-afd4-c4de792836ac}" error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" -testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.013Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to appendEntries to: peer="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:05:00.013Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to make requestVote RPC: target="{Voter server-7e5a1339-ba92-b6b9-afd4-c4de792836ac 7e5a1339-ba92-b6b9-afd4-c4de792836ac}" error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" - testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.013Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to appendEntries to: peer="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:05:00.013Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to make requestVote RPC: target="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:05:00.013Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to make requestVote RPC: target="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" -testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.030Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to heartbeat to: peer=772b73f7-89b2-6b24-3174-ae6a9841a41c error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" - testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.030Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to heartbeat to: peer=772b73f7-89b2-6b24-3174-ae6a9841a41c error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" -testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.030Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to appendEntries to: peer="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" - testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.030Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to appendEntries to: peer="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" -testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.047Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to heartbeat to: peer=772b73f7-89b2-6b24-3174-ae6a9841a41c error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" - testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.047Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to heartbeat to: peer=772b73f7-89b2-6b24-3174-ae6a9841a41c error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" -testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.053Z [WARN] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to contact: server-id=server-772b73f7-89b2-6b24-3174-ae6a9841a41c time=50.33111ms - testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.053Z [WARN] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to contact: server-id=server-772b73f7-89b2-6b24-3174-ae6a9841a41c time=50.33111ms -testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.058Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to appendEntries to: peer="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" - testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.058Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to appendEntries to: peer="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" -testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.077Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to heartbeat to: peer=772b73f7-89b2-6b24-3174-ae6a9841a41c error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" - testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.077Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to heartbeat to: peer=772b73f7-89b2-6b24-3174-ae6a9841a41c error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:05:00.078Z [WARN] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: Election timeout reached, restarting election - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:05:00.078Z [WARN] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: Election timeout reached, restarting election -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:05:00.078Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: entering candidate state: node="Node at 772b73f7-89b2-6b24-3174-ae6a9841a41c [Candidate]" term=4 - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:05:00.078Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: entering candidate state: node="Node at 772b73f7-89b2-6b24-3174-ae6a9841a41c [Candidate]" term=4 -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:05:00.078Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to make requestVote RPC: target="{Voter server-7e5a1339-ba92-b6b9-afd4-c4de792836ac 7e5a1339-ba92-b6b9-afd4-c4de792836ac}" error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:05:00.078Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to make requestVote RPC: target="{Voter server-7e5a1339-ba92-b6b9-afd4-c4de792836ac 7e5a1339-ba92-b6b9-afd4-c4de792836ac}" error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:05:00.078Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to make requestVote RPC: target="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:05:00.078Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to make requestVote RPC: target="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" -testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.103Z [WARN] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to contact: server-id=server-772b73f7-89b2-6b24-3174-ae6a9841a41c time=99.736131ms - testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.103Z [WARN] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to contact: server-id=server-772b73f7-89b2-6b24-3174-ae6a9841a41c time=99.736131ms -testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.107Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to appendEntries to: peer="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" - testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.107Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to appendEntries to: peer="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:00.119Z [INFO] stable state for Leader reached at 2021-07-10 01:05:00.01386486 +0000 UTC m=+5.402022830 (1 nodes), 147.361573ms from start of poll, 430.431077ms from cluster start. Timeout at 2021-07-10 01:05:00.118937068 +0000 UTC m=+5.507095038, 105.072208ms after stability - testing.go:149: cluster: 2021-07-10T01:05:00.119Z [INFO] stable state for Leader reached at 2021-07-10 01:05:00.01386486 +0000 UTC m=+5.402022830 (1 nodes), 147.361573ms from start of poll, 430.431077ms from cluster start. Timeout at 2021-07-10 01:05:00.118937068 +0000 UTC m=+5.507095038, 105.072208ms after stability - raft_test.go:371: [INFO] Reconnecting Node at 772b73f7-89b2-6b24-3174-ae6a9841a41c [Candidate] -testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:05:00.123Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: entering follower state: follower="Node at 772b73f7-89b2-6b24-3174-ae6a9841a41c [Follower]" leader=7e5a1339-ba92-b6b9-afd4-c4de792836ac - testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:05:00.123Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: entering follower state: follower="Node at 772b73f7-89b2-6b24-3174-ae6a9841a41c [Follower]" leader=7e5a1339-ba92-b6b9-afd4-c4de792836ac -testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.193Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: pipelining replication: peer="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" - testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.193Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: pipelining replication: peer="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" -testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.194Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: aborting pipeline replication: peer="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" - testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.194Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: aborting pipeline replication: peer="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" -testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.194Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: aborting pipeline replication: peer="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" - testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.194Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: aborting pipeline replication: peer="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" ---- PASS: TestRaft_LeaderFail (0.61s) +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.047Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-03239572-6daf-56e0-ced5-c92fdf8f5ddd Address:03239572-6daf-56e0-ced5-c92fdf8f5ddd} {Suffrage:Voter ID:server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 Address:534f03ce-f8ac-3ba8-34d1-fd8b10c21f43} {Suffrage:Voter ID:server-2c165020-96bb-92f2-561d-79045588e252 Address:2c165020-96bb-92f2-561d-79045588e252}]" + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.047Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-03239572-6daf-56e0-ced5-c92fdf8f5ddd Address:03239572-6daf-56e0-ced5-c92fdf8f5ddd} {Suffrage:Voter ID:server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 Address:534f03ce-f8ac-3ba8-34d1-fd8b10c21f43} {Suffrage:Voter ID:server-2c165020-96bb-92f2-561d-79045588e252 Address:2c165020-96bb-92f2-561d-79045588e252}]" +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.047Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: entering follower state: follower="Node at 03239572-6daf-56e0-ced5-c92fdf8f5ddd [Follower]" leader= + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.047Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: entering follower state: follower="Node at 03239572-6daf-56e0-ced5-c92fdf8f5ddd [Follower]" leader= +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.047Z [INFO] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-03239572-6daf-56e0-ced5-c92fdf8f5ddd Address:03239572-6daf-56e0-ced5-c92fdf8f5ddd} {Suffrage:Voter ID:server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 Address:534f03ce-f8ac-3ba8-34d1-fd8b10c21f43} {Suffrage:Voter ID:server-2c165020-96bb-92f2-561d-79045588e252 Address:2c165020-96bb-92f2-561d-79045588e252}]" + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.047Z [INFO] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-03239572-6daf-56e0-ced5-c92fdf8f5ddd Address:03239572-6daf-56e0-ced5-c92fdf8f5ddd} {Suffrage:Voter ID:server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 Address:534f03ce-f8ac-3ba8-34d1-fd8b10c21f43} {Suffrage:Voter ID:server-2c165020-96bb-92f2-561d-79045588e252 Address:2c165020-96bb-92f2-561d-79045588e252}]" +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.047Z [INFO] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: entering follower state: follower="Node at 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 [Follower]" leader= + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.047Z [INFO] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: entering follower state: follower="Node at 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 [Follower]" leader= +testLoggerAdapter verbose: server-2c165020-96bb-92f2-561d-79045588e252: 2022-08-12T07:30:04.047Z [INFO] server-2c165020-96bb-92f2-561d-79045588e252: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-03239572-6daf-56e0-ced5-c92fdf8f5ddd Address:03239572-6daf-56e0-ced5-c92fdf8f5ddd} {Suffrage:Voter ID:server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 Address:534f03ce-f8ac-3ba8-34d1-fd8b10c21f43} {Suffrage:Voter ID:server-2c165020-96bb-92f2-561d-79045588e252 Address:2c165020-96bb-92f2-561d-79045588e252}]" + testing.go:149: server-2c165020-96bb-92f2-561d-79045588e252: 2022-08-12T07:30:04.047Z [INFO] server-2c165020-96bb-92f2-561d-79045588e252: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-03239572-6daf-56e0-ced5-c92fdf8f5ddd Address:03239572-6daf-56e0-ced5-c92fdf8f5ddd} {Suffrage:Voter ID:server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 Address:534f03ce-f8ac-3ba8-34d1-fd8b10c21f43} {Suffrage:Voter ID:server-2c165020-96bb-92f2-561d-79045588e252 Address:2c165020-96bb-92f2-561d-79045588e252}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:04.047Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2022-08-12T07:30:04.047Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-2c165020-96bb-92f2-561d-79045588e252: 2022-08-12T07:30:04.047Z [INFO] server-2c165020-96bb-92f2-561d-79045588e252: entering follower state: follower="Node at 2c165020-96bb-92f2-561d-79045588e252 [Follower]" leader= + testing.go:149: server-2c165020-96bb-92f2-561d-79045588e252: 2022-08-12T07:30:04.047Z [INFO] server-2c165020-96bb-92f2-561d-79045588e252: entering follower state: follower="Node at 2c165020-96bb-92f2-561d-79045588e252 [Follower]" leader= +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.099Z [WARN] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.099Z [WARN] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.099Z [INFO] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: entering candidate state: node="Node at 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 [Candidate]" term=2 + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.099Z [INFO] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: entering candidate state: node="Node at 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 [Candidate]" term=2 +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.099Z [INFO] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: election won: tally=2 + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.099Z [INFO] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: election won: tally=2 +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.099Z [INFO] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: entering leader state: leader="Node at 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 [Leader]" + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.099Z [INFO] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: entering leader state: leader="Node at 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 [Leader]" +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.099Z [INFO] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: added peer, starting replication: peer=server-03239572-6daf-56e0-ced5-c92fdf8f5ddd + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.099Z [INFO] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: added peer, starting replication: peer=server-03239572-6daf-56e0-ced5-c92fdf8f5ddd +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.099Z [INFO] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: added peer, starting replication: peer=server-2c165020-96bb-92f2-561d-79045588e252 + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.099Z [INFO] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: added peer, starting replication: peer=server-2c165020-96bb-92f2-561d-79045588e252 +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.099Z [INFO] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: pipelining replication: peer="{Voter server-03239572-6daf-56e0-ced5-c92fdf8f5ddd 03239572-6daf-56e0-ced5-c92fdf8f5ddd}" + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.099Z [INFO] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: pipelining replication: peer="{Voter server-03239572-6daf-56e0-ced5-c92fdf8f5ddd 03239572-6daf-56e0-ced5-c92fdf8f5ddd}" +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.099Z [INFO] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: pipelining replication: peer="{Voter server-2c165020-96bb-92f2-561d-79045588e252 2c165020-96bb-92f2-561d-79045588e252}" + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.099Z [INFO] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: pipelining replication: peer="{Voter server-2c165020-96bb-92f2-561d-79045588e252 2c165020-96bb-92f2-561d-79045588e252}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:04.204Z [INFO] stable state for Follower reached at 2022-08-12 07:30:04.09975054 +0000 UTC m=+5.056437001 (2 nodes), 51.84688ms from start of poll, 52.77936ms from cluster start. Timeout at 2022-08-12 07:30:04.20481196 +0000 UTC m=+5.161498421, 105.06142ms after stability + testing.go:149: cluster: 2022-08-12T07:30:04.204Z [INFO] stable state for Follower reached at 2022-08-12 07:30:04.09975054 +0000 UTC m=+5.056437001 (2 nodes), 51.84688ms from start of poll, 52.77936ms from cluster start. Timeout at 2022-08-12 07:30:04.20481196 +0000 UTC m=+5.161498421, 105.06142ms after stability +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:04.204Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:04.204Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:04.310Z [INFO] stable state for Leader reached at 2022-08-12 07:30:04.20502022 +0000 UTC m=+5.161706701 (1 nodes), 2.16µs from start of poll, 158.04906ms from cluster start. Timeout at 2022-08-12 07:30:04.3100911 +0000 UTC m=+5.266777601, 105.0709ms after stability + testing.go:149: cluster: 2022-08-12T07:30:04.310Z [INFO] stable state for Leader reached at 2022-08-12 07:30:04.20502022 +0000 UTC m=+5.161706701 (1 nodes), 2.16µs from start of poll, 158.04906ms from cluster start. Timeout at 2022-08-12 07:30:04.3100911 +0000 UTC m=+5.266777601, 105.0709ms after stability + raft_test.go:336: [INFO] Disconnecting Node at 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 [Leader] +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.325Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to pipeline appendEntries: peer="{Voter server-2c165020-96bb-92f2-561d-79045588e252 2c165020-96bb-92f2-561d-79045588e252}" error="append pipeline closed" + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.325Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to pipeline appendEntries: peer="{Voter server-2c165020-96bb-92f2-561d-79045588e252 2c165020-96bb-92f2-561d-79045588e252}" error="append pipeline closed" +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.325Z [INFO] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: aborting pipeline replication: peer="{Voter server-2c165020-96bb-92f2-561d-79045588e252 2c165020-96bb-92f2-561d-79045588e252}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:04.325Z [INFO] starting stability test: raft-state=Leader + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.325Z [INFO] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: aborting pipeline replication: peer="{Voter server-2c165020-96bb-92f2-561d-79045588e252 2c165020-96bb-92f2-561d-79045588e252}" + testing.go:149: cluster: 2022-08-12T07:30:04.325Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.327Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to heartbeat to: peer=03239572-6daf-56e0-ced5-c92fdf8f5ddd error="failed to connect to peer: 03239572-6daf-56e0-ced5-c92fdf8f5ddd" + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.327Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to heartbeat to: peer=03239572-6daf-56e0-ced5-c92fdf8f5ddd error="failed to connect to peer: 03239572-6daf-56e0-ced5-c92fdf8f5ddd" +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.327Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to heartbeat to: peer=2c165020-96bb-92f2-561d-79045588e252 error="failed to connect to peer: 2c165020-96bb-92f2-561d-79045588e252" + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.327Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to heartbeat to: peer=2c165020-96bb-92f2-561d-79045588e252 error="failed to connect to peer: 2c165020-96bb-92f2-561d-79045588e252" +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.328Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to pipeline appendEntries: peer="{Voter server-03239572-6daf-56e0-ced5-c92fdf8f5ddd 03239572-6daf-56e0-ced5-c92fdf8f5ddd}" error="append pipeline closed" + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.328Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to pipeline appendEntries: peer="{Voter server-03239572-6daf-56e0-ced5-c92fdf8f5ddd 03239572-6daf-56e0-ced5-c92fdf8f5ddd}" error="append pipeline closed" +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.328Z [INFO] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: aborting pipeline replication: peer="{Voter server-03239572-6daf-56e0-ced5-c92fdf8f5ddd 03239572-6daf-56e0-ced5-c92fdf8f5ddd}" + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.328Z [INFO] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: aborting pipeline replication: peer="{Voter server-03239572-6daf-56e0-ced5-c92fdf8f5ddd 03239572-6daf-56e0-ced5-c92fdf8f5ddd}" +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.331Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to appendEntries to: peer="{Voter server-2c165020-96bb-92f2-561d-79045588e252 2c165020-96bb-92f2-561d-79045588e252}" error="failed to connect to peer: 2c165020-96bb-92f2-561d-79045588e252" + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.331Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to appendEntries to: peer="{Voter server-2c165020-96bb-92f2-561d-79045588e252 2c165020-96bb-92f2-561d-79045588e252}" error="failed to connect to peer: 2c165020-96bb-92f2-561d-79045588e252" +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.334Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to appendEntries to: peer="{Voter server-03239572-6daf-56e0-ced5-c92fdf8f5ddd 03239572-6daf-56e0-ced5-c92fdf8f5ddd}" error="failed to connect to peer: 03239572-6daf-56e0-ced5-c92fdf8f5ddd" + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.334Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to appendEntries to: peer="{Voter server-03239572-6daf-56e0-ced5-c92fdf8f5ddd 03239572-6daf-56e0-ced5-c92fdf8f5ddd}" error="failed to connect to peer: 03239572-6daf-56e0-ced5-c92fdf8f5ddd" +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.343Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to heartbeat to: peer=03239572-6daf-56e0-ced5-c92fdf8f5ddd error="failed to connect to peer: 03239572-6daf-56e0-ced5-c92fdf8f5ddd" + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.343Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to heartbeat to: peer=03239572-6daf-56e0-ced5-c92fdf8f5ddd error="failed to connect to peer: 03239572-6daf-56e0-ced5-c92fdf8f5ddd" +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.346Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to heartbeat to: peer=2c165020-96bb-92f2-561d-79045588e252 error="failed to connect to peer: 2c165020-96bb-92f2-561d-79045588e252" + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.346Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to heartbeat to: peer=2c165020-96bb-92f2-561d-79045588e252 error="failed to connect to peer: 2c165020-96bb-92f2-561d-79045588e252" +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.348Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to appendEntries to: peer="{Voter server-2c165020-96bb-92f2-561d-79045588e252 2c165020-96bb-92f2-561d-79045588e252}" error="failed to connect to peer: 2c165020-96bb-92f2-561d-79045588e252" + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.348Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to appendEntries to: peer="{Voter server-2c165020-96bb-92f2-561d-79045588e252 2c165020-96bb-92f2-561d-79045588e252}" error="failed to connect to peer: 2c165020-96bb-92f2-561d-79045588e252" +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.351Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to appendEntries to: peer="{Voter server-03239572-6daf-56e0-ced5-c92fdf8f5ddd 03239572-6daf-56e0-ced5-c92fdf8f5ddd}" error="failed to connect to peer: 03239572-6daf-56e0-ced5-c92fdf8f5ddd" + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.351Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to appendEntries to: peer="{Voter server-03239572-6daf-56e0-ced5-c92fdf8f5ddd 03239572-6daf-56e0-ced5-c92fdf8f5ddd}" error="failed to connect to peer: 03239572-6daf-56e0-ced5-c92fdf8f5ddd" +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.363Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to heartbeat to: peer=2c165020-96bb-92f2-561d-79045588e252 error="failed to connect to peer: 2c165020-96bb-92f2-561d-79045588e252" + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.363Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to heartbeat to: peer=2c165020-96bb-92f2-561d-79045588e252 error="failed to connect to peer: 2c165020-96bb-92f2-561d-79045588e252" +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.363Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to heartbeat to: peer=03239572-6daf-56e0-ced5-c92fdf8f5ddd error="failed to connect to peer: 03239572-6daf-56e0-ced5-c92fdf8f5ddd" + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.363Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to heartbeat to: peer=03239572-6daf-56e0-ced5-c92fdf8f5ddd error="failed to connect to peer: 03239572-6daf-56e0-ced5-c92fdf8f5ddd" +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.363Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to appendEntries to: peer="{Voter server-2c165020-96bb-92f2-561d-79045588e252 2c165020-96bb-92f2-561d-79045588e252}" error="failed to connect to peer: 2c165020-96bb-92f2-561d-79045588e252" + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.363Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to appendEntries to: peer="{Voter server-2c165020-96bb-92f2-561d-79045588e252 2c165020-96bb-92f2-561d-79045588e252}" error="failed to connect to peer: 2c165020-96bb-92f2-561d-79045588e252" +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.368Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to appendEntries to: peer="{Voter server-03239572-6daf-56e0-ced5-c92fdf8f5ddd 03239572-6daf-56e0-ced5-c92fdf8f5ddd}" error="failed to connect to peer: 03239572-6daf-56e0-ced5-c92fdf8f5ddd" + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.368Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to appendEntries to: peer="{Voter server-03239572-6daf-56e0-ced5-c92fdf8f5ddd 03239572-6daf-56e0-ced5-c92fdf8f5ddd}" error="failed to connect to peer: 03239572-6daf-56e0-ced5-c92fdf8f5ddd" +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.369Z [WARN] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to contact: server-id=server-03239572-6daf-56e0-ced5-c92fdf8f5ddd time=50.07894ms + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.369Z [WARN] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to contact: server-id=server-03239572-6daf-56e0-ced5-c92fdf8f5ddd time=50.07894ms +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.370Z [WARN] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to contact: server-id=server-2c165020-96bb-92f2-561d-79045588e252 time=50.01254ms + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.370Z [WARN] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to contact: server-id=server-2c165020-96bb-92f2-561d-79045588e252 time=50.01254ms +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.370Z [WARN] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to contact quorum of nodes, stepping down + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.370Z [WARN] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to contact quorum of nodes, stepping down +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.370Z [INFO] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: entering follower state: follower="Node at 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 [Follower]" leader= + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.370Z [INFO] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: entering follower state: follower="Node at 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 [Follower]" leader= +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.388Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to heartbeat to: peer=2c165020-96bb-92f2-561d-79045588e252 error="failed to connect to peer: 2c165020-96bb-92f2-561d-79045588e252" + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.388Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to heartbeat to: peer=2c165020-96bb-92f2-561d-79045588e252 error="failed to connect to peer: 2c165020-96bb-92f2-561d-79045588e252" +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.389Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to appendEntries to: peer="{Voter server-2c165020-96bb-92f2-561d-79045588e252 2c165020-96bb-92f2-561d-79045588e252}" error="failed to connect to peer: 2c165020-96bb-92f2-561d-79045588e252" + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.389Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to appendEntries to: peer="{Voter server-2c165020-96bb-92f2-561d-79045588e252 2c165020-96bb-92f2-561d-79045588e252}" error="failed to connect to peer: 2c165020-96bb-92f2-561d-79045588e252" +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.401Z [WARN] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: heartbeat timeout reached, starting election: last-leader=534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.401Z [WARN] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: heartbeat timeout reached, starting election: last-leader=534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.401Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: entering candidate state: node="Node at 03239572-6daf-56e0-ced5-c92fdf8f5ddd [Candidate]" term=3 + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.401Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: entering candidate state: node="Node at 03239572-6daf-56e0-ced5-c92fdf8f5ddd [Candidate]" term=3 +testLoggerAdapter verbose: server-2c165020-96bb-92f2-561d-79045588e252: 2022-08-12T07:30:04.402Z [WARN] server-2c165020-96bb-92f2-561d-79045588e252: rejecting vote request since we have a leader: from=03239572-6daf-56e0-ced5-c92fdf8f5ddd leader=534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.402Z [ERROR] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: failed to make requestVote RPC: target="{Voter server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43}" error="failed to connect to peer: 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43" + testing.go:149: server-2c165020-96bb-92f2-561d-79045588e252: 2022-08-12T07:30:04.402Z [WARN] server-2c165020-96bb-92f2-561d-79045588e252: rejecting vote request since we have a leader: from=03239572-6daf-56e0-ced5-c92fdf8f5ddd leader=534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.402Z [ERROR] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: failed to make requestVote RPC: target="{Voter server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43}" error="failed to connect to peer: 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43" +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.439Z [WARN] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.439Z [WARN] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.439Z [INFO] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: entering candidate state: node="Node at 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 [Candidate]" term=3 + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.439Z [INFO] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: entering candidate state: node="Node at 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 [Candidate]" term=3 +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.439Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to make requestVote RPC: target="{Voter server-2c165020-96bb-92f2-561d-79045588e252 2c165020-96bb-92f2-561d-79045588e252}" error="failed to connect to peer: 2c165020-96bb-92f2-561d-79045588e252" + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.439Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to make requestVote RPC: target="{Voter server-2c165020-96bb-92f2-561d-79045588e252 2c165020-96bb-92f2-561d-79045588e252}" error="failed to connect to peer: 2c165020-96bb-92f2-561d-79045588e252" +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.439Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to make requestVote RPC: target="{Voter server-03239572-6daf-56e0-ced5-c92fdf8f5ddd 03239572-6daf-56e0-ced5-c92fdf8f5ddd}" error="failed to connect to peer: 03239572-6daf-56e0-ced5-c92fdf8f5ddd" + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.439Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to make requestVote RPC: target="{Voter server-03239572-6daf-56e0-ced5-c92fdf8f5ddd 03239572-6daf-56e0-ced5-c92fdf8f5ddd}" error="failed to connect to peer: 03239572-6daf-56e0-ced5-c92fdf8f5ddd" +testLoggerAdapter verbose: server-2c165020-96bb-92f2-561d-79045588e252: 2022-08-12T07:30:04.443Z [WARN] server-2c165020-96bb-92f2-561d-79045588e252: heartbeat timeout reached, starting election: last-leader=534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 + testing.go:149: server-2c165020-96bb-92f2-561d-79045588e252: 2022-08-12T07:30:04.443Z [WARN] server-2c165020-96bb-92f2-561d-79045588e252: heartbeat timeout reached, starting election: last-leader=534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 +testLoggerAdapter verbose: server-2c165020-96bb-92f2-561d-79045588e252: 2022-08-12T07:30:04.444Z [INFO] server-2c165020-96bb-92f2-561d-79045588e252: entering candidate state: node="Node at 2c165020-96bb-92f2-561d-79045588e252 [Candidate]" term=3 + testing.go:149: server-2c165020-96bb-92f2-561d-79045588e252: 2022-08-12T07:30:04.444Z [INFO] server-2c165020-96bb-92f2-561d-79045588e252: entering candidate state: node="Node at 2c165020-96bb-92f2-561d-79045588e252 [Candidate]" term=3 +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.444Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: duplicate requestVote for same term: term=3 + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.444Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: duplicate requestVote for same term: term=3 +testLoggerAdapter verbose: server-2c165020-96bb-92f2-561d-79045588e252: 2022-08-12T07:30:04.444Z [ERROR] server-2c165020-96bb-92f2-561d-79045588e252: failed to make requestVote RPC: target="{Voter server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43}" error="failed to connect to peer: 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43" + testing.go:149: server-2c165020-96bb-92f2-561d-79045588e252: 2022-08-12T07:30:04.444Z [ERROR] server-2c165020-96bb-92f2-561d-79045588e252: failed to make requestVote RPC: target="{Voter server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43}" error="failed to connect to peer: 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43" +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.455Z [WARN] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: Election timeout reached, restarting election + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.455Z [WARN] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: Election timeout reached, restarting election +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.455Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: entering candidate state: node="Node at 03239572-6daf-56e0-ced5-c92fdf8f5ddd [Candidate]" term=4 + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.455Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: entering candidate state: node="Node at 03239572-6daf-56e0-ced5-c92fdf8f5ddd [Candidate]" term=4 +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.455Z [ERROR] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: failed to make requestVote RPC: target="{Voter server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43}" error="failed to connect to peer: 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43" + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.455Z [ERROR] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: failed to make requestVote RPC: target="{Voter server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43}" error="failed to connect to peer: 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43" +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.455Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: election won: tally=2 + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.455Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: election won: tally=2 +testLoggerAdapter verbose: server-2c165020-96bb-92f2-561d-79045588e252: 2022-08-12T07:30:04.455Z [INFO] server-2c165020-96bb-92f2-561d-79045588e252: entering follower state: follower="Node at 2c165020-96bb-92f2-561d-79045588e252 [Follower]" leader= +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.455Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: entering leader state: leader="Node at 03239572-6daf-56e0-ced5-c92fdf8f5ddd [Leader]" + testing.go:149: server-2c165020-96bb-92f2-561d-79045588e252: 2022-08-12T07:30:04.455Z [INFO] server-2c165020-96bb-92f2-561d-79045588e252: entering follower state: follower="Node at 2c165020-96bb-92f2-561d-79045588e252 [Follower]" leader= + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.455Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: entering leader state: leader="Node at 03239572-6daf-56e0-ced5-c92fdf8f5ddd [Leader]" +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.455Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: added peer, starting replication: peer=server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.455Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: added peer, starting replication: peer=server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.456Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: added peer, starting replication: peer=server-2c165020-96bb-92f2-561d-79045588e252 + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.456Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: added peer, starting replication: peer=server-2c165020-96bb-92f2-561d-79045588e252 +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.456Z [ERROR] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: failed to appendEntries to: peer="{Voter server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43}" error="failed to connect to peer: 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43" + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.456Z [ERROR] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: failed to appendEntries to: peer="{Voter server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43}" error="failed to connect to peer: 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43" +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.457Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: pipelining replication: peer="{Voter server-2c165020-96bb-92f2-561d-79045588e252 2c165020-96bb-92f2-561d-79045588e252}" + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.457Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: pipelining replication: peer="{Voter server-2c165020-96bb-92f2-561d-79045588e252 2c165020-96bb-92f2-561d-79045588e252}" +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.462Z [ERROR] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: failed to heartbeat to: peer=534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 error="failed to connect to peer: 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43" + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.462Z [ERROR] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: failed to heartbeat to: peer=534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 error="failed to connect to peer: 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43" +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.466Z [ERROR] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: failed to appendEntries to: peer="{Voter server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43}" error="failed to connect to peer: 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43" + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.466Z [ERROR] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: failed to appendEntries to: peer="{Voter server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43}" error="failed to connect to peer: 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43" +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.477Z [ERROR] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: failed to heartbeat to: peer=534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 error="failed to connect to peer: 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43" + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.477Z [ERROR] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: failed to heartbeat to: peer=534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 error="failed to connect to peer: 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43" +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.484Z [ERROR] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: failed to appendEntries to: peer="{Voter server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43}" error="failed to connect to peer: 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43" + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.484Z [ERROR] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: failed to appendEntries to: peer="{Voter server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43}" error="failed to connect to peer: 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43" +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.495Z [ERROR] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: failed to heartbeat to: peer=534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 error="failed to connect to peer: 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43" + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.495Z [ERROR] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: failed to heartbeat to: peer=534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 error="failed to connect to peer: 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43" +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.506Z [WARN] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: failed to contact: server-id=server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 time=50.82908ms + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.506Z [WARN] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: failed to contact: server-id=server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 time=50.82908ms +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.510Z [WARN] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: Election timeout reached, restarting election + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.510Z [WARN] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: Election timeout reached, restarting election +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.510Z [INFO] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: entering candidate state: node="Node at 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 [Candidate]" term=4 + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.510Z [INFO] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: entering candidate state: node="Node at 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 [Candidate]" term=4 +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.510Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to make requestVote RPC: target="{Voter server-2c165020-96bb-92f2-561d-79045588e252 2c165020-96bb-92f2-561d-79045588e252}" error="failed to connect to peer: 2c165020-96bb-92f2-561d-79045588e252" + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.510Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to make requestVote RPC: target="{Voter server-2c165020-96bb-92f2-561d-79045588e252 2c165020-96bb-92f2-561d-79045588e252}" error="failed to connect to peer: 2c165020-96bb-92f2-561d-79045588e252" +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.510Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to make requestVote RPC: target="{Voter server-03239572-6daf-56e0-ced5-c92fdf8f5ddd 03239572-6daf-56e0-ced5-c92fdf8f5ddd}" error="failed to connect to peer: 03239572-6daf-56e0-ced5-c92fdf8f5ddd" + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.510Z [ERROR] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to make requestVote RPC: target="{Voter server-03239572-6daf-56e0-ced5-c92fdf8f5ddd 03239572-6daf-56e0-ced5-c92fdf8f5ddd}" error="failed to connect to peer: 03239572-6daf-56e0-ced5-c92fdf8f5ddd" +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.511Z [ERROR] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: failed to appendEntries to: peer="{Voter server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43}" error="failed to connect to peer: 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43" + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.511Z [ERROR] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: failed to appendEntries to: peer="{Voter server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43}" error="failed to connect to peer: 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43" +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.521Z [ERROR] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: failed to heartbeat to: peer=534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 error="failed to connect to peer: 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43" + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.521Z [ERROR] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: failed to heartbeat to: peer=534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 error="failed to connect to peer: 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43" +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.555Z [WARN] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: failed to contact: server-id=server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 time=99.41344ms + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.555Z [WARN] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: failed to contact: server-id=server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 time=99.41344ms +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.557Z [ERROR] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: failed to appendEntries to: peer="{Voter server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43}" error="failed to connect to peer: 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43" + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.557Z [ERROR] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: failed to appendEntries to: peer="{Voter server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43}" error="failed to connect to peer: 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:04.561Z [INFO] stable state for Leader reached at 2022-08-12 07:30:04.45591982 +0000 UTC m=+5.412606301 (1 nodes), 129.96818ms from start of poll, 408.94866ms from cluster start. Timeout at 2022-08-12 07:30:04.560952 +0000 UTC m=+5.517638481, 105.03218ms after stability + testing.go:149: cluster: 2022-08-12T07:30:04.561Z [INFO] stable state for Leader reached at 2022-08-12 07:30:04.45591982 +0000 UTC m=+5.412606301 (1 nodes), 129.96818ms from start of poll, 408.94866ms from cluster start. Timeout at 2022-08-12 07:30:04.560952 +0000 UTC m=+5.517638481, 105.03218ms after stability + raft_test.go:371: [INFO] Reconnecting Node at 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 [Candidate] +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.567Z [WARN] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: Election timeout reached, restarting election + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.567Z [WARN] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: Election timeout reached, restarting election +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.567Z [INFO] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: entering candidate state: node="Node at 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 [Candidate]" term=5 + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.567Z [INFO] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: entering candidate state: node="Node at 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 [Candidate]" term=5 +testLoggerAdapter verbose: server-2c165020-96bb-92f2-561d-79045588e252: 2022-08-12T07:30:04.567Z [WARN] server-2c165020-96bb-92f2-561d-79045588e252: rejecting vote request since we have a leader: from=534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 leader=03239572-6daf-56e0-ced5-c92fdf8f5ddd + testing.go:149: server-2c165020-96bb-92f2-561d-79045588e252: 2022-08-12T07:30:04.567Z [WARN] server-2c165020-96bb-92f2-561d-79045588e252: rejecting vote request since we have a leader: from=534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 leader=03239572-6daf-56e0-ced5-c92fdf8f5ddd +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.567Z [WARN] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: rejecting vote request since we have a leader: from=534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 leader=03239572-6daf-56e0-ced5-c92fdf8f5ddd + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.567Z [WARN] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: rejecting vote request since we have a leader: from=534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 leader=03239572-6daf-56e0-ced5-c92fdf8f5ddd +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.622Z [WARN] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: Election timeout reached, restarting election + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.622Z [WARN] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: Election timeout reached, restarting election +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.622Z [INFO] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: entering candidate state: node="Node at 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 [Candidate]" term=6 + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.622Z [INFO] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: entering candidate state: node="Node at 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 [Candidate]" term=6 +testLoggerAdapter verbose: server-2c165020-96bb-92f2-561d-79045588e252: 2022-08-12T07:30:04.622Z [WARN] server-2c165020-96bb-92f2-561d-79045588e252: rejecting vote request since we have a leader: from=534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 leader=03239572-6daf-56e0-ced5-c92fdf8f5ddd +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.622Z [WARN] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: rejecting vote request since we have a leader: from=534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 leader=03239572-6daf-56e0-ced5-c92fdf8f5ddd + testing.go:149: server-2c165020-96bb-92f2-561d-79045588e252: 2022-08-12T07:30:04.622Z [WARN] server-2c165020-96bb-92f2-561d-79045588e252: rejecting vote request since we have a leader: from=534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 leader=03239572-6daf-56e0-ced5-c92fdf8f5ddd + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.622Z [WARN] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: rejecting vote request since we have a leader: from=534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 leader=03239572-6daf-56e0-ced5-c92fdf8f5ddd +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.641Z [ERROR] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: peer has newer term, stopping replication: peer="{Voter server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43}" + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.641Z [ERROR] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: peer has newer term, stopping replication: peer="{Voter server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43}" +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.641Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: entering follower state: follower="Node at 03239572-6daf-56e0-ced5-c92fdf8f5ddd [Follower]" leader= + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.641Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: entering follower state: follower="Node at 03239572-6daf-56e0-ced5-c92fdf8f5ddd [Follower]" leader= +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.641Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: aborting pipeline replication: peer="{Voter server-2c165020-96bb-92f2-561d-79045588e252 2c165020-96bb-92f2-561d-79045588e252}" + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.641Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: aborting pipeline replication: peer="{Voter server-2c165020-96bb-92f2-561d-79045588e252 2c165020-96bb-92f2-561d-79045588e252}" +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.680Z [WARN] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: Election timeout reached, restarting election + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.680Z [WARN] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: Election timeout reached, restarting election +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.680Z [INFO] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: entering candidate state: node="Node at 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 [Candidate]" term=7 + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.680Z [INFO] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: entering candidate state: node="Node at 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 [Candidate]" term=7 +testLoggerAdapter verbose: server-2c165020-96bb-92f2-561d-79045588e252: 2022-08-12T07:30:04.680Z [WARN] server-2c165020-96bb-92f2-561d-79045588e252: rejecting vote request since we have a leader: from=534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 leader=03239572-6daf-56e0-ced5-c92fdf8f5ddd +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.680Z [WARN] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: rejecting vote request since our last term is greater: candidate=534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 last-term=4 last-candidate-term=2 + testing.go:149: server-2c165020-96bb-92f2-561d-79045588e252: 2022-08-12T07:30:04.680Z [WARN] server-2c165020-96bb-92f2-561d-79045588e252: rejecting vote request since we have a leader: from=534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 leader=03239572-6daf-56e0-ced5-c92fdf8f5ddd + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.680Z [WARN] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: rejecting vote request since our last term is greater: candidate=534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 last-term=4 last-candidate-term=2 +testLoggerAdapter verbose: server-2c165020-96bb-92f2-561d-79045588e252: 2022-08-12T07:30:04.714Z [WARN] server-2c165020-96bb-92f2-561d-79045588e252: heartbeat timeout reached, starting election: last-leader=03239572-6daf-56e0-ced5-c92fdf8f5ddd + testing.go:149: server-2c165020-96bb-92f2-561d-79045588e252: 2022-08-12T07:30:04.714Z [WARN] server-2c165020-96bb-92f2-561d-79045588e252: heartbeat timeout reached, starting election: last-leader=03239572-6daf-56e0-ced5-c92fdf8f5ddd +testLoggerAdapter verbose: server-2c165020-96bb-92f2-561d-79045588e252: 2022-08-12T07:30:04.714Z [INFO] server-2c165020-96bb-92f2-561d-79045588e252: entering candidate state: node="Node at 2c165020-96bb-92f2-561d-79045588e252 [Candidate]" term=5 + testing.go:149: server-2c165020-96bb-92f2-561d-79045588e252: 2022-08-12T07:30:04.714Z [INFO] server-2c165020-96bb-92f2-561d-79045588e252: entering candidate state: node="Node at 2c165020-96bb-92f2-561d-79045588e252 [Candidate]" term=5 +testLoggerAdapter verbose: server-2c165020-96bb-92f2-561d-79045588e252: 2022-08-12T07:30:04.714Z [INFO] server-2c165020-96bb-92f2-561d-79045588e252: entering follower state: follower="Node at 2c165020-96bb-92f2-561d-79045588e252 [Follower]" leader= + testing.go:149: server-2c165020-96bb-92f2-561d-79045588e252: 2022-08-12T07:30:04.714Z [INFO] server-2c165020-96bb-92f2-561d-79045588e252: entering follower state: follower="Node at 2c165020-96bb-92f2-561d-79045588e252 [Follower]" leader= +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.732Z [WARN] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.732Z [WARN] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.733Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: entering candidate state: node="Node at 03239572-6daf-56e0-ced5-c92fdf8f5ddd [Candidate]" term=8 + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.733Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: entering candidate state: node="Node at 03239572-6daf-56e0-ced5-c92fdf8f5ddd [Candidate]" term=8 +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.733Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: election won: tally=2 + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.733Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: election won: tally=2 +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.733Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: entering leader state: leader="Node at 03239572-6daf-56e0-ced5-c92fdf8f5ddd [Leader]" +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.733Z [INFO] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: entering follower state: follower="Node at 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 [Follower]" leader= + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.733Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: entering leader state: leader="Node at 03239572-6daf-56e0-ced5-c92fdf8f5ddd [Leader]" +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.733Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: added peer, starting replication: peer=server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.733Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: added peer, starting replication: peer=server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.733Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: added peer, starting replication: peer=server-2c165020-96bb-92f2-561d-79045588e252 + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.733Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: added peer, starting replication: peer=server-2c165020-96bb-92f2-561d-79045588e252 + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.733Z [INFO] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: entering follower state: follower="Node at 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 [Follower]" leader= +testLoggerAdapter verbose: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.733Z [WARN] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to get previous log: previous-index=5 last-index=3 error="log not found" +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.733Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: pipelining replication: peer="{Voter server-2c165020-96bb-92f2-561d-79045588e252 2c165020-96bb-92f2-561d-79045588e252}" + testing.go:149: server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: 2022-08-12T07:30:04.733Z [WARN] server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43: failed to get previous log: previous-index=5 last-index=3 error="log not found" + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.733Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: pipelining replication: peer="{Voter server-2c165020-96bb-92f2-561d-79045588e252 2c165020-96bb-92f2-561d-79045588e252}" +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.733Z [WARN] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: appendEntries rejected, sending older logs: peer="{Voter server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43}" next=4 + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.733Z [WARN] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: appendEntries rejected, sending older logs: peer="{Voter server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43}" next=4 +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.733Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: pipelining replication: peer="{Voter server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43}" + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.733Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: pipelining replication: peer="{Voter server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43}" +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.738Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: aborting pipeline replication: peer="{Voter server-2c165020-96bb-92f2-561d-79045588e252 2c165020-96bb-92f2-561d-79045588e252}" + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.738Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: aborting pipeline replication: peer="{Voter server-2c165020-96bb-92f2-561d-79045588e252 2c165020-96bb-92f2-561d-79045588e252}" +testLoggerAdapter verbose: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.738Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: aborting pipeline replication: peer="{Voter server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43}" + testing.go:149: server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: 2022-08-12T07:30:04.738Z [INFO] server-03239572-6daf-56e0-ced5-c92fdf8f5ddd: aborting pipeline replication: peer="{Voter server-534f03ce-f8ac-3ba8-34d1-fd8b10c21f43 534f03ce-f8ac-3ba8-34d1-fd8b10c21f43}" +--- PASS: TestRaft_LeaderFail (0.69s) === RUN TestRaft_BehindFollower -testLoggerAdapter verbose: server-cc4def27-8c62-c75b-00e3-6804eaf9325d: 2021-07-10T01:05:00.196Z [INFO] server-cc4def27-8c62-c75b-00e3-6804eaf9325d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cc4def27-8c62-c75b-00e3-6804eaf9325d Address:cc4def27-8c62-c75b-00e3-6804eaf9325d} {Suffrage:Voter ID:server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a Address:8fa31dd9-db10-3057-8c4e-7025abaa6e2a} {Suffrage:Voter ID:server-7a0d7856-eca4-bf16-e907-5ebab1a3e751 Address:7a0d7856-eca4-bf16-e907-5ebab1a3e751}]" - testing.go:149: server-cc4def27-8c62-c75b-00e3-6804eaf9325d: 2021-07-10T01:05:00.196Z [INFO] server-cc4def27-8c62-c75b-00e3-6804eaf9325d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cc4def27-8c62-c75b-00e3-6804eaf9325d Address:cc4def27-8c62-c75b-00e3-6804eaf9325d} {Suffrage:Voter ID:server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a Address:8fa31dd9-db10-3057-8c4e-7025abaa6e2a} {Suffrage:Voter ID:server-7a0d7856-eca4-bf16-e907-5ebab1a3e751 Address:7a0d7856-eca4-bf16-e907-5ebab1a3e751}]" -testLoggerAdapter verbose: server-cc4def27-8c62-c75b-00e3-6804eaf9325d: 2021-07-10T01:05:00.196Z [INFO] server-cc4def27-8c62-c75b-00e3-6804eaf9325d: entering follower state: follower="Node at cc4def27-8c62-c75b-00e3-6804eaf9325d [Follower]" leader= - testing.go:149: server-cc4def27-8c62-c75b-00e3-6804eaf9325d: 2021-07-10T01:05:00.196Z [INFO] server-cc4def27-8c62-c75b-00e3-6804eaf9325d: entering follower state: follower="Node at cc4def27-8c62-c75b-00e3-6804eaf9325d [Follower]" leader= -testLoggerAdapter verbose: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.196Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cc4def27-8c62-c75b-00e3-6804eaf9325d Address:cc4def27-8c62-c75b-00e3-6804eaf9325d} {Suffrage:Voter ID:server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a Address:8fa31dd9-db10-3057-8c4e-7025abaa6e2a} {Suffrage:Voter ID:server-7a0d7856-eca4-bf16-e907-5ebab1a3e751 Address:7a0d7856-eca4-bf16-e907-5ebab1a3e751}]" - testing.go:149: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.196Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cc4def27-8c62-c75b-00e3-6804eaf9325d Address:cc4def27-8c62-c75b-00e3-6804eaf9325d} {Suffrage:Voter ID:server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a Address:8fa31dd9-db10-3057-8c4e-7025abaa6e2a} {Suffrage:Voter ID:server-7a0d7856-eca4-bf16-e907-5ebab1a3e751 Address:7a0d7856-eca4-bf16-e907-5ebab1a3e751}]" -testLoggerAdapter verbose: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.196Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: entering follower state: follower="Node at 8fa31dd9-db10-3057-8c4e-7025abaa6e2a [Follower]" leader= - testing.go:149: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.196Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: entering follower state: follower="Node at 8fa31dd9-db10-3057-8c4e-7025abaa6e2a [Follower]" leader= -testLoggerAdapter verbose: server-7a0d7856-eca4-bf16-e907-5ebab1a3e751: 2021-07-10T01:05:00.197Z [INFO] server-7a0d7856-eca4-bf16-e907-5ebab1a3e751: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cc4def27-8c62-c75b-00e3-6804eaf9325d Address:cc4def27-8c62-c75b-00e3-6804eaf9325d} {Suffrage:Voter ID:server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a Address:8fa31dd9-db10-3057-8c4e-7025abaa6e2a} {Suffrage:Voter ID:server-7a0d7856-eca4-bf16-e907-5ebab1a3e751 Address:7a0d7856-eca4-bf16-e907-5ebab1a3e751}]" - testing.go:149: server-7a0d7856-eca4-bf16-e907-5ebab1a3e751: 2021-07-10T01:05:00.197Z [INFO] server-7a0d7856-eca4-bf16-e907-5ebab1a3e751: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cc4def27-8c62-c75b-00e3-6804eaf9325d Address:cc4def27-8c62-c75b-00e3-6804eaf9325d} {Suffrage:Voter ID:server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a Address:8fa31dd9-db10-3057-8c4e-7025abaa6e2a} {Suffrage:Voter ID:server-7a0d7856-eca4-bf16-e907-5ebab1a3e751 Address:7a0d7856-eca4-bf16-e907-5ebab1a3e751}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:00.197Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:00.197Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-7a0d7856-eca4-bf16-e907-5ebab1a3e751: 2021-07-10T01:05:00.197Z [INFO] server-7a0d7856-eca4-bf16-e907-5ebab1a3e751: entering follower state: follower="Node at 7a0d7856-eca4-bf16-e907-5ebab1a3e751 [Follower]" leader= - testing.go:149: server-7a0d7856-eca4-bf16-e907-5ebab1a3e751: 2021-07-10T01:05:00.197Z [INFO] server-7a0d7856-eca4-bf16-e907-5ebab1a3e751: entering follower state: follower="Node at 7a0d7856-eca4-bf16-e907-5ebab1a3e751 [Follower]" leader= -testLoggerAdapter verbose: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.251Z [WARN] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.251Z [WARN] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.251Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: entering candidate state: node="Node at 8fa31dd9-db10-3057-8c4e-7025abaa6e2a [Candidate]" term=2 - testing.go:149: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.251Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: entering candidate state: node="Node at 8fa31dd9-db10-3057-8c4e-7025abaa6e2a [Candidate]" term=2 -testLoggerAdapter verbose: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.251Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: election won: tally=2 - testing.go:149: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.251Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: election won: tally=2 -testLoggerAdapter verbose: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.251Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: entering leader state: leader="Node at 8fa31dd9-db10-3057-8c4e-7025abaa6e2a [Leader]" - testing.go:149: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.251Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: entering leader state: leader="Node at 8fa31dd9-db10-3057-8c4e-7025abaa6e2a [Leader]" -testLoggerAdapter verbose: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.251Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: added peer, starting replication: peer=server-cc4def27-8c62-c75b-00e3-6804eaf9325d - testing.go:149: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.251Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: added peer, starting replication: peer=server-cc4def27-8c62-c75b-00e3-6804eaf9325d -testLoggerAdapter verbose: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.252Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: added peer, starting replication: peer=server-7a0d7856-eca4-bf16-e907-5ebab1a3e751 - testing.go:149: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.252Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: added peer, starting replication: peer=server-7a0d7856-eca4-bf16-e907-5ebab1a3e751 -testLoggerAdapter verbose: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.252Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: pipelining replication: peer="{Voter server-cc4def27-8c62-c75b-00e3-6804eaf9325d cc4def27-8c62-c75b-00e3-6804eaf9325d}" - testing.go:149: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.252Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: pipelining replication: peer="{Voter server-cc4def27-8c62-c75b-00e3-6804eaf9325d cc4def27-8c62-c75b-00e3-6804eaf9325d}" -testLoggerAdapter verbose: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.252Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: pipelining replication: peer="{Voter server-7a0d7856-eca4-bf16-e907-5ebab1a3e751 7a0d7856-eca4-bf16-e907-5ebab1a3e751}" - testing.go:149: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.252Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: pipelining replication: peer="{Voter server-7a0d7856-eca4-bf16-e907-5ebab1a3e751 7a0d7856-eca4-bf16-e907-5ebab1a3e751}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:00.357Z [INFO] stable state for Leader reached at 2021-07-10 01:05:00.251979811 +0000 UTC m=+5.640137801 (1 nodes), 54.825338ms from start of poll, 55.814969ms from cluster start. Timeout at 2021-07-10 01:05:00.357050238 +0000 UTC m=+5.745208208, 105.070407ms after stability - testing.go:149: cluster: 2021-07-10T01:05:00.357Z [INFO] stable state for Leader reached at 2021-07-10 01:05:00.251979811 +0000 UTC m=+5.640137801 (1 nodes), 54.825338ms from start of poll, 55.814969ms from cluster start. Timeout at 2021-07-10 01:05:00.357050238 +0000 UTC m=+5.745208208, 105.070407ms after stability -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:00.357Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-07-10T01:05:00.357Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:00.462Z [INFO] stable state for Follower reached at 2021-07-10 01:05:00.357348541 +0000 UTC m=+5.745506511 (2 nodes), 2.2µs from start of poll, 161.183679ms from cluster start. Timeout at 2021-07-10 01:05:00.462450769 +0000 UTC m=+5.850608759, 105.102248ms after stability - testing.go:149: cluster: 2021-07-10T01:05:00.462Z [INFO] stable state for Follower reached at 2021-07-10 01:05:00.357348541 +0000 UTC m=+5.745506511 (2 nodes), 2.2µs from start of poll, 161.183679ms from cluster start. Timeout at 2021-07-10 01:05:00.462450769 +0000 UTC m=+5.850608759, 105.102248ms after stability -testLoggerAdapter verbose: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.462Z [ERROR] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: failed to pipeline appendEntries: peer="{Voter server-cc4def27-8c62-c75b-00e3-6804eaf9325d cc4def27-8c62-c75b-00e3-6804eaf9325d}" error="append pipeline closed" - testing.go:149: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.462Z [ERROR] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: failed to pipeline appendEntries: peer="{Voter server-cc4def27-8c62-c75b-00e3-6804eaf9325d cc4def27-8c62-c75b-00e3-6804eaf9325d}" error="append pipeline closed" -testLoggerAdapter verbose: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.462Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: aborting pipeline replication: peer="{Voter server-cc4def27-8c62-c75b-00e3-6804eaf9325d cc4def27-8c62-c75b-00e3-6804eaf9325d}" - testing.go:149: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.462Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: aborting pipeline replication: peer="{Voter server-cc4def27-8c62-c75b-00e3-6804eaf9325d cc4def27-8c62-c75b-00e3-6804eaf9325d}" -testLoggerAdapter verbose: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.462Z [ERROR] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: failed to appendEntries to: peer="{Voter server-cc4def27-8c62-c75b-00e3-6804eaf9325d cc4def27-8c62-c75b-00e3-6804eaf9325d}" error="failed to connect to peer: cc4def27-8c62-c75b-00e3-6804eaf9325d" - testing.go:149: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.462Z [ERROR] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: failed to appendEntries to: peer="{Voter server-cc4def27-8c62-c75b-00e3-6804eaf9325d cc4def27-8c62-c75b-00e3-6804eaf9325d}" error="failed to connect to peer: cc4def27-8c62-c75b-00e3-6804eaf9325d" +testLoggerAdapter verbose: server-832ce5b1-2fca-fd9a-c4fb-7058f683091a: 2022-08-12T07:30:04.740Z [INFO] server-832ce5b1-2fca-fd9a-c4fb-7058f683091a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-832ce5b1-2fca-fd9a-c4fb-7058f683091a Address:832ce5b1-2fca-fd9a-c4fb-7058f683091a} {Suffrage:Voter ID:server-6ed299a4-5f85-50e4-61e3-bfb7068cea91 Address:6ed299a4-5f85-50e4-61e3-bfb7068cea91} {Suffrage:Voter ID:server-4462dcdc-5116-427c-5c7a-a2b0a960d8cb Address:4462dcdc-5116-427c-5c7a-a2b0a960d8cb}]" + testing.go:149: server-832ce5b1-2fca-fd9a-c4fb-7058f683091a: 2022-08-12T07:30:04.740Z [INFO] server-832ce5b1-2fca-fd9a-c4fb-7058f683091a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-832ce5b1-2fca-fd9a-c4fb-7058f683091a Address:832ce5b1-2fca-fd9a-c4fb-7058f683091a} {Suffrage:Voter ID:server-6ed299a4-5f85-50e4-61e3-bfb7068cea91 Address:6ed299a4-5f85-50e4-61e3-bfb7068cea91} {Suffrage:Voter ID:server-4462dcdc-5116-427c-5c7a-a2b0a960d8cb Address:4462dcdc-5116-427c-5c7a-a2b0a960d8cb}]" +testLoggerAdapter verbose: server-832ce5b1-2fca-fd9a-c4fb-7058f683091a: 2022-08-12T07:30:04.740Z [INFO] server-832ce5b1-2fca-fd9a-c4fb-7058f683091a: entering follower state: follower="Node at 832ce5b1-2fca-fd9a-c4fb-7058f683091a [Follower]" leader= + testing.go:149: server-832ce5b1-2fca-fd9a-c4fb-7058f683091a: 2022-08-12T07:30:04.740Z [INFO] server-832ce5b1-2fca-fd9a-c4fb-7058f683091a: entering follower state: follower="Node at 832ce5b1-2fca-fd9a-c4fb-7058f683091a [Follower]" leader= +testLoggerAdapter verbose: server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: 2022-08-12T07:30:04.740Z [INFO] server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-832ce5b1-2fca-fd9a-c4fb-7058f683091a Address:832ce5b1-2fca-fd9a-c4fb-7058f683091a} {Suffrage:Voter ID:server-6ed299a4-5f85-50e4-61e3-bfb7068cea91 Address:6ed299a4-5f85-50e4-61e3-bfb7068cea91} {Suffrage:Voter ID:server-4462dcdc-5116-427c-5c7a-a2b0a960d8cb Address:4462dcdc-5116-427c-5c7a-a2b0a960d8cb}]" + testing.go:149: server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: 2022-08-12T07:30:04.740Z [INFO] server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-832ce5b1-2fca-fd9a-c4fb-7058f683091a Address:832ce5b1-2fca-fd9a-c4fb-7058f683091a} {Suffrage:Voter ID:server-6ed299a4-5f85-50e4-61e3-bfb7068cea91 Address:6ed299a4-5f85-50e4-61e3-bfb7068cea91} {Suffrage:Voter ID:server-4462dcdc-5116-427c-5c7a-a2b0a960d8cb Address:4462dcdc-5116-427c-5c7a-a2b0a960d8cb}]" +testLoggerAdapter verbose: server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: 2022-08-12T07:30:04.740Z [INFO] server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: entering follower state: follower="Node at 6ed299a4-5f85-50e4-61e3-bfb7068cea91 [Follower]" leader= + testing.go:149: server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: 2022-08-12T07:30:04.740Z [INFO] server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: entering follower state: follower="Node at 6ed299a4-5f85-50e4-61e3-bfb7068cea91 [Follower]" leader= +testLoggerAdapter verbose: server-4462dcdc-5116-427c-5c7a-a2b0a960d8cb: 2022-08-12T07:30:04.741Z [INFO] server-4462dcdc-5116-427c-5c7a-a2b0a960d8cb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-832ce5b1-2fca-fd9a-c4fb-7058f683091a Address:832ce5b1-2fca-fd9a-c4fb-7058f683091a} {Suffrage:Voter ID:server-6ed299a4-5f85-50e4-61e3-bfb7068cea91 Address:6ed299a4-5f85-50e4-61e3-bfb7068cea91} {Suffrage:Voter ID:server-4462dcdc-5116-427c-5c7a-a2b0a960d8cb Address:4462dcdc-5116-427c-5c7a-a2b0a960d8cb}]" + testing.go:149: server-4462dcdc-5116-427c-5c7a-a2b0a960d8cb: 2022-08-12T07:30:04.741Z [INFO] server-4462dcdc-5116-427c-5c7a-a2b0a960d8cb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-832ce5b1-2fca-fd9a-c4fb-7058f683091a Address:832ce5b1-2fca-fd9a-c4fb-7058f683091a} {Suffrage:Voter ID:server-6ed299a4-5f85-50e4-61e3-bfb7068cea91 Address:6ed299a4-5f85-50e4-61e3-bfb7068cea91} {Suffrage:Voter ID:server-4462dcdc-5116-427c-5c7a-a2b0a960d8cb Address:4462dcdc-5116-427c-5c7a-a2b0a960d8cb}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:04.741Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:04.741Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-4462dcdc-5116-427c-5c7a-a2b0a960d8cb: 2022-08-12T07:30:04.741Z [INFO] server-4462dcdc-5116-427c-5c7a-a2b0a960d8cb: entering follower state: follower="Node at 4462dcdc-5116-427c-5c7a-a2b0a960d8cb [Follower]" leader= + testing.go:149: server-4462dcdc-5116-427c-5c7a-a2b0a960d8cb: 2022-08-12T07:30:04.741Z [INFO] server-4462dcdc-5116-427c-5c7a-a2b0a960d8cb: entering follower state: follower="Node at 4462dcdc-5116-427c-5c7a-a2b0a960d8cb [Follower]" leader= +testLoggerAdapter verbose: server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: 2022-08-12T07:30:04.804Z [WARN] server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: 2022-08-12T07:30:04.804Z [WARN] server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: 2022-08-12T07:30:04.805Z [INFO] server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: entering candidate state: node="Node at 6ed299a4-5f85-50e4-61e3-bfb7068cea91 [Candidate]" term=2 + testing.go:149: server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: 2022-08-12T07:30:04.805Z [INFO] server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: entering candidate state: node="Node at 6ed299a4-5f85-50e4-61e3-bfb7068cea91 [Candidate]" term=2 +testLoggerAdapter verbose: server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: 2022-08-12T07:30:04.805Z [INFO] server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: election won: tally=2 + testing.go:149: server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: 2022-08-12T07:30:04.805Z [INFO] server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: election won: tally=2 +testLoggerAdapter verbose: server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: 2022-08-12T07:30:04.805Z [INFO] server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: entering leader state: leader="Node at 6ed299a4-5f85-50e4-61e3-bfb7068cea91 [Leader]" + testing.go:149: server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: 2022-08-12T07:30:04.805Z [INFO] server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: entering leader state: leader="Node at 6ed299a4-5f85-50e4-61e3-bfb7068cea91 [Leader]" +testLoggerAdapter verbose: server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: 2022-08-12T07:30:04.805Z [INFO] server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: added peer, starting replication: peer=server-832ce5b1-2fca-fd9a-c4fb-7058f683091a + testing.go:149: server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: 2022-08-12T07:30:04.805Z [INFO] server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: added peer, starting replication: peer=server-832ce5b1-2fca-fd9a-c4fb-7058f683091a +testLoggerAdapter verbose: server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: 2022-08-12T07:30:04.805Z [INFO] server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: added peer, starting replication: peer=server-4462dcdc-5116-427c-5c7a-a2b0a960d8cb + testing.go:149: server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: 2022-08-12T07:30:04.805Z [INFO] server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: added peer, starting replication: peer=server-4462dcdc-5116-427c-5c7a-a2b0a960d8cb +testLoggerAdapter verbose: server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: 2022-08-12T07:30:04.805Z [INFO] server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: pipelining replication: peer="{Voter server-832ce5b1-2fca-fd9a-c4fb-7058f683091a 832ce5b1-2fca-fd9a-c4fb-7058f683091a}" + testing.go:149: server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: 2022-08-12T07:30:04.805Z [INFO] server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: pipelining replication: peer="{Voter server-832ce5b1-2fca-fd9a-c4fb-7058f683091a 832ce5b1-2fca-fd9a-c4fb-7058f683091a}" +testLoggerAdapter verbose: server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: 2022-08-12T07:30:04.805Z [INFO] server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: pipelining replication: peer="{Voter server-4462dcdc-5116-427c-5c7a-a2b0a960d8cb 4462dcdc-5116-427c-5c7a-a2b0a960d8cb}" + testing.go:149: server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: 2022-08-12T07:30:04.805Z [INFO] server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: pipelining replication: peer="{Voter server-4462dcdc-5116-427c-5c7a-a2b0a960d8cb 4462dcdc-5116-427c-5c7a-a2b0a960d8cb}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:04.910Z [INFO] stable state for Leader reached at 2022-08-12 07:30:04.80539356 +0000 UTC m=+5.762080041 (1 nodes), 64.16548ms from start of poll, 65.182ms from cluster start. Timeout at 2022-08-12 07:30:04.91045606 +0000 UTC m=+5.867142561, 105.06252ms after stability + testing.go:149: cluster: 2022-08-12T07:30:04.910Z [INFO] stable state for Leader reached at 2022-08-12 07:30:04.80539356 +0000 UTC m=+5.762080041 (1 nodes), 64.16548ms from start of poll, 65.182ms from cluster start. Timeout at 2022-08-12 07:30:04.91045606 +0000 UTC m=+5.867142561, 105.06252ms after stability +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:04.910Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2022-08-12T07:30:04.910Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:05.015Z [INFO] stable state for Follower reached at 2022-08-12 07:30:04.91069728 +0000 UTC m=+5.867383781 (2 nodes), 2.16µs from start of poll, 170.48574ms from cluster start. Timeout at 2022-08-12 07:30:05.0157 +0000 UTC m=+5.972386461, 105.00268ms after stability + testing.go:149: cluster: 2022-08-12T07:30:05.015Z [INFO] stable state for Follower reached at 2022-08-12 07:30:04.91069728 +0000 UTC m=+5.867383781 (2 nodes), 2.16µs from start of poll, 170.48574ms from cluster start. Timeout at 2022-08-12 07:30:05.0157 +0000 UTC m=+5.972386461, 105.00268ms after stability +testLoggerAdapter verbose: server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: 2022-08-12T07:30:05.015Z [ERROR] server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: failed to pipeline appendEntries: peer="{Voter server-832ce5b1-2fca-fd9a-c4fb-7058f683091a 832ce5b1-2fca-fd9a-c4fb-7058f683091a}" error="append pipeline closed" + testing.go:149: server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: 2022-08-12T07:30:05.015Z [ERROR] server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: failed to pipeline appendEntries: peer="{Voter server-832ce5b1-2fca-fd9a-c4fb-7058f683091a 832ce5b1-2fca-fd9a-c4fb-7058f683091a}" error="append pipeline closed" +testLoggerAdapter verbose: server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: 2022-08-12T07:30:05.016Z [INFO] server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: aborting pipeline replication: peer="{Voter server-832ce5b1-2fca-fd9a-c4fb-7058f683091a 832ce5b1-2fca-fd9a-c4fb-7058f683091a}" + testing.go:149: server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: 2022-08-12T07:30:05.016Z [INFO] server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: aborting pipeline replication: peer="{Voter server-832ce5b1-2fca-fd9a-c4fb-7058f683091a 832ce5b1-2fca-fd9a-c4fb-7058f683091a}" +testLoggerAdapter verbose: server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: 2022-08-12T07:30:05.016Z [ERROR] server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: failed to appendEntries to: peer="{Voter server-832ce5b1-2fca-fd9a-c4fb-7058f683091a 832ce5b1-2fca-fd9a-c4fb-7058f683091a}" error="failed to connect to peer: 832ce5b1-2fca-fd9a-c4fb-7058f683091a" + testing.go:149: server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: 2022-08-12T07:30:05.016Z [ERROR] server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: failed to appendEntries to: peer="{Voter server-832ce5b1-2fca-fd9a-c4fb-7058f683091a 832ce5b1-2fca-fd9a-c4fb-7058f683091a}" error="failed to connect to peer: 832ce5b1-2fca-fd9a-c4fb-7058f683091a" raft_test.go:420: [INFO] Finished apply without behind follower -testLoggerAdapter verbose: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.473Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: pipelining replication: peer="{Voter server-cc4def27-8c62-c75b-00e3-6804eaf9325d cc4def27-8c62-c75b-00e3-6804eaf9325d}" - testing.go:149: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.473Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: pipelining replication: peer="{Voter server-cc4def27-8c62-c75b-00e3-6804eaf9325d cc4def27-8c62-c75b-00e3-6804eaf9325d}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:00.485Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:00.485Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:00.591Z [INFO] stable state for Leader reached at 2021-07-10 01:05:00.485859776 +0000 UTC m=+5.874017726 (1 nodes), 2.2µs from start of poll, 289.694894ms from cluster start. Timeout at 2021-07-10 01:05:00.590938324 +0000 UTC m=+5.979096334, 105.078608ms after stability - testing.go:149: cluster: 2021-07-10T01:05:00.591Z [INFO] stable state for Leader reached at 2021-07-10 01:05:00.485859776 +0000 UTC m=+5.874017726 (1 nodes), 2.2µs from start of poll, 289.694894ms from cluster start. Timeout at 2021-07-10 01:05:00.590938324 +0000 UTC m=+5.979096334, 105.078608ms after stability -testLoggerAdapter verbose: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.591Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: aborting pipeline replication: peer="{Voter server-cc4def27-8c62-c75b-00e3-6804eaf9325d cc4def27-8c62-c75b-00e3-6804eaf9325d}" - testing.go:149: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.591Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: aborting pipeline replication: peer="{Voter server-cc4def27-8c62-c75b-00e3-6804eaf9325d cc4def27-8c62-c75b-00e3-6804eaf9325d}" -testLoggerAdapter verbose: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.591Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: aborting pipeline replication: peer="{Voter server-7a0d7856-eca4-bf16-e907-5ebab1a3e751 7a0d7856-eca4-bf16-e907-5ebab1a3e751}" - testing.go:149: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.591Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: aborting pipeline replication: peer="{Voter server-7a0d7856-eca4-bf16-e907-5ebab1a3e751 7a0d7856-eca4-bf16-e907-5ebab1a3e751}" ---- PASS: TestRaft_BehindFollower (0.40s) +testLoggerAdapter verbose: server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: 2022-08-12T07:30:05.026Z [INFO] server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: pipelining replication: peer="{Voter server-832ce5b1-2fca-fd9a-c4fb-7058f683091a 832ce5b1-2fca-fd9a-c4fb-7058f683091a}" + testing.go:149: server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: 2022-08-12T07:30:05.026Z [INFO] server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: pipelining replication: peer="{Voter server-832ce5b1-2fca-fd9a-c4fb-7058f683091a 832ce5b1-2fca-fd9a-c4fb-7058f683091a}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:05.038Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:05.038Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:05.144Z [INFO] stable state for Leader reached at 2022-08-12 07:30:05.03899686 +0000 UTC m=+5.995683341 (1 nodes), 1.82µs from start of poll, 298.7853ms from cluster start. Timeout at 2022-08-12 07:30:05.14406792 +0000 UTC m=+6.100754401, 105.07106ms after stability + testing.go:149: cluster: 2022-08-12T07:30:05.144Z [INFO] stable state for Leader reached at 2022-08-12 07:30:05.03899686 +0000 UTC m=+5.995683341 (1 nodes), 1.82µs from start of poll, 298.7853ms from cluster start. Timeout at 2022-08-12 07:30:05.14406792 +0000 UTC m=+6.100754401, 105.07106ms after stability +testLoggerAdapter verbose: server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: 2022-08-12T07:30:05.144Z [INFO] server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: aborting pipeline replication: peer="{Voter server-4462dcdc-5116-427c-5c7a-a2b0a960d8cb 4462dcdc-5116-427c-5c7a-a2b0a960d8cb}" + testing.go:149: server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: 2022-08-12T07:30:05.144Z [INFO] server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: aborting pipeline replication: peer="{Voter server-4462dcdc-5116-427c-5c7a-a2b0a960d8cb 4462dcdc-5116-427c-5c7a-a2b0a960d8cb}" +testLoggerAdapter verbose: server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: 2022-08-12T07:30:05.144Z [INFO] server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: aborting pipeline replication: peer="{Voter server-832ce5b1-2fca-fd9a-c4fb-7058f683091a 832ce5b1-2fca-fd9a-c4fb-7058f683091a}" + testing.go:149: server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: 2022-08-12T07:30:05.144Z [INFO] server-6ed299a4-5f85-50e4-61e3-bfb7068cea91: aborting pipeline replication: peer="{Voter server-832ce5b1-2fca-fd9a-c4fb-7058f683091a 832ce5b1-2fca-fd9a-c4fb-7058f683091a}" +--- PASS: TestRaft_BehindFollower (0.41s) === RUN TestRaft_ApplyNonLeader -testLoggerAdapter verbose: server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a: 2021-07-10T01:05:00.593Z [INFO] server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a Address:66db94f2-09bd-b5b9-1c59-dfab786a9e6a} {Suffrage:Voter ID:server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd Address:09dd28b7-7554-e5d9-369d-d0beeb9ccdcd} {Suffrage:Voter ID:server-724f68de-f93b-7654-af86-35f73f0d7331 Address:724f68de-f93b-7654-af86-35f73f0d7331}]" - testing.go:149: server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a: 2021-07-10T01:05:00.593Z [INFO] server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a Address:66db94f2-09bd-b5b9-1c59-dfab786a9e6a} {Suffrage:Voter ID:server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd Address:09dd28b7-7554-e5d9-369d-d0beeb9ccdcd} {Suffrage:Voter ID:server-724f68de-f93b-7654-af86-35f73f0d7331 Address:724f68de-f93b-7654-af86-35f73f0d7331}]" -testLoggerAdapter verbose: server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd: 2021-07-10T01:05:00.593Z [INFO] server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a Address:66db94f2-09bd-b5b9-1c59-dfab786a9e6a} {Suffrage:Voter ID:server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd Address:09dd28b7-7554-e5d9-369d-d0beeb9ccdcd} {Suffrage:Voter ID:server-724f68de-f93b-7654-af86-35f73f0d7331 Address:724f68de-f93b-7654-af86-35f73f0d7331}]" - testing.go:149: server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd: 2021-07-10T01:05:00.593Z [INFO] server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a Address:66db94f2-09bd-b5b9-1c59-dfab786a9e6a} {Suffrage:Voter ID:server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd Address:09dd28b7-7554-e5d9-369d-d0beeb9ccdcd} {Suffrage:Voter ID:server-724f68de-f93b-7654-af86-35f73f0d7331 Address:724f68de-f93b-7654-af86-35f73f0d7331}]" -testLoggerAdapter verbose: server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a: 2021-07-10T01:05:00.593Z [INFO] server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a: entering follower state: follower="Node at 66db94f2-09bd-b5b9-1c59-dfab786a9e6a [Follower]" leader= - testing.go:149: server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a: 2021-07-10T01:05:00.593Z [INFO] server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a: entering follower state: follower="Node at 66db94f2-09bd-b5b9-1c59-dfab786a9e6a [Follower]" leader= -testLoggerAdapter verbose: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.593Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a Address:66db94f2-09bd-b5b9-1c59-dfab786a9e6a} {Suffrage:Voter ID:server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd Address:09dd28b7-7554-e5d9-369d-d0beeb9ccdcd} {Suffrage:Voter ID:server-724f68de-f93b-7654-af86-35f73f0d7331 Address:724f68de-f93b-7654-af86-35f73f0d7331}]" - testing.go:149: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.593Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a Address:66db94f2-09bd-b5b9-1c59-dfab786a9e6a} {Suffrage:Voter ID:server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd Address:09dd28b7-7554-e5d9-369d-d0beeb9ccdcd} {Suffrage:Voter ID:server-724f68de-f93b-7654-af86-35f73f0d7331 Address:724f68de-f93b-7654-af86-35f73f0d7331}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:00.593Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:00.593Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.593Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: entering follower state: follower="Node at 724f68de-f93b-7654-af86-35f73f0d7331 [Follower]" leader= -testLoggerAdapter verbose: server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd: 2021-07-10T01:05:00.593Z [INFO] server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd: entering follower state: follower="Node at 09dd28b7-7554-e5d9-369d-d0beeb9ccdcd [Follower]" leader= - testing.go:149: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.593Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: entering follower state: follower="Node at 724f68de-f93b-7654-af86-35f73f0d7331 [Follower]" leader= - testing.go:149: server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd: 2021-07-10T01:05:00.593Z [INFO] server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd: entering follower state: follower="Node at 09dd28b7-7554-e5d9-369d-d0beeb9ccdcd [Follower]" leader= -testLoggerAdapter verbose: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.676Z [WARN] server-724f68de-f93b-7654-af86-35f73f0d7331: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.676Z [WARN] server-724f68de-f93b-7654-af86-35f73f0d7331: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.676Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: entering candidate state: node="Node at 724f68de-f93b-7654-af86-35f73f0d7331 [Candidate]" term=2 - testing.go:149: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.676Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: entering candidate state: node="Node at 724f68de-f93b-7654-af86-35f73f0d7331 [Candidate]" term=2 -testLoggerAdapter verbose: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.676Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: election won: tally=2 - testing.go:149: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.676Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: election won: tally=2 -testLoggerAdapter verbose: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.676Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: entering leader state: leader="Node at 724f68de-f93b-7654-af86-35f73f0d7331 [Leader]" - testing.go:149: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.676Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: entering leader state: leader="Node at 724f68de-f93b-7654-af86-35f73f0d7331 [Leader]" -testLoggerAdapter verbose: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.676Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: added peer, starting replication: peer=server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a - testing.go:149: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.676Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: added peer, starting replication: peer=server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a -testLoggerAdapter verbose: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.676Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: added peer, starting replication: peer=server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd - testing.go:149: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.676Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: added peer, starting replication: peer=server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd -testLoggerAdapter verbose: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.676Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: pipelining replication: peer="{Voter server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a 66db94f2-09bd-b5b9-1c59-dfab786a9e6a}" - testing.go:149: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.676Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: pipelining replication: peer="{Voter server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a 66db94f2-09bd-b5b9-1c59-dfab786a9e6a}" -testLoggerAdapter verbose: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.676Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: pipelining replication: peer="{Voter server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd 09dd28b7-7554-e5d9-369d-d0beeb9ccdcd}" - testing.go:149: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.676Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: pipelining replication: peer="{Voter server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd 09dd28b7-7554-e5d9-369d-d0beeb9ccdcd}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:00.781Z [INFO] stable state for Leader reached at 2021-07-10 01:05:00.676678028 +0000 UTC m=+6.064836018 (1 nodes), 82.795313ms from start of poll, 83.835964ms from cluster start. Timeout at 2021-07-10 01:05:00.781704375 +0000 UTC m=+6.169862345, 105.026327ms after stability - testing.go:149: cluster: 2021-07-10T01:05:00.781Z [INFO] stable state for Leader reached at 2021-07-10 01:05:00.676678028 +0000 UTC m=+6.064836018 (1 nodes), 82.795313ms from start of poll, 83.835964ms from cluster start. Timeout at 2021-07-10 01:05:00.781704375 +0000 UTC m=+6.169862345, 105.026327ms after stability -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:00.781Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-07-10T01:05:00.781Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:00.887Z [INFO] stable state for Follower reached at 2021-07-10 01:05:00.781942018 +0000 UTC m=+6.170099988 (2 nodes), 2.361µs from start of poll, 189.099934ms from cluster start. Timeout at 2021-07-10 01:05:00.886985905 +0000 UTC m=+6.275143895, 105.043907ms after stability - testing.go:149: cluster: 2021-07-10T01:05:00.887Z [INFO] stable state for Follower reached at 2021-07-10 01:05:00.781942018 +0000 UTC m=+6.170099988 (2 nodes), 2.361µs from start of poll, 189.099934ms from cluster start. Timeout at 2021-07-10 01:05:00.886985905 +0000 UTC m=+6.275143895, 105.043907ms after stability -testLoggerAdapter verbose: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.887Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: aborting pipeline replication: peer="{Voter server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd 09dd28b7-7554-e5d9-369d-d0beeb9ccdcd}" - testing.go:149: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.887Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: aborting pipeline replication: peer="{Voter server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd 09dd28b7-7554-e5d9-369d-d0beeb9ccdcd}" -testLoggerAdapter verbose: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.887Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: aborting pipeline replication: peer="{Voter server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a 66db94f2-09bd-b5b9-1c59-dfab786a9e6a}" - testing.go:149: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.887Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: aborting pipeline replication: peer="{Voter server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a 66db94f2-09bd-b5b9-1c59-dfab786a9e6a}" ---- PASS: TestRaft_ApplyNonLeader (0.30s) +testLoggerAdapter verbose: server-41cf36f5-7270-6ed5-f8c3-1009386dd5d8: 2022-08-12T07:30:05.145Z [INFO] server-41cf36f5-7270-6ed5-f8c3-1009386dd5d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-41cf36f5-7270-6ed5-f8c3-1009386dd5d8 Address:41cf36f5-7270-6ed5-f8c3-1009386dd5d8} {Suffrage:Voter ID:server-05d31422-bf12-0d52-73d2-75cd31e6af71 Address:05d31422-bf12-0d52-73d2-75cd31e6af71} {Suffrage:Voter ID:server-d9cc77f8-8ffa-4d59-176b-615164af52c3 Address:d9cc77f8-8ffa-4d59-176b-615164af52c3}]" + testing.go:149: server-41cf36f5-7270-6ed5-f8c3-1009386dd5d8: 2022-08-12T07:30:05.145Z [INFO] server-41cf36f5-7270-6ed5-f8c3-1009386dd5d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-41cf36f5-7270-6ed5-f8c3-1009386dd5d8 Address:41cf36f5-7270-6ed5-f8c3-1009386dd5d8} {Suffrage:Voter ID:server-05d31422-bf12-0d52-73d2-75cd31e6af71 Address:05d31422-bf12-0d52-73d2-75cd31e6af71} {Suffrage:Voter ID:server-d9cc77f8-8ffa-4d59-176b-615164af52c3 Address:d9cc77f8-8ffa-4d59-176b-615164af52c3}]" +testLoggerAdapter verbose: server-41cf36f5-7270-6ed5-f8c3-1009386dd5d8: 2022-08-12T07:30:05.145Z [INFO] server-41cf36f5-7270-6ed5-f8c3-1009386dd5d8: entering follower state: follower="Node at 41cf36f5-7270-6ed5-f8c3-1009386dd5d8 [Follower]" leader= + testing.go:149: server-41cf36f5-7270-6ed5-f8c3-1009386dd5d8: 2022-08-12T07:30:05.145Z [INFO] server-41cf36f5-7270-6ed5-f8c3-1009386dd5d8: entering follower state: follower="Node at 41cf36f5-7270-6ed5-f8c3-1009386dd5d8 [Follower]" leader= +testLoggerAdapter verbose: server-05d31422-bf12-0d52-73d2-75cd31e6af71: 2022-08-12T07:30:05.146Z [INFO] server-05d31422-bf12-0d52-73d2-75cd31e6af71: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-41cf36f5-7270-6ed5-f8c3-1009386dd5d8 Address:41cf36f5-7270-6ed5-f8c3-1009386dd5d8} {Suffrage:Voter ID:server-05d31422-bf12-0d52-73d2-75cd31e6af71 Address:05d31422-bf12-0d52-73d2-75cd31e6af71} {Suffrage:Voter ID:server-d9cc77f8-8ffa-4d59-176b-615164af52c3 Address:d9cc77f8-8ffa-4d59-176b-615164af52c3}]" + testing.go:149: server-05d31422-bf12-0d52-73d2-75cd31e6af71: 2022-08-12T07:30:05.146Z [INFO] server-05d31422-bf12-0d52-73d2-75cd31e6af71: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-41cf36f5-7270-6ed5-f8c3-1009386dd5d8 Address:41cf36f5-7270-6ed5-f8c3-1009386dd5d8} {Suffrage:Voter ID:server-05d31422-bf12-0d52-73d2-75cd31e6af71 Address:05d31422-bf12-0d52-73d2-75cd31e6af71} {Suffrage:Voter ID:server-d9cc77f8-8ffa-4d59-176b-615164af52c3 Address:d9cc77f8-8ffa-4d59-176b-615164af52c3}]" +testLoggerAdapter verbose: server-05d31422-bf12-0d52-73d2-75cd31e6af71: 2022-08-12T07:30:05.146Z [INFO] server-05d31422-bf12-0d52-73d2-75cd31e6af71: entering follower state: follower="Node at 05d31422-bf12-0d52-73d2-75cd31e6af71 [Follower]" leader= + testing.go:149: server-05d31422-bf12-0d52-73d2-75cd31e6af71: 2022-08-12T07:30:05.146Z [INFO] server-05d31422-bf12-0d52-73d2-75cd31e6af71: entering follower state: follower="Node at 05d31422-bf12-0d52-73d2-75cd31e6af71 [Follower]" leader= +testLoggerAdapter verbose: server-d9cc77f8-8ffa-4d59-176b-615164af52c3: 2022-08-12T07:30:05.146Z [INFO] server-d9cc77f8-8ffa-4d59-176b-615164af52c3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-41cf36f5-7270-6ed5-f8c3-1009386dd5d8 Address:41cf36f5-7270-6ed5-f8c3-1009386dd5d8} {Suffrage:Voter ID:server-05d31422-bf12-0d52-73d2-75cd31e6af71 Address:05d31422-bf12-0d52-73d2-75cd31e6af71} {Suffrage:Voter ID:server-d9cc77f8-8ffa-4d59-176b-615164af52c3 Address:d9cc77f8-8ffa-4d59-176b-615164af52c3}]" + testing.go:149: server-d9cc77f8-8ffa-4d59-176b-615164af52c3: 2022-08-12T07:30:05.146Z [INFO] server-d9cc77f8-8ffa-4d59-176b-615164af52c3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-41cf36f5-7270-6ed5-f8c3-1009386dd5d8 Address:41cf36f5-7270-6ed5-f8c3-1009386dd5d8} {Suffrage:Voter ID:server-05d31422-bf12-0d52-73d2-75cd31e6af71 Address:05d31422-bf12-0d52-73d2-75cd31e6af71} {Suffrage:Voter ID:server-d9cc77f8-8ffa-4d59-176b-615164af52c3 Address:d9cc77f8-8ffa-4d59-176b-615164af52c3}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:05.146Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:05.146Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-d9cc77f8-8ffa-4d59-176b-615164af52c3: 2022-08-12T07:30:05.146Z [INFO] server-d9cc77f8-8ffa-4d59-176b-615164af52c3: entering follower state: follower="Node at d9cc77f8-8ffa-4d59-176b-615164af52c3 [Follower]" leader= + testing.go:149: server-d9cc77f8-8ffa-4d59-176b-615164af52c3: 2022-08-12T07:30:05.146Z [INFO] server-d9cc77f8-8ffa-4d59-176b-615164af52c3: entering follower state: follower="Node at d9cc77f8-8ffa-4d59-176b-615164af52c3 [Follower]" leader= +testLoggerAdapter verbose: server-05d31422-bf12-0d52-73d2-75cd31e6af71: 2022-08-12T07:30:05.203Z [WARN] server-05d31422-bf12-0d52-73d2-75cd31e6af71: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-05d31422-bf12-0d52-73d2-75cd31e6af71: 2022-08-12T07:30:05.203Z [WARN] server-05d31422-bf12-0d52-73d2-75cd31e6af71: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-05d31422-bf12-0d52-73d2-75cd31e6af71: 2022-08-12T07:30:05.203Z [INFO] server-05d31422-bf12-0d52-73d2-75cd31e6af71: entering candidate state: node="Node at 05d31422-bf12-0d52-73d2-75cd31e6af71 [Candidate]" term=2 + testing.go:149: server-05d31422-bf12-0d52-73d2-75cd31e6af71: 2022-08-12T07:30:05.203Z [INFO] server-05d31422-bf12-0d52-73d2-75cd31e6af71: entering candidate state: node="Node at 05d31422-bf12-0d52-73d2-75cd31e6af71 [Candidate]" term=2 +testLoggerAdapter verbose: server-d9cc77f8-8ffa-4d59-176b-615164af52c3: 2022-08-12T07:30:05.203Z [WARN] server-d9cc77f8-8ffa-4d59-176b-615164af52c3: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-d9cc77f8-8ffa-4d59-176b-615164af52c3: 2022-08-12T07:30:05.203Z [WARN] server-d9cc77f8-8ffa-4d59-176b-615164af52c3: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-d9cc77f8-8ffa-4d59-176b-615164af52c3: 2022-08-12T07:30:05.203Z [INFO] server-d9cc77f8-8ffa-4d59-176b-615164af52c3: entering candidate state: node="Node at d9cc77f8-8ffa-4d59-176b-615164af52c3 [Candidate]" term=2 +testLoggerAdapter verbose: server-05d31422-bf12-0d52-73d2-75cd31e6af71: 2022-08-12T07:30:05.203Z [INFO] server-05d31422-bf12-0d52-73d2-75cd31e6af71: election won: tally=2 + testing.go:149: server-d9cc77f8-8ffa-4d59-176b-615164af52c3: 2022-08-12T07:30:05.203Z [INFO] server-d9cc77f8-8ffa-4d59-176b-615164af52c3: entering candidate state: node="Node at d9cc77f8-8ffa-4d59-176b-615164af52c3 [Candidate]" term=2 + testing.go:149: server-05d31422-bf12-0d52-73d2-75cd31e6af71: 2022-08-12T07:30:05.203Z [INFO] server-05d31422-bf12-0d52-73d2-75cd31e6af71: election won: tally=2 +testLoggerAdapter verbose: server-05d31422-bf12-0d52-73d2-75cd31e6af71: 2022-08-12T07:30:05.203Z [INFO] server-05d31422-bf12-0d52-73d2-75cd31e6af71: entering leader state: leader="Node at 05d31422-bf12-0d52-73d2-75cd31e6af71 [Leader]" +testLoggerAdapter verbose: server-d9cc77f8-8ffa-4d59-176b-615164af52c3: 2022-08-12T07:30:05.203Z [INFO] server-d9cc77f8-8ffa-4d59-176b-615164af52c3: duplicate requestVote for same term: term=2 + testing.go:149: server-05d31422-bf12-0d52-73d2-75cd31e6af71: 2022-08-12T07:30:05.203Z [INFO] server-05d31422-bf12-0d52-73d2-75cd31e6af71: entering leader state: leader="Node at 05d31422-bf12-0d52-73d2-75cd31e6af71 [Leader]" +testLoggerAdapter verbose: server-05d31422-bf12-0d52-73d2-75cd31e6af71: 2022-08-12T07:30:05.203Z [INFO] server-05d31422-bf12-0d52-73d2-75cd31e6af71: added peer, starting replication: peer=server-41cf36f5-7270-6ed5-f8c3-1009386dd5d8 + testing.go:149: server-05d31422-bf12-0d52-73d2-75cd31e6af71: 2022-08-12T07:30:05.203Z [INFO] server-05d31422-bf12-0d52-73d2-75cd31e6af71: added peer, starting replication: peer=server-41cf36f5-7270-6ed5-f8c3-1009386dd5d8 +testLoggerAdapter verbose: server-41cf36f5-7270-6ed5-f8c3-1009386dd5d8: 2022-08-12T07:30:05.204Z [INFO] server-41cf36f5-7270-6ed5-f8c3-1009386dd5d8: duplicate requestVote for same term: term=2 +testLoggerAdapter verbose: server-05d31422-bf12-0d52-73d2-75cd31e6af71: 2022-08-12T07:30:05.204Z [INFO] server-05d31422-bf12-0d52-73d2-75cd31e6af71: added peer, starting replication: peer=server-d9cc77f8-8ffa-4d59-176b-615164af52c3 + testing.go:149: server-d9cc77f8-8ffa-4d59-176b-615164af52c3: 2022-08-12T07:30:05.203Z [INFO] server-d9cc77f8-8ffa-4d59-176b-615164af52c3: duplicate requestVote for same term: term=2 + testing.go:149: server-05d31422-bf12-0d52-73d2-75cd31e6af71: 2022-08-12T07:30:05.204Z [INFO] server-05d31422-bf12-0d52-73d2-75cd31e6af71: added peer, starting replication: peer=server-d9cc77f8-8ffa-4d59-176b-615164af52c3 +testLoggerAdapter verbose: server-05d31422-bf12-0d52-73d2-75cd31e6af71: 2022-08-12T07:30:05.204Z [WARN] server-05d31422-bf12-0d52-73d2-75cd31e6af71: rejecting vote request since we have a leader: from=d9cc77f8-8ffa-4d59-176b-615164af52c3 leader=05d31422-bf12-0d52-73d2-75cd31e6af71 + testing.go:149: server-05d31422-bf12-0d52-73d2-75cd31e6af71: 2022-08-12T07:30:05.204Z [WARN] server-05d31422-bf12-0d52-73d2-75cd31e6af71: rejecting vote request since we have a leader: from=d9cc77f8-8ffa-4d59-176b-615164af52c3 leader=05d31422-bf12-0d52-73d2-75cd31e6af71 + testing.go:149: server-41cf36f5-7270-6ed5-f8c3-1009386dd5d8: 2022-08-12T07:30:05.204Z [INFO] server-41cf36f5-7270-6ed5-f8c3-1009386dd5d8: duplicate requestVote for same term: term=2 +testLoggerAdapter verbose: server-05d31422-bf12-0d52-73d2-75cd31e6af71: 2022-08-12T07:30:05.204Z [INFO] server-05d31422-bf12-0d52-73d2-75cd31e6af71: pipelining replication: peer="{Voter server-41cf36f5-7270-6ed5-f8c3-1009386dd5d8 41cf36f5-7270-6ed5-f8c3-1009386dd5d8}" + testing.go:149: server-05d31422-bf12-0d52-73d2-75cd31e6af71: 2022-08-12T07:30:05.204Z [INFO] server-05d31422-bf12-0d52-73d2-75cd31e6af71: pipelining replication: peer="{Voter server-41cf36f5-7270-6ed5-f8c3-1009386dd5d8 41cf36f5-7270-6ed5-f8c3-1009386dd5d8}" +testLoggerAdapter verbose: server-d9cc77f8-8ffa-4d59-176b-615164af52c3: 2022-08-12T07:30:05.204Z [INFO] server-d9cc77f8-8ffa-4d59-176b-615164af52c3: entering follower state: follower="Node at d9cc77f8-8ffa-4d59-176b-615164af52c3 [Follower]" leader=05d31422-bf12-0d52-73d2-75cd31e6af71 + testing.go:149: server-d9cc77f8-8ffa-4d59-176b-615164af52c3: 2022-08-12T07:30:05.204Z [INFO] server-d9cc77f8-8ffa-4d59-176b-615164af52c3: entering follower state: follower="Node at d9cc77f8-8ffa-4d59-176b-615164af52c3 [Follower]" leader=05d31422-bf12-0d52-73d2-75cd31e6af71 +testLoggerAdapter verbose: server-05d31422-bf12-0d52-73d2-75cd31e6af71: 2022-08-12T07:30:05.204Z [INFO] server-05d31422-bf12-0d52-73d2-75cd31e6af71: pipelining replication: peer="{Voter server-d9cc77f8-8ffa-4d59-176b-615164af52c3 d9cc77f8-8ffa-4d59-176b-615164af52c3}" + testing.go:149: server-05d31422-bf12-0d52-73d2-75cd31e6af71: 2022-08-12T07:30:05.204Z [INFO] server-05d31422-bf12-0d52-73d2-75cd31e6af71: pipelining replication: peer="{Voter server-d9cc77f8-8ffa-4d59-176b-615164af52c3 d9cc77f8-8ffa-4d59-176b-615164af52c3}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:05.309Z [INFO] stable state for Leader reached at 2022-08-12 07:30:05.20439812 +0000 UTC m=+6.161084601 (1 nodes), 57.90666ms from start of poll, 58.86042ms from cluster start. Timeout at 2022-08-12 07:30:05.30946256 +0000 UTC m=+6.266149041, 105.06444ms after stability + testing.go:149: cluster: 2022-08-12T07:30:05.309Z [INFO] stable state for Leader reached at 2022-08-12 07:30:05.20439812 +0000 UTC m=+6.161084601 (1 nodes), 57.90666ms from start of poll, 58.86042ms from cluster start. Timeout at 2022-08-12 07:30:05.30946256 +0000 UTC m=+6.266149041, 105.06444ms after stability +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:05.309Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2022-08-12T07:30:05.309Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:05.414Z [INFO] stable state for Follower reached at 2022-08-12 07:30:05.30976866 +0000 UTC m=+6.266455141 (2 nodes), 3.18µs from start of poll, 164.23096ms from cluster start. Timeout at 2022-08-12 07:30:05.41483744 +0000 UTC m=+6.371523921, 105.06878ms after stability + testing.go:149: cluster: 2022-08-12T07:30:05.414Z [INFO] stable state for Follower reached at 2022-08-12 07:30:05.30976866 +0000 UTC m=+6.266455141 (2 nodes), 3.18µs from start of poll, 164.23096ms from cluster start. Timeout at 2022-08-12 07:30:05.41483744 +0000 UTC m=+6.371523921, 105.06878ms after stability +testLoggerAdapter verbose: server-05d31422-bf12-0d52-73d2-75cd31e6af71: 2022-08-12T07:30:05.415Z [INFO] server-05d31422-bf12-0d52-73d2-75cd31e6af71: aborting pipeline replication: peer="{Voter server-d9cc77f8-8ffa-4d59-176b-615164af52c3 d9cc77f8-8ffa-4d59-176b-615164af52c3}" + testing.go:149: server-05d31422-bf12-0d52-73d2-75cd31e6af71: 2022-08-12T07:30:05.415Z [INFO] server-05d31422-bf12-0d52-73d2-75cd31e6af71: aborting pipeline replication: peer="{Voter server-d9cc77f8-8ffa-4d59-176b-615164af52c3 d9cc77f8-8ffa-4d59-176b-615164af52c3}" +testLoggerAdapter verbose: server-05d31422-bf12-0d52-73d2-75cd31e6af71: 2022-08-12T07:30:05.415Z [INFO] server-05d31422-bf12-0d52-73d2-75cd31e6af71: aborting pipeline replication: peer="{Voter server-41cf36f5-7270-6ed5-f8c3-1009386dd5d8 41cf36f5-7270-6ed5-f8c3-1009386dd5d8}" + testing.go:149: server-05d31422-bf12-0d52-73d2-75cd31e6af71: 2022-08-12T07:30:05.415Z [INFO] server-05d31422-bf12-0d52-73d2-75cd31e6af71: aborting pipeline replication: peer="{Voter server-41cf36f5-7270-6ed5-f8c3-1009386dd5d8 41cf36f5-7270-6ed5-f8c3-1009386dd5d8}" +--- PASS: TestRaft_ApplyNonLeader (0.27s) === RUN TestRaft_ApplyConcurrent -testLoggerAdapter verbose: server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3: 2021-07-10T01:05:00.888Z [INFO] server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3 Address:1e1d953e-3dae-f134-89b6-6cbcda5c8de3} {Suffrage:Voter ID:server-a827d17c-f21d-780f-dd5d-c9471da2e059 Address:a827d17c-f21d-780f-dd5d-c9471da2e059} {Suffrage:Voter ID:server-dc316ee9-f2eb-238f-1020-e5f5db597b81 Address:dc316ee9-f2eb-238f-1020-e5f5db597b81}]" - testing.go:149: server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3: 2021-07-10T01:05:00.888Z [INFO] server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3 Address:1e1d953e-3dae-f134-89b6-6cbcda5c8de3} {Suffrage:Voter ID:server-a827d17c-f21d-780f-dd5d-c9471da2e059 Address:a827d17c-f21d-780f-dd5d-c9471da2e059} {Suffrage:Voter ID:server-dc316ee9-f2eb-238f-1020-e5f5db597b81 Address:dc316ee9-f2eb-238f-1020-e5f5db597b81}]" -testLoggerAdapter verbose: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:00.889Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3 Address:1e1d953e-3dae-f134-89b6-6cbcda5c8de3} {Suffrage:Voter ID:server-a827d17c-f21d-780f-dd5d-c9471da2e059 Address:a827d17c-f21d-780f-dd5d-c9471da2e059} {Suffrage:Voter ID:server-dc316ee9-f2eb-238f-1020-e5f5db597b81 Address:dc316ee9-f2eb-238f-1020-e5f5db597b81}]" - testing.go:149: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:00.889Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3 Address:1e1d953e-3dae-f134-89b6-6cbcda5c8de3} {Suffrage:Voter ID:server-a827d17c-f21d-780f-dd5d-c9471da2e059 Address:a827d17c-f21d-780f-dd5d-c9471da2e059} {Suffrage:Voter ID:server-dc316ee9-f2eb-238f-1020-e5f5db597b81 Address:dc316ee9-f2eb-238f-1020-e5f5db597b81}]" -testLoggerAdapter verbose: server-dc316ee9-f2eb-238f-1020-e5f5db597b81: 2021-07-10T01:05:00.889Z [INFO] server-dc316ee9-f2eb-238f-1020-e5f5db597b81: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3 Address:1e1d953e-3dae-f134-89b6-6cbcda5c8de3} {Suffrage:Voter ID:server-a827d17c-f21d-780f-dd5d-c9471da2e059 Address:a827d17c-f21d-780f-dd5d-c9471da2e059} {Suffrage:Voter ID:server-dc316ee9-f2eb-238f-1020-e5f5db597b81 Address:dc316ee9-f2eb-238f-1020-e5f5db597b81}]" - testing.go:149: server-dc316ee9-f2eb-238f-1020-e5f5db597b81: 2021-07-10T01:05:00.889Z [INFO] server-dc316ee9-f2eb-238f-1020-e5f5db597b81: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3 Address:1e1d953e-3dae-f134-89b6-6cbcda5c8de3} {Suffrage:Voter ID:server-a827d17c-f21d-780f-dd5d-c9471da2e059 Address:a827d17c-f21d-780f-dd5d-c9471da2e059} {Suffrage:Voter ID:server-dc316ee9-f2eb-238f-1020-e5f5db597b81 Address:dc316ee9-f2eb-238f-1020-e5f5db597b81}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:00.889Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:00.889Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3: 2021-07-10T01:05:00.889Z [INFO] server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3: entering follower state: follower="Node at 1e1d953e-3dae-f134-89b6-6cbcda5c8de3 [Follower]" leader= - testing.go:149: server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3: 2021-07-10T01:05:00.889Z [INFO] server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3: entering follower state: follower="Node at 1e1d953e-3dae-f134-89b6-6cbcda5c8de3 [Follower]" leader= -testLoggerAdapter verbose: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:00.889Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: entering follower state: follower="Node at a827d17c-f21d-780f-dd5d-c9471da2e059 [Follower]" leader= - testing.go:149: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:00.889Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: entering follower state: follower="Node at a827d17c-f21d-780f-dd5d-c9471da2e059 [Follower]" leader= -testLoggerAdapter verbose: server-dc316ee9-f2eb-238f-1020-e5f5db597b81: 2021-07-10T01:05:00.889Z [INFO] server-dc316ee9-f2eb-238f-1020-e5f5db597b81: entering follower state: follower="Node at dc316ee9-f2eb-238f-1020-e5f5db597b81 [Follower]" leader= - testing.go:149: server-dc316ee9-f2eb-238f-1020-e5f5db597b81: 2021-07-10T01:05:00.889Z [INFO] server-dc316ee9-f2eb-238f-1020-e5f5db597b81: entering follower state: follower="Node at dc316ee9-f2eb-238f-1020-e5f5db597b81 [Follower]" leader= -testLoggerAdapter verbose: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.026Z [WARN] server-a827d17c-f21d-780f-dd5d-c9471da2e059: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.026Z [WARN] server-a827d17c-f21d-780f-dd5d-c9471da2e059: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.026Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: entering candidate state: node="Node at a827d17c-f21d-780f-dd5d-c9471da2e059 [Candidate]" term=2 - testing.go:149: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.026Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: entering candidate state: node="Node at a827d17c-f21d-780f-dd5d-c9471da2e059 [Candidate]" term=2 -testLoggerAdapter verbose: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.027Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: election won: tally=2 - testing.go:149: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.027Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: election won: tally=2 -testLoggerAdapter verbose: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.027Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: entering leader state: leader="Node at a827d17c-f21d-780f-dd5d-c9471da2e059 [Leader]" - testing.go:149: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.027Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: entering leader state: leader="Node at a827d17c-f21d-780f-dd5d-c9471da2e059 [Leader]" -testLoggerAdapter verbose: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.027Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: added peer, starting replication: peer=server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3 - testing.go:149: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.027Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: added peer, starting replication: peer=server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3 -testLoggerAdapter verbose: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.027Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: added peer, starting replication: peer=server-dc316ee9-f2eb-238f-1020-e5f5db597b81 - testing.go:149: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.027Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: added peer, starting replication: peer=server-dc316ee9-f2eb-238f-1020-e5f5db597b81 -testLoggerAdapter verbose: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.027Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: pipelining replication: peer="{Voter server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3 1e1d953e-3dae-f134-89b6-6cbcda5c8de3}" - testing.go:149: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.027Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: pipelining replication: peer="{Voter server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3 1e1d953e-3dae-f134-89b6-6cbcda5c8de3}" -testLoggerAdapter verbose: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.027Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: pipelining replication: peer="{Voter server-dc316ee9-f2eb-238f-1020-e5f5db597b81 dc316ee9-f2eb-238f-1020-e5f5db597b81}" - testing.go:149: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.027Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: pipelining replication: peer="{Voter server-dc316ee9-f2eb-238f-1020-e5f5db597b81 dc316ee9-f2eb-238f-1020-e5f5db597b81}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:01.232Z [INFO] stable state for Leader reached at 2021-07-10 01:05:01.027322584 +0000 UTC m=+6.415480574 (1 nodes), 137.796732ms from start of poll, 138.738142ms from cluster start. Timeout at 2021-07-10 01:05:01.232389546 +0000 UTC m=+6.620547496, 205.066922ms after stability - testing.go:149: cluster: 2021-07-10T01:05:01.232Z [INFO] stable state for Leader reached at 2021-07-10 01:05:01.027322584 +0000 UTC m=+6.415480574 (1 nodes), 137.796732ms from start of poll, 138.738142ms from cluster start. Timeout at 2021-07-10 01:05:01.232389546 +0000 UTC m=+6.620547496, 205.066922ms after stability -testLoggerAdapter verbose: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.245Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: aborting pipeline replication: peer="{Voter server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3 1e1d953e-3dae-f134-89b6-6cbcda5c8de3}" - testing.go:149: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.245Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: aborting pipeline replication: peer="{Voter server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3 1e1d953e-3dae-f134-89b6-6cbcda5c8de3}" -testLoggerAdapter verbose: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.245Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: aborting pipeline replication: peer="{Voter server-dc316ee9-f2eb-238f-1020-e5f5db597b81 dc316ee9-f2eb-238f-1020-e5f5db597b81}" - testing.go:149: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.245Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: aborting pipeline replication: peer="{Voter server-dc316ee9-f2eb-238f-1020-e5f5db597b81 dc316ee9-f2eb-238f-1020-e5f5db597b81}" ---- PASS: TestRaft_ApplyConcurrent (0.36s) +testLoggerAdapter verbose: server-09c41644-0456-7121-a771-4f78e759823a: 2022-08-12T07:30:05.416Z [INFO] server-09c41644-0456-7121-a771-4f78e759823a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-09c41644-0456-7121-a771-4f78e759823a Address:09c41644-0456-7121-a771-4f78e759823a} {Suffrage:Voter ID:server-42638d86-81a1-466d-fa2a-4874cdb201b8 Address:42638d86-81a1-466d-fa2a-4874cdb201b8} {Suffrage:Voter ID:server-f8457807-7874-1562-78aa-ee8519f6e29b Address:f8457807-7874-1562-78aa-ee8519f6e29b}]" + testing.go:149: server-09c41644-0456-7121-a771-4f78e759823a: 2022-08-12T07:30:05.416Z [INFO] server-09c41644-0456-7121-a771-4f78e759823a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-09c41644-0456-7121-a771-4f78e759823a Address:09c41644-0456-7121-a771-4f78e759823a} {Suffrage:Voter ID:server-42638d86-81a1-466d-fa2a-4874cdb201b8 Address:42638d86-81a1-466d-fa2a-4874cdb201b8} {Suffrage:Voter ID:server-f8457807-7874-1562-78aa-ee8519f6e29b Address:f8457807-7874-1562-78aa-ee8519f6e29b}]" +testLoggerAdapter verbose: server-09c41644-0456-7121-a771-4f78e759823a: 2022-08-12T07:30:05.416Z [INFO] server-09c41644-0456-7121-a771-4f78e759823a: entering follower state: follower="Node at 09c41644-0456-7121-a771-4f78e759823a [Follower]" leader= + testing.go:149: server-09c41644-0456-7121-a771-4f78e759823a: 2022-08-12T07:30:05.416Z [INFO] server-09c41644-0456-7121-a771-4f78e759823a: entering follower state: follower="Node at 09c41644-0456-7121-a771-4f78e759823a [Follower]" leader= +testLoggerAdapter verbose: server-42638d86-81a1-466d-fa2a-4874cdb201b8: 2022-08-12T07:30:05.416Z [INFO] server-42638d86-81a1-466d-fa2a-4874cdb201b8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-09c41644-0456-7121-a771-4f78e759823a Address:09c41644-0456-7121-a771-4f78e759823a} {Suffrage:Voter ID:server-42638d86-81a1-466d-fa2a-4874cdb201b8 Address:42638d86-81a1-466d-fa2a-4874cdb201b8} {Suffrage:Voter ID:server-f8457807-7874-1562-78aa-ee8519f6e29b Address:f8457807-7874-1562-78aa-ee8519f6e29b}]" + testing.go:149: server-42638d86-81a1-466d-fa2a-4874cdb201b8: 2022-08-12T07:30:05.416Z [INFO] server-42638d86-81a1-466d-fa2a-4874cdb201b8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-09c41644-0456-7121-a771-4f78e759823a Address:09c41644-0456-7121-a771-4f78e759823a} {Suffrage:Voter ID:server-42638d86-81a1-466d-fa2a-4874cdb201b8 Address:42638d86-81a1-466d-fa2a-4874cdb201b8} {Suffrage:Voter ID:server-f8457807-7874-1562-78aa-ee8519f6e29b Address:f8457807-7874-1562-78aa-ee8519f6e29b}]" +testLoggerAdapter verbose: server-42638d86-81a1-466d-fa2a-4874cdb201b8: 2022-08-12T07:30:05.416Z [INFO] server-42638d86-81a1-466d-fa2a-4874cdb201b8: entering follower state: follower="Node at 42638d86-81a1-466d-fa2a-4874cdb201b8 [Follower]" leader= + testing.go:149: server-42638d86-81a1-466d-fa2a-4874cdb201b8: 2022-08-12T07:30:05.416Z [INFO] server-42638d86-81a1-466d-fa2a-4874cdb201b8: entering follower state: follower="Node at 42638d86-81a1-466d-fa2a-4874cdb201b8 [Follower]" leader= +testLoggerAdapter verbose: server-f8457807-7874-1562-78aa-ee8519f6e29b: 2022-08-12T07:30:05.417Z [INFO] server-f8457807-7874-1562-78aa-ee8519f6e29b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-09c41644-0456-7121-a771-4f78e759823a Address:09c41644-0456-7121-a771-4f78e759823a} {Suffrage:Voter ID:server-42638d86-81a1-466d-fa2a-4874cdb201b8 Address:42638d86-81a1-466d-fa2a-4874cdb201b8} {Suffrage:Voter ID:server-f8457807-7874-1562-78aa-ee8519f6e29b Address:f8457807-7874-1562-78aa-ee8519f6e29b}]" + testing.go:149: server-f8457807-7874-1562-78aa-ee8519f6e29b: 2022-08-12T07:30:05.417Z [INFO] server-f8457807-7874-1562-78aa-ee8519f6e29b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-09c41644-0456-7121-a771-4f78e759823a Address:09c41644-0456-7121-a771-4f78e759823a} {Suffrage:Voter ID:server-42638d86-81a1-466d-fa2a-4874cdb201b8 Address:42638d86-81a1-466d-fa2a-4874cdb201b8} {Suffrage:Voter ID:server-f8457807-7874-1562-78aa-ee8519f6e29b Address:f8457807-7874-1562-78aa-ee8519f6e29b}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:05.417Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:05.417Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-f8457807-7874-1562-78aa-ee8519f6e29b: 2022-08-12T07:30:05.417Z [INFO] server-f8457807-7874-1562-78aa-ee8519f6e29b: entering follower state: follower="Node at f8457807-7874-1562-78aa-ee8519f6e29b [Follower]" leader= + testing.go:149: server-f8457807-7874-1562-78aa-ee8519f6e29b: 2022-08-12T07:30:05.417Z [INFO] server-f8457807-7874-1562-78aa-ee8519f6e29b: entering follower state: follower="Node at f8457807-7874-1562-78aa-ee8519f6e29b [Follower]" leader= +testLoggerAdapter verbose: server-42638d86-81a1-466d-fa2a-4874cdb201b8: 2022-08-12T07:30:05.517Z [WARN] server-42638d86-81a1-466d-fa2a-4874cdb201b8: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-42638d86-81a1-466d-fa2a-4874cdb201b8: 2022-08-12T07:30:05.517Z [WARN] server-42638d86-81a1-466d-fa2a-4874cdb201b8: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-42638d86-81a1-466d-fa2a-4874cdb201b8: 2022-08-12T07:30:05.517Z [INFO] server-42638d86-81a1-466d-fa2a-4874cdb201b8: entering candidate state: node="Node at 42638d86-81a1-466d-fa2a-4874cdb201b8 [Candidate]" term=2 + testing.go:149: server-42638d86-81a1-466d-fa2a-4874cdb201b8: 2022-08-12T07:30:05.517Z [INFO] server-42638d86-81a1-466d-fa2a-4874cdb201b8: entering candidate state: node="Node at 42638d86-81a1-466d-fa2a-4874cdb201b8 [Candidate]" term=2 +testLoggerAdapter verbose: server-42638d86-81a1-466d-fa2a-4874cdb201b8: 2022-08-12T07:30:05.517Z [INFO] server-42638d86-81a1-466d-fa2a-4874cdb201b8: election won: tally=2 + testing.go:149: server-42638d86-81a1-466d-fa2a-4874cdb201b8: 2022-08-12T07:30:05.517Z [INFO] server-42638d86-81a1-466d-fa2a-4874cdb201b8: election won: tally=2 +testLoggerAdapter verbose: server-42638d86-81a1-466d-fa2a-4874cdb201b8: 2022-08-12T07:30:05.517Z [INFO] server-42638d86-81a1-466d-fa2a-4874cdb201b8: entering leader state: leader="Node at 42638d86-81a1-466d-fa2a-4874cdb201b8 [Leader]" + testing.go:149: server-42638d86-81a1-466d-fa2a-4874cdb201b8: 2022-08-12T07:30:05.517Z [INFO] server-42638d86-81a1-466d-fa2a-4874cdb201b8: entering leader state: leader="Node at 42638d86-81a1-466d-fa2a-4874cdb201b8 [Leader]" +testLoggerAdapter verbose: server-42638d86-81a1-466d-fa2a-4874cdb201b8: 2022-08-12T07:30:05.518Z [INFO] server-42638d86-81a1-466d-fa2a-4874cdb201b8: added peer, starting replication: peer=server-09c41644-0456-7121-a771-4f78e759823a + testing.go:149: server-42638d86-81a1-466d-fa2a-4874cdb201b8: 2022-08-12T07:30:05.518Z [INFO] server-42638d86-81a1-466d-fa2a-4874cdb201b8: added peer, starting replication: peer=server-09c41644-0456-7121-a771-4f78e759823a +testLoggerAdapter verbose: server-42638d86-81a1-466d-fa2a-4874cdb201b8: 2022-08-12T07:30:05.518Z [INFO] server-42638d86-81a1-466d-fa2a-4874cdb201b8: added peer, starting replication: peer=server-f8457807-7874-1562-78aa-ee8519f6e29b + testing.go:149: server-42638d86-81a1-466d-fa2a-4874cdb201b8: 2022-08-12T07:30:05.518Z [INFO] server-42638d86-81a1-466d-fa2a-4874cdb201b8: added peer, starting replication: peer=server-f8457807-7874-1562-78aa-ee8519f6e29b +testLoggerAdapter verbose: server-42638d86-81a1-466d-fa2a-4874cdb201b8: 2022-08-12T07:30:05.518Z [INFO] server-42638d86-81a1-466d-fa2a-4874cdb201b8: pipelining replication: peer="{Voter server-09c41644-0456-7121-a771-4f78e759823a 09c41644-0456-7121-a771-4f78e759823a}" + testing.go:149: server-42638d86-81a1-466d-fa2a-4874cdb201b8: 2022-08-12T07:30:05.518Z [INFO] server-42638d86-81a1-466d-fa2a-4874cdb201b8: pipelining replication: peer="{Voter server-09c41644-0456-7121-a771-4f78e759823a 09c41644-0456-7121-a771-4f78e759823a}" +testLoggerAdapter verbose: server-42638d86-81a1-466d-fa2a-4874cdb201b8: 2022-08-12T07:30:05.518Z [INFO] server-42638d86-81a1-466d-fa2a-4874cdb201b8: pipelining replication: peer="{Voter server-f8457807-7874-1562-78aa-ee8519f6e29b f8457807-7874-1562-78aa-ee8519f6e29b}" + testing.go:149: server-42638d86-81a1-466d-fa2a-4874cdb201b8: 2022-08-12T07:30:05.518Z [INFO] server-42638d86-81a1-466d-fa2a-4874cdb201b8: pipelining replication: peer="{Voter server-f8457807-7874-1562-78aa-ee8519f6e29b f8457807-7874-1562-78aa-ee8519f6e29b}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:05.723Z [INFO] stable state for Leader reached at 2022-08-12 07:30:05.51802636 +0000 UTC m=+6.474712841 (1 nodes), 100.7222ms from start of poll, 101.70794ms from cluster start. Timeout at 2022-08-12 07:30:05.72304666 +0000 UTC m=+6.679733161, 205.02032ms after stability + testing.go:149: cluster: 2022-08-12T07:30:05.723Z [INFO] stable state for Leader reached at 2022-08-12 07:30:05.51802636 +0000 UTC m=+6.474712841 (1 nodes), 100.7222ms from start of poll, 101.70794ms from cluster start. Timeout at 2022-08-12 07:30:05.72304666 +0000 UTC m=+6.679733161, 205.02032ms after stability +testLoggerAdapter verbose: server-42638d86-81a1-466d-fa2a-4874cdb201b8: 2022-08-12T07:30:05.737Z [INFO] server-42638d86-81a1-466d-fa2a-4874cdb201b8: aborting pipeline replication: peer="{Voter server-f8457807-7874-1562-78aa-ee8519f6e29b f8457807-7874-1562-78aa-ee8519f6e29b}" + testing.go:149: server-42638d86-81a1-466d-fa2a-4874cdb201b8: 2022-08-12T07:30:05.737Z [INFO] server-42638d86-81a1-466d-fa2a-4874cdb201b8: aborting pipeline replication: peer="{Voter server-f8457807-7874-1562-78aa-ee8519f6e29b f8457807-7874-1562-78aa-ee8519f6e29b}" +testLoggerAdapter verbose: server-42638d86-81a1-466d-fa2a-4874cdb201b8: 2022-08-12T07:30:05.737Z [INFO] server-42638d86-81a1-466d-fa2a-4874cdb201b8: aborting pipeline replication: peer="{Voter server-09c41644-0456-7121-a771-4f78e759823a 09c41644-0456-7121-a771-4f78e759823a}" + testing.go:149: server-42638d86-81a1-466d-fa2a-4874cdb201b8: 2022-08-12T07:30:05.737Z [INFO] server-42638d86-81a1-466d-fa2a-4874cdb201b8: aborting pipeline replication: peer="{Voter server-09c41644-0456-7121-a771-4f78e759823a 09c41644-0456-7121-a771-4f78e759823a}" +--- PASS: TestRaft_ApplyConcurrent (0.32s) === RUN TestRaft_ApplyConcurrent_Timeout -testLoggerAdapter verbose: server-81607767-2f01-59f9-be42-330697676670: 2021-07-10T01:05:01.246Z [INFO] server-81607767-2f01-59f9-be42-330697676670: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-81607767-2f01-59f9-be42-330697676670 Address:81607767-2f01-59f9-be42-330697676670}]" - testing.go:149: server-81607767-2f01-59f9-be42-330697676670: 2021-07-10T01:05:01.246Z [INFO] server-81607767-2f01-59f9-be42-330697676670: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-81607767-2f01-59f9-be42-330697676670 Address:81607767-2f01-59f9-be42-330697676670}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:01.246Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:01.246Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-81607767-2f01-59f9-be42-330697676670: 2021-07-10T01:05:01.246Z [INFO] server-81607767-2f01-59f9-be42-330697676670: entering follower state: follower="Node at 81607767-2f01-59f9-be42-330697676670 [Follower]" leader= - testing.go:149: server-81607767-2f01-59f9-be42-330697676670: 2021-07-10T01:05:01.246Z [INFO] server-81607767-2f01-59f9-be42-330697676670: entering follower state: follower="Node at 81607767-2f01-59f9-be42-330697676670 [Follower]" leader= -testLoggerAdapter verbose: server-81607767-2f01-59f9-be42-330697676670: 2021-07-10T01:05:01.445Z [WARN] server-81607767-2f01-59f9-be42-330697676670: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-81607767-2f01-59f9-be42-330697676670: 2021-07-10T01:05:01.445Z [WARN] server-81607767-2f01-59f9-be42-330697676670: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-81607767-2f01-59f9-be42-330697676670: 2021-07-10T01:05:01.446Z [INFO] server-81607767-2f01-59f9-be42-330697676670: entering candidate state: node="Node at 81607767-2f01-59f9-be42-330697676670 [Candidate]" term=2 - testing.go:149: server-81607767-2f01-59f9-be42-330697676670: 2021-07-10T01:05:01.446Z [INFO] server-81607767-2f01-59f9-be42-330697676670: entering candidate state: node="Node at 81607767-2f01-59f9-be42-330697676670 [Candidate]" term=2 -testLoggerAdapter verbose: server-81607767-2f01-59f9-be42-330697676670: 2021-07-10T01:05:01.446Z [INFO] server-81607767-2f01-59f9-be42-330697676670: election won: tally=1 - testing.go:149: server-81607767-2f01-59f9-be42-330697676670: 2021-07-10T01:05:01.446Z [INFO] server-81607767-2f01-59f9-be42-330697676670: election won: tally=1 -testLoggerAdapter verbose: server-81607767-2f01-59f9-be42-330697676670: 2021-07-10T01:05:01.446Z [INFO] server-81607767-2f01-59f9-be42-330697676670: entering leader state: leader="Node at 81607767-2f01-59f9-be42-330697676670 [Leader]" - testing.go:149: server-81607767-2f01-59f9-be42-330697676670: 2021-07-10T01:05:01.446Z [INFO] server-81607767-2f01-59f9-be42-330697676670: entering leader state: leader="Node at 81607767-2f01-59f9-be42-330697676670 [Leader]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:01.647Z [INFO] stable state for Leader reached at 2021-07-10 01:05:01.446412503 +0000 UTC m=+6.834570513 (1 nodes), 199.699966ms from start of poll, 200.07635ms from cluster start. Timeout at 2021-07-10 01:05:01.647485242 +0000 UTC m=+7.035643212, 201.072699ms after stability - testing.go:149: cluster: 2021-07-10T01:05:01.647Z [INFO] stable state for Leader reached at 2021-07-10 01:05:01.446412503 +0000 UTC m=+6.834570513 (1 nodes), 199.699966ms from start of poll, 200.07635ms from cluster start. Timeout at 2021-07-10 01:05:01.647485242 +0000 UTC m=+7.035643212, 201.072699ms after stability ---- PASS: TestRaft_ApplyConcurrent_Timeout (0.40s) +testLoggerAdapter verbose: server-ff413874-d5dc-fe3e-c32b-b2f79c31b6f2: 2022-08-12T07:30:05.738Z [INFO] server-ff413874-d5dc-fe3e-c32b-b2f79c31b6f2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff413874-d5dc-fe3e-c32b-b2f79c31b6f2 Address:ff413874-d5dc-fe3e-c32b-b2f79c31b6f2}]" + testing.go:149: server-ff413874-d5dc-fe3e-c32b-b2f79c31b6f2: 2022-08-12T07:30:05.738Z [INFO] server-ff413874-d5dc-fe3e-c32b-b2f79c31b6f2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff413874-d5dc-fe3e-c32b-b2f79c31b6f2 Address:ff413874-d5dc-fe3e-c32b-b2f79c31b6f2}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:05.738Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:05.738Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-ff413874-d5dc-fe3e-c32b-b2f79c31b6f2: 2022-08-12T07:30:05.738Z [INFO] server-ff413874-d5dc-fe3e-c32b-b2f79c31b6f2: entering follower state: follower="Node at ff413874-d5dc-fe3e-c32b-b2f79c31b6f2 [Follower]" leader= + testing.go:149: server-ff413874-d5dc-fe3e-c32b-b2f79c31b6f2: 2022-08-12T07:30:05.738Z [INFO] server-ff413874-d5dc-fe3e-c32b-b2f79c31b6f2: entering follower state: follower="Node at ff413874-d5dc-fe3e-c32b-b2f79c31b6f2 [Follower]" leader= +testLoggerAdapter verbose: server-ff413874-d5dc-fe3e-c32b-b2f79c31b6f2: 2022-08-12T07:30:05.883Z [WARN] server-ff413874-d5dc-fe3e-c32b-b2f79c31b6f2: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-ff413874-d5dc-fe3e-c32b-b2f79c31b6f2: 2022-08-12T07:30:05.883Z [WARN] server-ff413874-d5dc-fe3e-c32b-b2f79c31b6f2: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-ff413874-d5dc-fe3e-c32b-b2f79c31b6f2: 2022-08-12T07:30:05.884Z [INFO] server-ff413874-d5dc-fe3e-c32b-b2f79c31b6f2: entering candidate state: node="Node at ff413874-d5dc-fe3e-c32b-b2f79c31b6f2 [Candidate]" term=2 + testing.go:149: server-ff413874-d5dc-fe3e-c32b-b2f79c31b6f2: 2022-08-12T07:30:05.884Z [INFO] server-ff413874-d5dc-fe3e-c32b-b2f79c31b6f2: entering candidate state: node="Node at ff413874-d5dc-fe3e-c32b-b2f79c31b6f2 [Candidate]" term=2 +testLoggerAdapter verbose: server-ff413874-d5dc-fe3e-c32b-b2f79c31b6f2: 2022-08-12T07:30:05.884Z [INFO] server-ff413874-d5dc-fe3e-c32b-b2f79c31b6f2: election won: tally=1 + testing.go:149: server-ff413874-d5dc-fe3e-c32b-b2f79c31b6f2: 2022-08-12T07:30:05.884Z [INFO] server-ff413874-d5dc-fe3e-c32b-b2f79c31b6f2: election won: tally=1 +testLoggerAdapter verbose: server-ff413874-d5dc-fe3e-c32b-b2f79c31b6f2: 2022-08-12T07:30:05.884Z [INFO] server-ff413874-d5dc-fe3e-c32b-b2f79c31b6f2: entering leader state: leader="Node at ff413874-d5dc-fe3e-c32b-b2f79c31b6f2 [Leader]" + testing.go:149: server-ff413874-d5dc-fe3e-c32b-b2f79c31b6f2: 2022-08-12T07:30:05.884Z [INFO] server-ff413874-d5dc-fe3e-c32b-b2f79c31b6f2: entering leader state: leader="Node at ff413874-d5dc-fe3e-c32b-b2f79c31b6f2 [Leader]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:06.085Z [INFO] stable state for Leader reached at 2022-08-12 07:30:05.88430368 +0000 UTC m=+6.840990161 (1 nodes), 145.82326ms from start of poll, 146.19852ms from cluster start. Timeout at 2022-08-12 07:30:06.08537698 +0000 UTC m=+7.042063461, 201.0733ms after stability + testing.go:149: cluster: 2022-08-12T07:30:06.085Z [INFO] stable state for Leader reached at 2022-08-12 07:30:05.88430368 +0000 UTC m=+6.840990161 (1 nodes), 145.82326ms from start of poll, 146.19852ms from cluster start. Timeout at 2022-08-12 07:30:06.08537698 +0000 UTC m=+7.042063461, 201.0733ms after stability +--- PASS: TestRaft_ApplyConcurrent_Timeout (0.35s) === RUN TestRaft_JoinNode -testLoggerAdapter verbose: server-77cd786a-e3b1-0556-30fa-9fa725f51fa2: 2021-07-10T01:05:01.648Z [INFO] server-77cd786a-e3b1-0556-30fa-9fa725f51fa2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77cd786a-e3b1-0556-30fa-9fa725f51fa2 Address:77cd786a-e3b1-0556-30fa-9fa725f51fa2} {Suffrage:Voter ID:server-547c628c-b604-48c9-f20b-4f20ab6dd0a0 Address:547c628c-b604-48c9-f20b-4f20ab6dd0a0}]" - testing.go:149: server-77cd786a-e3b1-0556-30fa-9fa725f51fa2: 2021-07-10T01:05:01.648Z [INFO] server-77cd786a-e3b1-0556-30fa-9fa725f51fa2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77cd786a-e3b1-0556-30fa-9fa725f51fa2 Address:77cd786a-e3b1-0556-30fa-9fa725f51fa2} {Suffrage:Voter ID:server-547c628c-b604-48c9-f20b-4f20ab6dd0a0 Address:547c628c-b604-48c9-f20b-4f20ab6dd0a0}]" -testLoggerAdapter verbose: server-77cd786a-e3b1-0556-30fa-9fa725f51fa2: 2021-07-10T01:05:01.649Z [INFO] server-77cd786a-e3b1-0556-30fa-9fa725f51fa2: entering follower state: follower="Node at 77cd786a-e3b1-0556-30fa-9fa725f51fa2 [Follower]" leader= - testing.go:149: server-77cd786a-e3b1-0556-30fa-9fa725f51fa2: 2021-07-10T01:05:01.649Z [INFO] server-77cd786a-e3b1-0556-30fa-9fa725f51fa2: entering follower state: follower="Node at 77cd786a-e3b1-0556-30fa-9fa725f51fa2 [Follower]" leader= -testLoggerAdapter verbose: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.649Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77cd786a-e3b1-0556-30fa-9fa725f51fa2 Address:77cd786a-e3b1-0556-30fa-9fa725f51fa2} {Suffrage:Voter ID:server-547c628c-b604-48c9-f20b-4f20ab6dd0a0 Address:547c628c-b604-48c9-f20b-4f20ab6dd0a0}]" - testing.go:149: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.649Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77cd786a-e3b1-0556-30fa-9fa725f51fa2 Address:77cd786a-e3b1-0556-30fa-9fa725f51fa2} {Suffrage:Voter ID:server-547c628c-b604-48c9-f20b-4f20ab6dd0a0 Address:547c628c-b604-48c9-f20b-4f20ab6dd0a0}]" -testLoggerAdapter verbose: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.649Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: entering follower state: follower="Node at 547c628c-b604-48c9-f20b-4f20ab6dd0a0 [Follower]" leader= - testing.go:149: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.649Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: entering follower state: follower="Node at 547c628c-b604-48c9-f20b-4f20ab6dd0a0 [Follower]" leader= -testLoggerAdapter verbose: server-d8ae26cb-1997-b273-9593-ba87939f154e: 2021-07-10T01:05:01.649Z [INFO] server-d8ae26cb-1997-b273-9593-ba87939f154e: initial configuration: index=0 servers=[] - testing.go:149: server-d8ae26cb-1997-b273-9593-ba87939f154e: 2021-07-10T01:05:01.649Z [INFO] server-d8ae26cb-1997-b273-9593-ba87939f154e: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:01.649Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-d8ae26cb-1997-b273-9593-ba87939f154e: 2021-07-10T01:05:01.649Z [INFO] server-d8ae26cb-1997-b273-9593-ba87939f154e: entering follower state: follower="Node at d8ae26cb-1997-b273-9593-ba87939f154e [Follower]" leader= - testing.go:149: cluster: 2021-07-10T01:05:01.649Z [INFO] starting stability test: raft-state=Leader - testing.go:149: server-d8ae26cb-1997-b273-9593-ba87939f154e: 2021-07-10T01:05:01.649Z [INFO] server-d8ae26cb-1997-b273-9593-ba87939f154e: entering follower state: follower="Node at d8ae26cb-1997-b273-9593-ba87939f154e [Follower]" leader= -testLoggerAdapter verbose: server-d8ae26cb-1997-b273-9593-ba87939f154e: 2021-07-10T01:05:01.702Z [WARN] server-d8ae26cb-1997-b273-9593-ba87939f154e: no known peers, aborting election - testing.go:149: server-d8ae26cb-1997-b273-9593-ba87939f154e: 2021-07-10T01:05:01.702Z [WARN] server-d8ae26cb-1997-b273-9593-ba87939f154e: no known peers, aborting election -testLoggerAdapter verbose: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.716Z [WARN] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.716Z [WARN] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.716Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: entering candidate state: node="Node at 547c628c-b604-48c9-f20b-4f20ab6dd0a0 [Candidate]" term=2 - testing.go:149: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.716Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: entering candidate state: node="Node at 547c628c-b604-48c9-f20b-4f20ab6dd0a0 [Candidate]" term=2 -testLoggerAdapter verbose: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.716Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: election won: tally=2 - testing.go:149: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.716Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: election won: tally=2 -testLoggerAdapter verbose: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.716Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: entering leader state: leader="Node at 547c628c-b604-48c9-f20b-4f20ab6dd0a0 [Leader]" - testing.go:149: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.716Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: entering leader state: leader="Node at 547c628c-b604-48c9-f20b-4f20ab6dd0a0 [Leader]" -testLoggerAdapter verbose: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.716Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: added peer, starting replication: peer=server-77cd786a-e3b1-0556-30fa-9fa725f51fa2 - testing.go:149: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.716Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: added peer, starting replication: peer=server-77cd786a-e3b1-0556-30fa-9fa725f51fa2 -testLoggerAdapter verbose: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.716Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: pipelining replication: peer="{Voter server-77cd786a-e3b1-0556-30fa-9fa725f51fa2 77cd786a-e3b1-0556-30fa-9fa725f51fa2}" - testing.go:149: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.716Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: pipelining replication: peer="{Voter server-77cd786a-e3b1-0556-30fa-9fa725f51fa2 77cd786a-e3b1-0556-30fa-9fa725f51fa2}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:01.821Z [INFO] stable state for Leader reached at 2021-07-10 01:05:01.716621291 +0000 UTC m=+7.104779261 (1 nodes), 66.883245ms from start of poll, 67.892336ms from cluster start. Timeout at 2021-07-10 01:05:01.821710419 +0000 UTC m=+7.209868389, 105.089128ms after stability - testing.go:149: cluster: 2021-07-10T01:05:01.821Z [INFO] stable state for Leader reached at 2021-07-10 01:05:01.716621291 +0000 UTC m=+7.104779261 (1 nodes), 66.883245ms from start of poll, 67.892336ms from cluster start. Timeout at 2021-07-10 01:05:01.821710419 +0000 UTC m=+7.209868389, 105.089128ms after stability -testLoggerAdapter verbose: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.821Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: updating configuration: command=AddStaging server-id=server-d8ae26cb-1997-b273-9593-ba87939f154e server-addr=d8ae26cb-1997-b273-9593-ba87939f154e servers="[{Suffrage:Voter ID:server-77cd786a-e3b1-0556-30fa-9fa725f51fa2 Address:77cd786a-e3b1-0556-30fa-9fa725f51fa2} {Suffrage:Voter ID:server-547c628c-b604-48c9-f20b-4f20ab6dd0a0 Address:547c628c-b604-48c9-f20b-4f20ab6dd0a0} {Suffrage:Voter ID:server-d8ae26cb-1997-b273-9593-ba87939f154e Address:d8ae26cb-1997-b273-9593-ba87939f154e}]" - testing.go:149: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.821Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: updating configuration: command=AddStaging server-id=server-d8ae26cb-1997-b273-9593-ba87939f154e server-addr=d8ae26cb-1997-b273-9593-ba87939f154e servers="[{Suffrage:Voter ID:server-77cd786a-e3b1-0556-30fa-9fa725f51fa2 Address:77cd786a-e3b1-0556-30fa-9fa725f51fa2} {Suffrage:Voter ID:server-547c628c-b604-48c9-f20b-4f20ab6dd0a0 Address:547c628c-b604-48c9-f20b-4f20ab6dd0a0} {Suffrage:Voter ID:server-d8ae26cb-1997-b273-9593-ba87939f154e Address:d8ae26cb-1997-b273-9593-ba87939f154e}]" -testLoggerAdapter verbose: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.822Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: added peer, starting replication: peer=server-d8ae26cb-1997-b273-9593-ba87939f154e - testing.go:149: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.822Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: added peer, starting replication: peer=server-d8ae26cb-1997-b273-9593-ba87939f154e -testLoggerAdapter verbose: server-d8ae26cb-1997-b273-9593-ba87939f154e: 2021-07-10T01:05:01.822Z [WARN] server-d8ae26cb-1997-b273-9593-ba87939f154e: failed to get previous log: previous-index=3 last-index=0 error="log not found" - testing.go:149: server-d8ae26cb-1997-b273-9593-ba87939f154e: 2021-07-10T01:05:01.822Z [WARN] server-d8ae26cb-1997-b273-9593-ba87939f154e: failed to get previous log: previous-index=3 last-index=0 error="log not found" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:01.822Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:01.822Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.822Z [WARN] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: appendEntries rejected, sending older logs: peer="{Voter server-d8ae26cb-1997-b273-9593-ba87939f154e d8ae26cb-1997-b273-9593-ba87939f154e}" next=1 - testing.go:149: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.822Z [WARN] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: appendEntries rejected, sending older logs: peer="{Voter server-d8ae26cb-1997-b273-9593-ba87939f154e d8ae26cb-1997-b273-9593-ba87939f154e}" next=1 -testLoggerAdapter verbose: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.822Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: pipelining replication: peer="{Voter server-d8ae26cb-1997-b273-9593-ba87939f154e d8ae26cb-1997-b273-9593-ba87939f154e}" - testing.go:149: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.822Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: pipelining replication: peer="{Voter server-d8ae26cb-1997-b273-9593-ba87939f154e d8ae26cb-1997-b273-9593-ba87939f154e}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:01.927Z [INFO] stable state for Leader reached at 2021-07-10 01:05:01.822609649 +0000 UTC m=+7.210767599 (1 nodes), 1.72µs from start of poll, 173.880674ms from cluster start. Timeout at 2021-07-10 01:05:01.927674576 +0000 UTC m=+7.315832546, 105.064947ms after stability - testing.go:149: cluster: 2021-07-10T01:05:01.927Z [INFO] stable state for Leader reached at 2021-07-10 01:05:01.822609649 +0000 UTC m=+7.210767599 (1 nodes), 1.72µs from start of poll, 173.880674ms from cluster start. Timeout at 2021-07-10 01:05:01.927674576 +0000 UTC m=+7.315832546, 105.064947ms after stability -testLoggerAdapter verbose: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.928Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: aborting pipeline replication: peer="{Voter server-d8ae26cb-1997-b273-9593-ba87939f154e d8ae26cb-1997-b273-9593-ba87939f154e}" - testing.go:149: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.928Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: aborting pipeline replication: peer="{Voter server-d8ae26cb-1997-b273-9593-ba87939f154e d8ae26cb-1997-b273-9593-ba87939f154e}" -testLoggerAdapter verbose: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.928Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: aborting pipeline replication: peer="{Voter server-77cd786a-e3b1-0556-30fa-9fa725f51fa2 77cd786a-e3b1-0556-30fa-9fa725f51fa2}" - testing.go:149: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.928Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: aborting pipeline replication: peer="{Voter server-77cd786a-e3b1-0556-30fa-9fa725f51fa2 77cd786a-e3b1-0556-30fa-9fa725f51fa2}" ---- PASS: TestRaft_JoinNode (0.28s) +testLoggerAdapter verbose: server-5fde384a-ce22-66c7-d5ce-b426768a2aa1: 2022-08-12T07:30:06.086Z [INFO] server-5fde384a-ce22-66c7-d5ce-b426768a2aa1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5fde384a-ce22-66c7-d5ce-b426768a2aa1 Address:5fde384a-ce22-66c7-d5ce-b426768a2aa1} {Suffrage:Voter ID:server-ecd4ca51-3951-d67f-0352-4c11c2226db4 Address:ecd4ca51-3951-d67f-0352-4c11c2226db4}]" + testing.go:149: server-5fde384a-ce22-66c7-d5ce-b426768a2aa1: 2022-08-12T07:30:06.086Z [INFO] server-5fde384a-ce22-66c7-d5ce-b426768a2aa1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5fde384a-ce22-66c7-d5ce-b426768a2aa1 Address:5fde384a-ce22-66c7-d5ce-b426768a2aa1} {Suffrage:Voter ID:server-ecd4ca51-3951-d67f-0352-4c11c2226db4 Address:ecd4ca51-3951-d67f-0352-4c11c2226db4}]" +testLoggerAdapter verbose: server-ecd4ca51-3951-d67f-0352-4c11c2226db4: 2022-08-12T07:30:06.087Z [INFO] server-ecd4ca51-3951-d67f-0352-4c11c2226db4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5fde384a-ce22-66c7-d5ce-b426768a2aa1 Address:5fde384a-ce22-66c7-d5ce-b426768a2aa1} {Suffrage:Voter ID:server-ecd4ca51-3951-d67f-0352-4c11c2226db4 Address:ecd4ca51-3951-d67f-0352-4c11c2226db4}]" + testing.go:149: server-ecd4ca51-3951-d67f-0352-4c11c2226db4: 2022-08-12T07:30:06.087Z [INFO] server-ecd4ca51-3951-d67f-0352-4c11c2226db4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5fde384a-ce22-66c7-d5ce-b426768a2aa1 Address:5fde384a-ce22-66c7-d5ce-b426768a2aa1} {Suffrage:Voter ID:server-ecd4ca51-3951-d67f-0352-4c11c2226db4 Address:ecd4ca51-3951-d67f-0352-4c11c2226db4}]" +testLoggerAdapter verbose: server-14bf5f20-ede0-9fa8-d77e-a278ac5745c9: 2022-08-12T07:30:06.087Z [INFO] server-14bf5f20-ede0-9fa8-d77e-a278ac5745c9: initial configuration: index=0 servers=[] + testing.go:149: server-14bf5f20-ede0-9fa8-d77e-a278ac5745c9: 2022-08-12T07:30:06.087Z [INFO] server-14bf5f20-ede0-9fa8-d77e-a278ac5745c9: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:06.087Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:06.087Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-5fde384a-ce22-66c7-d5ce-b426768a2aa1: 2022-08-12T07:30:06.087Z [INFO] server-5fde384a-ce22-66c7-d5ce-b426768a2aa1: entering follower state: follower="Node at 5fde384a-ce22-66c7-d5ce-b426768a2aa1 [Follower]" leader= + testing.go:149: server-5fde384a-ce22-66c7-d5ce-b426768a2aa1: 2022-08-12T07:30:06.087Z [INFO] server-5fde384a-ce22-66c7-d5ce-b426768a2aa1: entering follower state: follower="Node at 5fde384a-ce22-66c7-d5ce-b426768a2aa1 [Follower]" leader= +testLoggerAdapter verbose: server-ecd4ca51-3951-d67f-0352-4c11c2226db4: 2022-08-12T07:30:06.087Z [INFO] server-ecd4ca51-3951-d67f-0352-4c11c2226db4: entering follower state: follower="Node at ecd4ca51-3951-d67f-0352-4c11c2226db4 [Follower]" leader= + testing.go:149: server-ecd4ca51-3951-d67f-0352-4c11c2226db4: 2022-08-12T07:30:06.087Z [INFO] server-ecd4ca51-3951-d67f-0352-4c11c2226db4: entering follower state: follower="Node at ecd4ca51-3951-d67f-0352-4c11c2226db4 [Follower]" leader= +testLoggerAdapter verbose: server-14bf5f20-ede0-9fa8-d77e-a278ac5745c9: 2022-08-12T07:30:06.087Z [INFO] server-14bf5f20-ede0-9fa8-d77e-a278ac5745c9: entering follower state: follower="Node at 14bf5f20-ede0-9fa8-d77e-a278ac5745c9 [Follower]" leader= + testing.go:149: server-14bf5f20-ede0-9fa8-d77e-a278ac5745c9: 2022-08-12T07:30:06.087Z [INFO] server-14bf5f20-ede0-9fa8-d77e-a278ac5745c9: entering follower state: follower="Node at 14bf5f20-ede0-9fa8-d77e-a278ac5745c9 [Follower]" leader= +testLoggerAdapter verbose: server-ecd4ca51-3951-d67f-0352-4c11c2226db4: 2022-08-12T07:30:06.165Z [WARN] server-ecd4ca51-3951-d67f-0352-4c11c2226db4: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-ecd4ca51-3951-d67f-0352-4c11c2226db4: 2022-08-12T07:30:06.165Z [WARN] server-ecd4ca51-3951-d67f-0352-4c11c2226db4: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-ecd4ca51-3951-d67f-0352-4c11c2226db4: 2022-08-12T07:30:06.165Z [INFO] server-ecd4ca51-3951-d67f-0352-4c11c2226db4: entering candidate state: node="Node at ecd4ca51-3951-d67f-0352-4c11c2226db4 [Candidate]" term=2 + testing.go:149: server-ecd4ca51-3951-d67f-0352-4c11c2226db4: 2022-08-12T07:30:06.165Z [INFO] server-ecd4ca51-3951-d67f-0352-4c11c2226db4: entering candidate state: node="Node at ecd4ca51-3951-d67f-0352-4c11c2226db4 [Candidate]" term=2 +testLoggerAdapter verbose: server-ecd4ca51-3951-d67f-0352-4c11c2226db4: 2022-08-12T07:30:06.165Z [INFO] server-ecd4ca51-3951-d67f-0352-4c11c2226db4: election won: tally=2 + testing.go:149: server-ecd4ca51-3951-d67f-0352-4c11c2226db4: 2022-08-12T07:30:06.165Z [INFO] server-ecd4ca51-3951-d67f-0352-4c11c2226db4: election won: tally=2 +testLoggerAdapter verbose: server-ecd4ca51-3951-d67f-0352-4c11c2226db4: 2022-08-12T07:30:06.165Z [INFO] server-ecd4ca51-3951-d67f-0352-4c11c2226db4: entering leader state: leader="Node at ecd4ca51-3951-d67f-0352-4c11c2226db4 [Leader]" + testing.go:149: server-ecd4ca51-3951-d67f-0352-4c11c2226db4: 2022-08-12T07:30:06.165Z [INFO] server-ecd4ca51-3951-d67f-0352-4c11c2226db4: entering leader state: leader="Node at ecd4ca51-3951-d67f-0352-4c11c2226db4 [Leader]" +testLoggerAdapter verbose: server-ecd4ca51-3951-d67f-0352-4c11c2226db4: 2022-08-12T07:30:06.166Z [INFO] server-ecd4ca51-3951-d67f-0352-4c11c2226db4: added peer, starting replication: peer=server-5fde384a-ce22-66c7-d5ce-b426768a2aa1 +testLoggerAdapter verbose: server-14bf5f20-ede0-9fa8-d77e-a278ac5745c9: 2022-08-12T07:30:06.166Z [WARN] server-14bf5f20-ede0-9fa8-d77e-a278ac5745c9: no known peers, aborting election + testing.go:149: server-ecd4ca51-3951-d67f-0352-4c11c2226db4: 2022-08-12T07:30:06.166Z [INFO] server-ecd4ca51-3951-d67f-0352-4c11c2226db4: added peer, starting replication: peer=server-5fde384a-ce22-66c7-d5ce-b426768a2aa1 + testing.go:149: server-14bf5f20-ede0-9fa8-d77e-a278ac5745c9: 2022-08-12T07:30:06.166Z [WARN] server-14bf5f20-ede0-9fa8-d77e-a278ac5745c9: no known peers, aborting election +testLoggerAdapter verbose: server-ecd4ca51-3951-d67f-0352-4c11c2226db4: 2022-08-12T07:30:06.166Z [INFO] server-ecd4ca51-3951-d67f-0352-4c11c2226db4: pipelining replication: peer="{Voter server-5fde384a-ce22-66c7-d5ce-b426768a2aa1 5fde384a-ce22-66c7-d5ce-b426768a2aa1}" + testing.go:149: server-ecd4ca51-3951-d67f-0352-4c11c2226db4: 2022-08-12T07:30:06.166Z [INFO] server-ecd4ca51-3951-d67f-0352-4c11c2226db4: pipelining replication: peer="{Voter server-5fde384a-ce22-66c7-d5ce-b426768a2aa1 5fde384a-ce22-66c7-d5ce-b426768a2aa1}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:06.271Z [INFO] stable state for Leader reached at 2022-08-12 07:30:06.16602296 +0000 UTC m=+7.122709441 (1 nodes), 78.40216ms from start of poll, 79.3981ms from cluster start. Timeout at 2022-08-12 07:30:06.27108662 +0000 UTC m=+7.227773081, 105.06364ms after stability + testing.go:149: cluster: 2022-08-12T07:30:06.271Z [INFO] stable state for Leader reached at 2022-08-12 07:30:06.16602296 +0000 UTC m=+7.122709441 (1 nodes), 78.40216ms from start of poll, 79.3981ms from cluster start. Timeout at 2022-08-12 07:30:06.27108662 +0000 UTC m=+7.227773081, 105.06364ms after stability +testLoggerAdapter verbose: server-ecd4ca51-3951-d67f-0352-4c11c2226db4: 2022-08-12T07:30:06.271Z [INFO] server-ecd4ca51-3951-d67f-0352-4c11c2226db4: updating configuration: command=AddStaging server-id=server-14bf5f20-ede0-9fa8-d77e-a278ac5745c9 server-addr=14bf5f20-ede0-9fa8-d77e-a278ac5745c9 servers="[{Suffrage:Voter ID:server-5fde384a-ce22-66c7-d5ce-b426768a2aa1 Address:5fde384a-ce22-66c7-d5ce-b426768a2aa1} {Suffrage:Voter ID:server-ecd4ca51-3951-d67f-0352-4c11c2226db4 Address:ecd4ca51-3951-d67f-0352-4c11c2226db4} {Suffrage:Voter ID:server-14bf5f20-ede0-9fa8-d77e-a278ac5745c9 Address:14bf5f20-ede0-9fa8-d77e-a278ac5745c9}]" + testing.go:149: server-ecd4ca51-3951-d67f-0352-4c11c2226db4: 2022-08-12T07:30:06.271Z [INFO] server-ecd4ca51-3951-d67f-0352-4c11c2226db4: updating configuration: command=AddStaging server-id=server-14bf5f20-ede0-9fa8-d77e-a278ac5745c9 server-addr=14bf5f20-ede0-9fa8-d77e-a278ac5745c9 servers="[{Suffrage:Voter ID:server-5fde384a-ce22-66c7-d5ce-b426768a2aa1 Address:5fde384a-ce22-66c7-d5ce-b426768a2aa1} {Suffrage:Voter ID:server-ecd4ca51-3951-d67f-0352-4c11c2226db4 Address:ecd4ca51-3951-d67f-0352-4c11c2226db4} {Suffrage:Voter ID:server-14bf5f20-ede0-9fa8-d77e-a278ac5745c9 Address:14bf5f20-ede0-9fa8-d77e-a278ac5745c9}]" +testLoggerAdapter verbose: server-ecd4ca51-3951-d67f-0352-4c11c2226db4: 2022-08-12T07:30:06.271Z [INFO] server-ecd4ca51-3951-d67f-0352-4c11c2226db4: added peer, starting replication: peer=server-14bf5f20-ede0-9fa8-d77e-a278ac5745c9 + testing.go:149: server-ecd4ca51-3951-d67f-0352-4c11c2226db4: 2022-08-12T07:30:06.271Z [INFO] server-ecd4ca51-3951-d67f-0352-4c11c2226db4: added peer, starting replication: peer=server-14bf5f20-ede0-9fa8-d77e-a278ac5745c9 +testLoggerAdapter verbose: server-14bf5f20-ede0-9fa8-d77e-a278ac5745c9: 2022-08-12T07:30:06.271Z [WARN] server-14bf5f20-ede0-9fa8-d77e-a278ac5745c9: failed to get previous log: previous-index=3 last-index=0 error="log not found" + testing.go:149: server-14bf5f20-ede0-9fa8-d77e-a278ac5745c9: 2022-08-12T07:30:06.271Z [WARN] server-14bf5f20-ede0-9fa8-d77e-a278ac5745c9: failed to get previous log: previous-index=3 last-index=0 error="log not found" +testLoggerAdapter verbose: server-ecd4ca51-3951-d67f-0352-4c11c2226db4: 2022-08-12T07:30:06.271Z [WARN] server-ecd4ca51-3951-d67f-0352-4c11c2226db4: appendEntries rejected, sending older logs: peer="{Voter server-14bf5f20-ede0-9fa8-d77e-a278ac5745c9 14bf5f20-ede0-9fa8-d77e-a278ac5745c9}" next=1 +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:06.271Z [INFO] starting stability test: raft-state=Leader + testing.go:149: server-ecd4ca51-3951-d67f-0352-4c11c2226db4: 2022-08-12T07:30:06.271Z [WARN] server-ecd4ca51-3951-d67f-0352-4c11c2226db4: appendEntries rejected, sending older logs: peer="{Voter server-14bf5f20-ede0-9fa8-d77e-a278ac5745c9 14bf5f20-ede0-9fa8-d77e-a278ac5745c9}" next=1 + testing.go:149: cluster: 2022-08-12T07:30:06.271Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-ecd4ca51-3951-d67f-0352-4c11c2226db4: 2022-08-12T07:30:06.271Z [INFO] server-ecd4ca51-3951-d67f-0352-4c11c2226db4: pipelining replication: peer="{Voter server-14bf5f20-ede0-9fa8-d77e-a278ac5745c9 14bf5f20-ede0-9fa8-d77e-a278ac5745c9}" + testing.go:149: server-ecd4ca51-3951-d67f-0352-4c11c2226db4: 2022-08-12T07:30:06.271Z [INFO] server-ecd4ca51-3951-d67f-0352-4c11c2226db4: pipelining replication: peer="{Voter server-14bf5f20-ede0-9fa8-d77e-a278ac5745c9 14bf5f20-ede0-9fa8-d77e-a278ac5745c9}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:06.377Z [INFO] stable state for Leader reached at 2022-08-12 07:30:06.27191178 +0000 UTC m=+7.228598241 (1 nodes), 2.1µs from start of poll, 185.2869ms from cluster start. Timeout at 2022-08-12 07:30:06.37697604 +0000 UTC m=+7.333662521, 105.06428ms after stability + testing.go:149: cluster: 2022-08-12T07:30:06.377Z [INFO] stable state for Leader reached at 2022-08-12 07:30:06.27191178 +0000 UTC m=+7.228598241 (1 nodes), 2.1µs from start of poll, 185.2869ms from cluster start. Timeout at 2022-08-12 07:30:06.37697604 +0000 UTC m=+7.333662521, 105.06428ms after stability +testLoggerAdapter verbose: server-ecd4ca51-3951-d67f-0352-4c11c2226db4: 2022-08-12T07:30:06.377Z [INFO] server-ecd4ca51-3951-d67f-0352-4c11c2226db4: aborting pipeline replication: peer="{Voter server-14bf5f20-ede0-9fa8-d77e-a278ac5745c9 14bf5f20-ede0-9fa8-d77e-a278ac5745c9}" + testing.go:149: server-ecd4ca51-3951-d67f-0352-4c11c2226db4: 2022-08-12T07:30:06.377Z [INFO] server-ecd4ca51-3951-d67f-0352-4c11c2226db4: aborting pipeline replication: peer="{Voter server-14bf5f20-ede0-9fa8-d77e-a278ac5745c9 14bf5f20-ede0-9fa8-d77e-a278ac5745c9}" +testLoggerAdapter verbose: server-ecd4ca51-3951-d67f-0352-4c11c2226db4: 2022-08-12T07:30:06.377Z [INFO] server-ecd4ca51-3951-d67f-0352-4c11c2226db4: aborting pipeline replication: peer="{Voter server-5fde384a-ce22-66c7-d5ce-b426768a2aa1 5fde384a-ce22-66c7-d5ce-b426768a2aa1}" + testing.go:149: server-ecd4ca51-3951-d67f-0352-4c11c2226db4: 2022-08-12T07:30:06.377Z [INFO] server-ecd4ca51-3951-d67f-0352-4c11c2226db4: aborting pipeline replication: peer="{Voter server-5fde384a-ce22-66c7-d5ce-b426768a2aa1 5fde384a-ce22-66c7-d5ce-b426768a2aa1}" +--- PASS: TestRaft_JoinNode (0.29s) === RUN TestRaft_JoinNode_ConfigStore -testLoggerAdapter verbose: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:01.929Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b845038-ba98-0d58-7127-c9c89fd5fe60 Address:7b845038-ba98-0d58-7127-c9c89fd5fe60}]" - testing.go:149: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:01.929Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b845038-ba98-0d58-7127-c9c89fd5fe60 Address:7b845038-ba98-0d58-7127-c9c89fd5fe60}]" -testLoggerAdapter verbose: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:01.929Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: entering follower state: follower="Node at 7b845038-ba98-0d58-7127-c9c89fd5fe60 [Follower]" leader= - testing.go:149: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:01.929Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: entering follower state: follower="Node at 7b845038-ba98-0d58-7127-c9c89fd5fe60 [Follower]" leader= -testLoggerAdapter verbose: server-3b5470ae-10fd-3291-beba-a039712e0a65: 2021-07-10T01:05:01.929Z [INFO] server-3b5470ae-10fd-3291-beba-a039712e0a65: initial configuration: index=0 servers=[] - testing.go:149: server-3b5470ae-10fd-3291-beba-a039712e0a65: 2021-07-10T01:05:01.929Z [INFO] server-3b5470ae-10fd-3291-beba-a039712e0a65: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: server-3b5470ae-10fd-3291-beba-a039712e0a65: 2021-07-10T01:05:01.929Z [INFO] server-3b5470ae-10fd-3291-beba-a039712e0a65: entering follower state: follower="Node at 3b5470ae-10fd-3291-beba-a039712e0a65 [Follower]" leader= - testing.go:149: server-3b5470ae-10fd-3291-beba-a039712e0a65: 2021-07-10T01:05:01.929Z [INFO] server-3b5470ae-10fd-3291-beba-a039712e0a65: entering follower state: follower="Node at 3b5470ae-10fd-3291-beba-a039712e0a65 [Follower]" leader= -testLoggerAdapter verbose: server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690: 2021-07-10T01:05:01.930Z [INFO] server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690: initial configuration: index=0 servers=[] - testing.go:149: server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690: 2021-07-10T01:05:01.930Z [INFO] server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:01.930Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690: 2021-07-10T01:05:01.930Z [INFO] server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690: entering follower state: follower="Node at 1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690 [Follower]" leader= - testing.go:149: cluster: 2021-07-10T01:05:01.930Z [INFO] starting stability test: raft-state=Leader - testing.go:149: server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690: 2021-07-10T01:05:01.930Z [INFO] server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690: entering follower state: follower="Node at 1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690 [Follower]" leader= -testLoggerAdapter verbose: server-3b5470ae-10fd-3291-beba-a039712e0a65: 2021-07-10T01:05:01.987Z [WARN] server-3b5470ae-10fd-3291-beba-a039712e0a65: no known peers, aborting election - testing.go:149: server-3b5470ae-10fd-3291-beba-a039712e0a65: 2021-07-10T01:05:01.987Z [WARN] server-3b5470ae-10fd-3291-beba-a039712e0a65: no known peers, aborting election -testLoggerAdapter verbose: server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690: 2021-07-10T01:05:02.005Z [WARN] server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690: no known peers, aborting election - testing.go:149: server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690: 2021-07-10T01:05:02.005Z [WARN] server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690: no known peers, aborting election -testLoggerAdapter verbose: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.014Z [WARN] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.014Z [WARN] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.014Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: entering candidate state: node="Node at 7b845038-ba98-0d58-7127-c9c89fd5fe60 [Candidate]" term=2 - testing.go:149: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.014Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: entering candidate state: node="Node at 7b845038-ba98-0d58-7127-c9c89fd5fe60 [Candidate]" term=2 -testLoggerAdapter verbose: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.014Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: election won: tally=1 - testing.go:149: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.014Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: election won: tally=1 -testLoggerAdapter verbose: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.014Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: entering leader state: leader="Node at 7b845038-ba98-0d58-7127-c9c89fd5fe60 [Leader]" - testing.go:149: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.014Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: entering leader state: leader="Node at 7b845038-ba98-0d58-7127-c9c89fd5fe60 [Leader]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:02.120Z [INFO] stable state for Leader reached at 2021-07-10 01:05:02.014785554 +0000 UTC m=+7.402943544 (1 nodes), 84.614392ms from start of poll, 85.748383ms from cluster start. Timeout at 2021-07-10 01:05:02.119865762 +0000 UTC m=+7.508023732, 105.080188ms after stability - testing.go:149: cluster: 2021-07-10T01:05:02.120Z [INFO] stable state for Leader reached at 2021-07-10 01:05:02.014785554 +0000 UTC m=+7.402943544 (1 nodes), 84.614392ms from start of poll, 85.748383ms from cluster start. Timeout at 2021-07-10 01:05:02.119865762 +0000 UTC m=+7.508023732, 105.080188ms after stability -testLoggerAdapter verbose: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.120Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: updating configuration: command=AddStaging server-id=server-3b5470ae-10fd-3291-beba-a039712e0a65 server-addr=3b5470ae-10fd-3291-beba-a039712e0a65 servers="[{Suffrage:Voter ID:server-7b845038-ba98-0d58-7127-c9c89fd5fe60 Address:7b845038-ba98-0d58-7127-c9c89fd5fe60} {Suffrage:Voter ID:server-3b5470ae-10fd-3291-beba-a039712e0a65 Address:3b5470ae-10fd-3291-beba-a039712e0a65}]" - testing.go:149: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.120Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: updating configuration: command=AddStaging server-id=server-3b5470ae-10fd-3291-beba-a039712e0a65 server-addr=3b5470ae-10fd-3291-beba-a039712e0a65 servers="[{Suffrage:Voter ID:server-7b845038-ba98-0d58-7127-c9c89fd5fe60 Address:7b845038-ba98-0d58-7127-c9c89fd5fe60} {Suffrage:Voter ID:server-3b5470ae-10fd-3291-beba-a039712e0a65 Address:3b5470ae-10fd-3291-beba-a039712e0a65}]" -testLoggerAdapter verbose: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.120Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: added peer, starting replication: peer=server-3b5470ae-10fd-3291-beba-a039712e0a65 - testing.go:149: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.120Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: added peer, starting replication: peer=server-3b5470ae-10fd-3291-beba-a039712e0a65 -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:02.120Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:02.120Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-3b5470ae-10fd-3291-beba-a039712e0a65: 2021-07-10T01:05:02.120Z [WARN] server-3b5470ae-10fd-3291-beba-a039712e0a65: failed to get previous log: previous-index=3 last-index=0 error="log not found" - testing.go:149: server-3b5470ae-10fd-3291-beba-a039712e0a65: 2021-07-10T01:05:02.120Z [WARN] server-3b5470ae-10fd-3291-beba-a039712e0a65: failed to get previous log: previous-index=3 last-index=0 error="log not found" -testLoggerAdapter verbose: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.120Z [WARN] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: appendEntries rejected, sending older logs: peer="{Voter server-3b5470ae-10fd-3291-beba-a039712e0a65 3b5470ae-10fd-3291-beba-a039712e0a65}" next=1 - testing.go:149: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.120Z [WARN] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: appendEntries rejected, sending older logs: peer="{Voter server-3b5470ae-10fd-3291-beba-a039712e0a65 3b5470ae-10fd-3291-beba-a039712e0a65}" next=1 -testLoggerAdapter verbose: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.120Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: pipelining replication: peer="{Voter server-3b5470ae-10fd-3291-beba-a039712e0a65 3b5470ae-10fd-3291-beba-a039712e0a65}" - testing.go:149: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.120Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: pipelining replication: peer="{Voter server-3b5470ae-10fd-3291-beba-a039712e0a65 3b5470ae-10fd-3291-beba-a039712e0a65}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:02.225Z [INFO] stable state for Leader reached at 2021-07-10 01:05:02.12063029 +0000 UTC m=+7.508788280 (1 nodes), 2.14µs from start of poll, 191.593119ms from cluster start. Timeout at 2021-07-10 01:05:02.225687238 +0000 UTC m=+7.613845228, 105.056948ms after stability - testing.go:149: cluster: 2021-07-10T01:05:02.225Z [INFO] stable state for Leader reached at 2021-07-10 01:05:02.12063029 +0000 UTC m=+7.508788280 (1 nodes), 2.14µs from start of poll, 191.593119ms from cluster start. Timeout at 2021-07-10 01:05:02.225687238 +0000 UTC m=+7.613845228, 105.056948ms after stability -testLoggerAdapter verbose: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.225Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: updating configuration: command=AddStaging server-id=server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690 server-addr=1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690 servers="[{Suffrage:Voter ID:server-7b845038-ba98-0d58-7127-c9c89fd5fe60 Address:7b845038-ba98-0d58-7127-c9c89fd5fe60} {Suffrage:Voter ID:server-3b5470ae-10fd-3291-beba-a039712e0a65 Address:3b5470ae-10fd-3291-beba-a039712e0a65} {Suffrage:Voter ID:server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690 Address:1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690}]" - testing.go:149: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.225Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: updating configuration: command=AddStaging server-id=server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690 server-addr=1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690 servers="[{Suffrage:Voter ID:server-7b845038-ba98-0d58-7127-c9c89fd5fe60 Address:7b845038-ba98-0d58-7127-c9c89fd5fe60} {Suffrage:Voter ID:server-3b5470ae-10fd-3291-beba-a039712e0a65 Address:3b5470ae-10fd-3291-beba-a039712e0a65} {Suffrage:Voter ID:server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690 Address:1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690}]" -testLoggerAdapter verbose: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.226Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: added peer, starting replication: peer=server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690 - testing.go:149: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.226Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: added peer, starting replication: peer=server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690 -testLoggerAdapter verbose: server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690: 2021-07-10T01:05:02.226Z [WARN] server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690: failed to get previous log: previous-index=4 last-index=0 error="log not found" - testing.go:149: server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690: 2021-07-10T01:05:02.226Z [WARN] server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690: failed to get previous log: previous-index=4 last-index=0 error="log not found" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:02.226Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.226Z [WARN] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: appendEntries rejected, sending older logs: peer="{Voter server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690 1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690}" next=1 - testing.go:149: cluster: 2021-07-10T01:05:02.226Z [INFO] starting stability test: raft-state=Leader - testing.go:149: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.226Z [WARN] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: appendEntries rejected, sending older logs: peer="{Voter server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690 1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690}" next=1 -testLoggerAdapter verbose: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.226Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: pipelining replication: peer="{Voter server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690 1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690}" - testing.go:149: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.226Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: pipelining replication: peer="{Voter server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690 1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:02.331Z [INFO] stable state for Leader reached at 2021-07-10 01:05:02.226492646 +0000 UTC m=+7.614650596 (1 nodes), 1.32µs from start of poll, 297.455435ms from cluster start. Timeout at 2021-07-10 01:05:02.331558674 +0000 UTC m=+7.719716664, 105.066068ms after stability - testing.go:149: cluster: 2021-07-10T01:05:02.331Z [INFO] stable state for Leader reached at 2021-07-10 01:05:02.226492646 +0000 UTC m=+7.614650596 (1 nodes), 1.32µs from start of poll, 297.455435ms from cluster start. Timeout at 2021-07-10 01:05:02.331558674 +0000 UTC m=+7.719716664, 105.066068ms after stability -testLoggerAdapter verbose: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.331Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: aborting pipeline replication: peer="{Voter server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690 1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690}" - testing.go:149: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.331Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: aborting pipeline replication: peer="{Voter server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690 1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690}" -testLoggerAdapter verbose: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.331Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: aborting pipeline replication: peer="{Voter server-3b5470ae-10fd-3291-beba-a039712e0a65 3b5470ae-10fd-3291-beba-a039712e0a65}" - testing.go:149: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.331Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: aborting pipeline replication: peer="{Voter server-3b5470ae-10fd-3291-beba-a039712e0a65 3b5470ae-10fd-3291-beba-a039712e0a65}" ---- PASS: TestRaft_JoinNode_ConfigStore (0.40s) +testLoggerAdapter verbose: server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: 2022-08-12T07:30:06.378Z [INFO] server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c Address:ff6ee318-3f52-7f21-336c-8b6ec89cfa4c}]" + testing.go:149: server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: 2022-08-12T07:30:06.378Z [INFO] server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c Address:ff6ee318-3f52-7f21-336c-8b6ec89cfa4c}]" +testLoggerAdapter verbose: server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: 2022-08-12T07:30:06.378Z [INFO] server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: entering follower state: follower="Node at ff6ee318-3f52-7f21-336c-8b6ec89cfa4c [Follower]" leader= + testing.go:149: server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: 2022-08-12T07:30:06.378Z [INFO] server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: entering follower state: follower="Node at ff6ee318-3f52-7f21-336c-8b6ec89cfa4c [Follower]" leader= +testLoggerAdapter verbose: server-8f0882e3-7f71-b455-d943-4047d72a284f: 2022-08-12T07:30:06.379Z [INFO] server-8f0882e3-7f71-b455-d943-4047d72a284f: initial configuration: index=0 servers=[] + testing.go:149: server-8f0882e3-7f71-b455-d943-4047d72a284f: 2022-08-12T07:30:06.379Z [INFO] server-8f0882e3-7f71-b455-d943-4047d72a284f: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: server-8f0882e3-7f71-b455-d943-4047d72a284f: 2022-08-12T07:30:06.379Z [INFO] server-8f0882e3-7f71-b455-d943-4047d72a284f: entering follower state: follower="Node at 8f0882e3-7f71-b455-d943-4047d72a284f [Follower]" leader= + testing.go:149: server-8f0882e3-7f71-b455-d943-4047d72a284f: 2022-08-12T07:30:06.379Z [INFO] server-8f0882e3-7f71-b455-d943-4047d72a284f: entering follower state: follower="Node at 8f0882e3-7f71-b455-d943-4047d72a284f [Follower]" leader= +testLoggerAdapter verbose: server-c63fc8d6-aba6-a557-4d98-6d32054d2b16: 2022-08-12T07:30:06.379Z [INFO] server-c63fc8d6-aba6-a557-4d98-6d32054d2b16: initial configuration: index=0 servers=[] + testing.go:149: server-c63fc8d6-aba6-a557-4d98-6d32054d2b16: 2022-08-12T07:30:06.379Z [INFO] server-c63fc8d6-aba6-a557-4d98-6d32054d2b16: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:06.379Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:06.379Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-c63fc8d6-aba6-a557-4d98-6d32054d2b16: 2022-08-12T07:30:06.379Z [INFO] server-c63fc8d6-aba6-a557-4d98-6d32054d2b16: entering follower state: follower="Node at c63fc8d6-aba6-a557-4d98-6d32054d2b16 [Follower]" leader= + testing.go:149: server-c63fc8d6-aba6-a557-4d98-6d32054d2b16: 2022-08-12T07:30:06.379Z [INFO] server-c63fc8d6-aba6-a557-4d98-6d32054d2b16: entering follower state: follower="Node at c63fc8d6-aba6-a557-4d98-6d32054d2b16 [Follower]" leader= +testLoggerAdapter verbose: server-c63fc8d6-aba6-a557-4d98-6d32054d2b16: 2022-08-12T07:30:06.437Z [WARN] server-c63fc8d6-aba6-a557-4d98-6d32054d2b16: no known peers, aborting election + testing.go:149: server-c63fc8d6-aba6-a557-4d98-6d32054d2b16: 2022-08-12T07:30:06.437Z [WARN] server-c63fc8d6-aba6-a557-4d98-6d32054d2b16: no known peers, aborting election +testLoggerAdapter verbose: server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: 2022-08-12T07:30:06.445Z [WARN] server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: 2022-08-12T07:30:06.445Z [WARN] server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: 2022-08-12T07:30:06.445Z [INFO] server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: entering candidate state: node="Node at ff6ee318-3f52-7f21-336c-8b6ec89cfa4c [Candidate]" term=2 + testing.go:149: server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: 2022-08-12T07:30:06.445Z [INFO] server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: entering candidate state: node="Node at ff6ee318-3f52-7f21-336c-8b6ec89cfa4c [Candidate]" term=2 +testLoggerAdapter verbose: server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: 2022-08-12T07:30:06.445Z [INFO] server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: election won: tally=1 + testing.go:149: server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: 2022-08-12T07:30:06.445Z [INFO] server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: election won: tally=1 +testLoggerAdapter verbose: server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: 2022-08-12T07:30:06.445Z [INFO] server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: entering leader state: leader="Node at ff6ee318-3f52-7f21-336c-8b6ec89cfa4c [Leader]" + testing.go:149: server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: 2022-08-12T07:30:06.445Z [INFO] server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: entering leader state: leader="Node at ff6ee318-3f52-7f21-336c-8b6ec89cfa4c [Leader]" +testLoggerAdapter verbose: server-8f0882e3-7f71-b455-d943-4047d72a284f: 2022-08-12T07:30:06.470Z [WARN] server-8f0882e3-7f71-b455-d943-4047d72a284f: no known peers, aborting election + testing.go:149: server-8f0882e3-7f71-b455-d943-4047d72a284f: 2022-08-12T07:30:06.470Z [WARN] server-8f0882e3-7f71-b455-d943-4047d72a284f: no known peers, aborting election +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:06.550Z [INFO] stable state for Leader reached at 2022-08-12 07:30:06.44558276 +0000 UTC m=+7.402269261 (1 nodes), 66.04246ms from start of poll, 67.09588ms from cluster start. Timeout at 2022-08-12 07:30:06.550666 +0000 UTC m=+7.507352481, 105.08322ms after stability + testing.go:149: cluster: 2022-08-12T07:30:06.550Z [INFO] stable state for Leader reached at 2022-08-12 07:30:06.44558276 +0000 UTC m=+7.402269261 (1 nodes), 66.04246ms from start of poll, 67.09588ms from cluster start. Timeout at 2022-08-12 07:30:06.550666 +0000 UTC m=+7.507352481, 105.08322ms after stability +testLoggerAdapter verbose: server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: 2022-08-12T07:30:06.550Z [INFO] server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: updating configuration: command=AddStaging server-id=server-8f0882e3-7f71-b455-d943-4047d72a284f server-addr=8f0882e3-7f71-b455-d943-4047d72a284f servers="[{Suffrage:Voter ID:server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c Address:ff6ee318-3f52-7f21-336c-8b6ec89cfa4c} {Suffrage:Voter ID:server-8f0882e3-7f71-b455-d943-4047d72a284f Address:8f0882e3-7f71-b455-d943-4047d72a284f}]" + testing.go:149: server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: 2022-08-12T07:30:06.550Z [INFO] server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: updating configuration: command=AddStaging server-id=server-8f0882e3-7f71-b455-d943-4047d72a284f server-addr=8f0882e3-7f71-b455-d943-4047d72a284f servers="[{Suffrage:Voter ID:server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c Address:ff6ee318-3f52-7f21-336c-8b6ec89cfa4c} {Suffrage:Voter ID:server-8f0882e3-7f71-b455-d943-4047d72a284f Address:8f0882e3-7f71-b455-d943-4047d72a284f}]" +testLoggerAdapter verbose: server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: 2022-08-12T07:30:06.551Z [INFO] server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: added peer, starting replication: peer=server-8f0882e3-7f71-b455-d943-4047d72a284f + testing.go:149: server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: 2022-08-12T07:30:06.551Z [INFO] server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: added peer, starting replication: peer=server-8f0882e3-7f71-b455-d943-4047d72a284f +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:06.551Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:06.551Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-8f0882e3-7f71-b455-d943-4047d72a284f: 2022-08-12T07:30:06.551Z [WARN] server-8f0882e3-7f71-b455-d943-4047d72a284f: failed to get previous log: previous-index=3 last-index=0 error="log not found" + testing.go:149: server-8f0882e3-7f71-b455-d943-4047d72a284f: 2022-08-12T07:30:06.551Z [WARN] server-8f0882e3-7f71-b455-d943-4047d72a284f: failed to get previous log: previous-index=3 last-index=0 error="log not found" +testLoggerAdapter verbose: server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: 2022-08-12T07:30:06.551Z [WARN] server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: appendEntries rejected, sending older logs: peer="{Voter server-8f0882e3-7f71-b455-d943-4047d72a284f 8f0882e3-7f71-b455-d943-4047d72a284f}" next=1 + testing.go:149: server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: 2022-08-12T07:30:06.551Z [WARN] server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: appendEntries rejected, sending older logs: peer="{Voter server-8f0882e3-7f71-b455-d943-4047d72a284f 8f0882e3-7f71-b455-d943-4047d72a284f}" next=1 +testLoggerAdapter verbose: server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: 2022-08-12T07:30:06.551Z [INFO] server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: pipelining replication: peer="{Voter server-8f0882e3-7f71-b455-d943-4047d72a284f 8f0882e3-7f71-b455-d943-4047d72a284f}" + testing.go:149: server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: 2022-08-12T07:30:06.551Z [INFO] server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: pipelining replication: peer="{Voter server-8f0882e3-7f71-b455-d943-4047d72a284f 8f0882e3-7f71-b455-d943-4047d72a284f}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:06.656Z [INFO] stable state for Leader reached at 2022-08-12 07:30:06.55126494 +0000 UTC m=+7.507951421 (1 nodes), 1.64µs from start of poll, 172.77804ms from cluster start. Timeout at 2022-08-12 07:30:06.65633572 +0000 UTC m=+7.613022221, 105.0708ms after stability + testing.go:149: cluster: 2022-08-12T07:30:06.656Z [INFO] stable state for Leader reached at 2022-08-12 07:30:06.55126494 +0000 UTC m=+7.507951421 (1 nodes), 1.64µs from start of poll, 172.77804ms from cluster start. Timeout at 2022-08-12 07:30:06.65633572 +0000 UTC m=+7.613022221, 105.0708ms after stability +testLoggerAdapter verbose: server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: 2022-08-12T07:30:06.656Z [INFO] server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: updating configuration: command=AddStaging server-id=server-c63fc8d6-aba6-a557-4d98-6d32054d2b16 server-addr=c63fc8d6-aba6-a557-4d98-6d32054d2b16 servers="[{Suffrage:Voter ID:server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c Address:ff6ee318-3f52-7f21-336c-8b6ec89cfa4c} {Suffrage:Voter ID:server-8f0882e3-7f71-b455-d943-4047d72a284f Address:8f0882e3-7f71-b455-d943-4047d72a284f} {Suffrage:Voter ID:server-c63fc8d6-aba6-a557-4d98-6d32054d2b16 Address:c63fc8d6-aba6-a557-4d98-6d32054d2b16}]" + testing.go:149: server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: 2022-08-12T07:30:06.656Z [INFO] server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: updating configuration: command=AddStaging server-id=server-c63fc8d6-aba6-a557-4d98-6d32054d2b16 server-addr=c63fc8d6-aba6-a557-4d98-6d32054d2b16 servers="[{Suffrage:Voter ID:server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c Address:ff6ee318-3f52-7f21-336c-8b6ec89cfa4c} {Suffrage:Voter ID:server-8f0882e3-7f71-b455-d943-4047d72a284f Address:8f0882e3-7f71-b455-d943-4047d72a284f} {Suffrage:Voter ID:server-c63fc8d6-aba6-a557-4d98-6d32054d2b16 Address:c63fc8d6-aba6-a557-4d98-6d32054d2b16}]" +testLoggerAdapter verbose: server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: 2022-08-12T07:30:06.656Z [INFO] server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: added peer, starting replication: peer=server-c63fc8d6-aba6-a557-4d98-6d32054d2b16 + testing.go:149: server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: 2022-08-12T07:30:06.656Z [INFO] server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: added peer, starting replication: peer=server-c63fc8d6-aba6-a557-4d98-6d32054d2b16 +testLoggerAdapter verbose: server-c63fc8d6-aba6-a557-4d98-6d32054d2b16: 2022-08-12T07:30:06.656Z [WARN] server-c63fc8d6-aba6-a557-4d98-6d32054d2b16: failed to get previous log: previous-index=4 last-index=0 error="log not found" + testing.go:149: server-c63fc8d6-aba6-a557-4d98-6d32054d2b16: 2022-08-12T07:30:06.656Z [WARN] server-c63fc8d6-aba6-a557-4d98-6d32054d2b16: failed to get previous log: previous-index=4 last-index=0 error="log not found" +testLoggerAdapter verbose: server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: 2022-08-12T07:30:06.657Z [WARN] server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: appendEntries rejected, sending older logs: peer="{Voter server-c63fc8d6-aba6-a557-4d98-6d32054d2b16 c63fc8d6-aba6-a557-4d98-6d32054d2b16}" next=1 + testing.go:149: server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: 2022-08-12T07:30:06.657Z [WARN] server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: appendEntries rejected, sending older logs: peer="{Voter server-c63fc8d6-aba6-a557-4d98-6d32054d2b16 c63fc8d6-aba6-a557-4d98-6d32054d2b16}" next=1 +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:06.657Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:06.657Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: 2022-08-12T07:30:06.657Z [INFO] server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: pipelining replication: peer="{Voter server-c63fc8d6-aba6-a557-4d98-6d32054d2b16 c63fc8d6-aba6-a557-4d98-6d32054d2b16}" + testing.go:149: server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: 2022-08-12T07:30:06.657Z [INFO] server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: pipelining replication: peer="{Voter server-c63fc8d6-aba6-a557-4d98-6d32054d2b16 c63fc8d6-aba6-a557-4d98-6d32054d2b16}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:06.762Z [INFO] stable state for Leader reached at 2022-08-12 07:30:06.65720908 +0000 UTC m=+7.613895561 (1 nodes), 1.44µs from start of poll, 278.72218ms from cluster start. Timeout at 2022-08-12 07:30:06.76228004 +0000 UTC m=+7.718966541, 105.07098ms after stability + testing.go:149: cluster: 2022-08-12T07:30:06.762Z [INFO] stable state for Leader reached at 2022-08-12 07:30:06.65720908 +0000 UTC m=+7.613895561 (1 nodes), 1.44µs from start of poll, 278.72218ms from cluster start. Timeout at 2022-08-12 07:30:06.76228004 +0000 UTC m=+7.718966541, 105.07098ms after stability +testLoggerAdapter verbose: server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: 2022-08-12T07:30:06.762Z [INFO] server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: aborting pipeline replication: peer="{Voter server-c63fc8d6-aba6-a557-4d98-6d32054d2b16 c63fc8d6-aba6-a557-4d98-6d32054d2b16}" + testing.go:149: server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: 2022-08-12T07:30:06.762Z [INFO] server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: aborting pipeline replication: peer="{Voter server-c63fc8d6-aba6-a557-4d98-6d32054d2b16 c63fc8d6-aba6-a557-4d98-6d32054d2b16}" +testLoggerAdapter verbose: server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: 2022-08-12T07:30:06.762Z [INFO] server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: aborting pipeline replication: peer="{Voter server-8f0882e3-7f71-b455-d943-4047d72a284f 8f0882e3-7f71-b455-d943-4047d72a284f}" + testing.go:149: server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: 2022-08-12T07:30:06.762Z [INFO] server-ff6ee318-3f52-7f21-336c-8b6ec89cfa4c: aborting pipeline replication: peer="{Voter server-8f0882e3-7f71-b455-d943-4047d72a284f 8f0882e3-7f71-b455-d943-4047d72a284f}" +--- PASS: TestRaft_JoinNode_ConfigStore (0.39s) === RUN TestRaft_RemoveFollower -testLoggerAdapter verbose: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.333Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-19802584-fb94-dbc0-2edc-c35ec8be4ae1 Address:19802584-fb94-dbc0-2edc-c35ec8be4ae1} {Suffrage:Voter ID:server-37f9809a-6e60-f4c9-5aff-4b0b53e90820 Address:37f9809a-6e60-f4c9-5aff-4b0b53e90820} {Suffrage:Voter ID:server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1 Address:a18410a6-fd05-89ff-3d5b-fb3a48be0ec1}]" - testing.go:149: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.333Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-19802584-fb94-dbc0-2edc-c35ec8be4ae1 Address:19802584-fb94-dbc0-2edc-c35ec8be4ae1} {Suffrage:Voter ID:server-37f9809a-6e60-f4c9-5aff-4b0b53e90820 Address:37f9809a-6e60-f4c9-5aff-4b0b53e90820} {Suffrage:Voter ID:server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1 Address:a18410a6-fd05-89ff-3d5b-fb3a48be0ec1}]" -testLoggerAdapter verbose: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.333Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: entering follower state: follower="Node at 19802584-fb94-dbc0-2edc-c35ec8be4ae1 [Follower]" leader= - testing.go:149: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.333Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: entering follower state: follower="Node at 19802584-fb94-dbc0-2edc-c35ec8be4ae1 [Follower]" leader= -testLoggerAdapter verbose: server-37f9809a-6e60-f4c9-5aff-4b0b53e90820: 2021-07-10T01:05:02.333Z [INFO] server-37f9809a-6e60-f4c9-5aff-4b0b53e90820: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-19802584-fb94-dbc0-2edc-c35ec8be4ae1 Address:19802584-fb94-dbc0-2edc-c35ec8be4ae1} {Suffrage:Voter ID:server-37f9809a-6e60-f4c9-5aff-4b0b53e90820 Address:37f9809a-6e60-f4c9-5aff-4b0b53e90820} {Suffrage:Voter ID:server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1 Address:a18410a6-fd05-89ff-3d5b-fb3a48be0ec1}]" - testing.go:149: server-37f9809a-6e60-f4c9-5aff-4b0b53e90820: 2021-07-10T01:05:02.333Z [INFO] server-37f9809a-6e60-f4c9-5aff-4b0b53e90820: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-19802584-fb94-dbc0-2edc-c35ec8be4ae1 Address:19802584-fb94-dbc0-2edc-c35ec8be4ae1} {Suffrage:Voter ID:server-37f9809a-6e60-f4c9-5aff-4b0b53e90820 Address:37f9809a-6e60-f4c9-5aff-4b0b53e90820} {Suffrage:Voter ID:server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1 Address:a18410a6-fd05-89ff-3d5b-fb3a48be0ec1}]" -testLoggerAdapter verbose: server-37f9809a-6e60-f4c9-5aff-4b0b53e90820: 2021-07-10T01:05:02.333Z [INFO] server-37f9809a-6e60-f4c9-5aff-4b0b53e90820: entering follower state: follower="Node at 37f9809a-6e60-f4c9-5aff-4b0b53e90820 [Follower]" leader= - testing.go:149: server-37f9809a-6e60-f4c9-5aff-4b0b53e90820: 2021-07-10T01:05:02.333Z [INFO] server-37f9809a-6e60-f4c9-5aff-4b0b53e90820: entering follower state: follower="Node at 37f9809a-6e60-f4c9-5aff-4b0b53e90820 [Follower]" leader= -testLoggerAdapter verbose: server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1: 2021-07-10T01:05:02.333Z [INFO] server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-19802584-fb94-dbc0-2edc-c35ec8be4ae1 Address:19802584-fb94-dbc0-2edc-c35ec8be4ae1} {Suffrage:Voter ID:server-37f9809a-6e60-f4c9-5aff-4b0b53e90820 Address:37f9809a-6e60-f4c9-5aff-4b0b53e90820} {Suffrage:Voter ID:server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1 Address:a18410a6-fd05-89ff-3d5b-fb3a48be0ec1}]" - testing.go:149: server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1: 2021-07-10T01:05:02.333Z [INFO] server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-19802584-fb94-dbc0-2edc-c35ec8be4ae1 Address:19802584-fb94-dbc0-2edc-c35ec8be4ae1} {Suffrage:Voter ID:server-37f9809a-6e60-f4c9-5aff-4b0b53e90820 Address:37f9809a-6e60-f4c9-5aff-4b0b53e90820} {Suffrage:Voter ID:server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1 Address:a18410a6-fd05-89ff-3d5b-fb3a48be0ec1}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:02.333Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:02.333Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1: 2021-07-10T01:05:02.333Z [INFO] server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1: entering follower state: follower="Node at a18410a6-fd05-89ff-3d5b-fb3a48be0ec1 [Follower]" leader= - testing.go:149: server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1: 2021-07-10T01:05:02.333Z [INFO] server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1: entering follower state: follower="Node at a18410a6-fd05-89ff-3d5b-fb3a48be0ec1 [Follower]" leader= -testLoggerAdapter verbose: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.393Z [WARN] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.393Z [WARN] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.393Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: entering candidate state: node="Node at 19802584-fb94-dbc0-2edc-c35ec8be4ae1 [Candidate]" term=2 - testing.go:149: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.393Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: entering candidate state: node="Node at 19802584-fb94-dbc0-2edc-c35ec8be4ae1 [Candidate]" term=2 -testLoggerAdapter verbose: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.394Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: election won: tally=2 - testing.go:149: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.394Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: election won: tally=2 -testLoggerAdapter verbose: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.394Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: entering leader state: leader="Node at 19802584-fb94-dbc0-2edc-c35ec8be4ae1 [Leader]" - testing.go:149: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.394Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: entering leader state: leader="Node at 19802584-fb94-dbc0-2edc-c35ec8be4ae1 [Leader]" -testLoggerAdapter verbose: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.394Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: added peer, starting replication: peer=server-37f9809a-6e60-f4c9-5aff-4b0b53e90820 - testing.go:149: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.394Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: added peer, starting replication: peer=server-37f9809a-6e60-f4c9-5aff-4b0b53e90820 -testLoggerAdapter verbose: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.394Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: added peer, starting replication: peer=server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1 - testing.go:149: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.394Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: added peer, starting replication: peer=server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1 -testLoggerAdapter verbose: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.394Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: pipelining replication: peer="{Voter server-37f9809a-6e60-f4c9-5aff-4b0b53e90820 37f9809a-6e60-f4c9-5aff-4b0b53e90820}" - testing.go:149: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.394Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: pipelining replication: peer="{Voter server-37f9809a-6e60-f4c9-5aff-4b0b53e90820 37f9809a-6e60-f4c9-5aff-4b0b53e90820}" -testLoggerAdapter verbose: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.394Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: pipelining replication: peer="{Voter server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1 a18410a6-fd05-89ff-3d5b-fb3a48be0ec1}" - testing.go:149: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.394Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: pipelining replication: peer="{Voter server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1 a18410a6-fd05-89ff-3d5b-fb3a48be0ec1}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:02.499Z [INFO] stable state for Leader reached at 2021-07-10 01:05:02.394300515 +0000 UTC m=+7.782458505 (1 nodes), 60.278555ms from start of poll, 61.239945ms from cluster start. Timeout at 2021-07-10 01:05:02.499395283 +0000 UTC m=+7.887553273, 105.094768ms after stability - testing.go:149: cluster: 2021-07-10T01:05:02.499Z [INFO] stable state for Leader reached at 2021-07-10 01:05:02.394300515 +0000 UTC m=+7.782458505 (1 nodes), 60.278555ms from start of poll, 61.239945ms from cluster start. Timeout at 2021-07-10 01:05:02.499395283 +0000 UTC m=+7.887553273, 105.094768ms after stability -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:02.504Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-07-10T01:05:02.504Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:02.610Z [INFO] stable state for Follower reached at 2021-07-10 01:05:02.504901041 +0000 UTC m=+7.893058991 (2 nodes), 3.22µs from start of poll, 171.840431ms from cluster start. Timeout at 2021-07-10 01:05:02.609974229 +0000 UTC m=+7.998132199, 105.073208ms after stability - testing.go:149: cluster: 2021-07-10T01:05:02.610Z [INFO] stable state for Follower reached at 2021-07-10 01:05:02.504901041 +0000 UTC m=+7.893058991 (2 nodes), 3.22µs from start of poll, 171.840431ms from cluster start. Timeout at 2021-07-10 01:05:02.609974229 +0000 UTC m=+7.998132199, 105.073208ms after stability -testLoggerAdapter verbose: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.610Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: updating configuration: command=RemoveServer server-id=server-37f9809a-6e60-f4c9-5aff-4b0b53e90820 server-addr= servers="[{Suffrage:Voter ID:server-19802584-fb94-dbc0-2edc-c35ec8be4ae1 Address:19802584-fb94-dbc0-2edc-c35ec8be4ae1} {Suffrage:Voter ID:server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1 Address:a18410a6-fd05-89ff-3d5b-fb3a48be0ec1}]" - testing.go:149: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.610Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: updating configuration: command=RemoveServer server-id=server-37f9809a-6e60-f4c9-5aff-4b0b53e90820 server-addr= servers="[{Suffrage:Voter ID:server-19802584-fb94-dbc0-2edc-c35ec8be4ae1 Address:19802584-fb94-dbc0-2edc-c35ec8be4ae1} {Suffrage:Voter ID:server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1 Address:a18410a6-fd05-89ff-3d5b-fb3a48be0ec1}]" -testLoggerAdapter verbose: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.610Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: removed peer, stopping replication: peer=server-37f9809a-6e60-f4c9-5aff-4b0b53e90820 last-index=3 - testing.go:149: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.610Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: removed peer, stopping replication: peer=server-37f9809a-6e60-f4c9-5aff-4b0b53e90820 last-index=3 -testLoggerAdapter verbose: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.610Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: aborting pipeline replication: peer="{Voter server-37f9809a-6e60-f4c9-5aff-4b0b53e90820 37f9809a-6e60-f4c9-5aff-4b0b53e90820}" - testing.go:149: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.610Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: aborting pipeline replication: peer="{Voter server-37f9809a-6e60-f4c9-5aff-4b0b53e90820 37f9809a-6e60-f4c9-5aff-4b0b53e90820}" -testLoggerAdapter verbose: server-37f9809a-6e60-f4c9-5aff-4b0b53e90820: 2021-07-10T01:05:02.698Z [WARN] server-37f9809a-6e60-f4c9-5aff-4b0b53e90820: heartbeat timeout reached, starting election: last-leader=19802584-fb94-dbc0-2edc-c35ec8be4ae1 - testing.go:149: server-37f9809a-6e60-f4c9-5aff-4b0b53e90820: 2021-07-10T01:05:02.698Z [WARN] server-37f9809a-6e60-f4c9-5aff-4b0b53e90820: heartbeat timeout reached, starting election: last-leader=19802584-fb94-dbc0-2edc-c35ec8be4ae1 -testLoggerAdapter verbose: server-37f9809a-6e60-f4c9-5aff-4b0b53e90820: 2021-07-10T01:05:02.698Z [INFO] server-37f9809a-6e60-f4c9-5aff-4b0b53e90820: entering candidate state: node="Node at 37f9809a-6e60-f4c9-5aff-4b0b53e90820 [Candidate]" term=3 - testing.go:149: server-37f9809a-6e60-f4c9-5aff-4b0b53e90820: 2021-07-10T01:05:02.698Z [INFO] server-37f9809a-6e60-f4c9-5aff-4b0b53e90820: entering candidate state: node="Node at 37f9809a-6e60-f4c9-5aff-4b0b53e90820 [Candidate]" term=3 -testLoggerAdapter verbose: server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1: 2021-07-10T01:05:02.698Z [WARN] server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1: rejecting vote request since we have a leader: from=37f9809a-6e60-f4c9-5aff-4b0b53e90820 leader=19802584-fb94-dbc0-2edc-c35ec8be4ae1 - testing.go:149: server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1: 2021-07-10T01:05:02.698Z [WARN] server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1: rejecting vote request since we have a leader: from=37f9809a-6e60-f4c9-5aff-4b0b53e90820 leader=19802584-fb94-dbc0-2edc-c35ec8be4ae1 -testLoggerAdapter verbose: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.698Z [WARN] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: rejecting vote request since we have a leader: from=37f9809a-6e60-f4c9-5aff-4b0b53e90820 leader=19802584-fb94-dbc0-2edc-c35ec8be4ae1 - testing.go:149: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.698Z [WARN] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: rejecting vote request since we have a leader: from=37f9809a-6e60-f4c9-5aff-4b0b53e90820 leader=19802584-fb94-dbc0-2edc-c35ec8be4ae1 -testLoggerAdapter verbose: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.715Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: aborting pipeline replication: peer="{Voter server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1 a18410a6-fd05-89ff-3d5b-fb3a48be0ec1}" - testing.go:149: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.715Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: aborting pipeline replication: peer="{Voter server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1 a18410a6-fd05-89ff-3d5b-fb3a48be0ec1}" ---- PASS: TestRaft_RemoveFollower (0.38s) +testLoggerAdapter verbose: server-505707d2-6202-9e65-cd1a-452f2334eb87: 2022-08-12T07:30:06.764Z [INFO] server-505707d2-6202-9e65-cd1a-452f2334eb87: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-505707d2-6202-9e65-cd1a-452f2334eb87 Address:505707d2-6202-9e65-cd1a-452f2334eb87} {Suffrage:Voter ID:server-d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d Address:d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d} {Suffrage:Voter ID:server-69739415-fcab-87a4-70ad-de30284ab89f Address:69739415-fcab-87a4-70ad-de30284ab89f}]" + testing.go:149: server-505707d2-6202-9e65-cd1a-452f2334eb87: 2022-08-12T07:30:06.764Z [INFO] server-505707d2-6202-9e65-cd1a-452f2334eb87: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-505707d2-6202-9e65-cd1a-452f2334eb87 Address:505707d2-6202-9e65-cd1a-452f2334eb87} {Suffrage:Voter ID:server-d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d Address:d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d} {Suffrage:Voter ID:server-69739415-fcab-87a4-70ad-de30284ab89f Address:69739415-fcab-87a4-70ad-de30284ab89f}]" +testLoggerAdapter verbose: server-505707d2-6202-9e65-cd1a-452f2334eb87: 2022-08-12T07:30:06.764Z [INFO] server-505707d2-6202-9e65-cd1a-452f2334eb87: entering follower state: follower="Node at 505707d2-6202-9e65-cd1a-452f2334eb87 [Follower]" leader= + testing.go:149: server-505707d2-6202-9e65-cd1a-452f2334eb87: 2022-08-12T07:30:06.764Z [INFO] server-505707d2-6202-9e65-cd1a-452f2334eb87: entering follower state: follower="Node at 505707d2-6202-9e65-cd1a-452f2334eb87 [Follower]" leader= +testLoggerAdapter verbose: server-d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d: 2022-08-12T07:30:06.764Z [INFO] server-d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-505707d2-6202-9e65-cd1a-452f2334eb87 Address:505707d2-6202-9e65-cd1a-452f2334eb87} {Suffrage:Voter ID:server-d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d Address:d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d} {Suffrage:Voter ID:server-69739415-fcab-87a4-70ad-de30284ab89f Address:69739415-fcab-87a4-70ad-de30284ab89f}]" + testing.go:149: server-d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d: 2022-08-12T07:30:06.764Z [INFO] server-d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-505707d2-6202-9e65-cd1a-452f2334eb87 Address:505707d2-6202-9e65-cd1a-452f2334eb87} {Suffrage:Voter ID:server-d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d Address:d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d} {Suffrage:Voter ID:server-69739415-fcab-87a4-70ad-de30284ab89f Address:69739415-fcab-87a4-70ad-de30284ab89f}]" +testLoggerAdapter verbose: server-d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d: 2022-08-12T07:30:06.764Z [INFO] server-d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d: entering follower state: follower="Node at d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d [Follower]" leader= + testing.go:149: server-d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d: 2022-08-12T07:30:06.764Z [INFO] server-d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d: entering follower state: follower="Node at d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d [Follower]" leader= +testLoggerAdapter verbose: server-69739415-fcab-87a4-70ad-de30284ab89f: 2022-08-12T07:30:06.764Z [INFO] server-69739415-fcab-87a4-70ad-de30284ab89f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-505707d2-6202-9e65-cd1a-452f2334eb87 Address:505707d2-6202-9e65-cd1a-452f2334eb87} {Suffrage:Voter ID:server-d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d Address:d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d} {Suffrage:Voter ID:server-69739415-fcab-87a4-70ad-de30284ab89f Address:69739415-fcab-87a4-70ad-de30284ab89f}]" + testing.go:149: server-69739415-fcab-87a4-70ad-de30284ab89f: 2022-08-12T07:30:06.764Z [INFO] server-69739415-fcab-87a4-70ad-de30284ab89f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-505707d2-6202-9e65-cd1a-452f2334eb87 Address:505707d2-6202-9e65-cd1a-452f2334eb87} {Suffrage:Voter ID:server-d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d Address:d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d} {Suffrage:Voter ID:server-69739415-fcab-87a4-70ad-de30284ab89f Address:69739415-fcab-87a4-70ad-de30284ab89f}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:06.764Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:06.764Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-69739415-fcab-87a4-70ad-de30284ab89f: 2022-08-12T07:30:06.764Z [INFO] server-69739415-fcab-87a4-70ad-de30284ab89f: entering follower state: follower="Node at 69739415-fcab-87a4-70ad-de30284ab89f [Follower]" leader= + testing.go:149: server-69739415-fcab-87a4-70ad-de30284ab89f: 2022-08-12T07:30:06.764Z [INFO] server-69739415-fcab-87a4-70ad-de30284ab89f: entering follower state: follower="Node at 69739415-fcab-87a4-70ad-de30284ab89f [Follower]" leader= +testLoggerAdapter verbose: server-505707d2-6202-9e65-cd1a-452f2334eb87: 2022-08-12T07:30:06.832Z [WARN] server-505707d2-6202-9e65-cd1a-452f2334eb87: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-505707d2-6202-9e65-cd1a-452f2334eb87: 2022-08-12T07:30:06.832Z [WARN] server-505707d2-6202-9e65-cd1a-452f2334eb87: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-505707d2-6202-9e65-cd1a-452f2334eb87: 2022-08-12T07:30:06.832Z [INFO] server-505707d2-6202-9e65-cd1a-452f2334eb87: entering candidate state: node="Node at 505707d2-6202-9e65-cd1a-452f2334eb87 [Candidate]" term=2 + testing.go:149: server-505707d2-6202-9e65-cd1a-452f2334eb87: 2022-08-12T07:30:06.832Z [INFO] server-505707d2-6202-9e65-cd1a-452f2334eb87: entering candidate state: node="Node at 505707d2-6202-9e65-cd1a-452f2334eb87 [Candidate]" term=2 +testLoggerAdapter verbose: server-505707d2-6202-9e65-cd1a-452f2334eb87: 2022-08-12T07:30:06.832Z [INFO] server-505707d2-6202-9e65-cd1a-452f2334eb87: election won: tally=2 + testing.go:149: server-505707d2-6202-9e65-cd1a-452f2334eb87: 2022-08-12T07:30:06.832Z [INFO] server-505707d2-6202-9e65-cd1a-452f2334eb87: election won: tally=2 +testLoggerAdapter verbose: server-505707d2-6202-9e65-cd1a-452f2334eb87: 2022-08-12T07:30:06.832Z [INFO] server-505707d2-6202-9e65-cd1a-452f2334eb87: entering leader state: leader="Node at 505707d2-6202-9e65-cd1a-452f2334eb87 [Leader]" + testing.go:149: server-505707d2-6202-9e65-cd1a-452f2334eb87: 2022-08-12T07:30:06.832Z [INFO] server-505707d2-6202-9e65-cd1a-452f2334eb87: entering leader state: leader="Node at 505707d2-6202-9e65-cd1a-452f2334eb87 [Leader]" +testLoggerAdapter verbose: server-505707d2-6202-9e65-cd1a-452f2334eb87: 2022-08-12T07:30:06.832Z [INFO] server-505707d2-6202-9e65-cd1a-452f2334eb87: added peer, starting replication: peer=server-d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d + testing.go:149: server-505707d2-6202-9e65-cd1a-452f2334eb87: 2022-08-12T07:30:06.832Z [INFO] server-505707d2-6202-9e65-cd1a-452f2334eb87: added peer, starting replication: peer=server-d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d +testLoggerAdapter verbose: server-505707d2-6202-9e65-cd1a-452f2334eb87: 2022-08-12T07:30:06.832Z [INFO] server-505707d2-6202-9e65-cd1a-452f2334eb87: added peer, starting replication: peer=server-69739415-fcab-87a4-70ad-de30284ab89f + testing.go:149: server-505707d2-6202-9e65-cd1a-452f2334eb87: 2022-08-12T07:30:06.832Z [INFO] server-505707d2-6202-9e65-cd1a-452f2334eb87: added peer, starting replication: peer=server-69739415-fcab-87a4-70ad-de30284ab89f +testLoggerAdapter verbose: server-505707d2-6202-9e65-cd1a-452f2334eb87: 2022-08-12T07:30:06.832Z [INFO] server-505707d2-6202-9e65-cd1a-452f2334eb87: pipelining replication: peer="{Voter server-d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d}" + testing.go:149: server-505707d2-6202-9e65-cd1a-452f2334eb87: 2022-08-12T07:30:06.832Z [INFO] server-505707d2-6202-9e65-cd1a-452f2334eb87: pipelining replication: peer="{Voter server-d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d}" +testLoggerAdapter verbose: server-505707d2-6202-9e65-cd1a-452f2334eb87: 2022-08-12T07:30:06.832Z [INFO] server-505707d2-6202-9e65-cd1a-452f2334eb87: pipelining replication: peer="{Voter server-69739415-fcab-87a4-70ad-de30284ab89f 69739415-fcab-87a4-70ad-de30284ab89f}" + testing.go:149: server-505707d2-6202-9e65-cd1a-452f2334eb87: 2022-08-12T07:30:06.832Z [INFO] server-505707d2-6202-9e65-cd1a-452f2334eb87: pipelining replication: peer="{Voter server-69739415-fcab-87a4-70ad-de30284ab89f 69739415-fcab-87a4-70ad-de30284ab89f}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:06.937Z [INFO] stable state for Leader reached at 2022-08-12 07:30:06.83277908 +0000 UTC m=+7.789465581 (1 nodes), 68.03228ms from start of poll, 68.9601ms from cluster start. Timeout at 2022-08-12 07:30:06.9378604 +0000 UTC m=+7.894546881, 105.0813ms after stability + testing.go:149: cluster: 2022-08-12T07:30:06.937Z [INFO] stable state for Leader reached at 2022-08-12 07:30:06.83277908 +0000 UTC m=+7.789465581 (1 nodes), 68.03228ms from start of poll, 68.9601ms from cluster start. Timeout at 2022-08-12 07:30:06.9378604 +0000 UTC m=+7.894546881, 105.0813ms after stability +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:06.943Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2022-08-12T07:30:06.943Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:07.048Z [INFO] stable state for Follower reached at 2022-08-12 07:30:06.94329332 +0000 UTC m=+7.899979781 (2 nodes), 3.18µs from start of poll, 179.4743ms from cluster start. Timeout at 2022-08-12 07:30:07.04836926 +0000 UTC m=+8.005055741, 105.07596ms after stability + testing.go:149: cluster: 2022-08-12T07:30:07.048Z [INFO] stable state for Follower reached at 2022-08-12 07:30:06.94329332 +0000 UTC m=+7.899979781 (2 nodes), 3.18µs from start of poll, 179.4743ms from cluster start. Timeout at 2022-08-12 07:30:07.04836926 +0000 UTC m=+8.005055741, 105.07596ms after stability +testLoggerAdapter verbose: server-505707d2-6202-9e65-cd1a-452f2334eb87: 2022-08-12T07:30:07.048Z [INFO] server-505707d2-6202-9e65-cd1a-452f2334eb87: updating configuration: command=RemoveServer server-id=server-d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d server-addr= servers="[{Suffrage:Voter ID:server-505707d2-6202-9e65-cd1a-452f2334eb87 Address:505707d2-6202-9e65-cd1a-452f2334eb87} {Suffrage:Voter ID:server-69739415-fcab-87a4-70ad-de30284ab89f Address:69739415-fcab-87a4-70ad-de30284ab89f}]" + testing.go:149: server-505707d2-6202-9e65-cd1a-452f2334eb87: 2022-08-12T07:30:07.048Z [INFO] server-505707d2-6202-9e65-cd1a-452f2334eb87: updating configuration: command=RemoveServer server-id=server-d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d server-addr= servers="[{Suffrage:Voter ID:server-505707d2-6202-9e65-cd1a-452f2334eb87 Address:505707d2-6202-9e65-cd1a-452f2334eb87} {Suffrage:Voter ID:server-69739415-fcab-87a4-70ad-de30284ab89f Address:69739415-fcab-87a4-70ad-de30284ab89f}]" +testLoggerAdapter verbose: server-505707d2-6202-9e65-cd1a-452f2334eb87: 2022-08-12T07:30:07.048Z [INFO] server-505707d2-6202-9e65-cd1a-452f2334eb87: removed peer, stopping replication: peer=server-d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d last-index=3 + testing.go:149: server-505707d2-6202-9e65-cd1a-452f2334eb87: 2022-08-12T07:30:07.048Z [INFO] server-505707d2-6202-9e65-cd1a-452f2334eb87: removed peer, stopping replication: peer=server-d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d last-index=3 +testLoggerAdapter verbose: server-505707d2-6202-9e65-cd1a-452f2334eb87: 2022-08-12T07:30:07.048Z [INFO] server-505707d2-6202-9e65-cd1a-452f2334eb87: aborting pipeline replication: peer="{Voter server-d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d}" + testing.go:149: server-505707d2-6202-9e65-cd1a-452f2334eb87: 2022-08-12T07:30:07.048Z [INFO] server-505707d2-6202-9e65-cd1a-452f2334eb87: aborting pipeline replication: peer="{Voter server-d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d}" +testLoggerAdapter verbose: server-d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d: 2022-08-12T07:30:07.099Z [WARN] server-d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d: heartbeat timeout reached, starting election: last-leader=505707d2-6202-9e65-cd1a-452f2334eb87 + testing.go:149: server-d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d: 2022-08-12T07:30:07.099Z [WARN] server-d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d: heartbeat timeout reached, starting election: last-leader=505707d2-6202-9e65-cd1a-452f2334eb87 +testLoggerAdapter verbose: server-d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d: 2022-08-12T07:30:07.099Z [INFO] server-d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d: entering candidate state: node="Node at d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d [Candidate]" term=3 + testing.go:149: server-d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d: 2022-08-12T07:30:07.099Z [INFO] server-d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d: entering candidate state: node="Node at d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d [Candidate]" term=3 +testLoggerAdapter verbose: server-69739415-fcab-87a4-70ad-de30284ab89f: 2022-08-12T07:30:07.099Z [WARN] server-69739415-fcab-87a4-70ad-de30284ab89f: rejecting vote request since we have a leader: from=d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d leader=505707d2-6202-9e65-cd1a-452f2334eb87 +testLoggerAdapter verbose: server-505707d2-6202-9e65-cd1a-452f2334eb87: 2022-08-12T07:30:07.099Z [WARN] server-505707d2-6202-9e65-cd1a-452f2334eb87: rejecting vote request since we have a leader: from=d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d leader=505707d2-6202-9e65-cd1a-452f2334eb87 + testing.go:149: server-69739415-fcab-87a4-70ad-de30284ab89f: 2022-08-12T07:30:07.099Z [WARN] server-69739415-fcab-87a4-70ad-de30284ab89f: rejecting vote request since we have a leader: from=d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d leader=505707d2-6202-9e65-cd1a-452f2334eb87 + testing.go:149: server-505707d2-6202-9e65-cd1a-452f2334eb87: 2022-08-12T07:30:07.099Z [WARN] server-505707d2-6202-9e65-cd1a-452f2334eb87: rejecting vote request since we have a leader: from=d5bc3b0e-9c54-8eb2-8ae9-35d53131e16d leader=505707d2-6202-9e65-cd1a-452f2334eb87 +testLoggerAdapter verbose: server-505707d2-6202-9e65-cd1a-452f2334eb87: 2022-08-12T07:30:07.154Z [INFO] server-505707d2-6202-9e65-cd1a-452f2334eb87: aborting pipeline replication: peer="{Voter server-69739415-fcab-87a4-70ad-de30284ab89f 69739415-fcab-87a4-70ad-de30284ab89f}" + testing.go:149: server-505707d2-6202-9e65-cd1a-452f2334eb87: 2022-08-12T07:30:07.154Z [INFO] server-505707d2-6202-9e65-cd1a-452f2334eb87: aborting pipeline replication: peer="{Voter server-69739415-fcab-87a4-70ad-de30284ab89f 69739415-fcab-87a4-70ad-de30284ab89f}" +--- PASS: TestRaft_RemoveFollower (0.39s) === RUN TestRaft_RemoveLeader -testLoggerAdapter verbose: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.717Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab Address:2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab} {Suffrage:Voter ID:server-caae0ef8-5ef3-3038-2a73-a05d64d77146 Address:caae0ef8-5ef3-3038-2a73-a05d64d77146} {Suffrage:Voter ID:server-2bbd6b80-00c9-8a45-10e8-db43464eb80a Address:2bbd6b80-00c9-8a45-10e8-db43464eb80a}]" - testing.go:149: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.717Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab Address:2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab} {Suffrage:Voter ID:server-caae0ef8-5ef3-3038-2a73-a05d64d77146 Address:caae0ef8-5ef3-3038-2a73-a05d64d77146} {Suffrage:Voter ID:server-2bbd6b80-00c9-8a45-10e8-db43464eb80a Address:2bbd6b80-00c9-8a45-10e8-db43464eb80a}]" -testLoggerAdapter verbose: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.717Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: entering follower state: follower="Node at 2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab [Follower]" leader= - testing.go:149: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.717Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: entering follower state: follower="Node at 2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab [Follower]" leader= -testLoggerAdapter verbose: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:02.717Z [INFO] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab Address:2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab} {Suffrage:Voter ID:server-caae0ef8-5ef3-3038-2a73-a05d64d77146 Address:caae0ef8-5ef3-3038-2a73-a05d64d77146} {Suffrage:Voter ID:server-2bbd6b80-00c9-8a45-10e8-db43464eb80a Address:2bbd6b80-00c9-8a45-10e8-db43464eb80a}]" - testing.go:149: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:02.717Z [INFO] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab Address:2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab} {Suffrage:Voter ID:server-caae0ef8-5ef3-3038-2a73-a05d64d77146 Address:caae0ef8-5ef3-3038-2a73-a05d64d77146} {Suffrage:Voter ID:server-2bbd6b80-00c9-8a45-10e8-db43464eb80a Address:2bbd6b80-00c9-8a45-10e8-db43464eb80a}]" -testLoggerAdapter verbose: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:02.717Z [INFO] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: entering follower state: follower="Node at caae0ef8-5ef3-3038-2a73-a05d64d77146 [Follower]" leader= - testing.go:149: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:02.717Z [INFO] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: entering follower state: follower="Node at caae0ef8-5ef3-3038-2a73-a05d64d77146 [Follower]" leader= -testLoggerAdapter verbose: server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: 2021-07-10T01:05:02.718Z [INFO] server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab Address:2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab} {Suffrage:Voter ID:server-caae0ef8-5ef3-3038-2a73-a05d64d77146 Address:caae0ef8-5ef3-3038-2a73-a05d64d77146} {Suffrage:Voter ID:server-2bbd6b80-00c9-8a45-10e8-db43464eb80a Address:2bbd6b80-00c9-8a45-10e8-db43464eb80a}]" - testing.go:149: server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: 2021-07-10T01:05:02.718Z [INFO] server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab Address:2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab} {Suffrage:Voter ID:server-caae0ef8-5ef3-3038-2a73-a05d64d77146 Address:caae0ef8-5ef3-3038-2a73-a05d64d77146} {Suffrage:Voter ID:server-2bbd6b80-00c9-8a45-10e8-db43464eb80a Address:2bbd6b80-00c9-8a45-10e8-db43464eb80a}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:02.718Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:02.718Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: 2021-07-10T01:05:02.718Z [INFO] server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: entering follower state: follower="Node at 2bbd6b80-00c9-8a45-10e8-db43464eb80a [Follower]" leader= - testing.go:149: server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: 2021-07-10T01:05:02.718Z [INFO] server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: entering follower state: follower="Node at 2bbd6b80-00c9-8a45-10e8-db43464eb80a [Follower]" leader= -testLoggerAdapter verbose: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.774Z [WARN] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.774Z [WARN] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.775Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: entering candidate state: node="Node at 2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab [Candidate]" term=2 - testing.go:149: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.775Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: entering candidate state: node="Node at 2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab [Candidate]" term=2 -testLoggerAdapter verbose: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.775Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: election won: tally=2 - testing.go:149: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.775Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: election won: tally=2 -testLoggerAdapter verbose: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.775Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: entering leader state: leader="Node at 2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab [Leader]" - testing.go:149: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.775Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: entering leader state: leader="Node at 2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab [Leader]" -testLoggerAdapter verbose: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.775Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: added peer, starting replication: peer=server-caae0ef8-5ef3-3038-2a73-a05d64d77146 - testing.go:149: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.775Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: added peer, starting replication: peer=server-caae0ef8-5ef3-3038-2a73-a05d64d77146 -testLoggerAdapter verbose: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.775Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: added peer, starting replication: peer=server-2bbd6b80-00c9-8a45-10e8-db43464eb80a - testing.go:149: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.775Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: added peer, starting replication: peer=server-2bbd6b80-00c9-8a45-10e8-db43464eb80a -testLoggerAdapter verbose: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.775Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: pipelining replication: peer="{Voter server-caae0ef8-5ef3-3038-2a73-a05d64d77146 caae0ef8-5ef3-3038-2a73-a05d64d77146}" - testing.go:149: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.775Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: pipelining replication: peer="{Voter server-caae0ef8-5ef3-3038-2a73-a05d64d77146 caae0ef8-5ef3-3038-2a73-a05d64d77146}" -testLoggerAdapter verbose: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.775Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: pipelining replication: peer="{Voter server-2bbd6b80-00c9-8a45-10e8-db43464eb80a 2bbd6b80-00c9-8a45-10e8-db43464eb80a}" - testing.go:149: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.775Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: pipelining replication: peer="{Voter server-2bbd6b80-00c9-8a45-10e8-db43464eb80a 2bbd6b80-00c9-8a45-10e8-db43464eb80a}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:02.880Z [INFO] stable state for Leader reached at 2021-07-10 01:05:02.775450374 +0000 UTC m=+8.163608344 (1 nodes), 57.235604ms from start of poll, 58.260595ms from cluster start. Timeout at 2021-07-10 01:05:02.880514741 +0000 UTC m=+8.268672731, 105.064387ms after stability - testing.go:149: cluster: 2021-07-10T01:05:02.880Z [INFO] stable state for Leader reached at 2021-07-10 01:05:02.775450374 +0000 UTC m=+8.163608344 (1 nodes), 57.235604ms from start of poll, 58.260595ms from cluster start. Timeout at 2021-07-10 01:05:02.880514741 +0000 UTC m=+8.268672731, 105.064387ms after stability -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:02.885Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-07-10T01:05:02.885Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:02.991Z [INFO] stable state for Follower reached at 2021-07-10 01:05:02.885943219 +0000 UTC m=+8.274101189 (2 nodes), 3.441µs from start of poll, 168.75344ms from cluster start. Timeout at 2021-07-10 01:05:02.990997006 +0000 UTC m=+8.379154996, 105.053807ms after stability - testing.go:149: cluster: 2021-07-10T01:05:02.991Z [INFO] stable state for Follower reached at 2021-07-10 01:05:02.885943219 +0000 UTC m=+8.274101189 (2 nodes), 3.441µs from start of poll, 168.75344ms from cluster start. Timeout at 2021-07-10 01:05:02.990997006 +0000 UTC m=+8.379154996, 105.053807ms after stability -testLoggerAdapter verbose: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.991Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: updating configuration: command=RemoveServer server-id=server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab server-addr= servers="[{Suffrage:Voter ID:server-caae0ef8-5ef3-3038-2a73-a05d64d77146 Address:caae0ef8-5ef3-3038-2a73-a05d64d77146} {Suffrage:Voter ID:server-2bbd6b80-00c9-8a45-10e8-db43464eb80a Address:2bbd6b80-00c9-8a45-10e8-db43464eb80a}]" - testing.go:149: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.991Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: updating configuration: command=RemoveServer server-id=server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab server-addr= servers="[{Suffrage:Voter ID:server-caae0ef8-5ef3-3038-2a73-a05d64d77146 Address:caae0ef8-5ef3-3038-2a73-a05d64d77146} {Suffrage:Voter ID:server-2bbd6b80-00c9-8a45-10e8-db43464eb80a Address:2bbd6b80-00c9-8a45-10e8-db43464eb80a}]" -testLoggerAdapter verbose: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.991Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: removed ourself, shutting down - testing.go:149: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.991Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: removed ourself, shutting down -testLoggerAdapter verbose: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.991Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: aborting pipeline replication: peer="{Voter server-2bbd6b80-00c9-8a45-10e8-db43464eb80a 2bbd6b80-00c9-8a45-10e8-db43464eb80a}" - testing.go:149: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.991Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: aborting pipeline replication: peer="{Voter server-2bbd6b80-00c9-8a45-10e8-db43464eb80a 2bbd6b80-00c9-8a45-10e8-db43464eb80a}" -testLoggerAdapter verbose: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.991Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: aborting pipeline replication: peer="{Voter server-caae0ef8-5ef3-3038-2a73-a05d64d77146 caae0ef8-5ef3-3038-2a73-a05d64d77146}" - testing.go:149: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.991Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: aborting pipeline replication: peer="{Voter server-caae0ef8-5ef3-3038-2a73-a05d64d77146 caae0ef8-5ef3-3038-2a73-a05d64d77146}" -testLoggerAdapter verbose: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:03.041Z [WARN] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: heartbeat timeout reached, starting election: last-leader=2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab - testing.go:149: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:03.041Z [WARN] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: heartbeat timeout reached, starting election: last-leader=2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab -testLoggerAdapter verbose: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:03.042Z [INFO] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: entering candidate state: node="Node at caae0ef8-5ef3-3038-2a73-a05d64d77146 [Candidate]" term=3 - testing.go:149: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:03.042Z [INFO] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: entering candidate state: node="Node at caae0ef8-5ef3-3038-2a73-a05d64d77146 [Candidate]" term=3 -testLoggerAdapter verbose: server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: 2021-07-10T01:05:03.042Z [WARN] server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: rejecting vote request since we have a leader: from=caae0ef8-5ef3-3038-2a73-a05d64d77146 leader=2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab - testing.go:149: server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: 2021-07-10T01:05:03.042Z [WARN] server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: rejecting vote request since we have a leader: from=caae0ef8-5ef3-3038-2a73-a05d64d77146 leader=2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab -testLoggerAdapter verbose: server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: 2021-07-10T01:05:03.105Z [WARN] server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: heartbeat timeout reached, starting election: last-leader=2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab -testLoggerAdapter verbose: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:03.105Z [WARN] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: Election timeout reached, restarting election - testing.go:149: server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: 2021-07-10T01:05:03.105Z [WARN] server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: heartbeat timeout reached, starting election: last-leader=2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab -testLoggerAdapter verbose: server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: 2021-07-10T01:05:03.105Z [INFO] server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: entering candidate state: node="Node at 2bbd6b80-00c9-8a45-10e8-db43464eb80a [Candidate]" term=3 - testing.go:149: server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: 2021-07-10T01:05:03.105Z [INFO] server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: entering candidate state: node="Node at 2bbd6b80-00c9-8a45-10e8-db43464eb80a [Candidate]" term=3 - testing.go:149: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:03.105Z [WARN] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: Election timeout reached, restarting election -testLoggerAdapter verbose: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:03.105Z [INFO] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: entering candidate state: node="Node at caae0ef8-5ef3-3038-2a73-a05d64d77146 [Candidate]" term=4 - testing.go:149: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:03.105Z [INFO] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: entering candidate state: node="Node at caae0ef8-5ef3-3038-2a73-a05d64d77146 [Candidate]" term=4 -testLoggerAdapter verbose: server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: 2021-07-10T01:05:03.105Z [INFO] server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: entering follower state: follower="Node at 2bbd6b80-00c9-8a45-10e8-db43464eb80a [Follower]" leader= - testing.go:149: server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: 2021-07-10T01:05:03.105Z [INFO] server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: entering follower state: follower="Node at 2bbd6b80-00c9-8a45-10e8-db43464eb80a [Follower]" leader= -testLoggerAdapter verbose: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:03.106Z [INFO] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: election won: tally=2 - testing.go:149: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:03.106Z [INFO] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: election won: tally=2 -testLoggerAdapter verbose: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:03.106Z [INFO] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: entering leader state: leader="Node at caae0ef8-5ef3-3038-2a73-a05d64d77146 [Leader]" - testing.go:149: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:03.106Z [INFO] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: entering leader state: leader="Node at caae0ef8-5ef3-3038-2a73-a05d64d77146 [Leader]" -testLoggerAdapter verbose: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:03.106Z [INFO] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: added peer, starting replication: peer=server-2bbd6b80-00c9-8a45-10e8-db43464eb80a - testing.go:149: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:03.106Z [INFO] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: added peer, starting replication: peer=server-2bbd6b80-00c9-8a45-10e8-db43464eb80a -testLoggerAdapter verbose: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:03.106Z [INFO] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: pipelining replication: peer="{Voter server-2bbd6b80-00c9-8a45-10e8-db43464eb80a 2bbd6b80-00c9-8a45-10e8-db43464eb80a}" - testing.go:149: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:03.106Z [INFO] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: pipelining replication: peer="{Voter server-2bbd6b80-00c9-8a45-10e8-db43464eb80a 2bbd6b80-00c9-8a45-10e8-db43464eb80a}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:03.201Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:03.201Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:03.307Z [INFO] stable state for Leader reached at 2021-07-10 01:05:03.202199253 +0000 UTC m=+8.590357203 (1 nodes), 61.521µs from start of poll, 485.009454ms from cluster start. Timeout at 2021-07-10 01:05:03.30727416 +0000 UTC m=+8.695432150, 105.074947ms after stability - testing.go:149: cluster: 2021-07-10T01:05:03.307Z [INFO] stable state for Leader reached at 2021-07-10 01:05:03.202199253 +0000 UTC m=+8.590357203 (1 nodes), 61.521µs from start of poll, 485.009454ms from cluster start. Timeout at 2021-07-10 01:05:03.30727416 +0000 UTC m=+8.695432150, 105.074947ms after stability -testLoggerAdapter verbose: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:03.307Z [INFO] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: aborting pipeline replication: peer="{Voter server-2bbd6b80-00c9-8a45-10e8-db43464eb80a 2bbd6b80-00c9-8a45-10e8-db43464eb80a}" - testing.go:149: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:03.307Z [INFO] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: aborting pipeline replication: peer="{Voter server-2bbd6b80-00c9-8a45-10e8-db43464eb80a 2bbd6b80-00c9-8a45-10e8-db43464eb80a}" +testLoggerAdapter verbose: server-e89c0502-18b5-af37-f3bf-f951fac535cd: 2022-08-12T07:30:07.155Z [INFO] server-e89c0502-18b5-af37-f3bf-f951fac535cd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e89c0502-18b5-af37-f3bf-f951fac535cd Address:e89c0502-18b5-af37-f3bf-f951fac535cd} {Suffrage:Voter ID:server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95 Address:5ce4fa72-7fb2-476d-e1c9-1f62be13dd95} {Suffrage:Voter ID:server-f1132016-4a78-68a6-c1d7-09625ed86afa Address:f1132016-4a78-68a6-c1d7-09625ed86afa}]" + testing.go:149: server-e89c0502-18b5-af37-f3bf-f951fac535cd: 2022-08-12T07:30:07.155Z [INFO] server-e89c0502-18b5-af37-f3bf-f951fac535cd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e89c0502-18b5-af37-f3bf-f951fac535cd Address:e89c0502-18b5-af37-f3bf-f951fac535cd} {Suffrage:Voter ID:server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95 Address:5ce4fa72-7fb2-476d-e1c9-1f62be13dd95} {Suffrage:Voter ID:server-f1132016-4a78-68a6-c1d7-09625ed86afa Address:f1132016-4a78-68a6-c1d7-09625ed86afa}]" +testLoggerAdapter verbose: server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: 2022-08-12T07:30:07.156Z [INFO] server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e89c0502-18b5-af37-f3bf-f951fac535cd Address:e89c0502-18b5-af37-f3bf-f951fac535cd} {Suffrage:Voter ID:server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95 Address:5ce4fa72-7fb2-476d-e1c9-1f62be13dd95} {Suffrage:Voter ID:server-f1132016-4a78-68a6-c1d7-09625ed86afa Address:f1132016-4a78-68a6-c1d7-09625ed86afa}]" + testing.go:149: server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: 2022-08-12T07:30:07.156Z [INFO] server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e89c0502-18b5-af37-f3bf-f951fac535cd Address:e89c0502-18b5-af37-f3bf-f951fac535cd} {Suffrage:Voter ID:server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95 Address:5ce4fa72-7fb2-476d-e1c9-1f62be13dd95} {Suffrage:Voter ID:server-f1132016-4a78-68a6-c1d7-09625ed86afa Address:f1132016-4a78-68a6-c1d7-09625ed86afa}]" +testLoggerAdapter verbose: server-e89c0502-18b5-af37-f3bf-f951fac535cd: 2022-08-12T07:30:07.156Z [INFO] server-e89c0502-18b5-af37-f3bf-f951fac535cd: entering follower state: follower="Node at e89c0502-18b5-af37-f3bf-f951fac535cd [Follower]" leader= + testing.go:149: server-e89c0502-18b5-af37-f3bf-f951fac535cd: 2022-08-12T07:30:07.156Z [INFO] server-e89c0502-18b5-af37-f3bf-f951fac535cd: entering follower state: follower="Node at e89c0502-18b5-af37-f3bf-f951fac535cd [Follower]" leader= +testLoggerAdapter verbose: server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: 2022-08-12T07:30:07.156Z [INFO] server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: entering follower state: follower="Node at 5ce4fa72-7fb2-476d-e1c9-1f62be13dd95 [Follower]" leader= + testing.go:149: server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: 2022-08-12T07:30:07.156Z [INFO] server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: entering follower state: follower="Node at 5ce4fa72-7fb2-476d-e1c9-1f62be13dd95 [Follower]" leader= +testLoggerAdapter verbose: server-f1132016-4a78-68a6-c1d7-09625ed86afa: 2022-08-12T07:30:07.156Z [INFO] server-f1132016-4a78-68a6-c1d7-09625ed86afa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e89c0502-18b5-af37-f3bf-f951fac535cd Address:e89c0502-18b5-af37-f3bf-f951fac535cd} {Suffrage:Voter ID:server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95 Address:5ce4fa72-7fb2-476d-e1c9-1f62be13dd95} {Suffrage:Voter ID:server-f1132016-4a78-68a6-c1d7-09625ed86afa Address:f1132016-4a78-68a6-c1d7-09625ed86afa}]" + testing.go:149: server-f1132016-4a78-68a6-c1d7-09625ed86afa: 2022-08-12T07:30:07.156Z [INFO] server-f1132016-4a78-68a6-c1d7-09625ed86afa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e89c0502-18b5-af37-f3bf-f951fac535cd Address:e89c0502-18b5-af37-f3bf-f951fac535cd} {Suffrage:Voter ID:server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95 Address:5ce4fa72-7fb2-476d-e1c9-1f62be13dd95} {Suffrage:Voter ID:server-f1132016-4a78-68a6-c1d7-09625ed86afa Address:f1132016-4a78-68a6-c1d7-09625ed86afa}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:07.157Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:07.157Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-f1132016-4a78-68a6-c1d7-09625ed86afa: 2022-08-12T07:30:07.157Z [INFO] server-f1132016-4a78-68a6-c1d7-09625ed86afa: entering follower state: follower="Node at f1132016-4a78-68a6-c1d7-09625ed86afa [Follower]" leader= + testing.go:149: server-f1132016-4a78-68a6-c1d7-09625ed86afa: 2022-08-12T07:30:07.157Z [INFO] server-f1132016-4a78-68a6-c1d7-09625ed86afa: entering follower state: follower="Node at f1132016-4a78-68a6-c1d7-09625ed86afa [Follower]" leader= +testLoggerAdapter verbose: server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: 2022-08-12T07:30:07.207Z [WARN] server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: 2022-08-12T07:30:07.207Z [WARN] server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: 2022-08-12T07:30:07.207Z [INFO] server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: entering candidate state: node="Node at 5ce4fa72-7fb2-476d-e1c9-1f62be13dd95 [Candidate]" term=2 + testing.go:149: server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: 2022-08-12T07:30:07.207Z [INFO] server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: entering candidate state: node="Node at 5ce4fa72-7fb2-476d-e1c9-1f62be13dd95 [Candidate]" term=2 +testLoggerAdapter verbose: server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: 2022-08-12T07:30:07.208Z [INFO] server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: election won: tally=2 + testing.go:149: server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: 2022-08-12T07:30:07.208Z [INFO] server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: election won: tally=2 +testLoggerAdapter verbose: server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: 2022-08-12T07:30:07.208Z [INFO] server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: entering leader state: leader="Node at 5ce4fa72-7fb2-476d-e1c9-1f62be13dd95 [Leader]" + testing.go:149: server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: 2022-08-12T07:30:07.208Z [INFO] server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: entering leader state: leader="Node at 5ce4fa72-7fb2-476d-e1c9-1f62be13dd95 [Leader]" +testLoggerAdapter verbose: server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: 2022-08-12T07:30:07.208Z [INFO] server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: added peer, starting replication: peer=server-e89c0502-18b5-af37-f3bf-f951fac535cd + testing.go:149: server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: 2022-08-12T07:30:07.208Z [INFO] server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: added peer, starting replication: peer=server-e89c0502-18b5-af37-f3bf-f951fac535cd +testLoggerAdapter verbose: server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: 2022-08-12T07:30:07.208Z [INFO] server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: added peer, starting replication: peer=server-f1132016-4a78-68a6-c1d7-09625ed86afa + testing.go:149: server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: 2022-08-12T07:30:07.208Z [INFO] server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: added peer, starting replication: peer=server-f1132016-4a78-68a6-c1d7-09625ed86afa +testLoggerAdapter verbose: server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: 2022-08-12T07:30:07.208Z [INFO] server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: pipelining replication: peer="{Voter server-e89c0502-18b5-af37-f3bf-f951fac535cd e89c0502-18b5-af37-f3bf-f951fac535cd}" + testing.go:149: server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: 2022-08-12T07:30:07.208Z [INFO] server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: pipelining replication: peer="{Voter server-e89c0502-18b5-af37-f3bf-f951fac535cd e89c0502-18b5-af37-f3bf-f951fac535cd}" +testLoggerAdapter verbose: server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: 2022-08-12T07:30:07.208Z [INFO] server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: pipelining replication: peer="{Voter server-f1132016-4a78-68a6-c1d7-09625ed86afa f1132016-4a78-68a6-c1d7-09625ed86afa}" + testing.go:149: server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: 2022-08-12T07:30:07.208Z [INFO] server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: pipelining replication: peer="{Voter server-f1132016-4a78-68a6-c1d7-09625ed86afa f1132016-4a78-68a6-c1d7-09625ed86afa}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:07.313Z [INFO] stable state for Leader reached at 2022-08-12 07:30:07.20826904 +0000 UTC m=+8.164955521 (1 nodes), 51.18404ms from start of poll, 52.74658ms from cluster start. Timeout at 2022-08-12 07:30:07.31335384 +0000 UTC m=+8.270040341, 105.08482ms after stability + testing.go:149: cluster: 2022-08-12T07:30:07.313Z [INFO] stable state for Leader reached at 2022-08-12 07:30:07.20826904 +0000 UTC m=+8.164955521 (1 nodes), 51.18404ms from start of poll, 52.74658ms from cluster start. Timeout at 2022-08-12 07:30:07.31335384 +0000 UTC m=+8.270040341, 105.08482ms after stability +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:07.318Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2022-08-12T07:30:07.318Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:07.423Z [INFO] stable state for Follower reached at 2022-08-12 07:30:07.31884188 +0000 UTC m=+8.275528361 (2 nodes), 2.7µs from start of poll, 163.31942ms from cluster start. Timeout at 2022-08-12 07:30:07.42391798 +0000 UTC m=+8.380604441, 105.07608ms after stability + testing.go:149: cluster: 2022-08-12T07:30:07.423Z [INFO] stable state for Follower reached at 2022-08-12 07:30:07.31884188 +0000 UTC m=+8.275528361 (2 nodes), 2.7µs from start of poll, 163.31942ms from cluster start. Timeout at 2022-08-12 07:30:07.42391798 +0000 UTC m=+8.380604441, 105.07608ms after stability +testLoggerAdapter verbose: server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: 2022-08-12T07:30:07.424Z [INFO] server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: updating configuration: command=RemoveServer server-id=server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95 server-addr= servers="[{Suffrage:Voter ID:server-e89c0502-18b5-af37-f3bf-f951fac535cd Address:e89c0502-18b5-af37-f3bf-f951fac535cd} {Suffrage:Voter ID:server-f1132016-4a78-68a6-c1d7-09625ed86afa Address:f1132016-4a78-68a6-c1d7-09625ed86afa}]" + testing.go:149: server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: 2022-08-12T07:30:07.424Z [INFO] server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: updating configuration: command=RemoveServer server-id=server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95 server-addr= servers="[{Suffrage:Voter ID:server-e89c0502-18b5-af37-f3bf-f951fac535cd Address:e89c0502-18b5-af37-f3bf-f951fac535cd} {Suffrage:Voter ID:server-f1132016-4a78-68a6-c1d7-09625ed86afa Address:f1132016-4a78-68a6-c1d7-09625ed86afa}]" +testLoggerAdapter verbose: server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: 2022-08-12T07:30:07.424Z [INFO] server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: removed ourself, shutting down + testing.go:149: server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: 2022-08-12T07:30:07.424Z [INFO] server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: removed ourself, shutting down +testLoggerAdapter verbose: server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: 2022-08-12T07:30:07.424Z [INFO] server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: aborting pipeline replication: peer="{Voter server-f1132016-4a78-68a6-c1d7-09625ed86afa f1132016-4a78-68a6-c1d7-09625ed86afa}" + testing.go:149: server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: 2022-08-12T07:30:07.424Z [INFO] server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: aborting pipeline replication: peer="{Voter server-f1132016-4a78-68a6-c1d7-09625ed86afa f1132016-4a78-68a6-c1d7-09625ed86afa}" +testLoggerAdapter verbose: server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: 2022-08-12T07:30:07.424Z [INFO] server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: aborting pipeline replication: peer="{Voter server-e89c0502-18b5-af37-f3bf-f951fac535cd e89c0502-18b5-af37-f3bf-f951fac535cd}" + testing.go:149: server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: 2022-08-12T07:30:07.424Z [INFO] server-5ce4fa72-7fb2-476d-e1c9-1f62be13dd95: aborting pipeline replication: peer="{Voter server-e89c0502-18b5-af37-f3bf-f951fac535cd e89c0502-18b5-af37-f3bf-f951fac535cd}" +testLoggerAdapter verbose: server-e89c0502-18b5-af37-f3bf-f951fac535cd: 2022-08-12T07:30:07.474Z [WARN] server-e89c0502-18b5-af37-f3bf-f951fac535cd: heartbeat timeout reached, starting election: last-leader=5ce4fa72-7fb2-476d-e1c9-1f62be13dd95 + testing.go:149: server-e89c0502-18b5-af37-f3bf-f951fac535cd: 2022-08-12T07:30:07.474Z [WARN] server-e89c0502-18b5-af37-f3bf-f951fac535cd: heartbeat timeout reached, starting election: last-leader=5ce4fa72-7fb2-476d-e1c9-1f62be13dd95 +testLoggerAdapter verbose: server-e89c0502-18b5-af37-f3bf-f951fac535cd: 2022-08-12T07:30:07.474Z [INFO] server-e89c0502-18b5-af37-f3bf-f951fac535cd: entering candidate state: node="Node at e89c0502-18b5-af37-f3bf-f951fac535cd [Candidate]" term=3 + testing.go:149: server-e89c0502-18b5-af37-f3bf-f951fac535cd: 2022-08-12T07:30:07.474Z [INFO] server-e89c0502-18b5-af37-f3bf-f951fac535cd: entering candidate state: node="Node at e89c0502-18b5-af37-f3bf-f951fac535cd [Candidate]" term=3 +testLoggerAdapter verbose: server-f1132016-4a78-68a6-c1d7-09625ed86afa: 2022-08-12T07:30:07.475Z [WARN] server-f1132016-4a78-68a6-c1d7-09625ed86afa: rejecting vote request since we have a leader: from=e89c0502-18b5-af37-f3bf-f951fac535cd leader=5ce4fa72-7fb2-476d-e1c9-1f62be13dd95 + testing.go:149: server-f1132016-4a78-68a6-c1d7-09625ed86afa: 2022-08-12T07:30:07.475Z [WARN] server-f1132016-4a78-68a6-c1d7-09625ed86afa: rejecting vote request since we have a leader: from=e89c0502-18b5-af37-f3bf-f951fac535cd leader=5ce4fa72-7fb2-476d-e1c9-1f62be13dd95 +testLoggerAdapter verbose: server-f1132016-4a78-68a6-c1d7-09625ed86afa: 2022-08-12T07:30:07.499Z [WARN] server-f1132016-4a78-68a6-c1d7-09625ed86afa: heartbeat timeout reached, starting election: last-leader=5ce4fa72-7fb2-476d-e1c9-1f62be13dd95 + testing.go:149: server-f1132016-4a78-68a6-c1d7-09625ed86afa: 2022-08-12T07:30:07.499Z [WARN] server-f1132016-4a78-68a6-c1d7-09625ed86afa: heartbeat timeout reached, starting election: last-leader=5ce4fa72-7fb2-476d-e1c9-1f62be13dd95 +testLoggerAdapter verbose: server-f1132016-4a78-68a6-c1d7-09625ed86afa: 2022-08-12T07:30:07.499Z [INFO] server-f1132016-4a78-68a6-c1d7-09625ed86afa: entering candidate state: node="Node at f1132016-4a78-68a6-c1d7-09625ed86afa [Candidate]" term=3 + testing.go:149: server-f1132016-4a78-68a6-c1d7-09625ed86afa: 2022-08-12T07:30:07.499Z [INFO] server-f1132016-4a78-68a6-c1d7-09625ed86afa: entering candidate state: node="Node at f1132016-4a78-68a6-c1d7-09625ed86afa [Candidate]" term=3 +testLoggerAdapter verbose: server-e89c0502-18b5-af37-f3bf-f951fac535cd: 2022-08-12T07:30:07.499Z [INFO] server-e89c0502-18b5-af37-f3bf-f951fac535cd: duplicate requestVote for same term: term=3 + testing.go:149: server-e89c0502-18b5-af37-f3bf-f951fac535cd: 2022-08-12T07:30:07.499Z [INFO] server-e89c0502-18b5-af37-f3bf-f951fac535cd: duplicate requestVote for same term: term=3 +testLoggerAdapter verbose: server-e89c0502-18b5-af37-f3bf-f951fac535cd: 2022-08-12T07:30:07.542Z [WARN] server-e89c0502-18b5-af37-f3bf-f951fac535cd: Election timeout reached, restarting election + testing.go:149: server-e89c0502-18b5-af37-f3bf-f951fac535cd: 2022-08-12T07:30:07.542Z [WARN] server-e89c0502-18b5-af37-f3bf-f951fac535cd: Election timeout reached, restarting election +testLoggerAdapter verbose: server-e89c0502-18b5-af37-f3bf-f951fac535cd: 2022-08-12T07:30:07.542Z [INFO] server-e89c0502-18b5-af37-f3bf-f951fac535cd: entering candidate state: node="Node at e89c0502-18b5-af37-f3bf-f951fac535cd [Candidate]" term=4 + testing.go:149: server-e89c0502-18b5-af37-f3bf-f951fac535cd: 2022-08-12T07:30:07.542Z [INFO] server-e89c0502-18b5-af37-f3bf-f951fac535cd: entering candidate state: node="Node at e89c0502-18b5-af37-f3bf-f951fac535cd [Candidate]" term=4 +testLoggerAdapter verbose: server-f1132016-4a78-68a6-c1d7-09625ed86afa: 2022-08-12T07:30:07.542Z [INFO] server-f1132016-4a78-68a6-c1d7-09625ed86afa: entering follower state: follower="Node at f1132016-4a78-68a6-c1d7-09625ed86afa [Follower]" leader= + testing.go:149: server-f1132016-4a78-68a6-c1d7-09625ed86afa: 2022-08-12T07:30:07.542Z [INFO] server-f1132016-4a78-68a6-c1d7-09625ed86afa: entering follower state: follower="Node at f1132016-4a78-68a6-c1d7-09625ed86afa [Follower]" leader= +testLoggerAdapter verbose: server-e89c0502-18b5-af37-f3bf-f951fac535cd: 2022-08-12T07:30:07.542Z [INFO] server-e89c0502-18b5-af37-f3bf-f951fac535cd: election won: tally=2 + testing.go:149: server-e89c0502-18b5-af37-f3bf-f951fac535cd: 2022-08-12T07:30:07.542Z [INFO] server-e89c0502-18b5-af37-f3bf-f951fac535cd: election won: tally=2 +testLoggerAdapter verbose: server-e89c0502-18b5-af37-f3bf-f951fac535cd: 2022-08-12T07:30:07.542Z [INFO] server-e89c0502-18b5-af37-f3bf-f951fac535cd: entering leader state: leader="Node at e89c0502-18b5-af37-f3bf-f951fac535cd [Leader]" + testing.go:149: server-e89c0502-18b5-af37-f3bf-f951fac535cd: 2022-08-12T07:30:07.542Z [INFO] server-e89c0502-18b5-af37-f3bf-f951fac535cd: entering leader state: leader="Node at e89c0502-18b5-af37-f3bf-f951fac535cd [Leader]" +testLoggerAdapter verbose: server-e89c0502-18b5-af37-f3bf-f951fac535cd: 2022-08-12T07:30:07.542Z [INFO] server-e89c0502-18b5-af37-f3bf-f951fac535cd: added peer, starting replication: peer=server-f1132016-4a78-68a6-c1d7-09625ed86afa + testing.go:149: server-e89c0502-18b5-af37-f3bf-f951fac535cd: 2022-08-12T07:30:07.542Z [INFO] server-e89c0502-18b5-af37-f3bf-f951fac535cd: added peer, starting replication: peer=server-f1132016-4a78-68a6-c1d7-09625ed86afa +testLoggerAdapter verbose: server-e89c0502-18b5-af37-f3bf-f951fac535cd: 2022-08-12T07:30:07.542Z [INFO] server-e89c0502-18b5-af37-f3bf-f951fac535cd: pipelining replication: peer="{Voter server-f1132016-4a78-68a6-c1d7-09625ed86afa f1132016-4a78-68a6-c1d7-09625ed86afa}" + testing.go:149: server-e89c0502-18b5-af37-f3bf-f951fac535cd: 2022-08-12T07:30:07.542Z [INFO] server-e89c0502-18b5-af37-f3bf-f951fac535cd: pipelining replication: peer="{Voter server-f1132016-4a78-68a6-c1d7-09625ed86afa f1132016-4a78-68a6-c1d7-09625ed86afa}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:07.634Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:07.634Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:07.740Z [INFO] stable state for Leader reached at 2022-08-12 07:30:07.6349535 +0000 UTC m=+8.591639981 (1 nodes), 67.3µs from start of poll, 479.43104ms from cluster start. Timeout at 2022-08-12 07:30:07.74002098 +0000 UTC m=+8.696707461, 105.06748ms after stability + testing.go:149: cluster: 2022-08-12T07:30:07.740Z [INFO] stable state for Leader reached at 2022-08-12 07:30:07.6349535 +0000 UTC m=+8.591639981 (1 nodes), 67.3µs from start of poll, 479.43104ms from cluster start. Timeout at 2022-08-12 07:30:07.74002098 +0000 UTC m=+8.696707461, 105.06748ms after stability +testLoggerAdapter verbose: server-e89c0502-18b5-af37-f3bf-f951fac535cd: 2022-08-12T07:30:07.740Z [INFO] server-e89c0502-18b5-af37-f3bf-f951fac535cd: aborting pipeline replication: peer="{Voter server-f1132016-4a78-68a6-c1d7-09625ed86afa f1132016-4a78-68a6-c1d7-09625ed86afa}" + testing.go:149: server-e89c0502-18b5-af37-f3bf-f951fac535cd: 2022-08-12T07:30:07.740Z [INFO] server-e89c0502-18b5-af37-f3bf-f951fac535cd: aborting pipeline replication: peer="{Voter server-f1132016-4a78-68a6-c1d7-09625ed86afa f1132016-4a78-68a6-c1d7-09625ed86afa}" --- PASS: TestRaft_RemoveLeader (0.59s) === RUN TestRaft_RemoveLeader_NoShutdown -testLoggerAdapter verbose: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.309Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6 Address:8a0b128e-a408-73d9-bbe2-b4581e28fdb6} {Suffrage:Voter ID:server-ce7230b8-5299-7647-eb53-198e68cf4494 Address:ce7230b8-5299-7647-eb53-198e68cf4494} {Suffrage:Voter ID:server-af31b31c-daea-1470-20d1-b0245fe34199 Address:af31b31c-daea-1470-20d1-b0245fe34199}]" - testing.go:149: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.309Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6 Address:8a0b128e-a408-73d9-bbe2-b4581e28fdb6} {Suffrage:Voter ID:server-ce7230b8-5299-7647-eb53-198e68cf4494 Address:ce7230b8-5299-7647-eb53-198e68cf4494} {Suffrage:Voter ID:server-af31b31c-daea-1470-20d1-b0245fe34199 Address:af31b31c-daea-1470-20d1-b0245fe34199}]" -testLoggerAdapter verbose: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.309Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: entering follower state: follower="Node at 8a0b128e-a408-73d9-bbe2-b4581e28fdb6 [Follower]" leader= - testing.go:149: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.309Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: entering follower state: follower="Node at 8a0b128e-a408-73d9-bbe2-b4581e28fdb6 [Follower]" leader= -testLoggerAdapter verbose: server-ce7230b8-5299-7647-eb53-198e68cf4494: 2021-07-10T01:05:03.309Z [INFO] server-ce7230b8-5299-7647-eb53-198e68cf4494: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6 Address:8a0b128e-a408-73d9-bbe2-b4581e28fdb6} {Suffrage:Voter ID:server-ce7230b8-5299-7647-eb53-198e68cf4494 Address:ce7230b8-5299-7647-eb53-198e68cf4494} {Suffrage:Voter ID:server-af31b31c-daea-1470-20d1-b0245fe34199 Address:af31b31c-daea-1470-20d1-b0245fe34199}]" - testing.go:149: server-ce7230b8-5299-7647-eb53-198e68cf4494: 2021-07-10T01:05:03.309Z [INFO] server-ce7230b8-5299-7647-eb53-198e68cf4494: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6 Address:8a0b128e-a408-73d9-bbe2-b4581e28fdb6} {Suffrage:Voter ID:server-ce7230b8-5299-7647-eb53-198e68cf4494 Address:ce7230b8-5299-7647-eb53-198e68cf4494} {Suffrage:Voter ID:server-af31b31c-daea-1470-20d1-b0245fe34199 Address:af31b31c-daea-1470-20d1-b0245fe34199}]" -testLoggerAdapter verbose: server-ce7230b8-5299-7647-eb53-198e68cf4494: 2021-07-10T01:05:03.309Z [INFO] server-ce7230b8-5299-7647-eb53-198e68cf4494: entering follower state: follower="Node at ce7230b8-5299-7647-eb53-198e68cf4494 [Follower]" leader= - testing.go:149: server-ce7230b8-5299-7647-eb53-198e68cf4494: 2021-07-10T01:05:03.309Z [INFO] server-ce7230b8-5299-7647-eb53-198e68cf4494: entering follower state: follower="Node at ce7230b8-5299-7647-eb53-198e68cf4494 [Follower]" leader= -testLoggerAdapter verbose: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.309Z [INFO] server-af31b31c-daea-1470-20d1-b0245fe34199: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6 Address:8a0b128e-a408-73d9-bbe2-b4581e28fdb6} {Suffrage:Voter ID:server-ce7230b8-5299-7647-eb53-198e68cf4494 Address:ce7230b8-5299-7647-eb53-198e68cf4494} {Suffrage:Voter ID:server-af31b31c-daea-1470-20d1-b0245fe34199 Address:af31b31c-daea-1470-20d1-b0245fe34199}]" - testing.go:149: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.309Z [INFO] server-af31b31c-daea-1470-20d1-b0245fe34199: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6 Address:8a0b128e-a408-73d9-bbe2-b4581e28fdb6} {Suffrage:Voter ID:server-ce7230b8-5299-7647-eb53-198e68cf4494 Address:ce7230b8-5299-7647-eb53-198e68cf4494} {Suffrage:Voter ID:server-af31b31c-daea-1470-20d1-b0245fe34199 Address:af31b31c-daea-1470-20d1-b0245fe34199}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:03.309Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-07-10T01:05:03.309Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.309Z [INFO] server-af31b31c-daea-1470-20d1-b0245fe34199: entering follower state: follower="Node at af31b31c-daea-1470-20d1-b0245fe34199 [Follower]" leader= - testing.go:149: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.309Z [INFO] server-af31b31c-daea-1470-20d1-b0245fe34199: entering follower state: follower="Node at af31b31c-daea-1470-20d1-b0245fe34199 [Follower]" leader= -testLoggerAdapter verbose: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.366Z [WARN] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.366Z [WARN] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.366Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: entering candidate state: node="Node at 8a0b128e-a408-73d9-bbe2-b4581e28fdb6 [Candidate]" term=2 - testing.go:149: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.366Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: entering candidate state: node="Node at 8a0b128e-a408-73d9-bbe2-b4581e28fdb6 [Candidate]" term=2 -testLoggerAdapter verbose: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.366Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: election won: tally=2 - testing.go:149: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.366Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: election won: tally=2 -testLoggerAdapter verbose: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.366Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: entering leader state: leader="Node at 8a0b128e-a408-73d9-bbe2-b4581e28fdb6 [Leader]" - testing.go:149: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.366Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: entering leader state: leader="Node at 8a0b128e-a408-73d9-bbe2-b4581e28fdb6 [Leader]" -testLoggerAdapter verbose: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.366Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: added peer, starting replication: peer=server-ce7230b8-5299-7647-eb53-198e68cf4494 - testing.go:149: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.366Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: added peer, starting replication: peer=server-ce7230b8-5299-7647-eb53-198e68cf4494 -testLoggerAdapter verbose: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.366Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: added peer, starting replication: peer=server-af31b31c-daea-1470-20d1-b0245fe34199 - testing.go:149: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.366Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: added peer, starting replication: peer=server-af31b31c-daea-1470-20d1-b0245fe34199 -testLoggerAdapter verbose: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.366Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: pipelining replication: peer="{Voter server-ce7230b8-5299-7647-eb53-198e68cf4494 ce7230b8-5299-7647-eb53-198e68cf4494}" - testing.go:149: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.366Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: pipelining replication: peer="{Voter server-ce7230b8-5299-7647-eb53-198e68cf4494 ce7230b8-5299-7647-eb53-198e68cf4494}" -testLoggerAdapter verbose: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.366Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: pipelining replication: peer="{Voter server-af31b31c-daea-1470-20d1-b0245fe34199 af31b31c-daea-1470-20d1-b0245fe34199}" - testing.go:149: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.366Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: pipelining replication: peer="{Voter server-af31b31c-daea-1470-20d1-b0245fe34199 af31b31c-daea-1470-20d1-b0245fe34199}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:03.471Z [INFO] stable state for Follower reached at 2021-07-10 01:05:03.366643486 +0000 UTC m=+8.754801476 (2 nodes), 56.89888ms from start of poll, 57.83321ms from cluster start. Timeout at 2021-07-10 01:05:03.471708894 +0000 UTC m=+8.859866864, 105.065388ms after stability - testing.go:149: cluster: 2021-07-10T01:05:03.471Z [INFO] stable state for Follower reached at 2021-07-10 01:05:03.366643486 +0000 UTC m=+8.754801476 (2 nodes), 56.89888ms from start of poll, 57.83321ms from cluster start. Timeout at 2021-07-10 01:05:03.471708894 +0000 UTC m=+8.859866864, 105.065388ms after stability -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:03.471Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:03.471Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:03.577Z [INFO] stable state for Leader reached at 2021-07-10 01:05:03.471979617 +0000 UTC m=+8.860137587 (1 nodes), 1.82µs from start of poll, 163.169321ms from cluster start. Timeout at 2021-07-10 01:05:03.577049764 +0000 UTC m=+8.965207754, 105.070167ms after stability - testing.go:149: cluster: 2021-07-10T01:05:03.577Z [INFO] stable state for Leader reached at 2021-07-10 01:05:03.471979617 +0000 UTC m=+8.860137587 (1 nodes), 1.82µs from start of poll, 163.169321ms from cluster start. Timeout at 2021-07-10 01:05:03.577049764 +0000 UTC m=+8.965207754, 105.070167ms after stability -testLoggerAdapter verbose: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.579Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: updating configuration: command=RemoveServer server-id=server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6 server-addr= servers="[{Suffrage:Voter ID:server-ce7230b8-5299-7647-eb53-198e68cf4494 Address:ce7230b8-5299-7647-eb53-198e68cf4494} {Suffrage:Voter ID:server-af31b31c-daea-1470-20d1-b0245fe34199 Address:af31b31c-daea-1470-20d1-b0245fe34199}]" - testing.go:149: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.579Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: updating configuration: command=RemoveServer server-id=server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6 server-addr= servers="[{Suffrage:Voter ID:server-ce7230b8-5299-7647-eb53-198e68cf4494 Address:ce7230b8-5299-7647-eb53-198e68cf4494} {Suffrage:Voter ID:server-af31b31c-daea-1470-20d1-b0245fe34199 Address:af31b31c-daea-1470-20d1-b0245fe34199}]" -testLoggerAdapter verbose: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.579Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: removed ourself, transitioning to follower - testing.go:149: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.579Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: removed ourself, transitioning to follower -testLoggerAdapter verbose: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.579Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: entering follower state: follower="Node at 8a0b128e-a408-73d9-bbe2-b4581e28fdb6 [Follower]" leader= - testing.go:149: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.579Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: entering follower state: follower="Node at 8a0b128e-a408-73d9-bbe2-b4581e28fdb6 [Follower]" leader= -testLoggerAdapter verbose: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.579Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: aborting pipeline replication: peer="{Voter server-af31b31c-daea-1470-20d1-b0245fe34199 af31b31c-daea-1470-20d1-b0245fe34199}" - testing.go:149: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.579Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: aborting pipeline replication: peer="{Voter server-af31b31c-daea-1470-20d1-b0245fe34199 af31b31c-daea-1470-20d1-b0245fe34199}" -testLoggerAdapter verbose: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.579Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: aborting pipeline replication: peer="{Voter server-ce7230b8-5299-7647-eb53-198e68cf4494 ce7230b8-5299-7647-eb53-198e68cf4494}" - testing.go:149: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.579Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: aborting pipeline replication: peer="{Voter server-ce7230b8-5299-7647-eb53-198e68cf4494 ce7230b8-5299-7647-eb53-198e68cf4494}" -testLoggerAdapter verbose: server-ce7230b8-5299-7647-eb53-198e68cf4494: 2021-07-10T01:05:03.651Z [WARN] server-ce7230b8-5299-7647-eb53-198e68cf4494: heartbeat timeout reached, starting election: last-leader=8a0b128e-a408-73d9-bbe2-b4581e28fdb6 - testing.go:149: server-ce7230b8-5299-7647-eb53-198e68cf4494: 2021-07-10T01:05:03.651Z [WARN] server-ce7230b8-5299-7647-eb53-198e68cf4494: heartbeat timeout reached, starting election: last-leader=8a0b128e-a408-73d9-bbe2-b4581e28fdb6 -testLoggerAdapter verbose: server-ce7230b8-5299-7647-eb53-198e68cf4494: 2021-07-10T01:05:03.651Z [INFO] server-ce7230b8-5299-7647-eb53-198e68cf4494: entering candidate state: node="Node at ce7230b8-5299-7647-eb53-198e68cf4494 [Candidate]" term=3 - testing.go:149: server-ce7230b8-5299-7647-eb53-198e68cf4494: 2021-07-10T01:05:03.651Z [INFO] server-ce7230b8-5299-7647-eb53-198e68cf4494: entering candidate state: node="Node at ce7230b8-5299-7647-eb53-198e68cf4494 [Candidate]" term=3 -testLoggerAdapter verbose: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.652Z [WARN] server-af31b31c-daea-1470-20d1-b0245fe34199: rejecting vote request since we have a leader: from=ce7230b8-5299-7647-eb53-198e68cf4494 leader=8a0b128e-a408-73d9-bbe2-b4581e28fdb6 - testing.go:149: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.652Z [WARN] server-af31b31c-daea-1470-20d1-b0245fe34199: rejecting vote request since we have a leader: from=ce7230b8-5299-7647-eb53-198e68cf4494 leader=8a0b128e-a408-73d9-bbe2-b4581e28fdb6 -testLoggerAdapter verbose: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.659Z [WARN] server-af31b31c-daea-1470-20d1-b0245fe34199: heartbeat timeout reached, starting election: last-leader=8a0b128e-a408-73d9-bbe2-b4581e28fdb6 - testing.go:149: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.659Z [WARN] server-af31b31c-daea-1470-20d1-b0245fe34199: heartbeat timeout reached, starting election: last-leader=8a0b128e-a408-73d9-bbe2-b4581e28fdb6 -testLoggerAdapter verbose: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.659Z [INFO] server-af31b31c-daea-1470-20d1-b0245fe34199: entering candidate state: node="Node at af31b31c-daea-1470-20d1-b0245fe34199 [Candidate]" term=3 - testing.go:149: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.659Z [INFO] server-af31b31c-daea-1470-20d1-b0245fe34199: entering candidate state: node="Node at af31b31c-daea-1470-20d1-b0245fe34199 [Candidate]" term=3 -testLoggerAdapter verbose: server-ce7230b8-5299-7647-eb53-198e68cf4494: 2021-07-10T01:05:03.659Z [INFO] server-ce7230b8-5299-7647-eb53-198e68cf4494: duplicate requestVote for same term: term=3 - testing.go:149: server-ce7230b8-5299-7647-eb53-198e68cf4494: 2021-07-10T01:05:03.659Z [INFO] server-ce7230b8-5299-7647-eb53-198e68cf4494: duplicate requestVote for same term: term=3 -testLoggerAdapter verbose: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.677Z [WARN] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: not part of stable configuration, aborting election - testing.go:149: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.677Z [WARN] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: not part of stable configuration, aborting election -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:03.684Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:03.684Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.722Z [WARN] server-af31b31c-daea-1470-20d1-b0245fe34199: Election timeout reached, restarting election - testing.go:149: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.722Z [WARN] server-af31b31c-daea-1470-20d1-b0245fe34199: Election timeout reached, restarting election -testLoggerAdapter verbose: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.723Z [INFO] server-af31b31c-daea-1470-20d1-b0245fe34199: entering candidate state: node="Node at af31b31c-daea-1470-20d1-b0245fe34199 [Candidate]" term=4 - testing.go:149: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.723Z [INFO] server-af31b31c-daea-1470-20d1-b0245fe34199: entering candidate state: node="Node at af31b31c-daea-1470-20d1-b0245fe34199 [Candidate]" term=4 -testLoggerAdapter verbose: server-ce7230b8-5299-7647-eb53-198e68cf4494: 2021-07-10T01:05:03.723Z [INFO] server-ce7230b8-5299-7647-eb53-198e68cf4494: entering follower state: follower="Node at ce7230b8-5299-7647-eb53-198e68cf4494 [Follower]" leader= - testing.go:149: server-ce7230b8-5299-7647-eb53-198e68cf4494: 2021-07-10T01:05:03.723Z [INFO] server-ce7230b8-5299-7647-eb53-198e68cf4494: entering follower state: follower="Node at ce7230b8-5299-7647-eb53-198e68cf4494 [Follower]" leader= -testLoggerAdapter verbose: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.723Z [INFO] server-af31b31c-daea-1470-20d1-b0245fe34199: election won: tally=2 - testing.go:149: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.723Z [INFO] server-af31b31c-daea-1470-20d1-b0245fe34199: election won: tally=2 -testLoggerAdapter verbose: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.723Z [INFO] server-af31b31c-daea-1470-20d1-b0245fe34199: entering leader state: leader="Node at af31b31c-daea-1470-20d1-b0245fe34199 [Leader]" - testing.go:149: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.723Z [INFO] server-af31b31c-daea-1470-20d1-b0245fe34199: entering leader state: leader="Node at af31b31c-daea-1470-20d1-b0245fe34199 [Leader]" -testLoggerAdapter verbose: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.723Z [INFO] server-af31b31c-daea-1470-20d1-b0245fe34199: added peer, starting replication: peer=server-ce7230b8-5299-7647-eb53-198e68cf4494 - testing.go:149: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.723Z [INFO] server-af31b31c-daea-1470-20d1-b0245fe34199: added peer, starting replication: peer=server-ce7230b8-5299-7647-eb53-198e68cf4494 -testLoggerAdapter verbose: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.723Z [INFO] server-af31b31c-daea-1470-20d1-b0245fe34199: pipelining replication: peer="{Voter server-ce7230b8-5299-7647-eb53-198e68cf4494 ce7230b8-5299-7647-eb53-198e68cf4494}" - testing.go:149: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.723Z [INFO] server-af31b31c-daea-1470-20d1-b0245fe34199: pipelining replication: peer="{Voter server-ce7230b8-5299-7647-eb53-198e68cf4494 ce7230b8-5299-7647-eb53-198e68cf4494}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:03.828Z [INFO] stable state for Leader reached at 2021-07-10 01:05:03.723543569 +0000 UTC m=+9.111701559 (1 nodes), 38.499726ms from start of poll, 414.733293ms from cluster start. Timeout at 2021-07-10 01:05:03.828617076 +0000 UTC m=+9.216775046, 105.073487ms after stability - testing.go:149: cluster: 2021-07-10T01:05:03.828Z [INFO] stable state for Leader reached at 2021-07-10 01:05:03.723543569 +0000 UTC m=+9.111701559 (1 nodes), 38.499726ms from start of poll, 414.733293ms from cluster start. Timeout at 2021-07-10 01:05:03.828617076 +0000 UTC m=+9.216775046, 105.073487ms after stability -testLoggerAdapter verbose: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.933Z [INFO] server-af31b31c-daea-1470-20d1-b0245fe34199: aborting pipeline replication: peer="{Voter server-ce7230b8-5299-7647-eb53-198e68cf4494 ce7230b8-5299-7647-eb53-198e68cf4494}" - testing.go:149: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.933Z [INFO] server-af31b31c-daea-1470-20d1-b0245fe34199: aborting pipeline replication: peer="{Voter server-ce7230b8-5299-7647-eb53-198e68cf4494 ce7230b8-5299-7647-eb53-198e68cf4494}" ---- PASS: TestRaft_RemoveLeader_NoShutdown (0.63s) +testLoggerAdapter verbose: server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: 2022-08-12T07:30:07.742Z [INFO] server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3098eac-f146-ebec-ad6f-7fb29c171eb0 Address:d3098eac-f146-ebec-ad6f-7fb29c171eb0} {Suffrage:Voter ID:server-c9801220-3635-5f0a-cc85-efd24dbd57ea Address:c9801220-3635-5f0a-cc85-efd24dbd57ea} {Suffrage:Voter ID:server-cf81022f-4714-41c1-0fcb-c0313a53cb86 Address:cf81022f-4714-41c1-0fcb-c0313a53cb86}]" + testing.go:149: server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: 2022-08-12T07:30:07.742Z [INFO] server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3098eac-f146-ebec-ad6f-7fb29c171eb0 Address:d3098eac-f146-ebec-ad6f-7fb29c171eb0} {Suffrage:Voter ID:server-c9801220-3635-5f0a-cc85-efd24dbd57ea Address:c9801220-3635-5f0a-cc85-efd24dbd57ea} {Suffrage:Voter ID:server-cf81022f-4714-41c1-0fcb-c0313a53cb86 Address:cf81022f-4714-41c1-0fcb-c0313a53cb86}]" +testLoggerAdapter verbose: server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: 2022-08-12T07:30:07.742Z [INFO] server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: entering follower state: follower="Node at d3098eac-f146-ebec-ad6f-7fb29c171eb0 [Follower]" leader= + testing.go:149: server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: 2022-08-12T07:30:07.742Z [INFO] server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: entering follower state: follower="Node at d3098eac-f146-ebec-ad6f-7fb29c171eb0 [Follower]" leader= +testLoggerAdapter verbose: server-c9801220-3635-5f0a-cc85-efd24dbd57ea: 2022-08-12T07:30:07.742Z [INFO] server-c9801220-3635-5f0a-cc85-efd24dbd57ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3098eac-f146-ebec-ad6f-7fb29c171eb0 Address:d3098eac-f146-ebec-ad6f-7fb29c171eb0} {Suffrage:Voter ID:server-c9801220-3635-5f0a-cc85-efd24dbd57ea Address:c9801220-3635-5f0a-cc85-efd24dbd57ea} {Suffrage:Voter ID:server-cf81022f-4714-41c1-0fcb-c0313a53cb86 Address:cf81022f-4714-41c1-0fcb-c0313a53cb86}]" + testing.go:149: server-c9801220-3635-5f0a-cc85-efd24dbd57ea: 2022-08-12T07:30:07.742Z [INFO] server-c9801220-3635-5f0a-cc85-efd24dbd57ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3098eac-f146-ebec-ad6f-7fb29c171eb0 Address:d3098eac-f146-ebec-ad6f-7fb29c171eb0} {Suffrage:Voter ID:server-c9801220-3635-5f0a-cc85-efd24dbd57ea Address:c9801220-3635-5f0a-cc85-efd24dbd57ea} {Suffrage:Voter ID:server-cf81022f-4714-41c1-0fcb-c0313a53cb86 Address:cf81022f-4714-41c1-0fcb-c0313a53cb86}]" +testLoggerAdapter verbose: server-c9801220-3635-5f0a-cc85-efd24dbd57ea: 2022-08-12T07:30:07.742Z [INFO] server-c9801220-3635-5f0a-cc85-efd24dbd57ea: entering follower state: follower="Node at c9801220-3635-5f0a-cc85-efd24dbd57ea [Follower]" leader= + testing.go:149: server-c9801220-3635-5f0a-cc85-efd24dbd57ea: 2022-08-12T07:30:07.742Z [INFO] server-c9801220-3635-5f0a-cc85-efd24dbd57ea: entering follower state: follower="Node at c9801220-3635-5f0a-cc85-efd24dbd57ea [Follower]" leader= +testLoggerAdapter verbose: server-cf81022f-4714-41c1-0fcb-c0313a53cb86: 2022-08-12T07:30:07.742Z [INFO] server-cf81022f-4714-41c1-0fcb-c0313a53cb86: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3098eac-f146-ebec-ad6f-7fb29c171eb0 Address:d3098eac-f146-ebec-ad6f-7fb29c171eb0} {Suffrage:Voter ID:server-c9801220-3635-5f0a-cc85-efd24dbd57ea Address:c9801220-3635-5f0a-cc85-efd24dbd57ea} {Suffrage:Voter ID:server-cf81022f-4714-41c1-0fcb-c0313a53cb86 Address:cf81022f-4714-41c1-0fcb-c0313a53cb86}]" + testing.go:149: server-cf81022f-4714-41c1-0fcb-c0313a53cb86: 2022-08-12T07:30:07.742Z [INFO] server-cf81022f-4714-41c1-0fcb-c0313a53cb86: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3098eac-f146-ebec-ad6f-7fb29c171eb0 Address:d3098eac-f146-ebec-ad6f-7fb29c171eb0} {Suffrage:Voter ID:server-c9801220-3635-5f0a-cc85-efd24dbd57ea Address:c9801220-3635-5f0a-cc85-efd24dbd57ea} {Suffrage:Voter ID:server-cf81022f-4714-41c1-0fcb-c0313a53cb86 Address:cf81022f-4714-41c1-0fcb-c0313a53cb86}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:07.742Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-cf81022f-4714-41c1-0fcb-c0313a53cb86: 2022-08-12T07:30:07.742Z [INFO] server-cf81022f-4714-41c1-0fcb-c0313a53cb86: entering follower state: follower="Node at cf81022f-4714-41c1-0fcb-c0313a53cb86 [Follower]" leader= + testing.go:149: cluster: 2022-08-12T07:30:07.742Z [INFO] starting stability test: raft-state=Follower + testing.go:149: server-cf81022f-4714-41c1-0fcb-c0313a53cb86: 2022-08-12T07:30:07.742Z [INFO] server-cf81022f-4714-41c1-0fcb-c0313a53cb86: entering follower state: follower="Node at cf81022f-4714-41c1-0fcb-c0313a53cb86 [Follower]" leader= +testLoggerAdapter verbose: server-cf81022f-4714-41c1-0fcb-c0313a53cb86: 2022-08-12T07:30:07.818Z [WARN] server-cf81022f-4714-41c1-0fcb-c0313a53cb86: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-cf81022f-4714-41c1-0fcb-c0313a53cb86: 2022-08-12T07:30:07.818Z [WARN] server-cf81022f-4714-41c1-0fcb-c0313a53cb86: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-cf81022f-4714-41c1-0fcb-c0313a53cb86: 2022-08-12T07:30:07.818Z [INFO] server-cf81022f-4714-41c1-0fcb-c0313a53cb86: entering candidate state: node="Node at cf81022f-4714-41c1-0fcb-c0313a53cb86 [Candidate]" term=2 + testing.go:149: server-cf81022f-4714-41c1-0fcb-c0313a53cb86: 2022-08-12T07:30:07.818Z [INFO] server-cf81022f-4714-41c1-0fcb-c0313a53cb86: entering candidate state: node="Node at cf81022f-4714-41c1-0fcb-c0313a53cb86 [Candidate]" term=2 +testLoggerAdapter verbose: server-cf81022f-4714-41c1-0fcb-c0313a53cb86: 2022-08-12T07:30:07.818Z [INFO] server-cf81022f-4714-41c1-0fcb-c0313a53cb86: election won: tally=2 + testing.go:149: server-cf81022f-4714-41c1-0fcb-c0313a53cb86: 2022-08-12T07:30:07.818Z [INFO] server-cf81022f-4714-41c1-0fcb-c0313a53cb86: election won: tally=2 +testLoggerAdapter verbose: server-cf81022f-4714-41c1-0fcb-c0313a53cb86: 2022-08-12T07:30:07.818Z [INFO] server-cf81022f-4714-41c1-0fcb-c0313a53cb86: entering leader state: leader="Node at cf81022f-4714-41c1-0fcb-c0313a53cb86 [Leader]" + testing.go:149: server-cf81022f-4714-41c1-0fcb-c0313a53cb86: 2022-08-12T07:30:07.818Z [INFO] server-cf81022f-4714-41c1-0fcb-c0313a53cb86: entering leader state: leader="Node at cf81022f-4714-41c1-0fcb-c0313a53cb86 [Leader]" +testLoggerAdapter verbose: server-cf81022f-4714-41c1-0fcb-c0313a53cb86: 2022-08-12T07:30:07.818Z [INFO] server-cf81022f-4714-41c1-0fcb-c0313a53cb86: added peer, starting replication: peer=server-d3098eac-f146-ebec-ad6f-7fb29c171eb0 + testing.go:149: server-cf81022f-4714-41c1-0fcb-c0313a53cb86: 2022-08-12T07:30:07.818Z [INFO] server-cf81022f-4714-41c1-0fcb-c0313a53cb86: added peer, starting replication: peer=server-d3098eac-f146-ebec-ad6f-7fb29c171eb0 +testLoggerAdapter verbose: server-cf81022f-4714-41c1-0fcb-c0313a53cb86: 2022-08-12T07:30:07.819Z [INFO] server-cf81022f-4714-41c1-0fcb-c0313a53cb86: added peer, starting replication: peer=server-c9801220-3635-5f0a-cc85-efd24dbd57ea + testing.go:149: server-cf81022f-4714-41c1-0fcb-c0313a53cb86: 2022-08-12T07:30:07.819Z [INFO] server-cf81022f-4714-41c1-0fcb-c0313a53cb86: added peer, starting replication: peer=server-c9801220-3635-5f0a-cc85-efd24dbd57ea +testLoggerAdapter verbose: server-cf81022f-4714-41c1-0fcb-c0313a53cb86: 2022-08-12T07:30:07.819Z [INFO] server-cf81022f-4714-41c1-0fcb-c0313a53cb86: pipelining replication: peer="{Voter server-d3098eac-f146-ebec-ad6f-7fb29c171eb0 d3098eac-f146-ebec-ad6f-7fb29c171eb0}" + testing.go:149: server-cf81022f-4714-41c1-0fcb-c0313a53cb86: 2022-08-12T07:30:07.819Z [INFO] server-cf81022f-4714-41c1-0fcb-c0313a53cb86: pipelining replication: peer="{Voter server-d3098eac-f146-ebec-ad6f-7fb29c171eb0 d3098eac-f146-ebec-ad6f-7fb29c171eb0}" +testLoggerAdapter verbose: server-cf81022f-4714-41c1-0fcb-c0313a53cb86: 2022-08-12T07:30:07.819Z [INFO] server-cf81022f-4714-41c1-0fcb-c0313a53cb86: pipelining replication: peer="{Voter server-c9801220-3635-5f0a-cc85-efd24dbd57ea c9801220-3635-5f0a-cc85-efd24dbd57ea}" + testing.go:149: server-cf81022f-4714-41c1-0fcb-c0313a53cb86: 2022-08-12T07:30:07.819Z [INFO] server-cf81022f-4714-41c1-0fcb-c0313a53cb86: pipelining replication: peer="{Voter server-c9801220-3635-5f0a-cc85-efd24dbd57ea c9801220-3635-5f0a-cc85-efd24dbd57ea}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:07.924Z [INFO] stable state for Follower reached at 2022-08-12 07:30:07.8189447 +0000 UTC m=+8.775631181 (2 nodes), 76.17652ms from start of poll, 77.26164ms from cluster start. Timeout at 2022-08-12 07:30:07.92395944 +0000 UTC m=+8.880645901, 105.01472ms after stability + testing.go:149: cluster: 2022-08-12T07:30:07.924Z [INFO] stable state for Follower reached at 2022-08-12 07:30:07.8189447 +0000 UTC m=+8.775631181 (2 nodes), 76.17652ms from start of poll, 77.26164ms from cluster start. Timeout at 2022-08-12 07:30:07.92395944 +0000 UTC m=+8.880645901, 105.01472ms after stability +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:07.924Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:07.924Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:08.029Z [INFO] stable state for Leader reached at 2022-08-12 07:30:07.92424744 +0000 UTC m=+8.880933921 (1 nodes), 2.84µs from start of poll, 182.56438ms from cluster start. Timeout at 2022-08-12 07:30:08.02927578 +0000 UTC m=+8.985962281, 105.02836ms after stability + testing.go:149: cluster: 2022-08-12T07:30:08.029Z [INFO] stable state for Leader reached at 2022-08-12 07:30:07.92424744 +0000 UTC m=+8.880933921 (1 nodes), 2.84µs from start of poll, 182.56438ms from cluster start. Timeout at 2022-08-12 07:30:08.02927578 +0000 UTC m=+8.985962281, 105.02836ms after stability +testLoggerAdapter verbose: server-cf81022f-4714-41c1-0fcb-c0313a53cb86: 2022-08-12T07:30:08.031Z [INFO] server-cf81022f-4714-41c1-0fcb-c0313a53cb86: updating configuration: command=RemoveServer server-id=server-cf81022f-4714-41c1-0fcb-c0313a53cb86 server-addr= servers="[{Suffrage:Voter ID:server-d3098eac-f146-ebec-ad6f-7fb29c171eb0 Address:d3098eac-f146-ebec-ad6f-7fb29c171eb0} {Suffrage:Voter ID:server-c9801220-3635-5f0a-cc85-efd24dbd57ea Address:c9801220-3635-5f0a-cc85-efd24dbd57ea}]" + testing.go:149: server-cf81022f-4714-41c1-0fcb-c0313a53cb86: 2022-08-12T07:30:08.031Z [INFO] server-cf81022f-4714-41c1-0fcb-c0313a53cb86: updating configuration: command=RemoveServer server-id=server-cf81022f-4714-41c1-0fcb-c0313a53cb86 server-addr= servers="[{Suffrage:Voter ID:server-d3098eac-f146-ebec-ad6f-7fb29c171eb0 Address:d3098eac-f146-ebec-ad6f-7fb29c171eb0} {Suffrage:Voter ID:server-c9801220-3635-5f0a-cc85-efd24dbd57ea Address:c9801220-3635-5f0a-cc85-efd24dbd57ea}]" +testLoggerAdapter verbose: server-cf81022f-4714-41c1-0fcb-c0313a53cb86: 2022-08-12T07:30:08.031Z [INFO] server-cf81022f-4714-41c1-0fcb-c0313a53cb86: removed ourself, transitioning to follower + testing.go:149: server-cf81022f-4714-41c1-0fcb-c0313a53cb86: 2022-08-12T07:30:08.031Z [INFO] server-cf81022f-4714-41c1-0fcb-c0313a53cb86: removed ourself, transitioning to follower +testLoggerAdapter verbose: server-cf81022f-4714-41c1-0fcb-c0313a53cb86: 2022-08-12T07:30:08.031Z [INFO] server-cf81022f-4714-41c1-0fcb-c0313a53cb86: entering follower state: follower="Node at cf81022f-4714-41c1-0fcb-c0313a53cb86 [Follower]" leader= + testing.go:149: server-cf81022f-4714-41c1-0fcb-c0313a53cb86: 2022-08-12T07:30:08.031Z [INFO] server-cf81022f-4714-41c1-0fcb-c0313a53cb86: entering follower state: follower="Node at cf81022f-4714-41c1-0fcb-c0313a53cb86 [Follower]" leader= +testLoggerAdapter verbose: server-cf81022f-4714-41c1-0fcb-c0313a53cb86: 2022-08-12T07:30:08.031Z [INFO] server-cf81022f-4714-41c1-0fcb-c0313a53cb86: aborting pipeline replication: peer="{Voter server-c9801220-3635-5f0a-cc85-efd24dbd57ea c9801220-3635-5f0a-cc85-efd24dbd57ea}" + testing.go:149: server-cf81022f-4714-41c1-0fcb-c0313a53cb86: 2022-08-12T07:30:08.031Z [INFO] server-cf81022f-4714-41c1-0fcb-c0313a53cb86: aborting pipeline replication: peer="{Voter server-c9801220-3635-5f0a-cc85-efd24dbd57ea c9801220-3635-5f0a-cc85-efd24dbd57ea}" +testLoggerAdapter verbose: server-cf81022f-4714-41c1-0fcb-c0313a53cb86: 2022-08-12T07:30:08.032Z [INFO] server-cf81022f-4714-41c1-0fcb-c0313a53cb86: aborting pipeline replication: peer="{Voter server-d3098eac-f146-ebec-ad6f-7fb29c171eb0 d3098eac-f146-ebec-ad6f-7fb29c171eb0}" + testing.go:149: server-cf81022f-4714-41c1-0fcb-c0313a53cb86: 2022-08-12T07:30:08.032Z [INFO] server-cf81022f-4714-41c1-0fcb-c0313a53cb86: aborting pipeline replication: peer="{Voter server-d3098eac-f146-ebec-ad6f-7fb29c171eb0 d3098eac-f146-ebec-ad6f-7fb29c171eb0}" +testLoggerAdapter verbose: server-cf81022f-4714-41c1-0fcb-c0313a53cb86: 2022-08-12T07:30:08.100Z [WARN] server-cf81022f-4714-41c1-0fcb-c0313a53cb86: not part of stable configuration, aborting election + testing.go:149: server-cf81022f-4714-41c1-0fcb-c0313a53cb86: 2022-08-12T07:30:08.100Z [WARN] server-cf81022f-4714-41c1-0fcb-c0313a53cb86: not part of stable configuration, aborting election +testLoggerAdapter verbose: server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: 2022-08-12T07:30:08.101Z [WARN] server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: heartbeat timeout reached, starting election: last-leader=cf81022f-4714-41c1-0fcb-c0313a53cb86 + testing.go:149: server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: 2022-08-12T07:30:08.101Z [WARN] server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: heartbeat timeout reached, starting election: last-leader=cf81022f-4714-41c1-0fcb-c0313a53cb86 +testLoggerAdapter verbose: server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: 2022-08-12T07:30:08.101Z [INFO] server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: entering candidate state: node="Node at d3098eac-f146-ebec-ad6f-7fb29c171eb0 [Candidate]" term=3 + testing.go:149: server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: 2022-08-12T07:30:08.101Z [INFO] server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: entering candidate state: node="Node at d3098eac-f146-ebec-ad6f-7fb29c171eb0 [Candidate]" term=3 +testLoggerAdapter verbose: server-c9801220-3635-5f0a-cc85-efd24dbd57ea: 2022-08-12T07:30:08.101Z [WARN] server-c9801220-3635-5f0a-cc85-efd24dbd57ea: rejecting vote request since we have a leader: from=d3098eac-f146-ebec-ad6f-7fb29c171eb0 leader=cf81022f-4714-41c1-0fcb-c0313a53cb86 + testing.go:149: server-c9801220-3635-5f0a-cc85-efd24dbd57ea: 2022-08-12T07:30:08.101Z [WARN] server-c9801220-3635-5f0a-cc85-efd24dbd57ea: rejecting vote request since we have a leader: from=d3098eac-f146-ebec-ad6f-7fb29c171eb0 leader=cf81022f-4714-41c1-0fcb-c0313a53cb86 +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:08.137Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:08.137Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-c9801220-3635-5f0a-cc85-efd24dbd57ea: 2022-08-12T07:30:08.168Z [WARN] server-c9801220-3635-5f0a-cc85-efd24dbd57ea: heartbeat timeout reached, starting election: last-leader=cf81022f-4714-41c1-0fcb-c0313a53cb86 + testing.go:149: server-c9801220-3635-5f0a-cc85-efd24dbd57ea: 2022-08-12T07:30:08.168Z [WARN] server-c9801220-3635-5f0a-cc85-efd24dbd57ea: heartbeat timeout reached, starting election: last-leader=cf81022f-4714-41c1-0fcb-c0313a53cb86 +testLoggerAdapter verbose: server-c9801220-3635-5f0a-cc85-efd24dbd57ea: 2022-08-12T07:30:08.168Z [INFO] server-c9801220-3635-5f0a-cc85-efd24dbd57ea: entering candidate state: node="Node at c9801220-3635-5f0a-cc85-efd24dbd57ea [Candidate]" term=3 + testing.go:149: server-c9801220-3635-5f0a-cc85-efd24dbd57ea: 2022-08-12T07:30:08.168Z [INFO] server-c9801220-3635-5f0a-cc85-efd24dbd57ea: entering candidate state: node="Node at c9801220-3635-5f0a-cc85-efd24dbd57ea [Candidate]" term=3 +testLoggerAdapter verbose: server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: 2022-08-12T07:30:08.168Z [INFO] server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: duplicate requestVote for same term: term=3 + testing.go:149: server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: 2022-08-12T07:30:08.168Z [INFO] server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: duplicate requestVote for same term: term=3 +testLoggerAdapter verbose: server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: 2022-08-12T07:30:08.197Z [WARN] server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: Election timeout reached, restarting election + testing.go:149: server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: 2022-08-12T07:30:08.197Z [WARN] server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: Election timeout reached, restarting election +testLoggerAdapter verbose: server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: 2022-08-12T07:30:08.197Z [INFO] server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: entering candidate state: node="Node at d3098eac-f146-ebec-ad6f-7fb29c171eb0 [Candidate]" term=4 + testing.go:149: server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: 2022-08-12T07:30:08.197Z [INFO] server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: entering candidate state: node="Node at d3098eac-f146-ebec-ad6f-7fb29c171eb0 [Candidate]" term=4 +testLoggerAdapter verbose: server-c9801220-3635-5f0a-cc85-efd24dbd57ea: 2022-08-12T07:30:08.198Z [INFO] server-c9801220-3635-5f0a-cc85-efd24dbd57ea: entering follower state: follower="Node at c9801220-3635-5f0a-cc85-efd24dbd57ea [Follower]" leader= + testing.go:149: server-c9801220-3635-5f0a-cc85-efd24dbd57ea: 2022-08-12T07:30:08.198Z [INFO] server-c9801220-3635-5f0a-cc85-efd24dbd57ea: entering follower state: follower="Node at c9801220-3635-5f0a-cc85-efd24dbd57ea [Follower]" leader= +testLoggerAdapter verbose: server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: 2022-08-12T07:30:08.198Z [INFO] server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: election won: tally=2 + testing.go:149: server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: 2022-08-12T07:30:08.198Z [INFO] server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: election won: tally=2 +testLoggerAdapter verbose: server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: 2022-08-12T07:30:08.198Z [INFO] server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: entering leader state: leader="Node at d3098eac-f146-ebec-ad6f-7fb29c171eb0 [Leader]" + testing.go:149: server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: 2022-08-12T07:30:08.198Z [INFO] server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: entering leader state: leader="Node at d3098eac-f146-ebec-ad6f-7fb29c171eb0 [Leader]" +testLoggerAdapter verbose: server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: 2022-08-12T07:30:08.198Z [INFO] server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: added peer, starting replication: peer=server-c9801220-3635-5f0a-cc85-efd24dbd57ea + testing.go:149: server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: 2022-08-12T07:30:08.198Z [INFO] server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: added peer, starting replication: peer=server-c9801220-3635-5f0a-cc85-efd24dbd57ea +testLoggerAdapter verbose: server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: 2022-08-12T07:30:08.198Z [INFO] server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: pipelining replication: peer="{Voter server-c9801220-3635-5f0a-cc85-efd24dbd57ea c9801220-3635-5f0a-cc85-efd24dbd57ea}" + testing.go:149: server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: 2022-08-12T07:30:08.198Z [INFO] server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: pipelining replication: peer="{Voter server-c9801220-3635-5f0a-cc85-efd24dbd57ea c9801220-3635-5f0a-cc85-efd24dbd57ea}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:08.303Z [INFO] stable state for Leader reached at 2022-08-12 07:30:08.1982991 +0000 UTC m=+9.154985561 (1 nodes), 60.84794ms from start of poll, 456.61602ms from cluster start. Timeout at 2022-08-12 07:30:08.3033991 +0000 UTC m=+9.260085581, 105.10002ms after stability + testing.go:149: cluster: 2022-08-12T07:30:08.303Z [INFO] stable state for Leader reached at 2022-08-12 07:30:08.1982991 +0000 UTC m=+9.154985561 (1 nodes), 60.84794ms from start of poll, 456.61602ms from cluster start. Timeout at 2022-08-12 07:30:08.3033991 +0000 UTC m=+9.260085581, 105.10002ms after stability +testLoggerAdapter verbose: server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: 2022-08-12T07:30:08.408Z [INFO] server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: aborting pipeline replication: peer="{Voter server-c9801220-3635-5f0a-cc85-efd24dbd57ea c9801220-3635-5f0a-cc85-efd24dbd57ea}" + testing.go:149: server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: 2022-08-12T07:30:08.408Z [INFO] server-d3098eac-f146-ebec-ad6f-7fb29c171eb0: aborting pipeline replication: peer="{Voter server-c9801220-3635-5f0a-cc85-efd24dbd57ea c9801220-3635-5f0a-cc85-efd24dbd57ea}" +--- PASS: TestRaft_RemoveLeader_NoShutdown (0.67s) === RUN TestRaft_RemoveFollower_SplitCluster -testLoggerAdapter verbose: server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8: 2021-07-10T01:05:03.935Z [INFO] server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 Address:e2dbfe11-9d56-e8dc-9b73-209a54540fb8} {Suffrage:Voter ID:server-7c957af8-88bd-0379-90bd-080eb1a1f963 Address:7c957af8-88bd-0379-90bd-080eb1a1f963} {Suffrage:Voter ID:server-16924e6a-4df3-d71d-c931-45a38d6620a1 Address:16924e6a-4df3-d71d-c931-45a38d6620a1} {Suffrage:Voter ID:server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f Address:c30fad15-bc9e-dcf4-1045-c40bf71e2b1f}]" - testing.go:149: server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8: 2021-07-10T01:05:03.935Z [INFO] server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 Address:e2dbfe11-9d56-e8dc-9b73-209a54540fb8} {Suffrage:Voter ID:server-7c957af8-88bd-0379-90bd-080eb1a1f963 Address:7c957af8-88bd-0379-90bd-080eb1a1f963} {Suffrage:Voter ID:server-16924e6a-4df3-d71d-c931-45a38d6620a1 Address:16924e6a-4df3-d71d-c931-45a38d6620a1} {Suffrage:Voter ID:server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f Address:c30fad15-bc9e-dcf4-1045-c40bf71e2b1f}]" -testLoggerAdapter verbose: server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8: 2021-07-10T01:05:03.936Z [INFO] server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8: entering follower state: follower="Node at e2dbfe11-9d56-e8dc-9b73-209a54540fb8 [Follower]" leader= - testing.go:149: server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8: 2021-07-10T01:05:03.936Z [INFO] server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8: entering follower state: follower="Node at e2dbfe11-9d56-e8dc-9b73-209a54540fb8 [Follower]" leader= -testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:03.936Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 Address:e2dbfe11-9d56-e8dc-9b73-209a54540fb8} {Suffrage:Voter ID:server-7c957af8-88bd-0379-90bd-080eb1a1f963 Address:7c957af8-88bd-0379-90bd-080eb1a1f963} {Suffrage:Voter ID:server-16924e6a-4df3-d71d-c931-45a38d6620a1 Address:16924e6a-4df3-d71d-c931-45a38d6620a1} {Suffrage:Voter ID:server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f Address:c30fad15-bc9e-dcf4-1045-c40bf71e2b1f}]" - testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:03.936Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 Address:e2dbfe11-9d56-e8dc-9b73-209a54540fb8} {Suffrage:Voter ID:server-7c957af8-88bd-0379-90bd-080eb1a1f963 Address:7c957af8-88bd-0379-90bd-080eb1a1f963} {Suffrage:Voter ID:server-16924e6a-4df3-d71d-c931-45a38d6620a1 Address:16924e6a-4df3-d71d-c931-45a38d6620a1} {Suffrage:Voter ID:server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f Address:c30fad15-bc9e-dcf4-1045-c40bf71e2b1f}]" -testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:03.936Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: entering follower state: follower="Node at 7c957af8-88bd-0379-90bd-080eb1a1f963 [Follower]" leader= - testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:03.936Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: entering follower state: follower="Node at 7c957af8-88bd-0379-90bd-080eb1a1f963 [Follower]" leader= -testLoggerAdapter verbose: server-16924e6a-4df3-d71d-c931-45a38d6620a1: 2021-07-10T01:05:03.936Z [INFO] server-16924e6a-4df3-d71d-c931-45a38d6620a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 Address:e2dbfe11-9d56-e8dc-9b73-209a54540fb8} {Suffrage:Voter ID:server-7c957af8-88bd-0379-90bd-080eb1a1f963 Address:7c957af8-88bd-0379-90bd-080eb1a1f963} {Suffrage:Voter ID:server-16924e6a-4df3-d71d-c931-45a38d6620a1 Address:16924e6a-4df3-d71d-c931-45a38d6620a1} {Suffrage:Voter ID:server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f Address:c30fad15-bc9e-dcf4-1045-c40bf71e2b1f}]" - testing.go:149: server-16924e6a-4df3-d71d-c931-45a38d6620a1: 2021-07-10T01:05:03.936Z [INFO] server-16924e6a-4df3-d71d-c931-45a38d6620a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 Address:e2dbfe11-9d56-e8dc-9b73-209a54540fb8} {Suffrage:Voter ID:server-7c957af8-88bd-0379-90bd-080eb1a1f963 Address:7c957af8-88bd-0379-90bd-080eb1a1f963} {Suffrage:Voter ID:server-16924e6a-4df3-d71d-c931-45a38d6620a1 Address:16924e6a-4df3-d71d-c931-45a38d6620a1} {Suffrage:Voter ID:server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f Address:c30fad15-bc9e-dcf4-1045-c40bf71e2b1f}]" -testLoggerAdapter verbose: server-16924e6a-4df3-d71d-c931-45a38d6620a1: 2021-07-10T01:05:03.936Z [INFO] server-16924e6a-4df3-d71d-c931-45a38d6620a1: entering follower state: follower="Node at 16924e6a-4df3-d71d-c931-45a38d6620a1 [Follower]" leader= - testing.go:149: server-16924e6a-4df3-d71d-c931-45a38d6620a1: 2021-07-10T01:05:03.936Z [INFO] server-16924e6a-4df3-d71d-c931-45a38d6620a1: entering follower state: follower="Node at 16924e6a-4df3-d71d-c931-45a38d6620a1 [Follower]" leader= -testLoggerAdapter verbose: server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f: 2021-07-10T01:05:03.936Z [INFO] server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 Address:e2dbfe11-9d56-e8dc-9b73-209a54540fb8} {Suffrage:Voter ID:server-7c957af8-88bd-0379-90bd-080eb1a1f963 Address:7c957af8-88bd-0379-90bd-080eb1a1f963} {Suffrage:Voter ID:server-16924e6a-4df3-d71d-c931-45a38d6620a1 Address:16924e6a-4df3-d71d-c931-45a38d6620a1} {Suffrage:Voter ID:server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f Address:c30fad15-bc9e-dcf4-1045-c40bf71e2b1f}]" - testing.go:149: server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f: 2021-07-10T01:05:03.936Z [INFO] server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 Address:e2dbfe11-9d56-e8dc-9b73-209a54540fb8} {Suffrage:Voter ID:server-7c957af8-88bd-0379-90bd-080eb1a1f963 Address:7c957af8-88bd-0379-90bd-080eb1a1f963} {Suffrage:Voter ID:server-16924e6a-4df3-d71d-c931-45a38d6620a1 Address:16924e6a-4df3-d71d-c931-45a38d6620a1} {Suffrage:Voter ID:server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f Address:c30fad15-bc9e-dcf4-1045-c40bf71e2b1f}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:03.936Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:03.936Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f: 2021-07-10T01:05:03.936Z [INFO] server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f: entering follower state: follower="Node at c30fad15-bc9e-dcf4-1045-c40bf71e2b1f [Follower]" leader= - testing.go:149: server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f: 2021-07-10T01:05:03.936Z [INFO] server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f: entering follower state: follower="Node at c30fad15-bc9e-dcf4-1045-c40bf71e2b1f [Follower]" leader= -testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.001Z [WARN] server-7c957af8-88bd-0379-90bd-080eb1a1f963: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.001Z [WARN] server-7c957af8-88bd-0379-90bd-080eb1a1f963: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.001Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: entering candidate state: node="Node at 7c957af8-88bd-0379-90bd-080eb1a1f963 [Candidate]" term=2 - testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.001Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: entering candidate state: node="Node at 7c957af8-88bd-0379-90bd-080eb1a1f963 [Candidate]" term=2 -testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.001Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: election won: tally=3 - testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.001Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: election won: tally=3 -testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.002Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: entering leader state: leader="Node at 7c957af8-88bd-0379-90bd-080eb1a1f963 [Leader]" - testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.002Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: entering leader state: leader="Node at 7c957af8-88bd-0379-90bd-080eb1a1f963 [Leader]" -testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.002Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: added peer, starting replication: peer=server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 - testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.002Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: added peer, starting replication: peer=server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 -testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.002Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: added peer, starting replication: peer=server-16924e6a-4df3-d71d-c931-45a38d6620a1 - testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.002Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: added peer, starting replication: peer=server-16924e6a-4df3-d71d-c931-45a38d6620a1 -testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.002Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: added peer, starting replication: peer=server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f - testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.002Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: added peer, starting replication: peer=server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f -testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.002Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: pipelining replication: peer="{Voter server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 e2dbfe11-9d56-e8dc-9b73-209a54540fb8}" - testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.002Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: pipelining replication: peer="{Voter server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 e2dbfe11-9d56-e8dc-9b73-209a54540fb8}" -testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.002Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: pipelining replication: peer="{Voter server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f c30fad15-bc9e-dcf4-1045-c40bf71e2b1f}" - testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.002Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: pipelining replication: peer="{Voter server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f c30fad15-bc9e-dcf4-1045-c40bf71e2b1f}" -testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.002Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: pipelining replication: peer="{Voter server-16924e6a-4df3-d71d-c931-45a38d6620a1 16924e6a-4df3-d71d-c931-45a38d6620a1}" - testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.002Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: pipelining replication: peer="{Voter server-16924e6a-4df3-d71d-c931-45a38d6620a1 16924e6a-4df3-d71d-c931-45a38d6620a1}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:04.107Z [INFO] stable state for Leader reached at 2021-07-10 01:05:04.002091485 +0000 UTC m=+9.390249475 (1 nodes), 65.159827ms from start of poll, 66.494101ms from cluster start. Timeout at 2021-07-10 01:05:04.107159293 +0000 UTC m=+9.495317263, 105.067788ms after stability - testing.go:149: cluster: 2021-07-10T01:05:04.107Z [INFO] stable state for Leader reached at 2021-07-10 01:05:04.002091485 +0000 UTC m=+9.390249475 (1 nodes), 65.159827ms from start of poll, 66.494101ms from cluster start. Timeout at 2021-07-10 01:05:04.107159293 +0000 UTC m=+9.495317263, 105.067788ms after stability -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:04.212Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-07-10T01:05:04.212Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:04.317Z [INFO] stable state for Follower reached at 2021-07-10 01:05:04.212622505 +0000 UTC m=+9.600780475 (3 nodes), 2.58µs from start of poll, 277.025101ms from cluster start. Timeout at 2021-07-10 01:05:04.317686472 +0000 UTC m=+9.705844442, 105.063967ms after stability - testing.go:149: cluster: 2021-07-10T01:05:04.317Z [INFO] stable state for Follower reached at 2021-07-10 01:05:04.212622505 +0000 UTC m=+9.600780475 (3 nodes), 2.58µs from start of poll, 277.025101ms from cluster start. Timeout at 2021-07-10 01:05:04.317686472 +0000 UTC m=+9.705844442, 105.063967ms after stability -testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.317Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: updating configuration: command=RemoveServer server-id=server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f server-addr= servers="[{Suffrage:Voter ID:server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 Address:e2dbfe11-9d56-e8dc-9b73-209a54540fb8} {Suffrage:Voter ID:server-7c957af8-88bd-0379-90bd-080eb1a1f963 Address:7c957af8-88bd-0379-90bd-080eb1a1f963} {Suffrage:Voter ID:server-16924e6a-4df3-d71d-c931-45a38d6620a1 Address:16924e6a-4df3-d71d-c931-45a38d6620a1}]" - testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.317Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: updating configuration: command=RemoveServer server-id=server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f server-addr= servers="[{Suffrage:Voter ID:server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 Address:e2dbfe11-9d56-e8dc-9b73-209a54540fb8} {Suffrage:Voter ID:server-7c957af8-88bd-0379-90bd-080eb1a1f963 Address:7c957af8-88bd-0379-90bd-080eb1a1f963} {Suffrage:Voter ID:server-16924e6a-4df3-d71d-c931-45a38d6620a1 Address:16924e6a-4df3-d71d-c931-45a38d6620a1}]" -testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.318Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: removed peer, stopping replication: peer=server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f last-index=3 - testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.318Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: removed peer, stopping replication: peer=server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f last-index=3 -testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.318Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to pipeline appendEntries: peer="{Voter server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 e2dbfe11-9d56-e8dc-9b73-209a54540fb8}" error="append pipeline closed" - testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.318Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to pipeline appendEntries: peer="{Voter server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 e2dbfe11-9d56-e8dc-9b73-209a54540fb8}" error="append pipeline closed" -testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.318Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: aborting pipeline replication: peer="{Voter server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 e2dbfe11-9d56-e8dc-9b73-209a54540fb8}" - testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.318Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: aborting pipeline replication: peer="{Voter server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 e2dbfe11-9d56-e8dc-9b73-209a54540fb8}" -testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.318Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to pipeline appendEntries: peer="{Voter server-16924e6a-4df3-d71d-c931-45a38d6620a1 16924e6a-4df3-d71d-c931-45a38d6620a1}" error="append pipeline closed" - testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.318Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to pipeline appendEntries: peer="{Voter server-16924e6a-4df3-d71d-c931-45a38d6620a1 16924e6a-4df3-d71d-c931-45a38d6620a1}" error="append pipeline closed" -testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.318Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: aborting pipeline replication: peer="{Voter server-16924e6a-4df3-d71d-c931-45a38d6620a1 16924e6a-4df3-d71d-c931-45a38d6620a1}" - testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.318Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: aborting pipeline replication: peer="{Voter server-16924e6a-4df3-d71d-c931-45a38d6620a1 16924e6a-4df3-d71d-c931-45a38d6620a1}" -testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.318Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: aborting pipeline replication: peer="{Voter server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f c30fad15-bc9e-dcf4-1045-c40bf71e2b1f}" - testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.318Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: aborting pipeline replication: peer="{Voter server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f c30fad15-bc9e-dcf4-1045-c40bf71e2b1f}" -testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.321Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to heartbeat to: peer=e2dbfe11-9d56-e8dc-9b73-209a54540fb8 error="failed to connect to peer: e2dbfe11-9d56-e8dc-9b73-209a54540fb8" - testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.321Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to heartbeat to: peer=e2dbfe11-9d56-e8dc-9b73-209a54540fb8 error="failed to connect to peer: e2dbfe11-9d56-e8dc-9b73-209a54540fb8" -testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.327Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to appendEntries to: peer="{Voter server-16924e6a-4df3-d71d-c931-45a38d6620a1 16924e6a-4df3-d71d-c931-45a38d6620a1}" error="failed to connect to peer: 16924e6a-4df3-d71d-c931-45a38d6620a1" - testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.327Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to appendEntries to: peer="{Voter server-16924e6a-4df3-d71d-c931-45a38d6620a1 16924e6a-4df3-d71d-c931-45a38d6620a1}" error="failed to connect to peer: 16924e6a-4df3-d71d-c931-45a38d6620a1" -testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.327Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to appendEntries to: peer="{Voter server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 e2dbfe11-9d56-e8dc-9b73-209a54540fb8}" error="failed to connect to peer: e2dbfe11-9d56-e8dc-9b73-209a54540fb8" - testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.327Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to appendEntries to: peer="{Voter server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 e2dbfe11-9d56-e8dc-9b73-209a54540fb8}" error="failed to connect to peer: e2dbfe11-9d56-e8dc-9b73-209a54540fb8" -testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.327Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to heartbeat to: peer=16924e6a-4df3-d71d-c931-45a38d6620a1 error="failed to connect to peer: 16924e6a-4df3-d71d-c931-45a38d6620a1" - testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.327Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to heartbeat to: peer=16924e6a-4df3-d71d-c931-45a38d6620a1 error="failed to connect to peer: 16924e6a-4df3-d71d-c931-45a38d6620a1" -testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.338Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to heartbeat to: peer=e2dbfe11-9d56-e8dc-9b73-209a54540fb8 error="failed to connect to peer: e2dbfe11-9d56-e8dc-9b73-209a54540fb8" - testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.338Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to heartbeat to: peer=e2dbfe11-9d56-e8dc-9b73-209a54540fb8 error="failed to connect to peer: e2dbfe11-9d56-e8dc-9b73-209a54540fb8" -testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.345Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to appendEntries to: peer="{Voter server-16924e6a-4df3-d71d-c931-45a38d6620a1 16924e6a-4df3-d71d-c931-45a38d6620a1}" error="failed to connect to peer: 16924e6a-4df3-d71d-c931-45a38d6620a1" - testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.345Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to appendEntries to: peer="{Voter server-16924e6a-4df3-d71d-c931-45a38d6620a1 16924e6a-4df3-d71d-c931-45a38d6620a1}" error="failed to connect to peer: 16924e6a-4df3-d71d-c931-45a38d6620a1" -testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.346Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to appendEntries to: peer="{Voter server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 e2dbfe11-9d56-e8dc-9b73-209a54540fb8}" error="failed to connect to peer: e2dbfe11-9d56-e8dc-9b73-209a54540fb8" - testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.346Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to appendEntries to: peer="{Voter server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 e2dbfe11-9d56-e8dc-9b73-209a54540fb8}" error="failed to connect to peer: e2dbfe11-9d56-e8dc-9b73-209a54540fb8" -testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.347Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to heartbeat to: peer=16924e6a-4df3-d71d-c931-45a38d6620a1 error="failed to connect to peer: 16924e6a-4df3-d71d-c931-45a38d6620a1" - testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.347Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to heartbeat to: peer=16924e6a-4df3-d71d-c931-45a38d6620a1 error="failed to connect to peer: 16924e6a-4df3-d71d-c931-45a38d6620a1" -testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.355Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to heartbeat to: peer=e2dbfe11-9d56-e8dc-9b73-209a54540fb8 error="failed to connect to peer: e2dbfe11-9d56-e8dc-9b73-209a54540fb8" - testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.355Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to heartbeat to: peer=e2dbfe11-9d56-e8dc-9b73-209a54540fb8 error="failed to connect to peer: e2dbfe11-9d56-e8dc-9b73-209a54540fb8" -testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.361Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to appendEntries to: peer="{Voter server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 e2dbfe11-9d56-e8dc-9b73-209a54540fb8}" error="failed to connect to peer: e2dbfe11-9d56-e8dc-9b73-209a54540fb8" - testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.361Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to appendEntries to: peer="{Voter server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 e2dbfe11-9d56-e8dc-9b73-209a54540fb8}" error="failed to connect to peer: e2dbfe11-9d56-e8dc-9b73-209a54540fb8" -testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.364Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to appendEntries to: peer="{Voter server-16924e6a-4df3-d71d-c931-45a38d6620a1 16924e6a-4df3-d71d-c931-45a38d6620a1}" error="failed to connect to peer: 16924e6a-4df3-d71d-c931-45a38d6620a1" - testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.364Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to appendEntries to: peer="{Voter server-16924e6a-4df3-d71d-c931-45a38d6620a1 16924e6a-4df3-d71d-c931-45a38d6620a1}" error="failed to connect to peer: 16924e6a-4df3-d71d-c931-45a38d6620a1" -testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.365Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to heartbeat to: peer=16924e6a-4df3-d71d-c931-45a38d6620a1 error="failed to connect to peer: 16924e6a-4df3-d71d-c931-45a38d6620a1" - testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.365Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to heartbeat to: peer=16924e6a-4df3-d71d-c931-45a38d6620a1 error="failed to connect to peer: 16924e6a-4df3-d71d-c931-45a38d6620a1" -testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.366Z [WARN] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to contact: server-id=server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 time=50.073528ms - testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.366Z [WARN] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to contact: server-id=server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 time=50.073528ms -testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.376Z [WARN] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to contact: server-id=server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 time=60.268396ms - testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.376Z [WARN] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to contact: server-id=server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 time=60.268396ms -testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.376Z [WARN] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to contact: server-id=server-16924e6a-4df3-d71d-c931-45a38d6620a1 time=59.294765ms - testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.376Z [WARN] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to contact: server-id=server-16924e6a-4df3-d71d-c931-45a38d6620a1 time=59.294765ms -testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.376Z [WARN] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to contact quorum of nodes, stepping down - testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.376Z [WARN] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to contact quorum of nodes, stepping down -testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.376Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: entering follower state: follower="Node at 7c957af8-88bd-0379-90bd-080eb1a1f963 [Follower]" leader= - testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.376Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: entering follower state: follower="Node at 7c957af8-88bd-0379-90bd-080eb1a1f963 [Follower]" leader= -testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.376Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to appendEntries to: peer="{Voter server-16924e6a-4df3-d71d-c931-45a38d6620a1 16924e6a-4df3-d71d-c931-45a38d6620a1}" error="failed to connect to peer: 16924e6a-4df3-d71d-c931-45a38d6620a1" - testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.376Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to appendEntries to: peer="{Voter server-16924e6a-4df3-d71d-c931-45a38d6620a1 16924e6a-4df3-d71d-c931-45a38d6620a1}" error="failed to connect to peer: 16924e6a-4df3-d71d-c931-45a38d6620a1" -testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.376Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to appendEntries to: peer="{Voter server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 e2dbfe11-9d56-e8dc-9b73-209a54540fb8}" error="failed to connect to peer: e2dbfe11-9d56-e8dc-9b73-209a54540fb8" - testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.376Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to appendEntries to: peer="{Voter server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 e2dbfe11-9d56-e8dc-9b73-209a54540fb8}" error="failed to connect to peer: e2dbfe11-9d56-e8dc-9b73-209a54540fb8" ---- PASS: TestRaft_RemoveFollower_SplitCluster (0.44s) +testLoggerAdapter verbose: server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c: 2022-08-12T07:30:08.410Z [INFO] server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c Address:7e8240ba-d73b-96d5-971c-1548ba4dbd9c} {Suffrage:Voter ID:server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c Address:6f8ae9f0-a5ec-7240-322e-703d41a0af5c} {Suffrage:Voter ID:server-b862bd96-bf5f-ba9f-3d00-0b2144d70880 Address:b862bd96-bf5f-ba9f-3d00-0b2144d70880} {Suffrage:Voter ID:server-aca55d59-e8fa-2421-5ea2-5741a19cf017 Address:aca55d59-e8fa-2421-5ea2-5741a19cf017}]" + testing.go:149: server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c: 2022-08-12T07:30:08.410Z [INFO] server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c Address:7e8240ba-d73b-96d5-971c-1548ba4dbd9c} {Suffrage:Voter ID:server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c Address:6f8ae9f0-a5ec-7240-322e-703d41a0af5c} {Suffrage:Voter ID:server-b862bd96-bf5f-ba9f-3d00-0b2144d70880 Address:b862bd96-bf5f-ba9f-3d00-0b2144d70880} {Suffrage:Voter ID:server-aca55d59-e8fa-2421-5ea2-5741a19cf017 Address:aca55d59-e8fa-2421-5ea2-5741a19cf017}]" +testLoggerAdapter verbose: server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c: 2022-08-12T07:30:08.410Z [INFO] server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c: entering follower state: follower="Node at 7e8240ba-d73b-96d5-971c-1548ba4dbd9c [Follower]" leader= + testing.go:149: server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c: 2022-08-12T07:30:08.410Z [INFO] server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c: entering follower state: follower="Node at 7e8240ba-d73b-96d5-971c-1548ba4dbd9c [Follower]" leader= +testLoggerAdapter verbose: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.410Z [INFO] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c Address:7e8240ba-d73b-96d5-971c-1548ba4dbd9c} {Suffrage:Voter ID:server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c Address:6f8ae9f0-a5ec-7240-322e-703d41a0af5c} {Suffrage:Voter ID:server-b862bd96-bf5f-ba9f-3d00-0b2144d70880 Address:b862bd96-bf5f-ba9f-3d00-0b2144d70880} {Suffrage:Voter ID:server-aca55d59-e8fa-2421-5ea2-5741a19cf017 Address:aca55d59-e8fa-2421-5ea2-5741a19cf017}]" + testing.go:149: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.410Z [INFO] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c Address:7e8240ba-d73b-96d5-971c-1548ba4dbd9c} {Suffrage:Voter ID:server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c Address:6f8ae9f0-a5ec-7240-322e-703d41a0af5c} {Suffrage:Voter ID:server-b862bd96-bf5f-ba9f-3d00-0b2144d70880 Address:b862bd96-bf5f-ba9f-3d00-0b2144d70880} {Suffrage:Voter ID:server-aca55d59-e8fa-2421-5ea2-5741a19cf017 Address:aca55d59-e8fa-2421-5ea2-5741a19cf017}]" +testLoggerAdapter verbose: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.410Z [INFO] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: entering follower state: follower="Node at 6f8ae9f0-a5ec-7240-322e-703d41a0af5c [Follower]" leader= + testing.go:149: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.410Z [INFO] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: entering follower state: follower="Node at 6f8ae9f0-a5ec-7240-322e-703d41a0af5c [Follower]" leader= +testLoggerAdapter verbose: server-b862bd96-bf5f-ba9f-3d00-0b2144d70880: 2022-08-12T07:30:08.410Z [INFO] server-b862bd96-bf5f-ba9f-3d00-0b2144d70880: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c Address:7e8240ba-d73b-96d5-971c-1548ba4dbd9c} {Suffrage:Voter ID:server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c Address:6f8ae9f0-a5ec-7240-322e-703d41a0af5c} {Suffrage:Voter ID:server-b862bd96-bf5f-ba9f-3d00-0b2144d70880 Address:b862bd96-bf5f-ba9f-3d00-0b2144d70880} {Suffrage:Voter ID:server-aca55d59-e8fa-2421-5ea2-5741a19cf017 Address:aca55d59-e8fa-2421-5ea2-5741a19cf017}]" + testing.go:149: server-b862bd96-bf5f-ba9f-3d00-0b2144d70880: 2022-08-12T07:30:08.410Z [INFO] server-b862bd96-bf5f-ba9f-3d00-0b2144d70880: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c Address:7e8240ba-d73b-96d5-971c-1548ba4dbd9c} {Suffrage:Voter ID:server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c Address:6f8ae9f0-a5ec-7240-322e-703d41a0af5c} {Suffrage:Voter ID:server-b862bd96-bf5f-ba9f-3d00-0b2144d70880 Address:b862bd96-bf5f-ba9f-3d00-0b2144d70880} {Suffrage:Voter ID:server-aca55d59-e8fa-2421-5ea2-5741a19cf017 Address:aca55d59-e8fa-2421-5ea2-5741a19cf017}]" +testLoggerAdapter verbose: server-b862bd96-bf5f-ba9f-3d00-0b2144d70880: 2022-08-12T07:30:08.411Z [INFO] server-b862bd96-bf5f-ba9f-3d00-0b2144d70880: entering follower state: follower="Node at b862bd96-bf5f-ba9f-3d00-0b2144d70880 [Follower]" leader= + testing.go:149: server-b862bd96-bf5f-ba9f-3d00-0b2144d70880: 2022-08-12T07:30:08.411Z [INFO] server-b862bd96-bf5f-ba9f-3d00-0b2144d70880: entering follower state: follower="Node at b862bd96-bf5f-ba9f-3d00-0b2144d70880 [Follower]" leader= +testLoggerAdapter verbose: server-aca55d59-e8fa-2421-5ea2-5741a19cf017: 2022-08-12T07:30:08.411Z [INFO] server-aca55d59-e8fa-2421-5ea2-5741a19cf017: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c Address:7e8240ba-d73b-96d5-971c-1548ba4dbd9c} {Suffrage:Voter ID:server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c Address:6f8ae9f0-a5ec-7240-322e-703d41a0af5c} {Suffrage:Voter ID:server-b862bd96-bf5f-ba9f-3d00-0b2144d70880 Address:b862bd96-bf5f-ba9f-3d00-0b2144d70880} {Suffrage:Voter ID:server-aca55d59-e8fa-2421-5ea2-5741a19cf017 Address:aca55d59-e8fa-2421-5ea2-5741a19cf017}]" + testing.go:149: server-aca55d59-e8fa-2421-5ea2-5741a19cf017: 2022-08-12T07:30:08.411Z [INFO] server-aca55d59-e8fa-2421-5ea2-5741a19cf017: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c Address:7e8240ba-d73b-96d5-971c-1548ba4dbd9c} {Suffrage:Voter ID:server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c Address:6f8ae9f0-a5ec-7240-322e-703d41a0af5c} {Suffrage:Voter ID:server-b862bd96-bf5f-ba9f-3d00-0b2144d70880 Address:b862bd96-bf5f-ba9f-3d00-0b2144d70880} {Suffrage:Voter ID:server-aca55d59-e8fa-2421-5ea2-5741a19cf017 Address:aca55d59-e8fa-2421-5ea2-5741a19cf017}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:08.411Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-aca55d59-e8fa-2421-5ea2-5741a19cf017: 2022-08-12T07:30:08.411Z [INFO] server-aca55d59-e8fa-2421-5ea2-5741a19cf017: entering follower state: follower="Node at aca55d59-e8fa-2421-5ea2-5741a19cf017 [Follower]" leader= + testing.go:149: cluster: 2022-08-12T07:30:08.411Z [INFO] starting stability test: raft-state=Leader + testing.go:149: server-aca55d59-e8fa-2421-5ea2-5741a19cf017: 2022-08-12T07:30:08.411Z [INFO] server-aca55d59-e8fa-2421-5ea2-5741a19cf017: entering follower state: follower="Node at aca55d59-e8fa-2421-5ea2-5741a19cf017 [Follower]" leader= +testLoggerAdapter verbose: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.467Z [WARN] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.467Z [WARN] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.467Z [INFO] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: entering candidate state: node="Node at 6f8ae9f0-a5ec-7240-322e-703d41a0af5c [Candidate]" term=2 + testing.go:149: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.467Z [INFO] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: entering candidate state: node="Node at 6f8ae9f0-a5ec-7240-322e-703d41a0af5c [Candidate]" term=2 +testLoggerAdapter verbose: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.467Z [INFO] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: election won: tally=3 + testing.go:149: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.467Z [INFO] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: election won: tally=3 +testLoggerAdapter verbose: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.467Z [INFO] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: entering leader state: leader="Node at 6f8ae9f0-a5ec-7240-322e-703d41a0af5c [Leader]" + testing.go:149: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.467Z [INFO] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: entering leader state: leader="Node at 6f8ae9f0-a5ec-7240-322e-703d41a0af5c [Leader]" +testLoggerAdapter verbose: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.467Z [INFO] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: added peer, starting replication: peer=server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c + testing.go:149: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.467Z [INFO] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: added peer, starting replication: peer=server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c +testLoggerAdapter verbose: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.467Z [INFO] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: added peer, starting replication: peer=server-b862bd96-bf5f-ba9f-3d00-0b2144d70880 + testing.go:149: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.467Z [INFO] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: added peer, starting replication: peer=server-b862bd96-bf5f-ba9f-3d00-0b2144d70880 +testLoggerAdapter verbose: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.467Z [INFO] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: added peer, starting replication: peer=server-aca55d59-e8fa-2421-5ea2-5741a19cf017 + testing.go:149: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.467Z [INFO] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: added peer, starting replication: peer=server-aca55d59-e8fa-2421-5ea2-5741a19cf017 +testLoggerAdapter verbose: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.467Z [INFO] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: pipelining replication: peer="{Voter server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c 7e8240ba-d73b-96d5-971c-1548ba4dbd9c}" + testing.go:149: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.467Z [INFO] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: pipelining replication: peer="{Voter server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c 7e8240ba-d73b-96d5-971c-1548ba4dbd9c}" +testLoggerAdapter verbose: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.468Z [INFO] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: pipelining replication: peer="{Voter server-b862bd96-bf5f-ba9f-3d00-0b2144d70880 b862bd96-bf5f-ba9f-3d00-0b2144d70880}" + testing.go:149: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.468Z [INFO] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: pipelining replication: peer="{Voter server-b862bd96-bf5f-ba9f-3d00-0b2144d70880 b862bd96-bf5f-ba9f-3d00-0b2144d70880}" +testLoggerAdapter verbose: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.468Z [INFO] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: pipelining replication: peer="{Voter server-aca55d59-e8fa-2421-5ea2-5741a19cf017 aca55d59-e8fa-2421-5ea2-5741a19cf017}" + testing.go:149: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.468Z [INFO] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: pipelining replication: peer="{Voter server-aca55d59-e8fa-2421-5ea2-5741a19cf017 aca55d59-e8fa-2421-5ea2-5741a19cf017}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:08.572Z [INFO] stable state for Leader reached at 2022-08-12 07:30:08.46770222 +0000 UTC m=+9.424388701 (1 nodes), 56.2622ms from start of poll, 57.57086ms from cluster start. Timeout at 2022-08-12 07:30:08.5727587 +0000 UTC m=+9.529445181, 105.05648ms after stability + testing.go:149: cluster: 2022-08-12T07:30:08.572Z [INFO] stable state for Leader reached at 2022-08-12 07:30:08.46770222 +0000 UTC m=+9.424388701 (1 nodes), 56.2622ms from start of poll, 57.57086ms from cluster start. Timeout at 2022-08-12 07:30:08.5727587 +0000 UTC m=+9.529445181, 105.05648ms after stability +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:08.678Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2022-08-12T07:30:08.678Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:08.783Z [INFO] stable state for Follower reached at 2022-08-12 07:30:08.67820084 +0000 UTC m=+9.634887301 (3 nodes), 2.98µs from start of poll, 268.06946ms from cluster start. Timeout at 2022-08-12 07:30:08.7832656 +0000 UTC m=+9.739952081, 105.06478ms after stability + testing.go:149: cluster: 2022-08-12T07:30:08.783Z [INFO] stable state for Follower reached at 2022-08-12 07:30:08.67820084 +0000 UTC m=+9.634887301 (3 nodes), 2.98µs from start of poll, 268.06946ms from cluster start. Timeout at 2022-08-12 07:30:08.7832656 +0000 UTC m=+9.739952081, 105.06478ms after stability +testLoggerAdapter verbose: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.783Z [INFO] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: updating configuration: command=RemoveServer server-id=server-aca55d59-e8fa-2421-5ea2-5741a19cf017 server-addr= servers="[{Suffrage:Voter ID:server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c Address:7e8240ba-d73b-96d5-971c-1548ba4dbd9c} {Suffrage:Voter ID:server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c Address:6f8ae9f0-a5ec-7240-322e-703d41a0af5c} {Suffrage:Voter ID:server-b862bd96-bf5f-ba9f-3d00-0b2144d70880 Address:b862bd96-bf5f-ba9f-3d00-0b2144d70880}]" + testing.go:149: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.783Z [INFO] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: updating configuration: command=RemoveServer server-id=server-aca55d59-e8fa-2421-5ea2-5741a19cf017 server-addr= servers="[{Suffrage:Voter ID:server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c Address:7e8240ba-d73b-96d5-971c-1548ba4dbd9c} {Suffrage:Voter ID:server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c Address:6f8ae9f0-a5ec-7240-322e-703d41a0af5c} {Suffrage:Voter ID:server-b862bd96-bf5f-ba9f-3d00-0b2144d70880 Address:b862bd96-bf5f-ba9f-3d00-0b2144d70880}]" +testLoggerAdapter verbose: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.783Z [INFO] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: removed peer, stopping replication: peer=server-aca55d59-e8fa-2421-5ea2-5741a19cf017 last-index=3 + testing.go:149: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.783Z [INFO] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: removed peer, stopping replication: peer=server-aca55d59-e8fa-2421-5ea2-5741a19cf017 last-index=3 +testLoggerAdapter verbose: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.783Z [ERROR] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to pipeline appendEntries: peer="{Voter server-b862bd96-bf5f-ba9f-3d00-0b2144d70880 b862bd96-bf5f-ba9f-3d00-0b2144d70880}" error="append pipeline closed" + testing.go:149: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.783Z [ERROR] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to pipeline appendEntries: peer="{Voter server-b862bd96-bf5f-ba9f-3d00-0b2144d70880 b862bd96-bf5f-ba9f-3d00-0b2144d70880}" error="append pipeline closed" +testLoggerAdapter verbose: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.783Z [ERROR] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to pipeline appendEntries: peer="{Voter server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c 7e8240ba-d73b-96d5-971c-1548ba4dbd9c}" error="append pipeline closed" + testing.go:149: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.783Z [ERROR] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to pipeline appendEntries: peer="{Voter server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c 7e8240ba-d73b-96d5-971c-1548ba4dbd9c}" error="append pipeline closed" +testLoggerAdapter verbose: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.783Z [INFO] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: aborting pipeline replication: peer="{Voter server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c 7e8240ba-d73b-96d5-971c-1548ba4dbd9c}" + testing.go:149: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.783Z [INFO] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: aborting pipeline replication: peer="{Voter server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c 7e8240ba-d73b-96d5-971c-1548ba4dbd9c}" +testLoggerAdapter verbose: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.783Z [INFO] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: aborting pipeline replication: peer="{Voter server-aca55d59-e8fa-2421-5ea2-5741a19cf017 aca55d59-e8fa-2421-5ea2-5741a19cf017}" + testing.go:149: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.783Z [INFO] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: aborting pipeline replication: peer="{Voter server-aca55d59-e8fa-2421-5ea2-5741a19cf017 aca55d59-e8fa-2421-5ea2-5741a19cf017}" +testLoggerAdapter verbose: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.783Z [INFO] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: aborting pipeline replication: peer="{Voter server-b862bd96-bf5f-ba9f-3d00-0b2144d70880 b862bd96-bf5f-ba9f-3d00-0b2144d70880}" + testing.go:149: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.783Z [INFO] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: aborting pipeline replication: peer="{Voter server-b862bd96-bf5f-ba9f-3d00-0b2144d70880 b862bd96-bf5f-ba9f-3d00-0b2144d70880}" +testLoggerAdapter verbose: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.786Z [ERROR] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to heartbeat to: peer=7e8240ba-d73b-96d5-971c-1548ba4dbd9c error="failed to connect to peer: 7e8240ba-d73b-96d5-971c-1548ba4dbd9c" + testing.go:149: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.786Z [ERROR] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to heartbeat to: peer=7e8240ba-d73b-96d5-971c-1548ba4dbd9c error="failed to connect to peer: 7e8240ba-d73b-96d5-971c-1548ba4dbd9c" +testLoggerAdapter verbose: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.788Z [ERROR] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to heartbeat to: peer=b862bd96-bf5f-ba9f-3d00-0b2144d70880 error="failed to connect to peer: b862bd96-bf5f-ba9f-3d00-0b2144d70880" + testing.go:149: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.788Z [ERROR] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to heartbeat to: peer=b862bd96-bf5f-ba9f-3d00-0b2144d70880 error="failed to connect to peer: b862bd96-bf5f-ba9f-3d00-0b2144d70880" +testLoggerAdapter verbose: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.789Z [ERROR] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to appendEntries to: peer="{Voter server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c 7e8240ba-d73b-96d5-971c-1548ba4dbd9c}" error="failed to connect to peer: 7e8240ba-d73b-96d5-971c-1548ba4dbd9c" + testing.go:149: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.789Z [ERROR] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to appendEntries to: peer="{Voter server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c 7e8240ba-d73b-96d5-971c-1548ba4dbd9c}" error="failed to connect to peer: 7e8240ba-d73b-96d5-971c-1548ba4dbd9c" +testLoggerAdapter verbose: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.791Z [ERROR] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to appendEntries to: peer="{Voter server-b862bd96-bf5f-ba9f-3d00-0b2144d70880 b862bd96-bf5f-ba9f-3d00-0b2144d70880}" error="failed to connect to peer: b862bd96-bf5f-ba9f-3d00-0b2144d70880" + testing.go:149: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.791Z [ERROR] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to appendEntries to: peer="{Voter server-b862bd96-bf5f-ba9f-3d00-0b2144d70880 b862bd96-bf5f-ba9f-3d00-0b2144d70880}" error="failed to connect to peer: b862bd96-bf5f-ba9f-3d00-0b2144d70880" +testLoggerAdapter verbose: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.803Z [ERROR] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to heartbeat to: peer=7e8240ba-d73b-96d5-971c-1548ba4dbd9c error="failed to connect to peer: 7e8240ba-d73b-96d5-971c-1548ba4dbd9c" + testing.go:149: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.803Z [ERROR] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to heartbeat to: peer=7e8240ba-d73b-96d5-971c-1548ba4dbd9c error="failed to connect to peer: 7e8240ba-d73b-96d5-971c-1548ba4dbd9c" +testLoggerAdapter verbose: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.807Z [ERROR] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to appendEntries to: peer="{Voter server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c 7e8240ba-d73b-96d5-971c-1548ba4dbd9c}" error="failed to connect to peer: 7e8240ba-d73b-96d5-971c-1548ba4dbd9c" + testing.go:149: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.807Z [ERROR] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to appendEntries to: peer="{Voter server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c 7e8240ba-d73b-96d5-971c-1548ba4dbd9c}" error="failed to connect to peer: 7e8240ba-d73b-96d5-971c-1548ba4dbd9c" +testLoggerAdapter verbose: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.808Z [ERROR] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to heartbeat to: peer=b862bd96-bf5f-ba9f-3d00-0b2144d70880 error="failed to connect to peer: b862bd96-bf5f-ba9f-3d00-0b2144d70880" + testing.go:149: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.808Z [ERROR] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to heartbeat to: peer=b862bd96-bf5f-ba9f-3d00-0b2144d70880 error="failed to connect to peer: b862bd96-bf5f-ba9f-3d00-0b2144d70880" +testLoggerAdapter verbose: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.811Z [ERROR] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to appendEntries to: peer="{Voter server-b862bd96-bf5f-ba9f-3d00-0b2144d70880 b862bd96-bf5f-ba9f-3d00-0b2144d70880}" error="failed to connect to peer: b862bd96-bf5f-ba9f-3d00-0b2144d70880" + testing.go:149: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.811Z [ERROR] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to appendEntries to: peer="{Voter server-b862bd96-bf5f-ba9f-3d00-0b2144d70880 b862bd96-bf5f-ba9f-3d00-0b2144d70880}" error="failed to connect to peer: b862bd96-bf5f-ba9f-3d00-0b2144d70880" +testLoggerAdapter verbose: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.820Z [ERROR] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to heartbeat to: peer=7e8240ba-d73b-96d5-971c-1548ba4dbd9c error="failed to connect to peer: 7e8240ba-d73b-96d5-971c-1548ba4dbd9c" + testing.go:149: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.820Z [ERROR] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to heartbeat to: peer=7e8240ba-d73b-96d5-971c-1548ba4dbd9c error="failed to connect to peer: 7e8240ba-d73b-96d5-971c-1548ba4dbd9c" +testLoggerAdapter verbose: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.823Z [ERROR] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to appendEntries to: peer="{Voter server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c 7e8240ba-d73b-96d5-971c-1548ba4dbd9c}" error="failed to connect to peer: 7e8240ba-d73b-96d5-971c-1548ba4dbd9c" + testing.go:149: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.823Z [ERROR] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to appendEntries to: peer="{Voter server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c 7e8240ba-d73b-96d5-971c-1548ba4dbd9c}" error="failed to connect to peer: 7e8240ba-d73b-96d5-971c-1548ba4dbd9c" +testLoggerAdapter verbose: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.828Z [ERROR] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to heartbeat to: peer=b862bd96-bf5f-ba9f-3d00-0b2144d70880 error="failed to connect to peer: b862bd96-bf5f-ba9f-3d00-0b2144d70880" + testing.go:149: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.828Z [ERROR] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to heartbeat to: peer=b862bd96-bf5f-ba9f-3d00-0b2144d70880 error="failed to connect to peer: b862bd96-bf5f-ba9f-3d00-0b2144d70880" +testLoggerAdapter verbose: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.829Z [ERROR] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to appendEntries to: peer="{Voter server-b862bd96-bf5f-ba9f-3d00-0b2144d70880 b862bd96-bf5f-ba9f-3d00-0b2144d70880}" error="failed to connect to peer: b862bd96-bf5f-ba9f-3d00-0b2144d70880" + testing.go:149: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.829Z [ERROR] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to appendEntries to: peer="{Voter server-b862bd96-bf5f-ba9f-3d00-0b2144d70880 b862bd96-bf5f-ba9f-3d00-0b2144d70880}" error="failed to connect to peer: b862bd96-bf5f-ba9f-3d00-0b2144d70880" +testLoggerAdapter verbose: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.829Z [WARN] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to contact: server-id=server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c time=50.06962ms + testing.go:149: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.829Z [WARN] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to contact: server-id=server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c time=50.06962ms +testLoggerAdapter verbose: server-aca55d59-e8fa-2421-5ea2-5741a19cf017: 2022-08-12T07:30:08.837Z [WARN] server-aca55d59-e8fa-2421-5ea2-5741a19cf017: heartbeat timeout reached, starting election: last-leader=6f8ae9f0-a5ec-7240-322e-703d41a0af5c + testing.go:149: server-aca55d59-e8fa-2421-5ea2-5741a19cf017: 2022-08-12T07:30:08.837Z [WARN] server-aca55d59-e8fa-2421-5ea2-5741a19cf017: heartbeat timeout reached, starting election: last-leader=6f8ae9f0-a5ec-7240-322e-703d41a0af5c +testLoggerAdapter verbose: server-aca55d59-e8fa-2421-5ea2-5741a19cf017: 2022-08-12T07:30:08.837Z [INFO] server-aca55d59-e8fa-2421-5ea2-5741a19cf017: entering candidate state: node="Node at aca55d59-e8fa-2421-5ea2-5741a19cf017 [Candidate]" term=3 + testing.go:149: server-aca55d59-e8fa-2421-5ea2-5741a19cf017: 2022-08-12T07:30:08.837Z [INFO] server-aca55d59-e8fa-2421-5ea2-5741a19cf017: entering candidate state: node="Node at aca55d59-e8fa-2421-5ea2-5741a19cf017 [Candidate]" term=3 +testLoggerAdapter verbose: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.837Z [WARN] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: rejecting vote request since we have a leader: from=aca55d59-e8fa-2421-5ea2-5741a19cf017 leader=6f8ae9f0-a5ec-7240-322e-703d41a0af5c + testing.go:149: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.837Z [WARN] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: rejecting vote request since we have a leader: from=aca55d59-e8fa-2421-5ea2-5741a19cf017 leader=6f8ae9f0-a5ec-7240-322e-703d41a0af5c +testLoggerAdapter verbose: server-aca55d59-e8fa-2421-5ea2-5741a19cf017: 2022-08-12T07:30:08.837Z [ERROR] server-aca55d59-e8fa-2421-5ea2-5741a19cf017: failed to make requestVote RPC: target="{Voter server-b862bd96-bf5f-ba9f-3d00-0b2144d70880 b862bd96-bf5f-ba9f-3d00-0b2144d70880}" error="failed to connect to peer: b862bd96-bf5f-ba9f-3d00-0b2144d70880" + testing.go:149: server-aca55d59-e8fa-2421-5ea2-5741a19cf017: 2022-08-12T07:30:08.837Z [ERROR] server-aca55d59-e8fa-2421-5ea2-5741a19cf017: failed to make requestVote RPC: target="{Voter server-b862bd96-bf5f-ba9f-3d00-0b2144d70880 b862bd96-bf5f-ba9f-3d00-0b2144d70880}" error="failed to connect to peer: b862bd96-bf5f-ba9f-3d00-0b2144d70880" +testLoggerAdapter verbose: server-aca55d59-e8fa-2421-5ea2-5741a19cf017: 2022-08-12T07:30:08.837Z [ERROR] server-aca55d59-e8fa-2421-5ea2-5741a19cf017: failed to make requestVote RPC: target="{Voter server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c 7e8240ba-d73b-96d5-971c-1548ba4dbd9c}" error="failed to connect to peer: 7e8240ba-d73b-96d5-971c-1548ba4dbd9c" + testing.go:149: server-aca55d59-e8fa-2421-5ea2-5741a19cf017: 2022-08-12T07:30:08.837Z [ERROR] server-aca55d59-e8fa-2421-5ea2-5741a19cf017: failed to make requestVote RPC: target="{Voter server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c 7e8240ba-d73b-96d5-971c-1548ba4dbd9c}" error="failed to connect to peer: 7e8240ba-d73b-96d5-971c-1548ba4dbd9c" +testLoggerAdapter verbose: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.840Z [WARN] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to contact: server-id=server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c time=60.26716ms + testing.go:149: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.840Z [WARN] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to contact: server-id=server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c time=60.26716ms +testLoggerAdapter verbose: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.840Z [WARN] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to contact: server-id=server-b862bd96-bf5f-ba9f-3d00-0b2144d70880 time=58.02744ms + testing.go:149: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.840Z [WARN] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to contact: server-id=server-b862bd96-bf5f-ba9f-3d00-0b2144d70880 time=58.02744ms +testLoggerAdapter verbose: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.840Z [WARN] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to contact quorum of nodes, stepping down + testing.go:149: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.840Z [WARN] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to contact quorum of nodes, stepping down +testLoggerAdapter verbose: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.840Z [INFO] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: entering follower state: follower="Node at 6f8ae9f0-a5ec-7240-322e-703d41a0af5c [Follower]" leader= + testing.go:149: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.840Z [INFO] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: entering follower state: follower="Node at 6f8ae9f0-a5ec-7240-322e-703d41a0af5c [Follower]" leader= +testLoggerAdapter verbose: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.840Z [ERROR] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to appendEntries to: peer="{Voter server-b862bd96-bf5f-ba9f-3d00-0b2144d70880 b862bd96-bf5f-ba9f-3d00-0b2144d70880}" error="failed to connect to peer: b862bd96-bf5f-ba9f-3d00-0b2144d70880" + testing.go:149: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.840Z [ERROR] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to appendEntries to: peer="{Voter server-b862bd96-bf5f-ba9f-3d00-0b2144d70880 b862bd96-bf5f-ba9f-3d00-0b2144d70880}" error="failed to connect to peer: b862bd96-bf5f-ba9f-3d00-0b2144d70880" +testLoggerAdapter verbose: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.840Z [ERROR] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to appendEntries to: peer="{Voter server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c 7e8240ba-d73b-96d5-971c-1548ba4dbd9c}" error="failed to connect to peer: 7e8240ba-d73b-96d5-971c-1548ba4dbd9c" + testing.go:149: server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: 2022-08-12T07:30:08.840Z [ERROR] server-6f8ae9f0-a5ec-7240-322e-703d41a0af5c: failed to appendEntries to: peer="{Voter server-7e8240ba-d73b-96d5-971c-1548ba4dbd9c 7e8240ba-d73b-96d5-971c-1548ba4dbd9c}" error="failed to connect to peer: 7e8240ba-d73b-96d5-971c-1548ba4dbd9c" +--- PASS: TestRaft_RemoveFollower_SplitCluster (0.43s) === RUN TestRaft_AddKnownPeer -testLoggerAdapter verbose: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.378Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5 Address:46f9ebe0-96b0-0420-9cd5-c5b4d85585d5} {Suffrage:Voter ID:server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412 Address:1cd998d1-cbc4-6fe4-cd42-c8958b23e412} {Suffrage:Voter ID:server-8f4b8e8e-6a05-8718-184c-cd2e47306775 Address:8f4b8e8e-6a05-8718-184c-cd2e47306775}]" - testing.go:149: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.378Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5 Address:46f9ebe0-96b0-0420-9cd5-c5b4d85585d5} {Suffrage:Voter ID:server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412 Address:1cd998d1-cbc4-6fe4-cd42-c8958b23e412} {Suffrage:Voter ID:server-8f4b8e8e-6a05-8718-184c-cd2e47306775 Address:8f4b8e8e-6a05-8718-184c-cd2e47306775}]" -testLoggerAdapter verbose: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.378Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: entering follower state: follower="Node at 46f9ebe0-96b0-0420-9cd5-c5b4d85585d5 [Follower]" leader= - testing.go:149: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.378Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: entering follower state: follower="Node at 46f9ebe0-96b0-0420-9cd5-c5b4d85585d5 [Follower]" leader= -testLoggerAdapter verbose: server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412: 2021-07-10T01:05:04.378Z [INFO] server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5 Address:46f9ebe0-96b0-0420-9cd5-c5b4d85585d5} {Suffrage:Voter ID:server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412 Address:1cd998d1-cbc4-6fe4-cd42-c8958b23e412} {Suffrage:Voter ID:server-8f4b8e8e-6a05-8718-184c-cd2e47306775 Address:8f4b8e8e-6a05-8718-184c-cd2e47306775}]" - testing.go:149: server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412: 2021-07-10T01:05:04.378Z [INFO] server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5 Address:46f9ebe0-96b0-0420-9cd5-c5b4d85585d5} {Suffrage:Voter ID:server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412 Address:1cd998d1-cbc4-6fe4-cd42-c8958b23e412} {Suffrage:Voter ID:server-8f4b8e8e-6a05-8718-184c-cd2e47306775 Address:8f4b8e8e-6a05-8718-184c-cd2e47306775}]" -testLoggerAdapter verbose: server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412: 2021-07-10T01:05:04.378Z [INFO] server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412: entering follower state: follower="Node at 1cd998d1-cbc4-6fe4-cd42-c8958b23e412 [Follower]" leader= - testing.go:149: server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412: 2021-07-10T01:05:04.378Z [INFO] server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412: entering follower state: follower="Node at 1cd998d1-cbc4-6fe4-cd42-c8958b23e412 [Follower]" leader= -testLoggerAdapter verbose: server-8f4b8e8e-6a05-8718-184c-cd2e47306775: 2021-07-10T01:05:04.378Z [INFO] server-8f4b8e8e-6a05-8718-184c-cd2e47306775: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5 Address:46f9ebe0-96b0-0420-9cd5-c5b4d85585d5} {Suffrage:Voter ID:server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412 Address:1cd998d1-cbc4-6fe4-cd42-c8958b23e412} {Suffrage:Voter ID:server-8f4b8e8e-6a05-8718-184c-cd2e47306775 Address:8f4b8e8e-6a05-8718-184c-cd2e47306775}]" - testing.go:149: server-8f4b8e8e-6a05-8718-184c-cd2e47306775: 2021-07-10T01:05:04.378Z [INFO] server-8f4b8e8e-6a05-8718-184c-cd2e47306775: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5 Address:46f9ebe0-96b0-0420-9cd5-c5b4d85585d5} {Suffrage:Voter ID:server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412 Address:1cd998d1-cbc4-6fe4-cd42-c8958b23e412} {Suffrage:Voter ID:server-8f4b8e8e-6a05-8718-184c-cd2e47306775 Address:8f4b8e8e-6a05-8718-184c-cd2e47306775}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:04.378Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:04.378Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-8f4b8e8e-6a05-8718-184c-cd2e47306775: 2021-07-10T01:05:04.379Z [INFO] server-8f4b8e8e-6a05-8718-184c-cd2e47306775: entering follower state: follower="Node at 8f4b8e8e-6a05-8718-184c-cd2e47306775 [Follower]" leader= - testing.go:149: server-8f4b8e8e-6a05-8718-184c-cd2e47306775: 2021-07-10T01:05:04.379Z [INFO] server-8f4b8e8e-6a05-8718-184c-cd2e47306775: entering follower state: follower="Node at 8f4b8e8e-6a05-8718-184c-cd2e47306775 [Follower]" leader= -testLoggerAdapter verbose: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.439Z [WARN] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.439Z [WARN] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.440Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: entering candidate state: node="Node at 46f9ebe0-96b0-0420-9cd5-c5b4d85585d5 [Candidate]" term=2 - testing.go:149: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.440Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: entering candidate state: node="Node at 46f9ebe0-96b0-0420-9cd5-c5b4d85585d5 [Candidate]" term=2 -testLoggerAdapter verbose: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.440Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: election won: tally=2 - testing.go:149: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.440Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: election won: tally=2 -testLoggerAdapter verbose: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.440Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: entering leader state: leader="Node at 46f9ebe0-96b0-0420-9cd5-c5b4d85585d5 [Leader]" - testing.go:149: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.440Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: entering leader state: leader="Node at 46f9ebe0-96b0-0420-9cd5-c5b4d85585d5 [Leader]" -testLoggerAdapter verbose: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.440Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: added peer, starting replication: peer=server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412 - testing.go:149: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.440Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: added peer, starting replication: peer=server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412 -testLoggerAdapter verbose: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.440Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: added peer, starting replication: peer=server-8f4b8e8e-6a05-8718-184c-cd2e47306775 - testing.go:149: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.440Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: added peer, starting replication: peer=server-8f4b8e8e-6a05-8718-184c-cd2e47306775 -testLoggerAdapter verbose: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.440Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: pipelining replication: peer="{Voter server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412 1cd998d1-cbc4-6fe4-cd42-c8958b23e412}" - testing.go:149: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.440Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: pipelining replication: peer="{Voter server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412 1cd998d1-cbc4-6fe4-cd42-c8958b23e412}" -testLoggerAdapter verbose: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.440Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: pipelining replication: peer="{Voter server-8f4b8e8e-6a05-8718-184c-cd2e47306775 8f4b8e8e-6a05-8718-184c-cd2e47306775}" - testing.go:149: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.440Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: pipelining replication: peer="{Voter server-8f4b8e8e-6a05-8718-184c-cd2e47306775 8f4b8e8e-6a05-8718-184c-cd2e47306775}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:04.545Z [INFO] stable state for Leader reached at 2021-07-10 01:05:04.440455387 +0000 UTC m=+9.828613377 (1 nodes), 61.399648ms from start of poll, 62.375178ms from cluster start. Timeout at 2021-07-10 01:05:04.545477674 +0000 UTC m=+9.933635644, 105.022267ms after stability - testing.go:149: cluster: 2021-07-10T01:05:04.545Z [INFO] stable state for Leader reached at 2021-07-10 01:05:04.440455387 +0000 UTC m=+9.828613377 (1 nodes), 61.399648ms from start of poll, 62.375178ms from cluster start. Timeout at 2021-07-10 01:05:04.545477674 +0000 UTC m=+9.933635644, 105.022267ms after stability -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:04.545Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-07-10T01:05:04.545Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:04.650Z [INFO] stable state for Follower reached at 2021-07-10 01:05:04.545705256 +0000 UTC m=+9.933863226 (2 nodes), 2.54µs from start of poll, 167.625027ms from cluster start. Timeout at 2021-07-10 01:05:04.650770704 +0000 UTC m=+10.038928674, 105.065448ms after stability - testing.go:149: cluster: 2021-07-10T01:05:04.650Z [INFO] stable state for Follower reached at 2021-07-10 01:05:04.545705256 +0000 UTC m=+9.933863226 (2 nodes), 2.54µs from start of poll, 167.625027ms from cluster start. Timeout at 2021-07-10 01:05:04.650770704 +0000 UTC m=+10.038928674, 105.065448ms after stability -testLoggerAdapter verbose: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.651Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: updating configuration: command=AddStaging server-id=server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412 server-addr=1cd998d1-cbc4-6fe4-cd42-c8958b23e412 servers="[{Suffrage:Voter ID:server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5 Address:46f9ebe0-96b0-0420-9cd5-c5b4d85585d5} {Suffrage:Voter ID:server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412 Address:1cd998d1-cbc4-6fe4-cd42-c8958b23e412} {Suffrage:Voter ID:server-8f4b8e8e-6a05-8718-184c-cd2e47306775 Address:8f4b8e8e-6a05-8718-184c-cd2e47306775}]" - testing.go:149: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.651Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: updating configuration: command=AddStaging server-id=server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412 server-addr=1cd998d1-cbc4-6fe4-cd42-c8958b23e412 servers="[{Suffrage:Voter ID:server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5 Address:46f9ebe0-96b0-0420-9cd5-c5b4d85585d5} {Suffrage:Voter ID:server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412 Address:1cd998d1-cbc4-6fe4-cd42-c8958b23e412} {Suffrage:Voter ID:server-8f4b8e8e-6a05-8718-184c-cd2e47306775 Address:8f4b8e8e-6a05-8718-184c-cd2e47306775}]" -testLoggerAdapter verbose: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.651Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: aborting pipeline replication: peer="{Voter server-8f4b8e8e-6a05-8718-184c-cd2e47306775 8f4b8e8e-6a05-8718-184c-cd2e47306775}" - testing.go:149: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.651Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: aborting pipeline replication: peer="{Voter server-8f4b8e8e-6a05-8718-184c-cd2e47306775 8f4b8e8e-6a05-8718-184c-cd2e47306775}" -testLoggerAdapter verbose: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.651Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: aborting pipeline replication: peer="{Voter server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412 1cd998d1-cbc4-6fe4-cd42-c8958b23e412}" - testing.go:149: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.651Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: aborting pipeline replication: peer="{Voter server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412 1cd998d1-cbc4-6fe4-cd42-c8958b23e412}" +testLoggerAdapter verbose: server-2560cdc2-dfcf-5197-268a-9bd332f6d024: 2022-08-12T07:30:08.842Z [INFO] server-2560cdc2-dfcf-5197-268a-9bd332f6d024: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2560cdc2-dfcf-5197-268a-9bd332f6d024 Address:2560cdc2-dfcf-5197-268a-9bd332f6d024} {Suffrage:Voter ID:server-d669751d-1fd8-eae2-878e-3536fe8e01a1 Address:d669751d-1fd8-eae2-878e-3536fe8e01a1} {Suffrage:Voter ID:server-e3174366-4655-7535-027a-b299454358f9 Address:e3174366-4655-7535-027a-b299454358f9}]" + testing.go:149: server-2560cdc2-dfcf-5197-268a-9bd332f6d024: 2022-08-12T07:30:08.842Z [INFO] server-2560cdc2-dfcf-5197-268a-9bd332f6d024: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2560cdc2-dfcf-5197-268a-9bd332f6d024 Address:2560cdc2-dfcf-5197-268a-9bd332f6d024} {Suffrage:Voter ID:server-d669751d-1fd8-eae2-878e-3536fe8e01a1 Address:d669751d-1fd8-eae2-878e-3536fe8e01a1} {Suffrage:Voter ID:server-e3174366-4655-7535-027a-b299454358f9 Address:e3174366-4655-7535-027a-b299454358f9}]" +testLoggerAdapter verbose: server-2560cdc2-dfcf-5197-268a-9bd332f6d024: 2022-08-12T07:30:08.842Z [INFO] server-2560cdc2-dfcf-5197-268a-9bd332f6d024: entering follower state: follower="Node at 2560cdc2-dfcf-5197-268a-9bd332f6d024 [Follower]" leader= + testing.go:149: server-2560cdc2-dfcf-5197-268a-9bd332f6d024: 2022-08-12T07:30:08.842Z [INFO] server-2560cdc2-dfcf-5197-268a-9bd332f6d024: entering follower state: follower="Node at 2560cdc2-dfcf-5197-268a-9bd332f6d024 [Follower]" leader= +testLoggerAdapter verbose: server-d669751d-1fd8-eae2-878e-3536fe8e01a1: 2022-08-12T07:30:08.842Z [INFO] server-d669751d-1fd8-eae2-878e-3536fe8e01a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2560cdc2-dfcf-5197-268a-9bd332f6d024 Address:2560cdc2-dfcf-5197-268a-9bd332f6d024} {Suffrage:Voter ID:server-d669751d-1fd8-eae2-878e-3536fe8e01a1 Address:d669751d-1fd8-eae2-878e-3536fe8e01a1} {Suffrage:Voter ID:server-e3174366-4655-7535-027a-b299454358f9 Address:e3174366-4655-7535-027a-b299454358f9}]" + testing.go:149: server-d669751d-1fd8-eae2-878e-3536fe8e01a1: 2022-08-12T07:30:08.842Z [INFO] server-d669751d-1fd8-eae2-878e-3536fe8e01a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2560cdc2-dfcf-5197-268a-9bd332f6d024 Address:2560cdc2-dfcf-5197-268a-9bd332f6d024} {Suffrage:Voter ID:server-d669751d-1fd8-eae2-878e-3536fe8e01a1 Address:d669751d-1fd8-eae2-878e-3536fe8e01a1} {Suffrage:Voter ID:server-e3174366-4655-7535-027a-b299454358f9 Address:e3174366-4655-7535-027a-b299454358f9}]" +testLoggerAdapter verbose: server-d669751d-1fd8-eae2-878e-3536fe8e01a1: 2022-08-12T07:30:08.842Z [INFO] server-d669751d-1fd8-eae2-878e-3536fe8e01a1: entering follower state: follower="Node at d669751d-1fd8-eae2-878e-3536fe8e01a1 [Follower]" leader= + testing.go:149: server-d669751d-1fd8-eae2-878e-3536fe8e01a1: 2022-08-12T07:30:08.842Z [INFO] server-d669751d-1fd8-eae2-878e-3536fe8e01a1: entering follower state: follower="Node at d669751d-1fd8-eae2-878e-3536fe8e01a1 [Follower]" leader= +testLoggerAdapter verbose: server-e3174366-4655-7535-027a-b299454358f9: 2022-08-12T07:30:08.842Z [INFO] server-e3174366-4655-7535-027a-b299454358f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2560cdc2-dfcf-5197-268a-9bd332f6d024 Address:2560cdc2-dfcf-5197-268a-9bd332f6d024} {Suffrage:Voter ID:server-d669751d-1fd8-eae2-878e-3536fe8e01a1 Address:d669751d-1fd8-eae2-878e-3536fe8e01a1} {Suffrage:Voter ID:server-e3174366-4655-7535-027a-b299454358f9 Address:e3174366-4655-7535-027a-b299454358f9}]" + testing.go:149: server-e3174366-4655-7535-027a-b299454358f9: 2022-08-12T07:30:08.842Z [INFO] server-e3174366-4655-7535-027a-b299454358f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2560cdc2-dfcf-5197-268a-9bd332f6d024 Address:2560cdc2-dfcf-5197-268a-9bd332f6d024} {Suffrage:Voter ID:server-d669751d-1fd8-eae2-878e-3536fe8e01a1 Address:d669751d-1fd8-eae2-878e-3536fe8e01a1} {Suffrage:Voter ID:server-e3174366-4655-7535-027a-b299454358f9 Address:e3174366-4655-7535-027a-b299454358f9}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:08.843Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-e3174366-4655-7535-027a-b299454358f9: 2022-08-12T07:30:08.843Z [INFO] server-e3174366-4655-7535-027a-b299454358f9: entering follower state: follower="Node at e3174366-4655-7535-027a-b299454358f9 [Follower]" leader= + testing.go:149: cluster: 2022-08-12T07:30:08.843Z [INFO] starting stability test: raft-state=Leader + testing.go:149: server-e3174366-4655-7535-027a-b299454358f9: 2022-08-12T07:30:08.843Z [INFO] server-e3174366-4655-7535-027a-b299454358f9: entering follower state: follower="Node at e3174366-4655-7535-027a-b299454358f9 [Follower]" leader= +testLoggerAdapter verbose: server-2560cdc2-dfcf-5197-268a-9bd332f6d024: 2022-08-12T07:30:08.897Z [WARN] server-2560cdc2-dfcf-5197-268a-9bd332f6d024: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-2560cdc2-dfcf-5197-268a-9bd332f6d024: 2022-08-12T07:30:08.897Z [WARN] server-2560cdc2-dfcf-5197-268a-9bd332f6d024: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-2560cdc2-dfcf-5197-268a-9bd332f6d024: 2022-08-12T07:30:08.897Z [INFO] server-2560cdc2-dfcf-5197-268a-9bd332f6d024: entering candidate state: node="Node at 2560cdc2-dfcf-5197-268a-9bd332f6d024 [Candidate]" term=2 + testing.go:149: server-2560cdc2-dfcf-5197-268a-9bd332f6d024: 2022-08-12T07:30:08.897Z [INFO] server-2560cdc2-dfcf-5197-268a-9bd332f6d024: entering candidate state: node="Node at 2560cdc2-dfcf-5197-268a-9bd332f6d024 [Candidate]" term=2 +testLoggerAdapter verbose: server-2560cdc2-dfcf-5197-268a-9bd332f6d024: 2022-08-12T07:30:08.898Z [INFO] server-2560cdc2-dfcf-5197-268a-9bd332f6d024: election won: tally=2 + testing.go:149: server-2560cdc2-dfcf-5197-268a-9bd332f6d024: 2022-08-12T07:30:08.898Z [INFO] server-2560cdc2-dfcf-5197-268a-9bd332f6d024: election won: tally=2 +testLoggerAdapter verbose: server-2560cdc2-dfcf-5197-268a-9bd332f6d024: 2022-08-12T07:30:08.898Z [INFO] server-2560cdc2-dfcf-5197-268a-9bd332f6d024: entering leader state: leader="Node at 2560cdc2-dfcf-5197-268a-9bd332f6d024 [Leader]" + testing.go:149: server-2560cdc2-dfcf-5197-268a-9bd332f6d024: 2022-08-12T07:30:08.898Z [INFO] server-2560cdc2-dfcf-5197-268a-9bd332f6d024: entering leader state: leader="Node at 2560cdc2-dfcf-5197-268a-9bd332f6d024 [Leader]" +testLoggerAdapter verbose: server-2560cdc2-dfcf-5197-268a-9bd332f6d024: 2022-08-12T07:30:08.898Z [INFO] server-2560cdc2-dfcf-5197-268a-9bd332f6d024: added peer, starting replication: peer=server-d669751d-1fd8-eae2-878e-3536fe8e01a1 + testing.go:149: server-2560cdc2-dfcf-5197-268a-9bd332f6d024: 2022-08-12T07:30:08.898Z [INFO] server-2560cdc2-dfcf-5197-268a-9bd332f6d024: added peer, starting replication: peer=server-d669751d-1fd8-eae2-878e-3536fe8e01a1 +testLoggerAdapter verbose: server-2560cdc2-dfcf-5197-268a-9bd332f6d024: 2022-08-12T07:30:08.898Z [INFO] server-2560cdc2-dfcf-5197-268a-9bd332f6d024: added peer, starting replication: peer=server-e3174366-4655-7535-027a-b299454358f9 + testing.go:149: server-2560cdc2-dfcf-5197-268a-9bd332f6d024: 2022-08-12T07:30:08.898Z [INFO] server-2560cdc2-dfcf-5197-268a-9bd332f6d024: added peer, starting replication: peer=server-e3174366-4655-7535-027a-b299454358f9 +testLoggerAdapter verbose: server-2560cdc2-dfcf-5197-268a-9bd332f6d024: 2022-08-12T07:30:08.898Z [INFO] server-2560cdc2-dfcf-5197-268a-9bd332f6d024: pipelining replication: peer="{Voter server-d669751d-1fd8-eae2-878e-3536fe8e01a1 d669751d-1fd8-eae2-878e-3536fe8e01a1}" + testing.go:149: server-2560cdc2-dfcf-5197-268a-9bd332f6d024: 2022-08-12T07:30:08.898Z [INFO] server-2560cdc2-dfcf-5197-268a-9bd332f6d024: pipelining replication: peer="{Voter server-d669751d-1fd8-eae2-878e-3536fe8e01a1 d669751d-1fd8-eae2-878e-3536fe8e01a1}" +testLoggerAdapter verbose: server-2560cdc2-dfcf-5197-268a-9bd332f6d024: 2022-08-12T07:30:08.898Z [INFO] server-2560cdc2-dfcf-5197-268a-9bd332f6d024: pipelining replication: peer="{Voter server-e3174366-4655-7535-027a-b299454358f9 e3174366-4655-7535-027a-b299454358f9}" + testing.go:149: server-2560cdc2-dfcf-5197-268a-9bd332f6d024: 2022-08-12T07:30:08.898Z [INFO] server-2560cdc2-dfcf-5197-268a-9bd332f6d024: pipelining replication: peer="{Voter server-e3174366-4655-7535-027a-b299454358f9 e3174366-4655-7535-027a-b299454358f9}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:09.003Z [INFO] stable state for Leader reached at 2022-08-12 07:30:08.89831454 +0000 UTC m=+9.855001001 (1 nodes), 55.22826ms from start of poll, 56.23324ms from cluster start. Timeout at 2022-08-12 07:30:09.00337872 +0000 UTC m=+9.960065201, 105.0642ms after stability + testing.go:149: cluster: 2022-08-12T07:30:09.003Z [INFO] stable state for Leader reached at 2022-08-12 07:30:08.89831454 +0000 UTC m=+9.855001001 (1 nodes), 55.22826ms from start of poll, 56.23324ms from cluster start. Timeout at 2022-08-12 07:30:09.00337872 +0000 UTC m=+9.960065201, 105.0642ms after stability +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:09.003Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2022-08-12T07:30:09.003Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:09.108Z [INFO] stable state for Follower reached at 2022-08-12 07:30:09.00363012 +0000 UTC m=+9.960316581 (2 nodes), 2.2µs from start of poll, 161.54882ms from cluster start. Timeout at 2022-08-12 07:30:09.10870186 +0000 UTC m=+10.065388361, 105.07178ms after stability + testing.go:149: cluster: 2022-08-12T07:30:09.108Z [INFO] stable state for Follower reached at 2022-08-12 07:30:09.00363012 +0000 UTC m=+9.960316581 (2 nodes), 2.2µs from start of poll, 161.54882ms from cluster start. Timeout at 2022-08-12 07:30:09.10870186 +0000 UTC m=+10.065388361, 105.07178ms after stability +testLoggerAdapter verbose: server-2560cdc2-dfcf-5197-268a-9bd332f6d024: 2022-08-12T07:30:09.109Z [INFO] server-2560cdc2-dfcf-5197-268a-9bd332f6d024: updating configuration: command=AddStaging server-id=server-d669751d-1fd8-eae2-878e-3536fe8e01a1 server-addr=d669751d-1fd8-eae2-878e-3536fe8e01a1 servers="[{Suffrage:Voter ID:server-2560cdc2-dfcf-5197-268a-9bd332f6d024 Address:2560cdc2-dfcf-5197-268a-9bd332f6d024} {Suffrage:Voter ID:server-d669751d-1fd8-eae2-878e-3536fe8e01a1 Address:d669751d-1fd8-eae2-878e-3536fe8e01a1} {Suffrage:Voter ID:server-e3174366-4655-7535-027a-b299454358f9 Address:e3174366-4655-7535-027a-b299454358f9}]" + testing.go:149: server-2560cdc2-dfcf-5197-268a-9bd332f6d024: 2022-08-12T07:30:09.109Z [INFO] server-2560cdc2-dfcf-5197-268a-9bd332f6d024: updating configuration: command=AddStaging server-id=server-d669751d-1fd8-eae2-878e-3536fe8e01a1 server-addr=d669751d-1fd8-eae2-878e-3536fe8e01a1 servers="[{Suffrage:Voter ID:server-2560cdc2-dfcf-5197-268a-9bd332f6d024 Address:2560cdc2-dfcf-5197-268a-9bd332f6d024} {Suffrage:Voter ID:server-d669751d-1fd8-eae2-878e-3536fe8e01a1 Address:d669751d-1fd8-eae2-878e-3536fe8e01a1} {Suffrage:Voter ID:server-e3174366-4655-7535-027a-b299454358f9 Address:e3174366-4655-7535-027a-b299454358f9}]" +testLoggerAdapter verbose: server-2560cdc2-dfcf-5197-268a-9bd332f6d024: 2022-08-12T07:30:09.109Z [INFO] server-2560cdc2-dfcf-5197-268a-9bd332f6d024: aborting pipeline replication: peer="{Voter server-d669751d-1fd8-eae2-878e-3536fe8e01a1 d669751d-1fd8-eae2-878e-3536fe8e01a1}" + testing.go:149: server-2560cdc2-dfcf-5197-268a-9bd332f6d024: 2022-08-12T07:30:09.109Z [INFO] server-2560cdc2-dfcf-5197-268a-9bd332f6d024: aborting pipeline replication: peer="{Voter server-d669751d-1fd8-eae2-878e-3536fe8e01a1 d669751d-1fd8-eae2-878e-3536fe8e01a1}" +testLoggerAdapter verbose: server-2560cdc2-dfcf-5197-268a-9bd332f6d024: 2022-08-12T07:30:09.109Z [INFO] server-2560cdc2-dfcf-5197-268a-9bd332f6d024: aborting pipeline replication: peer="{Voter server-e3174366-4655-7535-027a-b299454358f9 e3174366-4655-7535-027a-b299454358f9}" + testing.go:149: server-2560cdc2-dfcf-5197-268a-9bd332f6d024: 2022-08-12T07:30:09.109Z [INFO] server-2560cdc2-dfcf-5197-268a-9bd332f6d024: aborting pipeline replication: peer="{Voter server-e3174366-4655-7535-027a-b299454358f9 e3174366-4655-7535-027a-b299454358f9}" --- PASS: TestRaft_AddKnownPeer (0.27s) === RUN TestRaft_RemoveUnknownPeer -testLoggerAdapter verbose: server-46668501-dcf3-613d-4f54-fde5c2a55bcf: 2021-07-10T01:05:04.653Z [INFO] server-46668501-dcf3-613d-4f54-fde5c2a55bcf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-46668501-dcf3-613d-4f54-fde5c2a55bcf Address:46668501-dcf3-613d-4f54-fde5c2a55bcf} {Suffrage:Voter ID:server-3497e30e-884c-48a8-5653-12d1a38507a4 Address:3497e30e-884c-48a8-5653-12d1a38507a4} {Suffrage:Voter ID:server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4 Address:9aa15ca3-275d-8bac-133b-09b2ee61c7d4}]" - testing.go:149: server-46668501-dcf3-613d-4f54-fde5c2a55bcf: 2021-07-10T01:05:04.653Z [INFO] server-46668501-dcf3-613d-4f54-fde5c2a55bcf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-46668501-dcf3-613d-4f54-fde5c2a55bcf Address:46668501-dcf3-613d-4f54-fde5c2a55bcf} {Suffrage:Voter ID:server-3497e30e-884c-48a8-5653-12d1a38507a4 Address:3497e30e-884c-48a8-5653-12d1a38507a4} {Suffrage:Voter ID:server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4 Address:9aa15ca3-275d-8bac-133b-09b2ee61c7d4}]" -testLoggerAdapter verbose: server-46668501-dcf3-613d-4f54-fde5c2a55bcf: 2021-07-10T01:05:04.653Z [INFO] server-46668501-dcf3-613d-4f54-fde5c2a55bcf: entering follower state: follower="Node at 46668501-dcf3-613d-4f54-fde5c2a55bcf [Follower]" leader= - testing.go:149: server-46668501-dcf3-613d-4f54-fde5c2a55bcf: 2021-07-10T01:05:04.653Z [INFO] server-46668501-dcf3-613d-4f54-fde5c2a55bcf: entering follower state: follower="Node at 46668501-dcf3-613d-4f54-fde5c2a55bcf [Follower]" leader= -testLoggerAdapter verbose: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.653Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-46668501-dcf3-613d-4f54-fde5c2a55bcf Address:46668501-dcf3-613d-4f54-fde5c2a55bcf} {Suffrage:Voter ID:server-3497e30e-884c-48a8-5653-12d1a38507a4 Address:3497e30e-884c-48a8-5653-12d1a38507a4} {Suffrage:Voter ID:server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4 Address:9aa15ca3-275d-8bac-133b-09b2ee61c7d4}]" - testing.go:149: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.653Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-46668501-dcf3-613d-4f54-fde5c2a55bcf Address:46668501-dcf3-613d-4f54-fde5c2a55bcf} {Suffrage:Voter ID:server-3497e30e-884c-48a8-5653-12d1a38507a4 Address:3497e30e-884c-48a8-5653-12d1a38507a4} {Suffrage:Voter ID:server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4 Address:9aa15ca3-275d-8bac-133b-09b2ee61c7d4}]" -testLoggerAdapter verbose: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.653Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: entering follower state: follower="Node at 3497e30e-884c-48a8-5653-12d1a38507a4 [Follower]" leader= - testing.go:149: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.653Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: entering follower state: follower="Node at 3497e30e-884c-48a8-5653-12d1a38507a4 [Follower]" leader= -testLoggerAdapter verbose: server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4: 2021-07-10T01:05:04.653Z [INFO] server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-46668501-dcf3-613d-4f54-fde5c2a55bcf Address:46668501-dcf3-613d-4f54-fde5c2a55bcf} {Suffrage:Voter ID:server-3497e30e-884c-48a8-5653-12d1a38507a4 Address:3497e30e-884c-48a8-5653-12d1a38507a4} {Suffrage:Voter ID:server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4 Address:9aa15ca3-275d-8bac-133b-09b2ee61c7d4}]" - testing.go:149: server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4: 2021-07-10T01:05:04.653Z [INFO] server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-46668501-dcf3-613d-4f54-fde5c2a55bcf Address:46668501-dcf3-613d-4f54-fde5c2a55bcf} {Suffrage:Voter ID:server-3497e30e-884c-48a8-5653-12d1a38507a4 Address:3497e30e-884c-48a8-5653-12d1a38507a4} {Suffrage:Voter ID:server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4 Address:9aa15ca3-275d-8bac-133b-09b2ee61c7d4}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:04.653Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:04.653Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4: 2021-07-10T01:05:04.653Z [INFO] server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4: entering follower state: follower="Node at 9aa15ca3-275d-8bac-133b-09b2ee61c7d4 [Follower]" leader= - testing.go:149: server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4: 2021-07-10T01:05:04.653Z [INFO] server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4: entering follower state: follower="Node at 9aa15ca3-275d-8bac-133b-09b2ee61c7d4 [Follower]" leader= -testLoggerAdapter verbose: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.704Z [WARN] server-3497e30e-884c-48a8-5653-12d1a38507a4: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.704Z [WARN] server-3497e30e-884c-48a8-5653-12d1a38507a4: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.704Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: entering candidate state: node="Node at 3497e30e-884c-48a8-5653-12d1a38507a4 [Candidate]" term=2 - testing.go:149: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.704Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: entering candidate state: node="Node at 3497e30e-884c-48a8-5653-12d1a38507a4 [Candidate]" term=2 -testLoggerAdapter verbose: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.705Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: election won: tally=2 - testing.go:149: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.705Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: election won: tally=2 -testLoggerAdapter verbose: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.705Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: entering leader state: leader="Node at 3497e30e-884c-48a8-5653-12d1a38507a4 [Leader]" - testing.go:149: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.705Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: entering leader state: leader="Node at 3497e30e-884c-48a8-5653-12d1a38507a4 [Leader]" -testLoggerAdapter verbose: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.705Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: added peer, starting replication: peer=server-46668501-dcf3-613d-4f54-fde5c2a55bcf - testing.go:149: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.705Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: added peer, starting replication: peer=server-46668501-dcf3-613d-4f54-fde5c2a55bcf -testLoggerAdapter verbose: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.705Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: added peer, starting replication: peer=server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4 - testing.go:149: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.705Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: added peer, starting replication: peer=server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4 -testLoggerAdapter verbose: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.705Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: pipelining replication: peer="{Voter server-46668501-dcf3-613d-4f54-fde5c2a55bcf 46668501-dcf3-613d-4f54-fde5c2a55bcf}" - testing.go:149: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.705Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: pipelining replication: peer="{Voter server-46668501-dcf3-613d-4f54-fde5c2a55bcf 46668501-dcf3-613d-4f54-fde5c2a55bcf}" -testLoggerAdapter verbose: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.705Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: pipelining replication: peer="{Voter server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4 9aa15ca3-275d-8bac-133b-09b2ee61c7d4}" - testing.go:149: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.705Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: pipelining replication: peer="{Voter server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4 9aa15ca3-275d-8bac-133b-09b2ee61c7d4}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:04.810Z [INFO] stable state for Leader reached at 2021-07-10 01:05:04.705224358 +0000 UTC m=+10.093382328 (1 nodes), 51.26126ms from start of poll, 52.254571ms from cluster start. Timeout at 2021-07-10 01:05:04.810289046 +0000 UTC m=+10.198447036, 105.064708ms after stability - testing.go:149: cluster: 2021-07-10T01:05:04.810Z [INFO] stable state for Leader reached at 2021-07-10 01:05:04.705224358 +0000 UTC m=+10.093382328 (1 nodes), 51.26126ms from start of poll, 52.254571ms from cluster start. Timeout at 2021-07-10 01:05:04.810289046 +0000 UTC m=+10.198447036, 105.064708ms after stability -testLoggerAdapter verbose: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.810Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: updating configuration: command=RemoveServer server-id=9b1b8d02-0f87-2fe1-81fa-12f8626b104b server-addr= servers="[{Suffrage:Voter ID:server-46668501-dcf3-613d-4f54-fde5c2a55bcf Address:46668501-dcf3-613d-4f54-fde5c2a55bcf} {Suffrage:Voter ID:server-3497e30e-884c-48a8-5653-12d1a38507a4 Address:3497e30e-884c-48a8-5653-12d1a38507a4} {Suffrage:Voter ID:server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4 Address:9aa15ca3-275d-8bac-133b-09b2ee61c7d4}]" - testing.go:149: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.810Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: updating configuration: command=RemoveServer server-id=9b1b8d02-0f87-2fe1-81fa-12f8626b104b server-addr= servers="[{Suffrage:Voter ID:server-46668501-dcf3-613d-4f54-fde5c2a55bcf Address:46668501-dcf3-613d-4f54-fde5c2a55bcf} {Suffrage:Voter ID:server-3497e30e-884c-48a8-5653-12d1a38507a4 Address:3497e30e-884c-48a8-5653-12d1a38507a4} {Suffrage:Voter ID:server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4 Address:9aa15ca3-275d-8bac-133b-09b2ee61c7d4}]" -testLoggerAdapter verbose: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.811Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: aborting pipeline replication: peer="{Voter server-46668501-dcf3-613d-4f54-fde5c2a55bcf 46668501-dcf3-613d-4f54-fde5c2a55bcf}" - testing.go:149: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.811Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: aborting pipeline replication: peer="{Voter server-46668501-dcf3-613d-4f54-fde5c2a55bcf 46668501-dcf3-613d-4f54-fde5c2a55bcf}" -testLoggerAdapter verbose: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.811Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: aborting pipeline replication: peer="{Voter server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4 9aa15ca3-275d-8bac-133b-09b2ee61c7d4}" - testing.go:149: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.811Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: aborting pipeline replication: peer="{Voter server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4 9aa15ca3-275d-8bac-133b-09b2ee61c7d4}" ---- PASS: TestRaft_RemoveUnknownPeer (0.16s) +testLoggerAdapter verbose: server-64c4ed11-e6af-8d8b-b401-b656c95bc608: 2022-08-12T07:30:09.110Z [INFO] server-64c4ed11-e6af-8d8b-b401-b656c95bc608: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-64c4ed11-e6af-8d8b-b401-b656c95bc608 Address:64c4ed11-e6af-8d8b-b401-b656c95bc608} {Suffrage:Voter ID:server-65cdcc94-bf03-1b87-4c8d-562b11352787 Address:65cdcc94-bf03-1b87-4c8d-562b11352787} {Suffrage:Voter ID:server-ecbc118a-0c08-1da2-5147-9da615203f29 Address:ecbc118a-0c08-1da2-5147-9da615203f29}]" + testing.go:149: server-64c4ed11-e6af-8d8b-b401-b656c95bc608: 2022-08-12T07:30:09.110Z [INFO] server-64c4ed11-e6af-8d8b-b401-b656c95bc608: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-64c4ed11-e6af-8d8b-b401-b656c95bc608 Address:64c4ed11-e6af-8d8b-b401-b656c95bc608} {Suffrage:Voter ID:server-65cdcc94-bf03-1b87-4c8d-562b11352787 Address:65cdcc94-bf03-1b87-4c8d-562b11352787} {Suffrage:Voter ID:server-ecbc118a-0c08-1da2-5147-9da615203f29 Address:ecbc118a-0c08-1da2-5147-9da615203f29}]" +testLoggerAdapter verbose: server-64c4ed11-e6af-8d8b-b401-b656c95bc608: 2022-08-12T07:30:09.111Z [INFO] server-64c4ed11-e6af-8d8b-b401-b656c95bc608: entering follower state: follower="Node at 64c4ed11-e6af-8d8b-b401-b656c95bc608 [Follower]" leader= + testing.go:149: server-64c4ed11-e6af-8d8b-b401-b656c95bc608: 2022-08-12T07:30:09.111Z [INFO] server-64c4ed11-e6af-8d8b-b401-b656c95bc608: entering follower state: follower="Node at 64c4ed11-e6af-8d8b-b401-b656c95bc608 [Follower]" leader= +testLoggerAdapter verbose: server-65cdcc94-bf03-1b87-4c8d-562b11352787: 2022-08-12T07:30:09.111Z [INFO] server-65cdcc94-bf03-1b87-4c8d-562b11352787: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-64c4ed11-e6af-8d8b-b401-b656c95bc608 Address:64c4ed11-e6af-8d8b-b401-b656c95bc608} {Suffrage:Voter ID:server-65cdcc94-bf03-1b87-4c8d-562b11352787 Address:65cdcc94-bf03-1b87-4c8d-562b11352787} {Suffrage:Voter ID:server-ecbc118a-0c08-1da2-5147-9da615203f29 Address:ecbc118a-0c08-1da2-5147-9da615203f29}]" + testing.go:149: server-65cdcc94-bf03-1b87-4c8d-562b11352787: 2022-08-12T07:30:09.111Z [INFO] server-65cdcc94-bf03-1b87-4c8d-562b11352787: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-64c4ed11-e6af-8d8b-b401-b656c95bc608 Address:64c4ed11-e6af-8d8b-b401-b656c95bc608} {Suffrage:Voter ID:server-65cdcc94-bf03-1b87-4c8d-562b11352787 Address:65cdcc94-bf03-1b87-4c8d-562b11352787} {Suffrage:Voter ID:server-ecbc118a-0c08-1da2-5147-9da615203f29 Address:ecbc118a-0c08-1da2-5147-9da615203f29}]" +testLoggerAdapter verbose: server-65cdcc94-bf03-1b87-4c8d-562b11352787: 2022-08-12T07:30:09.111Z [INFO] server-65cdcc94-bf03-1b87-4c8d-562b11352787: entering follower state: follower="Node at 65cdcc94-bf03-1b87-4c8d-562b11352787 [Follower]" leader= + testing.go:149: server-65cdcc94-bf03-1b87-4c8d-562b11352787: 2022-08-12T07:30:09.111Z [INFO] server-65cdcc94-bf03-1b87-4c8d-562b11352787: entering follower state: follower="Node at 65cdcc94-bf03-1b87-4c8d-562b11352787 [Follower]" leader= +testLoggerAdapter verbose: server-ecbc118a-0c08-1da2-5147-9da615203f29: 2022-08-12T07:30:09.111Z [INFO] server-ecbc118a-0c08-1da2-5147-9da615203f29: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-64c4ed11-e6af-8d8b-b401-b656c95bc608 Address:64c4ed11-e6af-8d8b-b401-b656c95bc608} {Suffrage:Voter ID:server-65cdcc94-bf03-1b87-4c8d-562b11352787 Address:65cdcc94-bf03-1b87-4c8d-562b11352787} {Suffrage:Voter ID:server-ecbc118a-0c08-1da2-5147-9da615203f29 Address:ecbc118a-0c08-1da2-5147-9da615203f29}]" + testing.go:149: server-ecbc118a-0c08-1da2-5147-9da615203f29: 2022-08-12T07:30:09.111Z [INFO] server-ecbc118a-0c08-1da2-5147-9da615203f29: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-64c4ed11-e6af-8d8b-b401-b656c95bc608 Address:64c4ed11-e6af-8d8b-b401-b656c95bc608} {Suffrage:Voter ID:server-65cdcc94-bf03-1b87-4c8d-562b11352787 Address:65cdcc94-bf03-1b87-4c8d-562b11352787} {Suffrage:Voter ID:server-ecbc118a-0c08-1da2-5147-9da615203f29 Address:ecbc118a-0c08-1da2-5147-9da615203f29}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:09.111Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:09.111Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-ecbc118a-0c08-1da2-5147-9da615203f29: 2022-08-12T07:30:09.111Z [INFO] server-ecbc118a-0c08-1da2-5147-9da615203f29: entering follower state: follower="Node at ecbc118a-0c08-1da2-5147-9da615203f29 [Follower]" leader= + testing.go:149: server-ecbc118a-0c08-1da2-5147-9da615203f29: 2022-08-12T07:30:09.111Z [INFO] server-ecbc118a-0c08-1da2-5147-9da615203f29: entering follower state: follower="Node at ecbc118a-0c08-1da2-5147-9da615203f29 [Follower]" leader= +testLoggerAdapter verbose: server-65cdcc94-bf03-1b87-4c8d-562b11352787: 2022-08-12T07:30:09.195Z [WARN] server-65cdcc94-bf03-1b87-4c8d-562b11352787: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-65cdcc94-bf03-1b87-4c8d-562b11352787: 2022-08-12T07:30:09.195Z [WARN] server-65cdcc94-bf03-1b87-4c8d-562b11352787: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-65cdcc94-bf03-1b87-4c8d-562b11352787: 2022-08-12T07:30:09.196Z [INFO] server-65cdcc94-bf03-1b87-4c8d-562b11352787: entering candidate state: node="Node at 65cdcc94-bf03-1b87-4c8d-562b11352787 [Candidate]" term=2 + testing.go:149: server-65cdcc94-bf03-1b87-4c8d-562b11352787: 2022-08-12T07:30:09.196Z [INFO] server-65cdcc94-bf03-1b87-4c8d-562b11352787: entering candidate state: node="Node at 65cdcc94-bf03-1b87-4c8d-562b11352787 [Candidate]" term=2 +testLoggerAdapter verbose: server-65cdcc94-bf03-1b87-4c8d-562b11352787: 2022-08-12T07:30:09.196Z [INFO] server-65cdcc94-bf03-1b87-4c8d-562b11352787: election won: tally=2 + testing.go:149: server-65cdcc94-bf03-1b87-4c8d-562b11352787: 2022-08-12T07:30:09.196Z [INFO] server-65cdcc94-bf03-1b87-4c8d-562b11352787: election won: tally=2 +testLoggerAdapter verbose: server-65cdcc94-bf03-1b87-4c8d-562b11352787: 2022-08-12T07:30:09.196Z [INFO] server-65cdcc94-bf03-1b87-4c8d-562b11352787: entering leader state: leader="Node at 65cdcc94-bf03-1b87-4c8d-562b11352787 [Leader]" + testing.go:149: server-65cdcc94-bf03-1b87-4c8d-562b11352787: 2022-08-12T07:30:09.196Z [INFO] server-65cdcc94-bf03-1b87-4c8d-562b11352787: entering leader state: leader="Node at 65cdcc94-bf03-1b87-4c8d-562b11352787 [Leader]" +testLoggerAdapter verbose: server-65cdcc94-bf03-1b87-4c8d-562b11352787: 2022-08-12T07:30:09.196Z [INFO] server-65cdcc94-bf03-1b87-4c8d-562b11352787: added peer, starting replication: peer=server-64c4ed11-e6af-8d8b-b401-b656c95bc608 + testing.go:149: server-65cdcc94-bf03-1b87-4c8d-562b11352787: 2022-08-12T07:30:09.196Z [INFO] server-65cdcc94-bf03-1b87-4c8d-562b11352787: added peer, starting replication: peer=server-64c4ed11-e6af-8d8b-b401-b656c95bc608 +testLoggerAdapter verbose: server-65cdcc94-bf03-1b87-4c8d-562b11352787: 2022-08-12T07:30:09.196Z [INFO] server-65cdcc94-bf03-1b87-4c8d-562b11352787: added peer, starting replication: peer=server-ecbc118a-0c08-1da2-5147-9da615203f29 + testing.go:149: server-65cdcc94-bf03-1b87-4c8d-562b11352787: 2022-08-12T07:30:09.196Z [INFO] server-65cdcc94-bf03-1b87-4c8d-562b11352787: added peer, starting replication: peer=server-ecbc118a-0c08-1da2-5147-9da615203f29 +testLoggerAdapter verbose: server-65cdcc94-bf03-1b87-4c8d-562b11352787: 2022-08-12T07:30:09.196Z [INFO] server-65cdcc94-bf03-1b87-4c8d-562b11352787: pipelining replication: peer="{Voter server-ecbc118a-0c08-1da2-5147-9da615203f29 ecbc118a-0c08-1da2-5147-9da615203f29}" + testing.go:149: server-65cdcc94-bf03-1b87-4c8d-562b11352787: 2022-08-12T07:30:09.196Z [INFO] server-65cdcc94-bf03-1b87-4c8d-562b11352787: pipelining replication: peer="{Voter server-ecbc118a-0c08-1da2-5147-9da615203f29 ecbc118a-0c08-1da2-5147-9da615203f29}" +testLoggerAdapter verbose: server-65cdcc94-bf03-1b87-4c8d-562b11352787: 2022-08-12T07:30:09.196Z [INFO] server-65cdcc94-bf03-1b87-4c8d-562b11352787: pipelining replication: peer="{Voter server-64c4ed11-e6af-8d8b-b401-b656c95bc608 64c4ed11-e6af-8d8b-b401-b656c95bc608}" + testing.go:149: server-65cdcc94-bf03-1b87-4c8d-562b11352787: 2022-08-12T07:30:09.196Z [INFO] server-65cdcc94-bf03-1b87-4c8d-562b11352787: pipelining replication: peer="{Voter server-64c4ed11-e6af-8d8b-b401-b656c95bc608 64c4ed11-e6af-8d8b-b401-b656c95bc608}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:09.301Z [INFO] stable state for Leader reached at 2022-08-12 07:30:09.19647208 +0000 UTC m=+10.153158561 (1 nodes), 84.8337ms from start of poll, 85.72774ms from cluster start. Timeout at 2022-08-12 07:30:09.30155712 +0000 UTC m=+10.258243581, 105.08502ms after stability + testing.go:149: cluster: 2022-08-12T07:30:09.301Z [INFO] stable state for Leader reached at 2022-08-12 07:30:09.19647208 +0000 UTC m=+10.153158561 (1 nodes), 84.8337ms from start of poll, 85.72774ms from cluster start. Timeout at 2022-08-12 07:30:09.30155712 +0000 UTC m=+10.258243581, 105.08502ms after stability +testLoggerAdapter verbose: server-65cdcc94-bf03-1b87-4c8d-562b11352787: 2022-08-12T07:30:09.301Z [INFO] server-65cdcc94-bf03-1b87-4c8d-562b11352787: updating configuration: command=RemoveServer server-id=9612bf3c-7e94-2e0b-baa5-b0edfeeb6cc3 server-addr= servers="[{Suffrage:Voter ID:server-64c4ed11-e6af-8d8b-b401-b656c95bc608 Address:64c4ed11-e6af-8d8b-b401-b656c95bc608} {Suffrage:Voter ID:server-65cdcc94-bf03-1b87-4c8d-562b11352787 Address:65cdcc94-bf03-1b87-4c8d-562b11352787} {Suffrage:Voter ID:server-ecbc118a-0c08-1da2-5147-9da615203f29 Address:ecbc118a-0c08-1da2-5147-9da615203f29}]" + testing.go:149: server-65cdcc94-bf03-1b87-4c8d-562b11352787: 2022-08-12T07:30:09.301Z [INFO] server-65cdcc94-bf03-1b87-4c8d-562b11352787: updating configuration: command=RemoveServer server-id=9612bf3c-7e94-2e0b-baa5-b0edfeeb6cc3 server-addr= servers="[{Suffrage:Voter ID:server-64c4ed11-e6af-8d8b-b401-b656c95bc608 Address:64c4ed11-e6af-8d8b-b401-b656c95bc608} {Suffrage:Voter ID:server-65cdcc94-bf03-1b87-4c8d-562b11352787 Address:65cdcc94-bf03-1b87-4c8d-562b11352787} {Suffrage:Voter ID:server-ecbc118a-0c08-1da2-5147-9da615203f29 Address:ecbc118a-0c08-1da2-5147-9da615203f29}]" +testLoggerAdapter verbose: server-65cdcc94-bf03-1b87-4c8d-562b11352787: 2022-08-12T07:30:09.302Z [INFO] server-65cdcc94-bf03-1b87-4c8d-562b11352787: aborting pipeline replication: peer="{Voter server-ecbc118a-0c08-1da2-5147-9da615203f29 ecbc118a-0c08-1da2-5147-9da615203f29}" + testing.go:149: server-65cdcc94-bf03-1b87-4c8d-562b11352787: 2022-08-12T07:30:09.302Z [INFO] server-65cdcc94-bf03-1b87-4c8d-562b11352787: aborting pipeline replication: peer="{Voter server-ecbc118a-0c08-1da2-5147-9da615203f29 ecbc118a-0c08-1da2-5147-9da615203f29}" +testLoggerAdapter verbose: server-65cdcc94-bf03-1b87-4c8d-562b11352787: 2022-08-12T07:30:09.302Z [INFO] server-65cdcc94-bf03-1b87-4c8d-562b11352787: aborting pipeline replication: peer="{Voter server-64c4ed11-e6af-8d8b-b401-b656c95bc608 64c4ed11-e6af-8d8b-b401-b656c95bc608}" + testing.go:149: server-65cdcc94-bf03-1b87-4c8d-562b11352787: 2022-08-12T07:30:09.302Z [INFO] server-65cdcc94-bf03-1b87-4c8d-562b11352787: aborting pipeline replication: peer="{Voter server-64c4ed11-e6af-8d8b-b401-b656c95bc608 64c4ed11-e6af-8d8b-b401-b656c95bc608}" +--- PASS: TestRaft_RemoveUnknownPeer (0.19s) === RUN TestRaft_SnapshotRestore -testLoggerAdapter verbose: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:04.812Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-72ddfcd9-c408-849c-f04f-07e68de80a4b Address:72ddfcd9-c408-849c-f04f-07e68de80a4b}]" - testing.go:149: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:04.812Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-72ddfcd9-c408-849c-f04f-07e68de80a4b Address:72ddfcd9-c408-849c-f04f-07e68de80a4b}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:04.812Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:04.812Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:04.812Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: entering follower state: follower="Node at 72ddfcd9-c408-849c-f04f-07e68de80a4b [Follower]" leader= - testing.go:149: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:04.812Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: entering follower state: follower="Node at 72ddfcd9-c408-849c-f04f-07e68de80a4b [Follower]" leader= -testLoggerAdapter verbose: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:04.911Z [WARN] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:04.911Z [WARN] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:04.911Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: entering candidate state: node="Node at 72ddfcd9-c408-849c-f04f-07e68de80a4b [Candidate]" term=2 - testing.go:149: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:04.911Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: entering candidate state: node="Node at 72ddfcd9-c408-849c-f04f-07e68de80a4b [Candidate]" term=2 -testLoggerAdapter verbose: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:04.911Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: election won: tally=1 - testing.go:149: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:04.911Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: election won: tally=1 -testLoggerAdapter verbose: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:04.911Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: entering leader state: leader="Node at 72ddfcd9-c408-849c-f04f-07e68de80a4b [Leader]" - testing.go:149: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:04.911Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: entering leader state: leader="Node at 72ddfcd9-c408-849c-f04f-07e68de80a4b [Leader]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:05.017Z [INFO] stable state for Leader reached at 2021-07-10 01:05:04.911998018 +0000 UTC m=+10.300156068 (1 nodes), 99.60725ms from start of poll, 99.965774ms from cluster start. Timeout at 2021-07-10 01:05:05.017103486 +0000 UTC m=+10.405261456, 105.105388ms after stability - testing.go:149: cluster: 2021-07-10T01:05:05.017Z [INFO] stable state for Leader reached at 2021-07-10 01:05:04.911998018 +0000 UTC m=+10.300156068 (1 nodes), 99.60725ms from start of poll, 99.965774ms from cluster start. Timeout at 2021-07-10 01:05:05.017103486 +0000 UTC m=+10.405261456, 105.105388ms after stability -testLoggerAdapter verbose: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:05.019Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: starting snapshot up to: index=102 - testing.go:149: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:05.019Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: starting snapshot up to: index=102 - testing.go:153: 2021-07-10T01:05:05.019Z [INFO] creating new snapshot: path=/tmp/raft700143239/snapshots/2-102-1625879105019.tmp -testLoggerAdapter verbose: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:05.020Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: compacting logs: from=1 to=92 - testing.go:149: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:05.020Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: compacting logs: from=1 to=92 -testLoggerAdapter verbose: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:05.020Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: snapshot complete up to: index=102 - testing.go:149: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:05.020Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: snapshot complete up to: index=102 -testLoggerAdapter verbose: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:05.020Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: restored from snapshot: id=2-102-1625879105019 - testing.go:149: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:05.020Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: restored from snapshot: id=2-102-1625879105019 -testLoggerAdapter verbose: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:05.021Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-72ddfcd9-c408-849c-f04f-07e68de80a4b Address:72ddfcd9-c408-849c-f04f-07e68de80a4b}]" - testing.go:149: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:05.021Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-72ddfcd9-c408-849c-f04f-07e68de80a4b Address:72ddfcd9-c408-849c-f04f-07e68de80a4b}]" ---- PASS: TestRaft_SnapshotRestore (0.21s) +testLoggerAdapter verbose: server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: 2022-08-12T07:30:09.303Z [INFO] server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258 Address:6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258}]" + testing.go:149: server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: 2022-08-12T07:30:09.303Z [INFO] server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258 Address:6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:09.303Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:09.303Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: 2022-08-12T07:30:09.303Z [INFO] server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: entering follower state: follower="Node at 6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258 [Follower]" leader= + testing.go:149: server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: 2022-08-12T07:30:09.303Z [INFO] server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: entering follower state: follower="Node at 6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258 [Follower]" leader= +testLoggerAdapter verbose: server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: 2022-08-12T07:30:09.354Z [WARN] server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: 2022-08-12T07:30:09.354Z [WARN] server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: 2022-08-12T07:30:09.354Z [INFO] server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: entering candidate state: node="Node at 6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258 [Candidate]" term=2 + testing.go:149: server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: 2022-08-12T07:30:09.354Z [INFO] server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: entering candidate state: node="Node at 6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258 [Candidate]" term=2 +testLoggerAdapter verbose: server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: 2022-08-12T07:30:09.354Z [INFO] server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: election won: tally=1 + testing.go:149: server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: 2022-08-12T07:30:09.354Z [INFO] server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: election won: tally=1 +testLoggerAdapter verbose: server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: 2022-08-12T07:30:09.354Z [INFO] server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: entering leader state: leader="Node at 6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258 [Leader]" + testing.go:149: server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: 2022-08-12T07:30:09.354Z [INFO] server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: entering leader state: leader="Node at 6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258 [Leader]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:09.459Z [INFO] stable state for Leader reached at 2022-08-12 07:30:09.354659 +0000 UTC m=+10.311345501 (1 nodes), 50.87ms from start of poll, 51.26272ms from cluster start. Timeout at 2022-08-12 07:30:09.45973534 +0000 UTC m=+10.416421841, 105.07634ms after stability + testing.go:149: cluster: 2022-08-12T07:30:09.459Z [INFO] stable state for Leader reached at 2022-08-12 07:30:09.354659 +0000 UTC m=+10.311345501 (1 nodes), 50.87ms from start of poll, 51.26272ms from cluster start. Timeout at 2022-08-12 07:30:09.45973534 +0000 UTC m=+10.416421841, 105.07634ms after stability +testLoggerAdapter verbose: server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: 2022-08-12T07:30:09.461Z [INFO] server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: starting snapshot up to: index=102 + testing.go:149: server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: 2022-08-12T07:30:09.461Z [INFO] server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: starting snapshot up to: index=102 + testing.go:153: 2022-08-12T07:30:09.461Z [INFO] creating new snapshot: path=/tmp/raft917709451/snapshots/2-102-1660289409461.tmp +testLoggerAdapter verbose: server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: 2022-08-12T07:30:09.462Z [INFO] server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: compacting logs: from=1 to=92 + testing.go:149: server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: 2022-08-12T07:30:09.462Z [INFO] server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: compacting logs: from=1 to=92 +testLoggerAdapter verbose: server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: 2022-08-12T07:30:09.462Z [INFO] server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: snapshot complete up to: index=102 + testing.go:149: server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: 2022-08-12T07:30:09.462Z [INFO] server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: snapshot complete up to: index=102 +testLoggerAdapter verbose: server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: 2022-08-12T07:30:09.463Z [INFO] server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: restored from snapshot: id=2-102-1660289409461 + testing.go:149: server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: 2022-08-12T07:30:09.463Z [INFO] server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: restored from snapshot: id=2-102-1660289409461 +testLoggerAdapter verbose: server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: 2022-08-12T07:30:09.463Z [INFO] server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258 Address:6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258}]" + testing.go:149: server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: 2022-08-12T07:30:09.463Z [INFO] server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258 Address:6f5696dc-ca5d-a209-8dbd-6d5dfbcc5258}]" +--- PASS: TestRaft_SnapshotRestore (0.16s) === RUN TestRaft_NoRestoreOnStart -testLoggerAdapter verbose: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.021Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1 Address:9de1fb4d-94f3-599a-7324-aa4df2d8ddf1}]" - testing.go:149: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.021Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1 Address:9de1fb4d-94f3-599a-7324-aa4df2d8ddf1}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:05.022Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:05.022Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.022Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: entering follower state: follower="Node at 9de1fb4d-94f3-599a-7324-aa4df2d8ddf1 [Follower]" leader= - testing.go:149: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.022Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: entering follower state: follower="Node at 9de1fb4d-94f3-599a-7324-aa4df2d8ddf1 [Follower]" leader= -testLoggerAdapter verbose: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.077Z [WARN] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.077Z [WARN] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.077Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: entering candidate state: node="Node at 9de1fb4d-94f3-599a-7324-aa4df2d8ddf1 [Candidate]" term=2 - testing.go:149: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.077Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: entering candidate state: node="Node at 9de1fb4d-94f3-599a-7324-aa4df2d8ddf1 [Candidate]" term=2 -testLoggerAdapter verbose: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.077Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: election won: tally=1 - testing.go:149: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.077Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: election won: tally=1 -testLoggerAdapter verbose: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.077Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: entering leader state: leader="Node at 9de1fb4d-94f3-599a-7324-aa4df2d8ddf1 [Leader]" - testing.go:149: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.077Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: entering leader state: leader="Node at 9de1fb4d-94f3-599a-7324-aa4df2d8ddf1 [Leader]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:05.183Z [INFO] stable state for Leader reached at 2021-07-10 01:05:05.077914387 +0000 UTC m=+10.466072397 (1 nodes), 55.796328ms from start of poll, 56.162772ms from cluster start. Timeout at 2021-07-10 01:05:05.183029495 +0000 UTC m=+10.571187465, 105.115068ms after stability - testing.go:149: cluster: 2021-07-10T01:05:05.183Z [INFO] stable state for Leader reached at 2021-07-10 01:05:05.077914387 +0000 UTC m=+10.466072397 (1 nodes), 55.796328ms from start of poll, 56.162772ms from cluster start. Timeout at 2021-07-10 01:05:05.183029495 +0000 UTC m=+10.571187465, 105.115068ms after stability -testLoggerAdapter verbose: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.185Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: starting snapshot up to: index=102 - testing.go:149: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.185Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: starting snapshot up to: index=102 - testing.go:153: 2021-07-10T01:05:05.185Z [INFO] creating new snapshot: path=/tmp/raft600398929/snapshots/2-102-1625879105185.tmp -testLoggerAdapter verbose: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.186Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: compacting logs: from=1 to=92 - testing.go:149: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.186Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: compacting logs: from=1 to=92 -testLoggerAdapter verbose: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.186Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: snapshot complete up to: index=102 - testing.go:149: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.186Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: snapshot complete up to: index=102 -testLoggerAdapter verbose: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.186Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1 Address:9de1fb4d-94f3-599a-7324-aa4df2d8ddf1}]" - testing.go:149: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.186Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1 Address:9de1fb4d-94f3-599a-7324-aa4df2d8ddf1}]" ---- PASS: TestRaft_NoRestoreOnStart (0.17s) +testLoggerAdapter verbose: server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: 2022-08-12T07:30:09.464Z [INFO] server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-df87334d-9be7-aa0f-ffa3-30090d3dd5da Address:df87334d-9be7-aa0f-ffa3-30090d3dd5da}]" + testing.go:149: server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: 2022-08-12T07:30:09.464Z [INFO] server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-df87334d-9be7-aa0f-ffa3-30090d3dd5da Address:df87334d-9be7-aa0f-ffa3-30090d3dd5da}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:09.464Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:09.464Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: 2022-08-12T07:30:09.464Z [INFO] server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: entering follower state: follower="Node at df87334d-9be7-aa0f-ffa3-30090d3dd5da [Follower]" leader= + testing.go:149: server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: 2022-08-12T07:30:09.464Z [INFO] server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: entering follower state: follower="Node at df87334d-9be7-aa0f-ffa3-30090d3dd5da [Follower]" leader= +testLoggerAdapter verbose: server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: 2022-08-12T07:30:09.552Z [WARN] server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: 2022-08-12T07:30:09.552Z [WARN] server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: 2022-08-12T07:30:09.552Z [INFO] server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: entering candidate state: node="Node at df87334d-9be7-aa0f-ffa3-30090d3dd5da [Candidate]" term=2 + testing.go:149: server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: 2022-08-12T07:30:09.552Z [INFO] server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: entering candidate state: node="Node at df87334d-9be7-aa0f-ffa3-30090d3dd5da [Candidate]" term=2 +testLoggerAdapter verbose: server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: 2022-08-12T07:30:09.552Z [INFO] server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: election won: tally=1 + testing.go:149: server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: 2022-08-12T07:30:09.552Z [INFO] server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: election won: tally=1 +testLoggerAdapter verbose: server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: 2022-08-12T07:30:09.552Z [INFO] server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: entering leader state: leader="Node at df87334d-9be7-aa0f-ffa3-30090d3dd5da [Leader]" + testing.go:149: server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: 2022-08-12T07:30:09.552Z [INFO] server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: entering leader state: leader="Node at df87334d-9be7-aa0f-ffa3-30090d3dd5da [Leader]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:09.657Z [INFO] stable state for Leader reached at 2022-08-12 07:30:09.5524907 +0000 UTC m=+10.509177181 (1 nodes), 88.23092ms from start of poll, 88.55386ms from cluster start. Timeout at 2022-08-12 07:30:09.65759856 +0000 UTC m=+10.614285061, 105.10788ms after stability + testing.go:149: cluster: 2022-08-12T07:30:09.657Z [INFO] stable state for Leader reached at 2022-08-12 07:30:09.5524907 +0000 UTC m=+10.509177181 (1 nodes), 88.23092ms from start of poll, 88.55386ms from cluster start. Timeout at 2022-08-12 07:30:09.65759856 +0000 UTC m=+10.614285061, 105.10788ms after stability +testLoggerAdapter verbose: server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: 2022-08-12T07:30:09.659Z [INFO] server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: starting snapshot up to: index=102 + testing.go:149: server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: 2022-08-12T07:30:09.659Z [INFO] server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: starting snapshot up to: index=102 + testing.go:153: 2022-08-12T07:30:09.659Z [INFO] creating new snapshot: path=/tmp/raft479431925/snapshots/2-102-1660289409659.tmp +testLoggerAdapter verbose: server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: 2022-08-12T07:30:09.660Z [INFO] server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: compacting logs: from=1 to=92 + testing.go:149: server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: 2022-08-12T07:30:09.660Z [INFO] server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: compacting logs: from=1 to=92 +testLoggerAdapter verbose: server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: 2022-08-12T07:30:09.660Z [INFO] server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: snapshot complete up to: index=102 + testing.go:149: server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: 2022-08-12T07:30:09.660Z [INFO] server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: snapshot complete up to: index=102 +testLoggerAdapter verbose: server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: 2022-08-12T07:30:09.661Z [INFO] server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-df87334d-9be7-aa0f-ffa3-30090d3dd5da Address:df87334d-9be7-aa0f-ffa3-30090d3dd5da}]" + testing.go:149: server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: 2022-08-12T07:30:09.661Z [INFO] server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-df87334d-9be7-aa0f-ffa3-30090d3dd5da Address:df87334d-9be7-aa0f-ffa3-30090d3dd5da}]" +--- PASS: TestRaft_NoRestoreOnStart (0.20s) === RUN TestRaft_SnapshotRestore_PeerChange raft_test.go:1026: Disabled unreliable test --- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s) +testLoggerAdapter verbose: server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: 2022-08-12T07:30:09.661Z [INFO] server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: entering follower state: follower="Node at df87334d-9be7-aa0f-ffa3-30090d3dd5da [Follower]" leader= === RUN TestRaft_AutoSnapshot raft_test.go:1132: Disabled unreliable test --- SKIP: TestRaft_AutoSnapshot (0.00s) === RUN TestRaft_UserSnapshot -testLoggerAdapter verbose: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.187Z [INFO] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5 Address:9f7bc916-22a0-854d-b0ec-12d138c3cdf5}]" - testing.go:149: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.187Z [INFO] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5 Address:9f7bc916-22a0-854d-b0ec-12d138c3cdf5}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:05.187Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:05.187Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.187Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: entering follower state: follower="Node at 9de1fb4d-94f3-599a-7324-aa4df2d8ddf1 [Follower]" leader= -testLoggerAdapter verbose: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.187Z [INFO] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: entering follower state: follower="Node at 9f7bc916-22a0-854d-b0ec-12d138c3cdf5 [Follower]" leader= - testing.go:149: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.187Z [INFO] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: entering follower state: follower="Node at 9f7bc916-22a0-854d-b0ec-12d138c3cdf5 [Follower]" leader= -testLoggerAdapter verbose: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.244Z [WARN] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.244Z [WARN] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.244Z [INFO] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: entering candidate state: node="Node at 9f7bc916-22a0-854d-b0ec-12d138c3cdf5 [Candidate]" term=2 - testing.go:149: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.244Z [INFO] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: entering candidate state: node="Node at 9f7bc916-22a0-854d-b0ec-12d138c3cdf5 [Candidate]" term=2 -testLoggerAdapter verbose: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.244Z [INFO] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: election won: tally=1 - testing.go:149: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.244Z [INFO] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: election won: tally=1 -testLoggerAdapter verbose: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.244Z [INFO] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: entering leader state: leader="Node at 9f7bc916-22a0-854d-b0ec-12d138c3cdf5 [Leader]" - testing.go:149: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.244Z [INFO] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: entering leader state: leader="Node at 9f7bc916-22a0-854d-b0ec-12d138c3cdf5 [Leader]" -testLoggerAdapter verbose: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.250Z [WARN] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.251Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: entering candidate state: node="Node at 9de1fb4d-94f3-599a-7324-aa4df2d8ddf1 [Candidate]" term=3 -testLoggerAdapter verbose: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.251Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: election won: tally=1 -testLoggerAdapter verbose: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.251Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: entering leader state: leader="Node at 9de1fb4d-94f3-599a-7324-aa4df2d8ddf1 [Leader]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:05.349Z [INFO] stable state for Leader reached at 2021-07-10 01:05:05.244475743 +0000 UTC m=+10.632633733 (1 nodes), 56.769659ms from start of poll, 57.178683ms from cluster start. Timeout at 2021-07-10 01:05:05.349577791 +0000 UTC m=+10.737735761, 105.102028ms after stability - testing.go:149: cluster: 2021-07-10T01:05:05.349Z [INFO] stable state for Leader reached at 2021-07-10 01:05:05.244475743 +0000 UTC m=+10.632633733 (1 nodes), 56.769659ms from start of poll, 57.178683ms from cluster start. Timeout at 2021-07-10 01:05:05.349577791 +0000 UTC m=+10.737735761, 105.102028ms after stability -testLoggerAdapter verbose: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.350Z [ERROR] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: failed to take snapshot: error="nothing new to snapshot" - testing.go:149: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.350Z [ERROR] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: failed to take snapshot: error="nothing new to snapshot" -testLoggerAdapter verbose: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.350Z [INFO] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: starting snapshot up to: index=12 - testing.go:149: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.350Z [INFO] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: starting snapshot up to: index=12 - testing.go:153: 2021-07-10T01:05:05.350Z [INFO] creating new snapshot: path=/tmp/raft874132139/snapshots/2-12-1625879105350.tmp -testLoggerAdapter verbose: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.351Z [INFO] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: compacting logs: from=1 to=2 - testing.go:149: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.351Z [INFO] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: compacting logs: from=1 to=2 -testLoggerAdapter verbose: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.351Z [INFO] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: snapshot complete up to: index=12 - testing.go:149: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.351Z [INFO] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: snapshot complete up to: index=12 ---- PASS: TestRaft_UserSnapshot (0.16s) +testLoggerAdapter verbose: server-4b02957e-6fce-23f6-4f96-f6c27974aa98: 2022-08-12T07:30:09.663Z [INFO] server-4b02957e-6fce-23f6-4f96-f6c27974aa98: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b02957e-6fce-23f6-4f96-f6c27974aa98 Address:4b02957e-6fce-23f6-4f96-f6c27974aa98}]" + testing.go:149: server-4b02957e-6fce-23f6-4f96-f6c27974aa98: 2022-08-12T07:30:09.663Z [INFO] server-4b02957e-6fce-23f6-4f96-f6c27974aa98: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b02957e-6fce-23f6-4f96-f6c27974aa98 Address:4b02957e-6fce-23f6-4f96-f6c27974aa98}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:09.663Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:09.663Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-4b02957e-6fce-23f6-4f96-f6c27974aa98: 2022-08-12T07:30:09.663Z [INFO] server-4b02957e-6fce-23f6-4f96-f6c27974aa98: entering follower state: follower="Node at 4b02957e-6fce-23f6-4f96-f6c27974aa98 [Follower]" leader= + testing.go:149: server-4b02957e-6fce-23f6-4f96-f6c27974aa98: 2022-08-12T07:30:09.663Z [INFO] server-4b02957e-6fce-23f6-4f96-f6c27974aa98: entering follower state: follower="Node at 4b02957e-6fce-23f6-4f96-f6c27974aa98 [Follower]" leader= +testLoggerAdapter verbose: server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: 2022-08-12T07:30:09.719Z [WARN] server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: 2022-08-12T07:30:09.719Z [INFO] server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: entering candidate state: node="Node at df87334d-9be7-aa0f-ffa3-30090d3dd5da [Candidate]" term=3 +testLoggerAdapter verbose: server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: 2022-08-12T07:30:09.719Z [INFO] server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: election won: tally=1 +testLoggerAdapter verbose: server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: 2022-08-12T07:30:09.719Z [INFO] server-df87334d-9be7-aa0f-ffa3-30090d3dd5da: entering leader state: leader="Node at df87334d-9be7-aa0f-ffa3-30090d3dd5da [Leader]" +testLoggerAdapter verbose: server-4b02957e-6fce-23f6-4f96-f6c27974aa98: 2022-08-12T07:30:09.727Z [WARN] server-4b02957e-6fce-23f6-4f96-f6c27974aa98: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-4b02957e-6fce-23f6-4f96-f6c27974aa98: 2022-08-12T07:30:09.727Z [WARN] server-4b02957e-6fce-23f6-4f96-f6c27974aa98: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-4b02957e-6fce-23f6-4f96-f6c27974aa98: 2022-08-12T07:30:09.727Z [INFO] server-4b02957e-6fce-23f6-4f96-f6c27974aa98: entering candidate state: node="Node at 4b02957e-6fce-23f6-4f96-f6c27974aa98 [Candidate]" term=2 + testing.go:149: server-4b02957e-6fce-23f6-4f96-f6c27974aa98: 2022-08-12T07:30:09.727Z [INFO] server-4b02957e-6fce-23f6-4f96-f6c27974aa98: entering candidate state: node="Node at 4b02957e-6fce-23f6-4f96-f6c27974aa98 [Candidate]" term=2 +testLoggerAdapter verbose: server-4b02957e-6fce-23f6-4f96-f6c27974aa98: 2022-08-12T07:30:09.727Z [INFO] server-4b02957e-6fce-23f6-4f96-f6c27974aa98: election won: tally=1 + testing.go:149: server-4b02957e-6fce-23f6-4f96-f6c27974aa98: 2022-08-12T07:30:09.727Z [INFO] server-4b02957e-6fce-23f6-4f96-f6c27974aa98: election won: tally=1 +testLoggerAdapter verbose: server-4b02957e-6fce-23f6-4f96-f6c27974aa98: 2022-08-12T07:30:09.727Z [INFO] server-4b02957e-6fce-23f6-4f96-f6c27974aa98: entering leader state: leader="Node at 4b02957e-6fce-23f6-4f96-f6c27974aa98 [Leader]" + testing.go:149: server-4b02957e-6fce-23f6-4f96-f6c27974aa98: 2022-08-12T07:30:09.727Z [INFO] server-4b02957e-6fce-23f6-4f96-f6c27974aa98: entering leader state: leader="Node at 4b02957e-6fce-23f6-4f96-f6c27974aa98 [Leader]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:09.833Z [INFO] stable state for Leader reached at 2022-08-12 07:30:09.72785428 +0000 UTC m=+10.684540781 (1 nodes), 64.6524ms from start of poll, 65.73514ms from cluster start. Timeout at 2022-08-12 07:30:09.8329379 +0000 UTC m=+10.789624381, 105.0836ms after stability + testing.go:149: cluster: 2022-08-12T07:30:09.833Z [INFO] stable state for Leader reached at 2022-08-12 07:30:09.72785428 +0000 UTC m=+10.684540781 (1 nodes), 64.6524ms from start of poll, 65.73514ms from cluster start. Timeout at 2022-08-12 07:30:09.8329379 +0000 UTC m=+10.789624381, 105.0836ms after stability +testLoggerAdapter verbose: server-4b02957e-6fce-23f6-4f96-f6c27974aa98: 2022-08-12T07:30:09.833Z [ERROR] server-4b02957e-6fce-23f6-4f96-f6c27974aa98: failed to take snapshot: error="nothing new to snapshot" + testing.go:149: server-4b02957e-6fce-23f6-4f96-f6c27974aa98: 2022-08-12T07:30:09.833Z [ERROR] server-4b02957e-6fce-23f6-4f96-f6c27974aa98: failed to take snapshot: error="nothing new to snapshot" +testLoggerAdapter verbose: server-4b02957e-6fce-23f6-4f96-f6c27974aa98: 2022-08-12T07:30:09.833Z [INFO] server-4b02957e-6fce-23f6-4f96-f6c27974aa98: starting snapshot up to: index=12 + testing.go:149: server-4b02957e-6fce-23f6-4f96-f6c27974aa98: 2022-08-12T07:30:09.833Z [INFO] server-4b02957e-6fce-23f6-4f96-f6c27974aa98: starting snapshot up to: index=12 + testing.go:153: 2022-08-12T07:30:09.833Z [INFO] creating new snapshot: path=/tmp/raft169393391/snapshots/2-12-1660289409833.tmp +testLoggerAdapter verbose: server-4b02957e-6fce-23f6-4f96-f6c27974aa98: 2022-08-12T07:30:09.834Z [INFO] server-4b02957e-6fce-23f6-4f96-f6c27974aa98: compacting logs: from=1 to=2 + testing.go:149: server-4b02957e-6fce-23f6-4f96-f6c27974aa98: 2022-08-12T07:30:09.834Z [INFO] server-4b02957e-6fce-23f6-4f96-f6c27974aa98: compacting logs: from=1 to=2 +testLoggerAdapter verbose: server-4b02957e-6fce-23f6-4f96-f6c27974aa98: 2022-08-12T07:30:09.834Z [INFO] server-4b02957e-6fce-23f6-4f96-f6c27974aa98: snapshot complete up to: index=12 + testing.go:149: server-4b02957e-6fce-23f6-4f96-f6c27974aa98: 2022-08-12T07:30:09.834Z [INFO] server-4b02957e-6fce-23f6-4f96-f6c27974aa98: snapshot complete up to: index=12 +--- PASS: TestRaft_UserSnapshot (0.17s) === RUN TestRaft_UserRestore -testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.352Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3d894380-81d6-593f-a2b3-516cefadd7e7 Address:3d894380-81d6-593f-a2b3-516cefadd7e7} {Suffrage:Voter ID:server-e818da0e-30b8-095b-47df-8156b2526404 Address:e818da0e-30b8-095b-47df-8156b2526404} {Suffrage:Voter ID:server-e9b07343-5cdf-0560-bed6-36501b160906 Address:e9b07343-5cdf-0560-bed6-36501b160906}]" - testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.352Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3d894380-81d6-593f-a2b3-516cefadd7e7 Address:3d894380-81d6-593f-a2b3-516cefadd7e7} {Suffrage:Voter ID:server-e818da0e-30b8-095b-47df-8156b2526404 Address:e818da0e-30b8-095b-47df-8156b2526404} {Suffrage:Voter ID:server-e9b07343-5cdf-0560-bed6-36501b160906 Address:e9b07343-5cdf-0560-bed6-36501b160906}]" -testLoggerAdapter verbose: server-e818da0e-30b8-095b-47df-8156b2526404: 2021-07-10T01:05:05.353Z [INFO] server-e818da0e-30b8-095b-47df-8156b2526404: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3d894380-81d6-593f-a2b3-516cefadd7e7 Address:3d894380-81d6-593f-a2b3-516cefadd7e7} {Suffrage:Voter ID:server-e818da0e-30b8-095b-47df-8156b2526404 Address:e818da0e-30b8-095b-47df-8156b2526404} {Suffrage:Voter ID:server-e9b07343-5cdf-0560-bed6-36501b160906 Address:e9b07343-5cdf-0560-bed6-36501b160906}]" - testing.go:149: server-e818da0e-30b8-095b-47df-8156b2526404: 2021-07-10T01:05:05.353Z [INFO] server-e818da0e-30b8-095b-47df-8156b2526404: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3d894380-81d6-593f-a2b3-516cefadd7e7 Address:3d894380-81d6-593f-a2b3-516cefadd7e7} {Suffrage:Voter ID:server-e818da0e-30b8-095b-47df-8156b2526404 Address:e818da0e-30b8-095b-47df-8156b2526404} {Suffrage:Voter ID:server-e9b07343-5cdf-0560-bed6-36501b160906 Address:e9b07343-5cdf-0560-bed6-36501b160906}]" -testLoggerAdapter verbose: server-e9b07343-5cdf-0560-bed6-36501b160906: 2021-07-10T01:05:05.353Z [INFO] server-e9b07343-5cdf-0560-bed6-36501b160906: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3d894380-81d6-593f-a2b3-516cefadd7e7 Address:3d894380-81d6-593f-a2b3-516cefadd7e7} {Suffrage:Voter ID:server-e818da0e-30b8-095b-47df-8156b2526404 Address:e818da0e-30b8-095b-47df-8156b2526404} {Suffrage:Voter ID:server-e9b07343-5cdf-0560-bed6-36501b160906 Address:e9b07343-5cdf-0560-bed6-36501b160906}]" - testing.go:149: server-e9b07343-5cdf-0560-bed6-36501b160906: 2021-07-10T01:05:05.353Z [INFO] server-e9b07343-5cdf-0560-bed6-36501b160906: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3d894380-81d6-593f-a2b3-516cefadd7e7 Address:3d894380-81d6-593f-a2b3-516cefadd7e7} {Suffrage:Voter ID:server-e818da0e-30b8-095b-47df-8156b2526404 Address:e818da0e-30b8-095b-47df-8156b2526404} {Suffrage:Voter ID:server-e9b07343-5cdf-0560-bed6-36501b160906 Address:e9b07343-5cdf-0560-bed6-36501b160906}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:05.353Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:05.353Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.353Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: entering follower state: follower="Node at 3d894380-81d6-593f-a2b3-516cefadd7e7 [Follower]" leader= - testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.353Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: entering follower state: follower="Node at 3d894380-81d6-593f-a2b3-516cefadd7e7 [Follower]" leader= -testLoggerAdapter verbose: server-e818da0e-30b8-095b-47df-8156b2526404: 2021-07-10T01:05:05.353Z [INFO] server-e818da0e-30b8-095b-47df-8156b2526404: entering follower state: follower="Node at e818da0e-30b8-095b-47df-8156b2526404 [Follower]" leader= - testing.go:149: server-e818da0e-30b8-095b-47df-8156b2526404: 2021-07-10T01:05:05.353Z [INFO] server-e818da0e-30b8-095b-47df-8156b2526404: entering follower state: follower="Node at e818da0e-30b8-095b-47df-8156b2526404 [Follower]" leader= -testLoggerAdapter verbose: server-e9b07343-5cdf-0560-bed6-36501b160906: 2021-07-10T01:05:05.353Z [INFO] server-e9b07343-5cdf-0560-bed6-36501b160906: entering follower state: follower="Node at e9b07343-5cdf-0560-bed6-36501b160906 [Follower]" leader= - testing.go:149: server-e9b07343-5cdf-0560-bed6-36501b160906: 2021-07-10T01:05:05.353Z [INFO] server-e9b07343-5cdf-0560-bed6-36501b160906: entering follower state: follower="Node at e9b07343-5cdf-0560-bed6-36501b160906 [Follower]" leader= -testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.438Z [WARN] server-3d894380-81d6-593f-a2b3-516cefadd7e7: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.438Z [WARN] server-3d894380-81d6-593f-a2b3-516cefadd7e7: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.439Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: entering candidate state: node="Node at 3d894380-81d6-593f-a2b3-516cefadd7e7 [Candidate]" term=2 - testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.439Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: entering candidate state: node="Node at 3d894380-81d6-593f-a2b3-516cefadd7e7 [Candidate]" term=2 -testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.439Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: election won: tally=2 - testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.439Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: election won: tally=2 -testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.439Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: entering leader state: leader="Node at 3d894380-81d6-593f-a2b3-516cefadd7e7 [Leader]" - testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.439Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: entering leader state: leader="Node at 3d894380-81d6-593f-a2b3-516cefadd7e7 [Leader]" -testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.439Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: added peer, starting replication: peer=server-e818da0e-30b8-095b-47df-8156b2526404 - testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.439Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: added peer, starting replication: peer=server-e818da0e-30b8-095b-47df-8156b2526404 -testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.439Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: added peer, starting replication: peer=server-e9b07343-5cdf-0560-bed6-36501b160906 - testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.439Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: added peer, starting replication: peer=server-e9b07343-5cdf-0560-bed6-36501b160906 -testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.439Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: pipelining replication: peer="{Voter server-e9b07343-5cdf-0560-bed6-36501b160906 e9b07343-5cdf-0560-bed6-36501b160906}" - testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.439Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: pipelining replication: peer="{Voter server-e9b07343-5cdf-0560-bed6-36501b160906 e9b07343-5cdf-0560-bed6-36501b160906}" -testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.439Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: pipelining replication: peer="{Voter server-e818da0e-30b8-095b-47df-8156b2526404 e818da0e-30b8-095b-47df-8156b2526404}" - testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.439Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: pipelining replication: peer="{Voter server-e818da0e-30b8-095b-47df-8156b2526404 e818da0e-30b8-095b-47df-8156b2526404}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:05.544Z [INFO] stable state for Leader reached at 2021-07-10 01:05:05.43957486 +0000 UTC m=+10.827732850 (1 nodes), 85.932266ms from start of poll, 86.935197ms from cluster start. Timeout at 2021-07-10 01:05:05.544666968 +0000 UTC m=+10.932824938, 105.092088ms after stability - testing.go:149: cluster: 2021-07-10T01:05:05.544Z [INFO] stable state for Leader reached at 2021-07-10 01:05:05.43957486 +0000 UTC m=+10.827732850 (1 nodes), 85.932266ms from start of poll, 86.935197ms from cluster start. Timeout at 2021-07-10 01:05:05.544666968 +0000 UTC m=+10.932824938, 105.092088ms after stability -testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.545Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: starting snapshot up to: index=12 - testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.545Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: starting snapshot up to: index=12 - testing.go:153: 2021-07-10T01:05:05.545Z [INFO] creating new snapshot: path=/tmp/raft110042645/snapshots/2-12-1625879105545.tmp -testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.546Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: snapshot complete up to: index=12 - testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.546Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: snapshot complete up to: index=12 - testing.go:153: 2021-07-10T01:05:05.547Z [INFO] creating new snapshot: path=/tmp/raft110042645/snapshots/2-23-1625879105547.tmp -testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.548Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: copied to local snapshot: bytes=71 - testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.548Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.549Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: restored user snapshot: index=1 - testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.549Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: restored user snapshot: index=1 -testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.549Z [ERROR] server-3d894380-81d6-593f-a2b3-516cefadd7e7: failed to get log: index=23 error="log not found" - testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.549Z [ERROR] server-3d894380-81d6-593f-a2b3-516cefadd7e7: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.549Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: aborting pipeline replication: peer="{Voter server-e818da0e-30b8-095b-47df-8156b2526404 e818da0e-30b8-095b-47df-8156b2526404}" - testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.549Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: aborting pipeline replication: peer="{Voter server-e818da0e-30b8-095b-47df-8156b2526404 e818da0e-30b8-095b-47df-8156b2526404}" -testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.549Z [ERROR] server-3d894380-81d6-593f-a2b3-516cefadd7e7: failed to get log: index=23 error="log not found" - testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.549Z [ERROR] server-3d894380-81d6-593f-a2b3-516cefadd7e7: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.549Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: aborting pipeline replication: peer="{Voter server-e9b07343-5cdf-0560-bed6-36501b160906 e9b07343-5cdf-0560-bed6-36501b160906}" - testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.549Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: aborting pipeline replication: peer="{Voter server-e9b07343-5cdf-0560-bed6-36501b160906 e9b07343-5cdf-0560-bed6-36501b160906}" -testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.557Z [ERROR] server-3d894380-81d6-593f-a2b3-516cefadd7e7: failed to get log: index=23 error="log not found" - testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.557Z [ERROR] server-3d894380-81d6-593f-a2b3-516cefadd7e7: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.558Z [ERROR] server-3d894380-81d6-593f-a2b3-516cefadd7e7: failed to get log: index=23 error="log not found" - testing.go:153: 2021-07-10T01:05:05.558Z [INFO] creating new snapshot: path=/tmp/raft184352015/snapshots/2-23-1625879105558.tmp - testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.558Z [ERROR] server-3d894380-81d6-593f-a2b3-516cefadd7e7: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-e818da0e-30b8-095b-47df-8156b2526404: 2021-07-10T01:05:05.558Z [INFO] server-e818da0e-30b8-095b-47df-8156b2526404: copied to local snapshot: bytes=71 - testing.go:153: 2021-07-10T01:05:05.558Z [INFO] creating new snapshot: path=/tmp/raft065387545/snapshots/2-23-1625879105558.tmp - testing.go:149: server-e818da0e-30b8-095b-47df-8156b2526404: 2021-07-10T01:05:05.558Z [INFO] server-e818da0e-30b8-095b-47df-8156b2526404: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-e818da0e-30b8-095b-47df-8156b2526404: 2021-07-10T01:05:05.559Z [INFO] server-e818da0e-30b8-095b-47df-8156b2526404: Installed remote snapshot - testing.go:149: server-e818da0e-30b8-095b-47df-8156b2526404: 2021-07-10T01:05:05.559Z [INFO] server-e818da0e-30b8-095b-47df-8156b2526404: Installed remote snapshot -testLoggerAdapter verbose: server-e818da0e-30b8-095b-47df-8156b2526404: 2021-07-10T01:05:05.559Z [WARN] server-e818da0e-30b8-095b-47df-8156b2526404: skipping application of old log: index=22 - testing.go:149: server-e818da0e-30b8-095b-47df-8156b2526404: 2021-07-10T01:05:05.559Z [WARN] server-e818da0e-30b8-095b-47df-8156b2526404: skipping application of old log: index=22 -testLoggerAdapter verbose: server-e9b07343-5cdf-0560-bed6-36501b160906: 2021-07-10T01:05:05.559Z [INFO] server-e9b07343-5cdf-0560-bed6-36501b160906: copied to local snapshot: bytes=71 - testing.go:149: server-e9b07343-5cdf-0560-bed6-36501b160906: 2021-07-10T01:05:05.559Z [INFO] server-e9b07343-5cdf-0560-bed6-36501b160906: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.559Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: pipelining replication: peer="{Voter server-e818da0e-30b8-095b-47df-8156b2526404 e818da0e-30b8-095b-47df-8156b2526404}" - testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.559Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: pipelining replication: peer="{Voter server-e818da0e-30b8-095b-47df-8156b2526404 e818da0e-30b8-095b-47df-8156b2526404}" -testLoggerAdapter verbose: server-e9b07343-5cdf-0560-bed6-36501b160906: 2021-07-10T01:05:05.559Z [INFO] server-e9b07343-5cdf-0560-bed6-36501b160906: Installed remote snapshot - testing.go:149: server-e9b07343-5cdf-0560-bed6-36501b160906: 2021-07-10T01:05:05.559Z [INFO] server-e9b07343-5cdf-0560-bed6-36501b160906: Installed remote snapshot -testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.559Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: pipelining replication: peer="{Voter server-e9b07343-5cdf-0560-bed6-36501b160906 e9b07343-5cdf-0560-bed6-36501b160906}" - testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.559Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: pipelining replication: peer="{Voter server-e9b07343-5cdf-0560-bed6-36501b160906 e9b07343-5cdf-0560-bed6-36501b160906}" -testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.575Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: aborting pipeline replication: peer="{Voter server-e818da0e-30b8-095b-47df-8156b2526404 e818da0e-30b8-095b-47df-8156b2526404}" - testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.575Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: aborting pipeline replication: peer="{Voter server-e818da0e-30b8-095b-47df-8156b2526404 e818da0e-30b8-095b-47df-8156b2526404}" -testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.575Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: aborting pipeline replication: peer="{Voter server-e9b07343-5cdf-0560-bed6-36501b160906 e9b07343-5cdf-0560-bed6-36501b160906}" - testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.575Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: aborting pipeline replication: peer="{Voter server-e9b07343-5cdf-0560-bed6-36501b160906 e9b07343-5cdf-0560-bed6-36501b160906}" -testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.577Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5df398f-3349-4628-2f92-4e243963cc61 Address:d5df398f-3349-4628-2f92-4e243963cc61} {Suffrage:Voter ID:server-6a233ea6-9f29-cb26-8f90-22b24de85079 Address:6a233ea6-9f29-cb26-8f90-22b24de85079} {Suffrage:Voter ID:server-4b63ed8c-a916-e473-8b4c-baa46c99b91b Address:4b63ed8c-a916-e473-8b4c-baa46c99b91b}]" - testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.577Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5df398f-3349-4628-2f92-4e243963cc61 Address:d5df398f-3349-4628-2f92-4e243963cc61} {Suffrage:Voter ID:server-6a233ea6-9f29-cb26-8f90-22b24de85079 Address:6a233ea6-9f29-cb26-8f90-22b24de85079} {Suffrage:Voter ID:server-4b63ed8c-a916-e473-8b4c-baa46c99b91b Address:4b63ed8c-a916-e473-8b4c-baa46c99b91b}]" -testLoggerAdapter verbose: server-6a233ea6-9f29-cb26-8f90-22b24de85079: 2021-07-10T01:05:05.577Z [INFO] server-6a233ea6-9f29-cb26-8f90-22b24de85079: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5df398f-3349-4628-2f92-4e243963cc61 Address:d5df398f-3349-4628-2f92-4e243963cc61} {Suffrage:Voter ID:server-6a233ea6-9f29-cb26-8f90-22b24de85079 Address:6a233ea6-9f29-cb26-8f90-22b24de85079} {Suffrage:Voter ID:server-4b63ed8c-a916-e473-8b4c-baa46c99b91b Address:4b63ed8c-a916-e473-8b4c-baa46c99b91b}]" - testing.go:149: server-6a233ea6-9f29-cb26-8f90-22b24de85079: 2021-07-10T01:05:05.577Z [INFO] server-6a233ea6-9f29-cb26-8f90-22b24de85079: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5df398f-3349-4628-2f92-4e243963cc61 Address:d5df398f-3349-4628-2f92-4e243963cc61} {Suffrage:Voter ID:server-6a233ea6-9f29-cb26-8f90-22b24de85079 Address:6a233ea6-9f29-cb26-8f90-22b24de85079} {Suffrage:Voter ID:server-4b63ed8c-a916-e473-8b4c-baa46c99b91b Address:4b63ed8c-a916-e473-8b4c-baa46c99b91b}]" -testLoggerAdapter verbose: server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: 2021-07-10T01:05:05.577Z [INFO] server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5df398f-3349-4628-2f92-4e243963cc61 Address:d5df398f-3349-4628-2f92-4e243963cc61} {Suffrage:Voter ID:server-6a233ea6-9f29-cb26-8f90-22b24de85079 Address:6a233ea6-9f29-cb26-8f90-22b24de85079} {Suffrage:Voter ID:server-4b63ed8c-a916-e473-8b4c-baa46c99b91b Address:4b63ed8c-a916-e473-8b4c-baa46c99b91b}]" - testing.go:149: server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: 2021-07-10T01:05:05.577Z [INFO] server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5df398f-3349-4628-2f92-4e243963cc61 Address:d5df398f-3349-4628-2f92-4e243963cc61} {Suffrage:Voter ID:server-6a233ea6-9f29-cb26-8f90-22b24de85079 Address:6a233ea6-9f29-cb26-8f90-22b24de85079} {Suffrage:Voter ID:server-4b63ed8c-a916-e473-8b4c-baa46c99b91b Address:4b63ed8c-a916-e473-8b4c-baa46c99b91b}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:05.577Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:05.577Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-6a233ea6-9f29-cb26-8f90-22b24de85079: 2021-07-10T01:05:05.577Z [INFO] server-6a233ea6-9f29-cb26-8f90-22b24de85079: entering follower state: follower="Node at 6a233ea6-9f29-cb26-8f90-22b24de85079 [Follower]" leader= -testLoggerAdapter verbose: server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: 2021-07-10T01:05:05.577Z [INFO] server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: entering follower state: follower="Node at 4b63ed8c-a916-e473-8b4c-baa46c99b91b [Follower]" leader= -testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.577Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: entering follower state: follower="Node at d5df398f-3349-4628-2f92-4e243963cc61 [Follower]" leader= - testing.go:149: server-6a233ea6-9f29-cb26-8f90-22b24de85079: 2021-07-10T01:05:05.577Z [INFO] server-6a233ea6-9f29-cb26-8f90-22b24de85079: entering follower state: follower="Node at 6a233ea6-9f29-cb26-8f90-22b24de85079 [Follower]" leader= - testing.go:149: server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: 2021-07-10T01:05:05.577Z [INFO] server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: entering follower state: follower="Node at 4b63ed8c-a916-e473-8b4c-baa46c99b91b [Follower]" leader= - testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.577Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: entering follower state: follower="Node at d5df398f-3349-4628-2f92-4e243963cc61 [Follower]" leader= -testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.629Z [WARN] server-d5df398f-3349-4628-2f92-4e243963cc61: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.629Z [WARN] server-d5df398f-3349-4628-2f92-4e243963cc61: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.629Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: entering candidate state: node="Node at d5df398f-3349-4628-2f92-4e243963cc61 [Candidate]" term=2 - testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.629Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: entering candidate state: node="Node at d5df398f-3349-4628-2f92-4e243963cc61 [Candidate]" term=2 -testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.629Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: election won: tally=2 - testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.629Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: election won: tally=2 -testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.629Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: entering leader state: leader="Node at d5df398f-3349-4628-2f92-4e243963cc61 [Leader]" - testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.629Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: entering leader state: leader="Node at d5df398f-3349-4628-2f92-4e243963cc61 [Leader]" -testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.629Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: added peer, starting replication: peer=server-6a233ea6-9f29-cb26-8f90-22b24de85079 - testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.629Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: added peer, starting replication: peer=server-6a233ea6-9f29-cb26-8f90-22b24de85079 -testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.629Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: added peer, starting replication: peer=server-4b63ed8c-a916-e473-8b4c-baa46c99b91b - testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.629Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: added peer, starting replication: peer=server-4b63ed8c-a916-e473-8b4c-baa46c99b91b -testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.629Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: pipelining replication: peer="{Voter server-4b63ed8c-a916-e473-8b4c-baa46c99b91b 4b63ed8c-a916-e473-8b4c-baa46c99b91b}" - testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.629Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: pipelining replication: peer="{Voter server-4b63ed8c-a916-e473-8b4c-baa46c99b91b 4b63ed8c-a916-e473-8b4c-baa46c99b91b}" -testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.629Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: pipelining replication: peer="{Voter server-6a233ea6-9f29-cb26-8f90-22b24de85079 6a233ea6-9f29-cb26-8f90-22b24de85079}" - testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.629Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: pipelining replication: peer="{Voter server-6a233ea6-9f29-cb26-8f90-22b24de85079 6a233ea6-9f29-cb26-8f90-22b24de85079}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:05.734Z [INFO] stable state for Leader reached at 2021-07-10 01:05:05.629807685 +0000 UTC m=+11.017965655 (1 nodes), 52.079169ms from start of poll, 52.951478ms from cluster start. Timeout at 2021-07-10 01:05:05.734879353 +0000 UTC m=+11.123037303, 105.071648ms after stability - testing.go:149: cluster: 2021-07-10T01:05:05.734Z [INFO] stable state for Leader reached at 2021-07-10 01:05:05.629807685 +0000 UTC m=+11.017965655 (1 nodes), 52.079169ms from start of poll, 52.951478ms from cluster start. Timeout at 2021-07-10 01:05:05.734879353 +0000 UTC m=+11.123037303, 105.071648ms after stability -testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.735Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: starting snapshot up to: index=12 - testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.735Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: starting snapshot up to: index=12 - testing.go:153: 2021-07-10T01:05:05.735Z [INFO] creating new snapshot: path=/tmp/raft563676595/snapshots/2-12-1625879105735.tmp -testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.736Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: snapshot complete up to: index=12 - testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.736Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: snapshot complete up to: index=12 - testing.go:153: 2021-07-10T01:05:05.736Z [INFO] creating new snapshot: path=/tmp/raft563676595/snapshots/2-23-1625879105736.tmp -testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.737Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: copied to local snapshot: bytes=71 - testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.737Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.738Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: restored user snapshot: index=1 - testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.738Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: restored user snapshot: index=1 -testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.738Z [ERROR] server-d5df398f-3349-4628-2f92-4e243963cc61: failed to get log: index=23 error="log not found" - testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.738Z [ERROR] server-d5df398f-3349-4628-2f92-4e243963cc61: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.738Z [ERROR] server-d5df398f-3349-4628-2f92-4e243963cc61: failed to get log: index=23 error="log not found" - testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.738Z [ERROR] server-d5df398f-3349-4628-2f92-4e243963cc61: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.738Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: aborting pipeline replication: peer="{Voter server-4b63ed8c-a916-e473-8b4c-baa46c99b91b 4b63ed8c-a916-e473-8b4c-baa46c99b91b}" - testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.738Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: aborting pipeline replication: peer="{Voter server-4b63ed8c-a916-e473-8b4c-baa46c99b91b 4b63ed8c-a916-e473-8b4c-baa46c99b91b}" -testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.738Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: aborting pipeline replication: peer="{Voter server-6a233ea6-9f29-cb26-8f90-22b24de85079 6a233ea6-9f29-cb26-8f90-22b24de85079}" - testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.738Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: aborting pipeline replication: peer="{Voter server-6a233ea6-9f29-cb26-8f90-22b24de85079 6a233ea6-9f29-cb26-8f90-22b24de85079}" -testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.743Z [ERROR] server-d5df398f-3349-4628-2f92-4e243963cc61: failed to get log: index=23 error="log not found" - testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.743Z [ERROR] server-d5df398f-3349-4628-2f92-4e243963cc61: failed to get log: index=23 error="log not found" - testing.go:153: 2021-07-10T01:05:05.744Z [INFO] creating new snapshot: path=/tmp/raft278181015/snapshots/2-23-1625879105744.tmp -testLoggerAdapter verbose: server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: 2021-07-10T01:05:05.744Z [INFO] server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: copied to local snapshot: bytes=71 - testing.go:149: server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: 2021-07-10T01:05:05.744Z [INFO] server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: 2021-07-10T01:05:05.745Z [INFO] server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: Installed remote snapshot - testing.go:149: server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: 2021-07-10T01:05:05.745Z [INFO] server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: Installed remote snapshot -testLoggerAdapter verbose: server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: 2021-07-10T01:05:05.745Z [WARN] server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: skipping application of old log: index=22 - testing.go:149: server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: 2021-07-10T01:05:05.745Z [WARN] server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: skipping application of old log: index=22 -testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.745Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: pipelining replication: peer="{Voter server-4b63ed8c-a916-e473-8b4c-baa46c99b91b 4b63ed8c-a916-e473-8b4c-baa46c99b91b}" - testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.745Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: pipelining replication: peer="{Voter server-4b63ed8c-a916-e473-8b4c-baa46c99b91b 4b63ed8c-a916-e473-8b4c-baa46c99b91b}" -testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.746Z [ERROR] server-d5df398f-3349-4628-2f92-4e243963cc61: failed to get log: index=23 error="log not found" - testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.746Z [ERROR] server-d5df398f-3349-4628-2f92-4e243963cc61: failed to get log: index=23 error="log not found" - testing.go:153: 2021-07-10T01:05:05.747Z [INFO] creating new snapshot: path=/tmp/raft347660893/snapshots/2-23-1625879105747.tmp -testLoggerAdapter verbose: server-6a233ea6-9f29-cb26-8f90-22b24de85079: 2021-07-10T01:05:05.747Z [INFO] server-6a233ea6-9f29-cb26-8f90-22b24de85079: copied to local snapshot: bytes=71 - testing.go:149: server-6a233ea6-9f29-cb26-8f90-22b24de85079: 2021-07-10T01:05:05.747Z [INFO] server-6a233ea6-9f29-cb26-8f90-22b24de85079: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-6a233ea6-9f29-cb26-8f90-22b24de85079: 2021-07-10T01:05:05.748Z [INFO] server-6a233ea6-9f29-cb26-8f90-22b24de85079: Installed remote snapshot - testing.go:149: server-6a233ea6-9f29-cb26-8f90-22b24de85079: 2021-07-10T01:05:05.748Z [INFO] server-6a233ea6-9f29-cb26-8f90-22b24de85079: Installed remote snapshot -testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.748Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: pipelining replication: peer="{Voter server-6a233ea6-9f29-cb26-8f90-22b24de85079 6a233ea6-9f29-cb26-8f90-22b24de85079}" - testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.748Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: pipelining replication: peer="{Voter server-6a233ea6-9f29-cb26-8f90-22b24de85079 6a233ea6-9f29-cb26-8f90-22b24de85079}" -testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.761Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: aborting pipeline replication: peer="{Voter server-4b63ed8c-a916-e473-8b4c-baa46c99b91b 4b63ed8c-a916-e473-8b4c-baa46c99b91b}" - testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.761Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: aborting pipeline replication: peer="{Voter server-4b63ed8c-a916-e473-8b4c-baa46c99b91b 4b63ed8c-a916-e473-8b4c-baa46c99b91b}" -testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.761Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: aborting pipeline replication: peer="{Voter server-6a233ea6-9f29-cb26-8f90-22b24de85079 6a233ea6-9f29-cb26-8f90-22b24de85079}" - testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.761Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: aborting pipeline replication: peer="{Voter server-6a233ea6-9f29-cb26-8f90-22b24de85079 6a233ea6-9f29-cb26-8f90-22b24de85079}" -testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.763Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8471d207-4de9-94a9-5793-b1487409cf54 Address:8471d207-4de9-94a9-5793-b1487409cf54} {Suffrage:Voter ID:server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0 Address:3ee9cfc8-34d3-969a-a04e-92671f5ff0d0} {Suffrage:Voter ID:server-f2ca7868-cf8e-471c-8066-09c8d7be878a Address:f2ca7868-cf8e-471c-8066-09c8d7be878a}]" - testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.763Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8471d207-4de9-94a9-5793-b1487409cf54 Address:8471d207-4de9-94a9-5793-b1487409cf54} {Suffrage:Voter ID:server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0 Address:3ee9cfc8-34d3-969a-a04e-92671f5ff0d0} {Suffrage:Voter ID:server-f2ca7868-cf8e-471c-8066-09c8d7be878a Address:f2ca7868-cf8e-471c-8066-09c8d7be878a}]" -testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.764Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: entering follower state: follower="Node at 8471d207-4de9-94a9-5793-b1487409cf54 [Follower]" leader= -testLoggerAdapter verbose: server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0: 2021-07-10T01:05:05.763Z [INFO] server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8471d207-4de9-94a9-5793-b1487409cf54 Address:8471d207-4de9-94a9-5793-b1487409cf54} {Suffrage:Voter ID:server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0 Address:3ee9cfc8-34d3-969a-a04e-92671f5ff0d0} {Suffrage:Voter ID:server-f2ca7868-cf8e-471c-8066-09c8d7be878a Address:f2ca7868-cf8e-471c-8066-09c8d7be878a}]" - testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.764Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: entering follower state: follower="Node at 8471d207-4de9-94a9-5793-b1487409cf54 [Follower]" leader= - testing.go:149: server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0: 2021-07-10T01:05:05.763Z [INFO] server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8471d207-4de9-94a9-5793-b1487409cf54 Address:8471d207-4de9-94a9-5793-b1487409cf54} {Suffrage:Voter ID:server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0 Address:3ee9cfc8-34d3-969a-a04e-92671f5ff0d0} {Suffrage:Voter ID:server-f2ca7868-cf8e-471c-8066-09c8d7be878a Address:f2ca7868-cf8e-471c-8066-09c8d7be878a}]" -testLoggerAdapter verbose: server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0: 2021-07-10T01:05:05.764Z [INFO] server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0: entering follower state: follower="Node at 3ee9cfc8-34d3-969a-a04e-92671f5ff0d0 [Follower]" leader= - testing.go:149: server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0: 2021-07-10T01:05:05.764Z [INFO] server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0: entering follower state: follower="Node at 3ee9cfc8-34d3-969a-a04e-92671f5ff0d0 [Follower]" leader= -testLoggerAdapter verbose: server-f2ca7868-cf8e-471c-8066-09c8d7be878a: 2021-07-10T01:05:05.764Z [INFO] server-f2ca7868-cf8e-471c-8066-09c8d7be878a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8471d207-4de9-94a9-5793-b1487409cf54 Address:8471d207-4de9-94a9-5793-b1487409cf54} {Suffrage:Voter ID:server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0 Address:3ee9cfc8-34d3-969a-a04e-92671f5ff0d0} {Suffrage:Voter ID:server-f2ca7868-cf8e-471c-8066-09c8d7be878a Address:f2ca7868-cf8e-471c-8066-09c8d7be878a}]" - testing.go:149: server-f2ca7868-cf8e-471c-8066-09c8d7be878a: 2021-07-10T01:05:05.764Z [INFO] server-f2ca7868-cf8e-471c-8066-09c8d7be878a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8471d207-4de9-94a9-5793-b1487409cf54 Address:8471d207-4de9-94a9-5793-b1487409cf54} {Suffrage:Voter ID:server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0 Address:3ee9cfc8-34d3-969a-a04e-92671f5ff0d0} {Suffrage:Voter ID:server-f2ca7868-cf8e-471c-8066-09c8d7be878a Address:f2ca7868-cf8e-471c-8066-09c8d7be878a}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:05.765Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:05.765Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-f2ca7868-cf8e-471c-8066-09c8d7be878a: 2021-07-10T01:05:05.765Z [INFO] server-f2ca7868-cf8e-471c-8066-09c8d7be878a: entering follower state: follower="Node at f2ca7868-cf8e-471c-8066-09c8d7be878a [Follower]" leader= - testing.go:149: server-f2ca7868-cf8e-471c-8066-09c8d7be878a: 2021-07-10T01:05:05.765Z [INFO] server-f2ca7868-cf8e-471c-8066-09c8d7be878a: entering follower state: follower="Node at f2ca7868-cf8e-471c-8066-09c8d7be878a [Follower]" leader= -testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.831Z [WARN] server-8471d207-4de9-94a9-5793-b1487409cf54: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.831Z [WARN] server-8471d207-4de9-94a9-5793-b1487409cf54: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.832Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: entering candidate state: node="Node at 8471d207-4de9-94a9-5793-b1487409cf54 [Candidate]" term=2 - testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.832Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: entering candidate state: node="Node at 8471d207-4de9-94a9-5793-b1487409cf54 [Candidate]" term=2 -testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.832Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: election won: tally=2 - testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.832Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: election won: tally=2 -testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.832Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: entering leader state: leader="Node at 8471d207-4de9-94a9-5793-b1487409cf54 [Leader]" - testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.832Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: entering leader state: leader="Node at 8471d207-4de9-94a9-5793-b1487409cf54 [Leader]" -testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.832Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: added peer, starting replication: peer=server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0 - testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.832Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: added peer, starting replication: peer=server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0 -testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.832Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: added peer, starting replication: peer=server-f2ca7868-cf8e-471c-8066-09c8d7be878a - testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.832Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: added peer, starting replication: peer=server-f2ca7868-cf8e-471c-8066-09c8d7be878a -testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.832Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: pipelining replication: peer="{Voter server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0 3ee9cfc8-34d3-969a-a04e-92671f5ff0d0}" - testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.832Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: pipelining replication: peer="{Voter server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0 3ee9cfc8-34d3-969a-a04e-92671f5ff0d0}" -testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.833Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: pipelining replication: peer="{Voter server-f2ca7868-cf8e-471c-8066-09c8d7be878a f2ca7868-cf8e-471c-8066-09c8d7be878a}" - testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.833Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: pipelining replication: peer="{Voter server-f2ca7868-cf8e-471c-8066-09c8d7be878a f2ca7868-cf8e-471c-8066-09c8d7be878a}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:05.937Z [INFO] stable state for Leader reached at 2021-07-10 01:05:05.832632884 +0000 UTC m=+11.220790874 (1 nodes), 67.511572ms from start of poll, 69.789296ms from cluster start. Timeout at 2021-07-10 01:05:05.937696771 +0000 UTC m=+11.325854741, 105.063867ms after stability - testing.go:149: cluster: 2021-07-10T01:05:05.937Z [INFO] stable state for Leader reached at 2021-07-10 01:05:05.832632884 +0000 UTC m=+11.220790874 (1 nodes), 67.511572ms from start of poll, 69.789296ms from cluster start. Timeout at 2021-07-10 01:05:05.937696771 +0000 UTC m=+11.325854741, 105.063867ms after stability -testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.938Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: starting snapshot up to: index=12 - testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.938Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: starting snapshot up to: index=12 - testing.go:153: 2021-07-10T01:05:05.938Z [INFO] creating new snapshot: path=/tmp/raft968864481/snapshots/2-12-1625879105938.tmp -testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.939Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: snapshot complete up to: index=12 - testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.939Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: snapshot complete up to: index=12 - testing.go:153: 2021-07-10T01:05:05.940Z [INFO] creating new snapshot: path=/tmp/raft968864481/snapshots/2-23-1625879105940.tmp -testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.940Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: copied to local snapshot: bytes=71 - testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.940Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.941Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: restored user snapshot: index=1 - testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.941Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: restored user snapshot: index=1 -testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.941Z [ERROR] server-8471d207-4de9-94a9-5793-b1487409cf54: failed to get log: index=23 error="log not found" - testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.941Z [ERROR] server-8471d207-4de9-94a9-5793-b1487409cf54: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.941Z [ERROR] server-8471d207-4de9-94a9-5793-b1487409cf54: failed to get log: index=23 error="log not found" - testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.941Z [ERROR] server-8471d207-4de9-94a9-5793-b1487409cf54: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.941Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: aborting pipeline replication: peer="{Voter server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0 3ee9cfc8-34d3-969a-a04e-92671f5ff0d0}" - testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.941Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: aborting pipeline replication: peer="{Voter server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0 3ee9cfc8-34d3-969a-a04e-92671f5ff0d0}" -testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.941Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: aborting pipeline replication: peer="{Voter server-f2ca7868-cf8e-471c-8066-09c8d7be878a f2ca7868-cf8e-471c-8066-09c8d7be878a}" - testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.941Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: aborting pipeline replication: peer="{Voter server-f2ca7868-cf8e-471c-8066-09c8d7be878a f2ca7868-cf8e-471c-8066-09c8d7be878a}" -testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.949Z [ERROR] server-8471d207-4de9-94a9-5793-b1487409cf54: failed to get log: index=23 error="log not found" - testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.949Z [ERROR] server-8471d207-4de9-94a9-5793-b1487409cf54: failed to get log: index=23 error="log not found" - testing.go:153: 2021-07-10T01:05:05.950Z [INFO] creating new snapshot: path=/tmp/raft494882213/snapshots/2-23-1625879105950.tmp -testLoggerAdapter verbose: server-f2ca7868-cf8e-471c-8066-09c8d7be878a: 2021-07-10T01:05:05.951Z [INFO] server-f2ca7868-cf8e-471c-8066-09c8d7be878a: copied to local snapshot: bytes=71 - testing.go:149: server-f2ca7868-cf8e-471c-8066-09c8d7be878a: 2021-07-10T01:05:05.951Z [INFO] server-f2ca7868-cf8e-471c-8066-09c8d7be878a: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.951Z [ERROR] server-8471d207-4de9-94a9-5793-b1487409cf54: failed to get log: index=23 error="log not found" - testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.951Z [ERROR] server-8471d207-4de9-94a9-5793-b1487409cf54: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-f2ca7868-cf8e-471c-8066-09c8d7be878a: 2021-07-10T01:05:05.951Z [INFO] server-f2ca7868-cf8e-471c-8066-09c8d7be878a: Installed remote snapshot - testing.go:149: server-f2ca7868-cf8e-471c-8066-09c8d7be878a: 2021-07-10T01:05:05.951Z [INFO] server-f2ca7868-cf8e-471c-8066-09c8d7be878a: Installed remote snapshot -testLoggerAdapter verbose: server-f2ca7868-cf8e-471c-8066-09c8d7be878a: 2021-07-10T01:05:05.951Z [WARN] server-f2ca7868-cf8e-471c-8066-09c8d7be878a: skipping application of old log: index=22 - testing.go:149: server-f2ca7868-cf8e-471c-8066-09c8d7be878a: 2021-07-10T01:05:05.951Z [WARN] server-f2ca7868-cf8e-471c-8066-09c8d7be878a: skipping application of old log: index=22 -testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.951Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: pipelining replication: peer="{Voter server-f2ca7868-cf8e-471c-8066-09c8d7be878a f2ca7868-cf8e-471c-8066-09c8d7be878a}" - testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.951Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: pipelining replication: peer="{Voter server-f2ca7868-cf8e-471c-8066-09c8d7be878a f2ca7868-cf8e-471c-8066-09c8d7be878a}" - testing.go:153: 2021-07-10T01:05:05.951Z [INFO] creating new snapshot: path=/tmp/raft740853691/snapshots/2-23-1625879105951.tmp -testLoggerAdapter verbose: server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0: 2021-07-10T01:05:05.952Z [INFO] server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0: copied to local snapshot: bytes=71 - testing.go:149: server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0: 2021-07-10T01:05:05.952Z [INFO] server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0: 2021-07-10T01:05:05.952Z [INFO] server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0: Installed remote snapshot - testing.go:149: server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0: 2021-07-10T01:05:05.952Z [INFO] server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0: Installed remote snapshot -testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.952Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: pipelining replication: peer="{Voter server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0 3ee9cfc8-34d3-969a-a04e-92671f5ff0d0}" - testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.952Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: pipelining replication: peer="{Voter server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0 3ee9cfc8-34d3-969a-a04e-92671f5ff0d0}" -testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.967Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: aborting pipeline replication: peer="{Voter server-f2ca7868-cf8e-471c-8066-09c8d7be878a f2ca7868-cf8e-471c-8066-09c8d7be878a}" - testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.967Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: aborting pipeline replication: peer="{Voter server-f2ca7868-cf8e-471c-8066-09c8d7be878a f2ca7868-cf8e-471c-8066-09c8d7be878a}" -testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.967Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: aborting pipeline replication: peer="{Voter server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0 3ee9cfc8-34d3-969a-a04e-92671f5ff0d0}" - testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.967Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: aborting pipeline replication: peer="{Voter server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0 3ee9cfc8-34d3-969a-a04e-92671f5ff0d0}" -testLoggerAdapter verbose: server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: 2021-07-10T01:05:05.969Z [INFO] server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-48eacc89-1ba5-1136-f5b7-64accdcd20c6 Address:48eacc89-1ba5-1136-f5b7-64accdcd20c6} {Suffrage:Voter ID:server-ca2f2293-543c-d2e7-acdd-df93c434c65b Address:ca2f2293-543c-d2e7-acdd-df93c434c65b} {Suffrage:Voter ID:server-7d0636ee-f718-97e6-3e8b-077575c1f1fb Address:7d0636ee-f718-97e6-3e8b-077575c1f1fb}]" - testing.go:149: server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: 2021-07-10T01:05:05.969Z [INFO] server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-48eacc89-1ba5-1136-f5b7-64accdcd20c6 Address:48eacc89-1ba5-1136-f5b7-64accdcd20c6} {Suffrage:Voter ID:server-ca2f2293-543c-d2e7-acdd-df93c434c65b Address:ca2f2293-543c-d2e7-acdd-df93c434c65b} {Suffrage:Voter ID:server-7d0636ee-f718-97e6-3e8b-077575c1f1fb Address:7d0636ee-f718-97e6-3e8b-077575c1f1fb}]" -testLoggerAdapter verbose: server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: 2021-07-10T01:05:05.969Z [INFO] server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: entering follower state: follower="Node at 48eacc89-1ba5-1136-f5b7-64accdcd20c6 [Follower]" leader= - testing.go:149: server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: 2021-07-10T01:05:05.969Z [INFO] server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: entering follower state: follower="Node at 48eacc89-1ba5-1136-f5b7-64accdcd20c6 [Follower]" leader= -testLoggerAdapter verbose: server-ca2f2293-543c-d2e7-acdd-df93c434c65b: 2021-07-10T01:05:05.969Z [INFO] server-ca2f2293-543c-d2e7-acdd-df93c434c65b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-48eacc89-1ba5-1136-f5b7-64accdcd20c6 Address:48eacc89-1ba5-1136-f5b7-64accdcd20c6} {Suffrage:Voter ID:server-ca2f2293-543c-d2e7-acdd-df93c434c65b Address:ca2f2293-543c-d2e7-acdd-df93c434c65b} {Suffrage:Voter ID:server-7d0636ee-f718-97e6-3e8b-077575c1f1fb Address:7d0636ee-f718-97e6-3e8b-077575c1f1fb}]" - testing.go:149: server-ca2f2293-543c-d2e7-acdd-df93c434c65b: 2021-07-10T01:05:05.969Z [INFO] server-ca2f2293-543c-d2e7-acdd-df93c434c65b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-48eacc89-1ba5-1136-f5b7-64accdcd20c6 Address:48eacc89-1ba5-1136-f5b7-64accdcd20c6} {Suffrage:Voter ID:server-ca2f2293-543c-d2e7-acdd-df93c434c65b Address:ca2f2293-543c-d2e7-acdd-df93c434c65b} {Suffrage:Voter ID:server-7d0636ee-f718-97e6-3e8b-077575c1f1fb Address:7d0636ee-f718-97e6-3e8b-077575c1f1fb}]" -testLoggerAdapter verbose: server-ca2f2293-543c-d2e7-acdd-df93c434c65b: 2021-07-10T01:05:05.969Z [INFO] server-ca2f2293-543c-d2e7-acdd-df93c434c65b: entering follower state: follower="Node at ca2f2293-543c-d2e7-acdd-df93c434c65b [Follower]" leader= - testing.go:149: server-ca2f2293-543c-d2e7-acdd-df93c434c65b: 2021-07-10T01:05:05.969Z [INFO] server-ca2f2293-543c-d2e7-acdd-df93c434c65b: entering follower state: follower="Node at ca2f2293-543c-d2e7-acdd-df93c434c65b [Follower]" leader= -testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:05.969Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-48eacc89-1ba5-1136-f5b7-64accdcd20c6 Address:48eacc89-1ba5-1136-f5b7-64accdcd20c6} {Suffrage:Voter ID:server-ca2f2293-543c-d2e7-acdd-df93c434c65b Address:ca2f2293-543c-d2e7-acdd-df93c434c65b} {Suffrage:Voter ID:server-7d0636ee-f718-97e6-3e8b-077575c1f1fb Address:7d0636ee-f718-97e6-3e8b-077575c1f1fb}]" - testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:05.969Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-48eacc89-1ba5-1136-f5b7-64accdcd20c6 Address:48eacc89-1ba5-1136-f5b7-64accdcd20c6} {Suffrage:Voter ID:server-ca2f2293-543c-d2e7-acdd-df93c434c65b Address:ca2f2293-543c-d2e7-acdd-df93c434c65b} {Suffrage:Voter ID:server-7d0636ee-f718-97e6-3e8b-077575c1f1fb Address:7d0636ee-f718-97e6-3e8b-077575c1f1fb}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:05.969Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:05.969Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:05.969Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: entering follower state: follower="Node at 7d0636ee-f718-97e6-3e8b-077575c1f1fb [Follower]" leader= - testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:05.969Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: entering follower state: follower="Node at 7d0636ee-f718-97e6-3e8b-077575c1f1fb [Follower]" leader= -testLoggerAdapter verbose: server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: 2021-07-10T01:05:06.045Z [WARN] server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: 2021-07-10T01:05:06.045Z [WARN] server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.046Z [WARN] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.046Z [WARN] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.046Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: entering candidate state: node="Node at 7d0636ee-f718-97e6-3e8b-077575c1f1fb [Candidate]" term=2 -testLoggerAdapter verbose: server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: 2021-07-10T01:05:06.046Z [INFO] server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: entering candidate state: node="Node at 48eacc89-1ba5-1136-f5b7-64accdcd20c6 [Candidate]" term=2 - testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.046Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: entering candidate state: node="Node at 7d0636ee-f718-97e6-3e8b-077575c1f1fb [Candidate]" term=2 - testing.go:149: server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: 2021-07-10T01:05:06.046Z [INFO] server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: entering candidate state: node="Node at 48eacc89-1ba5-1136-f5b7-64accdcd20c6 [Candidate]" term=2 -testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.046Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: election won: tally=2 - testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.046Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: election won: tally=2 -testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.046Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: entering leader state: leader="Node at 7d0636ee-f718-97e6-3e8b-077575c1f1fb [Leader]" - testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.046Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: entering leader state: leader="Node at 7d0636ee-f718-97e6-3e8b-077575c1f1fb [Leader]" -testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.046Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: added peer, starting replication: peer=server-48eacc89-1ba5-1136-f5b7-64accdcd20c6 -testLoggerAdapter verbose: server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: 2021-07-10T01:05:06.046Z [INFO] server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: duplicate requestVote for same term: term=2 - testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.046Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: added peer, starting replication: peer=server-48eacc89-1ba5-1136-f5b7-64accdcd20c6 -testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.046Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: added peer, starting replication: peer=server-ca2f2293-543c-d2e7-acdd-df93c434c65b -testLoggerAdapter verbose: server-ca2f2293-543c-d2e7-acdd-df93c434c65b: 2021-07-10T01:05:06.046Z [INFO] server-ca2f2293-543c-d2e7-acdd-df93c434c65b: duplicate requestVote for same term: term=2 - testing.go:149: server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: 2021-07-10T01:05:06.046Z [INFO] server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: duplicate requestVote for same term: term=2 - testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.046Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: added peer, starting replication: peer=server-ca2f2293-543c-d2e7-acdd-df93c434c65b -testLoggerAdapter verbose: server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: 2021-07-10T01:05:06.046Z [INFO] server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: entering follower state: follower="Node at 48eacc89-1ba5-1136-f5b7-64accdcd20c6 [Follower]" leader=7d0636ee-f718-97e6-3e8b-077575c1f1fb - testing.go:149: server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: 2021-07-10T01:05:06.046Z [INFO] server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: entering follower state: follower="Node at 48eacc89-1ba5-1136-f5b7-64accdcd20c6 [Follower]" leader=7d0636ee-f718-97e6-3e8b-077575c1f1fb -testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.046Z [WARN] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: rejecting vote request since we have a leader: from=48eacc89-1ba5-1136-f5b7-64accdcd20c6 leader=7d0636ee-f718-97e6-3e8b-077575c1f1fb - testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.046Z [WARN] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: rejecting vote request since we have a leader: from=48eacc89-1ba5-1136-f5b7-64accdcd20c6 leader=7d0636ee-f718-97e6-3e8b-077575c1f1fb - testing.go:149: server-ca2f2293-543c-d2e7-acdd-df93c434c65b: 2021-07-10T01:05:06.046Z [INFO] server-ca2f2293-543c-d2e7-acdd-df93c434c65b: duplicate requestVote for same term: term=2 -testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.046Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: pipelining replication: peer="{Voter server-48eacc89-1ba5-1136-f5b7-64accdcd20c6 48eacc89-1ba5-1136-f5b7-64accdcd20c6}" - testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.046Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: pipelining replication: peer="{Voter server-48eacc89-1ba5-1136-f5b7-64accdcd20c6 48eacc89-1ba5-1136-f5b7-64accdcd20c6}" -testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.046Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: pipelining replication: peer="{Voter server-ca2f2293-543c-d2e7-acdd-df93c434c65b ca2f2293-543c-d2e7-acdd-df93c434c65b}" - testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.046Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: pipelining replication: peer="{Voter server-ca2f2293-543c-d2e7-acdd-df93c434c65b ca2f2293-543c-d2e7-acdd-df93c434c65b}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:06.151Z [INFO] stable state for Leader reached at 2021-07-10 01:05:06.04671056 +0000 UTC m=+11.434868530 (1 nodes), 76.718808ms from start of poll, 77.650498ms from cluster start. Timeout at 2021-07-10 01:05:06.151791588 +0000 UTC m=+11.539949558, 105.081028ms after stability - testing.go:149: cluster: 2021-07-10T01:05:06.151Z [INFO] stable state for Leader reached at 2021-07-10 01:05:06.04671056 +0000 UTC m=+11.434868530 (1 nodes), 76.718808ms from start of poll, 77.650498ms from cluster start. Timeout at 2021-07-10 01:05:06.151791588 +0000 UTC m=+11.539949558, 105.081028ms after stability -testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.152Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: starting snapshot up to: index=12 - testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.152Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: starting snapshot up to: index=12 - testing.go:153: 2021-07-10T01:05:06.152Z [INFO] creating new snapshot: path=/tmp/raft014180035/snapshots/2-12-1625879106152.tmp -testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.153Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: snapshot complete up to: index=12 - testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.153Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: snapshot complete up to: index=12 - testing.go:153: 2021-07-10T01:05:06.154Z [INFO] creating new snapshot: path=/tmp/raft014180035/snapshots/2-113-1625879106154.tmp -testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.155Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: copied to local snapshot: bytes=71 - testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.155Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.155Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: restored user snapshot: index=1 - testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.155Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: restored user snapshot: index=1 -testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.155Z [ERROR] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: failed to get log: index=23 error="log not found" - testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.155Z [ERROR] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.155Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: aborting pipeline replication: peer="{Voter server-ca2f2293-543c-d2e7-acdd-df93c434c65b ca2f2293-543c-d2e7-acdd-df93c434c65b}" - testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.155Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: aborting pipeline replication: peer="{Voter server-ca2f2293-543c-d2e7-acdd-df93c434c65b ca2f2293-543c-d2e7-acdd-df93c434c65b}" -testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.155Z [ERROR] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: failed to get log: index=23 error="log not found" - testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.155Z [ERROR] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.155Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: aborting pipeline replication: peer="{Voter server-48eacc89-1ba5-1136-f5b7-64accdcd20c6 48eacc89-1ba5-1136-f5b7-64accdcd20c6}" - testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.155Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: aborting pipeline replication: peer="{Voter server-48eacc89-1ba5-1136-f5b7-64accdcd20c6 48eacc89-1ba5-1136-f5b7-64accdcd20c6}" -testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.161Z [ERROR] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: failed to get log: index=23 error="log not found" - testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.161Z [ERROR] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.161Z [ERROR] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: failed to get log: index=23 error="log not found" - testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.161Z [ERROR] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: failed to get log: index=23 error="log not found" - testing.go:153: 2021-07-10T01:05:06.161Z [INFO] creating new snapshot: path=/tmp/raft046324393/snapshots/2-113-1625879106161.tmp - testing.go:153: 2021-07-10T01:05:06.162Z [INFO] creating new snapshot: path=/tmp/raft434071327/snapshots/2-113-1625879106161.tmp -testLoggerAdapter verbose: server-ca2f2293-543c-d2e7-acdd-df93c434c65b: 2021-07-10T01:05:06.162Z [INFO] server-ca2f2293-543c-d2e7-acdd-df93c434c65b: copied to local snapshot: bytes=71 - testing.go:149: server-ca2f2293-543c-d2e7-acdd-df93c434c65b: 2021-07-10T01:05:06.162Z [INFO] server-ca2f2293-543c-d2e7-acdd-df93c434c65b: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: 2021-07-10T01:05:06.163Z [INFO] server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-ca2f2293-543c-d2e7-acdd-df93c434c65b: 2021-07-10T01:05:06.163Z [INFO] server-ca2f2293-543c-d2e7-acdd-df93c434c65b: Installed remote snapshot - testing.go:149: server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: 2021-07-10T01:05:06.163Z [INFO] server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: copied to local snapshot: bytes=71 - testing.go:149: server-ca2f2293-543c-d2e7-acdd-df93c434c65b: 2021-07-10T01:05:06.163Z [INFO] server-ca2f2293-543c-d2e7-acdd-df93c434c65b: Installed remote snapshot -testLoggerAdapter verbose: server-ca2f2293-543c-d2e7-acdd-df93c434c65b: 2021-07-10T01:05:06.164Z [WARN] server-ca2f2293-543c-d2e7-acdd-df93c434c65b: skipping application of old log: index=22 - testing.go:149: server-ca2f2293-543c-d2e7-acdd-df93c434c65b: 2021-07-10T01:05:06.164Z [WARN] server-ca2f2293-543c-d2e7-acdd-df93c434c65b: skipping application of old log: index=22 -testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.164Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: pipelining replication: peer="{Voter server-ca2f2293-543c-d2e7-acdd-df93c434c65b ca2f2293-543c-d2e7-acdd-df93c434c65b}" - testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.164Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: pipelining replication: peer="{Voter server-ca2f2293-543c-d2e7-acdd-df93c434c65b ca2f2293-543c-d2e7-acdd-df93c434c65b}" -testLoggerAdapter verbose: server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: 2021-07-10T01:05:06.164Z [INFO] server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: Installed remote snapshot - testing.go:149: server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: 2021-07-10T01:05:06.164Z [INFO] server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: Installed remote snapshot -testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.164Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: pipelining replication: peer="{Voter server-48eacc89-1ba5-1136-f5b7-64accdcd20c6 48eacc89-1ba5-1136-f5b7-64accdcd20c6}" - testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.164Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: pipelining replication: peer="{Voter server-48eacc89-1ba5-1136-f5b7-64accdcd20c6 48eacc89-1ba5-1136-f5b7-64accdcd20c6}" -testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.175Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: aborting pipeline replication: peer="{Voter server-ca2f2293-543c-d2e7-acdd-df93c434c65b ca2f2293-543c-d2e7-acdd-df93c434c65b}" - testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.175Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: aborting pipeline replication: peer="{Voter server-ca2f2293-543c-d2e7-acdd-df93c434c65b ca2f2293-543c-d2e7-acdd-df93c434c65b}" -testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.175Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: aborting pipeline replication: peer="{Voter server-48eacc89-1ba5-1136-f5b7-64accdcd20c6 48eacc89-1ba5-1136-f5b7-64accdcd20c6}" - testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.175Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: aborting pipeline replication: peer="{Voter server-48eacc89-1ba5-1136-f5b7-64accdcd20c6 48eacc89-1ba5-1136-f5b7-64accdcd20c6}" -testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.177Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8516e85f-e82e-9856-b696-b2add172e417 Address:8516e85f-e82e-9856-b696-b2add172e417} {Suffrage:Voter ID:server-186970eb-04e2-8b42-4a61-7c4501527a25 Address:186970eb-04e2-8b42-4a61-7c4501527a25} {Suffrage:Voter ID:server-79f52308-c464-7454-e3f1-a5714823a1b1 Address:79f52308-c464-7454-e3f1-a5714823a1b1}]" - testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.177Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8516e85f-e82e-9856-b696-b2add172e417 Address:8516e85f-e82e-9856-b696-b2add172e417} {Suffrage:Voter ID:server-186970eb-04e2-8b42-4a61-7c4501527a25 Address:186970eb-04e2-8b42-4a61-7c4501527a25} {Suffrage:Voter ID:server-79f52308-c464-7454-e3f1-a5714823a1b1 Address:79f52308-c464-7454-e3f1-a5714823a1b1}]" -testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.177Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: entering follower state: follower="Node at 8516e85f-e82e-9856-b696-b2add172e417 [Follower]" leader= - testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.177Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: entering follower state: follower="Node at 8516e85f-e82e-9856-b696-b2add172e417 [Follower]" leader= -testLoggerAdapter verbose: server-186970eb-04e2-8b42-4a61-7c4501527a25: 2021-07-10T01:05:06.177Z [INFO] server-186970eb-04e2-8b42-4a61-7c4501527a25: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8516e85f-e82e-9856-b696-b2add172e417 Address:8516e85f-e82e-9856-b696-b2add172e417} {Suffrage:Voter ID:server-186970eb-04e2-8b42-4a61-7c4501527a25 Address:186970eb-04e2-8b42-4a61-7c4501527a25} {Suffrage:Voter ID:server-79f52308-c464-7454-e3f1-a5714823a1b1 Address:79f52308-c464-7454-e3f1-a5714823a1b1}]" - testing.go:149: server-186970eb-04e2-8b42-4a61-7c4501527a25: 2021-07-10T01:05:06.177Z [INFO] server-186970eb-04e2-8b42-4a61-7c4501527a25: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8516e85f-e82e-9856-b696-b2add172e417 Address:8516e85f-e82e-9856-b696-b2add172e417} {Suffrage:Voter ID:server-186970eb-04e2-8b42-4a61-7c4501527a25 Address:186970eb-04e2-8b42-4a61-7c4501527a25} {Suffrage:Voter ID:server-79f52308-c464-7454-e3f1-a5714823a1b1 Address:79f52308-c464-7454-e3f1-a5714823a1b1}]" -testLoggerAdapter verbose: server-186970eb-04e2-8b42-4a61-7c4501527a25: 2021-07-10T01:05:06.177Z [INFO] server-186970eb-04e2-8b42-4a61-7c4501527a25: entering follower state: follower="Node at 186970eb-04e2-8b42-4a61-7c4501527a25 [Follower]" leader= - testing.go:149: server-186970eb-04e2-8b42-4a61-7c4501527a25: 2021-07-10T01:05:06.177Z [INFO] server-186970eb-04e2-8b42-4a61-7c4501527a25: entering follower state: follower="Node at 186970eb-04e2-8b42-4a61-7c4501527a25 [Follower]" leader= -testLoggerAdapter verbose: server-79f52308-c464-7454-e3f1-a5714823a1b1: 2021-07-10T01:05:06.177Z [INFO] server-79f52308-c464-7454-e3f1-a5714823a1b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8516e85f-e82e-9856-b696-b2add172e417 Address:8516e85f-e82e-9856-b696-b2add172e417} {Suffrage:Voter ID:server-186970eb-04e2-8b42-4a61-7c4501527a25 Address:186970eb-04e2-8b42-4a61-7c4501527a25} {Suffrage:Voter ID:server-79f52308-c464-7454-e3f1-a5714823a1b1 Address:79f52308-c464-7454-e3f1-a5714823a1b1}]" - testing.go:149: server-79f52308-c464-7454-e3f1-a5714823a1b1: 2021-07-10T01:05:06.177Z [INFO] server-79f52308-c464-7454-e3f1-a5714823a1b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8516e85f-e82e-9856-b696-b2add172e417 Address:8516e85f-e82e-9856-b696-b2add172e417} {Suffrage:Voter ID:server-186970eb-04e2-8b42-4a61-7c4501527a25 Address:186970eb-04e2-8b42-4a61-7c4501527a25} {Suffrage:Voter ID:server-79f52308-c464-7454-e3f1-a5714823a1b1 Address:79f52308-c464-7454-e3f1-a5714823a1b1}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:06.177Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:06.177Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-79f52308-c464-7454-e3f1-a5714823a1b1: 2021-07-10T01:05:06.177Z [INFO] server-79f52308-c464-7454-e3f1-a5714823a1b1: entering follower state: follower="Node at 79f52308-c464-7454-e3f1-a5714823a1b1 [Follower]" leader= - testing.go:149: server-79f52308-c464-7454-e3f1-a5714823a1b1: 2021-07-10T01:05:06.177Z [INFO] server-79f52308-c464-7454-e3f1-a5714823a1b1: entering follower state: follower="Node at 79f52308-c464-7454-e3f1-a5714823a1b1 [Follower]" leader= -testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.239Z [WARN] server-8516e85f-e82e-9856-b696-b2add172e417: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.239Z [WARN] server-8516e85f-e82e-9856-b696-b2add172e417: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.239Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: entering candidate state: node="Node at 8516e85f-e82e-9856-b696-b2add172e417 [Candidate]" term=2 - testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.239Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: entering candidate state: node="Node at 8516e85f-e82e-9856-b696-b2add172e417 [Candidate]" term=2 -testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.240Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: election won: tally=2 - testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.240Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: election won: tally=2 -testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.240Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: entering leader state: leader="Node at 8516e85f-e82e-9856-b696-b2add172e417 [Leader]" - testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.240Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: entering leader state: leader="Node at 8516e85f-e82e-9856-b696-b2add172e417 [Leader]" -testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.240Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: added peer, starting replication: peer=server-186970eb-04e2-8b42-4a61-7c4501527a25 - testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.240Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: added peer, starting replication: peer=server-186970eb-04e2-8b42-4a61-7c4501527a25 -testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.240Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: added peer, starting replication: peer=server-79f52308-c464-7454-e3f1-a5714823a1b1 - testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.240Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: added peer, starting replication: peer=server-79f52308-c464-7454-e3f1-a5714823a1b1 -testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.240Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: pipelining replication: peer="{Voter server-186970eb-04e2-8b42-4a61-7c4501527a25 186970eb-04e2-8b42-4a61-7c4501527a25}" - testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.240Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: pipelining replication: peer="{Voter server-186970eb-04e2-8b42-4a61-7c4501527a25 186970eb-04e2-8b42-4a61-7c4501527a25}" -testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.240Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: pipelining replication: peer="{Voter server-79f52308-c464-7454-e3f1-a5714823a1b1 79f52308-c464-7454-e3f1-a5714823a1b1}" - testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.240Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: pipelining replication: peer="{Voter server-79f52308-c464-7454-e3f1-a5714823a1b1 79f52308-c464-7454-e3f1-a5714823a1b1}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:06.345Z [INFO] stable state for Leader reached at 2021-07-10 01:05:06.240317321 +0000 UTC m=+11.628475311 (1 nodes), 62.323917ms from start of poll, 63.189406ms from cluster start. Timeout at 2021-07-10 01:05:06.345397809 +0000 UTC m=+11.733555799, 105.080488ms after stability - testing.go:149: cluster: 2021-07-10T01:05:06.345Z [INFO] stable state for Leader reached at 2021-07-10 01:05:06.240317321 +0000 UTC m=+11.628475311 (1 nodes), 62.323917ms from start of poll, 63.189406ms from cluster start. Timeout at 2021-07-10 01:05:06.345397809 +0000 UTC m=+11.733555799, 105.080488ms after stability -testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.346Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: starting snapshot up to: index=12 - testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.346Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: starting snapshot up to: index=12 - testing.go:153: 2021-07-10T01:05:06.346Z [INFO] creating new snapshot: path=/tmp/raft728465389/snapshots/2-12-1625879106346.tmp -testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.346Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: snapshot complete up to: index=12 - testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.346Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: snapshot complete up to: index=12 - testing.go:153: 2021-07-10T01:05:06.347Z [INFO] creating new snapshot: path=/tmp/raft728465389/snapshots/2-1013-1625879106347.tmp -testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.348Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: copied to local snapshot: bytes=71 - testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.348Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.348Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: restored user snapshot: index=1 - testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.348Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: restored user snapshot: index=1 -testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.348Z [ERROR] server-8516e85f-e82e-9856-b696-b2add172e417: failed to get log: index=23 error="log not found" - testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.348Z [ERROR] server-8516e85f-e82e-9856-b696-b2add172e417: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.348Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: aborting pipeline replication: peer="{Voter server-79f52308-c464-7454-e3f1-a5714823a1b1 79f52308-c464-7454-e3f1-a5714823a1b1}" - testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.348Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: aborting pipeline replication: peer="{Voter server-79f52308-c464-7454-e3f1-a5714823a1b1 79f52308-c464-7454-e3f1-a5714823a1b1}" -testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.348Z [ERROR] server-8516e85f-e82e-9856-b696-b2add172e417: failed to get log: index=23 error="log not found" - testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.348Z [ERROR] server-8516e85f-e82e-9856-b696-b2add172e417: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.349Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: aborting pipeline replication: peer="{Voter server-186970eb-04e2-8b42-4a61-7c4501527a25 186970eb-04e2-8b42-4a61-7c4501527a25}" - testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.349Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: aborting pipeline replication: peer="{Voter server-186970eb-04e2-8b42-4a61-7c4501527a25 186970eb-04e2-8b42-4a61-7c4501527a25}" -testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.357Z [ERROR] server-8516e85f-e82e-9856-b696-b2add172e417: failed to get log: index=23 error="log not found" - testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.357Z [ERROR] server-8516e85f-e82e-9856-b696-b2add172e417: failed to get log: index=23 error="log not found" - testing.go:153: 2021-07-10T01:05:06.357Z [INFO] creating new snapshot: path=/tmp/raft256942961/snapshots/2-1013-1625879106357.tmp -testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.357Z [ERROR] server-8516e85f-e82e-9856-b696-b2add172e417: failed to get log: index=23 error="log not found" - testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.357Z [ERROR] server-8516e85f-e82e-9856-b696-b2add172e417: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-79f52308-c464-7454-e3f1-a5714823a1b1: 2021-07-10T01:05:06.358Z [INFO] server-79f52308-c464-7454-e3f1-a5714823a1b1: copied to local snapshot: bytes=71 - testing.go:149: server-79f52308-c464-7454-e3f1-a5714823a1b1: 2021-07-10T01:05:06.358Z [INFO] server-79f52308-c464-7454-e3f1-a5714823a1b1: copied to local snapshot: bytes=71 - testing.go:153: 2021-07-10T01:05:06.358Z [INFO] creating new snapshot: path=/tmp/raft659057319/snapshots/2-1013-1625879106358.tmp -testLoggerAdapter verbose: server-79f52308-c464-7454-e3f1-a5714823a1b1: 2021-07-10T01:05:06.358Z [INFO] server-79f52308-c464-7454-e3f1-a5714823a1b1: Installed remote snapshot - testing.go:149: server-79f52308-c464-7454-e3f1-a5714823a1b1: 2021-07-10T01:05:06.358Z [INFO] server-79f52308-c464-7454-e3f1-a5714823a1b1: Installed remote snapshot -testLoggerAdapter verbose: server-79f52308-c464-7454-e3f1-a5714823a1b1: 2021-07-10T01:05:06.358Z [WARN] server-79f52308-c464-7454-e3f1-a5714823a1b1: skipping application of old log: index=22 - testing.go:149: server-79f52308-c464-7454-e3f1-a5714823a1b1: 2021-07-10T01:05:06.358Z [WARN] server-79f52308-c464-7454-e3f1-a5714823a1b1: skipping application of old log: index=22 -testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.358Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: pipelining replication: peer="{Voter server-79f52308-c464-7454-e3f1-a5714823a1b1 79f52308-c464-7454-e3f1-a5714823a1b1}" - testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.358Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: pipelining replication: peer="{Voter server-79f52308-c464-7454-e3f1-a5714823a1b1 79f52308-c464-7454-e3f1-a5714823a1b1}" -testLoggerAdapter verbose: server-186970eb-04e2-8b42-4a61-7c4501527a25: 2021-07-10T01:05:06.359Z [INFO] server-186970eb-04e2-8b42-4a61-7c4501527a25: copied to local snapshot: bytes=71 - testing.go:149: server-186970eb-04e2-8b42-4a61-7c4501527a25: 2021-07-10T01:05:06.359Z [INFO] server-186970eb-04e2-8b42-4a61-7c4501527a25: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-186970eb-04e2-8b42-4a61-7c4501527a25: 2021-07-10T01:05:06.359Z [INFO] server-186970eb-04e2-8b42-4a61-7c4501527a25: Installed remote snapshot - testing.go:149: server-186970eb-04e2-8b42-4a61-7c4501527a25: 2021-07-10T01:05:06.359Z [INFO] server-186970eb-04e2-8b42-4a61-7c4501527a25: Installed remote snapshot -testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.359Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: pipelining replication: peer="{Voter server-186970eb-04e2-8b42-4a61-7c4501527a25 186970eb-04e2-8b42-4a61-7c4501527a25}" - testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.359Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: pipelining replication: peer="{Voter server-186970eb-04e2-8b42-4a61-7c4501527a25 186970eb-04e2-8b42-4a61-7c4501527a25}" -testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.374Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: aborting pipeline replication: peer="{Voter server-79f52308-c464-7454-e3f1-a5714823a1b1 79f52308-c464-7454-e3f1-a5714823a1b1}" - testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.374Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: aborting pipeline replication: peer="{Voter server-79f52308-c464-7454-e3f1-a5714823a1b1 79f52308-c464-7454-e3f1-a5714823a1b1}" -testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.374Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: aborting pipeline replication: peer="{Voter server-186970eb-04e2-8b42-4a61-7c4501527a25 186970eb-04e2-8b42-4a61-7c4501527a25}" - testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.374Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: aborting pipeline replication: peer="{Voter server-186970eb-04e2-8b42-4a61-7c4501527a25 186970eb-04e2-8b42-4a61-7c4501527a25}" -testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.376Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8 Address:4b242baa-3ccf-bf7a-add2-3bc14bba42b8} {Suffrage:Voter ID:server-c04fd653-66f5-6ead-e982-aafeff94b195 Address:c04fd653-66f5-6ead-e982-aafeff94b195} {Suffrage:Voter ID:server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2 Address:7e2b8bf7-5150-d051-2dbd-f464df97f8a2}]" - testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.376Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8 Address:4b242baa-3ccf-bf7a-add2-3bc14bba42b8} {Suffrage:Voter ID:server-c04fd653-66f5-6ead-e982-aafeff94b195 Address:c04fd653-66f5-6ead-e982-aafeff94b195} {Suffrage:Voter ID:server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2 Address:7e2b8bf7-5150-d051-2dbd-f464df97f8a2}]" -testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.376Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: entering follower state: follower="Node at 4b242baa-3ccf-bf7a-add2-3bc14bba42b8 [Follower]" leader= - testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.376Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: entering follower state: follower="Node at 4b242baa-3ccf-bf7a-add2-3bc14bba42b8 [Follower]" leader= -testLoggerAdapter verbose: server-c04fd653-66f5-6ead-e982-aafeff94b195: 2021-07-10T01:05:06.376Z [INFO] server-c04fd653-66f5-6ead-e982-aafeff94b195: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8 Address:4b242baa-3ccf-bf7a-add2-3bc14bba42b8} {Suffrage:Voter ID:server-c04fd653-66f5-6ead-e982-aafeff94b195 Address:c04fd653-66f5-6ead-e982-aafeff94b195} {Suffrage:Voter ID:server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2 Address:7e2b8bf7-5150-d051-2dbd-f464df97f8a2}]" - testing.go:149: server-c04fd653-66f5-6ead-e982-aafeff94b195: 2021-07-10T01:05:06.376Z [INFO] server-c04fd653-66f5-6ead-e982-aafeff94b195: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8 Address:4b242baa-3ccf-bf7a-add2-3bc14bba42b8} {Suffrage:Voter ID:server-c04fd653-66f5-6ead-e982-aafeff94b195 Address:c04fd653-66f5-6ead-e982-aafeff94b195} {Suffrage:Voter ID:server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2 Address:7e2b8bf7-5150-d051-2dbd-f464df97f8a2}]" -testLoggerAdapter verbose: server-c04fd653-66f5-6ead-e982-aafeff94b195: 2021-07-10T01:05:06.376Z [INFO] server-c04fd653-66f5-6ead-e982-aafeff94b195: entering follower state: follower="Node at c04fd653-66f5-6ead-e982-aafeff94b195 [Follower]" leader= - testing.go:149: server-c04fd653-66f5-6ead-e982-aafeff94b195: 2021-07-10T01:05:06.376Z [INFO] server-c04fd653-66f5-6ead-e982-aafeff94b195: entering follower state: follower="Node at c04fd653-66f5-6ead-e982-aafeff94b195 [Follower]" leader= -testLoggerAdapter verbose: server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: 2021-07-10T01:05:06.376Z [INFO] server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8 Address:4b242baa-3ccf-bf7a-add2-3bc14bba42b8} {Suffrage:Voter ID:server-c04fd653-66f5-6ead-e982-aafeff94b195 Address:c04fd653-66f5-6ead-e982-aafeff94b195} {Suffrage:Voter ID:server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2 Address:7e2b8bf7-5150-d051-2dbd-f464df97f8a2}]" - testing.go:149: server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: 2021-07-10T01:05:06.376Z [INFO] server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8 Address:4b242baa-3ccf-bf7a-add2-3bc14bba42b8} {Suffrage:Voter ID:server-c04fd653-66f5-6ead-e982-aafeff94b195 Address:c04fd653-66f5-6ead-e982-aafeff94b195} {Suffrage:Voter ID:server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2 Address:7e2b8bf7-5150-d051-2dbd-f464df97f8a2}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:06.377Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:06.377Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: 2021-07-10T01:05:06.377Z [INFO] server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: entering follower state: follower="Node at 7e2b8bf7-5150-d051-2dbd-f464df97f8a2 [Follower]" leader= - testing.go:149: server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: 2021-07-10T01:05:06.377Z [INFO] server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: entering follower state: follower="Node at 7e2b8bf7-5150-d051-2dbd-f464df97f8a2 [Follower]" leader= -testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.447Z [WARN] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.447Z [WARN] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.447Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: entering candidate state: node="Node at 4b242baa-3ccf-bf7a-add2-3bc14bba42b8 [Candidate]" term=2 - testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.447Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: entering candidate state: node="Node at 4b242baa-3ccf-bf7a-add2-3bc14bba42b8 [Candidate]" term=2 -testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.448Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: election won: tally=2 - testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.448Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: election won: tally=2 -testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.448Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: entering leader state: leader="Node at 4b242baa-3ccf-bf7a-add2-3bc14bba42b8 [Leader]" - testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.448Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: entering leader state: leader="Node at 4b242baa-3ccf-bf7a-add2-3bc14bba42b8 [Leader]" -testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.448Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: added peer, starting replication: peer=server-c04fd653-66f5-6ead-e982-aafeff94b195 - testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.448Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: added peer, starting replication: peer=server-c04fd653-66f5-6ead-e982-aafeff94b195 -testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.448Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: added peer, starting replication: peer=server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2 - testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.448Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: added peer, starting replication: peer=server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2 -testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.448Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: pipelining replication: peer="{Voter server-c04fd653-66f5-6ead-e982-aafeff94b195 c04fd653-66f5-6ead-e982-aafeff94b195}" - testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.448Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: pipelining replication: peer="{Voter server-c04fd653-66f5-6ead-e982-aafeff94b195 c04fd653-66f5-6ead-e982-aafeff94b195}" -testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.448Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: pipelining replication: peer="{Voter server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2 7e2b8bf7-5150-d051-2dbd-f464df97f8a2}" - testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.448Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: pipelining replication: peer="{Voter server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2 7e2b8bf7-5150-d051-2dbd-f464df97f8a2}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:06.553Z [INFO] stable state for Leader reached at 2021-07-10 01:05:06.448239753 +0000 UTC m=+11.836397763 (1 nodes), 71.121669ms from start of poll, 72.09672ms from cluster start. Timeout at 2021-07-10 01:05:06.553318281 +0000 UTC m=+11.941476291, 105.078528ms after stability - testing.go:149: cluster: 2021-07-10T01:05:06.553Z [INFO] stable state for Leader reached at 2021-07-10 01:05:06.448239753 +0000 UTC m=+11.836397763 (1 nodes), 71.121669ms from start of poll, 72.09672ms from cluster start. Timeout at 2021-07-10 01:05:06.553318281 +0000 UTC m=+11.941476291, 105.078528ms after stability -testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.554Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: starting snapshot up to: index=12 - testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.554Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: starting snapshot up to: index=12 - testing.go:153: 2021-07-10T01:05:06.554Z [INFO] creating new snapshot: path=/tmp/raft174448843/snapshots/2-12-1625879106554.tmp -testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.556Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: snapshot complete up to: index=12 - testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.556Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: snapshot complete up to: index=12 - testing.go:153: 2021-07-10T01:05:06.557Z [INFO] creating new snapshot: path=/tmp/raft174448843/snapshots/2-10013-1625879106557.tmp -testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.558Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: copied to local snapshot: bytes=71 - testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.558Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.558Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: restored user snapshot: index=1 - testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.558Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: restored user snapshot: index=1 -testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.558Z [ERROR] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: failed to get log: index=23 error="log not found" - testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.558Z [ERROR] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.558Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: aborting pipeline replication: peer="{Voter server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2 7e2b8bf7-5150-d051-2dbd-f464df97f8a2}" - testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.558Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: aborting pipeline replication: peer="{Voter server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2 7e2b8bf7-5150-d051-2dbd-f464df97f8a2}" -testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.559Z [ERROR] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: failed to get log: index=23 error="log not found" - testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.559Z [ERROR] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.559Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: aborting pipeline replication: peer="{Voter server-c04fd653-66f5-6ead-e982-aafeff94b195 c04fd653-66f5-6ead-e982-aafeff94b195}" - testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.559Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: aborting pipeline replication: peer="{Voter server-c04fd653-66f5-6ead-e982-aafeff94b195 c04fd653-66f5-6ead-e982-aafeff94b195}" -testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.565Z [ERROR] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: failed to get log: index=23 error="log not found" - testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.565Z [ERROR] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: failed to get log: index=23 error="log not found" - testing.go:153: 2021-07-10T01:05:06.565Z [INFO] creating new snapshot: path=/tmp/raft362405679/snapshots/2-10013-1625879106565.tmp -testLoggerAdapter verbose: server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: 2021-07-10T01:05:06.566Z [INFO] server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: copied to local snapshot: bytes=71 - testing.go:149: server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: 2021-07-10T01:05:06.566Z [INFO] server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.566Z [ERROR] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: failed to get log: index=23 error="log not found" - testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.566Z [ERROR] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: failed to get log: index=23 error="log not found" -testLoggerAdapter verbose: server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: 2021-07-10T01:05:06.567Z [INFO] server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: Installed remote snapshot - testing.go:149: server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: 2021-07-10T01:05:06.567Z [INFO] server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: Installed remote snapshot -testLoggerAdapter verbose: server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: 2021-07-10T01:05:06.567Z [WARN] server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: skipping application of old log: index=22 - testing.go:149: server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: 2021-07-10T01:05:06.567Z [WARN] server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: skipping application of old log: index=22 -testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.567Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: pipelining replication: peer="{Voter server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2 7e2b8bf7-5150-d051-2dbd-f464df97f8a2}" - testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.567Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: pipelining replication: peer="{Voter server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2 7e2b8bf7-5150-d051-2dbd-f464df97f8a2}" - testing.go:153: 2021-07-10T01:05:06.567Z [INFO] creating new snapshot: path=/tmp/raft418321205/snapshots/2-10013-1625879106567.tmp -testLoggerAdapter verbose: server-c04fd653-66f5-6ead-e982-aafeff94b195: 2021-07-10T01:05:06.568Z [INFO] server-c04fd653-66f5-6ead-e982-aafeff94b195: copied to local snapshot: bytes=71 - testing.go:149: server-c04fd653-66f5-6ead-e982-aafeff94b195: 2021-07-10T01:05:06.568Z [INFO] server-c04fd653-66f5-6ead-e982-aafeff94b195: copied to local snapshot: bytes=71 -testLoggerAdapter verbose: server-c04fd653-66f5-6ead-e982-aafeff94b195: 2021-07-10T01:05:06.568Z [INFO] server-c04fd653-66f5-6ead-e982-aafeff94b195: Installed remote snapshot - testing.go:149: server-c04fd653-66f5-6ead-e982-aafeff94b195: 2021-07-10T01:05:06.568Z [INFO] server-c04fd653-66f5-6ead-e982-aafeff94b195: Installed remote snapshot -testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.568Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: pipelining replication: peer="{Voter server-c04fd653-66f5-6ead-e982-aafeff94b195 c04fd653-66f5-6ead-e982-aafeff94b195}" - testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.568Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: pipelining replication: peer="{Voter server-c04fd653-66f5-6ead-e982-aafeff94b195 c04fd653-66f5-6ead-e982-aafeff94b195}" -testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.578Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: aborting pipeline replication: peer="{Voter server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2 7e2b8bf7-5150-d051-2dbd-f464df97f8a2}" - testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.578Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: aborting pipeline replication: peer="{Voter server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2 7e2b8bf7-5150-d051-2dbd-f464df97f8a2}" -testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.578Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: aborting pipeline replication: peer="{Voter server-c04fd653-66f5-6ead-e982-aafeff94b195 c04fd653-66f5-6ead-e982-aafeff94b195}" - testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.578Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: aborting pipeline replication: peer="{Voter server-c04fd653-66f5-6ead-e982-aafeff94b195 c04fd653-66f5-6ead-e982-aafeff94b195}" +testLoggerAdapter verbose: server-9a74a43e-9fe7-8590-f8fc-a63a25d37a26: 2022-08-12T07:30:09.835Z [INFO] server-9a74a43e-9fe7-8590-f8fc-a63a25d37a26: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a74a43e-9fe7-8590-f8fc-a63a25d37a26 Address:9a74a43e-9fe7-8590-f8fc-a63a25d37a26} {Suffrage:Voter ID:server-f57ebfe6-b579-3a87-ad6c-eed1175f33da Address:f57ebfe6-b579-3a87-ad6c-eed1175f33da} {Suffrage:Voter ID:server-eb9bcc87-8265-a4db-9d54-54005d4b22ef Address:eb9bcc87-8265-a4db-9d54-54005d4b22ef}]" + testing.go:149: server-9a74a43e-9fe7-8590-f8fc-a63a25d37a26: 2022-08-12T07:30:09.835Z [INFO] server-9a74a43e-9fe7-8590-f8fc-a63a25d37a26: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a74a43e-9fe7-8590-f8fc-a63a25d37a26 Address:9a74a43e-9fe7-8590-f8fc-a63a25d37a26} {Suffrage:Voter ID:server-f57ebfe6-b579-3a87-ad6c-eed1175f33da Address:f57ebfe6-b579-3a87-ad6c-eed1175f33da} {Suffrage:Voter ID:server-eb9bcc87-8265-a4db-9d54-54005d4b22ef Address:eb9bcc87-8265-a4db-9d54-54005d4b22ef}]" +testLoggerAdapter verbose: server-9a74a43e-9fe7-8590-f8fc-a63a25d37a26: 2022-08-12T07:30:09.835Z [INFO] server-9a74a43e-9fe7-8590-f8fc-a63a25d37a26: entering follower state: follower="Node at 9a74a43e-9fe7-8590-f8fc-a63a25d37a26 [Follower]" leader= + testing.go:149: server-9a74a43e-9fe7-8590-f8fc-a63a25d37a26: 2022-08-12T07:30:09.835Z [INFO] server-9a74a43e-9fe7-8590-f8fc-a63a25d37a26: entering follower state: follower="Node at 9a74a43e-9fe7-8590-f8fc-a63a25d37a26 [Follower]" leader= +testLoggerAdapter verbose: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:09.835Z [INFO] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a74a43e-9fe7-8590-f8fc-a63a25d37a26 Address:9a74a43e-9fe7-8590-f8fc-a63a25d37a26} {Suffrage:Voter ID:server-f57ebfe6-b579-3a87-ad6c-eed1175f33da Address:f57ebfe6-b579-3a87-ad6c-eed1175f33da} {Suffrage:Voter ID:server-eb9bcc87-8265-a4db-9d54-54005d4b22ef Address:eb9bcc87-8265-a4db-9d54-54005d4b22ef}]" + testing.go:149: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:09.835Z [INFO] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a74a43e-9fe7-8590-f8fc-a63a25d37a26 Address:9a74a43e-9fe7-8590-f8fc-a63a25d37a26} {Suffrage:Voter ID:server-f57ebfe6-b579-3a87-ad6c-eed1175f33da Address:f57ebfe6-b579-3a87-ad6c-eed1175f33da} {Suffrage:Voter ID:server-eb9bcc87-8265-a4db-9d54-54005d4b22ef Address:eb9bcc87-8265-a4db-9d54-54005d4b22ef}]" +testLoggerAdapter verbose: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:09.836Z [INFO] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: entering follower state: follower="Node at f57ebfe6-b579-3a87-ad6c-eed1175f33da [Follower]" leader= + testing.go:149: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:09.836Z [INFO] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: entering follower state: follower="Node at f57ebfe6-b579-3a87-ad6c-eed1175f33da [Follower]" leader= +testLoggerAdapter verbose: server-eb9bcc87-8265-a4db-9d54-54005d4b22ef: 2022-08-12T07:30:09.836Z [INFO] server-eb9bcc87-8265-a4db-9d54-54005d4b22ef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a74a43e-9fe7-8590-f8fc-a63a25d37a26 Address:9a74a43e-9fe7-8590-f8fc-a63a25d37a26} {Suffrage:Voter ID:server-f57ebfe6-b579-3a87-ad6c-eed1175f33da Address:f57ebfe6-b579-3a87-ad6c-eed1175f33da} {Suffrage:Voter ID:server-eb9bcc87-8265-a4db-9d54-54005d4b22ef Address:eb9bcc87-8265-a4db-9d54-54005d4b22ef}]" + testing.go:149: server-eb9bcc87-8265-a4db-9d54-54005d4b22ef: 2022-08-12T07:30:09.836Z [INFO] server-eb9bcc87-8265-a4db-9d54-54005d4b22ef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a74a43e-9fe7-8590-f8fc-a63a25d37a26 Address:9a74a43e-9fe7-8590-f8fc-a63a25d37a26} {Suffrage:Voter ID:server-f57ebfe6-b579-3a87-ad6c-eed1175f33da Address:f57ebfe6-b579-3a87-ad6c-eed1175f33da} {Suffrage:Voter ID:server-eb9bcc87-8265-a4db-9d54-54005d4b22ef Address:eb9bcc87-8265-a4db-9d54-54005d4b22ef}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:09.836Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-eb9bcc87-8265-a4db-9d54-54005d4b22ef: 2022-08-12T07:30:09.836Z [INFO] server-eb9bcc87-8265-a4db-9d54-54005d4b22ef: entering follower state: follower="Node at eb9bcc87-8265-a4db-9d54-54005d4b22ef [Follower]" leader= + testing.go:149: cluster: 2022-08-12T07:30:09.836Z [INFO] starting stability test: raft-state=Leader + testing.go:149: server-eb9bcc87-8265-a4db-9d54-54005d4b22ef: 2022-08-12T07:30:09.836Z [INFO] server-eb9bcc87-8265-a4db-9d54-54005d4b22ef: entering follower state: follower="Node at eb9bcc87-8265-a4db-9d54-54005d4b22ef [Follower]" leader= +testLoggerAdapter verbose: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:09.894Z [WARN] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:09.894Z [WARN] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:09.894Z [INFO] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: entering candidate state: node="Node at f57ebfe6-b579-3a87-ad6c-eed1175f33da [Candidate]" term=2 + testing.go:149: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:09.894Z [INFO] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: entering candidate state: node="Node at f57ebfe6-b579-3a87-ad6c-eed1175f33da [Candidate]" term=2 +testLoggerAdapter verbose: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:09.894Z [INFO] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: election won: tally=2 + testing.go:149: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:09.894Z [INFO] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: election won: tally=2 +testLoggerAdapter verbose: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:09.894Z [INFO] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: entering leader state: leader="Node at f57ebfe6-b579-3a87-ad6c-eed1175f33da [Leader]" + testing.go:149: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:09.894Z [INFO] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: entering leader state: leader="Node at f57ebfe6-b579-3a87-ad6c-eed1175f33da [Leader]" +testLoggerAdapter verbose: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:09.894Z [INFO] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: added peer, starting replication: peer=server-9a74a43e-9fe7-8590-f8fc-a63a25d37a26 + testing.go:149: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:09.894Z [INFO] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: added peer, starting replication: peer=server-9a74a43e-9fe7-8590-f8fc-a63a25d37a26 +testLoggerAdapter verbose: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:09.894Z [INFO] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: added peer, starting replication: peer=server-eb9bcc87-8265-a4db-9d54-54005d4b22ef + testing.go:149: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:09.894Z [INFO] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: added peer, starting replication: peer=server-eb9bcc87-8265-a4db-9d54-54005d4b22ef +testLoggerAdapter verbose: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:09.894Z [INFO] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: pipelining replication: peer="{Voter server-9a74a43e-9fe7-8590-f8fc-a63a25d37a26 9a74a43e-9fe7-8590-f8fc-a63a25d37a26}" + testing.go:149: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:09.894Z [INFO] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: pipelining replication: peer="{Voter server-9a74a43e-9fe7-8590-f8fc-a63a25d37a26 9a74a43e-9fe7-8590-f8fc-a63a25d37a26}" +testLoggerAdapter verbose: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:09.894Z [INFO] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: pipelining replication: peer="{Voter server-eb9bcc87-8265-a4db-9d54-54005d4b22ef eb9bcc87-8265-a4db-9d54-54005d4b22ef}" + testing.go:149: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:09.894Z [INFO] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: pipelining replication: peer="{Voter server-eb9bcc87-8265-a4db-9d54-54005d4b22ef eb9bcc87-8265-a4db-9d54-54005d4b22ef}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:09.999Z [INFO] stable state for Leader reached at 2022-08-12 07:30:09.89470326 +0000 UTC m=+10.851389761 (1 nodes), 58.32876ms from start of poll, 59.20452ms from cluster start. Timeout at 2022-08-12 07:30:09.99977848 +0000 UTC m=+10.956464981, 105.07522ms after stability + testing.go:149: cluster: 2022-08-12T07:30:09.999Z [INFO] stable state for Leader reached at 2022-08-12 07:30:09.89470326 +0000 UTC m=+10.851389761 (1 nodes), 58.32876ms from start of poll, 59.20452ms from cluster start. Timeout at 2022-08-12 07:30:09.99977848 +0000 UTC m=+10.956464981, 105.07522ms after stability +testLoggerAdapter verbose: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:10.000Z [INFO] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: starting snapshot up to: index=12 + testing.go:149: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:10.000Z [INFO] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: starting snapshot up to: index=12 + testing.go:153: 2022-08-12T07:30:10.000Z [INFO] creating new snapshot: path=/tmp/raft995349395/snapshots/2-12-1660289410000.tmp +testLoggerAdapter verbose: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:10.001Z [INFO] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: snapshot complete up to: index=12 + testing.go:149: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:10.001Z [INFO] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: snapshot complete up to: index=12 + testing.go:153: 2022-08-12T07:30:10.001Z [INFO] creating new snapshot: path=/tmp/raft995349395/snapshots/2-23-1660289410001.tmp +testLoggerAdapter verbose: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:10.002Z [INFO] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: copied to local snapshot: bytes=71 + testing.go:149: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:10.002Z [INFO] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:10.002Z [INFO] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: restored user snapshot: index=1 + testing.go:149: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:10.002Z [INFO] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: restored user snapshot: index=1 +testLoggerAdapter verbose: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:10.002Z [ERROR] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: failed to get log: index=23 error="log not found" + testing.go:149: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:10.002Z [ERROR] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:10.002Z [ERROR] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: failed to get log: index=23 error="log not found" + testing.go:149: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:10.002Z [ERROR] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:10.003Z [INFO] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: aborting pipeline replication: peer="{Voter server-9a74a43e-9fe7-8590-f8fc-a63a25d37a26 9a74a43e-9fe7-8590-f8fc-a63a25d37a26}" + testing.go:149: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:10.003Z [INFO] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: aborting pipeline replication: peer="{Voter server-9a74a43e-9fe7-8590-f8fc-a63a25d37a26 9a74a43e-9fe7-8590-f8fc-a63a25d37a26}" +testLoggerAdapter verbose: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:10.003Z [INFO] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: aborting pipeline replication: peer="{Voter server-eb9bcc87-8265-a4db-9d54-54005d4b22ef eb9bcc87-8265-a4db-9d54-54005d4b22ef}" + testing.go:149: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:10.003Z [INFO] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: aborting pipeline replication: peer="{Voter server-eb9bcc87-8265-a4db-9d54-54005d4b22ef eb9bcc87-8265-a4db-9d54-54005d4b22ef}" +testLoggerAdapter verbose: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:10.010Z [ERROR] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: failed to get log: index=23 error="log not found" + testing.go:149: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:10.010Z [ERROR] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: failed to get log: index=23 error="log not found" + testing.go:153: 2022-08-12T07:30:10.011Z [INFO] creating new snapshot: path=/tmp/raft133583609/snapshots/2-23-1660289410011.tmp +testLoggerAdapter verbose: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:10.011Z [ERROR] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: failed to get log: index=23 error="log not found" + testing.go:149: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:10.011Z [ERROR] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-9a74a43e-9fe7-8590-f8fc-a63a25d37a26: 2022-08-12T07:30:10.011Z [INFO] server-9a74a43e-9fe7-8590-f8fc-a63a25d37a26: copied to local snapshot: bytes=71 + testing.go:149: server-9a74a43e-9fe7-8590-f8fc-a63a25d37a26: 2022-08-12T07:30:10.011Z [INFO] server-9a74a43e-9fe7-8590-f8fc-a63a25d37a26: copied to local snapshot: bytes=71 + testing.go:153: 2022-08-12T07:30:10.012Z [INFO] creating new snapshot: path=/tmp/raft249014589/snapshots/2-23-1660289410012.tmp +testLoggerAdapter verbose: server-9a74a43e-9fe7-8590-f8fc-a63a25d37a26: 2022-08-12T07:30:10.012Z [INFO] server-9a74a43e-9fe7-8590-f8fc-a63a25d37a26: Installed remote snapshot + testing.go:149: server-9a74a43e-9fe7-8590-f8fc-a63a25d37a26: 2022-08-12T07:30:10.012Z [INFO] server-9a74a43e-9fe7-8590-f8fc-a63a25d37a26: Installed remote snapshot +testLoggerAdapter verbose: server-9a74a43e-9fe7-8590-f8fc-a63a25d37a26: 2022-08-12T07:30:10.012Z [WARN] server-9a74a43e-9fe7-8590-f8fc-a63a25d37a26: skipping application of old log: index=22 + testing.go:149: server-9a74a43e-9fe7-8590-f8fc-a63a25d37a26: 2022-08-12T07:30:10.012Z [WARN] server-9a74a43e-9fe7-8590-f8fc-a63a25d37a26: skipping application of old log: index=22 +testLoggerAdapter verbose: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:10.012Z [INFO] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: pipelining replication: peer="{Voter server-9a74a43e-9fe7-8590-f8fc-a63a25d37a26 9a74a43e-9fe7-8590-f8fc-a63a25d37a26}" + testing.go:149: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:10.012Z [INFO] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: pipelining replication: peer="{Voter server-9a74a43e-9fe7-8590-f8fc-a63a25d37a26 9a74a43e-9fe7-8590-f8fc-a63a25d37a26}" +testLoggerAdapter verbose: server-eb9bcc87-8265-a4db-9d54-54005d4b22ef: 2022-08-12T07:30:10.012Z [INFO] server-eb9bcc87-8265-a4db-9d54-54005d4b22ef: copied to local snapshot: bytes=71 + testing.go:149: server-eb9bcc87-8265-a4db-9d54-54005d4b22ef: 2022-08-12T07:30:10.012Z [INFO] server-eb9bcc87-8265-a4db-9d54-54005d4b22ef: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-eb9bcc87-8265-a4db-9d54-54005d4b22ef: 2022-08-12T07:30:10.012Z [INFO] server-eb9bcc87-8265-a4db-9d54-54005d4b22ef: Installed remote snapshot + testing.go:149: server-eb9bcc87-8265-a4db-9d54-54005d4b22ef: 2022-08-12T07:30:10.012Z [INFO] server-eb9bcc87-8265-a4db-9d54-54005d4b22ef: Installed remote snapshot +testLoggerAdapter verbose: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:10.013Z [INFO] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: pipelining replication: peer="{Voter server-eb9bcc87-8265-a4db-9d54-54005d4b22ef eb9bcc87-8265-a4db-9d54-54005d4b22ef}" + testing.go:149: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:10.013Z [INFO] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: pipelining replication: peer="{Voter server-eb9bcc87-8265-a4db-9d54-54005d4b22ef eb9bcc87-8265-a4db-9d54-54005d4b22ef}" +testLoggerAdapter verbose: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:10.028Z [INFO] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: aborting pipeline replication: peer="{Voter server-eb9bcc87-8265-a4db-9d54-54005d4b22ef eb9bcc87-8265-a4db-9d54-54005d4b22ef}" + testing.go:149: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:10.028Z [INFO] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: aborting pipeline replication: peer="{Voter server-eb9bcc87-8265-a4db-9d54-54005d4b22ef eb9bcc87-8265-a4db-9d54-54005d4b22ef}" +testLoggerAdapter verbose: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:10.028Z [INFO] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: aborting pipeline replication: peer="{Voter server-9a74a43e-9fe7-8590-f8fc-a63a25d37a26 9a74a43e-9fe7-8590-f8fc-a63a25d37a26}" + testing.go:149: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:10.028Z [INFO] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: aborting pipeline replication: peer="{Voter server-9a74a43e-9fe7-8590-f8fc-a63a25d37a26 9a74a43e-9fe7-8590-f8fc-a63a25d37a26}" +testLoggerAdapter verbose: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:10.078Z [ERROR] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: failed to heartbeat to: peer=9a74a43e-9fe7-8590-f8fc-a63a25d37a26 error="command timed out" + testing.go:149: server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: 2022-08-12T07:30:10.078Z [ERROR] server-f57ebfe6-b579-3a87-ad6c-eed1175f33da: failed to heartbeat to: peer=9a74a43e-9fe7-8590-f8fc-a63a25d37a26 error="command timed out" +testLoggerAdapter verbose: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.080Z [INFO] server-f881e18a-a85a-2003-b1b9-277998be7d87: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f881e18a-a85a-2003-b1b9-277998be7d87 Address:f881e18a-a85a-2003-b1b9-277998be7d87} {Suffrage:Voter ID:server-b1555f43-50e9-11f6-474f-c520ad80480f Address:b1555f43-50e9-11f6-474f-c520ad80480f} {Suffrage:Voter ID:server-b75b87d5-15ed-3708-ca48-1afdae68e09d Address:b75b87d5-15ed-3708-ca48-1afdae68e09d}]" + testing.go:149: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.080Z [INFO] server-f881e18a-a85a-2003-b1b9-277998be7d87: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f881e18a-a85a-2003-b1b9-277998be7d87 Address:f881e18a-a85a-2003-b1b9-277998be7d87} {Suffrage:Voter ID:server-b1555f43-50e9-11f6-474f-c520ad80480f Address:b1555f43-50e9-11f6-474f-c520ad80480f} {Suffrage:Voter ID:server-b75b87d5-15ed-3708-ca48-1afdae68e09d Address:b75b87d5-15ed-3708-ca48-1afdae68e09d}]" +testLoggerAdapter verbose: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.080Z [INFO] server-f881e18a-a85a-2003-b1b9-277998be7d87: entering follower state: follower="Node at f881e18a-a85a-2003-b1b9-277998be7d87 [Follower]" leader= + testing.go:149: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.080Z [INFO] server-f881e18a-a85a-2003-b1b9-277998be7d87: entering follower state: follower="Node at f881e18a-a85a-2003-b1b9-277998be7d87 [Follower]" leader= +testLoggerAdapter verbose: server-b1555f43-50e9-11f6-474f-c520ad80480f: 2022-08-12T07:30:10.080Z [INFO] server-b1555f43-50e9-11f6-474f-c520ad80480f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f881e18a-a85a-2003-b1b9-277998be7d87 Address:f881e18a-a85a-2003-b1b9-277998be7d87} {Suffrage:Voter ID:server-b1555f43-50e9-11f6-474f-c520ad80480f Address:b1555f43-50e9-11f6-474f-c520ad80480f} {Suffrage:Voter ID:server-b75b87d5-15ed-3708-ca48-1afdae68e09d Address:b75b87d5-15ed-3708-ca48-1afdae68e09d}]" + testing.go:149: server-b1555f43-50e9-11f6-474f-c520ad80480f: 2022-08-12T07:30:10.080Z [INFO] server-b1555f43-50e9-11f6-474f-c520ad80480f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f881e18a-a85a-2003-b1b9-277998be7d87 Address:f881e18a-a85a-2003-b1b9-277998be7d87} {Suffrage:Voter ID:server-b1555f43-50e9-11f6-474f-c520ad80480f Address:b1555f43-50e9-11f6-474f-c520ad80480f} {Suffrage:Voter ID:server-b75b87d5-15ed-3708-ca48-1afdae68e09d Address:b75b87d5-15ed-3708-ca48-1afdae68e09d}]" +testLoggerAdapter verbose: server-b1555f43-50e9-11f6-474f-c520ad80480f: 2022-08-12T07:30:10.080Z [INFO] server-b1555f43-50e9-11f6-474f-c520ad80480f: entering follower state: follower="Node at b1555f43-50e9-11f6-474f-c520ad80480f [Follower]" leader= + testing.go:149: server-b1555f43-50e9-11f6-474f-c520ad80480f: 2022-08-12T07:30:10.080Z [INFO] server-b1555f43-50e9-11f6-474f-c520ad80480f: entering follower state: follower="Node at b1555f43-50e9-11f6-474f-c520ad80480f [Follower]" leader= +testLoggerAdapter verbose: server-b75b87d5-15ed-3708-ca48-1afdae68e09d: 2022-08-12T07:30:10.081Z [INFO] server-b75b87d5-15ed-3708-ca48-1afdae68e09d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f881e18a-a85a-2003-b1b9-277998be7d87 Address:f881e18a-a85a-2003-b1b9-277998be7d87} {Suffrage:Voter ID:server-b1555f43-50e9-11f6-474f-c520ad80480f Address:b1555f43-50e9-11f6-474f-c520ad80480f} {Suffrage:Voter ID:server-b75b87d5-15ed-3708-ca48-1afdae68e09d Address:b75b87d5-15ed-3708-ca48-1afdae68e09d}]" + testing.go:149: server-b75b87d5-15ed-3708-ca48-1afdae68e09d: 2022-08-12T07:30:10.081Z [INFO] server-b75b87d5-15ed-3708-ca48-1afdae68e09d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f881e18a-a85a-2003-b1b9-277998be7d87 Address:f881e18a-a85a-2003-b1b9-277998be7d87} {Suffrage:Voter ID:server-b1555f43-50e9-11f6-474f-c520ad80480f Address:b1555f43-50e9-11f6-474f-c520ad80480f} {Suffrage:Voter ID:server-b75b87d5-15ed-3708-ca48-1afdae68e09d Address:b75b87d5-15ed-3708-ca48-1afdae68e09d}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:10.081Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-b75b87d5-15ed-3708-ca48-1afdae68e09d: 2022-08-12T07:30:10.081Z [INFO] server-b75b87d5-15ed-3708-ca48-1afdae68e09d: entering follower state: follower="Node at b75b87d5-15ed-3708-ca48-1afdae68e09d [Follower]" leader= + testing.go:149: cluster: 2022-08-12T07:30:10.081Z [INFO] starting stability test: raft-state=Leader + testing.go:149: server-b75b87d5-15ed-3708-ca48-1afdae68e09d: 2022-08-12T07:30:10.081Z [INFO] server-b75b87d5-15ed-3708-ca48-1afdae68e09d: entering follower state: follower="Node at b75b87d5-15ed-3708-ca48-1afdae68e09d [Follower]" leader= +testLoggerAdapter verbose: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.135Z [WARN] server-f881e18a-a85a-2003-b1b9-277998be7d87: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.135Z [WARN] server-f881e18a-a85a-2003-b1b9-277998be7d87: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.135Z [INFO] server-f881e18a-a85a-2003-b1b9-277998be7d87: entering candidate state: node="Node at f881e18a-a85a-2003-b1b9-277998be7d87 [Candidate]" term=2 + testing.go:149: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.135Z [INFO] server-f881e18a-a85a-2003-b1b9-277998be7d87: entering candidate state: node="Node at f881e18a-a85a-2003-b1b9-277998be7d87 [Candidate]" term=2 +testLoggerAdapter verbose: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.136Z [INFO] server-f881e18a-a85a-2003-b1b9-277998be7d87: election won: tally=2 + testing.go:149: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.136Z [INFO] server-f881e18a-a85a-2003-b1b9-277998be7d87: election won: tally=2 +testLoggerAdapter verbose: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.136Z [INFO] server-f881e18a-a85a-2003-b1b9-277998be7d87: entering leader state: leader="Node at f881e18a-a85a-2003-b1b9-277998be7d87 [Leader]" + testing.go:149: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.136Z [INFO] server-f881e18a-a85a-2003-b1b9-277998be7d87: entering leader state: leader="Node at f881e18a-a85a-2003-b1b9-277998be7d87 [Leader]" +testLoggerAdapter verbose: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.136Z [INFO] server-f881e18a-a85a-2003-b1b9-277998be7d87: added peer, starting replication: peer=server-b1555f43-50e9-11f6-474f-c520ad80480f + testing.go:149: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.136Z [INFO] server-f881e18a-a85a-2003-b1b9-277998be7d87: added peer, starting replication: peer=server-b1555f43-50e9-11f6-474f-c520ad80480f +testLoggerAdapter verbose: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.136Z [INFO] server-f881e18a-a85a-2003-b1b9-277998be7d87: added peer, starting replication: peer=server-b75b87d5-15ed-3708-ca48-1afdae68e09d + testing.go:149: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.136Z [INFO] server-f881e18a-a85a-2003-b1b9-277998be7d87: added peer, starting replication: peer=server-b75b87d5-15ed-3708-ca48-1afdae68e09d +testLoggerAdapter verbose: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.136Z [INFO] server-f881e18a-a85a-2003-b1b9-277998be7d87: pipelining replication: peer="{Voter server-b1555f43-50e9-11f6-474f-c520ad80480f b1555f43-50e9-11f6-474f-c520ad80480f}" + testing.go:149: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.136Z [INFO] server-f881e18a-a85a-2003-b1b9-277998be7d87: pipelining replication: peer="{Voter server-b1555f43-50e9-11f6-474f-c520ad80480f b1555f43-50e9-11f6-474f-c520ad80480f}" +testLoggerAdapter verbose: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.136Z [INFO] server-f881e18a-a85a-2003-b1b9-277998be7d87: pipelining replication: peer="{Voter server-b75b87d5-15ed-3708-ca48-1afdae68e09d b75b87d5-15ed-3708-ca48-1afdae68e09d}" + testing.go:149: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.136Z [INFO] server-f881e18a-a85a-2003-b1b9-277998be7d87: pipelining replication: peer="{Voter server-b75b87d5-15ed-3708-ca48-1afdae68e09d b75b87d5-15ed-3708-ca48-1afdae68e09d}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:10.241Z [INFO] stable state for Leader reached at 2022-08-12 07:30:10.13629202 +0000 UTC m=+11.092978501 (1 nodes), 55.05754ms from start of poll, 56.02108ms from cluster start. Timeout at 2022-08-12 07:30:10.2413536 +0000 UTC m=+11.198040081, 105.06158ms after stability + testing.go:149: cluster: 2022-08-12T07:30:10.241Z [INFO] stable state for Leader reached at 2022-08-12 07:30:10.13629202 +0000 UTC m=+11.092978501 (1 nodes), 55.05754ms from start of poll, 56.02108ms from cluster start. Timeout at 2022-08-12 07:30:10.2413536 +0000 UTC m=+11.198040081, 105.06158ms after stability +testLoggerAdapter verbose: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.242Z [INFO] server-f881e18a-a85a-2003-b1b9-277998be7d87: starting snapshot up to: index=12 + testing.go:149: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.242Z [INFO] server-f881e18a-a85a-2003-b1b9-277998be7d87: starting snapshot up to: index=12 + testing.go:153: 2022-08-12T07:30:10.242Z [INFO] creating new snapshot: path=/tmp/raft349752951/snapshots/2-12-1660289410242.tmp +testLoggerAdapter verbose: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.242Z [INFO] server-f881e18a-a85a-2003-b1b9-277998be7d87: snapshot complete up to: index=12 + testing.go:149: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.242Z [INFO] server-f881e18a-a85a-2003-b1b9-277998be7d87: snapshot complete up to: index=12 + testing.go:153: 2022-08-12T07:30:10.243Z [INFO] creating new snapshot: path=/tmp/raft349752951/snapshots/2-23-1660289410243.tmp +testLoggerAdapter verbose: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.244Z [INFO] server-f881e18a-a85a-2003-b1b9-277998be7d87: copied to local snapshot: bytes=71 + testing.go:149: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.244Z [INFO] server-f881e18a-a85a-2003-b1b9-277998be7d87: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.244Z [INFO] server-f881e18a-a85a-2003-b1b9-277998be7d87: restored user snapshot: index=1 + testing.go:149: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.244Z [INFO] server-f881e18a-a85a-2003-b1b9-277998be7d87: restored user snapshot: index=1 +testLoggerAdapter verbose: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.244Z [ERROR] server-f881e18a-a85a-2003-b1b9-277998be7d87: failed to get log: index=23 error="log not found" + testing.go:149: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.244Z [ERROR] server-f881e18a-a85a-2003-b1b9-277998be7d87: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.244Z [ERROR] server-f881e18a-a85a-2003-b1b9-277998be7d87: failed to get log: index=23 error="log not found" + testing.go:149: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.244Z [ERROR] server-f881e18a-a85a-2003-b1b9-277998be7d87: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.245Z [INFO] server-f881e18a-a85a-2003-b1b9-277998be7d87: aborting pipeline replication: peer="{Voter server-b1555f43-50e9-11f6-474f-c520ad80480f b1555f43-50e9-11f6-474f-c520ad80480f}" + testing.go:149: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.245Z [INFO] server-f881e18a-a85a-2003-b1b9-277998be7d87: aborting pipeline replication: peer="{Voter server-b1555f43-50e9-11f6-474f-c520ad80480f b1555f43-50e9-11f6-474f-c520ad80480f}" +testLoggerAdapter verbose: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.244Z [INFO] server-f881e18a-a85a-2003-b1b9-277998be7d87: aborting pipeline replication: peer="{Voter server-b75b87d5-15ed-3708-ca48-1afdae68e09d b75b87d5-15ed-3708-ca48-1afdae68e09d}" + testing.go:149: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.244Z [INFO] server-f881e18a-a85a-2003-b1b9-277998be7d87: aborting pipeline replication: peer="{Voter server-b75b87d5-15ed-3708-ca48-1afdae68e09d b75b87d5-15ed-3708-ca48-1afdae68e09d}" +testLoggerAdapter verbose: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.254Z [ERROR] server-f881e18a-a85a-2003-b1b9-277998be7d87: failed to get log: index=23 error="log not found" + testing.go:149: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.254Z [ERROR] server-f881e18a-a85a-2003-b1b9-277998be7d87: failed to get log: index=23 error="log not found" + testing.go:153: 2022-08-12T07:30:10.255Z [INFO] creating new snapshot: path=/tmp/raft484105153/snapshots/2-23-1660289410255.tmp +testLoggerAdapter verbose: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.255Z [ERROR] server-f881e18a-a85a-2003-b1b9-277998be7d87: failed to get log: index=23 error="log not found" + testing.go:149: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.255Z [ERROR] server-f881e18a-a85a-2003-b1b9-277998be7d87: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-b1555f43-50e9-11f6-474f-c520ad80480f: 2022-08-12T07:30:10.256Z [INFO] server-b1555f43-50e9-11f6-474f-c520ad80480f: copied to local snapshot: bytes=71 + testing.go:149: server-b1555f43-50e9-11f6-474f-c520ad80480f: 2022-08-12T07:30:10.256Z [INFO] server-b1555f43-50e9-11f6-474f-c520ad80480f: copied to local snapshot: bytes=71 + testing.go:153: 2022-08-12T07:30:10.256Z [INFO] creating new snapshot: path=/tmp/raft780289947/snapshots/2-23-1660289410256.tmp +testLoggerAdapter verbose: server-b1555f43-50e9-11f6-474f-c520ad80480f: 2022-08-12T07:30:10.256Z [INFO] server-b1555f43-50e9-11f6-474f-c520ad80480f: Installed remote snapshot + testing.go:149: server-b1555f43-50e9-11f6-474f-c520ad80480f: 2022-08-12T07:30:10.256Z [INFO] server-b1555f43-50e9-11f6-474f-c520ad80480f: Installed remote snapshot +testLoggerAdapter verbose: server-b1555f43-50e9-11f6-474f-c520ad80480f: 2022-08-12T07:30:10.256Z [WARN] server-b1555f43-50e9-11f6-474f-c520ad80480f: skipping application of old log: index=22 + testing.go:149: server-b1555f43-50e9-11f6-474f-c520ad80480f: 2022-08-12T07:30:10.256Z [WARN] server-b1555f43-50e9-11f6-474f-c520ad80480f: skipping application of old log: index=22 +testLoggerAdapter verbose: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.256Z [INFO] server-f881e18a-a85a-2003-b1b9-277998be7d87: pipelining replication: peer="{Voter server-b1555f43-50e9-11f6-474f-c520ad80480f b1555f43-50e9-11f6-474f-c520ad80480f}" + testing.go:149: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.256Z [INFO] server-f881e18a-a85a-2003-b1b9-277998be7d87: pipelining replication: peer="{Voter server-b1555f43-50e9-11f6-474f-c520ad80480f b1555f43-50e9-11f6-474f-c520ad80480f}" +testLoggerAdapter verbose: server-b75b87d5-15ed-3708-ca48-1afdae68e09d: 2022-08-12T07:30:10.256Z [INFO] server-b75b87d5-15ed-3708-ca48-1afdae68e09d: copied to local snapshot: bytes=71 + testing.go:149: server-b75b87d5-15ed-3708-ca48-1afdae68e09d: 2022-08-12T07:30:10.256Z [INFO] server-b75b87d5-15ed-3708-ca48-1afdae68e09d: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-b75b87d5-15ed-3708-ca48-1afdae68e09d: 2022-08-12T07:30:10.257Z [INFO] server-b75b87d5-15ed-3708-ca48-1afdae68e09d: Installed remote snapshot + testing.go:149: server-b75b87d5-15ed-3708-ca48-1afdae68e09d: 2022-08-12T07:30:10.257Z [INFO] server-b75b87d5-15ed-3708-ca48-1afdae68e09d: Installed remote snapshot +testLoggerAdapter verbose: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.257Z [INFO] server-f881e18a-a85a-2003-b1b9-277998be7d87: pipelining replication: peer="{Voter server-b75b87d5-15ed-3708-ca48-1afdae68e09d b75b87d5-15ed-3708-ca48-1afdae68e09d}" + testing.go:149: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.257Z [INFO] server-f881e18a-a85a-2003-b1b9-277998be7d87: pipelining replication: peer="{Voter server-b75b87d5-15ed-3708-ca48-1afdae68e09d b75b87d5-15ed-3708-ca48-1afdae68e09d}" +testLoggerAdapter verbose: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.272Z [INFO] server-f881e18a-a85a-2003-b1b9-277998be7d87: aborting pipeline replication: peer="{Voter server-b1555f43-50e9-11f6-474f-c520ad80480f b1555f43-50e9-11f6-474f-c520ad80480f}" + testing.go:149: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.272Z [INFO] server-f881e18a-a85a-2003-b1b9-277998be7d87: aborting pipeline replication: peer="{Voter server-b1555f43-50e9-11f6-474f-c520ad80480f b1555f43-50e9-11f6-474f-c520ad80480f}" +testLoggerAdapter verbose: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.272Z [INFO] server-f881e18a-a85a-2003-b1b9-277998be7d87: aborting pipeline replication: peer="{Voter server-b75b87d5-15ed-3708-ca48-1afdae68e09d b75b87d5-15ed-3708-ca48-1afdae68e09d}" + testing.go:149: server-f881e18a-a85a-2003-b1b9-277998be7d87: 2022-08-12T07:30:10.272Z [INFO] server-f881e18a-a85a-2003-b1b9-277998be7d87: aborting pipeline replication: peer="{Voter server-b75b87d5-15ed-3708-ca48-1afdae68e09d b75b87d5-15ed-3708-ca48-1afdae68e09d}" +testLoggerAdapter verbose: server-6a5bce45-5390-2beb-45dd-7f157fc96e69: 2022-08-12T07:30:10.273Z [INFO] server-6a5bce45-5390-2beb-45dd-7f157fc96e69: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a5bce45-5390-2beb-45dd-7f157fc96e69 Address:6a5bce45-5390-2beb-45dd-7f157fc96e69} {Suffrage:Voter ID:server-b8fe715e-476a-d776-b535-ab2ac1f2dd73 Address:b8fe715e-476a-d776-b535-ab2ac1f2dd73} {Suffrage:Voter ID:server-89981a84-c9f9-4d23-22c3-f1fa42cac6c5 Address:89981a84-c9f9-4d23-22c3-f1fa42cac6c5}]" + testing.go:149: server-6a5bce45-5390-2beb-45dd-7f157fc96e69: 2022-08-12T07:30:10.273Z [INFO] server-6a5bce45-5390-2beb-45dd-7f157fc96e69: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a5bce45-5390-2beb-45dd-7f157fc96e69 Address:6a5bce45-5390-2beb-45dd-7f157fc96e69} {Suffrage:Voter ID:server-b8fe715e-476a-d776-b535-ab2ac1f2dd73 Address:b8fe715e-476a-d776-b535-ab2ac1f2dd73} {Suffrage:Voter ID:server-89981a84-c9f9-4d23-22c3-f1fa42cac6c5 Address:89981a84-c9f9-4d23-22c3-f1fa42cac6c5}]" +testLoggerAdapter verbose: server-6a5bce45-5390-2beb-45dd-7f157fc96e69: 2022-08-12T07:30:10.274Z [INFO] server-6a5bce45-5390-2beb-45dd-7f157fc96e69: entering follower state: follower="Node at 6a5bce45-5390-2beb-45dd-7f157fc96e69 [Follower]" leader= + testing.go:149: server-6a5bce45-5390-2beb-45dd-7f157fc96e69: 2022-08-12T07:30:10.274Z [INFO] server-6a5bce45-5390-2beb-45dd-7f157fc96e69: entering follower state: follower="Node at 6a5bce45-5390-2beb-45dd-7f157fc96e69 [Follower]" leader= +testLoggerAdapter verbose: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.274Z [INFO] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a5bce45-5390-2beb-45dd-7f157fc96e69 Address:6a5bce45-5390-2beb-45dd-7f157fc96e69} {Suffrage:Voter ID:server-b8fe715e-476a-d776-b535-ab2ac1f2dd73 Address:b8fe715e-476a-d776-b535-ab2ac1f2dd73} {Suffrage:Voter ID:server-89981a84-c9f9-4d23-22c3-f1fa42cac6c5 Address:89981a84-c9f9-4d23-22c3-f1fa42cac6c5}]" + testing.go:149: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.274Z [INFO] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a5bce45-5390-2beb-45dd-7f157fc96e69 Address:6a5bce45-5390-2beb-45dd-7f157fc96e69} {Suffrage:Voter ID:server-b8fe715e-476a-d776-b535-ab2ac1f2dd73 Address:b8fe715e-476a-d776-b535-ab2ac1f2dd73} {Suffrage:Voter ID:server-89981a84-c9f9-4d23-22c3-f1fa42cac6c5 Address:89981a84-c9f9-4d23-22c3-f1fa42cac6c5}]" +testLoggerAdapter verbose: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.274Z [INFO] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: entering follower state: follower="Node at b8fe715e-476a-d776-b535-ab2ac1f2dd73 [Follower]" leader= + testing.go:149: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.274Z [INFO] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: entering follower state: follower="Node at b8fe715e-476a-d776-b535-ab2ac1f2dd73 [Follower]" leader= +testLoggerAdapter verbose: server-89981a84-c9f9-4d23-22c3-f1fa42cac6c5: 2022-08-12T07:30:10.274Z [INFO] server-89981a84-c9f9-4d23-22c3-f1fa42cac6c5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a5bce45-5390-2beb-45dd-7f157fc96e69 Address:6a5bce45-5390-2beb-45dd-7f157fc96e69} {Suffrage:Voter ID:server-b8fe715e-476a-d776-b535-ab2ac1f2dd73 Address:b8fe715e-476a-d776-b535-ab2ac1f2dd73} {Suffrage:Voter ID:server-89981a84-c9f9-4d23-22c3-f1fa42cac6c5 Address:89981a84-c9f9-4d23-22c3-f1fa42cac6c5}]" + testing.go:149: server-89981a84-c9f9-4d23-22c3-f1fa42cac6c5: 2022-08-12T07:30:10.274Z [INFO] server-89981a84-c9f9-4d23-22c3-f1fa42cac6c5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a5bce45-5390-2beb-45dd-7f157fc96e69 Address:6a5bce45-5390-2beb-45dd-7f157fc96e69} {Suffrage:Voter ID:server-b8fe715e-476a-d776-b535-ab2ac1f2dd73 Address:b8fe715e-476a-d776-b535-ab2ac1f2dd73} {Suffrage:Voter ID:server-89981a84-c9f9-4d23-22c3-f1fa42cac6c5 Address:89981a84-c9f9-4d23-22c3-f1fa42cac6c5}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:10.274Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-89981a84-c9f9-4d23-22c3-f1fa42cac6c5: 2022-08-12T07:30:10.274Z [INFO] server-89981a84-c9f9-4d23-22c3-f1fa42cac6c5: entering follower state: follower="Node at 89981a84-c9f9-4d23-22c3-f1fa42cac6c5 [Follower]" leader= + testing.go:149: cluster: 2022-08-12T07:30:10.274Z [INFO] starting stability test: raft-state=Leader + testing.go:149: server-89981a84-c9f9-4d23-22c3-f1fa42cac6c5: 2022-08-12T07:30:10.274Z [INFO] server-89981a84-c9f9-4d23-22c3-f1fa42cac6c5: entering follower state: follower="Node at 89981a84-c9f9-4d23-22c3-f1fa42cac6c5 [Follower]" leader= +testLoggerAdapter verbose: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.342Z [WARN] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.342Z [WARN] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.342Z [INFO] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: entering candidate state: node="Node at b8fe715e-476a-d776-b535-ab2ac1f2dd73 [Candidate]" term=2 + testing.go:149: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.342Z [INFO] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: entering candidate state: node="Node at b8fe715e-476a-d776-b535-ab2ac1f2dd73 [Candidate]" term=2 +testLoggerAdapter verbose: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.342Z [INFO] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: election won: tally=2 + testing.go:149: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.342Z [INFO] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: election won: tally=2 +testLoggerAdapter verbose: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.342Z [INFO] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: entering leader state: leader="Node at b8fe715e-476a-d776-b535-ab2ac1f2dd73 [Leader]" + testing.go:149: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.342Z [INFO] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: entering leader state: leader="Node at b8fe715e-476a-d776-b535-ab2ac1f2dd73 [Leader]" +testLoggerAdapter verbose: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.342Z [INFO] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: added peer, starting replication: peer=server-6a5bce45-5390-2beb-45dd-7f157fc96e69 + testing.go:149: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.342Z [INFO] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: added peer, starting replication: peer=server-6a5bce45-5390-2beb-45dd-7f157fc96e69 +testLoggerAdapter verbose: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.342Z [INFO] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: added peer, starting replication: peer=server-89981a84-c9f9-4d23-22c3-f1fa42cac6c5 + testing.go:149: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.342Z [INFO] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: added peer, starting replication: peer=server-89981a84-c9f9-4d23-22c3-f1fa42cac6c5 +testLoggerAdapter verbose: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.343Z [INFO] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: pipelining replication: peer="{Voter server-6a5bce45-5390-2beb-45dd-7f157fc96e69 6a5bce45-5390-2beb-45dd-7f157fc96e69}" + testing.go:149: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.343Z [INFO] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: pipelining replication: peer="{Voter server-6a5bce45-5390-2beb-45dd-7f157fc96e69 6a5bce45-5390-2beb-45dd-7f157fc96e69}" +testLoggerAdapter verbose: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.343Z [INFO] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: pipelining replication: peer="{Voter server-89981a84-c9f9-4d23-22c3-f1fa42cac6c5 89981a84-c9f9-4d23-22c3-f1fa42cac6c5}" + testing.go:149: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.343Z [INFO] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: pipelining replication: peer="{Voter server-89981a84-c9f9-4d23-22c3-f1fa42cac6c5 89981a84-c9f9-4d23-22c3-f1fa42cac6c5}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:10.448Z [INFO] stable state for Leader reached at 2022-08-12 07:30:10.3429451 +0000 UTC m=+11.299631581 (1 nodes), 68.2758ms from start of poll, 69.1549ms from cluster start. Timeout at 2022-08-12 07:30:10.44796934 +0000 UTC m=+11.404655841, 105.02426ms after stability + testing.go:149: cluster: 2022-08-12T07:30:10.448Z [INFO] stable state for Leader reached at 2022-08-12 07:30:10.3429451 +0000 UTC m=+11.299631581 (1 nodes), 68.2758ms from start of poll, 69.1549ms from cluster start. Timeout at 2022-08-12 07:30:10.44796934 +0000 UTC m=+11.404655841, 105.02426ms after stability +testLoggerAdapter verbose: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.448Z [INFO] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: starting snapshot up to: index=12 + testing.go:149: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.448Z [INFO] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: starting snapshot up to: index=12 + testing.go:153: 2022-08-12T07:30:10.448Z [INFO] creating new snapshot: path=/tmp/raft541419775/snapshots/2-12-1660289410448.tmp +testLoggerAdapter verbose: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.449Z [INFO] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: snapshot complete up to: index=12 + testing.go:149: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.449Z [INFO] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: snapshot complete up to: index=12 + testing.go:153: 2022-08-12T07:30:10.450Z [INFO] creating new snapshot: path=/tmp/raft541419775/snapshots/2-23-1660289410450.tmp +testLoggerAdapter verbose: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.450Z [INFO] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: copied to local snapshot: bytes=71 + testing.go:149: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.450Z [INFO] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.451Z [INFO] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: restored user snapshot: index=1 + testing.go:149: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.451Z [INFO] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: restored user snapshot: index=1 +testLoggerAdapter verbose: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.451Z [ERROR] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: failed to get log: index=23 error="log not found" + testing.go:149: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.451Z [ERROR] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.451Z [ERROR] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: failed to get log: index=23 error="log not found" + testing.go:149: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.451Z [ERROR] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.451Z [INFO] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: aborting pipeline replication: peer="{Voter server-89981a84-c9f9-4d23-22c3-f1fa42cac6c5 89981a84-c9f9-4d23-22c3-f1fa42cac6c5}" + testing.go:149: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.451Z [INFO] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: aborting pipeline replication: peer="{Voter server-89981a84-c9f9-4d23-22c3-f1fa42cac6c5 89981a84-c9f9-4d23-22c3-f1fa42cac6c5}" +testLoggerAdapter verbose: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.451Z [INFO] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: aborting pipeline replication: peer="{Voter server-6a5bce45-5390-2beb-45dd-7f157fc96e69 6a5bce45-5390-2beb-45dd-7f157fc96e69}" + testing.go:149: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.451Z [INFO] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: aborting pipeline replication: peer="{Voter server-6a5bce45-5390-2beb-45dd-7f157fc96e69 6a5bce45-5390-2beb-45dd-7f157fc96e69}" +testLoggerAdapter verbose: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.457Z [ERROR] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: failed to get log: index=23 error="log not found" + testing.go:149: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.457Z [ERROR] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: failed to get log: index=23 error="log not found" + testing.go:153: 2022-08-12T07:30:10.457Z [INFO] creating new snapshot: path=/tmp/raft779801221/snapshots/2-23-1660289410457.tmp +testLoggerAdapter verbose: server-6a5bce45-5390-2beb-45dd-7f157fc96e69: 2022-08-12T07:30:10.458Z [INFO] server-6a5bce45-5390-2beb-45dd-7f157fc96e69: copied to local snapshot: bytes=71 + testing.go:149: server-6a5bce45-5390-2beb-45dd-7f157fc96e69: 2022-08-12T07:30:10.458Z [INFO] server-6a5bce45-5390-2beb-45dd-7f157fc96e69: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-6a5bce45-5390-2beb-45dd-7f157fc96e69: 2022-08-12T07:30:10.458Z [INFO] server-6a5bce45-5390-2beb-45dd-7f157fc96e69: Installed remote snapshot + testing.go:149: server-6a5bce45-5390-2beb-45dd-7f157fc96e69: 2022-08-12T07:30:10.458Z [INFO] server-6a5bce45-5390-2beb-45dd-7f157fc96e69: Installed remote snapshot +testLoggerAdapter verbose: server-6a5bce45-5390-2beb-45dd-7f157fc96e69: 2022-08-12T07:30:10.458Z [WARN] server-6a5bce45-5390-2beb-45dd-7f157fc96e69: skipping application of old log: index=22 + testing.go:149: server-6a5bce45-5390-2beb-45dd-7f157fc96e69: 2022-08-12T07:30:10.458Z [WARN] server-6a5bce45-5390-2beb-45dd-7f157fc96e69: skipping application of old log: index=22 +testLoggerAdapter verbose: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.458Z [INFO] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: pipelining replication: peer="{Voter server-6a5bce45-5390-2beb-45dd-7f157fc96e69 6a5bce45-5390-2beb-45dd-7f157fc96e69}" + testing.go:149: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.458Z [INFO] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: pipelining replication: peer="{Voter server-6a5bce45-5390-2beb-45dd-7f157fc96e69 6a5bce45-5390-2beb-45dd-7f157fc96e69}" +testLoggerAdapter verbose: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.461Z [ERROR] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: failed to get log: index=23 error="log not found" + testing.go:149: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.461Z [ERROR] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: failed to get log: index=23 error="log not found" + testing.go:153: 2022-08-12T07:30:10.461Z [INFO] creating new snapshot: path=/tmp/raft140800905/snapshots/2-23-1660289410461.tmp +testLoggerAdapter verbose: server-89981a84-c9f9-4d23-22c3-f1fa42cac6c5: 2022-08-12T07:30:10.462Z [INFO] server-89981a84-c9f9-4d23-22c3-f1fa42cac6c5: copied to local snapshot: bytes=71 + testing.go:149: server-89981a84-c9f9-4d23-22c3-f1fa42cac6c5: 2022-08-12T07:30:10.462Z [INFO] server-89981a84-c9f9-4d23-22c3-f1fa42cac6c5: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-89981a84-c9f9-4d23-22c3-f1fa42cac6c5: 2022-08-12T07:30:10.462Z [INFO] server-89981a84-c9f9-4d23-22c3-f1fa42cac6c5: Installed remote snapshot + testing.go:149: server-89981a84-c9f9-4d23-22c3-f1fa42cac6c5: 2022-08-12T07:30:10.462Z [INFO] server-89981a84-c9f9-4d23-22c3-f1fa42cac6c5: Installed remote snapshot +testLoggerAdapter verbose: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.462Z [INFO] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: pipelining replication: peer="{Voter server-89981a84-c9f9-4d23-22c3-f1fa42cac6c5 89981a84-c9f9-4d23-22c3-f1fa42cac6c5}" + testing.go:149: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.462Z [INFO] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: pipelining replication: peer="{Voter server-89981a84-c9f9-4d23-22c3-f1fa42cac6c5 89981a84-c9f9-4d23-22c3-f1fa42cac6c5}" +testLoggerAdapter verbose: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.474Z [INFO] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: aborting pipeline replication: peer="{Voter server-6a5bce45-5390-2beb-45dd-7f157fc96e69 6a5bce45-5390-2beb-45dd-7f157fc96e69}" + testing.go:149: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.474Z [INFO] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: aborting pipeline replication: peer="{Voter server-6a5bce45-5390-2beb-45dd-7f157fc96e69 6a5bce45-5390-2beb-45dd-7f157fc96e69}" +testLoggerAdapter verbose: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.474Z [INFO] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: aborting pipeline replication: peer="{Voter server-89981a84-c9f9-4d23-22c3-f1fa42cac6c5 89981a84-c9f9-4d23-22c3-f1fa42cac6c5}" + testing.go:149: server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: 2022-08-12T07:30:10.474Z [INFO] server-b8fe715e-476a-d776-b535-ab2ac1f2dd73: aborting pipeline replication: peer="{Voter server-89981a84-c9f9-4d23-22c3-f1fa42cac6c5 89981a84-c9f9-4d23-22c3-f1fa42cac6c5}" +testLoggerAdapter verbose: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.477Z [INFO] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186 Address:d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186} {Suffrage:Voter ID:server-7c9f407c-a623-3c85-6e76-a3377ce043a5 Address:7c9f407c-a623-3c85-6e76-a3377ce043a5} {Suffrage:Voter ID:server-57d89b2a-ba9d-158e-c9d0-5f6d721eb14c Address:57d89b2a-ba9d-158e-c9d0-5f6d721eb14c}]" + testing.go:149: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.477Z [INFO] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186 Address:d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186} {Suffrage:Voter ID:server-7c9f407c-a623-3c85-6e76-a3377ce043a5 Address:7c9f407c-a623-3c85-6e76-a3377ce043a5} {Suffrage:Voter ID:server-57d89b2a-ba9d-158e-c9d0-5f6d721eb14c Address:57d89b2a-ba9d-158e-c9d0-5f6d721eb14c}]" +testLoggerAdapter verbose: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.477Z [INFO] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: entering follower state: follower="Node at d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186 [Follower]" leader= + testing.go:149: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.477Z [INFO] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: entering follower state: follower="Node at d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186 [Follower]" leader= +testLoggerAdapter verbose: server-7c9f407c-a623-3c85-6e76-a3377ce043a5: 2022-08-12T07:30:10.477Z [INFO] server-7c9f407c-a623-3c85-6e76-a3377ce043a5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186 Address:d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186} {Suffrage:Voter ID:server-7c9f407c-a623-3c85-6e76-a3377ce043a5 Address:7c9f407c-a623-3c85-6e76-a3377ce043a5} {Suffrage:Voter ID:server-57d89b2a-ba9d-158e-c9d0-5f6d721eb14c Address:57d89b2a-ba9d-158e-c9d0-5f6d721eb14c}]" + testing.go:149: server-7c9f407c-a623-3c85-6e76-a3377ce043a5: 2022-08-12T07:30:10.477Z [INFO] server-7c9f407c-a623-3c85-6e76-a3377ce043a5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186 Address:d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186} {Suffrage:Voter ID:server-7c9f407c-a623-3c85-6e76-a3377ce043a5 Address:7c9f407c-a623-3c85-6e76-a3377ce043a5} {Suffrage:Voter ID:server-57d89b2a-ba9d-158e-c9d0-5f6d721eb14c Address:57d89b2a-ba9d-158e-c9d0-5f6d721eb14c}]" +testLoggerAdapter verbose: server-7c9f407c-a623-3c85-6e76-a3377ce043a5: 2022-08-12T07:30:10.478Z [INFO] server-7c9f407c-a623-3c85-6e76-a3377ce043a5: entering follower state: follower="Node at 7c9f407c-a623-3c85-6e76-a3377ce043a5 [Follower]" leader= + testing.go:149: server-7c9f407c-a623-3c85-6e76-a3377ce043a5: 2022-08-12T07:30:10.478Z [INFO] server-7c9f407c-a623-3c85-6e76-a3377ce043a5: entering follower state: follower="Node at 7c9f407c-a623-3c85-6e76-a3377ce043a5 [Follower]" leader= +testLoggerAdapter verbose: server-57d89b2a-ba9d-158e-c9d0-5f6d721eb14c: 2022-08-12T07:30:10.478Z [INFO] server-57d89b2a-ba9d-158e-c9d0-5f6d721eb14c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186 Address:d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186} {Suffrage:Voter ID:server-7c9f407c-a623-3c85-6e76-a3377ce043a5 Address:7c9f407c-a623-3c85-6e76-a3377ce043a5} {Suffrage:Voter ID:server-57d89b2a-ba9d-158e-c9d0-5f6d721eb14c Address:57d89b2a-ba9d-158e-c9d0-5f6d721eb14c}]" + testing.go:149: server-57d89b2a-ba9d-158e-c9d0-5f6d721eb14c: 2022-08-12T07:30:10.478Z [INFO] server-57d89b2a-ba9d-158e-c9d0-5f6d721eb14c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186 Address:d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186} {Suffrage:Voter ID:server-7c9f407c-a623-3c85-6e76-a3377ce043a5 Address:7c9f407c-a623-3c85-6e76-a3377ce043a5} {Suffrage:Voter ID:server-57d89b2a-ba9d-158e-c9d0-5f6d721eb14c Address:57d89b2a-ba9d-158e-c9d0-5f6d721eb14c}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:10.478Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:10.478Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-57d89b2a-ba9d-158e-c9d0-5f6d721eb14c: 2022-08-12T07:30:10.478Z [INFO] server-57d89b2a-ba9d-158e-c9d0-5f6d721eb14c: entering follower state: follower="Node at 57d89b2a-ba9d-158e-c9d0-5f6d721eb14c [Follower]" leader= + testing.go:149: server-57d89b2a-ba9d-158e-c9d0-5f6d721eb14c: 2022-08-12T07:30:10.478Z [INFO] server-57d89b2a-ba9d-158e-c9d0-5f6d721eb14c: entering follower state: follower="Node at 57d89b2a-ba9d-158e-c9d0-5f6d721eb14c [Follower]" leader= +testLoggerAdapter verbose: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.538Z [WARN] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.538Z [WARN] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.538Z [INFO] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: entering candidate state: node="Node at d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186 [Candidate]" term=2 + testing.go:149: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.538Z [INFO] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: entering candidate state: node="Node at d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186 [Candidate]" term=2 +testLoggerAdapter verbose: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.538Z [INFO] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: election won: tally=2 + testing.go:149: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.538Z [INFO] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: election won: tally=2 +testLoggerAdapter verbose: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.539Z [INFO] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: entering leader state: leader="Node at d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186 [Leader]" + testing.go:149: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.539Z [INFO] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: entering leader state: leader="Node at d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186 [Leader]" +testLoggerAdapter verbose: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.539Z [INFO] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: added peer, starting replication: peer=server-7c9f407c-a623-3c85-6e76-a3377ce043a5 + testing.go:149: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.539Z [INFO] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: added peer, starting replication: peer=server-7c9f407c-a623-3c85-6e76-a3377ce043a5 +testLoggerAdapter verbose: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.539Z [INFO] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: added peer, starting replication: peer=server-57d89b2a-ba9d-158e-c9d0-5f6d721eb14c + testing.go:149: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.539Z [INFO] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: added peer, starting replication: peer=server-57d89b2a-ba9d-158e-c9d0-5f6d721eb14c +testLoggerAdapter verbose: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.539Z [INFO] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: pipelining replication: peer="{Voter server-7c9f407c-a623-3c85-6e76-a3377ce043a5 7c9f407c-a623-3c85-6e76-a3377ce043a5}" + testing.go:149: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.539Z [INFO] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: pipelining replication: peer="{Voter server-7c9f407c-a623-3c85-6e76-a3377ce043a5 7c9f407c-a623-3c85-6e76-a3377ce043a5}" +testLoggerAdapter verbose: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.539Z [INFO] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: pipelining replication: peer="{Voter server-57d89b2a-ba9d-158e-c9d0-5f6d721eb14c 57d89b2a-ba9d-158e-c9d0-5f6d721eb14c}" + testing.go:149: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.539Z [INFO] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: pipelining replication: peer="{Voter server-57d89b2a-ba9d-158e-c9d0-5f6d721eb14c 57d89b2a-ba9d-158e-c9d0-5f6d721eb14c}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:10.644Z [INFO] stable state for Leader reached at 2022-08-12 07:30:10.53908002 +0000 UTC m=+11.495766501 (1 nodes), 60.61684ms from start of poll, 62.32072ms from cluster start. Timeout at 2022-08-12 07:30:10.64414344 +0000 UTC m=+11.600829921, 105.06342ms after stability + testing.go:149: cluster: 2022-08-12T07:30:10.644Z [INFO] stable state for Leader reached at 2022-08-12 07:30:10.53908002 +0000 UTC m=+11.495766501 (1 nodes), 60.61684ms from start of poll, 62.32072ms from cluster start. Timeout at 2022-08-12 07:30:10.64414344 +0000 UTC m=+11.600829921, 105.06342ms after stability +testLoggerAdapter verbose: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.644Z [INFO] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: starting snapshot up to: index=12 + testing.go:149: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.644Z [INFO] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: starting snapshot up to: index=12 + testing.go:153: 2022-08-12T07:30:10.644Z [INFO] creating new snapshot: path=/tmp/raft252219555/snapshots/2-12-1660289410644.tmp +testLoggerAdapter verbose: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.645Z [INFO] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: snapshot complete up to: index=12 + testing.go:149: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.645Z [INFO] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: snapshot complete up to: index=12 + testing.go:153: 2022-08-12T07:30:10.646Z [INFO] creating new snapshot: path=/tmp/raft252219555/snapshots/2-113-1660289410646.tmp +testLoggerAdapter verbose: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.647Z [INFO] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: copied to local snapshot: bytes=71 + testing.go:149: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.647Z [INFO] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.647Z [INFO] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: restored user snapshot: index=1 + testing.go:149: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.647Z [INFO] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: restored user snapshot: index=1 +testLoggerAdapter verbose: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.647Z [ERROR] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: failed to get log: index=23 error="log not found" + testing.go:149: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.647Z [ERROR] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.647Z [ERROR] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: failed to get log: index=23 error="log not found" + testing.go:149: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.647Z [ERROR] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.647Z [INFO] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: aborting pipeline replication: peer="{Voter server-57d89b2a-ba9d-158e-c9d0-5f6d721eb14c 57d89b2a-ba9d-158e-c9d0-5f6d721eb14c}" + testing.go:149: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.647Z [INFO] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: aborting pipeline replication: peer="{Voter server-57d89b2a-ba9d-158e-c9d0-5f6d721eb14c 57d89b2a-ba9d-158e-c9d0-5f6d721eb14c}" +testLoggerAdapter verbose: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.647Z [INFO] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: aborting pipeline replication: peer="{Voter server-7c9f407c-a623-3c85-6e76-a3377ce043a5 7c9f407c-a623-3c85-6e76-a3377ce043a5}" + testing.go:149: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.647Z [INFO] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: aborting pipeline replication: peer="{Voter server-7c9f407c-a623-3c85-6e76-a3377ce043a5 7c9f407c-a623-3c85-6e76-a3377ce043a5}" +testLoggerAdapter verbose: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.656Z [ERROR] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: failed to get log: index=23 error="log not found" + testing.go:149: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.656Z [ERROR] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: failed to get log: index=23 error="log not found" + testing.go:153: 2022-08-12T07:30:10.656Z [INFO] creating new snapshot: path=/tmp/raft771057799/snapshots/2-113-1660289410656.tmp +testLoggerAdapter verbose: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.657Z [ERROR] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: failed to get log: index=23 error="log not found" + testing.go:149: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.657Z [ERROR] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-57d89b2a-ba9d-158e-c9d0-5f6d721eb14c: 2022-08-12T07:30:10.657Z [INFO] server-57d89b2a-ba9d-158e-c9d0-5f6d721eb14c: copied to local snapshot: bytes=71 + testing.go:149: server-57d89b2a-ba9d-158e-c9d0-5f6d721eb14c: 2022-08-12T07:30:10.657Z [INFO] server-57d89b2a-ba9d-158e-c9d0-5f6d721eb14c: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-57d89b2a-ba9d-158e-c9d0-5f6d721eb14c: 2022-08-12T07:30:10.657Z [INFO] server-57d89b2a-ba9d-158e-c9d0-5f6d721eb14c: Installed remote snapshot + testing.go:149: server-57d89b2a-ba9d-158e-c9d0-5f6d721eb14c: 2022-08-12T07:30:10.657Z [INFO] server-57d89b2a-ba9d-158e-c9d0-5f6d721eb14c: Installed remote snapshot + testing.go:153: 2022-08-12T07:30:10.657Z [INFO] creating new snapshot: path=/tmp/raft934781133/snapshots/2-113-1660289410657.tmp +testLoggerAdapter verbose: server-57d89b2a-ba9d-158e-c9d0-5f6d721eb14c: 2022-08-12T07:30:10.657Z [WARN] server-57d89b2a-ba9d-158e-c9d0-5f6d721eb14c: skipping application of old log: index=22 + testing.go:149: server-57d89b2a-ba9d-158e-c9d0-5f6d721eb14c: 2022-08-12T07:30:10.657Z [WARN] server-57d89b2a-ba9d-158e-c9d0-5f6d721eb14c: skipping application of old log: index=22 +testLoggerAdapter verbose: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.658Z [INFO] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: pipelining replication: peer="{Voter server-57d89b2a-ba9d-158e-c9d0-5f6d721eb14c 57d89b2a-ba9d-158e-c9d0-5f6d721eb14c}" + testing.go:149: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.658Z [INFO] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: pipelining replication: peer="{Voter server-57d89b2a-ba9d-158e-c9d0-5f6d721eb14c 57d89b2a-ba9d-158e-c9d0-5f6d721eb14c}" +testLoggerAdapter verbose: server-7c9f407c-a623-3c85-6e76-a3377ce043a5: 2022-08-12T07:30:10.658Z [INFO] server-7c9f407c-a623-3c85-6e76-a3377ce043a5: copied to local snapshot: bytes=71 + testing.go:149: server-7c9f407c-a623-3c85-6e76-a3377ce043a5: 2022-08-12T07:30:10.658Z [INFO] server-7c9f407c-a623-3c85-6e76-a3377ce043a5: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-7c9f407c-a623-3c85-6e76-a3377ce043a5: 2022-08-12T07:30:10.658Z [INFO] server-7c9f407c-a623-3c85-6e76-a3377ce043a5: Installed remote snapshot + testing.go:149: server-7c9f407c-a623-3c85-6e76-a3377ce043a5: 2022-08-12T07:30:10.658Z [INFO] server-7c9f407c-a623-3c85-6e76-a3377ce043a5: Installed remote snapshot +testLoggerAdapter verbose: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.658Z [INFO] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: pipelining replication: peer="{Voter server-7c9f407c-a623-3c85-6e76-a3377ce043a5 7c9f407c-a623-3c85-6e76-a3377ce043a5}" + testing.go:149: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.658Z [INFO] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: pipelining replication: peer="{Voter server-7c9f407c-a623-3c85-6e76-a3377ce043a5 7c9f407c-a623-3c85-6e76-a3377ce043a5}" +testLoggerAdapter verbose: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.673Z [INFO] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: aborting pipeline replication: peer="{Voter server-57d89b2a-ba9d-158e-c9d0-5f6d721eb14c 57d89b2a-ba9d-158e-c9d0-5f6d721eb14c}" + testing.go:149: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.673Z [INFO] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: aborting pipeline replication: peer="{Voter server-57d89b2a-ba9d-158e-c9d0-5f6d721eb14c 57d89b2a-ba9d-158e-c9d0-5f6d721eb14c}" +testLoggerAdapter verbose: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.673Z [INFO] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: aborting pipeline replication: peer="{Voter server-7c9f407c-a623-3c85-6e76-a3377ce043a5 7c9f407c-a623-3c85-6e76-a3377ce043a5}" + testing.go:149: server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: 2022-08-12T07:30:10.673Z [INFO] server-d9e4e35d-db8d-e2f3-31e8-5b9d45f3c186: aborting pipeline replication: peer="{Voter server-7c9f407c-a623-3c85-6e76-a3377ce043a5 7c9f407c-a623-3c85-6e76-a3377ce043a5}" +testLoggerAdapter verbose: server-30967dfb-9de1-1ef4-5ec6-6fd8260f40d1: 2022-08-12T07:30:10.675Z [INFO] server-30967dfb-9de1-1ef4-5ec6-6fd8260f40d1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30967dfb-9de1-1ef4-5ec6-6fd8260f40d1 Address:30967dfb-9de1-1ef4-5ec6-6fd8260f40d1} {Suffrage:Voter ID:server-08d50ebc-e321-eb66-367a-d3e55a45f470 Address:08d50ebc-e321-eb66-367a-d3e55a45f470} {Suffrage:Voter ID:server-a737cfc9-2f87-2813-3298-56f5853f37a4 Address:a737cfc9-2f87-2813-3298-56f5853f37a4}]" + testing.go:149: server-30967dfb-9de1-1ef4-5ec6-6fd8260f40d1: 2022-08-12T07:30:10.675Z [INFO] server-30967dfb-9de1-1ef4-5ec6-6fd8260f40d1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30967dfb-9de1-1ef4-5ec6-6fd8260f40d1 Address:30967dfb-9de1-1ef4-5ec6-6fd8260f40d1} {Suffrage:Voter ID:server-08d50ebc-e321-eb66-367a-d3e55a45f470 Address:08d50ebc-e321-eb66-367a-d3e55a45f470} {Suffrage:Voter ID:server-a737cfc9-2f87-2813-3298-56f5853f37a4 Address:a737cfc9-2f87-2813-3298-56f5853f37a4}]" +testLoggerAdapter verbose: server-30967dfb-9de1-1ef4-5ec6-6fd8260f40d1: 2022-08-12T07:30:10.675Z [INFO] server-30967dfb-9de1-1ef4-5ec6-6fd8260f40d1: entering follower state: follower="Node at 30967dfb-9de1-1ef4-5ec6-6fd8260f40d1 [Follower]" leader= + testing.go:149: server-30967dfb-9de1-1ef4-5ec6-6fd8260f40d1: 2022-08-12T07:30:10.675Z [INFO] server-30967dfb-9de1-1ef4-5ec6-6fd8260f40d1: entering follower state: follower="Node at 30967dfb-9de1-1ef4-5ec6-6fd8260f40d1 [Follower]" leader= +testLoggerAdapter verbose: server-08d50ebc-e321-eb66-367a-d3e55a45f470: 2022-08-12T07:30:10.675Z [INFO] server-08d50ebc-e321-eb66-367a-d3e55a45f470: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30967dfb-9de1-1ef4-5ec6-6fd8260f40d1 Address:30967dfb-9de1-1ef4-5ec6-6fd8260f40d1} {Suffrage:Voter ID:server-08d50ebc-e321-eb66-367a-d3e55a45f470 Address:08d50ebc-e321-eb66-367a-d3e55a45f470} {Suffrage:Voter ID:server-a737cfc9-2f87-2813-3298-56f5853f37a4 Address:a737cfc9-2f87-2813-3298-56f5853f37a4}]" + testing.go:149: server-08d50ebc-e321-eb66-367a-d3e55a45f470: 2022-08-12T07:30:10.675Z [INFO] server-08d50ebc-e321-eb66-367a-d3e55a45f470: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30967dfb-9de1-1ef4-5ec6-6fd8260f40d1 Address:30967dfb-9de1-1ef4-5ec6-6fd8260f40d1} {Suffrage:Voter ID:server-08d50ebc-e321-eb66-367a-d3e55a45f470 Address:08d50ebc-e321-eb66-367a-d3e55a45f470} {Suffrage:Voter ID:server-a737cfc9-2f87-2813-3298-56f5853f37a4 Address:a737cfc9-2f87-2813-3298-56f5853f37a4}]" +testLoggerAdapter verbose: server-08d50ebc-e321-eb66-367a-d3e55a45f470: 2022-08-12T07:30:10.675Z [INFO] server-08d50ebc-e321-eb66-367a-d3e55a45f470: entering follower state: follower="Node at 08d50ebc-e321-eb66-367a-d3e55a45f470 [Follower]" leader= + testing.go:149: server-08d50ebc-e321-eb66-367a-d3e55a45f470: 2022-08-12T07:30:10.675Z [INFO] server-08d50ebc-e321-eb66-367a-d3e55a45f470: entering follower state: follower="Node at 08d50ebc-e321-eb66-367a-d3e55a45f470 [Follower]" leader= +testLoggerAdapter verbose: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.676Z [INFO] server-a737cfc9-2f87-2813-3298-56f5853f37a4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30967dfb-9de1-1ef4-5ec6-6fd8260f40d1 Address:30967dfb-9de1-1ef4-5ec6-6fd8260f40d1} {Suffrage:Voter ID:server-08d50ebc-e321-eb66-367a-d3e55a45f470 Address:08d50ebc-e321-eb66-367a-d3e55a45f470} {Suffrage:Voter ID:server-a737cfc9-2f87-2813-3298-56f5853f37a4 Address:a737cfc9-2f87-2813-3298-56f5853f37a4}]" + testing.go:149: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.676Z [INFO] server-a737cfc9-2f87-2813-3298-56f5853f37a4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30967dfb-9de1-1ef4-5ec6-6fd8260f40d1 Address:30967dfb-9de1-1ef4-5ec6-6fd8260f40d1} {Suffrage:Voter ID:server-08d50ebc-e321-eb66-367a-d3e55a45f470 Address:08d50ebc-e321-eb66-367a-d3e55a45f470} {Suffrage:Voter ID:server-a737cfc9-2f87-2813-3298-56f5853f37a4 Address:a737cfc9-2f87-2813-3298-56f5853f37a4}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:10.676Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:10.676Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.676Z [INFO] server-a737cfc9-2f87-2813-3298-56f5853f37a4: entering follower state: follower="Node at a737cfc9-2f87-2813-3298-56f5853f37a4 [Follower]" leader= + testing.go:149: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.676Z [INFO] server-a737cfc9-2f87-2813-3298-56f5853f37a4: entering follower state: follower="Node at a737cfc9-2f87-2813-3298-56f5853f37a4 [Follower]" leader= +testLoggerAdapter verbose: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.742Z [WARN] server-a737cfc9-2f87-2813-3298-56f5853f37a4: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.742Z [WARN] server-a737cfc9-2f87-2813-3298-56f5853f37a4: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.742Z [INFO] server-a737cfc9-2f87-2813-3298-56f5853f37a4: entering candidate state: node="Node at a737cfc9-2f87-2813-3298-56f5853f37a4 [Candidate]" term=2 + testing.go:149: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.742Z [INFO] server-a737cfc9-2f87-2813-3298-56f5853f37a4: entering candidate state: node="Node at a737cfc9-2f87-2813-3298-56f5853f37a4 [Candidate]" term=2 +testLoggerAdapter verbose: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.742Z [INFO] server-a737cfc9-2f87-2813-3298-56f5853f37a4: election won: tally=2 + testing.go:149: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.742Z [INFO] server-a737cfc9-2f87-2813-3298-56f5853f37a4: election won: tally=2 +testLoggerAdapter verbose: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.742Z [INFO] server-a737cfc9-2f87-2813-3298-56f5853f37a4: entering leader state: leader="Node at a737cfc9-2f87-2813-3298-56f5853f37a4 [Leader]" + testing.go:149: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.742Z [INFO] server-a737cfc9-2f87-2813-3298-56f5853f37a4: entering leader state: leader="Node at a737cfc9-2f87-2813-3298-56f5853f37a4 [Leader]" +testLoggerAdapter verbose: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.742Z [INFO] server-a737cfc9-2f87-2813-3298-56f5853f37a4: added peer, starting replication: peer=server-30967dfb-9de1-1ef4-5ec6-6fd8260f40d1 + testing.go:149: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.742Z [INFO] server-a737cfc9-2f87-2813-3298-56f5853f37a4: added peer, starting replication: peer=server-30967dfb-9de1-1ef4-5ec6-6fd8260f40d1 +testLoggerAdapter verbose: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.742Z [INFO] server-a737cfc9-2f87-2813-3298-56f5853f37a4: added peer, starting replication: peer=server-08d50ebc-e321-eb66-367a-d3e55a45f470 + testing.go:149: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.742Z [INFO] server-a737cfc9-2f87-2813-3298-56f5853f37a4: added peer, starting replication: peer=server-08d50ebc-e321-eb66-367a-d3e55a45f470 +testLoggerAdapter verbose: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.742Z [INFO] server-a737cfc9-2f87-2813-3298-56f5853f37a4: pipelining replication: peer="{Voter server-30967dfb-9de1-1ef4-5ec6-6fd8260f40d1 30967dfb-9de1-1ef4-5ec6-6fd8260f40d1}" + testing.go:149: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.742Z [INFO] server-a737cfc9-2f87-2813-3298-56f5853f37a4: pipelining replication: peer="{Voter server-30967dfb-9de1-1ef4-5ec6-6fd8260f40d1 30967dfb-9de1-1ef4-5ec6-6fd8260f40d1}" +testLoggerAdapter verbose: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.742Z [INFO] server-a737cfc9-2f87-2813-3298-56f5853f37a4: pipelining replication: peer="{Voter server-08d50ebc-e321-eb66-367a-d3e55a45f470 08d50ebc-e321-eb66-367a-d3e55a45f470}" + testing.go:149: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.742Z [INFO] server-a737cfc9-2f87-2813-3298-56f5853f37a4: pipelining replication: peer="{Voter server-08d50ebc-e321-eb66-367a-d3e55a45f470 08d50ebc-e321-eb66-367a-d3e55a45f470}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:10.847Z [INFO] stable state for Leader reached at 2022-08-12 07:30:10.74271512 +0000 UTC m=+11.699401581 (1 nodes), 66.55526ms from start of poll, 67.4334ms from cluster start. Timeout at 2022-08-12 07:30:10.84780512 +0000 UTC m=+11.804491641, 105.09006ms after stability + testing.go:149: cluster: 2022-08-12T07:30:10.847Z [INFO] stable state for Leader reached at 2022-08-12 07:30:10.74271512 +0000 UTC m=+11.699401581 (1 nodes), 66.55526ms from start of poll, 67.4334ms from cluster start. Timeout at 2022-08-12 07:30:10.84780512 +0000 UTC m=+11.804491641, 105.09006ms after stability +testLoggerAdapter verbose: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.848Z [INFO] server-a737cfc9-2f87-2813-3298-56f5853f37a4: starting snapshot up to: index=12 + testing.go:149: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.848Z [INFO] server-a737cfc9-2f87-2813-3298-56f5853f37a4: starting snapshot up to: index=12 + testing.go:153: 2022-08-12T07:30:10.848Z [INFO] creating new snapshot: path=/tmp/raft111104021/snapshots/2-12-1660289410848.tmp +testLoggerAdapter verbose: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.849Z [INFO] server-a737cfc9-2f87-2813-3298-56f5853f37a4: snapshot complete up to: index=12 + testing.go:149: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.849Z [INFO] server-a737cfc9-2f87-2813-3298-56f5853f37a4: snapshot complete up to: index=12 + testing.go:153: 2022-08-12T07:30:10.850Z [INFO] creating new snapshot: path=/tmp/raft111104021/snapshots/2-1013-1660289410850.tmp +testLoggerAdapter verbose: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.851Z [INFO] server-a737cfc9-2f87-2813-3298-56f5853f37a4: copied to local snapshot: bytes=71 + testing.go:149: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.851Z [INFO] server-a737cfc9-2f87-2813-3298-56f5853f37a4: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.851Z [INFO] server-a737cfc9-2f87-2813-3298-56f5853f37a4: restored user snapshot: index=1 + testing.go:149: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.851Z [INFO] server-a737cfc9-2f87-2813-3298-56f5853f37a4: restored user snapshot: index=1 +testLoggerAdapter verbose: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.851Z [ERROR] server-a737cfc9-2f87-2813-3298-56f5853f37a4: failed to get log: index=23 error="log not found" + testing.go:149: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.851Z [ERROR] server-a737cfc9-2f87-2813-3298-56f5853f37a4: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.851Z [ERROR] server-a737cfc9-2f87-2813-3298-56f5853f37a4: failed to get log: index=23 error="log not found" + testing.go:149: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.851Z [ERROR] server-a737cfc9-2f87-2813-3298-56f5853f37a4: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.851Z [INFO] server-a737cfc9-2f87-2813-3298-56f5853f37a4: aborting pipeline replication: peer="{Voter server-30967dfb-9de1-1ef4-5ec6-6fd8260f40d1 30967dfb-9de1-1ef4-5ec6-6fd8260f40d1}" + testing.go:149: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.851Z [INFO] server-a737cfc9-2f87-2813-3298-56f5853f37a4: aborting pipeline replication: peer="{Voter server-30967dfb-9de1-1ef4-5ec6-6fd8260f40d1 30967dfb-9de1-1ef4-5ec6-6fd8260f40d1}" +testLoggerAdapter verbose: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.851Z [INFO] server-a737cfc9-2f87-2813-3298-56f5853f37a4: aborting pipeline replication: peer="{Voter server-08d50ebc-e321-eb66-367a-d3e55a45f470 08d50ebc-e321-eb66-367a-d3e55a45f470}" + testing.go:149: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.851Z [INFO] server-a737cfc9-2f87-2813-3298-56f5853f37a4: aborting pipeline replication: peer="{Voter server-08d50ebc-e321-eb66-367a-d3e55a45f470 08d50ebc-e321-eb66-367a-d3e55a45f470}" +testLoggerAdapter verbose: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.857Z [ERROR] server-a737cfc9-2f87-2813-3298-56f5853f37a4: failed to get log: index=23 error="log not found" + testing.go:149: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.857Z [ERROR] server-a737cfc9-2f87-2813-3298-56f5853f37a4: failed to get log: index=23 error="log not found" + testing.go:153: 2022-08-12T07:30:10.858Z [INFO] creating new snapshot: path=/tmp/raft486485163/snapshots/2-1013-1660289410858.tmp +testLoggerAdapter verbose: server-08d50ebc-e321-eb66-367a-d3e55a45f470: 2022-08-12T07:30:10.858Z [INFO] server-08d50ebc-e321-eb66-367a-d3e55a45f470: copied to local snapshot: bytes=71 + testing.go:149: server-08d50ebc-e321-eb66-367a-d3e55a45f470: 2022-08-12T07:30:10.858Z [INFO] server-08d50ebc-e321-eb66-367a-d3e55a45f470: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-08d50ebc-e321-eb66-367a-d3e55a45f470: 2022-08-12T07:30:10.859Z [INFO] server-08d50ebc-e321-eb66-367a-d3e55a45f470: Installed remote snapshot + testing.go:149: server-08d50ebc-e321-eb66-367a-d3e55a45f470: 2022-08-12T07:30:10.859Z [INFO] server-08d50ebc-e321-eb66-367a-d3e55a45f470: Installed remote snapshot +testLoggerAdapter verbose: server-08d50ebc-e321-eb66-367a-d3e55a45f470: 2022-08-12T07:30:10.859Z [WARN] server-08d50ebc-e321-eb66-367a-d3e55a45f470: skipping application of old log: index=22 + testing.go:149: server-08d50ebc-e321-eb66-367a-d3e55a45f470: 2022-08-12T07:30:10.859Z [WARN] server-08d50ebc-e321-eb66-367a-d3e55a45f470: skipping application of old log: index=22 +testLoggerAdapter verbose: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.859Z [INFO] server-a737cfc9-2f87-2813-3298-56f5853f37a4: pipelining replication: peer="{Voter server-08d50ebc-e321-eb66-367a-d3e55a45f470 08d50ebc-e321-eb66-367a-d3e55a45f470}" + testing.go:149: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.859Z [INFO] server-a737cfc9-2f87-2813-3298-56f5853f37a4: pipelining replication: peer="{Voter server-08d50ebc-e321-eb66-367a-d3e55a45f470 08d50ebc-e321-eb66-367a-d3e55a45f470}" +testLoggerAdapter verbose: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.860Z [ERROR] server-a737cfc9-2f87-2813-3298-56f5853f37a4: failed to get log: index=23 error="log not found" + testing.go:149: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.860Z [ERROR] server-a737cfc9-2f87-2813-3298-56f5853f37a4: failed to get log: index=23 error="log not found" + testing.go:153: 2022-08-12T07:30:10.861Z [INFO] creating new snapshot: path=/tmp/raft569991761/snapshots/2-1013-1660289410861.tmp +testLoggerAdapter verbose: server-30967dfb-9de1-1ef4-5ec6-6fd8260f40d1: 2022-08-12T07:30:10.861Z [INFO] server-30967dfb-9de1-1ef4-5ec6-6fd8260f40d1: copied to local snapshot: bytes=71 + testing.go:149: server-30967dfb-9de1-1ef4-5ec6-6fd8260f40d1: 2022-08-12T07:30:10.861Z [INFO] server-30967dfb-9de1-1ef4-5ec6-6fd8260f40d1: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-30967dfb-9de1-1ef4-5ec6-6fd8260f40d1: 2022-08-12T07:30:10.862Z [INFO] server-30967dfb-9de1-1ef4-5ec6-6fd8260f40d1: Installed remote snapshot + testing.go:149: server-30967dfb-9de1-1ef4-5ec6-6fd8260f40d1: 2022-08-12T07:30:10.862Z [INFO] server-30967dfb-9de1-1ef4-5ec6-6fd8260f40d1: Installed remote snapshot +testLoggerAdapter verbose: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.862Z [INFO] server-a737cfc9-2f87-2813-3298-56f5853f37a4: pipelining replication: peer="{Voter server-30967dfb-9de1-1ef4-5ec6-6fd8260f40d1 30967dfb-9de1-1ef4-5ec6-6fd8260f40d1}" + testing.go:149: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.862Z [INFO] server-a737cfc9-2f87-2813-3298-56f5853f37a4: pipelining replication: peer="{Voter server-30967dfb-9de1-1ef4-5ec6-6fd8260f40d1 30967dfb-9de1-1ef4-5ec6-6fd8260f40d1}" +testLoggerAdapter verbose: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.875Z [INFO] server-a737cfc9-2f87-2813-3298-56f5853f37a4: aborting pipeline replication: peer="{Voter server-08d50ebc-e321-eb66-367a-d3e55a45f470 08d50ebc-e321-eb66-367a-d3e55a45f470}" + testing.go:149: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.875Z [INFO] server-a737cfc9-2f87-2813-3298-56f5853f37a4: aborting pipeline replication: peer="{Voter server-08d50ebc-e321-eb66-367a-d3e55a45f470 08d50ebc-e321-eb66-367a-d3e55a45f470}" +testLoggerAdapter verbose: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.875Z [INFO] server-a737cfc9-2f87-2813-3298-56f5853f37a4: aborting pipeline replication: peer="{Voter server-30967dfb-9de1-1ef4-5ec6-6fd8260f40d1 30967dfb-9de1-1ef4-5ec6-6fd8260f40d1}" + testing.go:149: server-a737cfc9-2f87-2813-3298-56f5853f37a4: 2022-08-12T07:30:10.875Z [INFO] server-a737cfc9-2f87-2813-3298-56f5853f37a4: aborting pipeline replication: peer="{Voter server-30967dfb-9de1-1ef4-5ec6-6fd8260f40d1 30967dfb-9de1-1ef4-5ec6-6fd8260f40d1}" +testLoggerAdapter verbose: server-65ebda27-dba2-3776-14a7-63a4a2c8a3c3: 2022-08-12T07:30:10.876Z [INFO] server-65ebda27-dba2-3776-14a7-63a4a2c8a3c3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-65ebda27-dba2-3776-14a7-63a4a2c8a3c3 Address:65ebda27-dba2-3776-14a7-63a4a2c8a3c3} {Suffrage:Voter ID:server-779dc010-d281-c1a4-bdd8-bca00c7bc259 Address:779dc010-d281-c1a4-bdd8-bca00c7bc259} {Suffrage:Voter ID:server-63140caa-c8bb-8a7a-ba12-9d22f43fe7aa Address:63140caa-c8bb-8a7a-ba12-9d22f43fe7aa}]" + testing.go:149: server-65ebda27-dba2-3776-14a7-63a4a2c8a3c3: 2022-08-12T07:30:10.876Z [INFO] server-65ebda27-dba2-3776-14a7-63a4a2c8a3c3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-65ebda27-dba2-3776-14a7-63a4a2c8a3c3 Address:65ebda27-dba2-3776-14a7-63a4a2c8a3c3} {Suffrage:Voter ID:server-779dc010-d281-c1a4-bdd8-bca00c7bc259 Address:779dc010-d281-c1a4-bdd8-bca00c7bc259} {Suffrage:Voter ID:server-63140caa-c8bb-8a7a-ba12-9d22f43fe7aa Address:63140caa-c8bb-8a7a-ba12-9d22f43fe7aa}]" +testLoggerAdapter verbose: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:10.877Z [INFO] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-65ebda27-dba2-3776-14a7-63a4a2c8a3c3 Address:65ebda27-dba2-3776-14a7-63a4a2c8a3c3} {Suffrage:Voter ID:server-779dc010-d281-c1a4-bdd8-bca00c7bc259 Address:779dc010-d281-c1a4-bdd8-bca00c7bc259} {Suffrage:Voter ID:server-63140caa-c8bb-8a7a-ba12-9d22f43fe7aa Address:63140caa-c8bb-8a7a-ba12-9d22f43fe7aa}]" + testing.go:149: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:10.877Z [INFO] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-65ebda27-dba2-3776-14a7-63a4a2c8a3c3 Address:65ebda27-dba2-3776-14a7-63a4a2c8a3c3} {Suffrage:Voter ID:server-779dc010-d281-c1a4-bdd8-bca00c7bc259 Address:779dc010-d281-c1a4-bdd8-bca00c7bc259} {Suffrage:Voter ID:server-63140caa-c8bb-8a7a-ba12-9d22f43fe7aa Address:63140caa-c8bb-8a7a-ba12-9d22f43fe7aa}]" +testLoggerAdapter verbose: server-65ebda27-dba2-3776-14a7-63a4a2c8a3c3: 2022-08-12T07:30:10.877Z [INFO] server-65ebda27-dba2-3776-14a7-63a4a2c8a3c3: entering follower state: follower="Node at 65ebda27-dba2-3776-14a7-63a4a2c8a3c3 [Follower]" leader= + testing.go:149: server-65ebda27-dba2-3776-14a7-63a4a2c8a3c3: 2022-08-12T07:30:10.877Z [INFO] server-65ebda27-dba2-3776-14a7-63a4a2c8a3c3: entering follower state: follower="Node at 65ebda27-dba2-3776-14a7-63a4a2c8a3c3 [Follower]" leader= +testLoggerAdapter verbose: server-63140caa-c8bb-8a7a-ba12-9d22f43fe7aa: 2022-08-12T07:30:10.877Z [INFO] server-63140caa-c8bb-8a7a-ba12-9d22f43fe7aa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-65ebda27-dba2-3776-14a7-63a4a2c8a3c3 Address:65ebda27-dba2-3776-14a7-63a4a2c8a3c3} {Suffrage:Voter ID:server-779dc010-d281-c1a4-bdd8-bca00c7bc259 Address:779dc010-d281-c1a4-bdd8-bca00c7bc259} {Suffrage:Voter ID:server-63140caa-c8bb-8a7a-ba12-9d22f43fe7aa Address:63140caa-c8bb-8a7a-ba12-9d22f43fe7aa}]" + testing.go:149: server-63140caa-c8bb-8a7a-ba12-9d22f43fe7aa: 2022-08-12T07:30:10.877Z [INFO] server-63140caa-c8bb-8a7a-ba12-9d22f43fe7aa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-65ebda27-dba2-3776-14a7-63a4a2c8a3c3 Address:65ebda27-dba2-3776-14a7-63a4a2c8a3c3} {Suffrage:Voter ID:server-779dc010-d281-c1a4-bdd8-bca00c7bc259 Address:779dc010-d281-c1a4-bdd8-bca00c7bc259} {Suffrage:Voter ID:server-63140caa-c8bb-8a7a-ba12-9d22f43fe7aa Address:63140caa-c8bb-8a7a-ba12-9d22f43fe7aa}]" +testLoggerAdapter verbose: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:10.877Z [INFO] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: entering follower state: follower="Node at 779dc010-d281-c1a4-bdd8-bca00c7bc259 [Follower]" leader= + testing.go:149: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:10.877Z [INFO] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: entering follower state: follower="Node at 779dc010-d281-c1a4-bdd8-bca00c7bc259 [Follower]" leader= +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:10.877Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:10.877Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-63140caa-c8bb-8a7a-ba12-9d22f43fe7aa: 2022-08-12T07:30:10.877Z [INFO] server-63140caa-c8bb-8a7a-ba12-9d22f43fe7aa: entering follower state: follower="Node at 63140caa-c8bb-8a7a-ba12-9d22f43fe7aa [Follower]" leader= + testing.go:149: server-63140caa-c8bb-8a7a-ba12-9d22f43fe7aa: 2022-08-12T07:30:10.877Z [INFO] server-63140caa-c8bb-8a7a-ba12-9d22f43fe7aa: entering follower state: follower="Node at 63140caa-c8bb-8a7a-ba12-9d22f43fe7aa [Follower]" leader= +testLoggerAdapter verbose: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:10.930Z [WARN] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:10.930Z [WARN] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:10.931Z [INFO] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: entering candidate state: node="Node at 779dc010-d281-c1a4-bdd8-bca00c7bc259 [Candidate]" term=2 + testing.go:149: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:10.931Z [INFO] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: entering candidate state: node="Node at 779dc010-d281-c1a4-bdd8-bca00c7bc259 [Candidate]" term=2 +testLoggerAdapter verbose: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:10.931Z [INFO] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: election won: tally=2 + testing.go:149: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:10.931Z [INFO] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: election won: tally=2 +testLoggerAdapter verbose: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:10.931Z [INFO] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: entering leader state: leader="Node at 779dc010-d281-c1a4-bdd8-bca00c7bc259 [Leader]" + testing.go:149: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:10.931Z [INFO] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: entering leader state: leader="Node at 779dc010-d281-c1a4-bdd8-bca00c7bc259 [Leader]" +testLoggerAdapter verbose: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:10.931Z [INFO] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: added peer, starting replication: peer=server-65ebda27-dba2-3776-14a7-63a4a2c8a3c3 + testing.go:149: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:10.931Z [INFO] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: added peer, starting replication: peer=server-65ebda27-dba2-3776-14a7-63a4a2c8a3c3 +testLoggerAdapter verbose: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:10.931Z [INFO] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: added peer, starting replication: peer=server-63140caa-c8bb-8a7a-ba12-9d22f43fe7aa + testing.go:149: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:10.931Z [INFO] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: added peer, starting replication: peer=server-63140caa-c8bb-8a7a-ba12-9d22f43fe7aa +testLoggerAdapter verbose: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:10.931Z [INFO] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: pipelining replication: peer="{Voter server-65ebda27-dba2-3776-14a7-63a4a2c8a3c3 65ebda27-dba2-3776-14a7-63a4a2c8a3c3}" + testing.go:149: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:10.931Z [INFO] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: pipelining replication: peer="{Voter server-65ebda27-dba2-3776-14a7-63a4a2c8a3c3 65ebda27-dba2-3776-14a7-63a4a2c8a3c3}" +testLoggerAdapter verbose: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:10.931Z [INFO] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: pipelining replication: peer="{Voter server-63140caa-c8bb-8a7a-ba12-9d22f43fe7aa 63140caa-c8bb-8a7a-ba12-9d22f43fe7aa}" + testing.go:149: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:10.931Z [INFO] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: pipelining replication: peer="{Voter server-63140caa-c8bb-8a7a-ba12-9d22f43fe7aa 63140caa-c8bb-8a7a-ba12-9d22f43fe7aa}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:11.036Z [INFO] stable state for Leader reached at 2022-08-12 07:30:10.931529 +0000 UTC m=+11.888215521 (1 nodes), 53.90704ms from start of poll, 54.9099ms from cluster start. Timeout at 2022-08-12 07:30:11.03657076 +0000 UTC m=+11.993257281, 105.04176ms after stability + testing.go:149: cluster: 2022-08-12T07:30:11.036Z [INFO] stable state for Leader reached at 2022-08-12 07:30:10.931529 +0000 UTC m=+11.888215521 (1 nodes), 53.90704ms from start of poll, 54.9099ms from cluster start. Timeout at 2022-08-12 07:30:11.03657076 +0000 UTC m=+11.993257281, 105.04176ms after stability +testLoggerAdapter verbose: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:11.037Z [INFO] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: starting snapshot up to: index=12 + testing.go:149: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:11.037Z [INFO] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: starting snapshot up to: index=12 + testing.go:153: 2022-08-12T07:30:11.037Z [INFO] creating new snapshot: path=/tmp/raft891451417/snapshots/2-12-1660289411037.tmp +testLoggerAdapter verbose: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:11.038Z [INFO] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: snapshot complete up to: index=12 + testing.go:149: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:11.038Z [INFO] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: snapshot complete up to: index=12 + testing.go:153: 2022-08-12T07:30:11.039Z [INFO] creating new snapshot: path=/tmp/raft891451417/snapshots/2-10013-1660289411039.tmp +testLoggerAdapter verbose: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:11.039Z [INFO] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: copied to local snapshot: bytes=71 + testing.go:149: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:11.039Z [INFO] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:11.040Z [INFO] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: restored user snapshot: index=1 + testing.go:149: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:11.040Z [INFO] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: restored user snapshot: index=1 +testLoggerAdapter verbose: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:11.040Z [ERROR] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: failed to get log: index=23 error="log not found" + testing.go:149: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:11.040Z [ERROR] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:11.040Z [ERROR] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: failed to get log: index=23 error="log not found" + testing.go:149: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:11.040Z [ERROR] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:11.040Z [INFO] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: aborting pipeline replication: peer="{Voter server-63140caa-c8bb-8a7a-ba12-9d22f43fe7aa 63140caa-c8bb-8a7a-ba12-9d22f43fe7aa}" + testing.go:149: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:11.040Z [INFO] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: aborting pipeline replication: peer="{Voter server-63140caa-c8bb-8a7a-ba12-9d22f43fe7aa 63140caa-c8bb-8a7a-ba12-9d22f43fe7aa}" +testLoggerAdapter verbose: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:11.040Z [INFO] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: aborting pipeline replication: peer="{Voter server-65ebda27-dba2-3776-14a7-63a4a2c8a3c3 65ebda27-dba2-3776-14a7-63a4a2c8a3c3}" + testing.go:149: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:11.040Z [INFO] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: aborting pipeline replication: peer="{Voter server-65ebda27-dba2-3776-14a7-63a4a2c8a3c3 65ebda27-dba2-3776-14a7-63a4a2c8a3c3}" +testLoggerAdapter verbose: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:11.046Z [ERROR] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: failed to get log: index=23 error="log not found" + testing.go:149: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:11.046Z [ERROR] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: failed to get log: index=23 error="log not found" + testing.go:153: 2022-08-12T07:30:11.047Z [INFO] creating new snapshot: path=/tmp/raft520850867/snapshots/2-10013-1660289411047.tmp +testLoggerAdapter verbose: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:11.047Z [ERROR] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: failed to get log: index=23 error="log not found" + testing.go:149: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:11.047Z [ERROR] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: failed to get log: index=23 error="log not found" +testLoggerAdapter verbose: server-63140caa-c8bb-8a7a-ba12-9d22f43fe7aa: 2022-08-12T07:30:11.047Z [INFO] server-63140caa-c8bb-8a7a-ba12-9d22f43fe7aa: copied to local snapshot: bytes=71 + testing.go:149: server-63140caa-c8bb-8a7a-ba12-9d22f43fe7aa: 2022-08-12T07:30:11.047Z [INFO] server-63140caa-c8bb-8a7a-ba12-9d22f43fe7aa: copied to local snapshot: bytes=71 + testing.go:153: 2022-08-12T07:30:11.048Z [INFO] creating new snapshot: path=/tmp/raft023943439/snapshots/2-10013-1660289411048.tmp +testLoggerAdapter verbose: server-63140caa-c8bb-8a7a-ba12-9d22f43fe7aa: 2022-08-12T07:30:11.048Z [INFO] server-63140caa-c8bb-8a7a-ba12-9d22f43fe7aa: Installed remote snapshot + testing.go:149: server-63140caa-c8bb-8a7a-ba12-9d22f43fe7aa: 2022-08-12T07:30:11.048Z [INFO] server-63140caa-c8bb-8a7a-ba12-9d22f43fe7aa: Installed remote snapshot +testLoggerAdapter verbose: server-63140caa-c8bb-8a7a-ba12-9d22f43fe7aa: 2022-08-12T07:30:11.048Z [WARN] server-63140caa-c8bb-8a7a-ba12-9d22f43fe7aa: skipping application of old log: index=22 + testing.go:149: server-63140caa-c8bb-8a7a-ba12-9d22f43fe7aa: 2022-08-12T07:30:11.048Z [WARN] server-63140caa-c8bb-8a7a-ba12-9d22f43fe7aa: skipping application of old log: index=22 +testLoggerAdapter verbose: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:11.048Z [INFO] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: pipelining replication: peer="{Voter server-63140caa-c8bb-8a7a-ba12-9d22f43fe7aa 63140caa-c8bb-8a7a-ba12-9d22f43fe7aa}" + testing.go:149: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:11.048Z [INFO] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: pipelining replication: peer="{Voter server-63140caa-c8bb-8a7a-ba12-9d22f43fe7aa 63140caa-c8bb-8a7a-ba12-9d22f43fe7aa}" +testLoggerAdapter verbose: server-65ebda27-dba2-3776-14a7-63a4a2c8a3c3: 2022-08-12T07:30:11.048Z [INFO] server-65ebda27-dba2-3776-14a7-63a4a2c8a3c3: copied to local snapshot: bytes=71 + testing.go:149: server-65ebda27-dba2-3776-14a7-63a4a2c8a3c3: 2022-08-12T07:30:11.048Z [INFO] server-65ebda27-dba2-3776-14a7-63a4a2c8a3c3: copied to local snapshot: bytes=71 +testLoggerAdapter verbose: server-65ebda27-dba2-3776-14a7-63a4a2c8a3c3: 2022-08-12T07:30:11.048Z [INFO] server-65ebda27-dba2-3776-14a7-63a4a2c8a3c3: Installed remote snapshot + testing.go:149: server-65ebda27-dba2-3776-14a7-63a4a2c8a3c3: 2022-08-12T07:30:11.048Z [INFO] server-65ebda27-dba2-3776-14a7-63a4a2c8a3c3: Installed remote snapshot +testLoggerAdapter verbose: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:11.049Z [INFO] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: pipelining replication: peer="{Voter server-65ebda27-dba2-3776-14a7-63a4a2c8a3c3 65ebda27-dba2-3776-14a7-63a4a2c8a3c3}" + testing.go:149: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:11.049Z [INFO] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: pipelining replication: peer="{Voter server-65ebda27-dba2-3776-14a7-63a4a2c8a3c3 65ebda27-dba2-3776-14a7-63a4a2c8a3c3}" +testLoggerAdapter verbose: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:11.064Z [INFO] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: aborting pipeline replication: peer="{Voter server-63140caa-c8bb-8a7a-ba12-9d22f43fe7aa 63140caa-c8bb-8a7a-ba12-9d22f43fe7aa}" + testing.go:149: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:11.064Z [INFO] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: aborting pipeline replication: peer="{Voter server-63140caa-c8bb-8a7a-ba12-9d22f43fe7aa 63140caa-c8bb-8a7a-ba12-9d22f43fe7aa}" +testLoggerAdapter verbose: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:11.064Z [INFO] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: aborting pipeline replication: peer="{Voter server-65ebda27-dba2-3776-14a7-63a4a2c8a3c3 65ebda27-dba2-3776-14a7-63a4a2c8a3c3}" + testing.go:149: server-779dc010-d281-c1a4-bdd8-bca00c7bc259: 2022-08-12T07:30:11.064Z [INFO] server-779dc010-d281-c1a4-bdd8-bca00c7bc259: aborting pipeline replication: peer="{Voter server-65ebda27-dba2-3776-14a7-63a4a2c8a3c3 65ebda27-dba2-3776-14a7-63a4a2c8a3c3}" --- PASS: TestRaft_UserRestore (1.23s) === RUN TestRaft_SendSnapshotFollower -testLoggerAdapter verbose: server-833f9884-bbe3-cac2-0483-712a20845245: 2021-07-10T01:05:06.580Z [INFO] server-833f9884-bbe3-cac2-0483-712a20845245: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-833f9884-bbe3-cac2-0483-712a20845245 Address:833f9884-bbe3-cac2-0483-712a20845245} {Suffrage:Voter ID:server-aff53f15-3429-aec5-85fc-802ba07ee431 Address:aff53f15-3429-aec5-85fc-802ba07ee431} {Suffrage:Voter ID:server-6f109fbf-935a-84c3-6933-fe72fe24c98a Address:6f109fbf-935a-84c3-6933-fe72fe24c98a}]" - testing.go:149: server-833f9884-bbe3-cac2-0483-712a20845245: 2021-07-10T01:05:06.580Z [INFO] server-833f9884-bbe3-cac2-0483-712a20845245: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-833f9884-bbe3-cac2-0483-712a20845245 Address:833f9884-bbe3-cac2-0483-712a20845245} {Suffrage:Voter ID:server-aff53f15-3429-aec5-85fc-802ba07ee431 Address:aff53f15-3429-aec5-85fc-802ba07ee431} {Suffrage:Voter ID:server-6f109fbf-935a-84c3-6933-fe72fe24c98a Address:6f109fbf-935a-84c3-6933-fe72fe24c98a}]" -testLoggerAdapter verbose: server-833f9884-bbe3-cac2-0483-712a20845245: 2021-07-10T01:05:06.580Z [INFO] server-833f9884-bbe3-cac2-0483-712a20845245: entering follower state: follower="Node at 833f9884-bbe3-cac2-0483-712a20845245 [Follower]" leader= - testing.go:149: server-833f9884-bbe3-cac2-0483-712a20845245: 2021-07-10T01:05:06.580Z [INFO] server-833f9884-bbe3-cac2-0483-712a20845245: entering follower state: follower="Node at 833f9884-bbe3-cac2-0483-712a20845245 [Follower]" leader= -testLoggerAdapter verbose: server-aff53f15-3429-aec5-85fc-802ba07ee431: 2021-07-10T01:05:06.580Z [INFO] server-aff53f15-3429-aec5-85fc-802ba07ee431: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-833f9884-bbe3-cac2-0483-712a20845245 Address:833f9884-bbe3-cac2-0483-712a20845245} {Suffrage:Voter ID:server-aff53f15-3429-aec5-85fc-802ba07ee431 Address:aff53f15-3429-aec5-85fc-802ba07ee431} {Suffrage:Voter ID:server-6f109fbf-935a-84c3-6933-fe72fe24c98a Address:6f109fbf-935a-84c3-6933-fe72fe24c98a}]" - testing.go:149: server-aff53f15-3429-aec5-85fc-802ba07ee431: 2021-07-10T01:05:06.580Z [INFO] server-aff53f15-3429-aec5-85fc-802ba07ee431: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-833f9884-bbe3-cac2-0483-712a20845245 Address:833f9884-bbe3-cac2-0483-712a20845245} {Suffrage:Voter ID:server-aff53f15-3429-aec5-85fc-802ba07ee431 Address:aff53f15-3429-aec5-85fc-802ba07ee431} {Suffrage:Voter ID:server-6f109fbf-935a-84c3-6933-fe72fe24c98a Address:6f109fbf-935a-84c3-6933-fe72fe24c98a}]" -testLoggerAdapter verbose: server-aff53f15-3429-aec5-85fc-802ba07ee431: 2021-07-10T01:05:06.580Z [INFO] server-aff53f15-3429-aec5-85fc-802ba07ee431: entering follower state: follower="Node at aff53f15-3429-aec5-85fc-802ba07ee431 [Follower]" leader= - testing.go:149: server-aff53f15-3429-aec5-85fc-802ba07ee431: 2021-07-10T01:05:06.580Z [INFO] server-aff53f15-3429-aec5-85fc-802ba07ee431: entering follower state: follower="Node at aff53f15-3429-aec5-85fc-802ba07ee431 [Follower]" leader= -testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.581Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-833f9884-bbe3-cac2-0483-712a20845245 Address:833f9884-bbe3-cac2-0483-712a20845245} {Suffrage:Voter ID:server-aff53f15-3429-aec5-85fc-802ba07ee431 Address:aff53f15-3429-aec5-85fc-802ba07ee431} {Suffrage:Voter ID:server-6f109fbf-935a-84c3-6933-fe72fe24c98a Address:6f109fbf-935a-84c3-6933-fe72fe24c98a}]" - testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.581Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-833f9884-bbe3-cac2-0483-712a20845245 Address:833f9884-bbe3-cac2-0483-712a20845245} {Suffrage:Voter ID:server-aff53f15-3429-aec5-85fc-802ba07ee431 Address:aff53f15-3429-aec5-85fc-802ba07ee431} {Suffrage:Voter ID:server-6f109fbf-935a-84c3-6933-fe72fe24c98a Address:6f109fbf-935a-84c3-6933-fe72fe24c98a}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:06.581Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-07-10T01:05:06.581Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.581Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: entering follower state: follower="Node at 6f109fbf-935a-84c3-6933-fe72fe24c98a [Follower]" leader= - testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.581Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: entering follower state: follower="Node at 6f109fbf-935a-84c3-6933-fe72fe24c98a [Follower]" leader= -testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.645Z [WARN] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.645Z [WARN] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.645Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: entering candidate state: node="Node at 6f109fbf-935a-84c3-6933-fe72fe24c98a [Candidate]" term=2 - testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.645Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: entering candidate state: node="Node at 6f109fbf-935a-84c3-6933-fe72fe24c98a [Candidate]" term=2 -testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.645Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: election won: tally=2 - testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.645Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: election won: tally=2 -testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.645Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: entering leader state: leader="Node at 6f109fbf-935a-84c3-6933-fe72fe24c98a [Leader]" - testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.645Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: entering leader state: leader="Node at 6f109fbf-935a-84c3-6933-fe72fe24c98a [Leader]" -testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.645Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: added peer, starting replication: peer=server-833f9884-bbe3-cac2-0483-712a20845245 - testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.645Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: added peer, starting replication: peer=server-833f9884-bbe3-cac2-0483-712a20845245 -testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.645Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: added peer, starting replication: peer=server-aff53f15-3429-aec5-85fc-802ba07ee431 - testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.645Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: added peer, starting replication: peer=server-aff53f15-3429-aec5-85fc-802ba07ee431 -testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.645Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: pipelining replication: peer="{Voter server-833f9884-bbe3-cac2-0483-712a20845245 833f9884-bbe3-cac2-0483-712a20845245}" - testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.645Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: pipelining replication: peer="{Voter server-833f9884-bbe3-cac2-0483-712a20845245 833f9884-bbe3-cac2-0483-712a20845245}" -testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.645Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: pipelining replication: peer="{Voter server-aff53f15-3429-aec5-85fc-802ba07ee431 aff53f15-3429-aec5-85fc-802ba07ee431}" - testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.645Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: pipelining replication: peer="{Voter server-aff53f15-3429-aec5-85fc-802ba07ee431 aff53f15-3429-aec5-85fc-802ba07ee431}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:06.750Z [INFO] stable state for Follower reached at 2021-07-10 01:05:06.645610134 +0000 UTC m=+12.033768124 (2 nodes), 64.416979ms from start of poll, 65.44569ms from cluster start. Timeout at 2021-07-10 01:05:06.750701142 +0000 UTC m=+12.138859112, 105.090988ms after stability - testing.go:149: cluster: 2021-07-10T01:05:06.750Z [INFO] stable state for Follower reached at 2021-07-10 01:05:06.645610134 +0000 UTC m=+12.033768124 (2 nodes), 64.416979ms from start of poll, 65.44569ms from cluster start. Timeout at 2021-07-10 01:05:06.750701142 +0000 UTC m=+12.138859112, 105.090988ms after stability -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:06.750Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:06.750Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:06.856Z [INFO] stable state for Leader reached at 2021-07-10 01:05:06.750958985 +0000 UTC m=+12.139116935 (1 nodes), 2.64µs from start of poll, 170.794501ms from cluster start. Timeout at 2021-07-10 01:05:06.856041593 +0000 UTC m=+12.244199583, 105.082648ms after stability - testing.go:149: cluster: 2021-07-10T01:05:06.856Z [INFO] stable state for Leader reached at 2021-07-10 01:05:06.750958985 +0000 UTC m=+12.139116935 (1 nodes), 2.64µs from start of poll, 170.794501ms from cluster start. Timeout at 2021-07-10 01:05:06.856041593 +0000 UTC m=+12.244199583, 105.082648ms after stability -testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.856Z [ERROR] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: failed to pipeline appendEntries: peer="{Voter server-833f9884-bbe3-cac2-0483-712a20845245 833f9884-bbe3-cac2-0483-712a20845245}" error="append pipeline closed" - testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.856Z [ERROR] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: failed to pipeline appendEntries: peer="{Voter server-833f9884-bbe3-cac2-0483-712a20845245 833f9884-bbe3-cac2-0483-712a20845245}" error="append pipeline closed" -testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.856Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: aborting pipeline replication: peer="{Voter server-833f9884-bbe3-cac2-0483-712a20845245 833f9884-bbe3-cac2-0483-712a20845245}" - testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.856Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: aborting pipeline replication: peer="{Voter server-833f9884-bbe3-cac2-0483-712a20845245 833f9884-bbe3-cac2-0483-712a20845245}" -testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.856Z [ERROR] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: failed to appendEntries to: peer="{Voter server-833f9884-bbe3-cac2-0483-712a20845245 833f9884-bbe3-cac2-0483-712a20845245}" error="failed to connect to peer: 833f9884-bbe3-cac2-0483-712a20845245" - testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.856Z [ERROR] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: failed to appendEntries to: peer="{Voter server-833f9884-bbe3-cac2-0483-712a20845245 833f9884-bbe3-cac2-0483-712a20845245}" error="failed to connect to peer: 833f9884-bbe3-cac2-0483-712a20845245" -testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.858Z [ERROR] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: failed to heartbeat to: peer=833f9884-bbe3-cac2-0483-712a20845245 error="failed to connect to peer: 833f9884-bbe3-cac2-0483-712a20845245" - testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.858Z [ERROR] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: failed to heartbeat to: peer=833f9884-bbe3-cac2-0483-712a20845245 error="failed to connect to peer: 833f9884-bbe3-cac2-0483-712a20845245" +testLoggerAdapter verbose: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.066Z [INFO] server-ce464892-e7a7-2e8f-b631-4594d607868a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce464892-e7a7-2e8f-b631-4594d607868a Address:ce464892-e7a7-2e8f-b631-4594d607868a} {Suffrage:Voter ID:server-d4e12240-e3c5-81b6-06a9-ea5e38969d01 Address:d4e12240-e3c5-81b6-06a9-ea5e38969d01} {Suffrage:Voter ID:server-2421386d-f1af-7ac4-efb1-7f737f36ab05 Address:2421386d-f1af-7ac4-efb1-7f737f36ab05}]" + testing.go:149: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.066Z [INFO] server-ce464892-e7a7-2e8f-b631-4594d607868a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce464892-e7a7-2e8f-b631-4594d607868a Address:ce464892-e7a7-2e8f-b631-4594d607868a} {Suffrage:Voter ID:server-d4e12240-e3c5-81b6-06a9-ea5e38969d01 Address:d4e12240-e3c5-81b6-06a9-ea5e38969d01} {Suffrage:Voter ID:server-2421386d-f1af-7ac4-efb1-7f737f36ab05 Address:2421386d-f1af-7ac4-efb1-7f737f36ab05}]" +testLoggerAdapter verbose: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.066Z [INFO] server-ce464892-e7a7-2e8f-b631-4594d607868a: entering follower state: follower="Node at ce464892-e7a7-2e8f-b631-4594d607868a [Follower]" leader= + testing.go:149: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.066Z [INFO] server-ce464892-e7a7-2e8f-b631-4594d607868a: entering follower state: follower="Node at ce464892-e7a7-2e8f-b631-4594d607868a [Follower]" leader= +testLoggerAdapter verbose: server-d4e12240-e3c5-81b6-06a9-ea5e38969d01: 2022-08-12T07:30:11.066Z [INFO] server-d4e12240-e3c5-81b6-06a9-ea5e38969d01: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce464892-e7a7-2e8f-b631-4594d607868a Address:ce464892-e7a7-2e8f-b631-4594d607868a} {Suffrage:Voter ID:server-d4e12240-e3c5-81b6-06a9-ea5e38969d01 Address:d4e12240-e3c5-81b6-06a9-ea5e38969d01} {Suffrage:Voter ID:server-2421386d-f1af-7ac4-efb1-7f737f36ab05 Address:2421386d-f1af-7ac4-efb1-7f737f36ab05}]" + testing.go:149: server-d4e12240-e3c5-81b6-06a9-ea5e38969d01: 2022-08-12T07:30:11.066Z [INFO] server-d4e12240-e3c5-81b6-06a9-ea5e38969d01: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce464892-e7a7-2e8f-b631-4594d607868a Address:ce464892-e7a7-2e8f-b631-4594d607868a} {Suffrage:Voter ID:server-d4e12240-e3c5-81b6-06a9-ea5e38969d01 Address:d4e12240-e3c5-81b6-06a9-ea5e38969d01} {Suffrage:Voter ID:server-2421386d-f1af-7ac4-efb1-7f737f36ab05 Address:2421386d-f1af-7ac4-efb1-7f737f36ab05}]" +testLoggerAdapter verbose: server-d4e12240-e3c5-81b6-06a9-ea5e38969d01: 2022-08-12T07:30:11.066Z [INFO] server-d4e12240-e3c5-81b6-06a9-ea5e38969d01: entering follower state: follower="Node at d4e12240-e3c5-81b6-06a9-ea5e38969d01 [Follower]" leader= + testing.go:149: server-d4e12240-e3c5-81b6-06a9-ea5e38969d01: 2022-08-12T07:30:11.066Z [INFO] server-d4e12240-e3c5-81b6-06a9-ea5e38969d01: entering follower state: follower="Node at d4e12240-e3c5-81b6-06a9-ea5e38969d01 [Follower]" leader= +testLoggerAdapter verbose: server-2421386d-f1af-7ac4-efb1-7f737f36ab05: 2022-08-12T07:30:11.066Z [INFO] server-2421386d-f1af-7ac4-efb1-7f737f36ab05: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce464892-e7a7-2e8f-b631-4594d607868a Address:ce464892-e7a7-2e8f-b631-4594d607868a} {Suffrage:Voter ID:server-d4e12240-e3c5-81b6-06a9-ea5e38969d01 Address:d4e12240-e3c5-81b6-06a9-ea5e38969d01} {Suffrage:Voter ID:server-2421386d-f1af-7ac4-efb1-7f737f36ab05 Address:2421386d-f1af-7ac4-efb1-7f737f36ab05}]" + testing.go:149: server-2421386d-f1af-7ac4-efb1-7f737f36ab05: 2022-08-12T07:30:11.066Z [INFO] server-2421386d-f1af-7ac4-efb1-7f737f36ab05: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce464892-e7a7-2e8f-b631-4594d607868a Address:ce464892-e7a7-2e8f-b631-4594d607868a} {Suffrage:Voter ID:server-d4e12240-e3c5-81b6-06a9-ea5e38969d01 Address:d4e12240-e3c5-81b6-06a9-ea5e38969d01} {Suffrage:Voter ID:server-2421386d-f1af-7ac4-efb1-7f737f36ab05 Address:2421386d-f1af-7ac4-efb1-7f737f36ab05}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:11.066Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2022-08-12T07:30:11.066Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-2421386d-f1af-7ac4-efb1-7f737f36ab05: 2022-08-12T07:30:11.066Z [INFO] server-2421386d-f1af-7ac4-efb1-7f737f36ab05: entering follower state: follower="Node at 2421386d-f1af-7ac4-efb1-7f737f36ab05 [Follower]" leader= + testing.go:149: server-2421386d-f1af-7ac4-efb1-7f737f36ab05: 2022-08-12T07:30:11.066Z [INFO] server-2421386d-f1af-7ac4-efb1-7f737f36ab05: entering follower state: follower="Node at 2421386d-f1af-7ac4-efb1-7f737f36ab05 [Follower]" leader= +testLoggerAdapter verbose: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.132Z [WARN] server-ce464892-e7a7-2e8f-b631-4594d607868a: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.132Z [WARN] server-ce464892-e7a7-2e8f-b631-4594d607868a: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.132Z [INFO] server-ce464892-e7a7-2e8f-b631-4594d607868a: entering candidate state: node="Node at ce464892-e7a7-2e8f-b631-4594d607868a [Candidate]" term=2 + testing.go:149: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.132Z [INFO] server-ce464892-e7a7-2e8f-b631-4594d607868a: entering candidate state: node="Node at ce464892-e7a7-2e8f-b631-4594d607868a [Candidate]" term=2 +testLoggerAdapter verbose: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.133Z [INFO] server-ce464892-e7a7-2e8f-b631-4594d607868a: election won: tally=2 + testing.go:149: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.133Z [INFO] server-ce464892-e7a7-2e8f-b631-4594d607868a: election won: tally=2 +testLoggerAdapter verbose: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.133Z [INFO] server-ce464892-e7a7-2e8f-b631-4594d607868a: entering leader state: leader="Node at ce464892-e7a7-2e8f-b631-4594d607868a [Leader]" + testing.go:149: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.133Z [INFO] server-ce464892-e7a7-2e8f-b631-4594d607868a: entering leader state: leader="Node at ce464892-e7a7-2e8f-b631-4594d607868a [Leader]" +testLoggerAdapter verbose: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.133Z [INFO] server-ce464892-e7a7-2e8f-b631-4594d607868a: added peer, starting replication: peer=server-d4e12240-e3c5-81b6-06a9-ea5e38969d01 + testing.go:149: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.133Z [INFO] server-ce464892-e7a7-2e8f-b631-4594d607868a: added peer, starting replication: peer=server-d4e12240-e3c5-81b6-06a9-ea5e38969d01 +testLoggerAdapter verbose: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.133Z [INFO] server-ce464892-e7a7-2e8f-b631-4594d607868a: added peer, starting replication: peer=server-2421386d-f1af-7ac4-efb1-7f737f36ab05 + testing.go:149: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.133Z [INFO] server-ce464892-e7a7-2e8f-b631-4594d607868a: added peer, starting replication: peer=server-2421386d-f1af-7ac4-efb1-7f737f36ab05 +testLoggerAdapter verbose: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.133Z [INFO] server-ce464892-e7a7-2e8f-b631-4594d607868a: pipelining replication: peer="{Voter server-d4e12240-e3c5-81b6-06a9-ea5e38969d01 d4e12240-e3c5-81b6-06a9-ea5e38969d01}" + testing.go:149: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.133Z [INFO] server-ce464892-e7a7-2e8f-b631-4594d607868a: pipelining replication: peer="{Voter server-d4e12240-e3c5-81b6-06a9-ea5e38969d01 d4e12240-e3c5-81b6-06a9-ea5e38969d01}" +testLoggerAdapter verbose: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.133Z [INFO] server-ce464892-e7a7-2e8f-b631-4594d607868a: pipelining replication: peer="{Voter server-2421386d-f1af-7ac4-efb1-7f737f36ab05 2421386d-f1af-7ac4-efb1-7f737f36ab05}" + testing.go:149: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.133Z [INFO] server-ce464892-e7a7-2e8f-b631-4594d607868a: pipelining replication: peer="{Voter server-2421386d-f1af-7ac4-efb1-7f737f36ab05 2421386d-f1af-7ac4-efb1-7f737f36ab05}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:11.238Z [INFO] stable state for Follower reached at 2022-08-12 07:30:11.13335284 +0000 UTC m=+12.090039321 (2 nodes), 66.47308ms from start of poll, 67.37976ms from cluster start. Timeout at 2022-08-12 07:30:11.23841618 +0000 UTC m=+12.195102661, 105.06334ms after stability + testing.go:149: cluster: 2022-08-12T07:30:11.238Z [INFO] stable state for Follower reached at 2022-08-12 07:30:11.13335284 +0000 UTC m=+12.090039321 (2 nodes), 66.47308ms from start of poll, 67.37976ms from cluster start. Timeout at 2022-08-12 07:30:11.23841618 +0000 UTC m=+12.195102661, 105.06334ms after stability +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:11.238Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:11.238Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:11.343Z [INFO] stable state for Leader reached at 2022-08-12 07:30:11.23867906 +0000 UTC m=+12.195365521 (1 nodes), 2.18µs from start of poll, 172.70596ms from cluster start. Timeout at 2022-08-12 07:30:11.34372856 +0000 UTC m=+12.300415041, 105.04952ms after stability + testing.go:149: cluster: 2022-08-12T07:30:11.343Z [INFO] stable state for Leader reached at 2022-08-12 07:30:11.23867906 +0000 UTC m=+12.195365521 (1 nodes), 2.18µs from start of poll, 172.70596ms from cluster start. Timeout at 2022-08-12 07:30:11.34372856 +0000 UTC m=+12.300415041, 105.04952ms after stability +testLoggerAdapter verbose: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.344Z [ERROR] server-ce464892-e7a7-2e8f-b631-4594d607868a: failed to pipeline appendEntries: peer="{Voter server-d4e12240-e3c5-81b6-06a9-ea5e38969d01 d4e12240-e3c5-81b6-06a9-ea5e38969d01}" error="append pipeline closed" + testing.go:149: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.344Z [ERROR] server-ce464892-e7a7-2e8f-b631-4594d607868a: failed to pipeline appendEntries: peer="{Voter server-d4e12240-e3c5-81b6-06a9-ea5e38969d01 d4e12240-e3c5-81b6-06a9-ea5e38969d01}" error="append pipeline closed" +testLoggerAdapter verbose: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.344Z [INFO] server-ce464892-e7a7-2e8f-b631-4594d607868a: aborting pipeline replication: peer="{Voter server-d4e12240-e3c5-81b6-06a9-ea5e38969d01 d4e12240-e3c5-81b6-06a9-ea5e38969d01}" + testing.go:149: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.344Z [INFO] server-ce464892-e7a7-2e8f-b631-4594d607868a: aborting pipeline replication: peer="{Voter server-d4e12240-e3c5-81b6-06a9-ea5e38969d01 d4e12240-e3c5-81b6-06a9-ea5e38969d01}" +testLoggerAdapter verbose: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.344Z [ERROR] server-ce464892-e7a7-2e8f-b631-4594d607868a: failed to appendEntries to: peer="{Voter server-d4e12240-e3c5-81b6-06a9-ea5e38969d01 d4e12240-e3c5-81b6-06a9-ea5e38969d01}" error="failed to connect to peer: d4e12240-e3c5-81b6-06a9-ea5e38969d01" + testing.go:149: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.344Z [ERROR] server-ce464892-e7a7-2e8f-b631-4594d607868a: failed to appendEntries to: peer="{Voter server-d4e12240-e3c5-81b6-06a9-ea5e38969d01 d4e12240-e3c5-81b6-06a9-ea5e38969d01}" error="failed to connect to peer: d4e12240-e3c5-81b6-06a9-ea5e38969d01" +testLoggerAdapter verbose: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.344Z [ERROR] server-ce464892-e7a7-2e8f-b631-4594d607868a: failed to heartbeat to: peer=d4e12240-e3c5-81b6-06a9-ea5e38969d01 error="failed to connect to peer: d4e12240-e3c5-81b6-06a9-ea5e38969d01" + testing.go:149: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.344Z [ERROR] server-ce464892-e7a7-2e8f-b631-4594d607868a: failed to heartbeat to: peer=d4e12240-e3c5-81b6-06a9-ea5e38969d01 error="failed to connect to peer: d4e12240-e3c5-81b6-06a9-ea5e38969d01" raft_test.go:1320: [INFO] Finished apply without behind follower -testLoggerAdapter verbose: server-833f9884-bbe3-cac2-0483-712a20845245: 2021-07-10T01:05:06.859Z [ERROR] server-833f9884-bbe3-cac2-0483-712a20845245: failed to take snapshot: error="nothing new to snapshot" - testing.go:149: server-833f9884-bbe3-cac2-0483-712a20845245: 2021-07-10T01:05:06.859Z [ERROR] server-833f9884-bbe3-cac2-0483-712a20845245: failed to take snapshot: error="nothing new to snapshot" -testLoggerAdapter verbose: server-aff53f15-3429-aec5-85fc-802ba07ee431: 2021-07-10T01:05:06.859Z [INFO] server-aff53f15-3429-aec5-85fc-802ba07ee431: starting snapshot up to: index=99 - testing.go:149: server-aff53f15-3429-aec5-85fc-802ba07ee431: 2021-07-10T01:05:06.859Z [INFO] server-aff53f15-3429-aec5-85fc-802ba07ee431: starting snapshot up to: index=99 - testing.go:153: 2021-07-10T01:05:06.859Z [INFO] creating new snapshot: path=/tmp/raft660393939/snapshots/2-99-1625879106859.tmp -testLoggerAdapter verbose: server-aff53f15-3429-aec5-85fc-802ba07ee431: 2021-07-10T01:05:06.860Z [INFO] server-aff53f15-3429-aec5-85fc-802ba07ee431: compacting logs: from=1 to=92 - testing.go:149: server-aff53f15-3429-aec5-85fc-802ba07ee431: 2021-07-10T01:05:06.860Z [INFO] server-aff53f15-3429-aec5-85fc-802ba07ee431: compacting logs: from=1 to=92 -testLoggerAdapter verbose: server-aff53f15-3429-aec5-85fc-802ba07ee431: 2021-07-10T01:05:06.860Z [INFO] server-aff53f15-3429-aec5-85fc-802ba07ee431: snapshot complete up to: index=99 - testing.go:149: server-aff53f15-3429-aec5-85fc-802ba07ee431: 2021-07-10T01:05:06.860Z [INFO] server-aff53f15-3429-aec5-85fc-802ba07ee431: snapshot complete up to: index=99 -testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.860Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: starting snapshot up to: index=102 - testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.860Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: starting snapshot up to: index=102 - testing.go:153: 2021-07-10T01:05:06.861Z [INFO] creating new snapshot: path=/tmp/raft560011645/snapshots/2-102-1625879106861.tmp -testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.861Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: compacting logs: from=1 to=92 - testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.861Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: compacting logs: from=1 to=92 -testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.861Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: snapshot complete up to: index=102 - testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.861Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: snapshot complete up to: index=102 -testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.867Z [ERROR] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: failed to get log: index=2 error="log not found" - testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.867Z [ERROR] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: failed to get log: index=2 error="log not found" - testing.go:153: 2021-07-10T01:05:06.867Z [INFO] creating new snapshot: path=/tmp/raft589080377/snapshots/2-102-1625879106867.tmp -testLoggerAdapter verbose: server-833f9884-bbe3-cac2-0483-712a20845245: 2021-07-10T01:05:06.868Z [INFO] server-833f9884-bbe3-cac2-0483-712a20845245: copied to local snapshot: bytes=693 - testing.go:149: server-833f9884-bbe3-cac2-0483-712a20845245: 2021-07-10T01:05:06.868Z [INFO] server-833f9884-bbe3-cac2-0483-712a20845245: copied to local snapshot: bytes=693 -testLoggerAdapter verbose: server-833f9884-bbe3-cac2-0483-712a20845245: 2021-07-10T01:05:06.868Z [INFO] server-833f9884-bbe3-cac2-0483-712a20845245: Installed remote snapshot - testing.go:149: server-833f9884-bbe3-cac2-0483-712a20845245: 2021-07-10T01:05:06.868Z [INFO] server-833f9884-bbe3-cac2-0483-712a20845245: Installed remote snapshot -testLoggerAdapter verbose: server-833f9884-bbe3-cac2-0483-712a20845245: 2021-07-10T01:05:06.868Z [WARN] server-833f9884-bbe3-cac2-0483-712a20845245: skipping application of old log: index=102 - testing.go:149: server-833f9884-bbe3-cac2-0483-712a20845245: 2021-07-10T01:05:06.868Z [WARN] server-833f9884-bbe3-cac2-0483-712a20845245: skipping application of old log: index=102 -testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.869Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: pipelining replication: peer="{Voter server-833f9884-bbe3-cac2-0483-712a20845245 833f9884-bbe3-cac2-0483-712a20845245}" - testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.869Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: pipelining replication: peer="{Voter server-833f9884-bbe3-cac2-0483-712a20845245 833f9884-bbe3-cac2-0483-712a20845245}" -testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.872Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: aborting pipeline replication: peer="{Voter server-833f9884-bbe3-cac2-0483-712a20845245 833f9884-bbe3-cac2-0483-712a20845245}" - testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.872Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: aborting pipeline replication: peer="{Voter server-833f9884-bbe3-cac2-0483-712a20845245 833f9884-bbe3-cac2-0483-712a20845245}" -testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.872Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: aborting pipeline replication: peer="{Voter server-aff53f15-3429-aec5-85fc-802ba07ee431 aff53f15-3429-aec5-85fc-802ba07ee431}" - testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.872Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: aborting pipeline replication: peer="{Voter server-aff53f15-3429-aec5-85fc-802ba07ee431 aff53f15-3429-aec5-85fc-802ba07ee431}" ---- PASS: TestRaft_SendSnapshotFollower (0.29s) +testLoggerAdapter verbose: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.348Z [INFO] server-ce464892-e7a7-2e8f-b631-4594d607868a: starting snapshot up to: index=102 + testing.go:149: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.348Z [INFO] server-ce464892-e7a7-2e8f-b631-4594d607868a: starting snapshot up to: index=102 + testing.go:153: 2022-08-12T07:30:11.348Z [INFO] creating new snapshot: path=/tmp/raft418968157/snapshots/2-102-1660289411348.tmp +testLoggerAdapter verbose: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.349Z [INFO] server-ce464892-e7a7-2e8f-b631-4594d607868a: compacting logs: from=1 to=92 + testing.go:149: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.349Z [INFO] server-ce464892-e7a7-2e8f-b631-4594d607868a: compacting logs: from=1 to=92 +testLoggerAdapter verbose: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.349Z [INFO] server-ce464892-e7a7-2e8f-b631-4594d607868a: snapshot complete up to: index=102 + testing.go:149: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.349Z [INFO] server-ce464892-e7a7-2e8f-b631-4594d607868a: snapshot complete up to: index=102 +testLoggerAdapter verbose: server-d4e12240-e3c5-81b6-06a9-ea5e38969d01: 2022-08-12T07:30:11.349Z [ERROR] server-d4e12240-e3c5-81b6-06a9-ea5e38969d01: failed to take snapshot: error="nothing new to snapshot" + testing.go:149: server-d4e12240-e3c5-81b6-06a9-ea5e38969d01: 2022-08-12T07:30:11.349Z [ERROR] server-d4e12240-e3c5-81b6-06a9-ea5e38969d01: failed to take snapshot: error="nothing new to snapshot" +testLoggerAdapter verbose: server-2421386d-f1af-7ac4-efb1-7f737f36ab05: 2022-08-12T07:30:11.349Z [INFO] server-2421386d-f1af-7ac4-efb1-7f737f36ab05: starting snapshot up to: index=100 + testing.go:149: server-2421386d-f1af-7ac4-efb1-7f737f36ab05: 2022-08-12T07:30:11.349Z [INFO] server-2421386d-f1af-7ac4-efb1-7f737f36ab05: starting snapshot up to: index=100 + testing.go:153: 2022-08-12T07:30:11.349Z [INFO] creating new snapshot: path=/tmp/raft781446881/snapshots/2-100-1660289411349.tmp +testLoggerAdapter verbose: server-2421386d-f1af-7ac4-efb1-7f737f36ab05: 2022-08-12T07:30:11.350Z [INFO] server-2421386d-f1af-7ac4-efb1-7f737f36ab05: compacting logs: from=1 to=92 + testing.go:149: server-2421386d-f1af-7ac4-efb1-7f737f36ab05: 2022-08-12T07:30:11.350Z [INFO] server-2421386d-f1af-7ac4-efb1-7f737f36ab05: compacting logs: from=1 to=92 +testLoggerAdapter verbose: server-2421386d-f1af-7ac4-efb1-7f737f36ab05: 2022-08-12T07:30:11.350Z [INFO] server-2421386d-f1af-7ac4-efb1-7f737f36ab05: snapshot complete up to: index=100 + testing.go:149: server-2421386d-f1af-7ac4-efb1-7f737f36ab05: 2022-08-12T07:30:11.350Z [INFO] server-2421386d-f1af-7ac4-efb1-7f737f36ab05: snapshot complete up to: index=100 +testLoggerAdapter verbose: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.354Z [ERROR] server-ce464892-e7a7-2e8f-b631-4594d607868a: failed to get log: index=2 error="log not found" + testing.go:149: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.354Z [ERROR] server-ce464892-e7a7-2e8f-b631-4594d607868a: failed to get log: index=2 error="log not found" + testing.go:153: 2022-08-12T07:30:11.355Z [INFO] creating new snapshot: path=/tmp/raft286298263/snapshots/2-102-1660289411355.tmp +testLoggerAdapter verbose: server-d4e12240-e3c5-81b6-06a9-ea5e38969d01: 2022-08-12T07:30:11.355Z [INFO] server-d4e12240-e3c5-81b6-06a9-ea5e38969d01: copied to local snapshot: bytes=693 + testing.go:149: server-d4e12240-e3c5-81b6-06a9-ea5e38969d01: 2022-08-12T07:30:11.355Z [INFO] server-d4e12240-e3c5-81b6-06a9-ea5e38969d01: copied to local snapshot: bytes=693 +testLoggerAdapter verbose: server-d4e12240-e3c5-81b6-06a9-ea5e38969d01: 2022-08-12T07:30:11.356Z [INFO] server-d4e12240-e3c5-81b6-06a9-ea5e38969d01: Installed remote snapshot + testing.go:149: server-d4e12240-e3c5-81b6-06a9-ea5e38969d01: 2022-08-12T07:30:11.356Z [INFO] server-d4e12240-e3c5-81b6-06a9-ea5e38969d01: Installed remote snapshot +testLoggerAdapter verbose: server-d4e12240-e3c5-81b6-06a9-ea5e38969d01: 2022-08-12T07:30:11.356Z [WARN] server-d4e12240-e3c5-81b6-06a9-ea5e38969d01: skipping application of old log: index=102 + testing.go:149: server-d4e12240-e3c5-81b6-06a9-ea5e38969d01: 2022-08-12T07:30:11.356Z [WARN] server-d4e12240-e3c5-81b6-06a9-ea5e38969d01: skipping application of old log: index=102 +testLoggerAdapter verbose: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.356Z [INFO] server-ce464892-e7a7-2e8f-b631-4594d607868a: pipelining replication: peer="{Voter server-d4e12240-e3c5-81b6-06a9-ea5e38969d01 d4e12240-e3c5-81b6-06a9-ea5e38969d01}" + testing.go:149: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.356Z [INFO] server-ce464892-e7a7-2e8f-b631-4594d607868a: pipelining replication: peer="{Voter server-d4e12240-e3c5-81b6-06a9-ea5e38969d01 d4e12240-e3c5-81b6-06a9-ea5e38969d01}" +testLoggerAdapter verbose: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.360Z [INFO] server-ce464892-e7a7-2e8f-b631-4594d607868a: aborting pipeline replication: peer="{Voter server-2421386d-f1af-7ac4-efb1-7f737f36ab05 2421386d-f1af-7ac4-efb1-7f737f36ab05}" + testing.go:149: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.360Z [INFO] server-ce464892-e7a7-2e8f-b631-4594d607868a: aborting pipeline replication: peer="{Voter server-2421386d-f1af-7ac4-efb1-7f737f36ab05 2421386d-f1af-7ac4-efb1-7f737f36ab05}" +testLoggerAdapter verbose: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.360Z [INFO] server-ce464892-e7a7-2e8f-b631-4594d607868a: aborting pipeline replication: peer="{Voter server-d4e12240-e3c5-81b6-06a9-ea5e38969d01 d4e12240-e3c5-81b6-06a9-ea5e38969d01}" + testing.go:149: server-ce464892-e7a7-2e8f-b631-4594d607868a: 2022-08-12T07:30:11.360Z [INFO] server-ce464892-e7a7-2e8f-b631-4594d607868a: aborting pipeline replication: peer="{Voter server-d4e12240-e3c5-81b6-06a9-ea5e38969d01 d4e12240-e3c5-81b6-06a9-ea5e38969d01}" +--- PASS: TestRaft_SendSnapshotFollower (0.30s) === RUN TestRaft_SendSnapshotAndLogsFollower -testLoggerAdapter verbose: server-6f4c103a-d978-ffc1-4b9d-66afe690d185: 2021-07-10T01:05:06.874Z [INFO] server-6f4c103a-d978-ffc1-4b9d-66afe690d185: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f4c103a-d978-ffc1-4b9d-66afe690d185 Address:6f4c103a-d978-ffc1-4b9d-66afe690d185} {Suffrage:Voter ID:server-1305ff47-0b5c-245e-8636-f3e7778aeda0 Address:1305ff47-0b5c-245e-8636-f3e7778aeda0} {Suffrage:Voter ID:server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8 Address:ab77d2b0-4d6f-3860-1a02-75b1f4032ac8}]" - testing.go:149: server-6f4c103a-d978-ffc1-4b9d-66afe690d185: 2021-07-10T01:05:06.874Z [INFO] server-6f4c103a-d978-ffc1-4b9d-66afe690d185: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f4c103a-d978-ffc1-4b9d-66afe690d185 Address:6f4c103a-d978-ffc1-4b9d-66afe690d185} {Suffrage:Voter ID:server-1305ff47-0b5c-245e-8636-f3e7778aeda0 Address:1305ff47-0b5c-245e-8636-f3e7778aeda0} {Suffrage:Voter ID:server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8 Address:ab77d2b0-4d6f-3860-1a02-75b1f4032ac8}]" -testLoggerAdapter verbose: server-6f4c103a-d978-ffc1-4b9d-66afe690d185: 2021-07-10T01:05:06.875Z [INFO] server-6f4c103a-d978-ffc1-4b9d-66afe690d185: entering follower state: follower="Node at 6f4c103a-d978-ffc1-4b9d-66afe690d185 [Follower]" leader= - testing.go:149: server-6f4c103a-d978-ffc1-4b9d-66afe690d185: 2021-07-10T01:05:06.875Z [INFO] server-6f4c103a-d978-ffc1-4b9d-66afe690d185: entering follower state: follower="Node at 6f4c103a-d978-ffc1-4b9d-66afe690d185 [Follower]" leader= -testLoggerAdapter verbose: server-1305ff47-0b5c-245e-8636-f3e7778aeda0: 2021-07-10T01:05:06.875Z [INFO] server-1305ff47-0b5c-245e-8636-f3e7778aeda0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f4c103a-d978-ffc1-4b9d-66afe690d185 Address:6f4c103a-d978-ffc1-4b9d-66afe690d185} {Suffrage:Voter ID:server-1305ff47-0b5c-245e-8636-f3e7778aeda0 Address:1305ff47-0b5c-245e-8636-f3e7778aeda0} {Suffrage:Voter ID:server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8 Address:ab77d2b0-4d6f-3860-1a02-75b1f4032ac8}]" - testing.go:149: server-1305ff47-0b5c-245e-8636-f3e7778aeda0: 2021-07-10T01:05:06.875Z [INFO] server-1305ff47-0b5c-245e-8636-f3e7778aeda0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f4c103a-d978-ffc1-4b9d-66afe690d185 Address:6f4c103a-d978-ffc1-4b9d-66afe690d185} {Suffrage:Voter ID:server-1305ff47-0b5c-245e-8636-f3e7778aeda0 Address:1305ff47-0b5c-245e-8636-f3e7778aeda0} {Suffrage:Voter ID:server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8 Address:ab77d2b0-4d6f-3860-1a02-75b1f4032ac8}]" -testLoggerAdapter verbose: server-1305ff47-0b5c-245e-8636-f3e7778aeda0: 2021-07-10T01:05:06.875Z [INFO] server-1305ff47-0b5c-245e-8636-f3e7778aeda0: entering follower state: follower="Node at 1305ff47-0b5c-245e-8636-f3e7778aeda0 [Follower]" leader= - testing.go:149: server-1305ff47-0b5c-245e-8636-f3e7778aeda0: 2021-07-10T01:05:06.875Z [INFO] server-1305ff47-0b5c-245e-8636-f3e7778aeda0: entering follower state: follower="Node at 1305ff47-0b5c-245e-8636-f3e7778aeda0 [Follower]" leader= -testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.875Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f4c103a-d978-ffc1-4b9d-66afe690d185 Address:6f4c103a-d978-ffc1-4b9d-66afe690d185} {Suffrage:Voter ID:server-1305ff47-0b5c-245e-8636-f3e7778aeda0 Address:1305ff47-0b5c-245e-8636-f3e7778aeda0} {Suffrage:Voter ID:server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8 Address:ab77d2b0-4d6f-3860-1a02-75b1f4032ac8}]" - testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.875Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f4c103a-d978-ffc1-4b9d-66afe690d185 Address:6f4c103a-d978-ffc1-4b9d-66afe690d185} {Suffrage:Voter ID:server-1305ff47-0b5c-245e-8636-f3e7778aeda0 Address:1305ff47-0b5c-245e-8636-f3e7778aeda0} {Suffrage:Voter ID:server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8 Address:ab77d2b0-4d6f-3860-1a02-75b1f4032ac8}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:06.875Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-07-10T01:05:06.875Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.875Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: entering follower state: follower="Node at ab77d2b0-4d6f-3860-1a02-75b1f4032ac8 [Follower]" leader= - testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.875Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: entering follower state: follower="Node at ab77d2b0-4d6f-3860-1a02-75b1f4032ac8 [Follower]" leader= -testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.933Z [WARN] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.933Z [WARN] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.933Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: entering candidate state: node="Node at ab77d2b0-4d6f-3860-1a02-75b1f4032ac8 [Candidate]" term=2 - testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.933Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: entering candidate state: node="Node at ab77d2b0-4d6f-3860-1a02-75b1f4032ac8 [Candidate]" term=2 -testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.933Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: election won: tally=2 - testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.933Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: election won: tally=2 -testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.933Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: entering leader state: leader="Node at ab77d2b0-4d6f-3860-1a02-75b1f4032ac8 [Leader]" - testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.933Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: entering leader state: leader="Node at ab77d2b0-4d6f-3860-1a02-75b1f4032ac8 [Leader]" -testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.933Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: added peer, starting replication: peer=server-6f4c103a-d978-ffc1-4b9d-66afe690d185 - testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.933Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: added peer, starting replication: peer=server-6f4c103a-d978-ffc1-4b9d-66afe690d185 -testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.933Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: added peer, starting replication: peer=server-1305ff47-0b5c-245e-8636-f3e7778aeda0 - testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.933Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: added peer, starting replication: peer=server-1305ff47-0b5c-245e-8636-f3e7778aeda0 -testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.933Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: pipelining replication: peer="{Voter server-6f4c103a-d978-ffc1-4b9d-66afe690d185 6f4c103a-d978-ffc1-4b9d-66afe690d185}" - testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.933Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: pipelining replication: peer="{Voter server-6f4c103a-d978-ffc1-4b9d-66afe690d185 6f4c103a-d978-ffc1-4b9d-66afe690d185}" -testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.933Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: pipelining replication: peer="{Voter server-1305ff47-0b5c-245e-8636-f3e7778aeda0 1305ff47-0b5c-245e-8636-f3e7778aeda0}" - testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.933Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: pipelining replication: peer="{Voter server-1305ff47-0b5c-245e-8636-f3e7778aeda0 1305ff47-0b5c-245e-8636-f3e7778aeda0}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:07.038Z [INFO] stable state for Follower reached at 2021-07-10 01:05:06.933776372 +0000 UTC m=+12.321934322 (2 nodes), 57.942631ms from start of poll, 59.008802ms from cluster start. Timeout at 2021-07-10 01:05:07.03885572 +0000 UTC m=+12.427013750, 105.079428ms after stability - testing.go:149: cluster: 2021-07-10T01:05:07.038Z [INFO] stable state for Follower reached at 2021-07-10 01:05:06.933776372 +0000 UTC m=+12.321934322 (2 nodes), 57.942631ms from start of poll, 59.008802ms from cluster start. Timeout at 2021-07-10 01:05:07.03885572 +0000 UTC m=+12.427013750, 105.079428ms after stability -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:07.039Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:07.039Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:07.144Z [INFO] stable state for Leader reached at 2021-07-10 01:05:07.039121863 +0000 UTC m=+12.427279913 (1 nodes), 2.44µs from start of poll, 164.354393ms from cluster start. Timeout at 2021-07-10 01:05:07.14419145 +0000 UTC m=+12.532349440, 105.069527ms after stability - testing.go:149: cluster: 2021-07-10T01:05:07.144Z [INFO] stable state for Leader reached at 2021-07-10 01:05:07.039121863 +0000 UTC m=+12.427279913 (1 nodes), 2.44µs from start of poll, 164.354393ms from cluster start. Timeout at 2021-07-10 01:05:07.14419145 +0000 UTC m=+12.532349440, 105.069527ms after stability -testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.144Z [ERROR] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: failed to pipeline appendEntries: peer="{Voter server-6f4c103a-d978-ffc1-4b9d-66afe690d185 6f4c103a-d978-ffc1-4b9d-66afe690d185}" error="append pipeline closed" - testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.144Z [ERROR] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: failed to pipeline appendEntries: peer="{Voter server-6f4c103a-d978-ffc1-4b9d-66afe690d185 6f4c103a-d978-ffc1-4b9d-66afe690d185}" error="append pipeline closed" -testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.144Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: aborting pipeline replication: peer="{Voter server-6f4c103a-d978-ffc1-4b9d-66afe690d185 6f4c103a-d978-ffc1-4b9d-66afe690d185}" - testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.144Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: aborting pipeline replication: peer="{Voter server-6f4c103a-d978-ffc1-4b9d-66afe690d185 6f4c103a-d978-ffc1-4b9d-66afe690d185}" -testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.144Z [ERROR] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: failed to appendEntries to: peer="{Voter server-6f4c103a-d978-ffc1-4b9d-66afe690d185 6f4c103a-d978-ffc1-4b9d-66afe690d185}" error="failed to connect to peer: 6f4c103a-d978-ffc1-4b9d-66afe690d185" - testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.144Z [ERROR] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: failed to appendEntries to: peer="{Voter server-6f4c103a-d978-ffc1-4b9d-66afe690d185 6f4c103a-d978-ffc1-4b9d-66afe690d185}" error="failed to connect to peer: 6f4c103a-d978-ffc1-4b9d-66afe690d185" +testLoggerAdapter verbose: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.362Z [INFO] server-6e22ea31-086e-67ac-5aa9-ae3312059151: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6e22ea31-086e-67ac-5aa9-ae3312059151 Address:6e22ea31-086e-67ac-5aa9-ae3312059151} {Suffrage:Voter ID:server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b Address:bd92d1cf-d04c-f6bc-b573-59cf998ab50b} {Suffrage:Voter ID:server-b632a8b0-96cc-4322-746f-d69ce5a6fd2e Address:b632a8b0-96cc-4322-746f-d69ce5a6fd2e}]" + testing.go:149: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.362Z [INFO] server-6e22ea31-086e-67ac-5aa9-ae3312059151: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6e22ea31-086e-67ac-5aa9-ae3312059151 Address:6e22ea31-086e-67ac-5aa9-ae3312059151} {Suffrage:Voter ID:server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b Address:bd92d1cf-d04c-f6bc-b573-59cf998ab50b} {Suffrage:Voter ID:server-b632a8b0-96cc-4322-746f-d69ce5a6fd2e Address:b632a8b0-96cc-4322-746f-d69ce5a6fd2e}]" +testLoggerAdapter verbose: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.362Z [INFO] server-6e22ea31-086e-67ac-5aa9-ae3312059151: entering follower state: follower="Node at 6e22ea31-086e-67ac-5aa9-ae3312059151 [Follower]" leader= + testing.go:149: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.362Z [INFO] server-6e22ea31-086e-67ac-5aa9-ae3312059151: entering follower state: follower="Node at 6e22ea31-086e-67ac-5aa9-ae3312059151 [Follower]" leader= +testLoggerAdapter verbose: server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b: 2022-08-12T07:30:11.362Z [INFO] server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6e22ea31-086e-67ac-5aa9-ae3312059151 Address:6e22ea31-086e-67ac-5aa9-ae3312059151} {Suffrage:Voter ID:server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b Address:bd92d1cf-d04c-f6bc-b573-59cf998ab50b} {Suffrage:Voter ID:server-b632a8b0-96cc-4322-746f-d69ce5a6fd2e Address:b632a8b0-96cc-4322-746f-d69ce5a6fd2e}]" + testing.go:149: server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b: 2022-08-12T07:30:11.362Z [INFO] server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6e22ea31-086e-67ac-5aa9-ae3312059151 Address:6e22ea31-086e-67ac-5aa9-ae3312059151} {Suffrage:Voter ID:server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b Address:bd92d1cf-d04c-f6bc-b573-59cf998ab50b} {Suffrage:Voter ID:server-b632a8b0-96cc-4322-746f-d69ce5a6fd2e Address:b632a8b0-96cc-4322-746f-d69ce5a6fd2e}]" +testLoggerAdapter verbose: server-b632a8b0-96cc-4322-746f-d69ce5a6fd2e: 2022-08-12T07:30:11.362Z [INFO] server-b632a8b0-96cc-4322-746f-d69ce5a6fd2e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6e22ea31-086e-67ac-5aa9-ae3312059151 Address:6e22ea31-086e-67ac-5aa9-ae3312059151} {Suffrage:Voter ID:server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b Address:bd92d1cf-d04c-f6bc-b573-59cf998ab50b} {Suffrage:Voter ID:server-b632a8b0-96cc-4322-746f-d69ce5a6fd2e Address:b632a8b0-96cc-4322-746f-d69ce5a6fd2e}]" +testLoggerAdapter verbose: server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b: 2022-08-12T07:30:11.362Z [INFO] server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b: entering follower state: follower="Node at bd92d1cf-d04c-f6bc-b573-59cf998ab50b [Follower]" leader= + testing.go:149: server-b632a8b0-96cc-4322-746f-d69ce5a6fd2e: 2022-08-12T07:30:11.362Z [INFO] server-b632a8b0-96cc-4322-746f-d69ce5a6fd2e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6e22ea31-086e-67ac-5aa9-ae3312059151 Address:6e22ea31-086e-67ac-5aa9-ae3312059151} {Suffrage:Voter ID:server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b Address:bd92d1cf-d04c-f6bc-b573-59cf998ab50b} {Suffrage:Voter ID:server-b632a8b0-96cc-4322-746f-d69ce5a6fd2e Address:b632a8b0-96cc-4322-746f-d69ce5a6fd2e}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:11.362Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-b632a8b0-96cc-4322-746f-d69ce5a6fd2e: 2022-08-12T07:30:11.362Z [INFO] server-b632a8b0-96cc-4322-746f-d69ce5a6fd2e: entering follower state: follower="Node at b632a8b0-96cc-4322-746f-d69ce5a6fd2e [Follower]" leader= + testing.go:149: cluster: 2022-08-12T07:30:11.362Z [INFO] starting stability test: raft-state=Follower + testing.go:149: server-b632a8b0-96cc-4322-746f-d69ce5a6fd2e: 2022-08-12T07:30:11.362Z [INFO] server-b632a8b0-96cc-4322-746f-d69ce5a6fd2e: entering follower state: follower="Node at b632a8b0-96cc-4322-746f-d69ce5a6fd2e [Follower]" leader= + testing.go:149: server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b: 2022-08-12T07:30:11.362Z [INFO] server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b: entering follower state: follower="Node at bd92d1cf-d04c-f6bc-b573-59cf998ab50b [Follower]" leader= +testLoggerAdapter verbose: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.448Z [WARN] server-6e22ea31-086e-67ac-5aa9-ae3312059151: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.448Z [WARN] server-6e22ea31-086e-67ac-5aa9-ae3312059151: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.448Z [INFO] server-6e22ea31-086e-67ac-5aa9-ae3312059151: entering candidate state: node="Node at 6e22ea31-086e-67ac-5aa9-ae3312059151 [Candidate]" term=2 + testing.go:149: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.448Z [INFO] server-6e22ea31-086e-67ac-5aa9-ae3312059151: entering candidate state: node="Node at 6e22ea31-086e-67ac-5aa9-ae3312059151 [Candidate]" term=2 +testLoggerAdapter verbose: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.448Z [INFO] server-6e22ea31-086e-67ac-5aa9-ae3312059151: election won: tally=2 + testing.go:149: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.448Z [INFO] server-6e22ea31-086e-67ac-5aa9-ae3312059151: election won: tally=2 +testLoggerAdapter verbose: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.448Z [INFO] server-6e22ea31-086e-67ac-5aa9-ae3312059151: entering leader state: leader="Node at 6e22ea31-086e-67ac-5aa9-ae3312059151 [Leader]" + testing.go:149: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.448Z [INFO] server-6e22ea31-086e-67ac-5aa9-ae3312059151: entering leader state: leader="Node at 6e22ea31-086e-67ac-5aa9-ae3312059151 [Leader]" +testLoggerAdapter verbose: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.448Z [INFO] server-6e22ea31-086e-67ac-5aa9-ae3312059151: added peer, starting replication: peer=server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b + testing.go:149: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.448Z [INFO] server-6e22ea31-086e-67ac-5aa9-ae3312059151: added peer, starting replication: peer=server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b +testLoggerAdapter verbose: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.448Z [INFO] server-6e22ea31-086e-67ac-5aa9-ae3312059151: added peer, starting replication: peer=server-b632a8b0-96cc-4322-746f-d69ce5a6fd2e + testing.go:149: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.448Z [INFO] server-6e22ea31-086e-67ac-5aa9-ae3312059151: added peer, starting replication: peer=server-b632a8b0-96cc-4322-746f-d69ce5a6fd2e +testLoggerAdapter verbose: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.448Z [INFO] server-6e22ea31-086e-67ac-5aa9-ae3312059151: pipelining replication: peer="{Voter server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b bd92d1cf-d04c-f6bc-b573-59cf998ab50b}" + testing.go:149: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.448Z [INFO] server-6e22ea31-086e-67ac-5aa9-ae3312059151: pipelining replication: peer="{Voter server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b bd92d1cf-d04c-f6bc-b573-59cf998ab50b}" +testLoggerAdapter verbose: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.448Z [INFO] server-6e22ea31-086e-67ac-5aa9-ae3312059151: pipelining replication: peer="{Voter server-b632a8b0-96cc-4322-746f-d69ce5a6fd2e b632a8b0-96cc-4322-746f-d69ce5a6fd2e}" + testing.go:149: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.448Z [INFO] server-6e22ea31-086e-67ac-5aa9-ae3312059151: pipelining replication: peer="{Voter server-b632a8b0-96cc-4322-746f-d69ce5a6fd2e b632a8b0-96cc-4322-746f-d69ce5a6fd2e}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:11.553Z [INFO] stable state for Follower reached at 2022-08-12 07:30:11.44857332 +0000 UTC m=+12.405259801 (2 nodes), 85.5684ms from start of poll, 86.49512ms from cluster start. Timeout at 2022-08-12 07:30:11.5536492 +0000 UTC m=+12.510335701, 105.0759ms after stability + testing.go:149: cluster: 2022-08-12T07:30:11.553Z [INFO] stable state for Follower reached at 2022-08-12 07:30:11.44857332 +0000 UTC m=+12.405259801 (2 nodes), 85.5684ms from start of poll, 86.49512ms from cluster start. Timeout at 2022-08-12 07:30:11.5536492 +0000 UTC m=+12.510335701, 105.0759ms after stability +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:11.553Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:11.553Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:11.659Z [INFO] stable state for Leader reached at 2022-08-12 07:30:11.55389768 +0000 UTC m=+12.510584161 (1 nodes), 2.8µs from start of poll, 191.81948ms from cluster start. Timeout at 2022-08-12 07:30:11.65897456 +0000 UTC m=+12.615661041, 105.07688ms after stability + testing.go:149: cluster: 2022-08-12T07:30:11.659Z [INFO] stable state for Leader reached at 2022-08-12 07:30:11.55389768 +0000 UTC m=+12.510584161 (1 nodes), 2.8µs from start of poll, 191.81948ms from cluster start. Timeout at 2022-08-12 07:30:11.65897456 +0000 UTC m=+12.615661041, 105.07688ms after stability +testLoggerAdapter verbose: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.659Z [ERROR] server-6e22ea31-086e-67ac-5aa9-ae3312059151: failed to pipeline appendEntries: peer="{Voter server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b bd92d1cf-d04c-f6bc-b573-59cf998ab50b}" error="append pipeline closed" + testing.go:149: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.659Z [ERROR] server-6e22ea31-086e-67ac-5aa9-ae3312059151: failed to pipeline appendEntries: peer="{Voter server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b bd92d1cf-d04c-f6bc-b573-59cf998ab50b}" error="append pipeline closed" +testLoggerAdapter verbose: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.659Z [INFO] server-6e22ea31-086e-67ac-5aa9-ae3312059151: aborting pipeline replication: peer="{Voter server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b bd92d1cf-d04c-f6bc-b573-59cf998ab50b}" + testing.go:149: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.659Z [INFO] server-6e22ea31-086e-67ac-5aa9-ae3312059151: aborting pipeline replication: peer="{Voter server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b bd92d1cf-d04c-f6bc-b573-59cf998ab50b}" +testLoggerAdapter verbose: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.659Z [ERROR] server-6e22ea31-086e-67ac-5aa9-ae3312059151: failed to appendEntries to: peer="{Voter server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b bd92d1cf-d04c-f6bc-b573-59cf998ab50b}" error="failed to connect to peer: bd92d1cf-d04c-f6bc-b573-59cf998ab50b" + testing.go:149: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.659Z [ERROR] server-6e22ea31-086e-67ac-5aa9-ae3312059151: failed to appendEntries to: peer="{Voter server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b bd92d1cf-d04c-f6bc-b573-59cf998ab50b}" error="failed to connect to peer: bd92d1cf-d04c-f6bc-b573-59cf998ab50b" +testLoggerAdapter verbose: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.660Z [ERROR] server-6e22ea31-086e-67ac-5aa9-ae3312059151: failed to heartbeat to: peer=bd92d1cf-d04c-f6bc-b573-59cf998ab50b error="failed to connect to peer: bd92d1cf-d04c-f6bc-b573-59cf998ab50b" + testing.go:149: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.660Z [ERROR] server-6e22ea31-086e-67ac-5aa9-ae3312059151: failed to heartbeat to: peer=bd92d1cf-d04c-f6bc-b573-59cf998ab50b error="failed to connect to peer: bd92d1cf-d04c-f6bc-b573-59cf998ab50b" raft_test.go:1362: [INFO] Finished apply without behind follower -testLoggerAdapter verbose: server-6f4c103a-d978-ffc1-4b9d-66afe690d185: 2021-07-10T01:05:07.147Z [ERROR] server-6f4c103a-d978-ffc1-4b9d-66afe690d185: failed to take snapshot: error="nothing new to snapshot" - testing.go:149: server-6f4c103a-d978-ffc1-4b9d-66afe690d185: 2021-07-10T01:05:07.147Z [ERROR] server-6f4c103a-d978-ffc1-4b9d-66afe690d185: failed to take snapshot: error="nothing new to snapshot" -testLoggerAdapter verbose: server-1305ff47-0b5c-245e-8636-f3e7778aeda0: 2021-07-10T01:05:07.147Z [INFO] server-1305ff47-0b5c-245e-8636-f3e7778aeda0: starting snapshot up to: index=100 - testing.go:149: server-1305ff47-0b5c-245e-8636-f3e7778aeda0: 2021-07-10T01:05:07.147Z [INFO] server-1305ff47-0b5c-245e-8636-f3e7778aeda0: starting snapshot up to: index=100 - testing.go:153: 2021-07-10T01:05:07.147Z [INFO] creating new snapshot: path=/tmp/raft332030977/snapshots/2-100-1625879107147.tmp -testLoggerAdapter verbose: server-1305ff47-0b5c-245e-8636-f3e7778aeda0: 2021-07-10T01:05:07.148Z [INFO] server-1305ff47-0b5c-245e-8636-f3e7778aeda0: compacting logs: from=1 to=92 - testing.go:149: server-1305ff47-0b5c-245e-8636-f3e7778aeda0: 2021-07-10T01:05:07.148Z [INFO] server-1305ff47-0b5c-245e-8636-f3e7778aeda0: compacting logs: from=1 to=92 -testLoggerAdapter verbose: server-1305ff47-0b5c-245e-8636-f3e7778aeda0: 2021-07-10T01:05:07.148Z [INFO] server-1305ff47-0b5c-245e-8636-f3e7778aeda0: snapshot complete up to: index=100 - testing.go:149: server-1305ff47-0b5c-245e-8636-f3e7778aeda0: 2021-07-10T01:05:07.148Z [INFO] server-1305ff47-0b5c-245e-8636-f3e7778aeda0: snapshot complete up to: index=100 -testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.148Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: starting snapshot up to: index=102 - testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.148Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: starting snapshot up to: index=102 - testing.go:153: 2021-07-10T01:05:07.148Z [INFO] creating new snapshot: path=/tmp/raft587756507/snapshots/2-102-1625879107148.tmp -testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.149Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: compacting logs: from=1 to=92 - testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.149Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: compacting logs: from=1 to=92 -testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.149Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: snapshot complete up to: index=102 - testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.149Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: snapshot complete up to: index=102 -testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.150Z [ERROR] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: failed to heartbeat to: peer=6f4c103a-d978-ffc1-4b9d-66afe690d185 error="failed to connect to peer: 6f4c103a-d978-ffc1-4b9d-66afe690d185" - testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.150Z [ERROR] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: failed to heartbeat to: peer=6f4c103a-d978-ffc1-4b9d-66afe690d185 error="failed to connect to peer: 6f4c103a-d978-ffc1-4b9d-66afe690d185" +testLoggerAdapter verbose: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.661Z [INFO] server-6e22ea31-086e-67ac-5aa9-ae3312059151: starting snapshot up to: index=102 + testing.go:149: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.661Z [INFO] server-6e22ea31-086e-67ac-5aa9-ae3312059151: starting snapshot up to: index=102 + testing.go:153: 2022-08-12T07:30:11.661Z [INFO] creating new snapshot: path=/tmp/raft835986363/snapshots/2-102-1660289411661.tmp +testLoggerAdapter verbose: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.662Z [INFO] server-6e22ea31-086e-67ac-5aa9-ae3312059151: compacting logs: from=1 to=92 + testing.go:149: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.662Z [INFO] server-6e22ea31-086e-67ac-5aa9-ae3312059151: compacting logs: from=1 to=92 +testLoggerAdapter verbose: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.662Z [INFO] server-6e22ea31-086e-67ac-5aa9-ae3312059151: snapshot complete up to: index=102 + testing.go:149: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.662Z [INFO] server-6e22ea31-086e-67ac-5aa9-ae3312059151: snapshot complete up to: index=102 +testLoggerAdapter verbose: server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b: 2022-08-12T07:30:11.662Z [ERROR] server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b: failed to take snapshot: error="nothing new to snapshot" + testing.go:149: server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b: 2022-08-12T07:30:11.662Z [ERROR] server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b: failed to take snapshot: error="nothing new to snapshot" +testLoggerAdapter verbose: server-b632a8b0-96cc-4322-746f-d69ce5a6fd2e: 2022-08-12T07:30:11.662Z [INFO] server-b632a8b0-96cc-4322-746f-d69ce5a6fd2e: starting snapshot up to: index=99 + testing.go:149: server-b632a8b0-96cc-4322-746f-d69ce5a6fd2e: 2022-08-12T07:30:11.662Z [INFO] server-b632a8b0-96cc-4322-746f-d69ce5a6fd2e: starting snapshot up to: index=99 + testing.go:153: 2022-08-12T07:30:11.662Z [INFO] creating new snapshot: path=/tmp/raft561089823/snapshots/2-99-1660289411662.tmp +testLoggerAdapter verbose: server-b632a8b0-96cc-4322-746f-d69ce5a6fd2e: 2022-08-12T07:30:11.663Z [INFO] server-b632a8b0-96cc-4322-746f-d69ce5a6fd2e: compacting logs: from=1 to=92 + testing.go:149: server-b632a8b0-96cc-4322-746f-d69ce5a6fd2e: 2022-08-12T07:30:11.663Z [INFO] server-b632a8b0-96cc-4322-746f-d69ce5a6fd2e: compacting logs: from=1 to=92 +testLoggerAdapter verbose: server-b632a8b0-96cc-4322-746f-d69ce5a6fd2e: 2022-08-12T07:30:11.663Z [INFO] server-b632a8b0-96cc-4322-746f-d69ce5a6fd2e: snapshot complete up to: index=99 + testing.go:149: server-b632a8b0-96cc-4322-746f-d69ce5a6fd2e: 2022-08-12T07:30:11.663Z [INFO] server-b632a8b0-96cc-4322-746f-d69ce5a6fd2e: snapshot complete up to: index=99 raft_test.go:1383: [INFO] Finished apply without behind follower -testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.154Z [ERROR] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: failed to get log: index=2 error="log not found" - testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.154Z [ERROR] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: failed to get log: index=2 error="log not found" - testing.go:153: 2021-07-10T01:05:07.155Z [INFO] creating new snapshot: path=/tmp/raft766657207/snapshots/2-102-1625879107155.tmp -testLoggerAdapter verbose: server-6f4c103a-d978-ffc1-4b9d-66afe690d185: 2021-07-10T01:05:07.155Z [INFO] server-6f4c103a-d978-ffc1-4b9d-66afe690d185: copied to local snapshot: bytes=693 - testing.go:149: server-6f4c103a-d978-ffc1-4b9d-66afe690d185: 2021-07-10T01:05:07.155Z [INFO] server-6f4c103a-d978-ffc1-4b9d-66afe690d185: copied to local snapshot: bytes=693 -testLoggerAdapter verbose: server-6f4c103a-d978-ffc1-4b9d-66afe690d185: 2021-07-10T01:05:07.156Z [INFO] server-6f4c103a-d978-ffc1-4b9d-66afe690d185: Installed remote snapshot - testing.go:149: server-6f4c103a-d978-ffc1-4b9d-66afe690d185: 2021-07-10T01:05:07.156Z [INFO] server-6f4c103a-d978-ffc1-4b9d-66afe690d185: Installed remote snapshot -testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.156Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: pipelining replication: peer="{Voter server-6f4c103a-d978-ffc1-4b9d-66afe690d185 6f4c103a-d978-ffc1-4b9d-66afe690d185}" - testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.156Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: pipelining replication: peer="{Voter server-6f4c103a-d978-ffc1-4b9d-66afe690d185 6f4c103a-d978-ffc1-4b9d-66afe690d185}" -testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.161Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: aborting pipeline replication: peer="{Voter server-1305ff47-0b5c-245e-8636-f3e7778aeda0 1305ff47-0b5c-245e-8636-f3e7778aeda0}" - testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.161Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: aborting pipeline replication: peer="{Voter server-1305ff47-0b5c-245e-8636-f3e7778aeda0 1305ff47-0b5c-245e-8636-f3e7778aeda0}" -testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.161Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: aborting pipeline replication: peer="{Voter server-6f4c103a-d978-ffc1-4b9d-66afe690d185 6f4c103a-d978-ffc1-4b9d-66afe690d185}" - testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.161Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: aborting pipeline replication: peer="{Voter server-6f4c103a-d978-ffc1-4b9d-66afe690d185 6f4c103a-d978-ffc1-4b9d-66afe690d185}" ---- PASS: TestRaft_SendSnapshotAndLogsFollower (0.29s) +testLoggerAdapter verbose: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.669Z [ERROR] server-6e22ea31-086e-67ac-5aa9-ae3312059151: failed to get log: index=2 error="log not found" + testing.go:149: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.669Z [ERROR] server-6e22ea31-086e-67ac-5aa9-ae3312059151: failed to get log: index=2 error="log not found" + testing.go:153: 2022-08-12T07:30:11.670Z [INFO] creating new snapshot: path=/tmp/raft521067941/snapshots/2-102-1660289411670.tmp +testLoggerAdapter verbose: server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b: 2022-08-12T07:30:11.670Z [INFO] server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b: copied to local snapshot: bytes=693 + testing.go:149: server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b: 2022-08-12T07:30:11.670Z [INFO] server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b: copied to local snapshot: bytes=693 +testLoggerAdapter verbose: server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b: 2022-08-12T07:30:11.671Z [INFO] server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b: Installed remote snapshot + testing.go:149: server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b: 2022-08-12T07:30:11.671Z [INFO] server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b: Installed remote snapshot +testLoggerAdapter verbose: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.671Z [INFO] server-6e22ea31-086e-67ac-5aa9-ae3312059151: pipelining replication: peer="{Voter server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b bd92d1cf-d04c-f6bc-b573-59cf998ab50b}" + testing.go:149: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.671Z [INFO] server-6e22ea31-086e-67ac-5aa9-ae3312059151: pipelining replication: peer="{Voter server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b bd92d1cf-d04c-f6bc-b573-59cf998ab50b}" +testLoggerAdapter verbose: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.676Z [INFO] server-6e22ea31-086e-67ac-5aa9-ae3312059151: aborting pipeline replication: peer="{Voter server-b632a8b0-96cc-4322-746f-d69ce5a6fd2e b632a8b0-96cc-4322-746f-d69ce5a6fd2e}" + testing.go:149: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.676Z [INFO] server-6e22ea31-086e-67ac-5aa9-ae3312059151: aborting pipeline replication: peer="{Voter server-b632a8b0-96cc-4322-746f-d69ce5a6fd2e b632a8b0-96cc-4322-746f-d69ce5a6fd2e}" +testLoggerAdapter verbose: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.676Z [INFO] server-6e22ea31-086e-67ac-5aa9-ae3312059151: aborting pipeline replication: peer="{Voter server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b bd92d1cf-d04c-f6bc-b573-59cf998ab50b}" + testing.go:149: server-6e22ea31-086e-67ac-5aa9-ae3312059151: 2022-08-12T07:30:11.676Z [INFO] server-6e22ea31-086e-67ac-5aa9-ae3312059151: aborting pipeline replication: peer="{Voter server-bd92d1cf-d04c-f6bc-b573-59cf998ab50b bd92d1cf-d04c-f6bc-b573-59cf998ab50b}" +--- PASS: TestRaft_SendSnapshotAndLogsFollower (0.32s) === RUN TestRaft_ReJoinFollower -testLoggerAdapter verbose: server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: 2021-07-10T01:05:07.163Z [INFO] server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 Address:0d2493c3-115c-992a-5a45-e72b1f0a85b1} {Suffrage:Voter ID:server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 Address:b76e5ee7-3f04-6b7d-2af0-4a98745184b4} {Suffrage:Voter ID:server-99962721-dd8b-44ac-1b09-7ed704d78cc5 Address:99962721-dd8b-44ac-1b09-7ed704d78cc5}]" - testing.go:149: server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: 2021-07-10T01:05:07.163Z [INFO] server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 Address:0d2493c3-115c-992a-5a45-e72b1f0a85b1} {Suffrage:Voter ID:server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 Address:b76e5ee7-3f04-6b7d-2af0-4a98745184b4} {Suffrage:Voter ID:server-99962721-dd8b-44ac-1b09-7ed704d78cc5 Address:99962721-dd8b-44ac-1b09-7ed704d78cc5}]" -testLoggerAdapter verbose: server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: 2021-07-10T01:05:07.163Z [INFO] server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: entering follower state: follower="Node at 0d2493c3-115c-992a-5a45-e72b1f0a85b1 [Follower]" leader= - testing.go:149: server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: 2021-07-10T01:05:07.163Z [INFO] server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: entering follower state: follower="Node at 0d2493c3-115c-992a-5a45-e72b1f0a85b1 [Follower]" leader= -testLoggerAdapter verbose: server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4: 2021-07-10T01:05:07.164Z [INFO] server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 Address:0d2493c3-115c-992a-5a45-e72b1f0a85b1} {Suffrage:Voter ID:server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 Address:b76e5ee7-3f04-6b7d-2af0-4a98745184b4} {Suffrage:Voter ID:server-99962721-dd8b-44ac-1b09-7ed704d78cc5 Address:99962721-dd8b-44ac-1b09-7ed704d78cc5}]" - testing.go:149: server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4: 2021-07-10T01:05:07.164Z [INFO] server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 Address:0d2493c3-115c-992a-5a45-e72b1f0a85b1} {Suffrage:Voter ID:server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 Address:b76e5ee7-3f04-6b7d-2af0-4a98745184b4} {Suffrage:Voter ID:server-99962721-dd8b-44ac-1b09-7ed704d78cc5 Address:99962721-dd8b-44ac-1b09-7ed704d78cc5}]" -testLoggerAdapter verbose: server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4: 2021-07-10T01:05:07.164Z [INFO] server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4: entering follower state: follower="Node at b76e5ee7-3f04-6b7d-2af0-4a98745184b4 [Follower]" leader= - testing.go:149: server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4: 2021-07-10T01:05:07.164Z [INFO] server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4: entering follower state: follower="Node at b76e5ee7-3f04-6b7d-2af0-4a98745184b4 [Follower]" leader= -testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.164Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 Address:0d2493c3-115c-992a-5a45-e72b1f0a85b1} {Suffrage:Voter ID:server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 Address:b76e5ee7-3f04-6b7d-2af0-4a98745184b4} {Suffrage:Voter ID:server-99962721-dd8b-44ac-1b09-7ed704d78cc5 Address:99962721-dd8b-44ac-1b09-7ed704d78cc5}]" - testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.164Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 Address:0d2493c3-115c-992a-5a45-e72b1f0a85b1} {Suffrage:Voter ID:server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 Address:b76e5ee7-3f04-6b7d-2af0-4a98745184b4} {Suffrage:Voter ID:server-99962721-dd8b-44ac-1b09-7ed704d78cc5 Address:99962721-dd8b-44ac-1b09-7ed704d78cc5}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:07.164Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:07.164Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.164Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: entering follower state: follower="Node at 99962721-dd8b-44ac-1b09-7ed704d78cc5 [Follower]" leader= - testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.164Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: entering follower state: follower="Node at 99962721-dd8b-44ac-1b09-7ed704d78cc5 [Follower]" leader= -testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.219Z [WARN] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.219Z [WARN] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.219Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: entering candidate state: node="Node at 99962721-dd8b-44ac-1b09-7ed704d78cc5 [Candidate]" term=2 - testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.219Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: entering candidate state: node="Node at 99962721-dd8b-44ac-1b09-7ed704d78cc5 [Candidate]" term=2 -testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.219Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: election won: tally=2 - testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.219Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: election won: tally=2 -testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.219Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: entering leader state: leader="Node at 99962721-dd8b-44ac-1b09-7ed704d78cc5 [Leader]" - testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.219Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: entering leader state: leader="Node at 99962721-dd8b-44ac-1b09-7ed704d78cc5 [Leader]" -testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.220Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: added peer, starting replication: peer=server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 - testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.220Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: added peer, starting replication: peer=server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 -testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.220Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: added peer, starting replication: peer=server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 - testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.220Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: added peer, starting replication: peer=server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 -testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.220Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: pipelining replication: peer="{Voter server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 0d2493c3-115c-992a-5a45-e72b1f0a85b1}" - testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.220Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: pipelining replication: peer="{Voter server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 0d2493c3-115c-992a-5a45-e72b1f0a85b1}" -testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.220Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: pipelining replication: peer="{Voter server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 b76e5ee7-3f04-6b7d-2af0-4a98745184b4}" - testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.220Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: pipelining replication: peer="{Voter server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 b76e5ee7-3f04-6b7d-2af0-4a98745184b4}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:07.325Z [INFO] stable state for Leader reached at 2021-07-10 01:05:07.22005621 +0000 UTC m=+12.608214200 (1 nodes), 55.512885ms from start of poll, 56.496055ms from cluster start. Timeout at 2021-07-10 01:05:07.325127738 +0000 UTC m=+12.713285728, 105.071528ms after stability - testing.go:149: cluster: 2021-07-10T01:05:07.325Z [INFO] stable state for Leader reached at 2021-07-10 01:05:07.22005621 +0000 UTC m=+12.608214200 (1 nodes), 55.512885ms from start of poll, 56.496055ms from cluster start. Timeout at 2021-07-10 01:05:07.325127738 +0000 UTC m=+12.713285728, 105.071528ms after stability -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:07.330Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-07-10T01:05:07.330Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:07.435Z [INFO] stable state for Follower reached at 2021-07-10 01:05:07.330558955 +0000 UTC m=+12.718716905 (2 nodes), 3.2µs from start of poll, 166.99876ms from cluster start. Timeout at 2021-07-10 01:05:07.435623403 +0000 UTC m=+12.823781373, 105.064468ms after stability - testing.go:149: cluster: 2021-07-10T01:05:07.435Z [INFO] stable state for Follower reached at 2021-07-10 01:05:07.330558955 +0000 UTC m=+12.718716905 (2 nodes), 3.2µs from start of poll, 166.99876ms from cluster start. Timeout at 2021-07-10 01:05:07.435623403 +0000 UTC m=+12.823781373, 105.064468ms after stability -testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.435Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: updating configuration: command=RemoveServer server-id=server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 server-addr= servers="[{Suffrage:Voter ID:server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 Address:b76e5ee7-3f04-6b7d-2af0-4a98745184b4} {Suffrage:Voter ID:server-99962721-dd8b-44ac-1b09-7ed704d78cc5 Address:99962721-dd8b-44ac-1b09-7ed704d78cc5}]" - testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.435Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: updating configuration: command=RemoveServer server-id=server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 server-addr= servers="[{Suffrage:Voter ID:server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 Address:b76e5ee7-3f04-6b7d-2af0-4a98745184b4} {Suffrage:Voter ID:server-99962721-dd8b-44ac-1b09-7ed704d78cc5 Address:99962721-dd8b-44ac-1b09-7ed704d78cc5}]" -testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.436Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: removed peer, stopping replication: peer=server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 last-index=3 - testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.436Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: removed peer, stopping replication: peer=server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 last-index=3 -testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.436Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: aborting pipeline replication: peer="{Voter server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 0d2493c3-115c-992a-5a45-e72b1f0a85b1}" - testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.436Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: aborting pipeline replication: peer="{Voter server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 0d2493c3-115c-992a-5a45-e72b1f0a85b1}" -testLoggerAdapter verbose: server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: 2021-07-10T01:05:07.494Z [WARN] server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: heartbeat timeout reached, starting election: last-leader=99962721-dd8b-44ac-1b09-7ed704d78cc5 - testing.go:149: server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: 2021-07-10T01:05:07.494Z [WARN] server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: heartbeat timeout reached, starting election: last-leader=99962721-dd8b-44ac-1b09-7ed704d78cc5 -testLoggerAdapter verbose: server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: 2021-07-10T01:05:07.494Z [INFO] server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: entering candidate state: node="Node at 0d2493c3-115c-992a-5a45-e72b1f0a85b1 [Candidate]" term=3 - testing.go:149: server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: 2021-07-10T01:05:07.494Z [INFO] server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: entering candidate state: node="Node at 0d2493c3-115c-992a-5a45-e72b1f0a85b1 [Candidate]" term=3 -testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.495Z [WARN] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: rejecting vote request since we have a leader: from=0d2493c3-115c-992a-5a45-e72b1f0a85b1 leader=99962721-dd8b-44ac-1b09-7ed704d78cc5 -testLoggerAdapter verbose: server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4: 2021-07-10T01:05:07.495Z [WARN] server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4: rejecting vote request since we have a leader: from=0d2493c3-115c-992a-5a45-e72b1f0a85b1 leader=99962721-dd8b-44ac-1b09-7ed704d78cc5 - testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.495Z [WARN] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: rejecting vote request since we have a leader: from=0d2493c3-115c-992a-5a45-e72b1f0a85b1 leader=99962721-dd8b-44ac-1b09-7ed704d78cc5 - testing.go:149: server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4: 2021-07-10T01:05:07.495Z [WARN] server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4: rejecting vote request since we have a leader: from=0d2493c3-115c-992a-5a45-e72b1f0a85b1 leader=99962721-dd8b-44ac-1b09-7ed704d78cc5 -testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.541Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: updating configuration: command=AddStaging server-id=server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 server-addr=0d2493c3-115c-992a-5a45-e72b1f0a85b1 servers="[{Suffrage:Voter ID:server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 Address:b76e5ee7-3f04-6b7d-2af0-4a98745184b4} {Suffrage:Voter ID:server-99962721-dd8b-44ac-1b09-7ed704d78cc5 Address:99962721-dd8b-44ac-1b09-7ed704d78cc5} {Suffrage:Voter ID:server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 Address:0d2493c3-115c-992a-5a45-e72b1f0a85b1}]" - testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.541Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: updating configuration: command=AddStaging server-id=server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 server-addr=0d2493c3-115c-992a-5a45-e72b1f0a85b1 servers="[{Suffrage:Voter ID:server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 Address:b76e5ee7-3f04-6b7d-2af0-4a98745184b4} {Suffrage:Voter ID:server-99962721-dd8b-44ac-1b09-7ed704d78cc5 Address:99962721-dd8b-44ac-1b09-7ed704d78cc5} {Suffrage:Voter ID:server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 Address:0d2493c3-115c-992a-5a45-e72b1f0a85b1}]" -testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.541Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: added peer, starting replication: peer=server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 - testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.541Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: added peer, starting replication: peer=server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 -testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.541Z [ERROR] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: peer has newer term, stopping replication: peer="{Voter server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 0d2493c3-115c-992a-5a45-e72b1f0a85b1}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:07.541Z [INFO] starting stability test: raft-state=Leader - testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.541Z [ERROR] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: peer has newer term, stopping replication: peer="{Voter server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 0d2493c3-115c-992a-5a45-e72b1f0a85b1}" - testing.go:149: cluster: 2021-07-10T01:05:07.541Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.542Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: entering follower state: follower="Node at 99962721-dd8b-44ac-1b09-7ed704d78cc5 [Follower]" leader= - testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.542Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: entering follower state: follower="Node at 99962721-dd8b-44ac-1b09-7ed704d78cc5 [Follower]" leader= -testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.542Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: aborting pipeline replication: peer="{Voter server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 b76e5ee7-3f04-6b7d-2af0-4a98745184b4}" - testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.542Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: aborting pipeline replication: peer="{Voter server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 b76e5ee7-3f04-6b7d-2af0-4a98745184b4}" -testLoggerAdapter verbose: server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: 2021-07-10T01:05:07.594Z [WARN] server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: Election timeout reached, restarting election - testing.go:149: server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: 2021-07-10T01:05:07.594Z [WARN] server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: Election timeout reached, restarting election -testLoggerAdapter verbose: server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: 2021-07-10T01:05:07.594Z [INFO] server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: entering candidate state: node="Node at 0d2493c3-115c-992a-5a45-e72b1f0a85b1 [Candidate]" term=4 - testing.go:149: server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: 2021-07-10T01:05:07.594Z [INFO] server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: entering candidate state: node="Node at 0d2493c3-115c-992a-5a45-e72b1f0a85b1 [Candidate]" term=4 -testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.594Z [WARN] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: rejecting vote request since our last index is greater: candidate=0d2493c3-115c-992a-5a45-e72b1f0a85b1 last-index=4 last-candidate-index=3 - testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.594Z [WARN] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: rejecting vote request since our last index is greater: candidate=0d2493c3-115c-992a-5a45-e72b1f0a85b1 last-index=4 last-candidate-index=3 -testLoggerAdapter verbose: server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4: 2021-07-10T01:05:07.594Z [WARN] server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4: rejecting vote request since we have a leader: from=0d2493c3-115c-992a-5a45-e72b1f0a85b1 leader=99962721-dd8b-44ac-1b09-7ed704d78cc5 - testing.go:149: server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4: 2021-07-10T01:05:07.594Z [WARN] server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4: rejecting vote request since we have a leader: from=0d2493c3-115c-992a-5a45-e72b1f0a85b1 leader=99962721-dd8b-44ac-1b09-7ed704d78cc5 -testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.617Z [WARN] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.617Z [WARN] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.617Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: entering candidate state: node="Node at 99962721-dd8b-44ac-1b09-7ed704d78cc5 [Candidate]" term=5 - testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.617Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: entering candidate state: node="Node at 99962721-dd8b-44ac-1b09-7ed704d78cc5 [Candidate]" term=5 -testLoggerAdapter verbose: server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: 2021-07-10T01:05:07.617Z [INFO] server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: entering follower state: follower="Node at 0d2493c3-115c-992a-5a45-e72b1f0a85b1 [Follower]" leader= -testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.617Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: election won: tally=2 - testing.go:149: server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: 2021-07-10T01:05:07.617Z [INFO] server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: entering follower state: follower="Node at 0d2493c3-115c-992a-5a45-e72b1f0a85b1 [Follower]" leader= - testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.617Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: election won: tally=2 -testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.617Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: entering leader state: leader="Node at 99962721-dd8b-44ac-1b09-7ed704d78cc5 [Leader]" - testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.617Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: entering leader state: leader="Node at 99962721-dd8b-44ac-1b09-7ed704d78cc5 [Leader]" -testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.618Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: added peer, starting replication: peer=server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 - testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.618Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: added peer, starting replication: peer=server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 -testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.618Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: added peer, starting replication: peer=server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 - testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.618Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: added peer, starting replication: peer=server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 -testLoggerAdapter verbose: server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: 2021-07-10T01:05:07.618Z [WARN] server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: failed to get previous log: previous-index=4 last-index=3 error="log not found" - testing.go:149: server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: 2021-07-10T01:05:07.618Z [WARN] server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: failed to get previous log: previous-index=4 last-index=3 error="log not found" -testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.618Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: pipelining replication: peer="{Voter server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 b76e5ee7-3f04-6b7d-2af0-4a98745184b4}" - testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.618Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: pipelining replication: peer="{Voter server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 b76e5ee7-3f04-6b7d-2af0-4a98745184b4}" -testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.618Z [WARN] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: appendEntries rejected, sending older logs: peer="{Voter server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 0d2493c3-115c-992a-5a45-e72b1f0a85b1}" next=4 - testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.618Z [WARN] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: appendEntries rejected, sending older logs: peer="{Voter server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 0d2493c3-115c-992a-5a45-e72b1f0a85b1}" next=4 -testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.619Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: pipelining replication: peer="{Voter server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 0d2493c3-115c-992a-5a45-e72b1f0a85b1}" - testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.619Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: pipelining replication: peer="{Voter server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 0d2493c3-115c-992a-5a45-e72b1f0a85b1}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:07.723Z [INFO] stable state for Leader reached at 2021-07-10 01:05:07.618001026 +0000 UTC m=+13.006159016 (1 nodes), 75.918241ms from start of poll, 454.440871ms from cluster start. Timeout at 2021-07-10 01:05:07.723069513 +0000 UTC m=+13.111227483, 105.068467ms after stability - testing.go:149: cluster: 2021-07-10T01:05:07.723Z [INFO] stable state for Leader reached at 2021-07-10 01:05:07.618001026 +0000 UTC m=+13.006159016 (1 nodes), 75.918241ms from start of poll, 454.440871ms from cluster start. Timeout at 2021-07-10 01:05:07.723069513 +0000 UTC m=+13.111227483, 105.068467ms after stability -testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.723Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: aborting pipeline replication: peer="{Voter server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 0d2493c3-115c-992a-5a45-e72b1f0a85b1}" - testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.723Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: aborting pipeline replication: peer="{Voter server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 0d2493c3-115c-992a-5a45-e72b1f0a85b1}" -testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.723Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: aborting pipeline replication: peer="{Voter server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 b76e5ee7-3f04-6b7d-2af0-4a98745184b4}" - testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.723Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: aborting pipeline replication: peer="{Voter server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 b76e5ee7-3f04-6b7d-2af0-4a98745184b4}" ---- PASS: TestRaft_ReJoinFollower (0.56s) +testLoggerAdapter verbose: server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9: 2022-08-12T07:30:11.678Z [INFO] server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9 Address:4c138c72-aa59-d43c-1ea3-22169a2ec8a9} {Suffrage:Voter ID:server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c Address:3a4fc9c3-4a54-4e7b-8e08-64676a68e33c} {Suffrage:Voter ID:server-a5759801-0e55-5f3d-850a-e88fd230beec Address:a5759801-0e55-5f3d-850a-e88fd230beec}]" + testing.go:149: server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9: 2022-08-12T07:30:11.678Z [INFO] server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9 Address:4c138c72-aa59-d43c-1ea3-22169a2ec8a9} {Suffrage:Voter ID:server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c Address:3a4fc9c3-4a54-4e7b-8e08-64676a68e33c} {Suffrage:Voter ID:server-a5759801-0e55-5f3d-850a-e88fd230beec Address:a5759801-0e55-5f3d-850a-e88fd230beec}]" +testLoggerAdapter verbose: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:11.679Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9 Address:4c138c72-aa59-d43c-1ea3-22169a2ec8a9} {Suffrage:Voter ID:server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c Address:3a4fc9c3-4a54-4e7b-8e08-64676a68e33c} {Suffrage:Voter ID:server-a5759801-0e55-5f3d-850a-e88fd230beec Address:a5759801-0e55-5f3d-850a-e88fd230beec}]" + testing.go:149: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:11.679Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9 Address:4c138c72-aa59-d43c-1ea3-22169a2ec8a9} {Suffrage:Voter ID:server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c Address:3a4fc9c3-4a54-4e7b-8e08-64676a68e33c} {Suffrage:Voter ID:server-a5759801-0e55-5f3d-850a-e88fd230beec Address:a5759801-0e55-5f3d-850a-e88fd230beec}]" +testLoggerAdapter verbose: server-a5759801-0e55-5f3d-850a-e88fd230beec: 2022-08-12T07:30:11.679Z [INFO] server-a5759801-0e55-5f3d-850a-e88fd230beec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9 Address:4c138c72-aa59-d43c-1ea3-22169a2ec8a9} {Suffrage:Voter ID:server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c Address:3a4fc9c3-4a54-4e7b-8e08-64676a68e33c} {Suffrage:Voter ID:server-a5759801-0e55-5f3d-850a-e88fd230beec Address:a5759801-0e55-5f3d-850a-e88fd230beec}]" + testing.go:149: server-a5759801-0e55-5f3d-850a-e88fd230beec: 2022-08-12T07:30:11.679Z [INFO] server-a5759801-0e55-5f3d-850a-e88fd230beec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9 Address:4c138c72-aa59-d43c-1ea3-22169a2ec8a9} {Suffrage:Voter ID:server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c Address:3a4fc9c3-4a54-4e7b-8e08-64676a68e33c} {Suffrage:Voter ID:server-a5759801-0e55-5f3d-850a-e88fd230beec Address:a5759801-0e55-5f3d-850a-e88fd230beec}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:11.679Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:11.679Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9: 2022-08-12T07:30:11.680Z [INFO] server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9: entering follower state: follower="Node at 4c138c72-aa59-d43c-1ea3-22169a2ec8a9 [Follower]" leader= + testing.go:149: server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9: 2022-08-12T07:30:11.680Z [INFO] server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9: entering follower state: follower="Node at 4c138c72-aa59-d43c-1ea3-22169a2ec8a9 [Follower]" leader= +testLoggerAdapter verbose: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:11.680Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: entering follower state: follower="Node at 3a4fc9c3-4a54-4e7b-8e08-64676a68e33c [Follower]" leader= + testing.go:149: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:11.680Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: entering follower state: follower="Node at 3a4fc9c3-4a54-4e7b-8e08-64676a68e33c [Follower]" leader= +testLoggerAdapter verbose: server-a5759801-0e55-5f3d-850a-e88fd230beec: 2022-08-12T07:30:11.680Z [INFO] server-a5759801-0e55-5f3d-850a-e88fd230beec: entering follower state: follower="Node at a5759801-0e55-5f3d-850a-e88fd230beec [Follower]" leader= + testing.go:149: server-a5759801-0e55-5f3d-850a-e88fd230beec: 2022-08-12T07:30:11.680Z [INFO] server-a5759801-0e55-5f3d-850a-e88fd230beec: entering follower state: follower="Node at a5759801-0e55-5f3d-850a-e88fd230beec [Follower]" leader= +testLoggerAdapter verbose: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:11.735Z [WARN] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:11.735Z [WARN] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:11.735Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: entering candidate state: node="Node at 3a4fc9c3-4a54-4e7b-8e08-64676a68e33c [Candidate]" term=2 + testing.go:149: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:11.735Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: entering candidate state: node="Node at 3a4fc9c3-4a54-4e7b-8e08-64676a68e33c [Candidate]" term=2 +testLoggerAdapter verbose: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:11.736Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: election won: tally=2 + testing.go:149: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:11.736Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: election won: tally=2 +testLoggerAdapter verbose: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:11.736Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: entering leader state: leader="Node at 3a4fc9c3-4a54-4e7b-8e08-64676a68e33c [Leader]" + testing.go:149: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:11.736Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: entering leader state: leader="Node at 3a4fc9c3-4a54-4e7b-8e08-64676a68e33c [Leader]" +testLoggerAdapter verbose: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:11.736Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: added peer, starting replication: peer=server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9 + testing.go:149: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:11.736Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: added peer, starting replication: peer=server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9 +testLoggerAdapter verbose: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:11.736Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: added peer, starting replication: peer=server-a5759801-0e55-5f3d-850a-e88fd230beec + testing.go:149: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:11.736Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: added peer, starting replication: peer=server-a5759801-0e55-5f3d-850a-e88fd230beec +testLoggerAdapter verbose: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:11.736Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: pipelining replication: peer="{Voter server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9 4c138c72-aa59-d43c-1ea3-22169a2ec8a9}" + testing.go:149: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:11.736Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: pipelining replication: peer="{Voter server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9 4c138c72-aa59-d43c-1ea3-22169a2ec8a9}" +testLoggerAdapter verbose: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:11.736Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: pipelining replication: peer="{Voter server-a5759801-0e55-5f3d-850a-e88fd230beec a5759801-0e55-5f3d-850a-e88fd230beec}" + testing.go:149: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:11.736Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: pipelining replication: peer="{Voter server-a5759801-0e55-5f3d-850a-e88fd230beec a5759801-0e55-5f3d-850a-e88fd230beec}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:11.841Z [INFO] stable state for Leader reached at 2022-08-12 07:30:11.73614888 +0000 UTC m=+12.692835381 (1 nodes), 56.24534ms from start of poll, 57.92612ms from cluster start. Timeout at 2022-08-12 07:30:11.84123544 +0000 UTC m=+12.797921941, 105.08656ms after stability + testing.go:149: cluster: 2022-08-12T07:30:11.841Z [INFO] stable state for Leader reached at 2022-08-12 07:30:11.73614888 +0000 UTC m=+12.692835381 (1 nodes), 56.24534ms from start of poll, 57.92612ms from cluster start. Timeout at 2022-08-12 07:30:11.84123544 +0000 UTC m=+12.797921941, 105.08656ms after stability +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:11.846Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2022-08-12T07:30:11.846Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:11.951Z [INFO] stable state for Follower reached at 2022-08-12 07:30:11.84667636 +0000 UTC m=+12.803362821 (2 nodes), 2.5µs from start of poll, 168.45356ms from cluster start. Timeout at 2022-08-12 07:30:11.9517556 +0000 UTC m=+12.908442081, 105.07926ms after stability + testing.go:149: cluster: 2022-08-12T07:30:11.951Z [INFO] stable state for Follower reached at 2022-08-12 07:30:11.84667636 +0000 UTC m=+12.803362821 (2 nodes), 2.5µs from start of poll, 168.45356ms from cluster start. Timeout at 2022-08-12 07:30:11.9517556 +0000 UTC m=+12.908442081, 105.07926ms after stability +testLoggerAdapter verbose: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:11.952Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: updating configuration: command=RemoveServer server-id=server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9 server-addr= servers="[{Suffrage:Voter ID:server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c Address:3a4fc9c3-4a54-4e7b-8e08-64676a68e33c} {Suffrage:Voter ID:server-a5759801-0e55-5f3d-850a-e88fd230beec Address:a5759801-0e55-5f3d-850a-e88fd230beec}]" + testing.go:149: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:11.952Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: updating configuration: command=RemoveServer server-id=server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9 server-addr= servers="[{Suffrage:Voter ID:server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c Address:3a4fc9c3-4a54-4e7b-8e08-64676a68e33c} {Suffrage:Voter ID:server-a5759801-0e55-5f3d-850a-e88fd230beec Address:a5759801-0e55-5f3d-850a-e88fd230beec}]" +testLoggerAdapter verbose: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:11.952Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: removed peer, stopping replication: peer=server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9 last-index=3 + testing.go:149: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:11.952Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: removed peer, stopping replication: peer=server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9 last-index=3 +testLoggerAdapter verbose: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:11.952Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: aborting pipeline replication: peer="{Voter server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9 4c138c72-aa59-d43c-1ea3-22169a2ec8a9}" + testing.go:149: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:11.952Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: aborting pipeline replication: peer="{Voter server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9 4c138c72-aa59-d43c-1ea3-22169a2ec8a9}" +testLoggerAdapter verbose: server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9: 2022-08-12T07:30:12.055Z [WARN] server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9: heartbeat timeout reached, starting election: last-leader=3a4fc9c3-4a54-4e7b-8e08-64676a68e33c + testing.go:149: server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9: 2022-08-12T07:30:12.055Z [WARN] server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9: heartbeat timeout reached, starting election: last-leader=3a4fc9c3-4a54-4e7b-8e08-64676a68e33c +testLoggerAdapter verbose: server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9: 2022-08-12T07:30:12.055Z [INFO] server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9: entering candidate state: node="Node at 4c138c72-aa59-d43c-1ea3-22169a2ec8a9 [Candidate]" term=3 + testing.go:149: server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9: 2022-08-12T07:30:12.055Z [INFO] server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9: entering candidate state: node="Node at 4c138c72-aa59-d43c-1ea3-22169a2ec8a9 [Candidate]" term=3 +testLoggerAdapter verbose: server-a5759801-0e55-5f3d-850a-e88fd230beec: 2022-08-12T07:30:12.055Z [WARN] server-a5759801-0e55-5f3d-850a-e88fd230beec: rejecting vote request since we have a leader: from=4c138c72-aa59-d43c-1ea3-22169a2ec8a9 leader=3a4fc9c3-4a54-4e7b-8e08-64676a68e33c +testLoggerAdapter verbose: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:12.055Z [WARN] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: rejecting vote request since we have a leader: from=4c138c72-aa59-d43c-1ea3-22169a2ec8a9 leader=3a4fc9c3-4a54-4e7b-8e08-64676a68e33c + testing.go:149: server-a5759801-0e55-5f3d-850a-e88fd230beec: 2022-08-12T07:30:12.055Z [WARN] server-a5759801-0e55-5f3d-850a-e88fd230beec: rejecting vote request since we have a leader: from=4c138c72-aa59-d43c-1ea3-22169a2ec8a9 leader=3a4fc9c3-4a54-4e7b-8e08-64676a68e33c + testing.go:149: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:12.055Z [WARN] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: rejecting vote request since we have a leader: from=4c138c72-aa59-d43c-1ea3-22169a2ec8a9 leader=3a4fc9c3-4a54-4e7b-8e08-64676a68e33c +testLoggerAdapter verbose: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:12.057Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: updating configuration: command=AddStaging server-id=server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9 server-addr=4c138c72-aa59-d43c-1ea3-22169a2ec8a9 servers="[{Suffrage:Voter ID:server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c Address:3a4fc9c3-4a54-4e7b-8e08-64676a68e33c} {Suffrage:Voter ID:server-a5759801-0e55-5f3d-850a-e88fd230beec Address:a5759801-0e55-5f3d-850a-e88fd230beec} {Suffrage:Voter ID:server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9 Address:4c138c72-aa59-d43c-1ea3-22169a2ec8a9}]" + testing.go:149: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:12.057Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: updating configuration: command=AddStaging server-id=server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9 server-addr=4c138c72-aa59-d43c-1ea3-22169a2ec8a9 servers="[{Suffrage:Voter ID:server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c Address:3a4fc9c3-4a54-4e7b-8e08-64676a68e33c} {Suffrage:Voter ID:server-a5759801-0e55-5f3d-850a-e88fd230beec Address:a5759801-0e55-5f3d-850a-e88fd230beec} {Suffrage:Voter ID:server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9 Address:4c138c72-aa59-d43c-1ea3-22169a2ec8a9}]" +testLoggerAdapter verbose: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:12.057Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: added peer, starting replication: peer=server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9 + testing.go:149: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:12.057Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: added peer, starting replication: peer=server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9 +testLoggerAdapter verbose: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:12.058Z [ERROR] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: peer has newer term, stopping replication: peer="{Voter server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9 4c138c72-aa59-d43c-1ea3-22169a2ec8a9}" + testing.go:149: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:12.058Z [ERROR] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: peer has newer term, stopping replication: peer="{Voter server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9 4c138c72-aa59-d43c-1ea3-22169a2ec8a9}" +testLoggerAdapter verbose: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:12.058Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: entering follower state: follower="Node at 3a4fc9c3-4a54-4e7b-8e08-64676a68e33c [Follower]" leader= + testing.go:149: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:12.058Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: entering follower state: follower="Node at 3a4fc9c3-4a54-4e7b-8e08-64676a68e33c [Follower]" leader= +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:12.058Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:12.058Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: aborting pipeline replication: peer="{Voter server-a5759801-0e55-5f3d-850a-e88fd230beec a5759801-0e55-5f3d-850a-e88fd230beec}" + testing.go:149: cluster: 2022-08-12T07:30:12.058Z [INFO] starting stability test: raft-state=Leader + testing.go:149: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:12.058Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: aborting pipeline replication: peer="{Voter server-a5759801-0e55-5f3d-850a-e88fd230beec a5759801-0e55-5f3d-850a-e88fd230beec}" +testLoggerAdapter verbose: server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9: 2022-08-12T07:30:12.106Z [WARN] server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9: Election timeout reached, restarting election + testing.go:149: server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9: 2022-08-12T07:30:12.106Z [WARN] server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9: Election timeout reached, restarting election +testLoggerAdapter verbose: server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9: 2022-08-12T07:30:12.106Z [INFO] server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9: entering candidate state: node="Node at 4c138c72-aa59-d43c-1ea3-22169a2ec8a9 [Candidate]" term=4 + testing.go:149: server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9: 2022-08-12T07:30:12.106Z [INFO] server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9: entering candidate state: node="Node at 4c138c72-aa59-d43c-1ea3-22169a2ec8a9 [Candidate]" term=4 +testLoggerAdapter verbose: server-a5759801-0e55-5f3d-850a-e88fd230beec: 2022-08-12T07:30:12.106Z [WARN] server-a5759801-0e55-5f3d-850a-e88fd230beec: rejecting vote request since we have a leader: from=4c138c72-aa59-d43c-1ea3-22169a2ec8a9 leader=3a4fc9c3-4a54-4e7b-8e08-64676a68e33c + testing.go:149: server-a5759801-0e55-5f3d-850a-e88fd230beec: 2022-08-12T07:30:12.106Z [WARN] server-a5759801-0e55-5f3d-850a-e88fd230beec: rejecting vote request since we have a leader: from=4c138c72-aa59-d43c-1ea3-22169a2ec8a9 leader=3a4fc9c3-4a54-4e7b-8e08-64676a68e33c +testLoggerAdapter verbose: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:12.106Z [WARN] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: rejecting vote request since our last index is greater: candidate=4c138c72-aa59-d43c-1ea3-22169a2ec8a9 last-index=4 last-candidate-index=3 + testing.go:149: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:12.106Z [WARN] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: rejecting vote request since our last index is greater: candidate=4c138c72-aa59-d43c-1ea3-22169a2ec8a9 last-index=4 last-candidate-index=3 +testLoggerAdapter verbose: server-a5759801-0e55-5f3d-850a-e88fd230beec: 2022-08-12T07:30:12.131Z [WARN] server-a5759801-0e55-5f3d-850a-e88fd230beec: heartbeat timeout reached, starting election: last-leader=3a4fc9c3-4a54-4e7b-8e08-64676a68e33c + testing.go:149: server-a5759801-0e55-5f3d-850a-e88fd230beec: 2022-08-12T07:30:12.131Z [WARN] server-a5759801-0e55-5f3d-850a-e88fd230beec: heartbeat timeout reached, starting election: last-leader=3a4fc9c3-4a54-4e7b-8e08-64676a68e33c +testLoggerAdapter verbose: server-a5759801-0e55-5f3d-850a-e88fd230beec: 2022-08-12T07:30:12.131Z [INFO] server-a5759801-0e55-5f3d-850a-e88fd230beec: entering candidate state: node="Node at a5759801-0e55-5f3d-850a-e88fd230beec [Candidate]" term=3 + testing.go:149: server-a5759801-0e55-5f3d-850a-e88fd230beec: 2022-08-12T07:30:12.131Z [INFO] server-a5759801-0e55-5f3d-850a-e88fd230beec: entering candidate state: node="Node at a5759801-0e55-5f3d-850a-e88fd230beec [Candidate]" term=3 +testLoggerAdapter verbose: server-a5759801-0e55-5f3d-850a-e88fd230beec: 2022-08-12T07:30:12.131Z [INFO] server-a5759801-0e55-5f3d-850a-e88fd230beec: entering follower state: follower="Node at a5759801-0e55-5f3d-850a-e88fd230beec [Follower]" leader= + testing.go:149: server-a5759801-0e55-5f3d-850a-e88fd230beec: 2022-08-12T07:30:12.131Z [INFO] server-a5759801-0e55-5f3d-850a-e88fd230beec: entering follower state: follower="Node at a5759801-0e55-5f3d-850a-e88fd230beec [Follower]" leader= +testLoggerAdapter verbose: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:12.138Z [WARN] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:12.138Z [WARN] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:12.138Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: entering candidate state: node="Node at 3a4fc9c3-4a54-4e7b-8e08-64676a68e33c [Candidate]" term=5 + testing.go:149: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:12.138Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: entering candidate state: node="Node at 3a4fc9c3-4a54-4e7b-8e08-64676a68e33c [Candidate]" term=5 +testLoggerAdapter verbose: server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9: 2022-08-12T07:30:12.138Z [INFO] server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9: entering follower state: follower="Node at 4c138c72-aa59-d43c-1ea3-22169a2ec8a9 [Follower]" leader= +testLoggerAdapter verbose: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:12.138Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: election won: tally=2 + testing.go:149: server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9: 2022-08-12T07:30:12.138Z [INFO] server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9: entering follower state: follower="Node at 4c138c72-aa59-d43c-1ea3-22169a2ec8a9 [Follower]" leader= + testing.go:149: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:12.138Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: election won: tally=2 +testLoggerAdapter verbose: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:12.138Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: entering leader state: leader="Node at 3a4fc9c3-4a54-4e7b-8e08-64676a68e33c [Leader]" + testing.go:149: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:12.138Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: entering leader state: leader="Node at 3a4fc9c3-4a54-4e7b-8e08-64676a68e33c [Leader]" +testLoggerAdapter verbose: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:12.139Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: added peer, starting replication: peer=server-a5759801-0e55-5f3d-850a-e88fd230beec + testing.go:149: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:12.139Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: added peer, starting replication: peer=server-a5759801-0e55-5f3d-850a-e88fd230beec +testLoggerAdapter verbose: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:12.139Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: added peer, starting replication: peer=server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9 + testing.go:149: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:12.139Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: added peer, starting replication: peer=server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9 +testLoggerAdapter verbose: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:12.139Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: pipelining replication: peer="{Voter server-a5759801-0e55-5f3d-850a-e88fd230beec a5759801-0e55-5f3d-850a-e88fd230beec}" +testLoggerAdapter verbose: server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9: 2022-08-12T07:30:12.139Z [WARN] server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9: failed to get previous log: previous-index=4 last-index=3 error="log not found" + testing.go:149: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:12.139Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: pipelining replication: peer="{Voter server-a5759801-0e55-5f3d-850a-e88fd230beec a5759801-0e55-5f3d-850a-e88fd230beec}" + testing.go:149: server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9: 2022-08-12T07:30:12.139Z [WARN] server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9: failed to get previous log: previous-index=4 last-index=3 error="log not found" +testLoggerAdapter verbose: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:12.139Z [WARN] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: appendEntries rejected, sending older logs: peer="{Voter server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9 4c138c72-aa59-d43c-1ea3-22169a2ec8a9}" next=4 + testing.go:149: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:12.139Z [WARN] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: appendEntries rejected, sending older logs: peer="{Voter server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9 4c138c72-aa59-d43c-1ea3-22169a2ec8a9}" next=4 +testLoggerAdapter verbose: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:12.139Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: pipelining replication: peer="{Voter server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9 4c138c72-aa59-d43c-1ea3-22169a2ec8a9}" + testing.go:149: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:12.139Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: pipelining replication: peer="{Voter server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9 4c138c72-aa59-d43c-1ea3-22169a2ec8a9}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:12.244Z [INFO] stable state for Leader reached at 2022-08-12 07:30:12.13907204 +0000 UTC m=+13.095758521 (1 nodes), 80.69382ms from start of poll, 460.84926ms from cluster start. Timeout at 2022-08-12 07:30:12.24414136 +0000 UTC m=+13.200827821, 105.0693ms after stability + testing.go:149: cluster: 2022-08-12T07:30:12.244Z [INFO] stable state for Leader reached at 2022-08-12 07:30:12.13907204 +0000 UTC m=+13.095758521 (1 nodes), 80.69382ms from start of poll, 460.84926ms from cluster start. Timeout at 2022-08-12 07:30:12.24414136 +0000 UTC m=+13.200827821, 105.0693ms after stability +testLoggerAdapter verbose: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:12.244Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: aborting pipeline replication: peer="{Voter server-a5759801-0e55-5f3d-850a-e88fd230beec a5759801-0e55-5f3d-850a-e88fd230beec}" + testing.go:149: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:12.244Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: aborting pipeline replication: peer="{Voter server-a5759801-0e55-5f3d-850a-e88fd230beec a5759801-0e55-5f3d-850a-e88fd230beec}" +testLoggerAdapter verbose: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:12.244Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: aborting pipeline replication: peer="{Voter server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9 4c138c72-aa59-d43c-1ea3-22169a2ec8a9}" + testing.go:149: server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: 2022-08-12T07:30:12.244Z [INFO] server-3a4fc9c3-4a54-4e7b-8e08-64676a68e33c: aborting pipeline replication: peer="{Voter server-4c138c72-aa59-d43c-1ea3-22169a2ec8a9 4c138c72-aa59-d43c-1ea3-22169a2ec8a9}" +--- PASS: TestRaft_ReJoinFollower (0.57s) === RUN TestRaft_LeaderLeaseExpire -testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:07.724Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7f0943e4-17b9-a000-0b75-5cba262cdcb7 Address:7f0943e4-17b9-a000-0b75-5cba262cdcb7} {Suffrage:Voter ID:server-edf8725a-05c2-da08-91e3-f04f36db93c4 Address:edf8725a-05c2-da08-91e3-f04f36db93c4}]" - testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:07.724Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7f0943e4-17b9-a000-0b75-5cba262cdcb7 Address:7f0943e4-17b9-a000-0b75-5cba262cdcb7} {Suffrage:Voter ID:server-edf8725a-05c2-da08-91e3-f04f36db93c4 Address:edf8725a-05c2-da08-91e3-f04f36db93c4}]" -testLoggerAdapter verbose: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:07.724Z [INFO] server-edf8725a-05c2-da08-91e3-f04f36db93c4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7f0943e4-17b9-a000-0b75-5cba262cdcb7 Address:7f0943e4-17b9-a000-0b75-5cba262cdcb7} {Suffrage:Voter ID:server-edf8725a-05c2-da08-91e3-f04f36db93c4 Address:edf8725a-05c2-da08-91e3-f04f36db93c4}]" - testing.go:149: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:07.724Z [INFO] server-edf8725a-05c2-da08-91e3-f04f36db93c4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7f0943e4-17b9-a000-0b75-5cba262cdcb7 Address:7f0943e4-17b9-a000-0b75-5cba262cdcb7} {Suffrage:Voter ID:server-edf8725a-05c2-da08-91e3-f04f36db93c4 Address:edf8725a-05c2-da08-91e3-f04f36db93c4}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:07.724Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:07.724Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:07.725Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: entering follower state: follower="Node at 7f0943e4-17b9-a000-0b75-5cba262cdcb7 [Follower]" leader= - testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:07.725Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: entering follower state: follower="Node at 7f0943e4-17b9-a000-0b75-5cba262cdcb7 [Follower]" leader= -testLoggerAdapter verbose: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:07.725Z [INFO] server-edf8725a-05c2-da08-91e3-f04f36db93c4: entering follower state: follower="Node at edf8725a-05c2-da08-91e3-f04f36db93c4 [Follower]" leader= - testing.go:149: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:07.725Z [INFO] server-edf8725a-05c2-da08-91e3-f04f36db93c4: entering follower state: follower="Node at edf8725a-05c2-da08-91e3-f04f36db93c4 [Follower]" leader= -testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:07.803Z [WARN] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:07.803Z [WARN] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:07.803Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: entering candidate state: node="Node at 7f0943e4-17b9-a000-0b75-5cba262cdcb7 [Candidate]" term=2 - testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:07.803Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: entering candidate state: node="Node at 7f0943e4-17b9-a000-0b75-5cba262cdcb7 [Candidate]" term=2 -testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:07.803Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: election won: tally=2 - testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:07.803Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: election won: tally=2 -testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:07.804Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: entering leader state: leader="Node at 7f0943e4-17b9-a000-0b75-5cba262cdcb7 [Leader]" - testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:07.804Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: entering leader state: leader="Node at 7f0943e4-17b9-a000-0b75-5cba262cdcb7 [Leader]" -testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:07.804Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: added peer, starting replication: peer=server-edf8725a-05c2-da08-91e3-f04f36db93c4 - testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:07.804Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: added peer, starting replication: peer=server-edf8725a-05c2-da08-91e3-f04f36db93c4 -testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:07.804Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: pipelining replication: peer="{Voter server-edf8725a-05c2-da08-91e3-f04f36db93c4 edf8725a-05c2-da08-91e3-f04f36db93c4}" - testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:07.804Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: pipelining replication: peer="{Voter server-edf8725a-05c2-da08-91e3-f04f36db93c4 edf8725a-05c2-da08-91e3-f04f36db93c4}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:07.909Z [INFO] stable state for Leader reached at 2021-07-10 01:05:07.804114068 +0000 UTC m=+13.192272038 (1 nodes), 79.088494ms from start of poll, 79.691761ms from cluster start. Timeout at 2021-07-10 01:05:07.909205636 +0000 UTC m=+13.297363606, 105.091568ms after stability - testing.go:149: cluster: 2021-07-10T01:05:07.909Z [INFO] stable state for Leader reached at 2021-07-10 01:05:07.804114068 +0000 UTC m=+13.192272038 (1 nodes), 79.088494ms from start of poll, 79.691761ms from cluster start. Timeout at 2021-07-10 01:05:07.909205636 +0000 UTC m=+13.297363606, 105.091568ms after stability -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:07.914Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-07-10T01:05:07.914Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:08.019Z [INFO] stable state for Follower reached at 2021-07-10 01:05:07.914683793 +0000 UTC m=+13.302841763 (1 nodes), 2.32µs from start of poll, 190.261486ms from cluster start. Timeout at 2021-07-10 01:05:08.019766581 +0000 UTC m=+13.407924551, 105.082788ms after stability - testing.go:149: cluster: 2021-07-10T01:05:08.019Z [INFO] stable state for Follower reached at 2021-07-10 01:05:07.914683793 +0000 UTC m=+13.302841763 (1 nodes), 2.32µs from start of poll, 190.261486ms from cluster start. Timeout at 2021-07-10 01:05:08.019766581 +0000 UTC m=+13.407924551, 105.082788ms after stability - raft_test.go:1492: [INFO] Disconnecting Node at edf8725a-05c2-da08-91e3-f04f36db93c4 [Follower] -testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.023Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to heartbeat to: peer=edf8725a-05c2-da08-91e3-f04f36db93c4 error="failed to connect to peer: edf8725a-05c2-da08-91e3-f04f36db93c4" - testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.023Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to heartbeat to: peer=edf8725a-05c2-da08-91e3-f04f36db93c4 error="failed to connect to peer: edf8725a-05c2-da08-91e3-f04f36db93c4" -testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.025Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to pipeline appendEntries: peer="{Voter server-edf8725a-05c2-da08-91e3-f04f36db93c4 edf8725a-05c2-da08-91e3-f04f36db93c4}" error="append pipeline closed" - testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.025Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to pipeline appendEntries: peer="{Voter server-edf8725a-05c2-da08-91e3-f04f36db93c4 edf8725a-05c2-da08-91e3-f04f36db93c4}" error="append pipeline closed" -testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.025Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: aborting pipeline replication: peer="{Voter server-edf8725a-05c2-da08-91e3-f04f36db93c4 edf8725a-05c2-da08-91e3-f04f36db93c4}" - testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.025Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: aborting pipeline replication: peer="{Voter server-edf8725a-05c2-da08-91e3-f04f36db93c4 edf8725a-05c2-da08-91e3-f04f36db93c4}" -testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.033Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to appendEntries to: peer="{Voter server-edf8725a-05c2-da08-91e3-f04f36db93c4 edf8725a-05c2-da08-91e3-f04f36db93c4}" error="failed to connect to peer: edf8725a-05c2-da08-91e3-f04f36db93c4" - testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.033Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to appendEntries to: peer="{Voter server-edf8725a-05c2-da08-91e3-f04f36db93c4 edf8725a-05c2-da08-91e3-f04f36db93c4}" error="failed to connect to peer: edf8725a-05c2-da08-91e3-f04f36db93c4" -testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.038Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to heartbeat to: peer=edf8725a-05c2-da08-91e3-f04f36db93c4 error="failed to connect to peer: edf8725a-05c2-da08-91e3-f04f36db93c4" - testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.038Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to heartbeat to: peer=edf8725a-05c2-da08-91e3-f04f36db93c4 error="failed to connect to peer: edf8725a-05c2-da08-91e3-f04f36db93c4" -testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.051Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to appendEntries to: peer="{Voter server-edf8725a-05c2-da08-91e3-f04f36db93c4 edf8725a-05c2-da08-91e3-f04f36db93c4}" error="failed to connect to peer: edf8725a-05c2-da08-91e3-f04f36db93c4" - testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.051Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to appendEntries to: peer="{Voter server-edf8725a-05c2-da08-91e3-f04f36db93c4 edf8725a-05c2-da08-91e3-f04f36db93c4}" error="failed to connect to peer: edf8725a-05c2-da08-91e3-f04f36db93c4" -testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.056Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to heartbeat to: peer=edf8725a-05c2-da08-91e3-f04f36db93c4 error="failed to connect to peer: edf8725a-05c2-da08-91e3-f04f36db93c4" - testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.056Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to heartbeat to: peer=edf8725a-05c2-da08-91e3-f04f36db93c4 error="failed to connect to peer: edf8725a-05c2-da08-91e3-f04f36db93c4" -testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.066Z [WARN] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to contact: server-id=server-edf8725a-05c2-da08-91e3-f04f36db93c4 time=50.059728ms - testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.066Z [WARN] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to contact: server-id=server-edf8725a-05c2-da08-91e3-f04f36db93c4 time=50.059728ms -testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.066Z [WARN] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to contact quorum of nodes, stepping down - testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.066Z [WARN] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to contact quorum of nodes, stepping down -testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.066Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: entering follower state: follower="Node at 7f0943e4-17b9-a000-0b75-5cba262cdcb7 [Follower]" leader= - testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.066Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: entering follower state: follower="Node at 7f0943e4-17b9-a000-0b75-5cba262cdcb7 [Follower]" leader= -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:08.066Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:08.066Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.071Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to appendEntries to: peer="{Voter server-edf8725a-05c2-da08-91e3-f04f36db93c4 edf8725a-05c2-da08-91e3-f04f36db93c4}" error="failed to connect to peer: edf8725a-05c2-da08-91e3-f04f36db93c4" - testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.071Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to appendEntries to: peer="{Voter server-edf8725a-05c2-da08-91e3-f04f36db93c4 edf8725a-05c2-da08-91e3-f04f36db93c4}" error="failed to connect to peer: edf8725a-05c2-da08-91e3-f04f36db93c4" -testLoggerAdapter verbose: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:08.109Z [WARN] server-edf8725a-05c2-da08-91e3-f04f36db93c4: heartbeat timeout reached, starting election: last-leader=7f0943e4-17b9-a000-0b75-5cba262cdcb7 - testing.go:149: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:08.109Z [WARN] server-edf8725a-05c2-da08-91e3-f04f36db93c4: heartbeat timeout reached, starting election: last-leader=7f0943e4-17b9-a000-0b75-5cba262cdcb7 -testLoggerAdapter verbose: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:08.109Z [INFO] server-edf8725a-05c2-da08-91e3-f04f36db93c4: entering candidate state: node="Node at edf8725a-05c2-da08-91e3-f04f36db93c4 [Candidate]" term=3 - testing.go:149: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:08.109Z [INFO] server-edf8725a-05c2-da08-91e3-f04f36db93c4: entering candidate state: node="Node at edf8725a-05c2-da08-91e3-f04f36db93c4 [Candidate]" term=3 -testLoggerAdapter verbose: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:08.109Z [ERROR] server-edf8725a-05c2-da08-91e3-f04f36db93c4: failed to make requestVote RPC: target="{Voter server-7f0943e4-17b9-a000-0b75-5cba262cdcb7 7f0943e4-17b9-a000-0b75-5cba262cdcb7}" error="failed to connect to peer: 7f0943e4-17b9-a000-0b75-5cba262cdcb7" - testing.go:149: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:08.109Z [ERROR] server-edf8725a-05c2-da08-91e3-f04f36db93c4: failed to make requestVote RPC: target="{Voter server-7f0943e4-17b9-a000-0b75-5cba262cdcb7 7f0943e4-17b9-a000-0b75-5cba262cdcb7}" error="failed to connect to peer: 7f0943e4-17b9-a000-0b75-5cba262cdcb7" -testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.159Z [WARN] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.159Z [WARN] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.159Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: entering candidate state: node="Node at 7f0943e4-17b9-a000-0b75-5cba262cdcb7 [Candidate]" term=3 - testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.159Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: entering candidate state: node="Node at 7f0943e4-17b9-a000-0b75-5cba262cdcb7 [Candidate]" term=3 -testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.159Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to make requestVote RPC: target="{Voter server-edf8725a-05c2-da08-91e3-f04f36db93c4 edf8725a-05c2-da08-91e3-f04f36db93c4}" error="failed to connect to peer: edf8725a-05c2-da08-91e3-f04f36db93c4" - testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.159Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to make requestVote RPC: target="{Voter server-edf8725a-05c2-da08-91e3-f04f36db93c4 edf8725a-05c2-da08-91e3-f04f36db93c4}" error="failed to connect to peer: edf8725a-05c2-da08-91e3-f04f36db93c4" -testLoggerAdapter verbose: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:08.166Z [WARN] server-edf8725a-05c2-da08-91e3-f04f36db93c4: Election timeout reached, restarting election - testing.go:149: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:08.166Z [WARN] server-edf8725a-05c2-da08-91e3-f04f36db93c4: Election timeout reached, restarting election -testLoggerAdapter verbose: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:08.166Z [INFO] server-edf8725a-05c2-da08-91e3-f04f36db93c4: entering candidate state: node="Node at edf8725a-05c2-da08-91e3-f04f36db93c4 [Candidate]" term=4 - testing.go:149: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:08.166Z [INFO] server-edf8725a-05c2-da08-91e3-f04f36db93c4: entering candidate state: node="Node at edf8725a-05c2-da08-91e3-f04f36db93c4 [Candidate]" term=4 -testLoggerAdapter verbose: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:08.166Z [ERROR] server-edf8725a-05c2-da08-91e3-f04f36db93c4: failed to make requestVote RPC: target="{Voter server-7f0943e4-17b9-a000-0b75-5cba262cdcb7 7f0943e4-17b9-a000-0b75-5cba262cdcb7}" error="failed to connect to peer: 7f0943e4-17b9-a000-0b75-5cba262cdcb7" - testing.go:149: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:08.166Z [ERROR] server-edf8725a-05c2-da08-91e3-f04f36db93c4: failed to make requestVote RPC: target="{Voter server-7f0943e4-17b9-a000-0b75-5cba262cdcb7 7f0943e4-17b9-a000-0b75-5cba262cdcb7}" error="failed to connect to peer: 7f0943e4-17b9-a000-0b75-5cba262cdcb7" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:08.215Z [INFO] stable state for Leader reached at 2021-07-10 01:05:08.109883391 +0000 UTC m=+13.498041381 (0 nodes), 43.147035ms from start of poll, 385.461104ms from cluster start. Timeout at 2021-07-10 01:05:08.214978619 +0000 UTC m=+13.603136589, 105.095208ms after stability - testing.go:149: cluster: 2021-07-10T01:05:08.215Z [INFO] stable state for Leader reached at 2021-07-10 01:05:08.109883391 +0000 UTC m=+13.498041381 (0 nodes), 43.147035ms from start of poll, 385.461104ms from cluster start. Timeout at 2021-07-10 01:05:08.214978619 +0000 UTC m=+13.603136589, 105.095208ms after stability -testLoggerAdapter verbose: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:08.244Z [WARN] server-edf8725a-05c2-da08-91e3-f04f36db93c4: Election timeout reached, restarting election - testing.go:149: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:08.244Z [WARN] server-edf8725a-05c2-da08-91e3-f04f36db93c4: Election timeout reached, restarting election -testLoggerAdapter verbose: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:08.244Z [INFO] server-edf8725a-05c2-da08-91e3-f04f36db93c4: entering candidate state: node="Node at edf8725a-05c2-da08-91e3-f04f36db93c4 [Candidate]" term=5 - testing.go:149: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:08.244Z [INFO] server-edf8725a-05c2-da08-91e3-f04f36db93c4: entering candidate state: node="Node at edf8725a-05c2-da08-91e3-f04f36db93c4 [Candidate]" term=5 -testLoggerAdapter verbose: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:08.245Z [ERROR] server-edf8725a-05c2-da08-91e3-f04f36db93c4: failed to make requestVote RPC: target="{Voter server-7f0943e4-17b9-a000-0b75-5cba262cdcb7 7f0943e4-17b9-a000-0b75-5cba262cdcb7}" error="failed to connect to peer: 7f0943e4-17b9-a000-0b75-5cba262cdcb7" - testing.go:149: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:08.245Z [ERROR] server-edf8725a-05c2-da08-91e3-f04f36db93c4: failed to make requestVote RPC: target="{Voter server-7f0943e4-17b9-a000-0b75-5cba262cdcb7 7f0943e4-17b9-a000-0b75-5cba262cdcb7}" error="failed to connect to peer: 7f0943e4-17b9-a000-0b75-5cba262cdcb7" -testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.251Z [WARN] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: Election timeout reached, restarting election - testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.251Z [WARN] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: Election timeout reached, restarting election -testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.251Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: entering candidate state: node="Node at 7f0943e4-17b9-a000-0b75-5cba262cdcb7 [Candidate]" term=4 - testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.251Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: entering candidate state: node="Node at 7f0943e4-17b9-a000-0b75-5cba262cdcb7 [Candidate]" term=4 -testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.251Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to make requestVote RPC: target="{Voter server-edf8725a-05c2-da08-91e3-f04f36db93c4 edf8725a-05c2-da08-91e3-f04f36db93c4}" error="failed to connect to peer: edf8725a-05c2-da08-91e3-f04f36db93c4" - testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.251Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to make requestVote RPC: target="{Voter server-edf8725a-05c2-da08-91e3-f04f36db93c4 edf8725a-05c2-da08-91e3-f04f36db93c4}" error="failed to connect to peer: edf8725a-05c2-da08-91e3-f04f36db93c4" -testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.313Z [WARN] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: Election timeout reached, restarting election - testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.313Z [WARN] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: Election timeout reached, restarting election -testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.314Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: entering candidate state: node="Node at 7f0943e4-17b9-a000-0b75-5cba262cdcb7 [Candidate]" term=5 - testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.314Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: entering candidate state: node="Node at 7f0943e4-17b9-a000-0b75-5cba262cdcb7 [Candidate]" term=5 -testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.314Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to make requestVote RPC: target="{Voter server-edf8725a-05c2-da08-91e3-f04f36db93c4 edf8725a-05c2-da08-91e3-f04f36db93c4}" error="failed to connect to peer: edf8725a-05c2-da08-91e3-f04f36db93c4" - testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.314Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to make requestVote RPC: target="{Voter server-edf8725a-05c2-da08-91e3-f04f36db93c4 edf8725a-05c2-da08-91e3-f04f36db93c4}" error="failed to connect to peer: edf8725a-05c2-da08-91e3-f04f36db93c4" ---- PASS: TestRaft_LeaderLeaseExpire (0.60s) +testLoggerAdapter verbose: server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: 2022-08-12T07:30:12.245Z [INFO] server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f Address:6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f} {Suffrage:Voter ID:server-d43512ec-dec4-a21a-fe90-da58d090a003 Address:d43512ec-dec4-a21a-fe90-da58d090a003}]" + testing.go:149: server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: 2022-08-12T07:30:12.245Z [INFO] server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f Address:6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f} {Suffrage:Voter ID:server-d43512ec-dec4-a21a-fe90-da58d090a003 Address:d43512ec-dec4-a21a-fe90-da58d090a003}]" +testLoggerAdapter verbose: server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: 2022-08-12T07:30:12.246Z [INFO] server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: entering follower state: follower="Node at 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f [Follower]" leader= + testing.go:149: server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: 2022-08-12T07:30:12.246Z [INFO] server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: entering follower state: follower="Node at 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f [Follower]" leader= +testLoggerAdapter verbose: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.246Z [INFO] server-d43512ec-dec4-a21a-fe90-da58d090a003: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f Address:6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f} {Suffrage:Voter ID:server-d43512ec-dec4-a21a-fe90-da58d090a003 Address:d43512ec-dec4-a21a-fe90-da58d090a003}]" + testing.go:149: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.246Z [INFO] server-d43512ec-dec4-a21a-fe90-da58d090a003: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f Address:6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f} {Suffrage:Voter ID:server-d43512ec-dec4-a21a-fe90-da58d090a003 Address:d43512ec-dec4-a21a-fe90-da58d090a003}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:12.246Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:12.246Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.246Z [INFO] server-d43512ec-dec4-a21a-fe90-da58d090a003: entering follower state: follower="Node at d43512ec-dec4-a21a-fe90-da58d090a003 [Follower]" leader= + testing.go:149: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.246Z [INFO] server-d43512ec-dec4-a21a-fe90-da58d090a003: entering follower state: follower="Node at d43512ec-dec4-a21a-fe90-da58d090a003 [Follower]" leader= +testLoggerAdapter verbose: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.334Z [WARN] server-d43512ec-dec4-a21a-fe90-da58d090a003: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.334Z [WARN] server-d43512ec-dec4-a21a-fe90-da58d090a003: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.334Z [INFO] server-d43512ec-dec4-a21a-fe90-da58d090a003: entering candidate state: node="Node at d43512ec-dec4-a21a-fe90-da58d090a003 [Candidate]" term=2 + testing.go:149: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.334Z [INFO] server-d43512ec-dec4-a21a-fe90-da58d090a003: entering candidate state: node="Node at d43512ec-dec4-a21a-fe90-da58d090a003 [Candidate]" term=2 +testLoggerAdapter verbose: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.334Z [INFO] server-d43512ec-dec4-a21a-fe90-da58d090a003: election won: tally=2 + testing.go:149: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.334Z [INFO] server-d43512ec-dec4-a21a-fe90-da58d090a003: election won: tally=2 +testLoggerAdapter verbose: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.335Z [INFO] server-d43512ec-dec4-a21a-fe90-da58d090a003: entering leader state: leader="Node at d43512ec-dec4-a21a-fe90-da58d090a003 [Leader]" + testing.go:149: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.335Z [INFO] server-d43512ec-dec4-a21a-fe90-da58d090a003: entering leader state: leader="Node at d43512ec-dec4-a21a-fe90-da58d090a003 [Leader]" +testLoggerAdapter verbose: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.335Z [INFO] server-d43512ec-dec4-a21a-fe90-da58d090a003: added peer, starting replication: peer=server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f + testing.go:149: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.335Z [INFO] server-d43512ec-dec4-a21a-fe90-da58d090a003: added peer, starting replication: peer=server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f +testLoggerAdapter verbose: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.335Z [INFO] server-d43512ec-dec4-a21a-fe90-da58d090a003: pipelining replication: peer="{Voter server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f}" + testing.go:149: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.335Z [INFO] server-d43512ec-dec4-a21a-fe90-da58d090a003: pipelining replication: peer="{Voter server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:12.440Z [INFO] stable state for Leader reached at 2022-08-12 07:30:12.33516844 +0000 UTC m=+13.291854921 (1 nodes), 88.7493ms from start of poll, 89.47ms from cluster start. Timeout at 2022-08-12 07:30:12.44023848 +0000 UTC m=+13.396924961, 105.07004ms after stability + testing.go:149: cluster: 2022-08-12T07:30:12.440Z [INFO] stable state for Leader reached at 2022-08-12 07:30:12.33516844 +0000 UTC m=+13.291854921 (1 nodes), 88.7493ms from start of poll, 89.47ms from cluster start. Timeout at 2022-08-12 07:30:12.44023848 +0000 UTC m=+13.396924961, 105.07004ms after stability +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:12.445Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2022-08-12T07:30:12.445Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:12.550Z [INFO] stable state for Follower reached at 2022-08-12 07:30:12.44569702 +0000 UTC m=+13.402383521 (1 nodes), 2.58µs from start of poll, 199.9986ms from cluster start. Timeout at 2022-08-12 07:30:12.55077478 +0000 UTC m=+13.507461261, 105.07774ms after stability + testing.go:149: cluster: 2022-08-12T07:30:12.550Z [INFO] stable state for Follower reached at 2022-08-12 07:30:12.44569702 +0000 UTC m=+13.402383521 (1 nodes), 2.58µs from start of poll, 199.9986ms from cluster start. Timeout at 2022-08-12 07:30:12.55077478 +0000 UTC m=+13.507461261, 105.07774ms after stability + raft_test.go:1492: [INFO] Disconnecting Node at 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f [Follower] +testLoggerAdapter verbose: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.551Z [ERROR] server-d43512ec-dec4-a21a-fe90-da58d090a003: failed to pipeline appendEntries: peer="{Voter server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f}" error="append pipeline closed" + testing.go:149: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.551Z [ERROR] server-d43512ec-dec4-a21a-fe90-da58d090a003: failed to pipeline appendEntries: peer="{Voter server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f}" error="append pipeline closed" +testLoggerAdapter verbose: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.551Z [INFO] server-d43512ec-dec4-a21a-fe90-da58d090a003: aborting pipeline replication: peer="{Voter server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f}" + testing.go:149: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.551Z [INFO] server-d43512ec-dec4-a21a-fe90-da58d090a003: aborting pipeline replication: peer="{Voter server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f}" +testLoggerAdapter verbose: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.551Z [ERROR] server-d43512ec-dec4-a21a-fe90-da58d090a003: failed to heartbeat to: peer=6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f error="failed to connect to peer: 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f" + testing.go:149: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.551Z [ERROR] server-d43512ec-dec4-a21a-fe90-da58d090a003: failed to heartbeat to: peer=6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f error="failed to connect to peer: 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f" +testLoggerAdapter verbose: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.560Z [ERROR] server-d43512ec-dec4-a21a-fe90-da58d090a003: failed to appendEntries to: peer="{Voter server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f}" error="failed to connect to peer: 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f" + testing.go:149: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.560Z [ERROR] server-d43512ec-dec4-a21a-fe90-da58d090a003: failed to appendEntries to: peer="{Voter server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f}" error="failed to connect to peer: 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f" +testLoggerAdapter verbose: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.570Z [ERROR] server-d43512ec-dec4-a21a-fe90-da58d090a003: failed to heartbeat to: peer=6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f error="failed to connect to peer: 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f" + testing.go:149: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.570Z [ERROR] server-d43512ec-dec4-a21a-fe90-da58d090a003: failed to heartbeat to: peer=6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f error="failed to connect to peer: 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f" +testLoggerAdapter verbose: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.579Z [ERROR] server-d43512ec-dec4-a21a-fe90-da58d090a003: failed to appendEntries to: peer="{Voter server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f}" error="failed to connect to peer: 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f" + testing.go:149: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.579Z [ERROR] server-d43512ec-dec4-a21a-fe90-da58d090a003: failed to appendEntries to: peer="{Voter server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f}" error="failed to connect to peer: 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f" +testLoggerAdapter verbose: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.590Z [ERROR] server-d43512ec-dec4-a21a-fe90-da58d090a003: failed to heartbeat to: peer=6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f error="failed to connect to peer: 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f" + testing.go:149: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.590Z [ERROR] server-d43512ec-dec4-a21a-fe90-da58d090a003: failed to heartbeat to: peer=6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f error="failed to connect to peer: 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f" +testLoggerAdapter verbose: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.596Z [ERROR] server-d43512ec-dec4-a21a-fe90-da58d090a003: failed to appendEntries to: peer="{Voter server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f}" error="failed to connect to peer: 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f" + testing.go:149: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.596Z [ERROR] server-d43512ec-dec4-a21a-fe90-da58d090a003: failed to appendEntries to: peer="{Voter server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f}" error="failed to connect to peer: 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f" +testLoggerAdapter verbose: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.596Z [WARN] server-d43512ec-dec4-a21a-fe90-da58d090a003: failed to contact: server-id=server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f time=50.0557ms + testing.go:149: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.596Z [WARN] server-d43512ec-dec4-a21a-fe90-da58d090a003: failed to contact: server-id=server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f time=50.0557ms +testLoggerAdapter verbose: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.596Z [WARN] server-d43512ec-dec4-a21a-fe90-da58d090a003: failed to contact quorum of nodes, stepping down + testing.go:149: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.596Z [WARN] server-d43512ec-dec4-a21a-fe90-da58d090a003: failed to contact quorum of nodes, stepping down +testLoggerAdapter verbose: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.596Z [INFO] server-d43512ec-dec4-a21a-fe90-da58d090a003: entering follower state: follower="Node at d43512ec-dec4-a21a-fe90-da58d090a003 [Follower]" leader= + testing.go:149: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.596Z [INFO] server-d43512ec-dec4-a21a-fe90-da58d090a003: entering follower state: follower="Node at d43512ec-dec4-a21a-fe90-da58d090a003 [Follower]" leader= +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:12.596Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:12.596Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: 2022-08-12T07:30:12.653Z [WARN] server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: heartbeat timeout reached, starting election: last-leader=d43512ec-dec4-a21a-fe90-da58d090a003 + testing.go:149: server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: 2022-08-12T07:30:12.653Z [WARN] server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: heartbeat timeout reached, starting election: last-leader=d43512ec-dec4-a21a-fe90-da58d090a003 +testLoggerAdapter verbose: server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: 2022-08-12T07:30:12.653Z [INFO] server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: entering candidate state: node="Node at 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f [Candidate]" term=3 + testing.go:149: server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: 2022-08-12T07:30:12.653Z [INFO] server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: entering candidate state: node="Node at 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f [Candidate]" term=3 +testLoggerAdapter verbose: server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: 2022-08-12T07:30:12.653Z [ERROR] server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: failed to make requestVote RPC: target="{Voter server-d43512ec-dec4-a21a-fe90-da58d090a003 d43512ec-dec4-a21a-fe90-da58d090a003}" error="failed to connect to peer: d43512ec-dec4-a21a-fe90-da58d090a003" + testing.go:149: server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: 2022-08-12T07:30:12.653Z [ERROR] server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: failed to make requestVote RPC: target="{Voter server-d43512ec-dec4-a21a-fe90-da58d090a003 d43512ec-dec4-a21a-fe90-da58d090a003}" error="failed to connect to peer: d43512ec-dec4-a21a-fe90-da58d090a003" +testLoggerAdapter verbose: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.668Z [WARN] server-d43512ec-dec4-a21a-fe90-da58d090a003: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.668Z [WARN] server-d43512ec-dec4-a21a-fe90-da58d090a003: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.669Z [INFO] server-d43512ec-dec4-a21a-fe90-da58d090a003: entering candidate state: node="Node at d43512ec-dec4-a21a-fe90-da58d090a003 [Candidate]" term=3 + testing.go:149: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.669Z [INFO] server-d43512ec-dec4-a21a-fe90-da58d090a003: entering candidate state: node="Node at d43512ec-dec4-a21a-fe90-da58d090a003 [Candidate]" term=3 +testLoggerAdapter verbose: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.669Z [ERROR] server-d43512ec-dec4-a21a-fe90-da58d090a003: failed to make requestVote RPC: target="{Voter server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f}" error="failed to connect to peer: 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f" + testing.go:149: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.669Z [ERROR] server-d43512ec-dec4-a21a-fe90-da58d090a003: failed to make requestVote RPC: target="{Voter server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f}" error="failed to connect to peer: 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f" +testLoggerAdapter verbose: server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: 2022-08-12T07:30:12.729Z [WARN] server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: Election timeout reached, restarting election + testing.go:149: server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: 2022-08-12T07:30:12.729Z [WARN] server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: Election timeout reached, restarting election +testLoggerAdapter verbose: server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: 2022-08-12T07:30:12.730Z [INFO] server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: entering candidate state: node="Node at 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f [Candidate]" term=4 + testing.go:149: server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: 2022-08-12T07:30:12.730Z [INFO] server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: entering candidate state: node="Node at 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f [Candidate]" term=4 +testLoggerAdapter verbose: server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: 2022-08-12T07:30:12.730Z [ERROR] server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: failed to make requestVote RPC: target="{Voter server-d43512ec-dec4-a21a-fe90-da58d090a003 d43512ec-dec4-a21a-fe90-da58d090a003}" error="failed to connect to peer: d43512ec-dec4-a21a-fe90-da58d090a003" + testing.go:149: server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: 2022-08-12T07:30:12.730Z [ERROR] server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: failed to make requestVote RPC: target="{Voter server-d43512ec-dec4-a21a-fe90-da58d090a003 d43512ec-dec4-a21a-fe90-da58d090a003}" error="failed to connect to peer: d43512ec-dec4-a21a-fe90-da58d090a003" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:12.759Z [INFO] stable state for Leader reached at 2022-08-12 07:30:12.6538319 +0000 UTC m=+13.610518401 (0 nodes), 56.82678ms from start of poll, 408.13348ms from cluster start. Timeout at 2022-08-12 07:30:12.75892522 +0000 UTC m=+13.715611721, 105.09332ms after stability + testing.go:149: cluster: 2022-08-12T07:30:12.759Z [INFO] stable state for Leader reached at 2022-08-12 07:30:12.6538319 +0000 UTC m=+13.610518401 (0 nodes), 56.82678ms from start of poll, 408.13348ms from cluster start. Timeout at 2022-08-12 07:30:12.75892522 +0000 UTC m=+13.715611721, 105.09332ms after stability +testLoggerAdapter verbose: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.768Z [WARN] server-d43512ec-dec4-a21a-fe90-da58d090a003: Election timeout reached, restarting election + testing.go:149: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.768Z [WARN] server-d43512ec-dec4-a21a-fe90-da58d090a003: Election timeout reached, restarting election +testLoggerAdapter verbose: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.768Z [INFO] server-d43512ec-dec4-a21a-fe90-da58d090a003: entering candidate state: node="Node at d43512ec-dec4-a21a-fe90-da58d090a003 [Candidate]" term=4 + testing.go:149: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.768Z [INFO] server-d43512ec-dec4-a21a-fe90-da58d090a003: entering candidate state: node="Node at d43512ec-dec4-a21a-fe90-da58d090a003 [Candidate]" term=4 +testLoggerAdapter verbose: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.768Z [ERROR] server-d43512ec-dec4-a21a-fe90-da58d090a003: failed to make requestVote RPC: target="{Voter server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f}" error="failed to connect to peer: 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f" + testing.go:149: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.768Z [ERROR] server-d43512ec-dec4-a21a-fe90-da58d090a003: failed to make requestVote RPC: target="{Voter server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f}" error="failed to connect to peer: 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f" +testLoggerAdapter verbose: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.820Z [WARN] server-d43512ec-dec4-a21a-fe90-da58d090a003: Election timeout reached, restarting election + testing.go:149: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.820Z [WARN] server-d43512ec-dec4-a21a-fe90-da58d090a003: Election timeout reached, restarting election +testLoggerAdapter verbose: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.820Z [INFO] server-d43512ec-dec4-a21a-fe90-da58d090a003: entering candidate state: node="Node at d43512ec-dec4-a21a-fe90-da58d090a003 [Candidate]" term=5 + testing.go:149: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.820Z [INFO] server-d43512ec-dec4-a21a-fe90-da58d090a003: entering candidate state: node="Node at d43512ec-dec4-a21a-fe90-da58d090a003 [Candidate]" term=5 +testLoggerAdapter verbose: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.821Z [ERROR] server-d43512ec-dec4-a21a-fe90-da58d090a003: failed to make requestVote RPC: target="{Voter server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f}" error="failed to connect to peer: 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f" + testing.go:149: server-d43512ec-dec4-a21a-fe90-da58d090a003: 2022-08-12T07:30:12.821Z [ERROR] server-d43512ec-dec4-a21a-fe90-da58d090a003: failed to make requestVote RPC: target="{Voter server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f}" error="failed to connect to peer: 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f" +testLoggerAdapter verbose: server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: 2022-08-12T07:30:12.827Z [WARN] server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: Election timeout reached, restarting election + testing.go:149: server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: 2022-08-12T07:30:12.827Z [WARN] server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: Election timeout reached, restarting election +testLoggerAdapter verbose: server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: 2022-08-12T07:30:12.827Z [INFO] server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: entering candidate state: node="Node at 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f [Candidate]" term=5 + testing.go:149: server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: 2022-08-12T07:30:12.827Z [INFO] server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: entering candidate state: node="Node at 6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f [Candidate]" term=5 +testLoggerAdapter verbose: server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: 2022-08-12T07:30:12.827Z [ERROR] server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: failed to make requestVote RPC: target="{Voter server-d43512ec-dec4-a21a-fe90-da58d090a003 d43512ec-dec4-a21a-fe90-da58d090a003}" error="failed to connect to peer: d43512ec-dec4-a21a-fe90-da58d090a003" + testing.go:149: server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: 2022-08-12T07:30:12.827Z [ERROR] server-6df8a4e8-e8fb-c83b-7ffb-d07d49d77d4f: failed to make requestVote RPC: target="{Voter server-d43512ec-dec4-a21a-fe90-da58d090a003 d43512ec-dec4-a21a-fe90-da58d090a003}" error="failed to connect to peer: d43512ec-dec4-a21a-fe90-da58d090a003" +--- PASS: TestRaft_LeaderLeaseExpire (0.62s) === RUN TestRaft_Barrier -testLoggerAdapter verbose: server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a: 2021-07-10T01:05:08.321Z [INFO] server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a Address:6c11a2fe-1efc-a2f9-acb4-068a2c4b024a} {Suffrage:Voter ID:server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7 Address:3ade38d3-0204-eb9e-f3db-53d78f60bfe7} {Suffrage:Voter ID:server-a01aed58-07aa-0ace-493c-e9efea909c1a Address:a01aed58-07aa-0ace-493c-e9efea909c1a}]" - testing.go:149: server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a: 2021-07-10T01:05:08.321Z [INFO] server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a Address:6c11a2fe-1efc-a2f9-acb4-068a2c4b024a} {Suffrage:Voter ID:server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7 Address:3ade38d3-0204-eb9e-f3db-53d78f60bfe7} {Suffrage:Voter ID:server-a01aed58-07aa-0ace-493c-e9efea909c1a Address:a01aed58-07aa-0ace-493c-e9efea909c1a}]" -testLoggerAdapter verbose: server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a: 2021-07-10T01:05:08.321Z [INFO] server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a: entering follower state: follower="Node at 6c11a2fe-1efc-a2f9-acb4-068a2c4b024a [Follower]" leader= - testing.go:149: server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a: 2021-07-10T01:05:08.321Z [INFO] server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a: entering follower state: follower="Node at 6c11a2fe-1efc-a2f9-acb4-068a2c4b024a [Follower]" leader= -testLoggerAdapter verbose: server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7: 2021-07-10T01:05:08.321Z [INFO] server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a Address:6c11a2fe-1efc-a2f9-acb4-068a2c4b024a} {Suffrage:Voter ID:server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7 Address:3ade38d3-0204-eb9e-f3db-53d78f60bfe7} {Suffrage:Voter ID:server-a01aed58-07aa-0ace-493c-e9efea909c1a Address:a01aed58-07aa-0ace-493c-e9efea909c1a}]" - testing.go:149: server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7: 2021-07-10T01:05:08.321Z [INFO] server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a Address:6c11a2fe-1efc-a2f9-acb4-068a2c4b024a} {Suffrage:Voter ID:server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7 Address:3ade38d3-0204-eb9e-f3db-53d78f60bfe7} {Suffrage:Voter ID:server-a01aed58-07aa-0ace-493c-e9efea909c1a Address:a01aed58-07aa-0ace-493c-e9efea909c1a}]" -testLoggerAdapter verbose: server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7: 2021-07-10T01:05:08.321Z [INFO] server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7: entering follower state: follower="Node at 3ade38d3-0204-eb9e-f3db-53d78f60bfe7 [Follower]" leader= - testing.go:149: server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7: 2021-07-10T01:05:08.321Z [INFO] server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7: entering follower state: follower="Node at 3ade38d3-0204-eb9e-f3db-53d78f60bfe7 [Follower]" leader= -testLoggerAdapter verbose: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.322Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a Address:6c11a2fe-1efc-a2f9-acb4-068a2c4b024a} {Suffrage:Voter ID:server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7 Address:3ade38d3-0204-eb9e-f3db-53d78f60bfe7} {Suffrage:Voter ID:server-a01aed58-07aa-0ace-493c-e9efea909c1a Address:a01aed58-07aa-0ace-493c-e9efea909c1a}]" - testing.go:149: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.322Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a Address:6c11a2fe-1efc-a2f9-acb4-068a2c4b024a} {Suffrage:Voter ID:server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7 Address:3ade38d3-0204-eb9e-f3db-53d78f60bfe7} {Suffrage:Voter ID:server-a01aed58-07aa-0ace-493c-e9efea909c1a Address:a01aed58-07aa-0ace-493c-e9efea909c1a}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:08.322Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:08.322Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.322Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: entering follower state: follower="Node at a01aed58-07aa-0ace-493c-e9efea909c1a [Follower]" leader= - testing.go:149: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.322Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: entering follower state: follower="Node at a01aed58-07aa-0ace-493c-e9efea909c1a [Follower]" leader= -testLoggerAdapter verbose: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.383Z [WARN] server-a01aed58-07aa-0ace-493c-e9efea909c1a: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.383Z [WARN] server-a01aed58-07aa-0ace-493c-e9efea909c1a: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.383Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: entering candidate state: node="Node at a01aed58-07aa-0ace-493c-e9efea909c1a [Candidate]" term=2 - testing.go:149: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.383Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: entering candidate state: node="Node at a01aed58-07aa-0ace-493c-e9efea909c1a [Candidate]" term=2 -testLoggerAdapter verbose: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.383Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: election won: tally=2 - testing.go:149: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.383Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: election won: tally=2 -testLoggerAdapter verbose: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.383Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: entering leader state: leader="Node at a01aed58-07aa-0ace-493c-e9efea909c1a [Leader]" - testing.go:149: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.383Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: entering leader state: leader="Node at a01aed58-07aa-0ace-493c-e9efea909c1a [Leader]" -testLoggerAdapter verbose: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.383Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: added peer, starting replication: peer=server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a - testing.go:149: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.383Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: added peer, starting replication: peer=server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a -testLoggerAdapter verbose: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.383Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: added peer, starting replication: peer=server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7 - testing.go:149: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.383Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: added peer, starting replication: peer=server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7 -testLoggerAdapter verbose: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.383Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: pipelining replication: peer="{Voter server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a 6c11a2fe-1efc-a2f9-acb4-068a2c4b024a}" - testing.go:149: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.383Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: pipelining replication: peer="{Voter server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a 6c11a2fe-1efc-a2f9-acb4-068a2c4b024a}" -testLoggerAdapter verbose: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.383Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: pipelining replication: peer="{Voter server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7 3ade38d3-0204-eb9e-f3db-53d78f60bfe7}" - testing.go:149: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.383Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: pipelining replication: peer="{Voter server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7 3ade38d3-0204-eb9e-f3db-53d78f60bfe7}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:08.488Z [INFO] stable state for Leader reached at 2021-07-10 01:05:08.383663857 +0000 UTC m=+13.771821807 (1 nodes), 61.413707ms from start of poll, 62.306136ms from cluster start. Timeout at 2021-07-10 01:05:08.488728565 +0000 UTC m=+13.876886535, 105.064728ms after stability - testing.go:149: cluster: 2021-07-10T01:05:08.488Z [INFO] stable state for Leader reached at 2021-07-10 01:05:08.383663857 +0000 UTC m=+13.771821807 (1 nodes), 61.413707ms from start of poll, 62.306136ms from cluster start. Timeout at 2021-07-10 01:05:08.488728565 +0000 UTC m=+13.876886535, 105.064728ms after stability -testLoggerAdapter verbose: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.501Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: aborting pipeline replication: peer="{Voter server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7 3ade38d3-0204-eb9e-f3db-53d78f60bfe7}" - testing.go:149: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.501Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: aborting pipeline replication: peer="{Voter server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7 3ade38d3-0204-eb9e-f3db-53d78f60bfe7}" -testLoggerAdapter verbose: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.501Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: aborting pipeline replication: peer="{Voter server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a 6c11a2fe-1efc-a2f9-acb4-068a2c4b024a}" - testing.go:149: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.501Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: aborting pipeline replication: peer="{Voter server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a 6c11a2fe-1efc-a2f9-acb4-068a2c4b024a}" ---- PASS: TestRaft_Barrier (0.18s) +testLoggerAdapter verbose: server-021f3dba-af2e-ed0f-317c-1c8f71f2b419: 2022-08-12T07:30:12.865Z [INFO] server-021f3dba-af2e-ed0f-317c-1c8f71f2b419: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-021f3dba-af2e-ed0f-317c-1c8f71f2b419 Address:021f3dba-af2e-ed0f-317c-1c8f71f2b419} {Suffrage:Voter ID:server-63827054-9b51-47dc-cd4a-0cbf24c5fa34 Address:63827054-9b51-47dc-cd4a-0cbf24c5fa34} {Suffrage:Voter ID:server-fdf73006-7189-f8eb-79a5-0794d6d24c40 Address:fdf73006-7189-f8eb-79a5-0794d6d24c40}]" + testing.go:149: server-021f3dba-af2e-ed0f-317c-1c8f71f2b419: 2022-08-12T07:30:12.865Z [INFO] server-021f3dba-af2e-ed0f-317c-1c8f71f2b419: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-021f3dba-af2e-ed0f-317c-1c8f71f2b419 Address:021f3dba-af2e-ed0f-317c-1c8f71f2b419} {Suffrage:Voter ID:server-63827054-9b51-47dc-cd4a-0cbf24c5fa34 Address:63827054-9b51-47dc-cd4a-0cbf24c5fa34} {Suffrage:Voter ID:server-fdf73006-7189-f8eb-79a5-0794d6d24c40 Address:fdf73006-7189-f8eb-79a5-0794d6d24c40}]" +testLoggerAdapter verbose: server-021f3dba-af2e-ed0f-317c-1c8f71f2b419: 2022-08-12T07:30:12.865Z [INFO] server-021f3dba-af2e-ed0f-317c-1c8f71f2b419: entering follower state: follower="Node at 021f3dba-af2e-ed0f-317c-1c8f71f2b419 [Follower]" leader= + testing.go:149: server-021f3dba-af2e-ed0f-317c-1c8f71f2b419: 2022-08-12T07:30:12.865Z [INFO] server-021f3dba-af2e-ed0f-317c-1c8f71f2b419: entering follower state: follower="Node at 021f3dba-af2e-ed0f-317c-1c8f71f2b419 [Follower]" leader= +testLoggerAdapter verbose: server-63827054-9b51-47dc-cd4a-0cbf24c5fa34: 2022-08-12T07:30:12.866Z [INFO] server-63827054-9b51-47dc-cd4a-0cbf24c5fa34: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-021f3dba-af2e-ed0f-317c-1c8f71f2b419 Address:021f3dba-af2e-ed0f-317c-1c8f71f2b419} {Suffrage:Voter ID:server-63827054-9b51-47dc-cd4a-0cbf24c5fa34 Address:63827054-9b51-47dc-cd4a-0cbf24c5fa34} {Suffrage:Voter ID:server-fdf73006-7189-f8eb-79a5-0794d6d24c40 Address:fdf73006-7189-f8eb-79a5-0794d6d24c40}]" + testing.go:149: server-63827054-9b51-47dc-cd4a-0cbf24c5fa34: 2022-08-12T07:30:12.866Z [INFO] server-63827054-9b51-47dc-cd4a-0cbf24c5fa34: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-021f3dba-af2e-ed0f-317c-1c8f71f2b419 Address:021f3dba-af2e-ed0f-317c-1c8f71f2b419} {Suffrage:Voter ID:server-63827054-9b51-47dc-cd4a-0cbf24c5fa34 Address:63827054-9b51-47dc-cd4a-0cbf24c5fa34} {Suffrage:Voter ID:server-fdf73006-7189-f8eb-79a5-0794d6d24c40 Address:fdf73006-7189-f8eb-79a5-0794d6d24c40}]" +testLoggerAdapter verbose: server-63827054-9b51-47dc-cd4a-0cbf24c5fa34: 2022-08-12T07:30:12.866Z [INFO] server-63827054-9b51-47dc-cd4a-0cbf24c5fa34: entering follower state: follower="Node at 63827054-9b51-47dc-cd4a-0cbf24c5fa34 [Follower]" leader= + testing.go:149: server-63827054-9b51-47dc-cd4a-0cbf24c5fa34: 2022-08-12T07:30:12.866Z [INFO] server-63827054-9b51-47dc-cd4a-0cbf24c5fa34: entering follower state: follower="Node at 63827054-9b51-47dc-cd4a-0cbf24c5fa34 [Follower]" leader= +testLoggerAdapter verbose: server-fdf73006-7189-f8eb-79a5-0794d6d24c40: 2022-08-12T07:30:12.866Z [INFO] server-fdf73006-7189-f8eb-79a5-0794d6d24c40: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-021f3dba-af2e-ed0f-317c-1c8f71f2b419 Address:021f3dba-af2e-ed0f-317c-1c8f71f2b419} {Suffrage:Voter ID:server-63827054-9b51-47dc-cd4a-0cbf24c5fa34 Address:63827054-9b51-47dc-cd4a-0cbf24c5fa34} {Suffrage:Voter ID:server-fdf73006-7189-f8eb-79a5-0794d6d24c40 Address:fdf73006-7189-f8eb-79a5-0794d6d24c40}]" + testing.go:149: server-fdf73006-7189-f8eb-79a5-0794d6d24c40: 2022-08-12T07:30:12.866Z [INFO] server-fdf73006-7189-f8eb-79a5-0794d6d24c40: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-021f3dba-af2e-ed0f-317c-1c8f71f2b419 Address:021f3dba-af2e-ed0f-317c-1c8f71f2b419} {Suffrage:Voter ID:server-63827054-9b51-47dc-cd4a-0cbf24c5fa34 Address:63827054-9b51-47dc-cd4a-0cbf24c5fa34} {Suffrage:Voter ID:server-fdf73006-7189-f8eb-79a5-0794d6d24c40 Address:fdf73006-7189-f8eb-79a5-0794d6d24c40}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:12.866Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:12.866Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-fdf73006-7189-f8eb-79a5-0794d6d24c40: 2022-08-12T07:30:12.866Z [INFO] server-fdf73006-7189-f8eb-79a5-0794d6d24c40: entering follower state: follower="Node at fdf73006-7189-f8eb-79a5-0794d6d24c40 [Follower]" leader= + testing.go:149: server-fdf73006-7189-f8eb-79a5-0794d6d24c40: 2022-08-12T07:30:12.866Z [INFO] server-fdf73006-7189-f8eb-79a5-0794d6d24c40: entering follower state: follower="Node at fdf73006-7189-f8eb-79a5-0794d6d24c40 [Follower]" leader= +testLoggerAdapter verbose: server-fdf73006-7189-f8eb-79a5-0794d6d24c40: 2022-08-12T07:30:12.956Z [WARN] server-fdf73006-7189-f8eb-79a5-0794d6d24c40: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-fdf73006-7189-f8eb-79a5-0794d6d24c40: 2022-08-12T07:30:12.956Z [WARN] server-fdf73006-7189-f8eb-79a5-0794d6d24c40: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-fdf73006-7189-f8eb-79a5-0794d6d24c40: 2022-08-12T07:30:12.956Z [INFO] server-fdf73006-7189-f8eb-79a5-0794d6d24c40: entering candidate state: node="Node at fdf73006-7189-f8eb-79a5-0794d6d24c40 [Candidate]" term=2 + testing.go:149: server-fdf73006-7189-f8eb-79a5-0794d6d24c40: 2022-08-12T07:30:12.956Z [INFO] server-fdf73006-7189-f8eb-79a5-0794d6d24c40: entering candidate state: node="Node at fdf73006-7189-f8eb-79a5-0794d6d24c40 [Candidate]" term=2 +testLoggerAdapter verbose: server-fdf73006-7189-f8eb-79a5-0794d6d24c40: 2022-08-12T07:30:12.956Z [INFO] server-fdf73006-7189-f8eb-79a5-0794d6d24c40: election won: tally=2 + testing.go:149: server-fdf73006-7189-f8eb-79a5-0794d6d24c40: 2022-08-12T07:30:12.956Z [INFO] server-fdf73006-7189-f8eb-79a5-0794d6d24c40: election won: tally=2 +testLoggerAdapter verbose: server-fdf73006-7189-f8eb-79a5-0794d6d24c40: 2022-08-12T07:30:12.956Z [INFO] server-fdf73006-7189-f8eb-79a5-0794d6d24c40: entering leader state: leader="Node at fdf73006-7189-f8eb-79a5-0794d6d24c40 [Leader]" + testing.go:149: server-fdf73006-7189-f8eb-79a5-0794d6d24c40: 2022-08-12T07:30:12.956Z [INFO] server-fdf73006-7189-f8eb-79a5-0794d6d24c40: entering leader state: leader="Node at fdf73006-7189-f8eb-79a5-0794d6d24c40 [Leader]" +testLoggerAdapter verbose: server-fdf73006-7189-f8eb-79a5-0794d6d24c40: 2022-08-12T07:30:12.956Z [INFO] server-fdf73006-7189-f8eb-79a5-0794d6d24c40: added peer, starting replication: peer=server-021f3dba-af2e-ed0f-317c-1c8f71f2b419 + testing.go:149: server-fdf73006-7189-f8eb-79a5-0794d6d24c40: 2022-08-12T07:30:12.956Z [INFO] server-fdf73006-7189-f8eb-79a5-0794d6d24c40: added peer, starting replication: peer=server-021f3dba-af2e-ed0f-317c-1c8f71f2b419 +testLoggerAdapter verbose: server-fdf73006-7189-f8eb-79a5-0794d6d24c40: 2022-08-12T07:30:12.956Z [INFO] server-fdf73006-7189-f8eb-79a5-0794d6d24c40: added peer, starting replication: peer=server-63827054-9b51-47dc-cd4a-0cbf24c5fa34 + testing.go:149: server-fdf73006-7189-f8eb-79a5-0794d6d24c40: 2022-08-12T07:30:12.956Z [INFO] server-fdf73006-7189-f8eb-79a5-0794d6d24c40: added peer, starting replication: peer=server-63827054-9b51-47dc-cd4a-0cbf24c5fa34 +testLoggerAdapter verbose: server-fdf73006-7189-f8eb-79a5-0794d6d24c40: 2022-08-12T07:30:12.956Z [INFO] server-fdf73006-7189-f8eb-79a5-0794d6d24c40: pipelining replication: peer="{Voter server-021f3dba-af2e-ed0f-317c-1c8f71f2b419 021f3dba-af2e-ed0f-317c-1c8f71f2b419}" + testing.go:149: server-fdf73006-7189-f8eb-79a5-0794d6d24c40: 2022-08-12T07:30:12.956Z [INFO] server-fdf73006-7189-f8eb-79a5-0794d6d24c40: pipelining replication: peer="{Voter server-021f3dba-af2e-ed0f-317c-1c8f71f2b419 021f3dba-af2e-ed0f-317c-1c8f71f2b419}" +testLoggerAdapter verbose: server-fdf73006-7189-f8eb-79a5-0794d6d24c40: 2022-08-12T07:30:12.956Z [INFO] server-fdf73006-7189-f8eb-79a5-0794d6d24c40: pipelining replication: peer="{Voter server-63827054-9b51-47dc-cd4a-0cbf24c5fa34 63827054-9b51-47dc-cd4a-0cbf24c5fa34}" + testing.go:149: server-fdf73006-7189-f8eb-79a5-0794d6d24c40: 2022-08-12T07:30:12.956Z [INFO] server-fdf73006-7189-f8eb-79a5-0794d6d24c40: pipelining replication: peer="{Voter server-63827054-9b51-47dc-cd4a-0cbf24c5fa34 63827054-9b51-47dc-cd4a-0cbf24c5fa34}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:13.061Z [INFO] stable state for Leader reached at 2022-08-12 07:30:12.95666466 +0000 UTC m=+13.913351121 (1 nodes), 90.25636ms from start of poll, 91.25738ms from cluster start. Timeout at 2022-08-12 07:30:13.0617425 +0000 UTC m=+14.018428981, 105.07786ms after stability + testing.go:149: cluster: 2022-08-12T07:30:13.061Z [INFO] stable state for Leader reached at 2022-08-12 07:30:12.95666466 +0000 UTC m=+13.913351121 (1 nodes), 90.25636ms from start of poll, 91.25738ms from cluster start. Timeout at 2022-08-12 07:30:13.0617425 +0000 UTC m=+14.018428981, 105.07786ms after stability +testLoggerAdapter verbose: server-fdf73006-7189-f8eb-79a5-0794d6d24c40: 2022-08-12T07:30:13.075Z [INFO] server-fdf73006-7189-f8eb-79a5-0794d6d24c40: aborting pipeline replication: peer="{Voter server-021f3dba-af2e-ed0f-317c-1c8f71f2b419 021f3dba-af2e-ed0f-317c-1c8f71f2b419}" + testing.go:149: server-fdf73006-7189-f8eb-79a5-0794d6d24c40: 2022-08-12T07:30:13.075Z [INFO] server-fdf73006-7189-f8eb-79a5-0794d6d24c40: aborting pipeline replication: peer="{Voter server-021f3dba-af2e-ed0f-317c-1c8f71f2b419 021f3dba-af2e-ed0f-317c-1c8f71f2b419}" +testLoggerAdapter verbose: server-fdf73006-7189-f8eb-79a5-0794d6d24c40: 2022-08-12T07:30:13.075Z [INFO] server-fdf73006-7189-f8eb-79a5-0794d6d24c40: aborting pipeline replication: peer="{Voter server-63827054-9b51-47dc-cd4a-0cbf24c5fa34 63827054-9b51-47dc-cd4a-0cbf24c5fa34}" + testing.go:149: server-fdf73006-7189-f8eb-79a5-0794d6d24c40: 2022-08-12T07:30:13.075Z [INFO] server-fdf73006-7189-f8eb-79a5-0794d6d24c40: aborting pipeline replication: peer="{Voter server-63827054-9b51-47dc-cd4a-0cbf24c5fa34 63827054-9b51-47dc-cd4a-0cbf24c5fa34}" +--- PASS: TestRaft_Barrier (0.21s) === RUN TestRaft_VerifyLeader -testLoggerAdapter verbose: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.503Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f80ef73e-f1d6-f137-0763-2550381591d3 Address:f80ef73e-f1d6-f137-0763-2550381591d3} {Suffrage:Voter ID:server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f Address:d7aee95d-d803-bb31-5fe2-03f7fc53ce0f} {Suffrage:Voter ID:server-6533e748-739f-3530-cef8-f8240bfb4c4d Address:6533e748-739f-3530-cef8-f8240bfb4c4d}]" - testing.go:149: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.503Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f80ef73e-f1d6-f137-0763-2550381591d3 Address:f80ef73e-f1d6-f137-0763-2550381591d3} {Suffrage:Voter ID:server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f Address:d7aee95d-d803-bb31-5fe2-03f7fc53ce0f} {Suffrage:Voter ID:server-6533e748-739f-3530-cef8-f8240bfb4c4d Address:6533e748-739f-3530-cef8-f8240bfb4c4d}]" -testLoggerAdapter verbose: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.503Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: entering follower state: follower="Node at f80ef73e-f1d6-f137-0763-2550381591d3 [Follower]" leader= - testing.go:149: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.503Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: entering follower state: follower="Node at f80ef73e-f1d6-f137-0763-2550381591d3 [Follower]" leader= -testLoggerAdapter verbose: server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f: 2021-07-10T01:05:08.503Z [INFO] server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f80ef73e-f1d6-f137-0763-2550381591d3 Address:f80ef73e-f1d6-f137-0763-2550381591d3} {Suffrage:Voter ID:server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f Address:d7aee95d-d803-bb31-5fe2-03f7fc53ce0f} {Suffrage:Voter ID:server-6533e748-739f-3530-cef8-f8240bfb4c4d Address:6533e748-739f-3530-cef8-f8240bfb4c4d}]" - testing.go:149: server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f: 2021-07-10T01:05:08.503Z [INFO] server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f80ef73e-f1d6-f137-0763-2550381591d3 Address:f80ef73e-f1d6-f137-0763-2550381591d3} {Suffrage:Voter ID:server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f Address:d7aee95d-d803-bb31-5fe2-03f7fc53ce0f} {Suffrage:Voter ID:server-6533e748-739f-3530-cef8-f8240bfb4c4d Address:6533e748-739f-3530-cef8-f8240bfb4c4d}]" -testLoggerAdapter verbose: server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f: 2021-07-10T01:05:08.503Z [INFO] server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f: entering follower state: follower="Node at d7aee95d-d803-bb31-5fe2-03f7fc53ce0f [Follower]" leader= - testing.go:149: server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f: 2021-07-10T01:05:08.503Z [INFO] server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f: entering follower state: follower="Node at d7aee95d-d803-bb31-5fe2-03f7fc53ce0f [Follower]" leader= -testLoggerAdapter verbose: server-6533e748-739f-3530-cef8-f8240bfb4c4d: 2021-07-10T01:05:08.503Z [INFO] server-6533e748-739f-3530-cef8-f8240bfb4c4d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f80ef73e-f1d6-f137-0763-2550381591d3 Address:f80ef73e-f1d6-f137-0763-2550381591d3} {Suffrage:Voter ID:server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f Address:d7aee95d-d803-bb31-5fe2-03f7fc53ce0f} {Suffrage:Voter ID:server-6533e748-739f-3530-cef8-f8240bfb4c4d Address:6533e748-739f-3530-cef8-f8240bfb4c4d}]" - testing.go:149: server-6533e748-739f-3530-cef8-f8240bfb4c4d: 2021-07-10T01:05:08.503Z [INFO] server-6533e748-739f-3530-cef8-f8240bfb4c4d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f80ef73e-f1d6-f137-0763-2550381591d3 Address:f80ef73e-f1d6-f137-0763-2550381591d3} {Suffrage:Voter ID:server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f Address:d7aee95d-d803-bb31-5fe2-03f7fc53ce0f} {Suffrage:Voter ID:server-6533e748-739f-3530-cef8-f8240bfb4c4d Address:6533e748-739f-3530-cef8-f8240bfb4c4d}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:08.504Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:08.504Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-6533e748-739f-3530-cef8-f8240bfb4c4d: 2021-07-10T01:05:08.504Z [INFO] server-6533e748-739f-3530-cef8-f8240bfb4c4d: entering follower state: follower="Node at 6533e748-739f-3530-cef8-f8240bfb4c4d [Follower]" leader= - testing.go:149: server-6533e748-739f-3530-cef8-f8240bfb4c4d: 2021-07-10T01:05:08.504Z [INFO] server-6533e748-739f-3530-cef8-f8240bfb4c4d: entering follower state: follower="Node at 6533e748-739f-3530-cef8-f8240bfb4c4d [Follower]" leader= -testLoggerAdapter verbose: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.556Z [WARN] server-f80ef73e-f1d6-f137-0763-2550381591d3: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.556Z [WARN] server-f80ef73e-f1d6-f137-0763-2550381591d3: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.556Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: entering candidate state: node="Node at f80ef73e-f1d6-f137-0763-2550381591d3 [Candidate]" term=2 - testing.go:149: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.556Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: entering candidate state: node="Node at f80ef73e-f1d6-f137-0763-2550381591d3 [Candidate]" term=2 -testLoggerAdapter verbose: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.556Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: election won: tally=2 - testing.go:149: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.556Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: election won: tally=2 -testLoggerAdapter verbose: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.557Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: entering leader state: leader="Node at f80ef73e-f1d6-f137-0763-2550381591d3 [Leader]" - testing.go:149: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.557Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: entering leader state: leader="Node at f80ef73e-f1d6-f137-0763-2550381591d3 [Leader]" -testLoggerAdapter verbose: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.557Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: added peer, starting replication: peer=server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f - testing.go:149: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.557Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: added peer, starting replication: peer=server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f -testLoggerAdapter verbose: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.557Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: added peer, starting replication: peer=server-6533e748-739f-3530-cef8-f8240bfb4c4d - testing.go:149: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.557Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: added peer, starting replication: peer=server-6533e748-739f-3530-cef8-f8240bfb4c4d -testLoggerAdapter verbose: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.557Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: pipelining replication: peer="{Voter server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f d7aee95d-d803-bb31-5fe2-03f7fc53ce0f}" - testing.go:149: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.557Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: pipelining replication: peer="{Voter server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f d7aee95d-d803-bb31-5fe2-03f7fc53ce0f}" -testLoggerAdapter verbose: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.557Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: pipelining replication: peer="{Voter server-6533e748-739f-3530-cef8-f8240bfb4c4d 6533e748-739f-3530-cef8-f8240bfb4c4d}" - testing.go:149: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.557Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: pipelining replication: peer="{Voter server-6533e748-739f-3530-cef8-f8240bfb4c4d 6533e748-739f-3530-cef8-f8240bfb4c4d}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:08.662Z [INFO] stable state for Leader reached at 2021-07-10 01:05:08.557147306 +0000 UTC m=+13.945305296 (1 nodes), 53.033919ms from start of poll, 54.022849ms from cluster start. Timeout at 2021-07-10 01:05:08.662232834 +0000 UTC m=+14.050390824, 105.085528ms after stability - testing.go:149: cluster: 2021-07-10T01:05:08.662Z [INFO] stable state for Leader reached at 2021-07-10 01:05:08.557147306 +0000 UTC m=+13.945305296 (1 nodes), 53.033919ms from start of poll, 54.022849ms from cluster start. Timeout at 2021-07-10 01:05:08.662232834 +0000 UTC m=+14.050390824, 105.085528ms after stability -testLoggerAdapter verbose: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.662Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: aborting pipeline replication: peer="{Voter server-6533e748-739f-3530-cef8-f8240bfb4c4d 6533e748-739f-3530-cef8-f8240bfb4c4d}" - testing.go:149: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.662Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: aborting pipeline replication: peer="{Voter server-6533e748-739f-3530-cef8-f8240bfb4c4d 6533e748-739f-3530-cef8-f8240bfb4c4d}" -testLoggerAdapter verbose: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.662Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: aborting pipeline replication: peer="{Voter server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f d7aee95d-d803-bb31-5fe2-03f7fc53ce0f}" - testing.go:149: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.662Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: aborting pipeline replication: peer="{Voter server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f d7aee95d-d803-bb31-5fe2-03f7fc53ce0f}" ---- PASS: TestRaft_VerifyLeader (0.16s) +testLoggerAdapter verbose: server-1f0d975c-0e9a-0ef8-efa5-0f64ee3678ef: 2022-08-12T07:30:13.077Z [INFO] server-1f0d975c-0e9a-0ef8-efa5-0f64ee3678ef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f0d975c-0e9a-0ef8-efa5-0f64ee3678ef Address:1f0d975c-0e9a-0ef8-efa5-0f64ee3678ef} {Suffrage:Voter ID:server-39777192-27b6-9ea9-cd3b-b6f2f8843c08 Address:39777192-27b6-9ea9-cd3b-b6f2f8843c08} {Suffrage:Voter ID:server-4a902b2f-4171-e319-3418-90e89629a652 Address:4a902b2f-4171-e319-3418-90e89629a652}]" + testing.go:149: server-1f0d975c-0e9a-0ef8-efa5-0f64ee3678ef: 2022-08-12T07:30:13.077Z [INFO] server-1f0d975c-0e9a-0ef8-efa5-0f64ee3678ef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f0d975c-0e9a-0ef8-efa5-0f64ee3678ef Address:1f0d975c-0e9a-0ef8-efa5-0f64ee3678ef} {Suffrage:Voter ID:server-39777192-27b6-9ea9-cd3b-b6f2f8843c08 Address:39777192-27b6-9ea9-cd3b-b6f2f8843c08} {Suffrage:Voter ID:server-4a902b2f-4171-e319-3418-90e89629a652 Address:4a902b2f-4171-e319-3418-90e89629a652}]" +testLoggerAdapter verbose: server-39777192-27b6-9ea9-cd3b-b6f2f8843c08: 2022-08-12T07:30:13.077Z [INFO] server-39777192-27b6-9ea9-cd3b-b6f2f8843c08: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f0d975c-0e9a-0ef8-efa5-0f64ee3678ef Address:1f0d975c-0e9a-0ef8-efa5-0f64ee3678ef} {Suffrage:Voter ID:server-39777192-27b6-9ea9-cd3b-b6f2f8843c08 Address:39777192-27b6-9ea9-cd3b-b6f2f8843c08} {Suffrage:Voter ID:server-4a902b2f-4171-e319-3418-90e89629a652 Address:4a902b2f-4171-e319-3418-90e89629a652}]" + testing.go:149: server-39777192-27b6-9ea9-cd3b-b6f2f8843c08: 2022-08-12T07:30:13.077Z [INFO] server-39777192-27b6-9ea9-cd3b-b6f2f8843c08: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f0d975c-0e9a-0ef8-efa5-0f64ee3678ef Address:1f0d975c-0e9a-0ef8-efa5-0f64ee3678ef} {Suffrage:Voter ID:server-39777192-27b6-9ea9-cd3b-b6f2f8843c08 Address:39777192-27b6-9ea9-cd3b-b6f2f8843c08} {Suffrage:Voter ID:server-4a902b2f-4171-e319-3418-90e89629a652 Address:4a902b2f-4171-e319-3418-90e89629a652}]" +testLoggerAdapter verbose: server-1f0d975c-0e9a-0ef8-efa5-0f64ee3678ef: 2022-08-12T07:30:13.077Z [INFO] server-1f0d975c-0e9a-0ef8-efa5-0f64ee3678ef: entering follower state: follower="Node at 1f0d975c-0e9a-0ef8-efa5-0f64ee3678ef [Follower]" leader= + testing.go:149: server-1f0d975c-0e9a-0ef8-efa5-0f64ee3678ef: 2022-08-12T07:30:13.077Z [INFO] server-1f0d975c-0e9a-0ef8-efa5-0f64ee3678ef: entering follower state: follower="Node at 1f0d975c-0e9a-0ef8-efa5-0f64ee3678ef [Follower]" leader= +testLoggerAdapter verbose: server-39777192-27b6-9ea9-cd3b-b6f2f8843c08: 2022-08-12T07:30:13.077Z [INFO] server-39777192-27b6-9ea9-cd3b-b6f2f8843c08: entering follower state: follower="Node at 39777192-27b6-9ea9-cd3b-b6f2f8843c08 [Follower]" leader= + testing.go:149: server-39777192-27b6-9ea9-cd3b-b6f2f8843c08: 2022-08-12T07:30:13.077Z [INFO] server-39777192-27b6-9ea9-cd3b-b6f2f8843c08: entering follower state: follower="Node at 39777192-27b6-9ea9-cd3b-b6f2f8843c08 [Follower]" leader= +testLoggerAdapter verbose: server-4a902b2f-4171-e319-3418-90e89629a652: 2022-08-12T07:30:13.078Z [INFO] server-4a902b2f-4171-e319-3418-90e89629a652: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f0d975c-0e9a-0ef8-efa5-0f64ee3678ef Address:1f0d975c-0e9a-0ef8-efa5-0f64ee3678ef} {Suffrage:Voter ID:server-39777192-27b6-9ea9-cd3b-b6f2f8843c08 Address:39777192-27b6-9ea9-cd3b-b6f2f8843c08} {Suffrage:Voter ID:server-4a902b2f-4171-e319-3418-90e89629a652 Address:4a902b2f-4171-e319-3418-90e89629a652}]" + testing.go:149: server-4a902b2f-4171-e319-3418-90e89629a652: 2022-08-12T07:30:13.078Z [INFO] server-4a902b2f-4171-e319-3418-90e89629a652: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f0d975c-0e9a-0ef8-efa5-0f64ee3678ef Address:1f0d975c-0e9a-0ef8-efa5-0f64ee3678ef} {Suffrage:Voter ID:server-39777192-27b6-9ea9-cd3b-b6f2f8843c08 Address:39777192-27b6-9ea9-cd3b-b6f2f8843c08} {Suffrage:Voter ID:server-4a902b2f-4171-e319-3418-90e89629a652 Address:4a902b2f-4171-e319-3418-90e89629a652}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:13.078Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:13.078Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-4a902b2f-4171-e319-3418-90e89629a652: 2022-08-12T07:30:13.078Z [INFO] server-4a902b2f-4171-e319-3418-90e89629a652: entering follower state: follower="Node at 4a902b2f-4171-e319-3418-90e89629a652 [Follower]" leader= + testing.go:149: server-4a902b2f-4171-e319-3418-90e89629a652: 2022-08-12T07:30:13.078Z [INFO] server-4a902b2f-4171-e319-3418-90e89629a652: entering follower state: follower="Node at 4a902b2f-4171-e319-3418-90e89629a652 [Follower]" leader= +testLoggerAdapter verbose: server-4a902b2f-4171-e319-3418-90e89629a652: 2022-08-12T07:30:13.157Z [WARN] server-4a902b2f-4171-e319-3418-90e89629a652: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-4a902b2f-4171-e319-3418-90e89629a652: 2022-08-12T07:30:13.157Z [WARN] server-4a902b2f-4171-e319-3418-90e89629a652: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-4a902b2f-4171-e319-3418-90e89629a652: 2022-08-12T07:30:13.158Z [INFO] server-4a902b2f-4171-e319-3418-90e89629a652: entering candidate state: node="Node at 4a902b2f-4171-e319-3418-90e89629a652 [Candidate]" term=2 + testing.go:149: server-4a902b2f-4171-e319-3418-90e89629a652: 2022-08-12T07:30:13.158Z [INFO] server-4a902b2f-4171-e319-3418-90e89629a652: entering candidate state: node="Node at 4a902b2f-4171-e319-3418-90e89629a652 [Candidate]" term=2 +testLoggerAdapter verbose: server-4a902b2f-4171-e319-3418-90e89629a652: 2022-08-12T07:30:13.158Z [INFO] server-4a902b2f-4171-e319-3418-90e89629a652: election won: tally=2 + testing.go:149: server-4a902b2f-4171-e319-3418-90e89629a652: 2022-08-12T07:30:13.158Z [INFO] server-4a902b2f-4171-e319-3418-90e89629a652: election won: tally=2 +testLoggerAdapter verbose: server-4a902b2f-4171-e319-3418-90e89629a652: 2022-08-12T07:30:13.158Z [INFO] server-4a902b2f-4171-e319-3418-90e89629a652: entering leader state: leader="Node at 4a902b2f-4171-e319-3418-90e89629a652 [Leader]" + testing.go:149: server-4a902b2f-4171-e319-3418-90e89629a652: 2022-08-12T07:30:13.158Z [INFO] server-4a902b2f-4171-e319-3418-90e89629a652: entering leader state: leader="Node at 4a902b2f-4171-e319-3418-90e89629a652 [Leader]" +testLoggerAdapter verbose: server-4a902b2f-4171-e319-3418-90e89629a652: 2022-08-12T07:30:13.158Z [INFO] server-4a902b2f-4171-e319-3418-90e89629a652: added peer, starting replication: peer=server-1f0d975c-0e9a-0ef8-efa5-0f64ee3678ef + testing.go:149: server-4a902b2f-4171-e319-3418-90e89629a652: 2022-08-12T07:30:13.158Z [INFO] server-4a902b2f-4171-e319-3418-90e89629a652: added peer, starting replication: peer=server-1f0d975c-0e9a-0ef8-efa5-0f64ee3678ef +testLoggerAdapter verbose: server-4a902b2f-4171-e319-3418-90e89629a652: 2022-08-12T07:30:13.158Z [INFO] server-4a902b2f-4171-e319-3418-90e89629a652: added peer, starting replication: peer=server-39777192-27b6-9ea9-cd3b-b6f2f8843c08 + testing.go:149: server-4a902b2f-4171-e319-3418-90e89629a652: 2022-08-12T07:30:13.158Z [INFO] server-4a902b2f-4171-e319-3418-90e89629a652: added peer, starting replication: peer=server-39777192-27b6-9ea9-cd3b-b6f2f8843c08 +testLoggerAdapter verbose: server-4a902b2f-4171-e319-3418-90e89629a652: 2022-08-12T07:30:13.158Z [INFO] server-4a902b2f-4171-e319-3418-90e89629a652: pipelining replication: peer="{Voter server-1f0d975c-0e9a-0ef8-efa5-0f64ee3678ef 1f0d975c-0e9a-0ef8-efa5-0f64ee3678ef}" + testing.go:149: server-4a902b2f-4171-e319-3418-90e89629a652: 2022-08-12T07:30:13.158Z [INFO] server-4a902b2f-4171-e319-3418-90e89629a652: pipelining replication: peer="{Voter server-1f0d975c-0e9a-0ef8-efa5-0f64ee3678ef 1f0d975c-0e9a-0ef8-efa5-0f64ee3678ef}" +testLoggerAdapter verbose: server-4a902b2f-4171-e319-3418-90e89629a652: 2022-08-12T07:30:13.158Z [INFO] server-4a902b2f-4171-e319-3418-90e89629a652: pipelining replication: peer="{Voter server-39777192-27b6-9ea9-cd3b-b6f2f8843c08 39777192-27b6-9ea9-cd3b-b6f2f8843c08}" + testing.go:149: server-4a902b2f-4171-e319-3418-90e89629a652: 2022-08-12T07:30:13.158Z [INFO] server-4a902b2f-4171-e319-3418-90e89629a652: pipelining replication: peer="{Voter server-39777192-27b6-9ea9-cd3b-b6f2f8843c08 39777192-27b6-9ea9-cd3b-b6f2f8843c08}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:13.263Z [INFO] stable state for Leader reached at 2022-08-12 07:30:13.15839826 +0000 UTC m=+14.115084721 (1 nodes), 80.2568ms from start of poll, 81.3198ms from cluster start. Timeout at 2022-08-12 07:30:13.2634691 +0000 UTC m=+14.220155581, 105.07086ms after stability + testing.go:149: cluster: 2022-08-12T07:30:13.263Z [INFO] stable state for Leader reached at 2022-08-12 07:30:13.15839826 +0000 UTC m=+14.115084721 (1 nodes), 80.2568ms from start of poll, 81.3198ms from cluster start. Timeout at 2022-08-12 07:30:13.2634691 +0000 UTC m=+14.220155581, 105.07086ms after stability +testLoggerAdapter verbose: server-4a902b2f-4171-e319-3418-90e89629a652: 2022-08-12T07:30:13.263Z [INFO] server-4a902b2f-4171-e319-3418-90e89629a652: aborting pipeline replication: peer="{Voter server-39777192-27b6-9ea9-cd3b-b6f2f8843c08 39777192-27b6-9ea9-cd3b-b6f2f8843c08}" + testing.go:149: server-4a902b2f-4171-e319-3418-90e89629a652: 2022-08-12T07:30:13.263Z [INFO] server-4a902b2f-4171-e319-3418-90e89629a652: aborting pipeline replication: peer="{Voter server-39777192-27b6-9ea9-cd3b-b6f2f8843c08 39777192-27b6-9ea9-cd3b-b6f2f8843c08}" +testLoggerAdapter verbose: server-4a902b2f-4171-e319-3418-90e89629a652: 2022-08-12T07:30:13.263Z [INFO] server-4a902b2f-4171-e319-3418-90e89629a652: aborting pipeline replication: peer="{Voter server-1f0d975c-0e9a-0ef8-efa5-0f64ee3678ef 1f0d975c-0e9a-0ef8-efa5-0f64ee3678ef}" + testing.go:149: server-4a902b2f-4171-e319-3418-90e89629a652: 2022-08-12T07:30:13.263Z [INFO] server-4a902b2f-4171-e319-3418-90e89629a652: aborting pipeline replication: peer="{Voter server-1f0d975c-0e9a-0ef8-efa5-0f64ee3678ef 1f0d975c-0e9a-0ef8-efa5-0f64ee3678ef}" +--- PASS: TestRaft_VerifyLeader (0.19s) === RUN TestRaft_VerifyLeader_Single -testLoggerAdapter verbose: server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: 2021-07-10T01:05:08.664Z [INFO] server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572 Address:5a7c4091-dc6f-8fb9-8854-c30bb1eaf572}]" - testing.go:149: server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: 2021-07-10T01:05:08.664Z [INFO] server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572 Address:5a7c4091-dc6f-8fb9-8854-c30bb1eaf572}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:08.664Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:08.664Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: 2021-07-10T01:05:08.664Z [INFO] server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: entering follower state: follower="Node at 5a7c4091-dc6f-8fb9-8854-c30bb1eaf572 [Follower]" leader= - testing.go:149: server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: 2021-07-10T01:05:08.664Z [INFO] server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: entering follower state: follower="Node at 5a7c4091-dc6f-8fb9-8854-c30bb1eaf572 [Follower]" leader= -testLoggerAdapter verbose: server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: 2021-07-10T01:05:08.720Z [WARN] server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: 2021-07-10T01:05:08.720Z [WARN] server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: 2021-07-10T01:05:08.721Z [INFO] server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: entering candidate state: node="Node at 5a7c4091-dc6f-8fb9-8854-c30bb1eaf572 [Candidate]" term=2 - testing.go:149: server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: 2021-07-10T01:05:08.721Z [INFO] server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: entering candidate state: node="Node at 5a7c4091-dc6f-8fb9-8854-c30bb1eaf572 [Candidate]" term=2 -testLoggerAdapter verbose: server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: 2021-07-10T01:05:08.721Z [INFO] server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: election won: tally=1 - testing.go:149: server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: 2021-07-10T01:05:08.721Z [INFO] server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: election won: tally=1 -testLoggerAdapter verbose: server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: 2021-07-10T01:05:08.721Z [INFO] server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: entering leader state: leader="Node at 5a7c4091-dc6f-8fb9-8854-c30bb1eaf572 [Leader]" - testing.go:149: server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: 2021-07-10T01:05:08.721Z [INFO] server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: entering leader state: leader="Node at 5a7c4091-dc6f-8fb9-8854-c30bb1eaf572 [Leader]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:08.826Z [INFO] stable state for Leader reached at 2021-07-10 01:05:08.721306097 +0000 UTC m=+14.109464127 (1 nodes), 57.096002ms from start of poll, 57.588787ms from cluster start. Timeout at 2021-07-10 01:05:08.826391765 +0000 UTC m=+14.214549735, 105.085608ms after stability - testing.go:149: cluster: 2021-07-10T01:05:08.826Z [INFO] stable state for Leader reached at 2021-07-10 01:05:08.721306097 +0000 UTC m=+14.109464127 (1 nodes), 57.096002ms from start of poll, 57.588787ms from cluster start. Timeout at 2021-07-10 01:05:08.826391765 +0000 UTC m=+14.214549735, 105.085608ms after stability ---- PASS: TestRaft_VerifyLeader_Single (0.16s) +testLoggerAdapter verbose: server-dfd6d35e-fff9-db51-f4e7-bcf0b00680d8: 2022-08-12T07:30:13.266Z [INFO] server-dfd6d35e-fff9-db51-f4e7-bcf0b00680d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dfd6d35e-fff9-db51-f4e7-bcf0b00680d8 Address:dfd6d35e-fff9-db51-f4e7-bcf0b00680d8}]" + testing.go:149: server-dfd6d35e-fff9-db51-f4e7-bcf0b00680d8: 2022-08-12T07:30:13.266Z [INFO] server-dfd6d35e-fff9-db51-f4e7-bcf0b00680d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dfd6d35e-fff9-db51-f4e7-bcf0b00680d8 Address:dfd6d35e-fff9-db51-f4e7-bcf0b00680d8}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:13.266Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:13.266Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-dfd6d35e-fff9-db51-f4e7-bcf0b00680d8: 2022-08-12T07:30:13.266Z [INFO] server-dfd6d35e-fff9-db51-f4e7-bcf0b00680d8: entering follower state: follower="Node at dfd6d35e-fff9-db51-f4e7-bcf0b00680d8 [Follower]" leader= + testing.go:149: server-dfd6d35e-fff9-db51-f4e7-bcf0b00680d8: 2022-08-12T07:30:13.266Z [INFO] server-dfd6d35e-fff9-db51-f4e7-bcf0b00680d8: entering follower state: follower="Node at dfd6d35e-fff9-db51-f4e7-bcf0b00680d8 [Follower]" leader= +testLoggerAdapter verbose: server-dfd6d35e-fff9-db51-f4e7-bcf0b00680d8: 2022-08-12T07:30:13.336Z [WARN] server-dfd6d35e-fff9-db51-f4e7-bcf0b00680d8: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-dfd6d35e-fff9-db51-f4e7-bcf0b00680d8: 2022-08-12T07:30:13.336Z [WARN] server-dfd6d35e-fff9-db51-f4e7-bcf0b00680d8: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-dfd6d35e-fff9-db51-f4e7-bcf0b00680d8: 2022-08-12T07:30:13.337Z [INFO] server-dfd6d35e-fff9-db51-f4e7-bcf0b00680d8: entering candidate state: node="Node at dfd6d35e-fff9-db51-f4e7-bcf0b00680d8 [Candidate]" term=2 + testing.go:149: server-dfd6d35e-fff9-db51-f4e7-bcf0b00680d8: 2022-08-12T07:30:13.337Z [INFO] server-dfd6d35e-fff9-db51-f4e7-bcf0b00680d8: entering candidate state: node="Node at dfd6d35e-fff9-db51-f4e7-bcf0b00680d8 [Candidate]" term=2 +testLoggerAdapter verbose: server-dfd6d35e-fff9-db51-f4e7-bcf0b00680d8: 2022-08-12T07:30:13.337Z [INFO] server-dfd6d35e-fff9-db51-f4e7-bcf0b00680d8: election won: tally=1 + testing.go:149: server-dfd6d35e-fff9-db51-f4e7-bcf0b00680d8: 2022-08-12T07:30:13.337Z [INFO] server-dfd6d35e-fff9-db51-f4e7-bcf0b00680d8: election won: tally=1 +testLoggerAdapter verbose: server-dfd6d35e-fff9-db51-f4e7-bcf0b00680d8: 2022-08-12T07:30:13.337Z [INFO] server-dfd6d35e-fff9-db51-f4e7-bcf0b00680d8: entering leader state: leader="Node at dfd6d35e-fff9-db51-f4e7-bcf0b00680d8 [Leader]" + testing.go:149: server-dfd6d35e-fff9-db51-f4e7-bcf0b00680d8: 2022-08-12T07:30:13.337Z [INFO] server-dfd6d35e-fff9-db51-f4e7-bcf0b00680d8: entering leader state: leader="Node at dfd6d35e-fff9-db51-f4e7-bcf0b00680d8 [Leader]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:13.442Z [INFO] stable state for Leader reached at 2022-08-12 07:30:13.3374335 +0000 UTC m=+14.294119981 (1 nodes), 71.15262ms from start of poll, 71.72468ms from cluster start. Timeout at 2022-08-12 07:30:13.44252402 +0000 UTC m=+14.399210501, 105.09052ms after stability + testing.go:149: cluster: 2022-08-12T07:30:13.442Z [INFO] stable state for Leader reached at 2022-08-12 07:30:13.3374335 +0000 UTC m=+14.294119981 (1 nodes), 71.15262ms from start of poll, 71.72468ms from cluster start. Timeout at 2022-08-12 07:30:13.44252402 +0000 UTC m=+14.399210501, 105.09052ms after stability +--- PASS: TestRaft_VerifyLeader_Single (0.18s) === RUN TestRaft_VerifyLeader_Fail -testLoggerAdapter verbose: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:08.828Z [INFO] server-49248667-6b34-68b4-1b72-125153ca0cd9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49248667-6b34-68b4-1b72-125153ca0cd9 Address:49248667-6b34-68b4-1b72-125153ca0cd9} {Suffrage:Voter ID:server-5d8b8708-f49b-4d49-a364-61238cdaf4ea Address:5d8b8708-f49b-4d49-a364-61238cdaf4ea}]" - testing.go:149: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:08.828Z [INFO] server-49248667-6b34-68b4-1b72-125153ca0cd9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49248667-6b34-68b4-1b72-125153ca0cd9 Address:49248667-6b34-68b4-1b72-125153ca0cd9} {Suffrage:Voter ID:server-5d8b8708-f49b-4d49-a364-61238cdaf4ea Address:5d8b8708-f49b-4d49-a364-61238cdaf4ea}]" -testLoggerAdapter verbose: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:08.829Z [INFO] server-49248667-6b34-68b4-1b72-125153ca0cd9: entering follower state: follower="Node at 49248667-6b34-68b4-1b72-125153ca0cd9 [Follower]" leader= - testing.go:149: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:08.829Z [INFO] server-49248667-6b34-68b4-1b72-125153ca0cd9: entering follower state: follower="Node at 49248667-6b34-68b4-1b72-125153ca0cd9 [Follower]" leader= -testLoggerAdapter verbose: server-5d8b8708-f49b-4d49-a364-61238cdaf4ea: 2021-07-10T01:05:08.829Z [INFO] server-5d8b8708-f49b-4d49-a364-61238cdaf4ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49248667-6b34-68b4-1b72-125153ca0cd9 Address:49248667-6b34-68b4-1b72-125153ca0cd9} {Suffrage:Voter ID:server-5d8b8708-f49b-4d49-a364-61238cdaf4ea Address:5d8b8708-f49b-4d49-a364-61238cdaf4ea}]" - testing.go:149: server-5d8b8708-f49b-4d49-a364-61238cdaf4ea: 2021-07-10T01:05:08.829Z [INFO] server-5d8b8708-f49b-4d49-a364-61238cdaf4ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49248667-6b34-68b4-1b72-125153ca0cd9 Address:49248667-6b34-68b4-1b72-125153ca0cd9} {Suffrage:Voter ID:server-5d8b8708-f49b-4d49-a364-61238cdaf4ea Address:5d8b8708-f49b-4d49-a364-61238cdaf4ea}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:08.829Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:08.829Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-5d8b8708-f49b-4d49-a364-61238cdaf4ea: 2021-07-10T01:05:08.829Z [INFO] server-5d8b8708-f49b-4d49-a364-61238cdaf4ea: entering follower state: follower="Node at 5d8b8708-f49b-4d49-a364-61238cdaf4ea [Follower]" leader= - testing.go:149: server-5d8b8708-f49b-4d49-a364-61238cdaf4ea: 2021-07-10T01:05:08.829Z [INFO] server-5d8b8708-f49b-4d49-a364-61238cdaf4ea: entering follower state: follower="Node at 5d8b8708-f49b-4d49-a364-61238cdaf4ea [Follower]" leader= -testLoggerAdapter verbose: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:08.898Z [WARN] server-49248667-6b34-68b4-1b72-125153ca0cd9: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:08.898Z [WARN] server-49248667-6b34-68b4-1b72-125153ca0cd9: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:08.898Z [INFO] server-49248667-6b34-68b4-1b72-125153ca0cd9: entering candidate state: node="Node at 49248667-6b34-68b4-1b72-125153ca0cd9 [Candidate]" term=2 - testing.go:149: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:08.898Z [INFO] server-49248667-6b34-68b4-1b72-125153ca0cd9: entering candidate state: node="Node at 49248667-6b34-68b4-1b72-125153ca0cd9 [Candidate]" term=2 -testLoggerAdapter verbose: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:08.898Z [INFO] server-49248667-6b34-68b4-1b72-125153ca0cd9: election won: tally=2 - testing.go:149: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:08.898Z [INFO] server-49248667-6b34-68b4-1b72-125153ca0cd9: election won: tally=2 -testLoggerAdapter verbose: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:08.898Z [INFO] server-49248667-6b34-68b4-1b72-125153ca0cd9: entering leader state: leader="Node at 49248667-6b34-68b4-1b72-125153ca0cd9 [Leader]" - testing.go:149: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:08.898Z [INFO] server-49248667-6b34-68b4-1b72-125153ca0cd9: entering leader state: leader="Node at 49248667-6b34-68b4-1b72-125153ca0cd9 [Leader]" -testLoggerAdapter verbose: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:08.898Z [INFO] server-49248667-6b34-68b4-1b72-125153ca0cd9: added peer, starting replication: peer=server-5d8b8708-f49b-4d49-a364-61238cdaf4ea - testing.go:149: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:08.898Z [INFO] server-49248667-6b34-68b4-1b72-125153ca0cd9: added peer, starting replication: peer=server-5d8b8708-f49b-4d49-a364-61238cdaf4ea -testLoggerAdapter verbose: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:08.899Z [INFO] server-49248667-6b34-68b4-1b72-125153ca0cd9: pipelining replication: peer="{Voter server-5d8b8708-f49b-4d49-a364-61238cdaf4ea 5d8b8708-f49b-4d49-a364-61238cdaf4ea}" - testing.go:149: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:08.899Z [INFO] server-49248667-6b34-68b4-1b72-125153ca0cd9: pipelining replication: peer="{Voter server-5d8b8708-f49b-4d49-a364-61238cdaf4ea 5d8b8708-f49b-4d49-a364-61238cdaf4ea}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:09.004Z [INFO] stable state for Leader reached at 2021-07-10 01:05:08.89892045 +0000 UTC m=+14.287078440 (1 nodes), 69.635135ms from start of poll, 70.353762ms from cluster start. Timeout at 2021-07-10 01:05:09.003994117 +0000 UTC m=+14.392152087, 105.073647ms after stability - testing.go:149: cluster: 2021-07-10T01:05:09.004Z [INFO] stable state for Leader reached at 2021-07-10 01:05:08.89892045 +0000 UTC m=+14.287078440 (1 nodes), 69.635135ms from start of poll, 70.353762ms from cluster start. Timeout at 2021-07-10 01:05:09.003994117 +0000 UTC m=+14.392152087, 105.073647ms after stability -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:09.004Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-07-10T01:05:09.004Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:09.109Z [INFO] stable state for Follower reached at 2021-07-10 01:05:09.004318801 +0000 UTC m=+14.392476751 (1 nodes), 1.86µs from start of poll, 175.752073ms from cluster start. Timeout at 2021-07-10 01:05:09.109388488 +0000 UTC m=+14.497546458, 105.069707ms after stability - testing.go:149: cluster: 2021-07-10T01:05:09.109Z [INFO] stable state for Follower reached at 2021-07-10 01:05:09.004318801 +0000 UTC m=+14.392476751 (1 nodes), 1.86µs from start of poll, 175.752073ms from cluster start. Timeout at 2021-07-10 01:05:09.109388488 +0000 UTC m=+14.497546458, 105.069707ms after stability -testLoggerAdapter verbose: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:09.109Z [WARN] server-49248667-6b34-68b4-1b72-125153ca0cd9: new leader elected, stepping down - testing.go:149: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:09.109Z [WARN] server-49248667-6b34-68b4-1b72-125153ca0cd9: new leader elected, stepping down -testLoggerAdapter verbose: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:09.109Z [INFO] server-49248667-6b34-68b4-1b72-125153ca0cd9: entering follower state: follower="Node at 49248667-6b34-68b4-1b72-125153ca0cd9 [Shutdown]" leader= - testing.go:149: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:09.109Z [INFO] server-49248667-6b34-68b4-1b72-125153ca0cd9: entering follower state: follower="Node at 49248667-6b34-68b4-1b72-125153ca0cd9 [Shutdown]" leader= -testLoggerAdapter verbose: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:09.109Z [INFO] server-49248667-6b34-68b4-1b72-125153ca0cd9: aborting pipeline replication: peer="{Voter server-5d8b8708-f49b-4d49-a364-61238cdaf4ea 5d8b8708-f49b-4d49-a364-61238cdaf4ea}" - testing.go:149: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:09.109Z [INFO] server-49248667-6b34-68b4-1b72-125153ca0cd9: aborting pipeline replication: peer="{Voter server-5d8b8708-f49b-4d49-a364-61238cdaf4ea 5d8b8708-f49b-4d49-a364-61238cdaf4ea}" +testLoggerAdapter verbose: server-fa7f3bfc-d3d4-143d-a09c-a2659ecd3c48: 2022-08-12T07:30:13.443Z [INFO] server-fa7f3bfc-d3d4-143d-a09c-a2659ecd3c48: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fa7f3bfc-d3d4-143d-a09c-a2659ecd3c48 Address:fa7f3bfc-d3d4-143d-a09c-a2659ecd3c48} {Suffrage:Voter ID:server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b Address:2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b}]" + testing.go:149: server-fa7f3bfc-d3d4-143d-a09c-a2659ecd3c48: 2022-08-12T07:30:13.443Z [INFO] server-fa7f3bfc-d3d4-143d-a09c-a2659ecd3c48: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fa7f3bfc-d3d4-143d-a09c-a2659ecd3c48 Address:fa7f3bfc-d3d4-143d-a09c-a2659ecd3c48} {Suffrage:Voter ID:server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b Address:2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b}]" +testLoggerAdapter verbose: server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: 2022-08-12T07:30:13.444Z [INFO] server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fa7f3bfc-d3d4-143d-a09c-a2659ecd3c48 Address:fa7f3bfc-d3d4-143d-a09c-a2659ecd3c48} {Suffrage:Voter ID:server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b Address:2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b}]" + testing.go:149: server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: 2022-08-12T07:30:13.444Z [INFO] server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fa7f3bfc-d3d4-143d-a09c-a2659ecd3c48 Address:fa7f3bfc-d3d4-143d-a09c-a2659ecd3c48} {Suffrage:Voter ID:server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b Address:2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:13.444Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:13.444Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-fa7f3bfc-d3d4-143d-a09c-a2659ecd3c48: 2022-08-12T07:30:13.444Z [INFO] server-fa7f3bfc-d3d4-143d-a09c-a2659ecd3c48: entering follower state: follower="Node at fa7f3bfc-d3d4-143d-a09c-a2659ecd3c48 [Follower]" leader= + testing.go:149: server-fa7f3bfc-d3d4-143d-a09c-a2659ecd3c48: 2022-08-12T07:30:13.444Z [INFO] server-fa7f3bfc-d3d4-143d-a09c-a2659ecd3c48: entering follower state: follower="Node at fa7f3bfc-d3d4-143d-a09c-a2659ecd3c48 [Follower]" leader= +testLoggerAdapter verbose: server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: 2022-08-12T07:30:13.444Z [INFO] server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: entering follower state: follower="Node at 2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b [Follower]" leader= + testing.go:149: server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: 2022-08-12T07:30:13.444Z [INFO] server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: entering follower state: follower="Node at 2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b [Follower]" leader= +testLoggerAdapter verbose: server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: 2022-08-12T07:30:13.510Z [WARN] server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: 2022-08-12T07:30:13.510Z [WARN] server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: 2022-08-12T07:30:13.511Z [INFO] server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: entering candidate state: node="Node at 2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b [Candidate]" term=2 + testing.go:149: server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: 2022-08-12T07:30:13.511Z [INFO] server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: entering candidate state: node="Node at 2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b [Candidate]" term=2 +testLoggerAdapter verbose: server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: 2022-08-12T07:30:13.511Z [INFO] server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: election won: tally=2 + testing.go:149: server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: 2022-08-12T07:30:13.511Z [INFO] server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: election won: tally=2 +testLoggerAdapter verbose: server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: 2022-08-12T07:30:13.511Z [INFO] server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: entering leader state: leader="Node at 2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b [Leader]" + testing.go:149: server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: 2022-08-12T07:30:13.511Z [INFO] server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: entering leader state: leader="Node at 2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b [Leader]" +testLoggerAdapter verbose: server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: 2022-08-12T07:30:13.511Z [INFO] server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: added peer, starting replication: peer=server-fa7f3bfc-d3d4-143d-a09c-a2659ecd3c48 + testing.go:149: server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: 2022-08-12T07:30:13.511Z [INFO] server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: added peer, starting replication: peer=server-fa7f3bfc-d3d4-143d-a09c-a2659ecd3c48 +testLoggerAdapter verbose: server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: 2022-08-12T07:30:13.511Z [INFO] server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: pipelining replication: peer="{Voter server-fa7f3bfc-d3d4-143d-a09c-a2659ecd3c48 fa7f3bfc-d3d4-143d-a09c-a2659ecd3c48}" + testing.go:149: server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: 2022-08-12T07:30:13.511Z [INFO] server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: pipelining replication: peer="{Voter server-fa7f3bfc-d3d4-143d-a09c-a2659ecd3c48 fa7f3bfc-d3d4-143d-a09c-a2659ecd3c48}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:13.616Z [INFO] stable state for Leader reached at 2022-08-12 07:30:13.5115267 +0000 UTC m=+14.468213181 (1 nodes), 67.27768ms from start of poll, 67.97758ms from cluster start. Timeout at 2022-08-12 07:30:13.61659138 +0000 UTC m=+14.573277881, 105.0647ms after stability + testing.go:149: cluster: 2022-08-12T07:30:13.616Z [INFO] stable state for Leader reached at 2022-08-12 07:30:13.5115267 +0000 UTC m=+14.468213181 (1 nodes), 67.27768ms from start of poll, 67.97758ms from cluster start. Timeout at 2022-08-12 07:30:13.61659138 +0000 UTC m=+14.573277881, 105.0647ms after stability +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:13.616Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2022-08-12T07:30:13.616Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:13.722Z [INFO] stable state for Follower reached at 2022-08-12 07:30:13.61684252 +0000 UTC m=+14.573529001 (1 nodes), 2.28µs from start of poll, 173.2934ms from cluster start. Timeout at 2022-08-12 07:30:13.72192734 +0000 UTC m=+14.678613821, 105.08482ms after stability + testing.go:149: cluster: 2022-08-12T07:30:13.722Z [INFO] stable state for Follower reached at 2022-08-12 07:30:13.61684252 +0000 UTC m=+14.573529001 (1 nodes), 2.28µs from start of poll, 173.2934ms from cluster start. Timeout at 2022-08-12 07:30:13.72192734 +0000 UTC m=+14.678613821, 105.08482ms after stability +testLoggerAdapter verbose: server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: 2022-08-12T07:30:13.722Z [WARN] server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: new leader elected, stepping down + testing.go:149: server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: 2022-08-12T07:30:13.722Z [WARN] server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: new leader elected, stepping down +testLoggerAdapter verbose: server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: 2022-08-12T07:30:13.722Z [INFO] server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: entering follower state: follower="Node at 2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b [Follower]" leader= + testing.go:149: server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: 2022-08-12T07:30:13.722Z [INFO] server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: entering follower state: follower="Node at 2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b [Follower]" leader= +testLoggerAdapter verbose: server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: 2022-08-12T07:30:13.722Z [INFO] server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: aborting pipeline replication: peer="{Voter server-fa7f3bfc-d3d4-143d-a09c-a2659ecd3c48 fa7f3bfc-d3d4-143d-a09c-a2659ecd3c48}" + testing.go:149: server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: 2022-08-12T07:30:13.722Z [INFO] server-2e0525cd-3c9d-2e53-dc6a-faa84bfdb89b: aborting pipeline replication: peer="{Voter server-fa7f3bfc-d3d4-143d-a09c-a2659ecd3c48 fa7f3bfc-d3d4-143d-a09c-a2659ecd3c48}" --- PASS: TestRaft_VerifyLeader_Fail (0.28s) === RUN TestRaft_VerifyLeader_PartialConnect -testLoggerAdapter verbose: server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0: 2021-07-10T01:05:09.111Z [INFO] server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0 Address:0296557a-4dbe-3b4d-b2d5-4c199b97d6d0} {Suffrage:Voter ID:server-463bc25b-73bf-1f88-9ff5-b53a151968a8 Address:463bc25b-73bf-1f88-9ff5-b53a151968a8} {Suffrage:Voter ID:server-a21e9604-f93b-7718-8a1d-7c0f031126d4 Address:a21e9604-f93b-7718-8a1d-7c0f031126d4}]" - testing.go:149: server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0: 2021-07-10T01:05:09.111Z [INFO] server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0 Address:0296557a-4dbe-3b4d-b2d5-4c199b97d6d0} {Suffrage:Voter ID:server-463bc25b-73bf-1f88-9ff5-b53a151968a8 Address:463bc25b-73bf-1f88-9ff5-b53a151968a8} {Suffrage:Voter ID:server-a21e9604-f93b-7718-8a1d-7c0f031126d4 Address:a21e9604-f93b-7718-8a1d-7c0f031126d4}]" -testLoggerAdapter verbose: server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0: 2021-07-10T01:05:09.111Z [INFO] server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0: entering follower state: follower="Node at 0296557a-4dbe-3b4d-b2d5-4c199b97d6d0 [Follower]" leader= - testing.go:149: server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0: 2021-07-10T01:05:09.111Z [INFO] server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0: entering follower state: follower="Node at 0296557a-4dbe-3b4d-b2d5-4c199b97d6d0 [Follower]" leader= -testLoggerAdapter verbose: server-463bc25b-73bf-1f88-9ff5-b53a151968a8: 2021-07-10T01:05:09.111Z [INFO] server-463bc25b-73bf-1f88-9ff5-b53a151968a8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0 Address:0296557a-4dbe-3b4d-b2d5-4c199b97d6d0} {Suffrage:Voter ID:server-463bc25b-73bf-1f88-9ff5-b53a151968a8 Address:463bc25b-73bf-1f88-9ff5-b53a151968a8} {Suffrage:Voter ID:server-a21e9604-f93b-7718-8a1d-7c0f031126d4 Address:a21e9604-f93b-7718-8a1d-7c0f031126d4}]" - testing.go:149: server-463bc25b-73bf-1f88-9ff5-b53a151968a8: 2021-07-10T01:05:09.111Z [INFO] server-463bc25b-73bf-1f88-9ff5-b53a151968a8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0 Address:0296557a-4dbe-3b4d-b2d5-4c199b97d6d0} {Suffrage:Voter ID:server-463bc25b-73bf-1f88-9ff5-b53a151968a8 Address:463bc25b-73bf-1f88-9ff5-b53a151968a8} {Suffrage:Voter ID:server-a21e9604-f93b-7718-8a1d-7c0f031126d4 Address:a21e9604-f93b-7718-8a1d-7c0f031126d4}]" -testLoggerAdapter verbose: server-463bc25b-73bf-1f88-9ff5-b53a151968a8: 2021-07-10T01:05:09.111Z [INFO] server-463bc25b-73bf-1f88-9ff5-b53a151968a8: entering follower state: follower="Node at 463bc25b-73bf-1f88-9ff5-b53a151968a8 [Follower]" leader= - testing.go:149: server-463bc25b-73bf-1f88-9ff5-b53a151968a8: 2021-07-10T01:05:09.111Z [INFO] server-463bc25b-73bf-1f88-9ff5-b53a151968a8: entering follower state: follower="Node at 463bc25b-73bf-1f88-9ff5-b53a151968a8 [Follower]" leader= -testLoggerAdapter verbose: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.112Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0 Address:0296557a-4dbe-3b4d-b2d5-4c199b97d6d0} {Suffrage:Voter ID:server-463bc25b-73bf-1f88-9ff5-b53a151968a8 Address:463bc25b-73bf-1f88-9ff5-b53a151968a8} {Suffrage:Voter ID:server-a21e9604-f93b-7718-8a1d-7c0f031126d4 Address:a21e9604-f93b-7718-8a1d-7c0f031126d4}]" - testing.go:149: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.112Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0 Address:0296557a-4dbe-3b4d-b2d5-4c199b97d6d0} {Suffrage:Voter ID:server-463bc25b-73bf-1f88-9ff5-b53a151968a8 Address:463bc25b-73bf-1f88-9ff5-b53a151968a8} {Suffrage:Voter ID:server-a21e9604-f93b-7718-8a1d-7c0f031126d4 Address:a21e9604-f93b-7718-8a1d-7c0f031126d4}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:09.112Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:09.112Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.112Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: entering follower state: follower="Node at a21e9604-f93b-7718-8a1d-7c0f031126d4 [Follower]" leader= - testing.go:149: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.112Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: entering follower state: follower="Node at a21e9604-f93b-7718-8a1d-7c0f031126d4 [Follower]" leader= -testLoggerAdapter verbose: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.164Z [WARN] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.164Z [WARN] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.164Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: entering candidate state: node="Node at a21e9604-f93b-7718-8a1d-7c0f031126d4 [Candidate]" term=2 - testing.go:149: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.164Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: entering candidate state: node="Node at a21e9604-f93b-7718-8a1d-7c0f031126d4 [Candidate]" term=2 -testLoggerAdapter verbose: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.164Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: election won: tally=2 - testing.go:149: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.164Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: election won: tally=2 -testLoggerAdapter verbose: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.164Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: entering leader state: leader="Node at a21e9604-f93b-7718-8a1d-7c0f031126d4 [Leader]" - testing.go:149: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.164Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: entering leader state: leader="Node at a21e9604-f93b-7718-8a1d-7c0f031126d4 [Leader]" -testLoggerAdapter verbose: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.164Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: added peer, starting replication: peer=server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0 - testing.go:149: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.164Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: added peer, starting replication: peer=server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0 -testLoggerAdapter verbose: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.164Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: added peer, starting replication: peer=server-463bc25b-73bf-1f88-9ff5-b53a151968a8 - testing.go:149: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.164Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: added peer, starting replication: peer=server-463bc25b-73bf-1f88-9ff5-b53a151968a8 -testLoggerAdapter verbose: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.164Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: pipelining replication: peer="{Voter server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0 0296557a-4dbe-3b4d-b2d5-4c199b97d6d0}" - testing.go:149: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.164Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: pipelining replication: peer="{Voter server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0 0296557a-4dbe-3b4d-b2d5-4c199b97d6d0}" -testLoggerAdapter verbose: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.164Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: pipelining replication: peer="{Voter server-463bc25b-73bf-1f88-9ff5-b53a151968a8 463bc25b-73bf-1f88-9ff5-b53a151968a8}" - testing.go:149: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.164Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: pipelining replication: peer="{Voter server-463bc25b-73bf-1f88-9ff5-b53a151968a8 463bc25b-73bf-1f88-9ff5-b53a151968a8}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:09.269Z [INFO] stable state for Leader reached at 2021-07-10 01:05:09.164669171 +0000 UTC m=+14.552827181 (1 nodes), 52.480873ms from start of poll, 53.532344ms from cluster start. Timeout at 2021-07-10 01:05:09.269750999 +0000 UTC m=+14.657908969, 105.081788ms after stability - testing.go:149: cluster: 2021-07-10T01:05:09.269Z [INFO] stable state for Leader reached at 2021-07-10 01:05:09.164669171 +0000 UTC m=+14.552827181 (1 nodes), 52.480873ms from start of poll, 53.532344ms from cluster start. Timeout at 2021-07-10 01:05:09.269750999 +0000 UTC m=+14.657908969, 105.081788ms after stability -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:09.275Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-07-10T01:05:09.275Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:09.380Z [INFO] stable state for Follower reached at 2021-07-10 01:05:09.275163356 +0000 UTC m=+14.663321306 (2 nodes), 2.86µs from start of poll, 164.026469ms from cluster start. Timeout at 2021-07-10 01:05:09.380229664 +0000 UTC m=+14.768387634, 105.066328ms after stability - testing.go:149: cluster: 2021-07-10T01:05:09.380Z [INFO] stable state for Follower reached at 2021-07-10 01:05:09.275163356 +0000 UTC m=+14.663321306 (2 nodes), 2.86µs from start of poll, 164.026469ms from cluster start. Timeout at 2021-07-10 01:05:09.380229664 +0000 UTC m=+14.768387634, 105.066328ms after stability - raft_test.go:1647: [INFO] Disconnecting Node at 0296557a-4dbe-3b4d-b2d5-4c199b97d6d0 [Follower] -testLoggerAdapter verbose: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.380Z [ERROR] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: failed to heartbeat to: peer=0296557a-4dbe-3b4d-b2d5-4c199b97d6d0 error="failed to connect to peer: 0296557a-4dbe-3b4d-b2d5-4c199b97d6d0" - testing.go:149: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.380Z [ERROR] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: failed to heartbeat to: peer=0296557a-4dbe-3b4d-b2d5-4c199b97d6d0 error="failed to connect to peer: 0296557a-4dbe-3b4d-b2d5-4c199b97d6d0" -testLoggerAdapter verbose: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.380Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: aborting pipeline replication: peer="{Voter server-463bc25b-73bf-1f88-9ff5-b53a151968a8 463bc25b-73bf-1f88-9ff5-b53a151968a8}" - testing.go:149: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.380Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: aborting pipeline replication: peer="{Voter server-463bc25b-73bf-1f88-9ff5-b53a151968a8 463bc25b-73bf-1f88-9ff5-b53a151968a8}" -testLoggerAdapter verbose: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.380Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: aborting pipeline replication: peer="{Voter server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0 0296557a-4dbe-3b4d-b2d5-4c199b97d6d0}" - testing.go:149: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.380Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: aborting pipeline replication: peer="{Voter server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0 0296557a-4dbe-3b4d-b2d5-4c199b97d6d0}" ---- PASS: TestRaft_VerifyLeader_PartialConnect (0.27s) +testLoggerAdapter verbose: server-851ffecf-5899-5c53-f503-0c3ac4b01e11: 2022-08-12T07:30:13.723Z [INFO] server-851ffecf-5899-5c53-f503-0c3ac4b01e11: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-851ffecf-5899-5c53-f503-0c3ac4b01e11 Address:851ffecf-5899-5c53-f503-0c3ac4b01e11} {Suffrage:Voter ID:server-1bf2513c-320c-7c82-bd29-5102d2babaa3 Address:1bf2513c-320c-7c82-bd29-5102d2babaa3} {Suffrage:Voter ID:server-37633145-577f-1e00-7ef9-8d01888e907a Address:37633145-577f-1e00-7ef9-8d01888e907a}]" + testing.go:149: server-851ffecf-5899-5c53-f503-0c3ac4b01e11: 2022-08-12T07:30:13.723Z [INFO] server-851ffecf-5899-5c53-f503-0c3ac4b01e11: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-851ffecf-5899-5c53-f503-0c3ac4b01e11 Address:851ffecf-5899-5c53-f503-0c3ac4b01e11} {Suffrage:Voter ID:server-1bf2513c-320c-7c82-bd29-5102d2babaa3 Address:1bf2513c-320c-7c82-bd29-5102d2babaa3} {Suffrage:Voter ID:server-37633145-577f-1e00-7ef9-8d01888e907a Address:37633145-577f-1e00-7ef9-8d01888e907a}]" +testLoggerAdapter verbose: server-1bf2513c-320c-7c82-bd29-5102d2babaa3: 2022-08-12T07:30:13.724Z [INFO] server-1bf2513c-320c-7c82-bd29-5102d2babaa3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-851ffecf-5899-5c53-f503-0c3ac4b01e11 Address:851ffecf-5899-5c53-f503-0c3ac4b01e11} {Suffrage:Voter ID:server-1bf2513c-320c-7c82-bd29-5102d2babaa3 Address:1bf2513c-320c-7c82-bd29-5102d2babaa3} {Suffrage:Voter ID:server-37633145-577f-1e00-7ef9-8d01888e907a Address:37633145-577f-1e00-7ef9-8d01888e907a}]" + testing.go:149: server-1bf2513c-320c-7c82-bd29-5102d2babaa3: 2022-08-12T07:30:13.724Z [INFO] server-1bf2513c-320c-7c82-bd29-5102d2babaa3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-851ffecf-5899-5c53-f503-0c3ac4b01e11 Address:851ffecf-5899-5c53-f503-0c3ac4b01e11} {Suffrage:Voter ID:server-1bf2513c-320c-7c82-bd29-5102d2babaa3 Address:1bf2513c-320c-7c82-bd29-5102d2babaa3} {Suffrage:Voter ID:server-37633145-577f-1e00-7ef9-8d01888e907a Address:37633145-577f-1e00-7ef9-8d01888e907a}]" +testLoggerAdapter verbose: server-37633145-577f-1e00-7ef9-8d01888e907a: 2022-08-12T07:30:13.724Z [INFO] server-37633145-577f-1e00-7ef9-8d01888e907a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-851ffecf-5899-5c53-f503-0c3ac4b01e11 Address:851ffecf-5899-5c53-f503-0c3ac4b01e11} {Suffrage:Voter ID:server-1bf2513c-320c-7c82-bd29-5102d2babaa3 Address:1bf2513c-320c-7c82-bd29-5102d2babaa3} {Suffrage:Voter ID:server-37633145-577f-1e00-7ef9-8d01888e907a Address:37633145-577f-1e00-7ef9-8d01888e907a}]" + testing.go:149: server-37633145-577f-1e00-7ef9-8d01888e907a: 2022-08-12T07:30:13.724Z [INFO] server-37633145-577f-1e00-7ef9-8d01888e907a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-851ffecf-5899-5c53-f503-0c3ac4b01e11 Address:851ffecf-5899-5c53-f503-0c3ac4b01e11} {Suffrage:Voter ID:server-1bf2513c-320c-7c82-bd29-5102d2babaa3 Address:1bf2513c-320c-7c82-bd29-5102d2babaa3} {Suffrage:Voter ID:server-37633145-577f-1e00-7ef9-8d01888e907a Address:37633145-577f-1e00-7ef9-8d01888e907a}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:13.724Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:13.724Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-851ffecf-5899-5c53-f503-0c3ac4b01e11: 2022-08-12T07:30:13.724Z [INFO] server-851ffecf-5899-5c53-f503-0c3ac4b01e11: entering follower state: follower="Node at 851ffecf-5899-5c53-f503-0c3ac4b01e11 [Follower]" leader= + testing.go:149: server-851ffecf-5899-5c53-f503-0c3ac4b01e11: 2022-08-12T07:30:13.724Z [INFO] server-851ffecf-5899-5c53-f503-0c3ac4b01e11: entering follower state: follower="Node at 851ffecf-5899-5c53-f503-0c3ac4b01e11 [Follower]" leader= +testLoggerAdapter verbose: server-1bf2513c-320c-7c82-bd29-5102d2babaa3: 2022-08-12T07:30:13.724Z [INFO] server-1bf2513c-320c-7c82-bd29-5102d2babaa3: entering follower state: follower="Node at 1bf2513c-320c-7c82-bd29-5102d2babaa3 [Follower]" leader= + testing.go:149: server-1bf2513c-320c-7c82-bd29-5102d2babaa3: 2022-08-12T07:30:13.724Z [INFO] server-1bf2513c-320c-7c82-bd29-5102d2babaa3: entering follower state: follower="Node at 1bf2513c-320c-7c82-bd29-5102d2babaa3 [Follower]" leader= +testLoggerAdapter verbose: server-37633145-577f-1e00-7ef9-8d01888e907a: 2022-08-12T07:30:13.724Z [INFO] server-37633145-577f-1e00-7ef9-8d01888e907a: entering follower state: follower="Node at 37633145-577f-1e00-7ef9-8d01888e907a [Follower]" leader= + testing.go:149: server-37633145-577f-1e00-7ef9-8d01888e907a: 2022-08-12T07:30:13.724Z [INFO] server-37633145-577f-1e00-7ef9-8d01888e907a: entering follower state: follower="Node at 37633145-577f-1e00-7ef9-8d01888e907a [Follower]" leader= +testLoggerAdapter verbose: server-37633145-577f-1e00-7ef9-8d01888e907a: 2022-08-12T07:30:13.782Z [WARN] server-37633145-577f-1e00-7ef9-8d01888e907a: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-37633145-577f-1e00-7ef9-8d01888e907a: 2022-08-12T07:30:13.782Z [WARN] server-37633145-577f-1e00-7ef9-8d01888e907a: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-37633145-577f-1e00-7ef9-8d01888e907a: 2022-08-12T07:30:13.782Z [INFO] server-37633145-577f-1e00-7ef9-8d01888e907a: entering candidate state: node="Node at 37633145-577f-1e00-7ef9-8d01888e907a [Candidate]" term=2 + testing.go:149: server-37633145-577f-1e00-7ef9-8d01888e907a: 2022-08-12T07:30:13.782Z [INFO] server-37633145-577f-1e00-7ef9-8d01888e907a: entering candidate state: node="Node at 37633145-577f-1e00-7ef9-8d01888e907a [Candidate]" term=2 +testLoggerAdapter verbose: server-37633145-577f-1e00-7ef9-8d01888e907a: 2022-08-12T07:30:13.783Z [INFO] server-37633145-577f-1e00-7ef9-8d01888e907a: election won: tally=2 + testing.go:149: server-37633145-577f-1e00-7ef9-8d01888e907a: 2022-08-12T07:30:13.783Z [INFO] server-37633145-577f-1e00-7ef9-8d01888e907a: election won: tally=2 +testLoggerAdapter verbose: server-37633145-577f-1e00-7ef9-8d01888e907a: 2022-08-12T07:30:13.783Z [INFO] server-37633145-577f-1e00-7ef9-8d01888e907a: entering leader state: leader="Node at 37633145-577f-1e00-7ef9-8d01888e907a [Leader]" + testing.go:149: server-37633145-577f-1e00-7ef9-8d01888e907a: 2022-08-12T07:30:13.783Z [INFO] server-37633145-577f-1e00-7ef9-8d01888e907a: entering leader state: leader="Node at 37633145-577f-1e00-7ef9-8d01888e907a [Leader]" +testLoggerAdapter verbose: server-37633145-577f-1e00-7ef9-8d01888e907a: 2022-08-12T07:30:13.783Z [INFO] server-37633145-577f-1e00-7ef9-8d01888e907a: added peer, starting replication: peer=server-851ffecf-5899-5c53-f503-0c3ac4b01e11 + testing.go:149: server-37633145-577f-1e00-7ef9-8d01888e907a: 2022-08-12T07:30:13.783Z [INFO] server-37633145-577f-1e00-7ef9-8d01888e907a: added peer, starting replication: peer=server-851ffecf-5899-5c53-f503-0c3ac4b01e11 +testLoggerAdapter verbose: server-37633145-577f-1e00-7ef9-8d01888e907a: 2022-08-12T07:30:13.783Z [INFO] server-37633145-577f-1e00-7ef9-8d01888e907a: added peer, starting replication: peer=server-1bf2513c-320c-7c82-bd29-5102d2babaa3 + testing.go:149: server-37633145-577f-1e00-7ef9-8d01888e907a: 2022-08-12T07:30:13.783Z [INFO] server-37633145-577f-1e00-7ef9-8d01888e907a: added peer, starting replication: peer=server-1bf2513c-320c-7c82-bd29-5102d2babaa3 +testLoggerAdapter verbose: server-37633145-577f-1e00-7ef9-8d01888e907a: 2022-08-12T07:30:13.783Z [INFO] server-37633145-577f-1e00-7ef9-8d01888e907a: pipelining replication: peer="{Voter server-1bf2513c-320c-7c82-bd29-5102d2babaa3 1bf2513c-320c-7c82-bd29-5102d2babaa3}" + testing.go:149: server-37633145-577f-1e00-7ef9-8d01888e907a: 2022-08-12T07:30:13.783Z [INFO] server-37633145-577f-1e00-7ef9-8d01888e907a: pipelining replication: peer="{Voter server-1bf2513c-320c-7c82-bd29-5102d2babaa3 1bf2513c-320c-7c82-bd29-5102d2babaa3}" +testLoggerAdapter verbose: server-37633145-577f-1e00-7ef9-8d01888e907a: 2022-08-12T07:30:13.783Z [INFO] server-37633145-577f-1e00-7ef9-8d01888e907a: pipelining replication: peer="{Voter server-851ffecf-5899-5c53-f503-0c3ac4b01e11 851ffecf-5899-5c53-f503-0c3ac4b01e11}" + testing.go:149: server-37633145-577f-1e00-7ef9-8d01888e907a: 2022-08-12T07:30:13.783Z [INFO] server-37633145-577f-1e00-7ef9-8d01888e907a: pipelining replication: peer="{Voter server-851ffecf-5899-5c53-f503-0c3ac4b01e11 851ffecf-5899-5c53-f503-0c3ac4b01e11}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:13.888Z [INFO] stable state for Leader reached at 2022-08-12 07:30:13.78341272 +0000 UTC m=+14.740099201 (1 nodes), 58.86814ms from start of poll, 59.9085ms from cluster start. Timeout at 2022-08-12 07:30:13.88848982 +0000 UTC m=+14.845176321, 105.07712ms after stability + testing.go:149: cluster: 2022-08-12T07:30:13.888Z [INFO] stable state for Leader reached at 2022-08-12 07:30:13.78341272 +0000 UTC m=+14.740099201 (1 nodes), 58.86814ms from start of poll, 59.9085ms from cluster start. Timeout at 2022-08-12 07:30:13.88848982 +0000 UTC m=+14.845176321, 105.07712ms after stability +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:13.893Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2022-08-12T07:30:13.893Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:13.999Z [INFO] stable state for Follower reached at 2022-08-12 07:30:13.8939575 +0000 UTC m=+14.850643961 (2 nodes), 2.74µs from start of poll, 170.45326ms from cluster start. Timeout at 2022-08-12 07:30:13.99903276 +0000 UTC m=+14.955719261, 105.0753ms after stability + testing.go:149: cluster: 2022-08-12T07:30:13.999Z [INFO] stable state for Follower reached at 2022-08-12 07:30:13.8939575 +0000 UTC m=+14.850643961 (2 nodes), 2.74µs from start of poll, 170.45326ms from cluster start. Timeout at 2022-08-12 07:30:13.99903276 +0000 UTC m=+14.955719261, 105.0753ms after stability + raft_test.go:1647: [INFO] Disconnecting Node at 851ffecf-5899-5c53-f503-0c3ac4b01e11 [Follower] +testLoggerAdapter verbose: server-37633145-577f-1e00-7ef9-8d01888e907a: 2022-08-12T07:30:13.999Z [ERROR] server-37633145-577f-1e00-7ef9-8d01888e907a: failed to heartbeat to: peer=851ffecf-5899-5c53-f503-0c3ac4b01e11 error="failed to connect to peer: 851ffecf-5899-5c53-f503-0c3ac4b01e11" + testing.go:149: server-37633145-577f-1e00-7ef9-8d01888e907a: 2022-08-12T07:30:13.999Z [ERROR] server-37633145-577f-1e00-7ef9-8d01888e907a: failed to heartbeat to: peer=851ffecf-5899-5c53-f503-0c3ac4b01e11 error="failed to connect to peer: 851ffecf-5899-5c53-f503-0c3ac4b01e11" +testLoggerAdapter verbose: server-37633145-577f-1e00-7ef9-8d01888e907a: 2022-08-12T07:30:13.999Z [INFO] server-37633145-577f-1e00-7ef9-8d01888e907a: aborting pipeline replication: peer="{Voter server-1bf2513c-320c-7c82-bd29-5102d2babaa3 1bf2513c-320c-7c82-bd29-5102d2babaa3}" + testing.go:149: server-37633145-577f-1e00-7ef9-8d01888e907a: 2022-08-12T07:30:13.999Z [INFO] server-37633145-577f-1e00-7ef9-8d01888e907a: aborting pipeline replication: peer="{Voter server-1bf2513c-320c-7c82-bd29-5102d2babaa3 1bf2513c-320c-7c82-bd29-5102d2babaa3}" +testLoggerAdapter verbose: server-37633145-577f-1e00-7ef9-8d01888e907a: 2022-08-12T07:30:13.999Z [INFO] server-37633145-577f-1e00-7ef9-8d01888e907a: aborting pipeline replication: peer="{Voter server-851ffecf-5899-5c53-f503-0c3ac4b01e11 851ffecf-5899-5c53-f503-0c3ac4b01e11}" + testing.go:149: server-37633145-577f-1e00-7ef9-8d01888e907a: 2022-08-12T07:30:13.999Z [INFO] server-37633145-577f-1e00-7ef9-8d01888e907a: aborting pipeline replication: peer="{Voter server-851ffecf-5899-5c53-f503-0c3ac4b01e11 851ffecf-5899-5c53-f503-0c3ac4b01e11}" +--- PASS: TestRaft_VerifyLeader_PartialConnect (0.28s) === RUN TestRaft_StartAsLeader -testLoggerAdapter verbose: server-7b8f9ddf-b93c-2668-7308-bbed0d0d90d5: 2021-07-10T01:05:09.382Z [INFO] server-7b8f9ddf-b93c-2668-7308-bbed0d0d90d5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b8f9ddf-b93c-2668-7308-bbed0d0d90d5 Address:7b8f9ddf-b93c-2668-7308-bbed0d0d90d5}]" - testing.go:149: server-7b8f9ddf-b93c-2668-7308-bbed0d0d90d5: 2021-07-10T01:05:09.382Z [INFO] server-7b8f9ddf-b93c-2668-7308-bbed0d0d90d5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b8f9ddf-b93c-2668-7308-bbed0d0d90d5 Address:7b8f9ddf-b93c-2668-7308-bbed0d0d90d5}]" -testLoggerAdapter verbose: server-7b8f9ddf-b93c-2668-7308-bbed0d0d90d5: 2021-07-10T01:05:09.382Z [INFO] server-7b8f9ddf-b93c-2668-7308-bbed0d0d90d5: entering leader state: leader="Node at 7b8f9ddf-b93c-2668-7308-bbed0d0d90d5 [Leader]" - testing.go:149: server-7b8f9ddf-b93c-2668-7308-bbed0d0d90d5: 2021-07-10T01:05:09.382Z [INFO] server-7b8f9ddf-b93c-2668-7308-bbed0d0d90d5: entering leader state: leader="Node at 7b8f9ddf-b93c-2668-7308-bbed0d0d90d5 [Leader]" +testLoggerAdapter verbose: server-52e62f06-1d0a-7771-2034-47890702e6bc: 2022-08-12T07:30:14.000Z [INFO] server-52e62f06-1d0a-7771-2034-47890702e6bc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-52e62f06-1d0a-7771-2034-47890702e6bc Address:52e62f06-1d0a-7771-2034-47890702e6bc}]" + testing.go:149: server-52e62f06-1d0a-7771-2034-47890702e6bc: 2022-08-12T07:30:14.000Z [INFO] server-52e62f06-1d0a-7771-2034-47890702e6bc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-52e62f06-1d0a-7771-2034-47890702e6bc Address:52e62f06-1d0a-7771-2034-47890702e6bc}]" +testLoggerAdapter verbose: server-52e62f06-1d0a-7771-2034-47890702e6bc: 2022-08-12T07:30:14.000Z [INFO] server-52e62f06-1d0a-7771-2034-47890702e6bc: entering leader state: leader="Node at 52e62f06-1d0a-7771-2034-47890702e6bc [Leader]" + testing.go:149: server-52e62f06-1d0a-7771-2034-47890702e6bc: 2022-08-12T07:30:14.000Z [INFO] server-52e62f06-1d0a-7771-2034-47890702e6bc: entering leader state: leader="Node at 52e62f06-1d0a-7771-2034-47890702e6bc [Leader]" --- PASS: TestRaft_StartAsLeader (0.00s) === RUN TestRaft_NotifyCh -testLoggerAdapter verbose: server-f8896627-a809-21db-6ab5-129eacf80e8e: 2021-07-10T01:05:09.382Z [INFO] server-f8896627-a809-21db-6ab5-129eacf80e8e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f8896627-a809-21db-6ab5-129eacf80e8e Address:f8896627-a809-21db-6ab5-129eacf80e8e}]" - testing.go:149: server-f8896627-a809-21db-6ab5-129eacf80e8e: 2021-07-10T01:05:09.382Z [INFO] server-f8896627-a809-21db-6ab5-129eacf80e8e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f8896627-a809-21db-6ab5-129eacf80e8e Address:f8896627-a809-21db-6ab5-129eacf80e8e}]" -testLoggerAdapter verbose: server-f8896627-a809-21db-6ab5-129eacf80e8e: 2021-07-10T01:05:09.383Z [INFO] server-f8896627-a809-21db-6ab5-129eacf80e8e: entering follower state: follower="Node at f8896627-a809-21db-6ab5-129eacf80e8e [Follower]" leader= - testing.go:149: server-f8896627-a809-21db-6ab5-129eacf80e8e: 2021-07-10T01:05:09.383Z [INFO] server-f8896627-a809-21db-6ab5-129eacf80e8e: entering follower state: follower="Node at f8896627-a809-21db-6ab5-129eacf80e8e [Follower]" leader= -testLoggerAdapter verbose: server-f8896627-a809-21db-6ab5-129eacf80e8e: 2021-07-10T01:05:09.481Z [WARN] server-f8896627-a809-21db-6ab5-129eacf80e8e: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-f8896627-a809-21db-6ab5-129eacf80e8e: 2021-07-10T01:05:09.481Z [WARN] server-f8896627-a809-21db-6ab5-129eacf80e8e: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-f8896627-a809-21db-6ab5-129eacf80e8e: 2021-07-10T01:05:09.481Z [INFO] server-f8896627-a809-21db-6ab5-129eacf80e8e: entering candidate state: node="Node at f8896627-a809-21db-6ab5-129eacf80e8e [Candidate]" term=2 - testing.go:149: server-f8896627-a809-21db-6ab5-129eacf80e8e: 2021-07-10T01:05:09.481Z [INFO] server-f8896627-a809-21db-6ab5-129eacf80e8e: entering candidate state: node="Node at f8896627-a809-21db-6ab5-129eacf80e8e [Candidate]" term=2 -testLoggerAdapter verbose: server-f8896627-a809-21db-6ab5-129eacf80e8e: 2021-07-10T01:05:09.481Z [INFO] server-f8896627-a809-21db-6ab5-129eacf80e8e: election won: tally=1 - testing.go:149: server-f8896627-a809-21db-6ab5-129eacf80e8e: 2021-07-10T01:05:09.481Z [INFO] server-f8896627-a809-21db-6ab5-129eacf80e8e: election won: tally=1 -testLoggerAdapter verbose: server-f8896627-a809-21db-6ab5-129eacf80e8e: 2021-07-10T01:05:09.481Z [INFO] server-f8896627-a809-21db-6ab5-129eacf80e8e: entering leader state: leader="Node at f8896627-a809-21db-6ab5-129eacf80e8e [Leader]" - testing.go:149: server-f8896627-a809-21db-6ab5-129eacf80e8e: 2021-07-10T01:05:09.481Z [INFO] server-f8896627-a809-21db-6ab5-129eacf80e8e: entering leader state: leader="Node at f8896627-a809-21db-6ab5-129eacf80e8e [Leader]" ---- PASS: TestRaft_NotifyCh (0.10s) +testLoggerAdapter verbose: server-482259e8-b4a5-ff10-b290-4ae0cdffb41c: 2022-08-12T07:30:14.001Z [INFO] server-482259e8-b4a5-ff10-b290-4ae0cdffb41c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-482259e8-b4a5-ff10-b290-4ae0cdffb41c Address:482259e8-b4a5-ff10-b290-4ae0cdffb41c}]" + testing.go:149: server-482259e8-b4a5-ff10-b290-4ae0cdffb41c: 2022-08-12T07:30:14.001Z [INFO] server-482259e8-b4a5-ff10-b290-4ae0cdffb41c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-482259e8-b4a5-ff10-b290-4ae0cdffb41c Address:482259e8-b4a5-ff10-b290-4ae0cdffb41c}]" +testLoggerAdapter verbose: server-482259e8-b4a5-ff10-b290-4ae0cdffb41c: 2022-08-12T07:30:14.001Z [INFO] server-482259e8-b4a5-ff10-b290-4ae0cdffb41c: entering follower state: follower="Node at 482259e8-b4a5-ff10-b290-4ae0cdffb41c [Follower]" leader= + testing.go:149: server-482259e8-b4a5-ff10-b290-4ae0cdffb41c: 2022-08-12T07:30:14.001Z [INFO] server-482259e8-b4a5-ff10-b290-4ae0cdffb41c: entering follower state: follower="Node at 482259e8-b4a5-ff10-b290-4ae0cdffb41c [Follower]" leader= +testLoggerAdapter verbose: server-482259e8-b4a5-ff10-b290-4ae0cdffb41c: 2022-08-12T07:30:14.088Z [WARN] server-482259e8-b4a5-ff10-b290-4ae0cdffb41c: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-482259e8-b4a5-ff10-b290-4ae0cdffb41c: 2022-08-12T07:30:14.088Z [WARN] server-482259e8-b4a5-ff10-b290-4ae0cdffb41c: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-482259e8-b4a5-ff10-b290-4ae0cdffb41c: 2022-08-12T07:30:14.088Z [INFO] server-482259e8-b4a5-ff10-b290-4ae0cdffb41c: entering candidate state: node="Node at 482259e8-b4a5-ff10-b290-4ae0cdffb41c [Candidate]" term=2 + testing.go:149: server-482259e8-b4a5-ff10-b290-4ae0cdffb41c: 2022-08-12T07:30:14.088Z [INFO] server-482259e8-b4a5-ff10-b290-4ae0cdffb41c: entering candidate state: node="Node at 482259e8-b4a5-ff10-b290-4ae0cdffb41c [Candidate]" term=2 +testLoggerAdapter verbose: server-482259e8-b4a5-ff10-b290-4ae0cdffb41c: 2022-08-12T07:30:14.088Z [INFO] server-482259e8-b4a5-ff10-b290-4ae0cdffb41c: election won: tally=1 + testing.go:149: server-482259e8-b4a5-ff10-b290-4ae0cdffb41c: 2022-08-12T07:30:14.088Z [INFO] server-482259e8-b4a5-ff10-b290-4ae0cdffb41c: election won: tally=1 +testLoggerAdapter verbose: server-482259e8-b4a5-ff10-b290-4ae0cdffb41c: 2022-08-12T07:30:14.088Z [INFO] server-482259e8-b4a5-ff10-b290-4ae0cdffb41c: entering leader state: leader="Node at 482259e8-b4a5-ff10-b290-4ae0cdffb41c [Leader]" + testing.go:149: server-482259e8-b4a5-ff10-b290-4ae0cdffb41c: 2022-08-12T07:30:14.088Z [INFO] server-482259e8-b4a5-ff10-b290-4ae0cdffb41c: entering leader state: leader="Node at 482259e8-b4a5-ff10-b290-4ae0cdffb41c [Leader]" +--- PASS: TestRaft_NotifyCh (0.09s) === RUN TestRaft_Voting -testLoggerAdapter verbose: server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d: 2021-07-10T01:05:09.483Z [INFO] server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d Address:6316c959-88d4-e67a-a5ff-cd3975fa8d2d} {Suffrage:Voter ID:server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e Address:0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e} {Suffrage:Voter ID:server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460 Address:77a1df6c-8e57-68f7-a83d-e1f8b2f67460}]" - testing.go:149: server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d: 2021-07-10T01:05:09.483Z [INFO] server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d Address:6316c959-88d4-e67a-a5ff-cd3975fa8d2d} {Suffrage:Voter ID:server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e Address:0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e} {Suffrage:Voter ID:server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460 Address:77a1df6c-8e57-68f7-a83d-e1f8b2f67460}]" -testLoggerAdapter verbose: server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d: 2021-07-10T01:05:09.483Z [INFO] server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d: entering follower state: follower="Node at 6316c959-88d4-e67a-a5ff-cd3975fa8d2d [Follower]" leader= - testing.go:149: server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d: 2021-07-10T01:05:09.483Z [INFO] server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d: entering follower state: follower="Node at 6316c959-88d4-e67a-a5ff-cd3975fa8d2d [Follower]" leader= -testLoggerAdapter verbose: server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e: 2021-07-10T01:05:09.483Z [INFO] server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d Address:6316c959-88d4-e67a-a5ff-cd3975fa8d2d} {Suffrage:Voter ID:server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e Address:0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e} {Suffrage:Voter ID:server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460 Address:77a1df6c-8e57-68f7-a83d-e1f8b2f67460}]" - testing.go:149: server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e: 2021-07-10T01:05:09.483Z [INFO] server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d Address:6316c959-88d4-e67a-a5ff-cd3975fa8d2d} {Suffrage:Voter ID:server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e Address:0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e} {Suffrage:Voter ID:server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460 Address:77a1df6c-8e57-68f7-a83d-e1f8b2f67460}]" -testLoggerAdapter verbose: server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e: 2021-07-10T01:05:09.483Z [INFO] server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e: entering follower state: follower="Node at 0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e [Follower]" leader= - testing.go:149: server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e: 2021-07-10T01:05:09.483Z [INFO] server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e: entering follower state: follower="Node at 0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e [Follower]" leader= -testLoggerAdapter verbose: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.483Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d Address:6316c959-88d4-e67a-a5ff-cd3975fa8d2d} {Suffrage:Voter ID:server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e Address:0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e} {Suffrage:Voter ID:server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460 Address:77a1df6c-8e57-68f7-a83d-e1f8b2f67460}]" - testing.go:149: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.483Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d Address:6316c959-88d4-e67a-a5ff-cd3975fa8d2d} {Suffrage:Voter ID:server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e Address:0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e} {Suffrage:Voter ID:server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460 Address:77a1df6c-8e57-68f7-a83d-e1f8b2f67460}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:09.483Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.483Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: entering follower state: follower="Node at 77a1df6c-8e57-68f7-a83d-e1f8b2f67460 [Follower]" leader= - testing.go:149: cluster: 2021-07-10T01:05:09.483Z [INFO] starting stability test: raft-state=Follower - testing.go:149: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.483Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: entering follower state: follower="Node at 77a1df6c-8e57-68f7-a83d-e1f8b2f67460 [Follower]" leader= -testLoggerAdapter verbose: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.544Z [WARN] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.544Z [WARN] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.544Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: entering candidate state: node="Node at 77a1df6c-8e57-68f7-a83d-e1f8b2f67460 [Candidate]" term=2 - testing.go:149: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.544Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: entering candidate state: node="Node at 77a1df6c-8e57-68f7-a83d-e1f8b2f67460 [Candidate]" term=2 -testLoggerAdapter verbose: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.544Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: election won: tally=2 - testing.go:149: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.544Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: election won: tally=2 -testLoggerAdapter verbose: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.545Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: entering leader state: leader="Node at 77a1df6c-8e57-68f7-a83d-e1f8b2f67460 [Leader]" - testing.go:149: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.545Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: entering leader state: leader="Node at 77a1df6c-8e57-68f7-a83d-e1f8b2f67460 [Leader]" -testLoggerAdapter verbose: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.545Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: added peer, starting replication: peer=server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d - testing.go:149: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.545Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: added peer, starting replication: peer=server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d -testLoggerAdapter verbose: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.545Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: added peer, starting replication: peer=server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e - testing.go:149: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.545Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: added peer, starting replication: peer=server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e -testLoggerAdapter verbose: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.545Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: pipelining replication: peer="{Voter server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d 6316c959-88d4-e67a-a5ff-cd3975fa8d2d}" - testing.go:149: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.545Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: pipelining replication: peer="{Voter server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d 6316c959-88d4-e67a-a5ff-cd3975fa8d2d}" -testLoggerAdapter verbose: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.545Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: pipelining replication: peer="{Voter server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e 0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e}" - testing.go:149: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.545Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: pipelining replication: peer="{Voter server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e 0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:09.650Z [INFO] stable state for Follower reached at 2021-07-10 01:05:09.545132642 +0000 UTC m=+14.933290612 (2 nodes), 61.118424ms from start of poll, 62.175515ms from cluster start. Timeout at 2021-07-10 01:05:09.65019925 +0000 UTC m=+15.038357240, 105.066628ms after stability - testing.go:149: cluster: 2021-07-10T01:05:09.650Z [INFO] stable state for Follower reached at 2021-07-10 01:05:09.545132642 +0000 UTC m=+14.933290612 (2 nodes), 61.118424ms from start of poll, 62.175515ms from cluster start. Timeout at 2021-07-10 01:05:09.65019925 +0000 UTC m=+15.038357240, 105.066628ms after stability -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:09.650Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:09.650Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:09.755Z [INFO] stable state for Leader reached at 2021-07-10 01:05:09.650469493 +0000 UTC m=+15.038627463 (1 nodes), 2.12µs from start of poll, 167.512366ms from cluster start. Timeout at 2021-07-10 01:05:09.75548068 +0000 UTC m=+15.143638650, 105.011187ms after stability - testing.go:149: cluster: 2021-07-10T01:05:09.755Z [INFO] stable state for Leader reached at 2021-07-10 01:05:09.650469493 +0000 UTC m=+15.038627463 (1 nodes), 2.12µs from start of poll, 167.512366ms from cluster start. Timeout at 2021-07-10 01:05:09.75548068 +0000 UTC m=+15.143638650, 105.011187ms after stability -testLoggerAdapter verbose: server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e: 2021-07-10T01:05:09.755Z [WARN] server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e: rejecting vote request since we have a leader: from=6316c959-88d4-e67a-a5ff-cd3975fa8d2d leader=77a1df6c-8e57-68f7-a83d-e1f8b2f67460 - testing.go:149: server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e: 2021-07-10T01:05:09.755Z [WARN] server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e: rejecting vote request since we have a leader: from=6316c959-88d4-e67a-a5ff-cd3975fa8d2d leader=77a1df6c-8e57-68f7-a83d-e1f8b2f67460 -testLoggerAdapter verbose: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.755Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: aborting pipeline replication: peer="{Voter server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e 0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e}" - testing.go:149: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.755Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: aborting pipeline replication: peer="{Voter server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e 0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e}" -testLoggerAdapter verbose: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.756Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: aborting pipeline replication: peer="{Voter server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d 6316c959-88d4-e67a-a5ff-cd3975fa8d2d}" - testing.go:149: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.756Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: aborting pipeline replication: peer="{Voter server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d 6316c959-88d4-e67a-a5ff-cd3975fa8d2d}" +testLoggerAdapter verbose: server-87b4c0b1-8047-f726-514e-ea71c1fe3754: 2022-08-12T07:30:14.090Z [INFO] server-87b4c0b1-8047-f726-514e-ea71c1fe3754: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-87b4c0b1-8047-f726-514e-ea71c1fe3754 Address:87b4c0b1-8047-f726-514e-ea71c1fe3754} {Suffrage:Voter ID:server-68472621-51a6-48bc-27c7-5ee5594c2763 Address:68472621-51a6-48bc-27c7-5ee5594c2763} {Suffrage:Voter ID:server-111f9aca-e4b4-24c7-c5e6-27ddfcb568b6 Address:111f9aca-e4b4-24c7-c5e6-27ddfcb568b6}]" + testing.go:149: server-87b4c0b1-8047-f726-514e-ea71c1fe3754: 2022-08-12T07:30:14.090Z [INFO] server-87b4c0b1-8047-f726-514e-ea71c1fe3754: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-87b4c0b1-8047-f726-514e-ea71c1fe3754 Address:87b4c0b1-8047-f726-514e-ea71c1fe3754} {Suffrage:Voter ID:server-68472621-51a6-48bc-27c7-5ee5594c2763 Address:68472621-51a6-48bc-27c7-5ee5594c2763} {Suffrage:Voter ID:server-111f9aca-e4b4-24c7-c5e6-27ddfcb568b6 Address:111f9aca-e4b4-24c7-c5e6-27ddfcb568b6}]" +testLoggerAdapter verbose: server-68472621-51a6-48bc-27c7-5ee5594c2763: 2022-08-12T07:30:14.090Z [INFO] server-68472621-51a6-48bc-27c7-5ee5594c2763: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-87b4c0b1-8047-f726-514e-ea71c1fe3754 Address:87b4c0b1-8047-f726-514e-ea71c1fe3754} {Suffrage:Voter ID:server-68472621-51a6-48bc-27c7-5ee5594c2763 Address:68472621-51a6-48bc-27c7-5ee5594c2763} {Suffrage:Voter ID:server-111f9aca-e4b4-24c7-c5e6-27ddfcb568b6 Address:111f9aca-e4b4-24c7-c5e6-27ddfcb568b6}]" + testing.go:149: server-68472621-51a6-48bc-27c7-5ee5594c2763: 2022-08-12T07:30:14.090Z [INFO] server-68472621-51a6-48bc-27c7-5ee5594c2763: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-87b4c0b1-8047-f726-514e-ea71c1fe3754 Address:87b4c0b1-8047-f726-514e-ea71c1fe3754} {Suffrage:Voter ID:server-68472621-51a6-48bc-27c7-5ee5594c2763 Address:68472621-51a6-48bc-27c7-5ee5594c2763} {Suffrage:Voter ID:server-111f9aca-e4b4-24c7-c5e6-27ddfcb568b6 Address:111f9aca-e4b4-24c7-c5e6-27ddfcb568b6}]" +testLoggerAdapter verbose: server-111f9aca-e4b4-24c7-c5e6-27ddfcb568b6: 2022-08-12T07:30:14.090Z [INFO] server-111f9aca-e4b4-24c7-c5e6-27ddfcb568b6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-87b4c0b1-8047-f726-514e-ea71c1fe3754 Address:87b4c0b1-8047-f726-514e-ea71c1fe3754} {Suffrage:Voter ID:server-68472621-51a6-48bc-27c7-5ee5594c2763 Address:68472621-51a6-48bc-27c7-5ee5594c2763} {Suffrage:Voter ID:server-111f9aca-e4b4-24c7-c5e6-27ddfcb568b6 Address:111f9aca-e4b4-24c7-c5e6-27ddfcb568b6}]" + testing.go:149: server-111f9aca-e4b4-24c7-c5e6-27ddfcb568b6: 2022-08-12T07:30:14.090Z [INFO] server-111f9aca-e4b4-24c7-c5e6-27ddfcb568b6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-87b4c0b1-8047-f726-514e-ea71c1fe3754 Address:87b4c0b1-8047-f726-514e-ea71c1fe3754} {Suffrage:Voter ID:server-68472621-51a6-48bc-27c7-5ee5594c2763 Address:68472621-51a6-48bc-27c7-5ee5594c2763} {Suffrage:Voter ID:server-111f9aca-e4b4-24c7-c5e6-27ddfcb568b6 Address:111f9aca-e4b4-24c7-c5e6-27ddfcb568b6}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:14.090Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2022-08-12T07:30:14.090Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-87b4c0b1-8047-f726-514e-ea71c1fe3754: 2022-08-12T07:30:14.090Z [INFO] server-87b4c0b1-8047-f726-514e-ea71c1fe3754: entering follower state: follower="Node at 87b4c0b1-8047-f726-514e-ea71c1fe3754 [Follower]" leader= + testing.go:149: server-87b4c0b1-8047-f726-514e-ea71c1fe3754: 2022-08-12T07:30:14.090Z [INFO] server-87b4c0b1-8047-f726-514e-ea71c1fe3754: entering follower state: follower="Node at 87b4c0b1-8047-f726-514e-ea71c1fe3754 [Follower]" leader= +testLoggerAdapter verbose: server-68472621-51a6-48bc-27c7-5ee5594c2763: 2022-08-12T07:30:14.091Z [INFO] server-68472621-51a6-48bc-27c7-5ee5594c2763: entering follower state: follower="Node at 68472621-51a6-48bc-27c7-5ee5594c2763 [Follower]" leader= + testing.go:149: server-68472621-51a6-48bc-27c7-5ee5594c2763: 2022-08-12T07:30:14.091Z [INFO] server-68472621-51a6-48bc-27c7-5ee5594c2763: entering follower state: follower="Node at 68472621-51a6-48bc-27c7-5ee5594c2763 [Follower]" leader= +testLoggerAdapter verbose: server-111f9aca-e4b4-24c7-c5e6-27ddfcb568b6: 2022-08-12T07:30:14.091Z [INFO] server-111f9aca-e4b4-24c7-c5e6-27ddfcb568b6: entering follower state: follower="Node at 111f9aca-e4b4-24c7-c5e6-27ddfcb568b6 [Follower]" leader= + testing.go:149: server-111f9aca-e4b4-24c7-c5e6-27ddfcb568b6: 2022-08-12T07:30:14.091Z [INFO] server-111f9aca-e4b4-24c7-c5e6-27ddfcb568b6: entering follower state: follower="Node at 111f9aca-e4b4-24c7-c5e6-27ddfcb568b6 [Follower]" leader= +testLoggerAdapter verbose: server-68472621-51a6-48bc-27c7-5ee5594c2763: 2022-08-12T07:30:14.151Z [WARN] server-68472621-51a6-48bc-27c7-5ee5594c2763: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-68472621-51a6-48bc-27c7-5ee5594c2763: 2022-08-12T07:30:14.151Z [WARN] server-68472621-51a6-48bc-27c7-5ee5594c2763: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-68472621-51a6-48bc-27c7-5ee5594c2763: 2022-08-12T07:30:14.151Z [INFO] server-68472621-51a6-48bc-27c7-5ee5594c2763: entering candidate state: node="Node at 68472621-51a6-48bc-27c7-5ee5594c2763 [Candidate]" term=2 + testing.go:149: server-68472621-51a6-48bc-27c7-5ee5594c2763: 2022-08-12T07:30:14.151Z [INFO] server-68472621-51a6-48bc-27c7-5ee5594c2763: entering candidate state: node="Node at 68472621-51a6-48bc-27c7-5ee5594c2763 [Candidate]" term=2 +testLoggerAdapter verbose: server-68472621-51a6-48bc-27c7-5ee5594c2763: 2022-08-12T07:30:14.151Z [INFO] server-68472621-51a6-48bc-27c7-5ee5594c2763: election won: tally=2 + testing.go:149: server-68472621-51a6-48bc-27c7-5ee5594c2763: 2022-08-12T07:30:14.151Z [INFO] server-68472621-51a6-48bc-27c7-5ee5594c2763: election won: tally=2 +testLoggerAdapter verbose: server-68472621-51a6-48bc-27c7-5ee5594c2763: 2022-08-12T07:30:14.151Z [INFO] server-68472621-51a6-48bc-27c7-5ee5594c2763: entering leader state: leader="Node at 68472621-51a6-48bc-27c7-5ee5594c2763 [Leader]" + testing.go:149: server-68472621-51a6-48bc-27c7-5ee5594c2763: 2022-08-12T07:30:14.151Z [INFO] server-68472621-51a6-48bc-27c7-5ee5594c2763: entering leader state: leader="Node at 68472621-51a6-48bc-27c7-5ee5594c2763 [Leader]" +testLoggerAdapter verbose: server-68472621-51a6-48bc-27c7-5ee5594c2763: 2022-08-12T07:30:14.152Z [INFO] server-68472621-51a6-48bc-27c7-5ee5594c2763: added peer, starting replication: peer=server-87b4c0b1-8047-f726-514e-ea71c1fe3754 + testing.go:149: server-68472621-51a6-48bc-27c7-5ee5594c2763: 2022-08-12T07:30:14.152Z [INFO] server-68472621-51a6-48bc-27c7-5ee5594c2763: added peer, starting replication: peer=server-87b4c0b1-8047-f726-514e-ea71c1fe3754 +testLoggerAdapter verbose: server-68472621-51a6-48bc-27c7-5ee5594c2763: 2022-08-12T07:30:14.152Z [INFO] server-68472621-51a6-48bc-27c7-5ee5594c2763: added peer, starting replication: peer=server-111f9aca-e4b4-24c7-c5e6-27ddfcb568b6 + testing.go:149: server-68472621-51a6-48bc-27c7-5ee5594c2763: 2022-08-12T07:30:14.152Z [INFO] server-68472621-51a6-48bc-27c7-5ee5594c2763: added peer, starting replication: peer=server-111f9aca-e4b4-24c7-c5e6-27ddfcb568b6 +testLoggerAdapter verbose: server-68472621-51a6-48bc-27c7-5ee5594c2763: 2022-08-12T07:30:14.152Z [INFO] server-68472621-51a6-48bc-27c7-5ee5594c2763: pipelining replication: peer="{Voter server-87b4c0b1-8047-f726-514e-ea71c1fe3754 87b4c0b1-8047-f726-514e-ea71c1fe3754}" + testing.go:149: server-68472621-51a6-48bc-27c7-5ee5594c2763: 2022-08-12T07:30:14.152Z [INFO] server-68472621-51a6-48bc-27c7-5ee5594c2763: pipelining replication: peer="{Voter server-87b4c0b1-8047-f726-514e-ea71c1fe3754 87b4c0b1-8047-f726-514e-ea71c1fe3754}" +testLoggerAdapter verbose: server-68472621-51a6-48bc-27c7-5ee5594c2763: 2022-08-12T07:30:14.152Z [INFO] server-68472621-51a6-48bc-27c7-5ee5594c2763: pipelining replication: peer="{Voter server-111f9aca-e4b4-24c7-c5e6-27ddfcb568b6 111f9aca-e4b4-24c7-c5e6-27ddfcb568b6}" + testing.go:149: server-68472621-51a6-48bc-27c7-5ee5594c2763: 2022-08-12T07:30:14.152Z [INFO] server-68472621-51a6-48bc-27c7-5ee5594c2763: pipelining replication: peer="{Voter server-111f9aca-e4b4-24c7-c5e6-27ddfcb568b6 111f9aca-e4b4-24c7-c5e6-27ddfcb568b6}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:14.257Z [INFO] stable state for Follower reached at 2022-08-12 07:30:14.15206658 +0000 UTC m=+15.108753061 (2 nodes), 61.18014ms from start of poll, 62.0897ms from cluster start. Timeout at 2022-08-12 07:30:14.25710318 +0000 UTC m=+15.213789661, 105.0366ms after stability + testing.go:149: cluster: 2022-08-12T07:30:14.257Z [INFO] stable state for Follower reached at 2022-08-12 07:30:14.15206658 +0000 UTC m=+15.108753061 (2 nodes), 61.18014ms from start of poll, 62.0897ms from cluster start. Timeout at 2022-08-12 07:30:14.25710318 +0000 UTC m=+15.213789661, 105.0366ms after stability +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:14.257Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:14.257Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:14.362Z [INFO] stable state for Leader reached at 2022-08-12 07:30:14.25737188 +0000 UTC m=+15.214058361 (1 nodes), 1.96µs from start of poll, 167.395ms from cluster start. Timeout at 2022-08-12 07:30:14.36244922 +0000 UTC m=+15.319135721, 105.07736ms after stability + testing.go:149: cluster: 2022-08-12T07:30:14.362Z [INFO] stable state for Leader reached at 2022-08-12 07:30:14.25737188 +0000 UTC m=+15.214058361 (1 nodes), 1.96µs from start of poll, 167.395ms from cluster start. Timeout at 2022-08-12 07:30:14.36244922 +0000 UTC m=+15.319135721, 105.07736ms after stability +testLoggerAdapter verbose: server-111f9aca-e4b4-24c7-c5e6-27ddfcb568b6: 2022-08-12T07:30:14.362Z [WARN] server-111f9aca-e4b4-24c7-c5e6-27ddfcb568b6: rejecting vote request since we have a leader: from=87b4c0b1-8047-f726-514e-ea71c1fe3754 leader=68472621-51a6-48bc-27c7-5ee5594c2763 + testing.go:149: server-111f9aca-e4b4-24c7-c5e6-27ddfcb568b6: 2022-08-12T07:30:14.362Z [WARN] server-111f9aca-e4b4-24c7-c5e6-27ddfcb568b6: rejecting vote request since we have a leader: from=87b4c0b1-8047-f726-514e-ea71c1fe3754 leader=68472621-51a6-48bc-27c7-5ee5594c2763 +testLoggerAdapter verbose: server-68472621-51a6-48bc-27c7-5ee5594c2763: 2022-08-12T07:30:14.362Z [INFO] server-68472621-51a6-48bc-27c7-5ee5594c2763: aborting pipeline replication: peer="{Voter server-87b4c0b1-8047-f726-514e-ea71c1fe3754 87b4c0b1-8047-f726-514e-ea71c1fe3754}" + testing.go:149: server-68472621-51a6-48bc-27c7-5ee5594c2763: 2022-08-12T07:30:14.362Z [INFO] server-68472621-51a6-48bc-27c7-5ee5594c2763: aborting pipeline replication: peer="{Voter server-87b4c0b1-8047-f726-514e-ea71c1fe3754 87b4c0b1-8047-f726-514e-ea71c1fe3754}" +testLoggerAdapter verbose: server-68472621-51a6-48bc-27c7-5ee5594c2763: 2022-08-12T07:30:14.362Z [INFO] server-68472621-51a6-48bc-27c7-5ee5594c2763: aborting pipeline replication: peer="{Voter server-111f9aca-e4b4-24c7-c5e6-27ddfcb568b6 111f9aca-e4b4-24c7-c5e6-27ddfcb568b6}" + testing.go:149: server-68472621-51a6-48bc-27c7-5ee5594c2763: 2022-08-12T07:30:14.362Z [INFO] server-68472621-51a6-48bc-27c7-5ee5594c2763: aborting pipeline replication: peer="{Voter server-111f9aca-e4b4-24c7-c5e6-27ddfcb568b6 111f9aca-e4b4-24c7-c5e6-27ddfcb568b6}" --- PASS: TestRaft_Voting (0.27s) === RUN TestRaft_ProtocolVersion_RejectRPC -testLoggerAdapter verbose: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.757Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31 Address:5f6ce37a-9c3a-c951-74b3-d324f3ed3e31} {Suffrage:Voter ID:server-1000e46c-dd80-5b3d-08c7-02868a2f867c Address:1000e46c-dd80-5b3d-08c7-02868a2f867c} {Suffrage:Voter ID:server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0 Address:3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0}]" - testing.go:149: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.757Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31 Address:5f6ce37a-9c3a-c951-74b3-d324f3ed3e31} {Suffrage:Voter ID:server-1000e46c-dd80-5b3d-08c7-02868a2f867c Address:1000e46c-dd80-5b3d-08c7-02868a2f867c} {Suffrage:Voter ID:server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0 Address:3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0}]" -testLoggerAdapter verbose: server-1000e46c-dd80-5b3d-08c7-02868a2f867c: 2021-07-10T01:05:09.757Z [INFO] server-1000e46c-dd80-5b3d-08c7-02868a2f867c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31 Address:5f6ce37a-9c3a-c951-74b3-d324f3ed3e31} {Suffrage:Voter ID:server-1000e46c-dd80-5b3d-08c7-02868a2f867c Address:1000e46c-dd80-5b3d-08c7-02868a2f867c} {Suffrage:Voter ID:server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0 Address:3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0}]" - testing.go:149: server-1000e46c-dd80-5b3d-08c7-02868a2f867c: 2021-07-10T01:05:09.757Z [INFO] server-1000e46c-dd80-5b3d-08c7-02868a2f867c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31 Address:5f6ce37a-9c3a-c951-74b3-d324f3ed3e31} {Suffrage:Voter ID:server-1000e46c-dd80-5b3d-08c7-02868a2f867c Address:1000e46c-dd80-5b3d-08c7-02868a2f867c} {Suffrage:Voter ID:server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0 Address:3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0}]" -testLoggerAdapter verbose: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.757Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: entering follower state: follower="Node at 5f6ce37a-9c3a-c951-74b3-d324f3ed3e31 [Follower]" leader= - testing.go:149: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.757Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: entering follower state: follower="Node at 5f6ce37a-9c3a-c951-74b3-d324f3ed3e31 [Follower]" leader= -testLoggerAdapter verbose: server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0: 2021-07-10T01:05:09.757Z [INFO] server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31 Address:5f6ce37a-9c3a-c951-74b3-d324f3ed3e31} {Suffrage:Voter ID:server-1000e46c-dd80-5b3d-08c7-02868a2f867c Address:1000e46c-dd80-5b3d-08c7-02868a2f867c} {Suffrage:Voter ID:server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0 Address:3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0}]" -testLoggerAdapter verbose: server-1000e46c-dd80-5b3d-08c7-02868a2f867c: 2021-07-10T01:05:09.757Z [INFO] server-1000e46c-dd80-5b3d-08c7-02868a2f867c: entering follower state: follower="Node at 1000e46c-dd80-5b3d-08c7-02868a2f867c [Follower]" leader= - testing.go:149: server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0: 2021-07-10T01:05:09.757Z [INFO] server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31 Address:5f6ce37a-9c3a-c951-74b3-d324f3ed3e31} {Suffrage:Voter ID:server-1000e46c-dd80-5b3d-08c7-02868a2f867c Address:1000e46c-dd80-5b3d-08c7-02868a2f867c} {Suffrage:Voter ID:server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0 Address:3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:09.757Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-07-10T01:05:09.757Z [INFO] starting stability test: raft-state=Follower - testing.go:149: server-1000e46c-dd80-5b3d-08c7-02868a2f867c: 2021-07-10T01:05:09.757Z [INFO] server-1000e46c-dd80-5b3d-08c7-02868a2f867c: entering follower state: follower="Node at 1000e46c-dd80-5b3d-08c7-02868a2f867c [Follower]" leader= -testLoggerAdapter verbose: server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0: 2021-07-10T01:05:09.757Z [INFO] server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0: entering follower state: follower="Node at 3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0 [Follower]" leader= - testing.go:149: server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0: 2021-07-10T01:05:09.757Z [INFO] server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0: entering follower state: follower="Node at 3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0 [Follower]" leader= -testLoggerAdapter verbose: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.843Z [WARN] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.843Z [WARN] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.843Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: entering candidate state: node="Node at 5f6ce37a-9c3a-c951-74b3-d324f3ed3e31 [Candidate]" term=2 - testing.go:149: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.843Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: entering candidate state: node="Node at 5f6ce37a-9c3a-c951-74b3-d324f3ed3e31 [Candidate]" term=2 -testLoggerAdapter verbose: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.844Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: election won: tally=2 - testing.go:149: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.844Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: election won: tally=2 -testLoggerAdapter verbose: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.844Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: entering leader state: leader="Node at 5f6ce37a-9c3a-c951-74b3-d324f3ed3e31 [Leader]" - testing.go:149: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.844Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: entering leader state: leader="Node at 5f6ce37a-9c3a-c951-74b3-d324f3ed3e31 [Leader]" -testLoggerAdapter verbose: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.844Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: added peer, starting replication: peer=server-1000e46c-dd80-5b3d-08c7-02868a2f867c - testing.go:149: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.844Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: added peer, starting replication: peer=server-1000e46c-dd80-5b3d-08c7-02868a2f867c -testLoggerAdapter verbose: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.844Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: added peer, starting replication: peer=server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0 - testing.go:149: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.844Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: added peer, starting replication: peer=server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0 -testLoggerAdapter verbose: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.844Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: pipelining replication: peer="{Voter server-1000e46c-dd80-5b3d-08c7-02868a2f867c 1000e46c-dd80-5b3d-08c7-02868a2f867c}" - testing.go:149: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.844Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: pipelining replication: peer="{Voter server-1000e46c-dd80-5b3d-08c7-02868a2f867c 1000e46c-dd80-5b3d-08c7-02868a2f867c}" -testLoggerAdapter verbose: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.844Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: pipelining replication: peer="{Voter server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0 3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0}" - testing.go:149: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.844Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: pipelining replication: peer="{Voter server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0 3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:09.949Z [INFO] stable state for Follower reached at 2021-07-10 01:05:09.844171775 +0000 UTC m=+15.232329865 (2 nodes), 86.197449ms from start of poll, 87.025518ms from cluster start. Timeout at 2021-07-10 01:05:09.949229882 +0000 UTC m=+15.337387852, 105.057987ms after stability - testing.go:149: cluster: 2021-07-10T01:05:09.949Z [INFO] stable state for Follower reached at 2021-07-10 01:05:09.844171775 +0000 UTC m=+15.232329865 (2 nodes), 86.197449ms from start of poll, 87.025518ms from cluster start. Timeout at 2021-07-10 01:05:09.949229882 +0000 UTC m=+15.337387852, 105.057987ms after stability -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:09.949Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:09.949Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:10.054Z [INFO] stable state for Leader reached at 2021-07-10 01:05:09.949474185 +0000 UTC m=+15.337632155 (1 nodes), 2.06µs from start of poll, 192.327808ms from cluster start. Timeout at 2021-07-10 01:05:10.054542133 +0000 UTC m=+15.442700103, 105.067948ms after stability - testing.go:149: cluster: 2021-07-10T01:05:10.054Z [INFO] stable state for Leader reached at 2021-07-10 01:05:09.949474185 +0000 UTC m=+15.337632155 (1 nodes), 2.06µs from start of poll, 192.327808ms from cluster start. Timeout at 2021-07-10 01:05:10.054542133 +0000 UTC m=+15.442700103, 105.067948ms after stability -testLoggerAdapter verbose: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:10.054Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: aborting pipeline replication: peer="{Voter server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0 3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0}" - testing.go:149: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:10.054Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: aborting pipeline replication: peer="{Voter server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0 3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0}" -testLoggerAdapter verbose: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:10.054Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: aborting pipeline replication: peer="{Voter server-1000e46c-dd80-5b3d-08c7-02868a2f867c 1000e46c-dd80-5b3d-08c7-02868a2f867c}" - testing.go:149: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:10.054Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: aborting pipeline replication: peer="{Voter server-1000e46c-dd80-5b3d-08c7-02868a2f867c 1000e46c-dd80-5b3d-08c7-02868a2f867c}" ---- PASS: TestRaft_ProtocolVersion_RejectRPC (0.30s) +testLoggerAdapter verbose: server-257cc924-eaff-115a-4f5f-764233b4a43e: 2022-08-12T07:30:14.364Z [INFO] server-257cc924-eaff-115a-4f5f-764233b4a43e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-257cc924-eaff-115a-4f5f-764233b4a43e Address:257cc924-eaff-115a-4f5f-764233b4a43e} {Suffrage:Voter ID:server-dbe4da3a-aee5-e3ca-c701-278247e46176 Address:dbe4da3a-aee5-e3ca-c701-278247e46176} {Suffrage:Voter ID:server-53755df8-30f9-afcf-f5bb-d66b02dcf997 Address:53755df8-30f9-afcf-f5bb-d66b02dcf997}]" + testing.go:149: server-257cc924-eaff-115a-4f5f-764233b4a43e: 2022-08-12T07:30:14.364Z [INFO] server-257cc924-eaff-115a-4f5f-764233b4a43e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-257cc924-eaff-115a-4f5f-764233b4a43e Address:257cc924-eaff-115a-4f5f-764233b4a43e} {Suffrage:Voter ID:server-dbe4da3a-aee5-e3ca-c701-278247e46176 Address:dbe4da3a-aee5-e3ca-c701-278247e46176} {Suffrage:Voter ID:server-53755df8-30f9-afcf-f5bb-d66b02dcf997 Address:53755df8-30f9-afcf-f5bb-d66b02dcf997}]" +testLoggerAdapter verbose: server-dbe4da3a-aee5-e3ca-c701-278247e46176: 2022-08-12T07:30:14.364Z [INFO] server-dbe4da3a-aee5-e3ca-c701-278247e46176: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-257cc924-eaff-115a-4f5f-764233b4a43e Address:257cc924-eaff-115a-4f5f-764233b4a43e} {Suffrage:Voter ID:server-dbe4da3a-aee5-e3ca-c701-278247e46176 Address:dbe4da3a-aee5-e3ca-c701-278247e46176} {Suffrage:Voter ID:server-53755df8-30f9-afcf-f5bb-d66b02dcf997 Address:53755df8-30f9-afcf-f5bb-d66b02dcf997}]" + testing.go:149: server-dbe4da3a-aee5-e3ca-c701-278247e46176: 2022-08-12T07:30:14.364Z [INFO] server-dbe4da3a-aee5-e3ca-c701-278247e46176: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-257cc924-eaff-115a-4f5f-764233b4a43e Address:257cc924-eaff-115a-4f5f-764233b4a43e} {Suffrage:Voter ID:server-dbe4da3a-aee5-e3ca-c701-278247e46176 Address:dbe4da3a-aee5-e3ca-c701-278247e46176} {Suffrage:Voter ID:server-53755df8-30f9-afcf-f5bb-d66b02dcf997 Address:53755df8-30f9-afcf-f5bb-d66b02dcf997}]" +testLoggerAdapter verbose: server-257cc924-eaff-115a-4f5f-764233b4a43e: 2022-08-12T07:30:14.364Z [INFO] server-257cc924-eaff-115a-4f5f-764233b4a43e: entering follower state: follower="Node at 257cc924-eaff-115a-4f5f-764233b4a43e [Follower]" leader= + testing.go:149: server-257cc924-eaff-115a-4f5f-764233b4a43e: 2022-08-12T07:30:14.364Z [INFO] server-257cc924-eaff-115a-4f5f-764233b4a43e: entering follower state: follower="Node at 257cc924-eaff-115a-4f5f-764233b4a43e [Follower]" leader= +testLoggerAdapter verbose: server-53755df8-30f9-afcf-f5bb-d66b02dcf997: 2022-08-12T07:30:14.364Z [INFO] server-53755df8-30f9-afcf-f5bb-d66b02dcf997: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-257cc924-eaff-115a-4f5f-764233b4a43e Address:257cc924-eaff-115a-4f5f-764233b4a43e} {Suffrage:Voter ID:server-dbe4da3a-aee5-e3ca-c701-278247e46176 Address:dbe4da3a-aee5-e3ca-c701-278247e46176} {Suffrage:Voter ID:server-53755df8-30f9-afcf-f5bb-d66b02dcf997 Address:53755df8-30f9-afcf-f5bb-d66b02dcf997}]" +testLoggerAdapter verbose: server-dbe4da3a-aee5-e3ca-c701-278247e46176: 2022-08-12T07:30:14.364Z [INFO] server-dbe4da3a-aee5-e3ca-c701-278247e46176: entering follower state: follower="Node at dbe4da3a-aee5-e3ca-c701-278247e46176 [Follower]" leader= + testing.go:149: server-53755df8-30f9-afcf-f5bb-d66b02dcf997: 2022-08-12T07:30:14.364Z [INFO] server-53755df8-30f9-afcf-f5bb-d66b02dcf997: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-257cc924-eaff-115a-4f5f-764233b4a43e Address:257cc924-eaff-115a-4f5f-764233b4a43e} {Suffrage:Voter ID:server-dbe4da3a-aee5-e3ca-c701-278247e46176 Address:dbe4da3a-aee5-e3ca-c701-278247e46176} {Suffrage:Voter ID:server-53755df8-30f9-afcf-f5bb-d66b02dcf997 Address:53755df8-30f9-afcf-f5bb-d66b02dcf997}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:14.365Z [INFO] starting stability test: raft-state=Follower + testing.go:149: server-dbe4da3a-aee5-e3ca-c701-278247e46176: 2022-08-12T07:30:14.364Z [INFO] server-dbe4da3a-aee5-e3ca-c701-278247e46176: entering follower state: follower="Node at dbe4da3a-aee5-e3ca-c701-278247e46176 [Follower]" leader= + testing.go:149: cluster: 2022-08-12T07:30:14.365Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-53755df8-30f9-afcf-f5bb-d66b02dcf997: 2022-08-12T07:30:14.365Z [INFO] server-53755df8-30f9-afcf-f5bb-d66b02dcf997: entering follower state: follower="Node at 53755df8-30f9-afcf-f5bb-d66b02dcf997 [Follower]" leader= + testing.go:149: server-53755df8-30f9-afcf-f5bb-d66b02dcf997: 2022-08-12T07:30:14.365Z [INFO] server-53755df8-30f9-afcf-f5bb-d66b02dcf997: entering follower state: follower="Node at 53755df8-30f9-afcf-f5bb-d66b02dcf997 [Follower]" leader= +testLoggerAdapter verbose: server-dbe4da3a-aee5-e3ca-c701-278247e46176: 2022-08-12T07:30:14.425Z [WARN] server-dbe4da3a-aee5-e3ca-c701-278247e46176: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-dbe4da3a-aee5-e3ca-c701-278247e46176: 2022-08-12T07:30:14.425Z [WARN] server-dbe4da3a-aee5-e3ca-c701-278247e46176: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-dbe4da3a-aee5-e3ca-c701-278247e46176: 2022-08-12T07:30:14.425Z [INFO] server-dbe4da3a-aee5-e3ca-c701-278247e46176: entering candidate state: node="Node at dbe4da3a-aee5-e3ca-c701-278247e46176 [Candidate]" term=2 + testing.go:149: server-dbe4da3a-aee5-e3ca-c701-278247e46176: 2022-08-12T07:30:14.425Z [INFO] server-dbe4da3a-aee5-e3ca-c701-278247e46176: entering candidate state: node="Node at dbe4da3a-aee5-e3ca-c701-278247e46176 [Candidate]" term=2 +testLoggerAdapter verbose: server-dbe4da3a-aee5-e3ca-c701-278247e46176: 2022-08-12T07:30:14.426Z [INFO] server-dbe4da3a-aee5-e3ca-c701-278247e46176: election won: tally=2 + testing.go:149: server-dbe4da3a-aee5-e3ca-c701-278247e46176: 2022-08-12T07:30:14.426Z [INFO] server-dbe4da3a-aee5-e3ca-c701-278247e46176: election won: tally=2 +testLoggerAdapter verbose: server-dbe4da3a-aee5-e3ca-c701-278247e46176: 2022-08-12T07:30:14.426Z [INFO] server-dbe4da3a-aee5-e3ca-c701-278247e46176: entering leader state: leader="Node at dbe4da3a-aee5-e3ca-c701-278247e46176 [Leader]" + testing.go:149: server-dbe4da3a-aee5-e3ca-c701-278247e46176: 2022-08-12T07:30:14.426Z [INFO] server-dbe4da3a-aee5-e3ca-c701-278247e46176: entering leader state: leader="Node at dbe4da3a-aee5-e3ca-c701-278247e46176 [Leader]" +testLoggerAdapter verbose: server-dbe4da3a-aee5-e3ca-c701-278247e46176: 2022-08-12T07:30:14.426Z [INFO] server-dbe4da3a-aee5-e3ca-c701-278247e46176: added peer, starting replication: peer=server-257cc924-eaff-115a-4f5f-764233b4a43e + testing.go:149: server-dbe4da3a-aee5-e3ca-c701-278247e46176: 2022-08-12T07:30:14.426Z [INFO] server-dbe4da3a-aee5-e3ca-c701-278247e46176: added peer, starting replication: peer=server-257cc924-eaff-115a-4f5f-764233b4a43e +testLoggerAdapter verbose: server-dbe4da3a-aee5-e3ca-c701-278247e46176: 2022-08-12T07:30:14.426Z [INFO] server-dbe4da3a-aee5-e3ca-c701-278247e46176: added peer, starting replication: peer=server-53755df8-30f9-afcf-f5bb-d66b02dcf997 + testing.go:149: server-dbe4da3a-aee5-e3ca-c701-278247e46176: 2022-08-12T07:30:14.426Z [INFO] server-dbe4da3a-aee5-e3ca-c701-278247e46176: added peer, starting replication: peer=server-53755df8-30f9-afcf-f5bb-d66b02dcf997 +testLoggerAdapter verbose: server-dbe4da3a-aee5-e3ca-c701-278247e46176: 2022-08-12T07:30:14.426Z [INFO] server-dbe4da3a-aee5-e3ca-c701-278247e46176: pipelining replication: peer="{Voter server-257cc924-eaff-115a-4f5f-764233b4a43e 257cc924-eaff-115a-4f5f-764233b4a43e}" + testing.go:149: server-dbe4da3a-aee5-e3ca-c701-278247e46176: 2022-08-12T07:30:14.426Z [INFO] server-dbe4da3a-aee5-e3ca-c701-278247e46176: pipelining replication: peer="{Voter server-257cc924-eaff-115a-4f5f-764233b4a43e 257cc924-eaff-115a-4f5f-764233b4a43e}" +testLoggerAdapter verbose: server-dbe4da3a-aee5-e3ca-c701-278247e46176: 2022-08-12T07:30:14.426Z [INFO] server-dbe4da3a-aee5-e3ca-c701-278247e46176: pipelining replication: peer="{Voter server-53755df8-30f9-afcf-f5bb-d66b02dcf997 53755df8-30f9-afcf-f5bb-d66b02dcf997}" + testing.go:149: server-dbe4da3a-aee5-e3ca-c701-278247e46176: 2022-08-12T07:30:14.426Z [INFO] server-dbe4da3a-aee5-e3ca-c701-278247e46176: pipelining replication: peer="{Voter server-53755df8-30f9-afcf-f5bb-d66b02dcf997 53755df8-30f9-afcf-f5bb-d66b02dcf997}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:14.531Z [INFO] stable state for Follower reached at 2022-08-12 07:30:14.42621548 +0000 UTC m=+15.382901941 (2 nodes), 61.09714ms from start of poll, 62.04076ms from cluster start. Timeout at 2022-08-12 07:30:14.5312231 +0000 UTC m=+15.487909581, 105.00764ms after stability + testing.go:149: cluster: 2022-08-12T07:30:14.531Z [INFO] stable state for Follower reached at 2022-08-12 07:30:14.42621548 +0000 UTC m=+15.382901941 (2 nodes), 61.09714ms from start of poll, 62.04076ms from cluster start. Timeout at 2022-08-12 07:30:14.5312231 +0000 UTC m=+15.487909581, 105.00764ms after stability +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:14.531Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:14.531Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:14.636Z [INFO] stable state for Leader reached at 2022-08-12 07:30:14.53147714 +0000 UTC m=+15.488163601 (1 nodes), 2.44µs from start of poll, 167.30242ms from cluster start. Timeout at 2022-08-12 07:30:14.63651516 +0000 UTC m=+15.593201641, 105.03804ms after stability + testing.go:149: cluster: 2022-08-12T07:30:14.636Z [INFO] stable state for Leader reached at 2022-08-12 07:30:14.53147714 +0000 UTC m=+15.488163601 (1 nodes), 2.44µs from start of poll, 167.30242ms from cluster start. Timeout at 2022-08-12 07:30:14.63651516 +0000 UTC m=+15.593201641, 105.03804ms after stability +testLoggerAdapter verbose: server-dbe4da3a-aee5-e3ca-c701-278247e46176: 2022-08-12T07:30:14.636Z [INFO] server-dbe4da3a-aee5-e3ca-c701-278247e46176: aborting pipeline replication: peer="{Voter server-53755df8-30f9-afcf-f5bb-d66b02dcf997 53755df8-30f9-afcf-f5bb-d66b02dcf997}" + testing.go:149: server-dbe4da3a-aee5-e3ca-c701-278247e46176: 2022-08-12T07:30:14.636Z [INFO] server-dbe4da3a-aee5-e3ca-c701-278247e46176: aborting pipeline replication: peer="{Voter server-53755df8-30f9-afcf-f5bb-d66b02dcf997 53755df8-30f9-afcf-f5bb-d66b02dcf997}" +testLoggerAdapter verbose: server-dbe4da3a-aee5-e3ca-c701-278247e46176: 2022-08-12T07:30:14.636Z [INFO] server-dbe4da3a-aee5-e3ca-c701-278247e46176: aborting pipeline replication: peer="{Voter server-257cc924-eaff-115a-4f5f-764233b4a43e 257cc924-eaff-115a-4f5f-764233b4a43e}" + testing.go:149: server-dbe4da3a-aee5-e3ca-c701-278247e46176: 2022-08-12T07:30:14.636Z [INFO] server-dbe4da3a-aee5-e3ca-c701-278247e46176: aborting pipeline replication: peer="{Voter server-257cc924-eaff-115a-4f5f-764233b4a43e 257cc924-eaff-115a-4f5f-764233b4a43e}" +--- PASS: TestRaft_ProtocolVersion_RejectRPC (0.27s) === RUN TestRaft_ProtocolVersion_Upgrade_1_2 -testLoggerAdapter verbose: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.056Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: initial configuration: index=1 servers="[{Suffrage:Voter ID:9a872e48-d9d3-d627-f6d2-461f9c91991c Address:9a872e48-d9d3-d627-f6d2-461f9c91991c} {Suffrage:Voter ID:020322e8-db03-a3eb-6c7f-aa0385507418 Address:020322e8-db03-a3eb-6c7f-aa0385507418}]" - testing.go:149: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.056Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: initial configuration: index=1 servers="[{Suffrage:Voter ID:9a872e48-d9d3-d627-f6d2-461f9c91991c Address:9a872e48-d9d3-d627-f6d2-461f9c91991c} {Suffrage:Voter ID:020322e8-db03-a3eb-6c7f-aa0385507418 Address:020322e8-db03-a3eb-6c7f-aa0385507418}]" -testLoggerAdapter verbose: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.056Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: entering follower state: follower="Node at 9a872e48-d9d3-d627-f6d2-461f9c91991c [Follower]" leader= - testing.go:149: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.056Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: entering follower state: follower="Node at 9a872e48-d9d3-d627-f6d2-461f9c91991c [Follower]" leader= -testLoggerAdapter verbose: 020322e8-db03-a3eb-6c7f-aa0385507418: 2021-07-10T01:05:10.056Z [INFO] 020322e8-db03-a3eb-6c7f-aa0385507418: initial configuration: index=1 servers="[{Suffrage:Voter ID:9a872e48-d9d3-d627-f6d2-461f9c91991c Address:9a872e48-d9d3-d627-f6d2-461f9c91991c} {Suffrage:Voter ID:020322e8-db03-a3eb-6c7f-aa0385507418 Address:020322e8-db03-a3eb-6c7f-aa0385507418}]" - testing.go:149: 020322e8-db03-a3eb-6c7f-aa0385507418: 2021-07-10T01:05:10.056Z [INFO] 020322e8-db03-a3eb-6c7f-aa0385507418: initial configuration: index=1 servers="[{Suffrage:Voter ID:9a872e48-d9d3-d627-f6d2-461f9c91991c Address:9a872e48-d9d3-d627-f6d2-461f9c91991c} {Suffrage:Voter ID:020322e8-db03-a3eb-6c7f-aa0385507418 Address:020322e8-db03-a3eb-6c7f-aa0385507418}]" -testLoggerAdapter verbose: 020322e8-db03-a3eb-6c7f-aa0385507418: 2021-07-10T01:05:10.056Z [INFO] 020322e8-db03-a3eb-6c7f-aa0385507418: entering follower state: follower="Node at 020322e8-db03-a3eb-6c7f-aa0385507418 [Follower]" leader= - testing.go:149: 020322e8-db03-a3eb-6c7f-aa0385507418: 2021-07-10T01:05:10.056Z [INFO] 020322e8-db03-a3eb-6c7f-aa0385507418: entering follower state: follower="Node at 020322e8-db03-a3eb-6c7f-aa0385507418 [Follower]" leader= -testLoggerAdapter verbose: 363d840e-a128-8f51-7ba8-2e437195bfa9: 2021-07-10T01:05:10.056Z [INFO] 363d840e-a128-8f51-7ba8-2e437195bfa9: initial configuration: index=0 servers=[] - testing.go:149: 363d840e-a128-8f51-7ba8-2e437195bfa9: 2021-07-10T01:05:10.056Z [INFO] 363d840e-a128-8f51-7ba8-2e437195bfa9: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:10.056Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: 363d840e-a128-8f51-7ba8-2e437195bfa9: 2021-07-10T01:05:10.056Z [INFO] 363d840e-a128-8f51-7ba8-2e437195bfa9: entering follower state: follower="Node at 363d840e-a128-8f51-7ba8-2e437195bfa9 [Follower]" leader= - testing.go:149: cluster: 2021-07-10T01:05:10.056Z [INFO] starting stability test: raft-state=Leader - testing.go:149: 363d840e-a128-8f51-7ba8-2e437195bfa9: 2021-07-10T01:05:10.056Z [INFO] 363d840e-a128-8f51-7ba8-2e437195bfa9: entering follower state: follower="Node at 363d840e-a128-8f51-7ba8-2e437195bfa9 [Follower]" leader= -testLoggerAdapter verbose: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.129Z [WARN] 9a872e48-d9d3-d627-f6d2-461f9c91991c: heartbeat timeout reached, starting election: last-leader= - testing.go:149: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.129Z [WARN] 9a872e48-d9d3-d627-f6d2-461f9c91991c: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.130Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: entering candidate state: node="Node at 9a872e48-d9d3-d627-f6d2-461f9c91991c [Candidate]" term=2 - testing.go:149: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.130Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: entering candidate state: node="Node at 9a872e48-d9d3-d627-f6d2-461f9c91991c [Candidate]" term=2 -testLoggerAdapter verbose: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.130Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: election won: tally=2 - testing.go:149: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.130Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: election won: tally=2 -testLoggerAdapter verbose: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.130Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: entering leader state: leader="Node at 9a872e48-d9d3-d627-f6d2-461f9c91991c [Leader]" - testing.go:149: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.130Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: entering leader state: leader="Node at 9a872e48-d9d3-d627-f6d2-461f9c91991c [Leader]" -testLoggerAdapter verbose: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.130Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: added peer, starting replication: peer=020322e8-db03-a3eb-6c7f-aa0385507418 - testing.go:149: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.130Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: added peer, starting replication: peer=020322e8-db03-a3eb-6c7f-aa0385507418 -testLoggerAdapter verbose: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.130Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: pipelining replication: peer="{Voter 020322e8-db03-a3eb-6c7f-aa0385507418 020322e8-db03-a3eb-6c7f-aa0385507418}" - testing.go:149: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.130Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: pipelining replication: peer="{Voter 020322e8-db03-a3eb-6c7f-aa0385507418 020322e8-db03-a3eb-6c7f-aa0385507418}" -testLoggerAdapter verbose: 363d840e-a128-8f51-7ba8-2e437195bfa9: 2021-07-10T01:05:10.135Z [WARN] 363d840e-a128-8f51-7ba8-2e437195bfa9: no known peers, aborting election - testing.go:149: 363d840e-a128-8f51-7ba8-2e437195bfa9: 2021-07-10T01:05:10.135Z [WARN] 363d840e-a128-8f51-7ba8-2e437195bfa9: no known peers, aborting election -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:10.235Z [INFO] stable state for Leader reached at 2021-07-10 01:05:10.130416533 +0000 UTC m=+15.518574563 (1 nodes), 73.355534ms from start of poll, 74.363264ms from cluster start. Timeout at 2021-07-10 01:05:10.23550362 +0000 UTC m=+15.623661610, 105.087047ms after stability - testing.go:149: cluster: 2021-07-10T01:05:10.235Z [INFO] stable state for Leader reached at 2021-07-10 01:05:10.130416533 +0000 UTC m=+15.518574563 (1 nodes), 73.355534ms from start of poll, 74.363264ms from cluster start. Timeout at 2021-07-10 01:05:10.23550362 +0000 UTC m=+15.623661610, 105.087047ms after stability -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:10.235Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:10.235Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:10.340Z [INFO] stable state for Leader reached at 2021-07-10 01:05:10.235777443 +0000 UTC m=+15.623935413 (1 nodes), 2.06µs from start of poll, 179.724114ms from cluster start. Timeout at 2021-07-10 01:05:10.340861471 +0000 UTC m=+15.729019441, 105.084028ms after stability - testing.go:149: cluster: 2021-07-10T01:05:10.340Z [INFO] stable state for Leader reached at 2021-07-10 01:05:10.235777443 +0000 UTC m=+15.623935413 (1 nodes), 2.06µs from start of poll, 179.724114ms from cluster start. Timeout at 2021-07-10 01:05:10.340861471 +0000 UTC m=+15.729019441, 105.084028ms after stability -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:10.341Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:10.341Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:10.446Z [INFO] stable state for Leader reached at 2021-07-10 01:05:10.341137394 +0000 UTC m=+15.729295364 (1 nodes), 1.92µs from start of poll, 285.084065ms from cluster start. Timeout at 2021-07-10 01:05:10.446239282 +0000 UTC m=+15.834397252, 105.101888ms after stability - testing.go:149: cluster: 2021-07-10T01:05:10.446Z [INFO] stable state for Leader reached at 2021-07-10 01:05:10.341137394 +0000 UTC m=+15.729295364 (1 nodes), 1.92µs from start of poll, 285.084065ms from cluster start. Timeout at 2021-07-10 01:05:10.446239282 +0000 UTC m=+15.834397252, 105.101888ms after stability -testLoggerAdapter verbose: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.446Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: updating configuration: command=AddStaging server-id=363d840e-a128-8f51-7ba8-2e437195bfa9 server-addr=363d840e-a128-8f51-7ba8-2e437195bfa9 servers="[{Suffrage:Voter ID:9a872e48-d9d3-d627-f6d2-461f9c91991c Address:9a872e48-d9d3-d627-f6d2-461f9c91991c} {Suffrage:Voter ID:020322e8-db03-a3eb-6c7f-aa0385507418 Address:020322e8-db03-a3eb-6c7f-aa0385507418} {Suffrage:Voter ID:363d840e-a128-8f51-7ba8-2e437195bfa9 Address:363d840e-a128-8f51-7ba8-2e437195bfa9}]" - testing.go:149: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.446Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: updating configuration: command=AddStaging server-id=363d840e-a128-8f51-7ba8-2e437195bfa9 server-addr=363d840e-a128-8f51-7ba8-2e437195bfa9 servers="[{Suffrage:Voter ID:9a872e48-d9d3-d627-f6d2-461f9c91991c Address:9a872e48-d9d3-d627-f6d2-461f9c91991c} {Suffrage:Voter ID:020322e8-db03-a3eb-6c7f-aa0385507418 Address:020322e8-db03-a3eb-6c7f-aa0385507418} {Suffrage:Voter ID:363d840e-a128-8f51-7ba8-2e437195bfa9 Address:363d840e-a128-8f51-7ba8-2e437195bfa9}]" -testLoggerAdapter verbose: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.446Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: added peer, starting replication: peer=363d840e-a128-8f51-7ba8-2e437195bfa9 - testing.go:149: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.446Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: added peer, starting replication: peer=363d840e-a128-8f51-7ba8-2e437195bfa9 -testLoggerAdapter verbose: 363d840e-a128-8f51-7ba8-2e437195bfa9: 2021-07-10T01:05:10.446Z [WARN] 363d840e-a128-8f51-7ba8-2e437195bfa9: failed to get previous log: previous-index=3 last-index=0 error="log not found" - testing.go:149: 363d840e-a128-8f51-7ba8-2e437195bfa9: 2021-07-10T01:05:10.446Z [WARN] 363d840e-a128-8f51-7ba8-2e437195bfa9: failed to get previous log: previous-index=3 last-index=0 error="log not found" -testLoggerAdapter verbose: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.447Z [WARN] 9a872e48-d9d3-d627-f6d2-461f9c91991c: appendEntries rejected, sending older logs: peer="{Voter 363d840e-a128-8f51-7ba8-2e437195bfa9 363d840e-a128-8f51-7ba8-2e437195bfa9}" next=1 - testing.go:149: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.447Z [WARN] 9a872e48-d9d3-d627-f6d2-461f9c91991c: appendEntries rejected, sending older logs: peer="{Voter 363d840e-a128-8f51-7ba8-2e437195bfa9 363d840e-a128-8f51-7ba8-2e437195bfa9}" next=1 -testLoggerAdapter verbose: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.447Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: pipelining replication: peer="{Voter 363d840e-a128-8f51-7ba8-2e437195bfa9 363d840e-a128-8f51-7ba8-2e437195bfa9}" - testing.go:149: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.447Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: pipelining replication: peer="{Voter 363d840e-a128-8f51-7ba8-2e437195bfa9 363d840e-a128-8f51-7ba8-2e437195bfa9}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:10.452Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:10.452Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:10.557Z [INFO] stable state for Leader reached at 2021-07-10 01:05:10.452377007 +0000 UTC m=+15.840534977 (1 nodes), 4.32µs from start of poll, 396.323678ms from cluster start. Timeout at 2021-07-10 01:05:10.557445414 +0000 UTC m=+15.945603384, 105.068407ms after stability - testing.go:149: cluster: 2021-07-10T01:05:10.557Z [INFO] stable state for Leader reached at 2021-07-10 01:05:10.452377007 +0000 UTC m=+15.840534977 (1 nodes), 4.32µs from start of poll, 396.323678ms from cluster start. Timeout at 2021-07-10 01:05:10.557445414 +0000 UTC m=+15.945603384, 105.068407ms after stability -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:10.557Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:10.557Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:10.662Z [INFO] stable state for Leader reached at 2021-07-10 01:05:10.557672477 +0000 UTC m=+15.945830447 (1 nodes), 2.4µs from start of poll, 501.619148ms from cluster start. Timeout at 2021-07-10 01:05:10.662688684 +0000 UTC m=+16.050846654, 105.016207ms after stability - testing.go:149: cluster: 2021-07-10T01:05:10.662Z [INFO] stable state for Leader reached at 2021-07-10 01:05:10.557672477 +0000 UTC m=+15.945830447 (1 nodes), 2.4µs from start of poll, 501.619148ms from cluster start. Timeout at 2021-07-10 01:05:10.662688684 +0000 UTC m=+16.050846654, 105.016207ms after stability -testLoggerAdapter verbose: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.663Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: updating configuration: command=RemoveServer server-id=363d840e-a128-8f51-7ba8-2e437195bfa9 server-addr= servers="[{Suffrage:Voter ID:9a872e48-d9d3-d627-f6d2-461f9c91991c Address:9a872e48-d9d3-d627-f6d2-461f9c91991c} {Suffrage:Voter ID:020322e8-db03-a3eb-6c7f-aa0385507418 Address:020322e8-db03-a3eb-6c7f-aa0385507418}]" - testing.go:149: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.663Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: updating configuration: command=RemoveServer server-id=363d840e-a128-8f51-7ba8-2e437195bfa9 server-addr= servers="[{Suffrage:Voter ID:9a872e48-d9d3-d627-f6d2-461f9c91991c Address:9a872e48-d9d3-d627-f6d2-461f9c91991c} {Suffrage:Voter ID:020322e8-db03-a3eb-6c7f-aa0385507418 Address:020322e8-db03-a3eb-6c7f-aa0385507418}]" -testLoggerAdapter verbose: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.663Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: removed peer, stopping replication: peer=363d840e-a128-8f51-7ba8-2e437195bfa9 last-index=4 - testing.go:149: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.663Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: removed peer, stopping replication: peer=363d840e-a128-8f51-7ba8-2e437195bfa9 last-index=4 -testLoggerAdapter verbose: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.663Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: aborting pipeline replication: peer="{Voter 363d840e-a128-8f51-7ba8-2e437195bfa9 363d840e-a128-8f51-7ba8-2e437195bfa9}" - testing.go:149: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.663Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: aborting pipeline replication: peer="{Voter 363d840e-a128-8f51-7ba8-2e437195bfa9 363d840e-a128-8f51-7ba8-2e437195bfa9}" -testLoggerAdapter verbose: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.663Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: aborting pipeline replication: peer="{Voter 020322e8-db03-a3eb-6c7f-aa0385507418 020322e8-db03-a3eb-6c7f-aa0385507418}" - testing.go:149: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.663Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: aborting pipeline replication: peer="{Voter 020322e8-db03-a3eb-6c7f-aa0385507418 020322e8-db03-a3eb-6c7f-aa0385507418}" ---- PASS: TestRaft_ProtocolVersion_Upgrade_1_2 (0.61s) +testLoggerAdapter verbose: 2917a807-97fc-a914-a414-0d0ab7a89ace: 2022-08-12T07:30:14.638Z [INFO] 2917a807-97fc-a914-a414-0d0ab7a89ace: initial configuration: index=1 servers="[{Suffrage:Voter ID:2917a807-97fc-a914-a414-0d0ab7a89ace Address:2917a807-97fc-a914-a414-0d0ab7a89ace} {Suffrage:Voter ID:e85152e7-9f1c-6f9b-fd5f-a83ca8f11d2c Address:e85152e7-9f1c-6f9b-fd5f-a83ca8f11d2c}]" + testing.go:149: 2917a807-97fc-a914-a414-0d0ab7a89ace: 2022-08-12T07:30:14.638Z [INFO] 2917a807-97fc-a914-a414-0d0ab7a89ace: initial configuration: index=1 servers="[{Suffrage:Voter ID:2917a807-97fc-a914-a414-0d0ab7a89ace Address:2917a807-97fc-a914-a414-0d0ab7a89ace} {Suffrage:Voter ID:e85152e7-9f1c-6f9b-fd5f-a83ca8f11d2c Address:e85152e7-9f1c-6f9b-fd5f-a83ca8f11d2c}]" +testLoggerAdapter verbose: e85152e7-9f1c-6f9b-fd5f-a83ca8f11d2c: 2022-08-12T07:30:14.638Z [INFO] e85152e7-9f1c-6f9b-fd5f-a83ca8f11d2c: initial configuration: index=1 servers="[{Suffrage:Voter ID:2917a807-97fc-a914-a414-0d0ab7a89ace Address:2917a807-97fc-a914-a414-0d0ab7a89ace} {Suffrage:Voter ID:e85152e7-9f1c-6f9b-fd5f-a83ca8f11d2c Address:e85152e7-9f1c-6f9b-fd5f-a83ca8f11d2c}]" + testing.go:149: e85152e7-9f1c-6f9b-fd5f-a83ca8f11d2c: 2022-08-12T07:30:14.638Z [INFO] e85152e7-9f1c-6f9b-fd5f-a83ca8f11d2c: initial configuration: index=1 servers="[{Suffrage:Voter ID:2917a807-97fc-a914-a414-0d0ab7a89ace Address:2917a807-97fc-a914-a414-0d0ab7a89ace} {Suffrage:Voter ID:e85152e7-9f1c-6f9b-fd5f-a83ca8f11d2c Address:e85152e7-9f1c-6f9b-fd5f-a83ca8f11d2c}]" +testLoggerAdapter verbose: e85152e7-9f1c-6f9b-fd5f-a83ca8f11d2c: 2022-08-12T07:30:14.639Z [INFO] e85152e7-9f1c-6f9b-fd5f-a83ca8f11d2c: entering follower state: follower="Node at e85152e7-9f1c-6f9b-fd5f-a83ca8f11d2c [Follower]" leader= +testLoggerAdapter verbose: 2917a807-97fc-a914-a414-0d0ab7a89ace: 2022-08-12T07:30:14.638Z [INFO] 2917a807-97fc-a914-a414-0d0ab7a89ace: entering follower state: follower="Node at 2917a807-97fc-a914-a414-0d0ab7a89ace [Follower]" leader= + testing.go:149: e85152e7-9f1c-6f9b-fd5f-a83ca8f11d2c: 2022-08-12T07:30:14.639Z [INFO] e85152e7-9f1c-6f9b-fd5f-a83ca8f11d2c: entering follower state: follower="Node at e85152e7-9f1c-6f9b-fd5f-a83ca8f11d2c [Follower]" leader= + testing.go:149: 2917a807-97fc-a914-a414-0d0ab7a89ace: 2022-08-12T07:30:14.638Z [INFO] 2917a807-97fc-a914-a414-0d0ab7a89ace: entering follower state: follower="Node at 2917a807-97fc-a914-a414-0d0ab7a89ace [Follower]" leader= +testLoggerAdapter verbose: d007266e-a784-feb8-9ad6-ce7d20c63422: 2022-08-12T07:30:14.639Z [INFO] d007266e-a784-feb8-9ad6-ce7d20c63422: initial configuration: index=0 servers=[] + testing.go:149: d007266e-a784-feb8-9ad6-ce7d20c63422: 2022-08-12T07:30:14.639Z [INFO] d007266e-a784-feb8-9ad6-ce7d20c63422: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:14.640Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:14.640Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: d007266e-a784-feb8-9ad6-ce7d20c63422: 2022-08-12T07:30:14.640Z [INFO] d007266e-a784-feb8-9ad6-ce7d20c63422: entering follower state: follower="Node at d007266e-a784-feb8-9ad6-ce7d20c63422 [Follower]" leader= + testing.go:149: d007266e-a784-feb8-9ad6-ce7d20c63422: 2022-08-12T07:30:14.640Z [INFO] d007266e-a784-feb8-9ad6-ce7d20c63422: entering follower state: follower="Node at d007266e-a784-feb8-9ad6-ce7d20c63422 [Follower]" leader= +testLoggerAdapter verbose: d007266e-a784-feb8-9ad6-ce7d20c63422: 2022-08-12T07:30:14.699Z [WARN] d007266e-a784-feb8-9ad6-ce7d20c63422: no known peers, aborting election + testing.go:149: d007266e-a784-feb8-9ad6-ce7d20c63422: 2022-08-12T07:30:14.699Z [WARN] d007266e-a784-feb8-9ad6-ce7d20c63422: no known peers, aborting election +testLoggerAdapter verbose: 2917a807-97fc-a914-a414-0d0ab7a89ace: 2022-08-12T07:30:14.731Z [WARN] 2917a807-97fc-a914-a414-0d0ab7a89ace: heartbeat timeout reached, starting election: last-leader= + testing.go:149: 2917a807-97fc-a914-a414-0d0ab7a89ace: 2022-08-12T07:30:14.731Z [WARN] 2917a807-97fc-a914-a414-0d0ab7a89ace: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: 2917a807-97fc-a914-a414-0d0ab7a89ace: 2022-08-12T07:30:14.731Z [INFO] 2917a807-97fc-a914-a414-0d0ab7a89ace: entering candidate state: node="Node at 2917a807-97fc-a914-a414-0d0ab7a89ace [Candidate]" term=2 + testing.go:149: 2917a807-97fc-a914-a414-0d0ab7a89ace: 2022-08-12T07:30:14.731Z [INFO] 2917a807-97fc-a914-a414-0d0ab7a89ace: entering candidate state: node="Node at 2917a807-97fc-a914-a414-0d0ab7a89ace [Candidate]" term=2 +testLoggerAdapter verbose: 2917a807-97fc-a914-a414-0d0ab7a89ace: 2022-08-12T07:30:14.732Z [INFO] 2917a807-97fc-a914-a414-0d0ab7a89ace: election won: tally=2 + testing.go:149: 2917a807-97fc-a914-a414-0d0ab7a89ace: 2022-08-12T07:30:14.732Z [INFO] 2917a807-97fc-a914-a414-0d0ab7a89ace: election won: tally=2 +testLoggerAdapter verbose: 2917a807-97fc-a914-a414-0d0ab7a89ace: 2022-08-12T07:30:14.732Z [INFO] 2917a807-97fc-a914-a414-0d0ab7a89ace: entering leader state: leader="Node at 2917a807-97fc-a914-a414-0d0ab7a89ace [Leader]" + testing.go:149: 2917a807-97fc-a914-a414-0d0ab7a89ace: 2022-08-12T07:30:14.732Z [INFO] 2917a807-97fc-a914-a414-0d0ab7a89ace: entering leader state: leader="Node at 2917a807-97fc-a914-a414-0d0ab7a89ace [Leader]" +testLoggerAdapter verbose: 2917a807-97fc-a914-a414-0d0ab7a89ace: 2022-08-12T07:30:14.732Z [INFO] 2917a807-97fc-a914-a414-0d0ab7a89ace: added peer, starting replication: peer=e85152e7-9f1c-6f9b-fd5f-a83ca8f11d2c + testing.go:149: 2917a807-97fc-a914-a414-0d0ab7a89ace: 2022-08-12T07:30:14.732Z [INFO] 2917a807-97fc-a914-a414-0d0ab7a89ace: added peer, starting replication: peer=e85152e7-9f1c-6f9b-fd5f-a83ca8f11d2c +testLoggerAdapter verbose: 2917a807-97fc-a914-a414-0d0ab7a89ace: 2022-08-12T07:30:14.732Z [INFO] 2917a807-97fc-a914-a414-0d0ab7a89ace: pipelining replication: peer="{Voter e85152e7-9f1c-6f9b-fd5f-a83ca8f11d2c e85152e7-9f1c-6f9b-fd5f-a83ca8f11d2c}" + testing.go:149: 2917a807-97fc-a914-a414-0d0ab7a89ace: 2022-08-12T07:30:14.732Z [INFO] 2917a807-97fc-a914-a414-0d0ab7a89ace: pipelining replication: peer="{Voter e85152e7-9f1c-6f9b-fd5f-a83ca8f11d2c e85152e7-9f1c-6f9b-fd5f-a83ca8f11d2c}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:14.837Z [INFO] stable state for Leader reached at 2022-08-12 07:30:14.7323176 +0000 UTC m=+15.689004061 (1 nodes), 91.93568ms from start of poll, 94.16512ms from cluster start. Timeout at 2022-08-12 07:30:14.83738684 +0000 UTC m=+15.794073301, 105.06924ms after stability + testing.go:149: cluster: 2022-08-12T07:30:14.837Z [INFO] stable state for Leader reached at 2022-08-12 07:30:14.7323176 +0000 UTC m=+15.689004061 (1 nodes), 91.93568ms from start of poll, 94.16512ms from cluster start. Timeout at 2022-08-12 07:30:14.83738684 +0000 UTC m=+15.794073301, 105.06924ms after stability +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:14.837Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:14.837Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:14.942Z [INFO] stable state for Leader reached at 2022-08-12 07:30:14.8376598 +0000 UTC m=+15.794346281 (1 nodes), 1.82µs from start of poll, 199.50734ms from cluster start. Timeout at 2022-08-12 07:30:14.94274982 +0000 UTC m=+15.899436281, 105.09ms after stability + testing.go:149: cluster: 2022-08-12T07:30:14.942Z [INFO] stable state for Leader reached at 2022-08-12 07:30:14.8376598 +0000 UTC m=+15.794346281 (1 nodes), 1.82µs from start of poll, 199.50734ms from cluster start. Timeout at 2022-08-12 07:30:14.94274982 +0000 UTC m=+15.899436281, 105.09ms after stability +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:14.942Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:14.942Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:15.048Z [INFO] stable state for Leader reached at 2022-08-12 07:30:14.94303102 +0000 UTC m=+15.899717501 (1 nodes), 4.04µs from start of poll, 304.87856ms from cluster start. Timeout at 2022-08-12 07:30:15.04810476 +0000 UTC m=+16.004791261, 105.07376ms after stability + testing.go:149: cluster: 2022-08-12T07:30:15.048Z [INFO] stable state for Leader reached at 2022-08-12 07:30:14.94303102 +0000 UTC m=+15.899717501 (1 nodes), 4.04µs from start of poll, 304.87856ms from cluster start. Timeout at 2022-08-12 07:30:15.04810476 +0000 UTC m=+16.004791261, 105.07376ms after stability +testLoggerAdapter verbose: 2917a807-97fc-a914-a414-0d0ab7a89ace: 2022-08-12T07:30:15.048Z [INFO] 2917a807-97fc-a914-a414-0d0ab7a89ace: updating configuration: command=AddStaging server-id=d007266e-a784-feb8-9ad6-ce7d20c63422 server-addr=d007266e-a784-feb8-9ad6-ce7d20c63422 servers="[{Suffrage:Voter ID:2917a807-97fc-a914-a414-0d0ab7a89ace Address:2917a807-97fc-a914-a414-0d0ab7a89ace} {Suffrage:Voter ID:e85152e7-9f1c-6f9b-fd5f-a83ca8f11d2c Address:e85152e7-9f1c-6f9b-fd5f-a83ca8f11d2c} {Suffrage:Voter ID:d007266e-a784-feb8-9ad6-ce7d20c63422 Address:d007266e-a784-feb8-9ad6-ce7d20c63422}]" + testing.go:149: 2917a807-97fc-a914-a414-0d0ab7a89ace: 2022-08-12T07:30:15.048Z [INFO] 2917a807-97fc-a914-a414-0d0ab7a89ace: updating configuration: command=AddStaging server-id=d007266e-a784-feb8-9ad6-ce7d20c63422 server-addr=d007266e-a784-feb8-9ad6-ce7d20c63422 servers="[{Suffrage:Voter ID:2917a807-97fc-a914-a414-0d0ab7a89ace Address:2917a807-97fc-a914-a414-0d0ab7a89ace} {Suffrage:Voter ID:e85152e7-9f1c-6f9b-fd5f-a83ca8f11d2c Address:e85152e7-9f1c-6f9b-fd5f-a83ca8f11d2c} {Suffrage:Voter ID:d007266e-a784-feb8-9ad6-ce7d20c63422 Address:d007266e-a784-feb8-9ad6-ce7d20c63422}]" +testLoggerAdapter verbose: 2917a807-97fc-a914-a414-0d0ab7a89ace: 2022-08-12T07:30:15.048Z [INFO] 2917a807-97fc-a914-a414-0d0ab7a89ace: added peer, starting replication: peer=d007266e-a784-feb8-9ad6-ce7d20c63422 + testing.go:149: 2917a807-97fc-a914-a414-0d0ab7a89ace: 2022-08-12T07:30:15.048Z [INFO] 2917a807-97fc-a914-a414-0d0ab7a89ace: added peer, starting replication: peer=d007266e-a784-feb8-9ad6-ce7d20c63422 +testLoggerAdapter verbose: d007266e-a784-feb8-9ad6-ce7d20c63422: 2022-08-12T07:30:15.048Z [WARN] d007266e-a784-feb8-9ad6-ce7d20c63422: failed to get previous log: previous-index=3 last-index=0 error="log not found" + testing.go:149: d007266e-a784-feb8-9ad6-ce7d20c63422: 2022-08-12T07:30:15.048Z [WARN] d007266e-a784-feb8-9ad6-ce7d20c63422: failed to get previous log: previous-index=3 last-index=0 error="log not found" +testLoggerAdapter verbose: 2917a807-97fc-a914-a414-0d0ab7a89ace: 2022-08-12T07:30:15.048Z [WARN] 2917a807-97fc-a914-a414-0d0ab7a89ace: appendEntries rejected, sending older logs: peer="{Voter d007266e-a784-feb8-9ad6-ce7d20c63422 d007266e-a784-feb8-9ad6-ce7d20c63422}" next=1 + testing.go:149: 2917a807-97fc-a914-a414-0d0ab7a89ace: 2022-08-12T07:30:15.048Z [WARN] 2917a807-97fc-a914-a414-0d0ab7a89ace: appendEntries rejected, sending older logs: peer="{Voter d007266e-a784-feb8-9ad6-ce7d20c63422 d007266e-a784-feb8-9ad6-ce7d20c63422}" next=1 +testLoggerAdapter verbose: 2917a807-97fc-a914-a414-0d0ab7a89ace: 2022-08-12T07:30:15.049Z [INFO] 2917a807-97fc-a914-a414-0d0ab7a89ace: pipelining replication: peer="{Voter d007266e-a784-feb8-9ad6-ce7d20c63422 d007266e-a784-feb8-9ad6-ce7d20c63422}" + testing.go:149: 2917a807-97fc-a914-a414-0d0ab7a89ace: 2022-08-12T07:30:15.049Z [INFO] 2917a807-97fc-a914-a414-0d0ab7a89ace: pipelining replication: peer="{Voter d007266e-a784-feb8-9ad6-ce7d20c63422 d007266e-a784-feb8-9ad6-ce7d20c63422}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:15.053Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:15.053Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:15.159Z [INFO] stable state for Leader reached at 2022-08-12 07:30:15.05402858 +0000 UTC m=+16.010715061 (1 nodes), 3.76µs from start of poll, 415.87612ms from cluster start. Timeout at 2022-08-12 07:30:15.15909784 +0000 UTC m=+16.115784301, 105.06924ms after stability + testing.go:149: cluster: 2022-08-12T07:30:15.159Z [INFO] stable state for Leader reached at 2022-08-12 07:30:15.05402858 +0000 UTC m=+16.010715061 (1 nodes), 3.76µs from start of poll, 415.87612ms from cluster start. Timeout at 2022-08-12 07:30:15.15909784 +0000 UTC m=+16.115784301, 105.06924ms after stability +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:15.159Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:15.159Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:15.264Z [INFO] stable state for Leader reached at 2022-08-12 07:30:15.15937636 +0000 UTC m=+16.116062841 (1 nodes), 3.04µs from start of poll, 521.2239ms from cluster start. Timeout at 2022-08-12 07:30:15.26445274 +0000 UTC m=+16.221139241, 105.0764ms after stability + testing.go:149: cluster: 2022-08-12T07:30:15.264Z [INFO] stable state for Leader reached at 2022-08-12 07:30:15.15937636 +0000 UTC m=+16.116062841 (1 nodes), 3.04µs from start of poll, 521.2239ms from cluster start. Timeout at 2022-08-12 07:30:15.26445274 +0000 UTC m=+16.221139241, 105.0764ms after stability +testLoggerAdapter verbose: 2917a807-97fc-a914-a414-0d0ab7a89ace: 2022-08-12T07:30:15.264Z [INFO] 2917a807-97fc-a914-a414-0d0ab7a89ace: updating configuration: command=RemoveServer server-id=d007266e-a784-feb8-9ad6-ce7d20c63422 server-addr= servers="[{Suffrage:Voter ID:2917a807-97fc-a914-a414-0d0ab7a89ace Address:2917a807-97fc-a914-a414-0d0ab7a89ace} {Suffrage:Voter ID:e85152e7-9f1c-6f9b-fd5f-a83ca8f11d2c Address:e85152e7-9f1c-6f9b-fd5f-a83ca8f11d2c}]" + testing.go:149: 2917a807-97fc-a914-a414-0d0ab7a89ace: 2022-08-12T07:30:15.264Z [INFO] 2917a807-97fc-a914-a414-0d0ab7a89ace: updating configuration: command=RemoveServer server-id=d007266e-a784-feb8-9ad6-ce7d20c63422 server-addr= servers="[{Suffrage:Voter ID:2917a807-97fc-a914-a414-0d0ab7a89ace Address:2917a807-97fc-a914-a414-0d0ab7a89ace} {Suffrage:Voter ID:e85152e7-9f1c-6f9b-fd5f-a83ca8f11d2c Address:e85152e7-9f1c-6f9b-fd5f-a83ca8f11d2c}]" +testLoggerAdapter verbose: 2917a807-97fc-a914-a414-0d0ab7a89ace: 2022-08-12T07:30:15.264Z [INFO] 2917a807-97fc-a914-a414-0d0ab7a89ace: removed peer, stopping replication: peer=d007266e-a784-feb8-9ad6-ce7d20c63422 last-index=4 + testing.go:149: 2917a807-97fc-a914-a414-0d0ab7a89ace: 2022-08-12T07:30:15.264Z [INFO] 2917a807-97fc-a914-a414-0d0ab7a89ace: removed peer, stopping replication: peer=d007266e-a784-feb8-9ad6-ce7d20c63422 last-index=4 +testLoggerAdapter verbose: 2917a807-97fc-a914-a414-0d0ab7a89ace: 2022-08-12T07:30:15.265Z [INFO] 2917a807-97fc-a914-a414-0d0ab7a89ace: aborting pipeline replication: peer="{Voter d007266e-a784-feb8-9ad6-ce7d20c63422 d007266e-a784-feb8-9ad6-ce7d20c63422}" + testing.go:149: 2917a807-97fc-a914-a414-0d0ab7a89ace: 2022-08-12T07:30:15.265Z [INFO] 2917a807-97fc-a914-a414-0d0ab7a89ace: aborting pipeline replication: peer="{Voter d007266e-a784-feb8-9ad6-ce7d20c63422 d007266e-a784-feb8-9ad6-ce7d20c63422}" +testLoggerAdapter verbose: 2917a807-97fc-a914-a414-0d0ab7a89ace: 2022-08-12T07:30:15.265Z [INFO] 2917a807-97fc-a914-a414-0d0ab7a89ace: aborting pipeline replication: peer="{Voter e85152e7-9f1c-6f9b-fd5f-a83ca8f11d2c e85152e7-9f1c-6f9b-fd5f-a83ca8f11d2c}" + testing.go:149: 2917a807-97fc-a914-a414-0d0ab7a89ace: 2022-08-12T07:30:15.265Z [INFO] 2917a807-97fc-a914-a414-0d0ab7a89ace: aborting pipeline replication: peer="{Voter e85152e7-9f1c-6f9b-fd5f-a83ca8f11d2c e85152e7-9f1c-6f9b-fd5f-a83ca8f11d2c}" +--- PASS: TestRaft_ProtocolVersion_Upgrade_1_2 (0.63s) === RUN TestRaft_ProtocolVersion_Upgrade_2_3 -testLoggerAdapter verbose: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.664Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: initial configuration: index=1 servers="[{Suffrage:Voter ID:1305795e-8f52-6a93-4f5f-2c58c5b5d119 Address:1305795e-8f52-6a93-4f5f-2c58c5b5d119} {Suffrage:Voter ID:a0850083-6008-3bdc-77a7-b2a2091c4613 Address:a0850083-6008-3bdc-77a7-b2a2091c4613}]" - testing.go:149: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.664Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: initial configuration: index=1 servers="[{Suffrage:Voter ID:1305795e-8f52-6a93-4f5f-2c58c5b5d119 Address:1305795e-8f52-6a93-4f5f-2c58c5b5d119} {Suffrage:Voter ID:a0850083-6008-3bdc-77a7-b2a2091c4613 Address:a0850083-6008-3bdc-77a7-b2a2091c4613}]" -testLoggerAdapter verbose: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.664Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: entering follower state: follower="Node at 1305795e-8f52-6a93-4f5f-2c58c5b5d119 [Follower]" leader= - testing.go:149: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.664Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: entering follower state: follower="Node at 1305795e-8f52-6a93-4f5f-2c58c5b5d119 [Follower]" leader= -testLoggerAdapter verbose: a0850083-6008-3bdc-77a7-b2a2091c4613: 2021-07-10T01:05:10.664Z [INFO] a0850083-6008-3bdc-77a7-b2a2091c4613: initial configuration: index=1 servers="[{Suffrage:Voter ID:1305795e-8f52-6a93-4f5f-2c58c5b5d119 Address:1305795e-8f52-6a93-4f5f-2c58c5b5d119} {Suffrage:Voter ID:a0850083-6008-3bdc-77a7-b2a2091c4613 Address:a0850083-6008-3bdc-77a7-b2a2091c4613}]" - testing.go:149: a0850083-6008-3bdc-77a7-b2a2091c4613: 2021-07-10T01:05:10.664Z [INFO] a0850083-6008-3bdc-77a7-b2a2091c4613: initial configuration: index=1 servers="[{Suffrage:Voter ID:1305795e-8f52-6a93-4f5f-2c58c5b5d119 Address:1305795e-8f52-6a93-4f5f-2c58c5b5d119} {Suffrage:Voter ID:a0850083-6008-3bdc-77a7-b2a2091c4613 Address:a0850083-6008-3bdc-77a7-b2a2091c4613}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:10.665Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-07-10T01:05:10.665Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: a0850083-6008-3bdc-77a7-b2a2091c4613: 2021-07-10T01:05:10.665Z [INFO] a0850083-6008-3bdc-77a7-b2a2091c4613: entering follower state: follower="Node at a0850083-6008-3bdc-77a7-b2a2091c4613 [Follower]" leader= - testing.go:149: a0850083-6008-3bdc-77a7-b2a2091c4613: 2021-07-10T01:05:10.665Z [INFO] a0850083-6008-3bdc-77a7-b2a2091c4613: entering follower state: follower="Node at a0850083-6008-3bdc-77a7-b2a2091c4613 [Follower]" leader= -testLoggerAdapter verbose: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.750Z [WARN] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: heartbeat timeout reached, starting election: last-leader= - testing.go:149: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.750Z [WARN] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.750Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: entering candidate state: node="Node at 1305795e-8f52-6a93-4f5f-2c58c5b5d119 [Candidate]" term=2 - testing.go:149: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.750Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: entering candidate state: node="Node at 1305795e-8f52-6a93-4f5f-2c58c5b5d119 [Candidate]" term=2 -testLoggerAdapter verbose: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.751Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: election won: tally=2 - testing.go:149: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.751Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: election won: tally=2 -testLoggerAdapter verbose: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.751Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: entering leader state: leader="Node at 1305795e-8f52-6a93-4f5f-2c58c5b5d119 [Leader]" - testing.go:149: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.751Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: entering leader state: leader="Node at 1305795e-8f52-6a93-4f5f-2c58c5b5d119 [Leader]" -testLoggerAdapter verbose: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.751Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: added peer, starting replication: peer=a0850083-6008-3bdc-77a7-b2a2091c4613 - testing.go:149: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.751Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: added peer, starting replication: peer=a0850083-6008-3bdc-77a7-b2a2091c4613 -testLoggerAdapter verbose: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.751Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: pipelining replication: peer="{Voter a0850083-6008-3bdc-77a7-b2a2091c4613 a0850083-6008-3bdc-77a7-b2a2091c4613}" - testing.go:149: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.751Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: pipelining replication: peer="{Voter a0850083-6008-3bdc-77a7-b2a2091c4613 a0850083-6008-3bdc-77a7-b2a2091c4613}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:10.856Z [INFO] stable state for Follower reached at 2021-07-10 01:05:10.751230797 +0000 UTC m=+16.139388787 (1 nodes), 86.169048ms from start of poll, 86.741934ms from cluster start. Timeout at 2021-07-10 01:05:10.856306805 +0000 UTC m=+16.244464775, 105.075988ms after stability - testing.go:149: cluster: 2021-07-10T01:05:10.856Z [INFO] stable state for Follower reached at 2021-07-10 01:05:10.751230797 +0000 UTC m=+16.139388787 (1 nodes), 86.169048ms from start of poll, 86.741934ms from cluster start. Timeout at 2021-07-10 01:05:10.856306805 +0000 UTC m=+16.244464775, 105.075988ms after stability -testLoggerAdapter verbose: server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d: 2021-07-10T01:05:10.856Z [INFO] server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d: initial configuration: index=0 servers=[] - testing.go:149: server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d: 2021-07-10T01:05:10.856Z [INFO] server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d: initial configuration: index=0 servers=[] -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:10.857Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:10.857Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d: 2021-07-10T01:05:10.857Z [INFO] server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d: entering follower state: follower="Node at a8678b99-61f6-a3e5-6f78-5df9e6bafe9d [Follower]" leader= - testing.go:149: server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d: 2021-07-10T01:05:10.857Z [INFO] server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d: entering follower state: follower="Node at a8678b99-61f6-a3e5-6f78-5df9e6bafe9d [Follower]" leader= -testLoggerAdapter verbose: server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d: 2021-07-10T01:05:10.919Z [WARN] server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d: no known peers, aborting election - testing.go:149: server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d: 2021-07-10T01:05:10.919Z [WARN] server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d: no known peers, aborting election -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:10.962Z [INFO] stable state for Leader reached at 2021-07-10 01:05:10.857127914 +0000 UTC m=+16.245285884 (1 nodes), 1.46µs from start of poll, 192.639031ms from cluster start. Timeout at 2021-07-10 01:05:10.962197782 +0000 UTC m=+16.350355752, 105.069868ms after stability - testing.go:149: cluster: 2021-07-10T01:05:10.962Z [INFO] stable state for Leader reached at 2021-07-10 01:05:10.857127914 +0000 UTC m=+16.245285884 (1 nodes), 1.46µs from start of poll, 192.639031ms from cluster start. Timeout at 2021-07-10 01:05:10.962197782 +0000 UTC m=+16.350355752, 105.069868ms after stability -testLoggerAdapter verbose: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.962Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: updating configuration: command=AddStaging server-id=server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d server-addr=a8678b99-61f6-a3e5-6f78-5df9e6bafe9d servers="[{Suffrage:Voter ID:1305795e-8f52-6a93-4f5f-2c58c5b5d119 Address:1305795e-8f52-6a93-4f5f-2c58c5b5d119} {Suffrage:Voter ID:a0850083-6008-3bdc-77a7-b2a2091c4613 Address:a0850083-6008-3bdc-77a7-b2a2091c4613} {Suffrage:Voter ID:server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d Address:a8678b99-61f6-a3e5-6f78-5df9e6bafe9d}]" - testing.go:149: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.962Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: updating configuration: command=AddStaging server-id=server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d server-addr=a8678b99-61f6-a3e5-6f78-5df9e6bafe9d servers="[{Suffrage:Voter ID:1305795e-8f52-6a93-4f5f-2c58c5b5d119 Address:1305795e-8f52-6a93-4f5f-2c58c5b5d119} {Suffrage:Voter ID:a0850083-6008-3bdc-77a7-b2a2091c4613 Address:a0850083-6008-3bdc-77a7-b2a2091c4613} {Suffrage:Voter ID:server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d Address:a8678b99-61f6-a3e5-6f78-5df9e6bafe9d}]" -testLoggerAdapter verbose: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.962Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: added peer, starting replication: peer=server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d - testing.go:149: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.962Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: added peer, starting replication: peer=server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d -testLoggerAdapter verbose: server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d: 2021-07-10T01:05:10.962Z [WARN] server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d: failed to get previous log: previous-index=3 last-index=0 error="log not found" - testing.go:149: server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d: 2021-07-10T01:05:10.962Z [WARN] server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d: failed to get previous log: previous-index=3 last-index=0 error="log not found" -testLoggerAdapter verbose: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.962Z [WARN] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: appendEntries rejected, sending older logs: peer="{Voter server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d a8678b99-61f6-a3e5-6f78-5df9e6bafe9d}" next=1 - testing.go:149: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.962Z [WARN] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: appendEntries rejected, sending older logs: peer="{Voter server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d a8678b99-61f6-a3e5-6f78-5df9e6bafe9d}" next=1 -testLoggerAdapter verbose: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.963Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: pipelining replication: peer="{Voter server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d a8678b99-61f6-a3e5-6f78-5df9e6bafe9d}" - testing.go:149: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.963Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: pipelining replication: peer="{Voter server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d a8678b99-61f6-a3e5-6f78-5df9e6bafe9d}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:10.968Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:10.968Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:11.073Z [INFO] stable state for Leader reached at 2021-07-10 01:05:10.968225845 +0000 UTC m=+16.356383815 (1 nodes), 2.02µs from start of poll, 303.736962ms from cluster start. Timeout at 2021-07-10 01:05:11.073302193 +0000 UTC m=+16.461460143, 105.076328ms after stability - testing.go:149: cluster: 2021-07-10T01:05:11.073Z [INFO] stable state for Leader reached at 2021-07-10 01:05:10.968225845 +0000 UTC m=+16.356383815 (1 nodes), 2.02µs from start of poll, 303.736962ms from cluster start. Timeout at 2021-07-10 01:05:11.073302193 +0000 UTC m=+16.461460143, 105.076328ms after stability -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:11.073Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:11.073Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:11.178Z [INFO] stable state for Leader reached at 2021-07-10 01:05:11.073530575 +0000 UTC m=+16.461688545 (1 nodes), 2.7µs from start of poll, 409.041692ms from cluster start. Timeout at 2021-07-10 01:05:11.178608463 +0000 UTC m=+16.566766433, 105.077888ms after stability - testing.go:149: cluster: 2021-07-10T01:05:11.178Z [INFO] stable state for Leader reached at 2021-07-10 01:05:11.073530575 +0000 UTC m=+16.461688545 (1 nodes), 2.7µs from start of poll, 409.041692ms from cluster start. Timeout at 2021-07-10 01:05:11.178608463 +0000 UTC m=+16.566766433, 105.077888ms after stability -testLoggerAdapter verbose: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:11.178Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: updating configuration: command=RemoveServer server-id=a0850083-6008-3bdc-77a7-b2a2091c4613 server-addr= servers="[{Suffrage:Voter ID:1305795e-8f52-6a93-4f5f-2c58c5b5d119 Address:1305795e-8f52-6a93-4f5f-2c58c5b5d119} {Suffrage:Voter ID:server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d Address:a8678b99-61f6-a3e5-6f78-5df9e6bafe9d}]" - testing.go:149: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:11.178Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: updating configuration: command=RemoveServer server-id=a0850083-6008-3bdc-77a7-b2a2091c4613 server-addr= servers="[{Suffrage:Voter ID:1305795e-8f52-6a93-4f5f-2c58c5b5d119 Address:1305795e-8f52-6a93-4f5f-2c58c5b5d119} {Suffrage:Voter ID:server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d Address:a8678b99-61f6-a3e5-6f78-5df9e6bafe9d}]" -testLoggerAdapter verbose: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:11.179Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: removed peer, stopping replication: peer=a0850083-6008-3bdc-77a7-b2a2091c4613 last-index=4 - testing.go:149: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:11.179Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: removed peer, stopping replication: peer=a0850083-6008-3bdc-77a7-b2a2091c4613 last-index=4 -testLoggerAdapter verbose: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:11.179Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: aborting pipeline replication: peer="{Voter a0850083-6008-3bdc-77a7-b2a2091c4613 a0850083-6008-3bdc-77a7-b2a2091c4613}" - testing.go:149: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:11.179Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: aborting pipeline replication: peer="{Voter a0850083-6008-3bdc-77a7-b2a2091c4613 a0850083-6008-3bdc-77a7-b2a2091c4613}" -testLoggerAdapter verbose: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:11.179Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: aborting pipeline replication: peer="{Voter server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d a8678b99-61f6-a3e5-6f78-5df9e6bafe9d}" - testing.go:149: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:11.179Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: aborting pipeline replication: peer="{Voter server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d a8678b99-61f6-a3e5-6f78-5df9e6bafe9d}" ---- PASS: TestRaft_ProtocolVersion_Upgrade_2_3 (0.52s) +testLoggerAdapter verbose: 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: 2022-08-12T07:30:15.266Z [INFO] 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: initial configuration: index=1 servers="[{Suffrage:Voter ID:0ef310aa-9188-b8c1-10ff-a2b36ece5a1b Address:0ef310aa-9188-b8c1-10ff-a2b36ece5a1b} {Suffrage:Voter ID:2c6c91fe-e3d3-a39b-1341-2e854c4257d4 Address:2c6c91fe-e3d3-a39b-1341-2e854c4257d4}]" + testing.go:149: 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: 2022-08-12T07:30:15.266Z [INFO] 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: initial configuration: index=1 servers="[{Suffrage:Voter ID:0ef310aa-9188-b8c1-10ff-a2b36ece5a1b Address:0ef310aa-9188-b8c1-10ff-a2b36ece5a1b} {Suffrage:Voter ID:2c6c91fe-e3d3-a39b-1341-2e854c4257d4 Address:2c6c91fe-e3d3-a39b-1341-2e854c4257d4}]" +testLoggerAdapter verbose: 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: 2022-08-12T07:30:15.266Z [INFO] 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: entering follower state: follower="Node at 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b [Follower]" leader= + testing.go:149: 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: 2022-08-12T07:30:15.266Z [INFO] 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: entering follower state: follower="Node at 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b [Follower]" leader= +testLoggerAdapter verbose: 2c6c91fe-e3d3-a39b-1341-2e854c4257d4: 2022-08-12T07:30:15.266Z [INFO] 2c6c91fe-e3d3-a39b-1341-2e854c4257d4: initial configuration: index=1 servers="[{Suffrage:Voter ID:0ef310aa-9188-b8c1-10ff-a2b36ece5a1b Address:0ef310aa-9188-b8c1-10ff-a2b36ece5a1b} {Suffrage:Voter ID:2c6c91fe-e3d3-a39b-1341-2e854c4257d4 Address:2c6c91fe-e3d3-a39b-1341-2e854c4257d4}]" + testing.go:149: 2c6c91fe-e3d3-a39b-1341-2e854c4257d4: 2022-08-12T07:30:15.266Z [INFO] 2c6c91fe-e3d3-a39b-1341-2e854c4257d4: initial configuration: index=1 servers="[{Suffrage:Voter ID:0ef310aa-9188-b8c1-10ff-a2b36ece5a1b Address:0ef310aa-9188-b8c1-10ff-a2b36ece5a1b} {Suffrage:Voter ID:2c6c91fe-e3d3-a39b-1341-2e854c4257d4 Address:2c6c91fe-e3d3-a39b-1341-2e854c4257d4}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:15.266Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2022-08-12T07:30:15.266Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: 2c6c91fe-e3d3-a39b-1341-2e854c4257d4: 2022-08-12T07:30:15.266Z [INFO] 2c6c91fe-e3d3-a39b-1341-2e854c4257d4: entering follower state: follower="Node at 2c6c91fe-e3d3-a39b-1341-2e854c4257d4 [Follower]" leader= + testing.go:149: 2c6c91fe-e3d3-a39b-1341-2e854c4257d4: 2022-08-12T07:30:15.266Z [INFO] 2c6c91fe-e3d3-a39b-1341-2e854c4257d4: entering follower state: follower="Node at 2c6c91fe-e3d3-a39b-1341-2e854c4257d4 [Follower]" leader= +testLoggerAdapter verbose: 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: 2022-08-12T07:30:15.334Z [WARN] 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: heartbeat timeout reached, starting election: last-leader= + testing.go:149: 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: 2022-08-12T07:30:15.334Z [WARN] 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: 2022-08-12T07:30:15.334Z [INFO] 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: entering candidate state: node="Node at 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b [Candidate]" term=2 + testing.go:149: 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: 2022-08-12T07:30:15.334Z [INFO] 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: entering candidate state: node="Node at 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b [Candidate]" term=2 +testLoggerAdapter verbose: 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: 2022-08-12T07:30:15.334Z [INFO] 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: election won: tally=2 + testing.go:149: 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: 2022-08-12T07:30:15.334Z [INFO] 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: election won: tally=2 +testLoggerAdapter verbose: 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: 2022-08-12T07:30:15.334Z [INFO] 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: entering leader state: leader="Node at 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b [Leader]" + testing.go:149: 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: 2022-08-12T07:30:15.334Z [INFO] 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: entering leader state: leader="Node at 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b [Leader]" +testLoggerAdapter verbose: 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: 2022-08-12T07:30:15.334Z [INFO] 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: added peer, starting replication: peer=2c6c91fe-e3d3-a39b-1341-2e854c4257d4 + testing.go:149: 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: 2022-08-12T07:30:15.334Z [INFO] 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: added peer, starting replication: peer=2c6c91fe-e3d3-a39b-1341-2e854c4257d4 +testLoggerAdapter verbose: 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: 2022-08-12T07:30:15.335Z [INFO] 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: pipelining replication: peer="{Voter 2c6c91fe-e3d3-a39b-1341-2e854c4257d4 2c6c91fe-e3d3-a39b-1341-2e854c4257d4}" + testing.go:149: 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: 2022-08-12T07:30:15.335Z [INFO] 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: pipelining replication: peer="{Voter 2c6c91fe-e3d3-a39b-1341-2e854c4257d4 2c6c91fe-e3d3-a39b-1341-2e854c4257d4}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:15.439Z [INFO] stable state for Follower reached at 2022-08-12 07:30:15.33486332 +0000 UTC m=+16.291549801 (1 nodes), 68.05084ms from start of poll, 68.64896ms from cluster start. Timeout at 2022-08-12 07:30:15.43990542 +0000 UTC m=+16.396591901, 105.0421ms after stability + testing.go:149: cluster: 2022-08-12T07:30:15.439Z [INFO] stable state for Follower reached at 2022-08-12 07:30:15.33486332 +0000 UTC m=+16.291549801 (1 nodes), 68.05084ms from start of poll, 68.64896ms from cluster start. Timeout at 2022-08-12 07:30:15.43990542 +0000 UTC m=+16.396591901, 105.0421ms after stability +testLoggerAdapter verbose: server-fd38b192-17a5-b476-fe4b-b44007f22fe8: 2022-08-12T07:30:15.440Z [INFO] server-fd38b192-17a5-b476-fe4b-b44007f22fe8: initial configuration: index=0 servers=[] + testing.go:149: server-fd38b192-17a5-b476-fe4b-b44007f22fe8: 2022-08-12T07:30:15.440Z [INFO] server-fd38b192-17a5-b476-fe4b-b44007f22fe8: initial configuration: index=0 servers=[] +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:15.440Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:15.440Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-fd38b192-17a5-b476-fe4b-b44007f22fe8: 2022-08-12T07:30:15.440Z [INFO] server-fd38b192-17a5-b476-fe4b-b44007f22fe8: entering follower state: follower="Node at fd38b192-17a5-b476-fe4b-b44007f22fe8 [Follower]" leader= + testing.go:149: server-fd38b192-17a5-b476-fe4b-b44007f22fe8: 2022-08-12T07:30:15.440Z [INFO] server-fd38b192-17a5-b476-fe4b-b44007f22fe8: entering follower state: follower="Node at fd38b192-17a5-b476-fe4b-b44007f22fe8 [Follower]" leader= +testLoggerAdapter verbose: server-fd38b192-17a5-b476-fe4b-b44007f22fe8: 2022-08-12T07:30:15.517Z [WARN] server-fd38b192-17a5-b476-fe4b-b44007f22fe8: no known peers, aborting election + testing.go:149: server-fd38b192-17a5-b476-fe4b-b44007f22fe8: 2022-08-12T07:30:15.517Z [WARN] server-fd38b192-17a5-b476-fe4b-b44007f22fe8: no known peers, aborting election +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:15.545Z [INFO] stable state for Leader reached at 2022-08-12 07:30:15.44063382 +0000 UTC m=+16.397320301 (1 nodes), 1.66µs from start of poll, 174.41946ms from cluster start. Timeout at 2022-08-12 07:30:15.54570056 +0000 UTC m=+16.502387041, 105.06674ms after stability + testing.go:149: cluster: 2022-08-12T07:30:15.545Z [INFO] stable state for Leader reached at 2022-08-12 07:30:15.44063382 +0000 UTC m=+16.397320301 (1 nodes), 1.66µs from start of poll, 174.41946ms from cluster start. Timeout at 2022-08-12 07:30:15.54570056 +0000 UTC m=+16.502387041, 105.06674ms after stability +testLoggerAdapter verbose: 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: 2022-08-12T07:30:15.545Z [INFO] 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: updating configuration: command=AddStaging server-id=server-fd38b192-17a5-b476-fe4b-b44007f22fe8 server-addr=fd38b192-17a5-b476-fe4b-b44007f22fe8 servers="[{Suffrage:Voter ID:0ef310aa-9188-b8c1-10ff-a2b36ece5a1b Address:0ef310aa-9188-b8c1-10ff-a2b36ece5a1b} {Suffrage:Voter ID:2c6c91fe-e3d3-a39b-1341-2e854c4257d4 Address:2c6c91fe-e3d3-a39b-1341-2e854c4257d4} {Suffrage:Voter ID:server-fd38b192-17a5-b476-fe4b-b44007f22fe8 Address:fd38b192-17a5-b476-fe4b-b44007f22fe8}]" + testing.go:149: 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: 2022-08-12T07:30:15.545Z [INFO] 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: updating configuration: command=AddStaging server-id=server-fd38b192-17a5-b476-fe4b-b44007f22fe8 server-addr=fd38b192-17a5-b476-fe4b-b44007f22fe8 servers="[{Suffrage:Voter ID:0ef310aa-9188-b8c1-10ff-a2b36ece5a1b Address:0ef310aa-9188-b8c1-10ff-a2b36ece5a1b} {Suffrage:Voter ID:2c6c91fe-e3d3-a39b-1341-2e854c4257d4 Address:2c6c91fe-e3d3-a39b-1341-2e854c4257d4} {Suffrage:Voter ID:server-fd38b192-17a5-b476-fe4b-b44007f22fe8 Address:fd38b192-17a5-b476-fe4b-b44007f22fe8}]" +testLoggerAdapter verbose: 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: 2022-08-12T07:30:15.546Z [INFO] 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: added peer, starting replication: peer=server-fd38b192-17a5-b476-fe4b-b44007f22fe8 + testing.go:149: 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: 2022-08-12T07:30:15.546Z [INFO] 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: added peer, starting replication: peer=server-fd38b192-17a5-b476-fe4b-b44007f22fe8 +testLoggerAdapter verbose: server-fd38b192-17a5-b476-fe4b-b44007f22fe8: 2022-08-12T07:30:15.546Z [WARN] server-fd38b192-17a5-b476-fe4b-b44007f22fe8: failed to get previous log: previous-index=3 last-index=0 error="log not found" + testing.go:149: server-fd38b192-17a5-b476-fe4b-b44007f22fe8: 2022-08-12T07:30:15.546Z [WARN] server-fd38b192-17a5-b476-fe4b-b44007f22fe8: failed to get previous log: previous-index=3 last-index=0 error="log not found" +testLoggerAdapter verbose: 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: 2022-08-12T07:30:15.546Z [WARN] 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: appendEntries rejected, sending older logs: peer="{Voter server-fd38b192-17a5-b476-fe4b-b44007f22fe8 fd38b192-17a5-b476-fe4b-b44007f22fe8}" next=1 + testing.go:149: 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: 2022-08-12T07:30:15.546Z [WARN] 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: appendEntries rejected, sending older logs: peer="{Voter server-fd38b192-17a5-b476-fe4b-b44007f22fe8 fd38b192-17a5-b476-fe4b-b44007f22fe8}" next=1 +testLoggerAdapter verbose: 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: 2022-08-12T07:30:15.546Z [INFO] 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: pipelining replication: peer="{Voter server-fd38b192-17a5-b476-fe4b-b44007f22fe8 fd38b192-17a5-b476-fe4b-b44007f22fe8}" + testing.go:149: 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: 2022-08-12T07:30:15.546Z [INFO] 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: pipelining replication: peer="{Voter server-fd38b192-17a5-b476-fe4b-b44007f22fe8 fd38b192-17a5-b476-fe4b-b44007f22fe8}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:15.551Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:15.551Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:15.656Z [INFO] stable state for Leader reached at 2022-08-12 07:30:15.55157874 +0000 UTC m=+16.508265221 (1 nodes), 1.62µs from start of poll, 285.36438ms from cluster start. Timeout at 2022-08-12 07:30:15.6566348 +0000 UTC m=+16.613321281, 105.05606ms after stability + testing.go:149: cluster: 2022-08-12T07:30:15.656Z [INFO] stable state for Leader reached at 2022-08-12 07:30:15.55157874 +0000 UTC m=+16.508265221 (1 nodes), 1.62µs from start of poll, 285.36438ms from cluster start. Timeout at 2022-08-12 07:30:15.6566348 +0000 UTC m=+16.613321281, 105.05606ms after stability +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:15.656Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:15.656Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:15.762Z [INFO] stable state for Leader reached at 2022-08-12 07:30:15.65686632 +0000 UTC m=+16.613552801 (1 nodes), 2.2µs from start of poll, 390.65196ms from cluster start. Timeout at 2022-08-12 07:30:15.76194404 +0000 UTC m=+16.718630521, 105.07772ms after stability + testing.go:149: cluster: 2022-08-12T07:30:15.762Z [INFO] stable state for Leader reached at 2022-08-12 07:30:15.65686632 +0000 UTC m=+16.613552801 (1 nodes), 2.2µs from start of poll, 390.65196ms from cluster start. Timeout at 2022-08-12 07:30:15.76194404 +0000 UTC m=+16.718630521, 105.07772ms after stability +testLoggerAdapter verbose: 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: 2022-08-12T07:30:15.762Z [INFO] 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: updating configuration: command=RemoveServer server-id=2c6c91fe-e3d3-a39b-1341-2e854c4257d4 server-addr= servers="[{Suffrage:Voter ID:0ef310aa-9188-b8c1-10ff-a2b36ece5a1b Address:0ef310aa-9188-b8c1-10ff-a2b36ece5a1b} {Suffrage:Voter ID:server-fd38b192-17a5-b476-fe4b-b44007f22fe8 Address:fd38b192-17a5-b476-fe4b-b44007f22fe8}]" + testing.go:149: 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: 2022-08-12T07:30:15.762Z [INFO] 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: updating configuration: command=RemoveServer server-id=2c6c91fe-e3d3-a39b-1341-2e854c4257d4 server-addr= servers="[{Suffrage:Voter ID:0ef310aa-9188-b8c1-10ff-a2b36ece5a1b Address:0ef310aa-9188-b8c1-10ff-a2b36ece5a1b} {Suffrage:Voter ID:server-fd38b192-17a5-b476-fe4b-b44007f22fe8 Address:fd38b192-17a5-b476-fe4b-b44007f22fe8}]" +testLoggerAdapter verbose: 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: 2022-08-12T07:30:15.762Z [INFO] 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: removed peer, stopping replication: peer=2c6c91fe-e3d3-a39b-1341-2e854c4257d4 last-index=4 + testing.go:149: 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: 2022-08-12T07:30:15.762Z [INFO] 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: removed peer, stopping replication: peer=2c6c91fe-e3d3-a39b-1341-2e854c4257d4 last-index=4 +testLoggerAdapter verbose: 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: 2022-08-12T07:30:15.762Z [INFO] 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: aborting pipeline replication: peer="{Voter 2c6c91fe-e3d3-a39b-1341-2e854c4257d4 2c6c91fe-e3d3-a39b-1341-2e854c4257d4}" + testing.go:149: 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: 2022-08-12T07:30:15.762Z [INFO] 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: aborting pipeline replication: peer="{Voter 2c6c91fe-e3d3-a39b-1341-2e854c4257d4 2c6c91fe-e3d3-a39b-1341-2e854c4257d4}" +testLoggerAdapter verbose: 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: 2022-08-12T07:30:15.762Z [INFO] 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: aborting pipeline replication: peer="{Voter server-fd38b192-17a5-b476-fe4b-b44007f22fe8 fd38b192-17a5-b476-fe4b-b44007f22fe8}" + testing.go:149: 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: 2022-08-12T07:30:15.762Z [INFO] 0ef310aa-9188-b8c1-10ff-a2b36ece5a1b: aborting pipeline replication: peer="{Voter server-fd38b192-17a5-b476-fe4b-b44007f22fe8 fd38b192-17a5-b476-fe4b-b44007f22fe8}" +--- PASS: TestRaft_ProtocolVersion_Upgrade_2_3 (0.50s) === RUN TestRaft_LeadershipTransferInProgress --- PASS: TestRaft_LeadershipTransferInProgress (0.00s) === RUN TestRaft_LeadershipTransferPickServer --- PASS: TestRaft_LeadershipTransferPickServer (0.00s) === RUN TestRaft_LeadershipTransfer -testLoggerAdapter verbose: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.180Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a76f879-c778-63b4-3230-37e80df5f9a7 Address:6a76f879-c778-63b4-3230-37e80df5f9a7} {Suffrage:Voter ID:server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d Address:de91a26b-f14a-bd86-fc3c-9dd7e80a495d} {Suffrage:Voter ID:server-f6290c41-6e88-8e2c-a14b-87f717de5dee Address:f6290c41-6e88-8e2c-a14b-87f717de5dee}]" - testing.go:149: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.180Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a76f879-c778-63b4-3230-37e80df5f9a7 Address:6a76f879-c778-63b4-3230-37e80df5f9a7} {Suffrage:Voter ID:server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d Address:de91a26b-f14a-bd86-fc3c-9dd7e80a495d} {Suffrage:Voter ID:server-f6290c41-6e88-8e2c-a14b-87f717de5dee Address:f6290c41-6e88-8e2c-a14b-87f717de5dee}]" -testLoggerAdapter verbose: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.181Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: entering follower state: follower="Node at 6a76f879-c778-63b4-3230-37e80df5f9a7 [Follower]" leader= - testing.go:149: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.181Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: entering follower state: follower="Node at 6a76f879-c778-63b4-3230-37e80df5f9a7 [Follower]" leader= -testLoggerAdapter verbose: server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d: 2021-07-10T01:05:11.181Z [INFO] server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a76f879-c778-63b4-3230-37e80df5f9a7 Address:6a76f879-c778-63b4-3230-37e80df5f9a7} {Suffrage:Voter ID:server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d Address:de91a26b-f14a-bd86-fc3c-9dd7e80a495d} {Suffrage:Voter ID:server-f6290c41-6e88-8e2c-a14b-87f717de5dee Address:f6290c41-6e88-8e2c-a14b-87f717de5dee}]" - testing.go:149: server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d: 2021-07-10T01:05:11.181Z [INFO] server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a76f879-c778-63b4-3230-37e80df5f9a7 Address:6a76f879-c778-63b4-3230-37e80df5f9a7} {Suffrage:Voter ID:server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d Address:de91a26b-f14a-bd86-fc3c-9dd7e80a495d} {Suffrage:Voter ID:server-f6290c41-6e88-8e2c-a14b-87f717de5dee Address:f6290c41-6e88-8e2c-a14b-87f717de5dee}]" -testLoggerAdapter verbose: server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d: 2021-07-10T01:05:11.181Z [INFO] server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d: entering follower state: follower="Node at de91a26b-f14a-bd86-fc3c-9dd7e80a495d [Follower]" leader= - testing.go:149: server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d: 2021-07-10T01:05:11.181Z [INFO] server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d: entering follower state: follower="Node at de91a26b-f14a-bd86-fc3c-9dd7e80a495d [Follower]" leader= -testLoggerAdapter verbose: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.181Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a76f879-c778-63b4-3230-37e80df5f9a7 Address:6a76f879-c778-63b4-3230-37e80df5f9a7} {Suffrage:Voter ID:server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d Address:de91a26b-f14a-bd86-fc3c-9dd7e80a495d} {Suffrage:Voter ID:server-f6290c41-6e88-8e2c-a14b-87f717de5dee Address:f6290c41-6e88-8e2c-a14b-87f717de5dee}]" - testing.go:149: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.181Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a76f879-c778-63b4-3230-37e80df5f9a7 Address:6a76f879-c778-63b4-3230-37e80df5f9a7} {Suffrage:Voter ID:server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d Address:de91a26b-f14a-bd86-fc3c-9dd7e80a495d} {Suffrage:Voter ID:server-f6290c41-6e88-8e2c-a14b-87f717de5dee Address:f6290c41-6e88-8e2c-a14b-87f717de5dee}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:11.181Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:11.181Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.181Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: entering follower state: follower="Node at f6290c41-6e88-8e2c-a14b-87f717de5dee [Follower]" leader= - testing.go:149: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.181Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: entering follower state: follower="Node at f6290c41-6e88-8e2c-a14b-87f717de5dee [Follower]" leader= -testLoggerAdapter verbose: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.245Z [WARN] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.245Z [WARN] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.245Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: entering candidate state: node="Node at f6290c41-6e88-8e2c-a14b-87f717de5dee [Candidate]" term=2 - testing.go:149: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.245Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: entering candidate state: node="Node at f6290c41-6e88-8e2c-a14b-87f717de5dee [Candidate]" term=2 -testLoggerAdapter verbose: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.246Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: election won: tally=2 - testing.go:149: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.246Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: election won: tally=2 -testLoggerAdapter verbose: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.246Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: entering leader state: leader="Node at f6290c41-6e88-8e2c-a14b-87f717de5dee [Leader]" - testing.go:149: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.246Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: entering leader state: leader="Node at f6290c41-6e88-8e2c-a14b-87f717de5dee [Leader]" -testLoggerAdapter verbose: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.246Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: added peer, starting replication: peer=server-6a76f879-c778-63b4-3230-37e80df5f9a7 - testing.go:149: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.246Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: added peer, starting replication: peer=server-6a76f879-c778-63b4-3230-37e80df5f9a7 -testLoggerAdapter verbose: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.246Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: added peer, starting replication: peer=server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d - testing.go:149: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.246Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: added peer, starting replication: peer=server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d -testLoggerAdapter verbose: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.246Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: pipelining replication: peer="{Voter server-6a76f879-c778-63b4-3230-37e80df5f9a7 6a76f879-c778-63b4-3230-37e80df5f9a7}" - testing.go:149: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.246Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: pipelining replication: peer="{Voter server-6a76f879-c778-63b4-3230-37e80df5f9a7 6a76f879-c778-63b4-3230-37e80df5f9a7}" -testLoggerAdapter verbose: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.246Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: pipelining replication: peer="{Voter server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d de91a26b-f14a-bd86-fc3c-9dd7e80a495d}" - testing.go:149: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.246Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: pipelining replication: peer="{Voter server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d de91a26b-f14a-bd86-fc3c-9dd7e80a495d}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:11.351Z [INFO] stable state for Leader reached at 2021-07-10 01:05:11.246184535 +0000 UTC m=+16.634342505 (1 nodes), 64.51988ms from start of poll, 65.438049ms from cluster start. Timeout at 2021-07-10 01:05:11.351278343 +0000 UTC m=+16.739436293, 105.093788ms after stability - testing.go:149: cluster: 2021-07-10T01:05:11.351Z [INFO] stable state for Leader reached at 2021-07-10 01:05:11.246184535 +0000 UTC m=+16.634342505 (1 nodes), 64.51988ms from start of poll, 65.438049ms from cluster start. Timeout at 2021-07-10 01:05:11.351278343 +0000 UTC m=+16.739436293, 105.093788ms after stability -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:11.351Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:11.351Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:11.456Z [INFO] stable state for Leader reached at 2021-07-10 01:05:11.351594147 +0000 UTC m=+16.739752117 (1 nodes), 2.4µs from start of poll, 170.847661ms from cluster start. Timeout at 2021-07-10 01:05:11.456649034 +0000 UTC m=+16.844807004, 105.054887ms after stability - testing.go:149: cluster: 2021-07-10T01:05:11.456Z [INFO] stable state for Leader reached at 2021-07-10 01:05:11.351594147 +0000 UTC m=+16.739752117 (1 nodes), 2.4µs from start of poll, 170.847661ms from cluster start. Timeout at 2021-07-10 01:05:11.456649034 +0000 UTC m=+16.844807004, 105.054887ms after stability -testLoggerAdapter verbose: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.456Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: entering candidate state: node="Node at 6a76f879-c778-63b4-3230-37e80df5f9a7 [Candidate]" term=3 -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:11.457Z [INFO] starting stability test: raft-state=Leader - testing.go:149: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.456Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: entering candidate state: node="Node at 6a76f879-c778-63b4-3230-37e80df5f9a7 [Candidate]" term=3 - testing.go:149: cluster: 2021-07-10T01:05:11.457Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.457Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: entering follower state: follower="Node at f6290c41-6e88-8e2c-a14b-87f717de5dee [Follower]" leader= - testing.go:149: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.457Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: entering follower state: follower="Node at f6290c41-6e88-8e2c-a14b-87f717de5dee [Follower]" leader= -testLoggerAdapter verbose: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.457Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: aborting pipeline replication: peer="{Voter server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d de91a26b-f14a-bd86-fc3c-9dd7e80a495d}" - testing.go:149: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.457Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: aborting pipeline replication: peer="{Voter server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d de91a26b-f14a-bd86-fc3c-9dd7e80a495d}" -testLoggerAdapter verbose: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.457Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: election won: tally=2 -testLoggerAdapter verbose: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.457Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: aborting pipeline replication: peer="{Voter server-6a76f879-c778-63b4-3230-37e80df5f9a7 6a76f879-c778-63b4-3230-37e80df5f9a7}" - testing.go:149: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.457Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: election won: tally=2 -testLoggerAdapter verbose: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.457Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: entering leader state: leader="Node at 6a76f879-c778-63b4-3230-37e80df5f9a7 [Leader]" - testing.go:149: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.457Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: aborting pipeline replication: peer="{Voter server-6a76f879-c778-63b4-3230-37e80df5f9a7 6a76f879-c778-63b4-3230-37e80df5f9a7}" - testing.go:149: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.457Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: entering leader state: leader="Node at 6a76f879-c778-63b4-3230-37e80df5f9a7 [Leader]" -testLoggerAdapter verbose: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.457Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: added peer, starting replication: peer=server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d - testing.go:149: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.457Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: added peer, starting replication: peer=server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d -testLoggerAdapter verbose: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.457Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: added peer, starting replication: peer=server-f6290c41-6e88-8e2c-a14b-87f717de5dee - testing.go:149: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.457Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: added peer, starting replication: peer=server-f6290c41-6e88-8e2c-a14b-87f717de5dee -testLoggerAdapter verbose: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.457Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: pipelining replication: peer="{Voter server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d de91a26b-f14a-bd86-fc3c-9dd7e80a495d}" - testing.go:149: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.457Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: pipelining replication: peer="{Voter server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d de91a26b-f14a-bd86-fc3c-9dd7e80a495d}" -testLoggerAdapter verbose: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.457Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: pipelining replication: peer="{Voter server-f6290c41-6e88-8e2c-a14b-87f717de5dee f6290c41-6e88-8e2c-a14b-87f717de5dee}" - testing.go:149: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.457Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: pipelining replication: peer="{Voter server-f6290c41-6e88-8e2c-a14b-87f717de5dee f6290c41-6e88-8e2c-a14b-87f717de5dee}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:11.562Z [INFO] stable state for Leader reached at 2021-07-10 01:05:11.457546824 +0000 UTC m=+16.845704814 (1 nodes), 320.504µs from start of poll, 276.800358ms from cluster start. Timeout at 2021-07-10 01:05:11.562643112 +0000 UTC m=+16.950801102, 105.096288ms after stability - testing.go:149: cluster: 2021-07-10T01:05:11.562Z [INFO] stable state for Leader reached at 2021-07-10 01:05:11.457546824 +0000 UTC m=+16.845704814 (1 nodes), 320.504µs from start of poll, 276.800358ms from cluster start. Timeout at 2021-07-10 01:05:11.562643112 +0000 UTC m=+16.950801102, 105.096288ms after stability -testLoggerAdapter verbose: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.562Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: aborting pipeline replication: peer="{Voter server-f6290c41-6e88-8e2c-a14b-87f717de5dee f6290c41-6e88-8e2c-a14b-87f717de5dee}" - testing.go:149: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.562Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: aborting pipeline replication: peer="{Voter server-f6290c41-6e88-8e2c-a14b-87f717de5dee f6290c41-6e88-8e2c-a14b-87f717de5dee}" -testLoggerAdapter verbose: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.563Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: aborting pipeline replication: peer="{Voter server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d de91a26b-f14a-bd86-fc3c-9dd7e80a495d}" - testing.go:149: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.563Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: aborting pipeline replication: peer="{Voter server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d de91a26b-f14a-bd86-fc3c-9dd7e80a495d}" +testLoggerAdapter verbose: server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: 2022-08-12T07:30:15.764Z [INFO] server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc Address:663080d9-3f9a-2838-8ffa-f9d6dcb7eadc} {Suffrage:Voter ID:server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef Address:8a5af767-25fb-ec8a-326f-b5221bf2a4ef} {Suffrage:Voter ID:server-ee7a67e5-0596-fb24-92ee-17b7503a586c Address:ee7a67e5-0596-fb24-92ee-17b7503a586c}]" + testing.go:149: server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: 2022-08-12T07:30:15.764Z [INFO] server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc Address:663080d9-3f9a-2838-8ffa-f9d6dcb7eadc} {Suffrage:Voter ID:server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef Address:8a5af767-25fb-ec8a-326f-b5221bf2a4ef} {Suffrage:Voter ID:server-ee7a67e5-0596-fb24-92ee-17b7503a586c Address:ee7a67e5-0596-fb24-92ee-17b7503a586c}]" +testLoggerAdapter verbose: server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: 2022-08-12T07:30:15.764Z [INFO] server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc Address:663080d9-3f9a-2838-8ffa-f9d6dcb7eadc} {Suffrage:Voter ID:server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef Address:8a5af767-25fb-ec8a-326f-b5221bf2a4ef} {Suffrage:Voter ID:server-ee7a67e5-0596-fb24-92ee-17b7503a586c Address:ee7a67e5-0596-fb24-92ee-17b7503a586c}]" + testing.go:149: server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: 2022-08-12T07:30:15.764Z [INFO] server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc Address:663080d9-3f9a-2838-8ffa-f9d6dcb7eadc} {Suffrage:Voter ID:server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef Address:8a5af767-25fb-ec8a-326f-b5221bf2a4ef} {Suffrage:Voter ID:server-ee7a67e5-0596-fb24-92ee-17b7503a586c Address:ee7a67e5-0596-fb24-92ee-17b7503a586c}]" +testLoggerAdapter verbose: server-ee7a67e5-0596-fb24-92ee-17b7503a586c: 2022-08-12T07:30:15.764Z [INFO] server-ee7a67e5-0596-fb24-92ee-17b7503a586c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc Address:663080d9-3f9a-2838-8ffa-f9d6dcb7eadc} {Suffrage:Voter ID:server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef Address:8a5af767-25fb-ec8a-326f-b5221bf2a4ef} {Suffrage:Voter ID:server-ee7a67e5-0596-fb24-92ee-17b7503a586c Address:ee7a67e5-0596-fb24-92ee-17b7503a586c}]" + testing.go:149: server-ee7a67e5-0596-fb24-92ee-17b7503a586c: 2022-08-12T07:30:15.764Z [INFO] server-ee7a67e5-0596-fb24-92ee-17b7503a586c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc Address:663080d9-3f9a-2838-8ffa-f9d6dcb7eadc} {Suffrage:Voter ID:server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef Address:8a5af767-25fb-ec8a-326f-b5221bf2a4ef} {Suffrage:Voter ID:server-ee7a67e5-0596-fb24-92ee-17b7503a586c Address:ee7a67e5-0596-fb24-92ee-17b7503a586c}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:15.764Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:15.764Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: 2022-08-12T07:30:15.764Z [INFO] server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: entering follower state: follower="Node at 663080d9-3f9a-2838-8ffa-f9d6dcb7eadc [Follower]" leader= + testing.go:149: server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: 2022-08-12T07:30:15.764Z [INFO] server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: entering follower state: follower="Node at 663080d9-3f9a-2838-8ffa-f9d6dcb7eadc [Follower]" leader= +testLoggerAdapter verbose: server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: 2022-08-12T07:30:15.764Z [INFO] server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: entering follower state: follower="Node at 8a5af767-25fb-ec8a-326f-b5221bf2a4ef [Follower]" leader= + testing.go:149: server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: 2022-08-12T07:30:15.764Z [INFO] server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: entering follower state: follower="Node at 8a5af767-25fb-ec8a-326f-b5221bf2a4ef [Follower]" leader= +testLoggerAdapter verbose: server-ee7a67e5-0596-fb24-92ee-17b7503a586c: 2022-08-12T07:30:15.765Z [INFO] server-ee7a67e5-0596-fb24-92ee-17b7503a586c: entering follower state: follower="Node at ee7a67e5-0596-fb24-92ee-17b7503a586c [Follower]" leader= + testing.go:149: server-ee7a67e5-0596-fb24-92ee-17b7503a586c: 2022-08-12T07:30:15.765Z [INFO] server-ee7a67e5-0596-fb24-92ee-17b7503a586c: entering follower state: follower="Node at ee7a67e5-0596-fb24-92ee-17b7503a586c [Follower]" leader= +testLoggerAdapter verbose: server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: 2022-08-12T07:30:15.824Z [WARN] server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: 2022-08-12T07:30:15.824Z [WARN] server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: 2022-08-12T07:30:15.825Z [INFO] server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: entering candidate state: node="Node at 663080d9-3f9a-2838-8ffa-f9d6dcb7eadc [Candidate]" term=2 + testing.go:149: server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: 2022-08-12T07:30:15.825Z [INFO] server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: entering candidate state: node="Node at 663080d9-3f9a-2838-8ffa-f9d6dcb7eadc [Candidate]" term=2 +testLoggerAdapter verbose: server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: 2022-08-12T07:30:15.825Z [INFO] server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: election won: tally=2 + testing.go:149: server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: 2022-08-12T07:30:15.825Z [INFO] server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: election won: tally=2 +testLoggerAdapter verbose: server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: 2022-08-12T07:30:15.825Z [INFO] server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: entering leader state: leader="Node at 663080d9-3f9a-2838-8ffa-f9d6dcb7eadc [Leader]" + testing.go:149: server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: 2022-08-12T07:30:15.825Z [INFO] server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: entering leader state: leader="Node at 663080d9-3f9a-2838-8ffa-f9d6dcb7eadc [Leader]" +testLoggerAdapter verbose: server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: 2022-08-12T07:30:15.825Z [INFO] server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: added peer, starting replication: peer=server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef + testing.go:149: server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: 2022-08-12T07:30:15.825Z [INFO] server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: added peer, starting replication: peer=server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef +testLoggerAdapter verbose: server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: 2022-08-12T07:30:15.825Z [INFO] server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: added peer, starting replication: peer=server-ee7a67e5-0596-fb24-92ee-17b7503a586c + testing.go:149: server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: 2022-08-12T07:30:15.825Z [INFO] server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: added peer, starting replication: peer=server-ee7a67e5-0596-fb24-92ee-17b7503a586c +testLoggerAdapter verbose: server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: 2022-08-12T07:30:15.825Z [INFO] server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: pipelining replication: peer="{Voter server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef 8a5af767-25fb-ec8a-326f-b5221bf2a4ef}" + testing.go:149: server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: 2022-08-12T07:30:15.825Z [INFO] server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: pipelining replication: peer="{Voter server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef 8a5af767-25fb-ec8a-326f-b5221bf2a4ef}" +testLoggerAdapter verbose: server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: 2022-08-12T07:30:15.825Z [INFO] server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: pipelining replication: peer="{Voter server-ee7a67e5-0596-fb24-92ee-17b7503a586c ee7a67e5-0596-fb24-92ee-17b7503a586c}" + testing.go:149: server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: 2022-08-12T07:30:15.825Z [INFO] server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: pipelining replication: peer="{Voter server-ee7a67e5-0596-fb24-92ee-17b7503a586c ee7a67e5-0596-fb24-92ee-17b7503a586c}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:15.930Z [INFO] stable state for Leader reached at 2022-08-12 07:30:15.82535744 +0000 UTC m=+16.782043901 (1 nodes), 60.61546ms from start of poll, 61.45564ms from cluster start. Timeout at 2022-08-12 07:30:15.9303802 +0000 UTC m=+16.887066701, 105.0228ms after stability + testing.go:149: cluster: 2022-08-12T07:30:15.930Z [INFO] stable state for Leader reached at 2022-08-12 07:30:15.82535744 +0000 UTC m=+16.782043901 (1 nodes), 60.61546ms from start of poll, 61.45564ms from cluster start. Timeout at 2022-08-12 07:30:15.9303802 +0000 UTC m=+16.887066701, 105.0228ms after stability +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:15.930Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:15.930Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:16.035Z [INFO] stable state for Leader reached at 2022-08-12 07:30:15.9306343 +0000 UTC m=+16.887320761 (1 nodes), 1.82µs from start of poll, 166.7325ms from cluster start. Timeout at 2022-08-12 07:30:16.03570096 +0000 UTC m=+16.992387441, 105.06668ms after stability + testing.go:149: cluster: 2022-08-12T07:30:16.035Z [INFO] stable state for Leader reached at 2022-08-12 07:30:15.9306343 +0000 UTC m=+16.887320761 (1 nodes), 1.82µs from start of poll, 166.7325ms from cluster start. Timeout at 2022-08-12 07:30:16.03570096 +0000 UTC m=+16.992387441, 105.06668ms after stability +testLoggerAdapter verbose: server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: 2022-08-12T07:30:16.036Z [INFO] server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: entering candidate state: node="Node at 8a5af767-25fb-ec8a-326f-b5221bf2a4ef [Candidate]" term=3 + testing.go:149: server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: 2022-08-12T07:30:16.036Z [INFO] server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: entering candidate state: node="Node at 8a5af767-25fb-ec8a-326f-b5221bf2a4ef [Candidate]" term=3 +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:16.036Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:16.036Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: 2022-08-12T07:30:16.036Z [INFO] server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: election won: tally=2 +testLoggerAdapter verbose: server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: 2022-08-12T07:30:16.036Z [INFO] server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: entering follower state: follower="Node at 663080d9-3f9a-2838-8ffa-f9d6dcb7eadc [Follower]" leader= + testing.go:149: server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: 2022-08-12T07:30:16.036Z [INFO] server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: election won: tally=2 +testLoggerAdapter verbose: server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: 2022-08-12T07:30:16.036Z [INFO] server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: entering leader state: leader="Node at 8a5af767-25fb-ec8a-326f-b5221bf2a4ef [Leader]" + testing.go:149: server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: 2022-08-12T07:30:16.036Z [INFO] server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: entering leader state: leader="Node at 8a5af767-25fb-ec8a-326f-b5221bf2a4ef [Leader]" +testLoggerAdapter verbose: server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: 2022-08-12T07:30:16.036Z [INFO] server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: added peer, starting replication: peer=server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc + testing.go:149: server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: 2022-08-12T07:30:16.036Z [INFO] server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: entering follower state: follower="Node at 663080d9-3f9a-2838-8ffa-f9d6dcb7eadc [Follower]" leader= +testLoggerAdapter verbose: server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: 2022-08-12T07:30:16.036Z [INFO] server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: aborting pipeline replication: peer="{Voter server-ee7a67e5-0596-fb24-92ee-17b7503a586c ee7a67e5-0596-fb24-92ee-17b7503a586c}" + testing.go:149: server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: 2022-08-12T07:30:16.036Z [INFO] server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: aborting pipeline replication: peer="{Voter server-ee7a67e5-0596-fb24-92ee-17b7503a586c ee7a67e5-0596-fb24-92ee-17b7503a586c}" + testing.go:149: server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: 2022-08-12T07:30:16.036Z [INFO] server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: added peer, starting replication: peer=server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc +testLoggerAdapter verbose: server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: 2022-08-12T07:30:16.036Z [INFO] server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: added peer, starting replication: peer=server-ee7a67e5-0596-fb24-92ee-17b7503a586c + testing.go:149: server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: 2022-08-12T07:30:16.036Z [INFO] server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: added peer, starting replication: peer=server-ee7a67e5-0596-fb24-92ee-17b7503a586c +testLoggerAdapter verbose: server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: 2022-08-12T07:30:16.036Z [INFO] server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: pipelining replication: peer="{Voter server-ee7a67e5-0596-fb24-92ee-17b7503a586c ee7a67e5-0596-fb24-92ee-17b7503a586c}" + testing.go:149: server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: 2022-08-12T07:30:16.036Z [INFO] server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: pipelining replication: peer="{Voter server-ee7a67e5-0596-fb24-92ee-17b7503a586c ee7a67e5-0596-fb24-92ee-17b7503a586c}" +testLoggerAdapter verbose: server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: 2022-08-12T07:30:16.036Z [INFO] server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: aborting pipeline replication: peer="{Voter server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef 8a5af767-25fb-ec8a-326f-b5221bf2a4ef}" + testing.go:149: server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: 2022-08-12T07:30:16.036Z [INFO] server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc: aborting pipeline replication: peer="{Voter server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef 8a5af767-25fb-ec8a-326f-b5221bf2a4ef}" +testLoggerAdapter verbose: server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: 2022-08-12T07:30:16.036Z [INFO] server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: pipelining replication: peer="{Voter server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc 663080d9-3f9a-2838-8ffa-f9d6dcb7eadc}" + testing.go:149: server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: 2022-08-12T07:30:16.036Z [INFO] server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: pipelining replication: peer="{Voter server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc 663080d9-3f9a-2838-8ffa-f9d6dcb7eadc}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:16.141Z [INFO] stable state for Leader reached at 2022-08-12 07:30:16.03631632 +0000 UTC m=+16.993002801 (1 nodes), 111.8µs from start of poll, 272.41454ms from cluster start. Timeout at 2022-08-12 07:30:16.14139578 +0000 UTC m=+17.098082241, 105.07944ms after stability + testing.go:149: cluster: 2022-08-12T07:30:16.141Z [INFO] stable state for Leader reached at 2022-08-12 07:30:16.03631632 +0000 UTC m=+16.993002801 (1 nodes), 111.8µs from start of poll, 272.41454ms from cluster start. Timeout at 2022-08-12 07:30:16.14139578 +0000 UTC m=+17.098082241, 105.07944ms after stability +testLoggerAdapter verbose: server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: 2022-08-12T07:30:16.141Z [INFO] server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: aborting pipeline replication: peer="{Voter server-ee7a67e5-0596-fb24-92ee-17b7503a586c ee7a67e5-0596-fb24-92ee-17b7503a586c}" + testing.go:149: server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: 2022-08-12T07:30:16.141Z [INFO] server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: aborting pipeline replication: peer="{Voter server-ee7a67e5-0596-fb24-92ee-17b7503a586c ee7a67e5-0596-fb24-92ee-17b7503a586c}" +testLoggerAdapter verbose: server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: 2022-08-12T07:30:16.141Z [INFO] server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: aborting pipeline replication: peer="{Voter server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc 663080d9-3f9a-2838-8ffa-f9d6dcb7eadc}" + testing.go:149: server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: 2022-08-12T07:30:16.141Z [INFO] server-8a5af767-25fb-ec8a-326f-b5221bf2a4ef: aborting pipeline replication: peer="{Voter server-663080d9-3f9a-2838-8ffa-f9d6dcb7eadc 663080d9-3f9a-2838-8ffa-f9d6dcb7eadc}" --- PASS: TestRaft_LeadershipTransfer (0.38s) === RUN TestRaft_LeadershipTransferWithOneNode -testLoggerAdapter verbose: server-674e2b05-a148-c0de-3845-668cd7394e8e: 2021-07-10T01:05:11.564Z [INFO] server-674e2b05-a148-c0de-3845-668cd7394e8e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-674e2b05-a148-c0de-3845-668cd7394e8e Address:674e2b05-a148-c0de-3845-668cd7394e8e}]" - testing.go:149: server-674e2b05-a148-c0de-3845-668cd7394e8e: 2021-07-10T01:05:11.564Z [INFO] server-674e2b05-a148-c0de-3845-668cd7394e8e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-674e2b05-a148-c0de-3845-668cd7394e8e Address:674e2b05-a148-c0de-3845-668cd7394e8e}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:11.564Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:11.564Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-674e2b05-a148-c0de-3845-668cd7394e8e: 2021-07-10T01:05:11.564Z [INFO] server-674e2b05-a148-c0de-3845-668cd7394e8e: entering follower state: follower="Node at 674e2b05-a148-c0de-3845-668cd7394e8e [Follower]" leader= - testing.go:149: server-674e2b05-a148-c0de-3845-668cd7394e8e: 2021-07-10T01:05:11.564Z [INFO] server-674e2b05-a148-c0de-3845-668cd7394e8e: entering follower state: follower="Node at 674e2b05-a148-c0de-3845-668cd7394e8e [Follower]" leader= -testLoggerAdapter verbose: server-674e2b05-a148-c0de-3845-668cd7394e8e: 2021-07-10T01:05:11.628Z [WARN] server-674e2b05-a148-c0de-3845-668cd7394e8e: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-674e2b05-a148-c0de-3845-668cd7394e8e: 2021-07-10T01:05:11.628Z [WARN] server-674e2b05-a148-c0de-3845-668cd7394e8e: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-674e2b05-a148-c0de-3845-668cd7394e8e: 2021-07-10T01:05:11.628Z [INFO] server-674e2b05-a148-c0de-3845-668cd7394e8e: entering candidate state: node="Node at 674e2b05-a148-c0de-3845-668cd7394e8e [Candidate]" term=2 - testing.go:149: server-674e2b05-a148-c0de-3845-668cd7394e8e: 2021-07-10T01:05:11.628Z [INFO] server-674e2b05-a148-c0de-3845-668cd7394e8e: entering candidate state: node="Node at 674e2b05-a148-c0de-3845-668cd7394e8e [Candidate]" term=2 -testLoggerAdapter verbose: server-674e2b05-a148-c0de-3845-668cd7394e8e: 2021-07-10T01:05:11.628Z [INFO] server-674e2b05-a148-c0de-3845-668cd7394e8e: election won: tally=1 - testing.go:149: server-674e2b05-a148-c0de-3845-668cd7394e8e: 2021-07-10T01:05:11.628Z [INFO] server-674e2b05-a148-c0de-3845-668cd7394e8e: election won: tally=1 -testLoggerAdapter verbose: server-674e2b05-a148-c0de-3845-668cd7394e8e: 2021-07-10T01:05:11.628Z [INFO] server-674e2b05-a148-c0de-3845-668cd7394e8e: entering leader state: leader="Node at 674e2b05-a148-c0de-3845-668cd7394e8e [Leader]" - testing.go:149: server-674e2b05-a148-c0de-3845-668cd7394e8e: 2021-07-10T01:05:11.628Z [INFO] server-674e2b05-a148-c0de-3845-668cd7394e8e: entering leader state: leader="Node at 674e2b05-a148-c0de-3845-668cd7394e8e [Leader]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:11.734Z [INFO] stable state for Leader reached at 2021-07-10 01:05:11.628818329 +0000 UTC m=+17.016976439 (1 nodes), 64.133956ms from start of poll, 64.632401ms from cluster start. Timeout at 2021-07-10 01:05:11.733911817 +0000 UTC m=+17.122069807, 105.093368ms after stability - testing.go:149: cluster: 2021-07-10T01:05:11.734Z [INFO] stable state for Leader reached at 2021-07-10 01:05:11.628818329 +0000 UTC m=+17.016976439 (1 nodes), 64.133956ms from start of poll, 64.632401ms from cluster start. Timeout at 2021-07-10 01:05:11.733911817 +0000 UTC m=+17.122069807, 105.093368ms after stability ---- PASS: TestRaft_LeadershipTransferWithOneNode (0.17s) +testLoggerAdapter verbose: server-c119d3ef-17b3-9526-06df-8c85bc187584: 2022-08-12T07:30:16.143Z [INFO] server-c119d3ef-17b3-9526-06df-8c85bc187584: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c119d3ef-17b3-9526-06df-8c85bc187584 Address:c119d3ef-17b3-9526-06df-8c85bc187584}]" + testing.go:149: server-c119d3ef-17b3-9526-06df-8c85bc187584: 2022-08-12T07:30:16.143Z [INFO] server-c119d3ef-17b3-9526-06df-8c85bc187584: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c119d3ef-17b3-9526-06df-8c85bc187584 Address:c119d3ef-17b3-9526-06df-8c85bc187584}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:16.143Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:16.143Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-c119d3ef-17b3-9526-06df-8c85bc187584: 2022-08-12T07:30:16.143Z [INFO] server-c119d3ef-17b3-9526-06df-8c85bc187584: entering follower state: follower="Node at c119d3ef-17b3-9526-06df-8c85bc187584 [Follower]" leader= + testing.go:149: server-c119d3ef-17b3-9526-06df-8c85bc187584: 2022-08-12T07:30:16.143Z [INFO] server-c119d3ef-17b3-9526-06df-8c85bc187584: entering follower state: follower="Node at c119d3ef-17b3-9526-06df-8c85bc187584 [Follower]" leader= +testLoggerAdapter verbose: server-c119d3ef-17b3-9526-06df-8c85bc187584: 2022-08-12T07:30:16.196Z [WARN] server-c119d3ef-17b3-9526-06df-8c85bc187584: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-c119d3ef-17b3-9526-06df-8c85bc187584: 2022-08-12T07:30:16.196Z [WARN] server-c119d3ef-17b3-9526-06df-8c85bc187584: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-c119d3ef-17b3-9526-06df-8c85bc187584: 2022-08-12T07:30:16.196Z [INFO] server-c119d3ef-17b3-9526-06df-8c85bc187584: entering candidate state: node="Node at c119d3ef-17b3-9526-06df-8c85bc187584 [Candidate]" term=2 + testing.go:149: server-c119d3ef-17b3-9526-06df-8c85bc187584: 2022-08-12T07:30:16.196Z [INFO] server-c119d3ef-17b3-9526-06df-8c85bc187584: entering candidate state: node="Node at c119d3ef-17b3-9526-06df-8c85bc187584 [Candidate]" term=2 +testLoggerAdapter verbose: server-c119d3ef-17b3-9526-06df-8c85bc187584: 2022-08-12T07:30:16.196Z [INFO] server-c119d3ef-17b3-9526-06df-8c85bc187584: election won: tally=1 + testing.go:149: server-c119d3ef-17b3-9526-06df-8c85bc187584: 2022-08-12T07:30:16.196Z [INFO] server-c119d3ef-17b3-9526-06df-8c85bc187584: election won: tally=1 +testLoggerAdapter verbose: server-c119d3ef-17b3-9526-06df-8c85bc187584: 2022-08-12T07:30:16.196Z [INFO] server-c119d3ef-17b3-9526-06df-8c85bc187584: entering leader state: leader="Node at c119d3ef-17b3-9526-06df-8c85bc187584 [Leader]" + testing.go:149: server-c119d3ef-17b3-9526-06df-8c85bc187584: 2022-08-12T07:30:16.196Z [INFO] server-c119d3ef-17b3-9526-06df-8c85bc187584: entering leader state: leader="Node at c119d3ef-17b3-9526-06df-8c85bc187584 [Leader]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:16.302Z [INFO] stable state for Leader reached at 2022-08-12 07:30:16.1969024 +0000 UTC m=+17.153588901 (1 nodes), 53.55844ms from start of poll, 54.00862ms from cluster start. Timeout at 2022-08-12 07:30:16.3019211 +0000 UTC m=+17.258607701, 105.0188ms after stability + testing.go:149: cluster: 2022-08-12T07:30:16.302Z [INFO] stable state for Leader reached at 2022-08-12 07:30:16.1969024 +0000 UTC m=+17.153588901 (1 nodes), 53.55844ms from start of poll, 54.00862ms from cluster start. Timeout at 2022-08-12 07:30:16.3019211 +0000 UTC m=+17.258607701, 105.0188ms after stability +--- PASS: TestRaft_LeadershipTransferWithOneNode (0.16s) === RUN TestRaft_LeadershipTransferWithSevenNodes -testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.736Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace9227e-5165-dd6e-c019-1d2527aa2089 Address:ace9227e-5165-dd6e-c019-1d2527aa2089} {Suffrage:Voter ID:server-1568b52b-331a-882f-74a0-d70b659fb2e4 Address:1568b52b-331a-882f-74a0-d70b659fb2e4} {Suffrage:Voter ID:server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 Address:f55d7e90-36a2-ee2a-f65d-95c1e28cdb58} {Suffrage:Voter ID:server-95d157c1-a440-c46a-0d3e-40707d4563ce Address:95d157c1-a440-c46a-0d3e-40707d4563ce} {Suffrage:Voter ID:server-20ce517a-3433-6266-3126-63cfd16a1a27 Address:20ce517a-3433-6266-3126-63cfd16a1a27} {Suffrage:Voter ID:server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 Address:a33a7d16-94dc-ff82-86ab-a4333f4b91d3} {Suffrage:Voter ID:server-744fc635-8d74-914d-af27-318108d034bd Address:744fc635-8d74-914d-af27-318108d034bd}]" - testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.736Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace9227e-5165-dd6e-c019-1d2527aa2089 Address:ace9227e-5165-dd6e-c019-1d2527aa2089} {Suffrage:Voter ID:server-1568b52b-331a-882f-74a0-d70b659fb2e4 Address:1568b52b-331a-882f-74a0-d70b659fb2e4} {Suffrage:Voter ID:server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 Address:f55d7e90-36a2-ee2a-f65d-95c1e28cdb58} {Suffrage:Voter ID:server-95d157c1-a440-c46a-0d3e-40707d4563ce Address:95d157c1-a440-c46a-0d3e-40707d4563ce} {Suffrage:Voter ID:server-20ce517a-3433-6266-3126-63cfd16a1a27 Address:20ce517a-3433-6266-3126-63cfd16a1a27} {Suffrage:Voter ID:server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 Address:a33a7d16-94dc-ff82-86ab-a4333f4b91d3} {Suffrage:Voter ID:server-744fc635-8d74-914d-af27-318108d034bd Address:744fc635-8d74-914d-af27-318108d034bd}]" -testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.736Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: entering follower state: follower="Node at ace9227e-5165-dd6e-c019-1d2527aa2089 [Follower]" leader= - testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.736Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: entering follower state: follower="Node at ace9227e-5165-dd6e-c019-1d2527aa2089 [Follower]" leader= -testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:11.737Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace9227e-5165-dd6e-c019-1d2527aa2089 Address:ace9227e-5165-dd6e-c019-1d2527aa2089} {Suffrage:Voter ID:server-1568b52b-331a-882f-74a0-d70b659fb2e4 Address:1568b52b-331a-882f-74a0-d70b659fb2e4} {Suffrage:Voter ID:server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 Address:f55d7e90-36a2-ee2a-f65d-95c1e28cdb58} {Suffrage:Voter ID:server-95d157c1-a440-c46a-0d3e-40707d4563ce Address:95d157c1-a440-c46a-0d3e-40707d4563ce} {Suffrage:Voter ID:server-20ce517a-3433-6266-3126-63cfd16a1a27 Address:20ce517a-3433-6266-3126-63cfd16a1a27} {Suffrage:Voter ID:server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 Address:a33a7d16-94dc-ff82-86ab-a4333f4b91d3} {Suffrage:Voter ID:server-744fc635-8d74-914d-af27-318108d034bd Address:744fc635-8d74-914d-af27-318108d034bd}]" - testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:11.737Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace9227e-5165-dd6e-c019-1d2527aa2089 Address:ace9227e-5165-dd6e-c019-1d2527aa2089} {Suffrage:Voter ID:server-1568b52b-331a-882f-74a0-d70b659fb2e4 Address:1568b52b-331a-882f-74a0-d70b659fb2e4} {Suffrage:Voter ID:server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 Address:f55d7e90-36a2-ee2a-f65d-95c1e28cdb58} {Suffrage:Voter ID:server-95d157c1-a440-c46a-0d3e-40707d4563ce Address:95d157c1-a440-c46a-0d3e-40707d4563ce} {Suffrage:Voter ID:server-20ce517a-3433-6266-3126-63cfd16a1a27 Address:20ce517a-3433-6266-3126-63cfd16a1a27} {Suffrage:Voter ID:server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 Address:a33a7d16-94dc-ff82-86ab-a4333f4b91d3} {Suffrage:Voter ID:server-744fc635-8d74-914d-af27-318108d034bd Address:744fc635-8d74-914d-af27-318108d034bd}]" -testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:11.737Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: entering follower state: follower="Node at 1568b52b-331a-882f-74a0-d70b659fb2e4 [Follower]" leader= - testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:11.737Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: entering follower state: follower="Node at 1568b52b-331a-882f-74a0-d70b659fb2e4 [Follower]" leader= -testLoggerAdapter verbose: server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58: 2021-07-10T01:05:11.737Z [INFO] server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace9227e-5165-dd6e-c019-1d2527aa2089 Address:ace9227e-5165-dd6e-c019-1d2527aa2089} {Suffrage:Voter ID:server-1568b52b-331a-882f-74a0-d70b659fb2e4 Address:1568b52b-331a-882f-74a0-d70b659fb2e4} {Suffrage:Voter ID:server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 Address:f55d7e90-36a2-ee2a-f65d-95c1e28cdb58} {Suffrage:Voter ID:server-95d157c1-a440-c46a-0d3e-40707d4563ce Address:95d157c1-a440-c46a-0d3e-40707d4563ce} {Suffrage:Voter ID:server-20ce517a-3433-6266-3126-63cfd16a1a27 Address:20ce517a-3433-6266-3126-63cfd16a1a27} {Suffrage:Voter ID:server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 Address:a33a7d16-94dc-ff82-86ab-a4333f4b91d3} {Suffrage:Voter ID:server-744fc635-8d74-914d-af27-318108d034bd Address:744fc635-8d74-914d-af27-318108d034bd}]" - testing.go:149: server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58: 2021-07-10T01:05:11.737Z [INFO] server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace9227e-5165-dd6e-c019-1d2527aa2089 Address:ace9227e-5165-dd6e-c019-1d2527aa2089} {Suffrage:Voter ID:server-1568b52b-331a-882f-74a0-d70b659fb2e4 Address:1568b52b-331a-882f-74a0-d70b659fb2e4} {Suffrage:Voter ID:server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 Address:f55d7e90-36a2-ee2a-f65d-95c1e28cdb58} {Suffrage:Voter ID:server-95d157c1-a440-c46a-0d3e-40707d4563ce Address:95d157c1-a440-c46a-0d3e-40707d4563ce} {Suffrage:Voter ID:server-20ce517a-3433-6266-3126-63cfd16a1a27 Address:20ce517a-3433-6266-3126-63cfd16a1a27} {Suffrage:Voter ID:server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 Address:a33a7d16-94dc-ff82-86ab-a4333f4b91d3} {Suffrage:Voter ID:server-744fc635-8d74-914d-af27-318108d034bd Address:744fc635-8d74-914d-af27-318108d034bd}]" -testLoggerAdapter verbose: server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58: 2021-07-10T01:05:11.737Z [INFO] server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58: entering follower state: follower="Node at f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 [Follower]" leader= - testing.go:149: server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58: 2021-07-10T01:05:11.737Z [INFO] server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58: entering follower state: follower="Node at f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 [Follower]" leader= -testLoggerAdapter verbose: server-95d157c1-a440-c46a-0d3e-40707d4563ce: 2021-07-10T01:05:11.738Z [INFO] server-95d157c1-a440-c46a-0d3e-40707d4563ce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace9227e-5165-dd6e-c019-1d2527aa2089 Address:ace9227e-5165-dd6e-c019-1d2527aa2089} {Suffrage:Voter ID:server-1568b52b-331a-882f-74a0-d70b659fb2e4 Address:1568b52b-331a-882f-74a0-d70b659fb2e4} {Suffrage:Voter ID:server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 Address:f55d7e90-36a2-ee2a-f65d-95c1e28cdb58} {Suffrage:Voter ID:server-95d157c1-a440-c46a-0d3e-40707d4563ce Address:95d157c1-a440-c46a-0d3e-40707d4563ce} {Suffrage:Voter ID:server-20ce517a-3433-6266-3126-63cfd16a1a27 Address:20ce517a-3433-6266-3126-63cfd16a1a27} {Suffrage:Voter ID:server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 Address:a33a7d16-94dc-ff82-86ab-a4333f4b91d3} {Suffrage:Voter ID:server-744fc635-8d74-914d-af27-318108d034bd Address:744fc635-8d74-914d-af27-318108d034bd}]" - testing.go:149: server-95d157c1-a440-c46a-0d3e-40707d4563ce: 2021-07-10T01:05:11.738Z [INFO] server-95d157c1-a440-c46a-0d3e-40707d4563ce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace9227e-5165-dd6e-c019-1d2527aa2089 Address:ace9227e-5165-dd6e-c019-1d2527aa2089} {Suffrage:Voter ID:server-1568b52b-331a-882f-74a0-d70b659fb2e4 Address:1568b52b-331a-882f-74a0-d70b659fb2e4} {Suffrage:Voter ID:server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 Address:f55d7e90-36a2-ee2a-f65d-95c1e28cdb58} {Suffrage:Voter ID:server-95d157c1-a440-c46a-0d3e-40707d4563ce Address:95d157c1-a440-c46a-0d3e-40707d4563ce} {Suffrage:Voter ID:server-20ce517a-3433-6266-3126-63cfd16a1a27 Address:20ce517a-3433-6266-3126-63cfd16a1a27} {Suffrage:Voter ID:server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 Address:a33a7d16-94dc-ff82-86ab-a4333f4b91d3} {Suffrage:Voter ID:server-744fc635-8d74-914d-af27-318108d034bd Address:744fc635-8d74-914d-af27-318108d034bd}]" -testLoggerAdapter verbose: server-95d157c1-a440-c46a-0d3e-40707d4563ce: 2021-07-10T01:05:11.738Z [INFO] server-95d157c1-a440-c46a-0d3e-40707d4563ce: entering follower state: follower="Node at 95d157c1-a440-c46a-0d3e-40707d4563ce [Follower]" leader= - testing.go:149: server-95d157c1-a440-c46a-0d3e-40707d4563ce: 2021-07-10T01:05:11.738Z [INFO] server-95d157c1-a440-c46a-0d3e-40707d4563ce: entering follower state: follower="Node at 95d157c1-a440-c46a-0d3e-40707d4563ce [Follower]" leader= -testLoggerAdapter verbose: server-20ce517a-3433-6266-3126-63cfd16a1a27: 2021-07-10T01:05:11.738Z [INFO] server-20ce517a-3433-6266-3126-63cfd16a1a27: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace9227e-5165-dd6e-c019-1d2527aa2089 Address:ace9227e-5165-dd6e-c019-1d2527aa2089} {Suffrage:Voter ID:server-1568b52b-331a-882f-74a0-d70b659fb2e4 Address:1568b52b-331a-882f-74a0-d70b659fb2e4} {Suffrage:Voter ID:server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 Address:f55d7e90-36a2-ee2a-f65d-95c1e28cdb58} {Suffrage:Voter ID:server-95d157c1-a440-c46a-0d3e-40707d4563ce Address:95d157c1-a440-c46a-0d3e-40707d4563ce} {Suffrage:Voter ID:server-20ce517a-3433-6266-3126-63cfd16a1a27 Address:20ce517a-3433-6266-3126-63cfd16a1a27} {Suffrage:Voter ID:server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 Address:a33a7d16-94dc-ff82-86ab-a4333f4b91d3} {Suffrage:Voter ID:server-744fc635-8d74-914d-af27-318108d034bd Address:744fc635-8d74-914d-af27-318108d034bd}]" - testing.go:149: server-20ce517a-3433-6266-3126-63cfd16a1a27: 2021-07-10T01:05:11.738Z [INFO] server-20ce517a-3433-6266-3126-63cfd16a1a27: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace9227e-5165-dd6e-c019-1d2527aa2089 Address:ace9227e-5165-dd6e-c019-1d2527aa2089} {Suffrage:Voter ID:server-1568b52b-331a-882f-74a0-d70b659fb2e4 Address:1568b52b-331a-882f-74a0-d70b659fb2e4} {Suffrage:Voter ID:server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 Address:f55d7e90-36a2-ee2a-f65d-95c1e28cdb58} {Suffrage:Voter ID:server-95d157c1-a440-c46a-0d3e-40707d4563ce Address:95d157c1-a440-c46a-0d3e-40707d4563ce} {Suffrage:Voter ID:server-20ce517a-3433-6266-3126-63cfd16a1a27 Address:20ce517a-3433-6266-3126-63cfd16a1a27} {Suffrage:Voter ID:server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 Address:a33a7d16-94dc-ff82-86ab-a4333f4b91d3} {Suffrage:Voter ID:server-744fc635-8d74-914d-af27-318108d034bd Address:744fc635-8d74-914d-af27-318108d034bd}]" -testLoggerAdapter verbose: server-20ce517a-3433-6266-3126-63cfd16a1a27: 2021-07-10T01:05:11.738Z [INFO] server-20ce517a-3433-6266-3126-63cfd16a1a27: entering follower state: follower="Node at 20ce517a-3433-6266-3126-63cfd16a1a27 [Follower]" leader= - testing.go:149: server-20ce517a-3433-6266-3126-63cfd16a1a27: 2021-07-10T01:05:11.738Z [INFO] server-20ce517a-3433-6266-3126-63cfd16a1a27: entering follower state: follower="Node at 20ce517a-3433-6266-3126-63cfd16a1a27 [Follower]" leader= -testLoggerAdapter verbose: server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3: 2021-07-10T01:05:11.738Z [INFO] server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace9227e-5165-dd6e-c019-1d2527aa2089 Address:ace9227e-5165-dd6e-c019-1d2527aa2089} {Suffrage:Voter ID:server-1568b52b-331a-882f-74a0-d70b659fb2e4 Address:1568b52b-331a-882f-74a0-d70b659fb2e4} {Suffrage:Voter ID:server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 Address:f55d7e90-36a2-ee2a-f65d-95c1e28cdb58} {Suffrage:Voter ID:server-95d157c1-a440-c46a-0d3e-40707d4563ce Address:95d157c1-a440-c46a-0d3e-40707d4563ce} {Suffrage:Voter ID:server-20ce517a-3433-6266-3126-63cfd16a1a27 Address:20ce517a-3433-6266-3126-63cfd16a1a27} {Suffrage:Voter ID:server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 Address:a33a7d16-94dc-ff82-86ab-a4333f4b91d3} {Suffrage:Voter ID:server-744fc635-8d74-914d-af27-318108d034bd Address:744fc635-8d74-914d-af27-318108d034bd}]" - testing.go:149: server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3: 2021-07-10T01:05:11.738Z [INFO] server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace9227e-5165-dd6e-c019-1d2527aa2089 Address:ace9227e-5165-dd6e-c019-1d2527aa2089} {Suffrage:Voter ID:server-1568b52b-331a-882f-74a0-d70b659fb2e4 Address:1568b52b-331a-882f-74a0-d70b659fb2e4} {Suffrage:Voter ID:server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 Address:f55d7e90-36a2-ee2a-f65d-95c1e28cdb58} {Suffrage:Voter ID:server-95d157c1-a440-c46a-0d3e-40707d4563ce Address:95d157c1-a440-c46a-0d3e-40707d4563ce} {Suffrage:Voter ID:server-20ce517a-3433-6266-3126-63cfd16a1a27 Address:20ce517a-3433-6266-3126-63cfd16a1a27} {Suffrage:Voter ID:server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 Address:a33a7d16-94dc-ff82-86ab-a4333f4b91d3} {Suffrage:Voter ID:server-744fc635-8d74-914d-af27-318108d034bd Address:744fc635-8d74-914d-af27-318108d034bd}]" -testLoggerAdapter verbose: server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3: 2021-07-10T01:05:11.738Z [INFO] server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3: entering follower state: follower="Node at a33a7d16-94dc-ff82-86ab-a4333f4b91d3 [Follower]" leader= - testing.go:149: server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3: 2021-07-10T01:05:11.738Z [INFO] server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3: entering follower state: follower="Node at a33a7d16-94dc-ff82-86ab-a4333f4b91d3 [Follower]" leader= -testLoggerAdapter verbose: server-744fc635-8d74-914d-af27-318108d034bd: 2021-07-10T01:05:11.738Z [INFO] server-744fc635-8d74-914d-af27-318108d034bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace9227e-5165-dd6e-c019-1d2527aa2089 Address:ace9227e-5165-dd6e-c019-1d2527aa2089} {Suffrage:Voter ID:server-1568b52b-331a-882f-74a0-d70b659fb2e4 Address:1568b52b-331a-882f-74a0-d70b659fb2e4} {Suffrage:Voter ID:server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 Address:f55d7e90-36a2-ee2a-f65d-95c1e28cdb58} {Suffrage:Voter ID:server-95d157c1-a440-c46a-0d3e-40707d4563ce Address:95d157c1-a440-c46a-0d3e-40707d4563ce} {Suffrage:Voter ID:server-20ce517a-3433-6266-3126-63cfd16a1a27 Address:20ce517a-3433-6266-3126-63cfd16a1a27} {Suffrage:Voter ID:server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 Address:a33a7d16-94dc-ff82-86ab-a4333f4b91d3} {Suffrage:Voter ID:server-744fc635-8d74-914d-af27-318108d034bd Address:744fc635-8d74-914d-af27-318108d034bd}]" - testing.go:149: server-744fc635-8d74-914d-af27-318108d034bd: 2021-07-10T01:05:11.738Z [INFO] server-744fc635-8d74-914d-af27-318108d034bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace9227e-5165-dd6e-c019-1d2527aa2089 Address:ace9227e-5165-dd6e-c019-1d2527aa2089} {Suffrage:Voter ID:server-1568b52b-331a-882f-74a0-d70b659fb2e4 Address:1568b52b-331a-882f-74a0-d70b659fb2e4} {Suffrage:Voter ID:server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 Address:f55d7e90-36a2-ee2a-f65d-95c1e28cdb58} {Suffrage:Voter ID:server-95d157c1-a440-c46a-0d3e-40707d4563ce Address:95d157c1-a440-c46a-0d3e-40707d4563ce} {Suffrage:Voter ID:server-20ce517a-3433-6266-3126-63cfd16a1a27 Address:20ce517a-3433-6266-3126-63cfd16a1a27} {Suffrage:Voter ID:server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 Address:a33a7d16-94dc-ff82-86ab-a4333f4b91d3} {Suffrage:Voter ID:server-744fc635-8d74-914d-af27-318108d034bd Address:744fc635-8d74-914d-af27-318108d034bd}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:11.739Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:11.739Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-744fc635-8d74-914d-af27-318108d034bd: 2021-07-10T01:05:11.739Z [INFO] server-744fc635-8d74-914d-af27-318108d034bd: entering follower state: follower="Node at 744fc635-8d74-914d-af27-318108d034bd [Follower]" leader= - testing.go:149: server-744fc635-8d74-914d-af27-318108d034bd: 2021-07-10T01:05:11.739Z [INFO] server-744fc635-8d74-914d-af27-318108d034bd: entering follower state: follower="Node at 744fc635-8d74-914d-af27-318108d034bd [Follower]" leader= -testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.796Z [WARN] server-ace9227e-5165-dd6e-c019-1d2527aa2089: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.796Z [WARN] server-ace9227e-5165-dd6e-c019-1d2527aa2089: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: entering candidate state: node="Node at ace9227e-5165-dd6e-c019-1d2527aa2089 [Candidate]" term=2 - testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: entering candidate state: node="Node at ace9227e-5165-dd6e-c019-1d2527aa2089 [Candidate]" term=2 -testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: election won: tally=4 - testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: election won: tally=4 -testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: entering leader state: leader="Node at ace9227e-5165-dd6e-c019-1d2527aa2089 [Leader]" - testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: entering leader state: leader="Node at ace9227e-5165-dd6e-c019-1d2527aa2089 [Leader]" -testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: added peer, starting replication: peer=server-1568b52b-331a-882f-74a0-d70b659fb2e4 - testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: added peer, starting replication: peer=server-1568b52b-331a-882f-74a0-d70b659fb2e4 -testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: added peer, starting replication: peer=server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 - testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: added peer, starting replication: peer=server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 -testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: added peer, starting replication: peer=server-95d157c1-a440-c46a-0d3e-40707d4563ce - testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: added peer, starting replication: peer=server-95d157c1-a440-c46a-0d3e-40707d4563ce -testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: added peer, starting replication: peer=server-20ce517a-3433-6266-3126-63cfd16a1a27 - testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: added peer, starting replication: peer=server-20ce517a-3433-6266-3126-63cfd16a1a27 -testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: added peer, starting replication: peer=server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 - testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: added peer, starting replication: peer=server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 -testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: pipelining replication: peer="{Voter server-95d157c1-a440-c46a-0d3e-40707d4563ce 95d157c1-a440-c46a-0d3e-40707d4563ce}" - testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: pipelining replication: peer="{Voter server-95d157c1-a440-c46a-0d3e-40707d4563ce 95d157c1-a440-c46a-0d3e-40707d4563ce}" -testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: pipelining replication: peer="{Voter server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 f55d7e90-36a2-ee2a-f65d-95c1e28cdb58}" - testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: pipelining replication: peer="{Voter server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 f55d7e90-36a2-ee2a-f65d-95c1e28cdb58}" -testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: added peer, starting replication: peer=server-744fc635-8d74-914d-af27-318108d034bd - testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: added peer, starting replication: peer=server-744fc635-8d74-914d-af27-318108d034bd -testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: pipelining replication: peer="{Voter server-1568b52b-331a-882f-74a0-d70b659fb2e4 1568b52b-331a-882f-74a0-d70b659fb2e4}" - testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: pipelining replication: peer="{Voter server-1568b52b-331a-882f-74a0-d70b659fb2e4 1568b52b-331a-882f-74a0-d70b659fb2e4}" -testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.798Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: pipelining replication: peer="{Voter server-744fc635-8d74-914d-af27-318108d034bd 744fc635-8d74-914d-af27-318108d034bd}" - testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.798Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: pipelining replication: peer="{Voter server-744fc635-8d74-914d-af27-318108d034bd 744fc635-8d74-914d-af27-318108d034bd}" -testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: pipelining replication: peer="{Voter server-20ce517a-3433-6266-3126-63cfd16a1a27 20ce517a-3433-6266-3126-63cfd16a1a27}" - testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: pipelining replication: peer="{Voter server-20ce517a-3433-6266-3126-63cfd16a1a27 20ce517a-3433-6266-3126-63cfd16a1a27}" -testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: pipelining replication: peer="{Voter server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 a33a7d16-94dc-ff82-86ab-a4333f4b91d3}" - testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: pipelining replication: peer="{Voter server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 a33a7d16-94dc-ff82-86ab-a4333f4b91d3}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:11.902Z [INFO] stable state for Leader reached at 2021-07-10 01:05:11.797518348 +0000 UTC m=+17.185676318 (1 nodes), 58.388076ms from start of poll, 61.897213ms from cluster start. Timeout at 2021-07-10 01:05:11.902583956 +0000 UTC m=+17.290741926, 105.065608ms after stability - testing.go:149: cluster: 2021-07-10T01:05:11.902Z [INFO] stable state for Leader reached at 2021-07-10 01:05:11.797518348 +0000 UTC m=+17.185676318 (1 nodes), 58.388076ms from start of poll, 61.897213ms from cluster start. Timeout at 2021-07-10 01:05:11.902583956 +0000 UTC m=+17.290741926, 105.065608ms after stability -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:11.902Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-07-10T01:05:11.902Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:12.008Z [INFO] stable state for Follower reached at 2021-07-10 01:05:11.902839378 +0000 UTC m=+17.290997328 (6 nodes), 4.7µs from start of poll, 167.218223ms from cluster start. Timeout at 2021-07-10 01:05:12.007918026 +0000 UTC m=+17.396076016, 105.078688ms after stability - testing.go:149: cluster: 2021-07-10T01:05:12.008Z [INFO] stable state for Follower reached at 2021-07-10 01:05:11.902839378 +0000 UTC m=+17.290997328 (6 nodes), 4.7µs from start of poll, 167.218223ms from cluster start. Timeout at 2021-07-10 01:05:12.007918026 +0000 UTC m=+17.396076016, 105.078688ms after stability -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:12.008Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:12.008Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:12.113Z [INFO] stable state for Leader reached at 2021-07-10 01:05:12.008201189 +0000 UTC m=+17.396359139 (1 nodes), 2.78µs from start of poll, 272.580034ms from cluster start. Timeout at 2021-07-10 01:05:12.113270557 +0000 UTC m=+17.501428547, 105.069408ms after stability - testing.go:149: cluster: 2021-07-10T01:05:12.113Z [INFO] stable state for Leader reached at 2021-07-10 01:05:12.008201189 +0000 UTC m=+17.396359139 (1 nodes), 2.78µs from start of poll, 272.580034ms from cluster start. Timeout at 2021-07-10 01:05:12.113270557 +0000 UTC m=+17.501428547, 105.069408ms after stability -testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.113Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: entering candidate state: node="Node at 1568b52b-331a-882f-74a0-d70b659fb2e4 [Candidate]" term=3 - testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.113Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: entering candidate state: node="Node at 1568b52b-331a-882f-74a0-d70b659fb2e4 [Candidate]" term=3 -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:12.113Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:12.113Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:12.113Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: entering follower state: follower="Node at ace9227e-5165-dd6e-c019-1d2527aa2089 [Follower]" leader= - testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:12.113Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: entering follower state: follower="Node at ace9227e-5165-dd6e-c019-1d2527aa2089 [Follower]" leader= -testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.113Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: election won: tally=4 - testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.113Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: election won: tally=4 -testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:12.113Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: aborting pipeline replication: peer="{Voter server-744fc635-8d74-914d-af27-318108d034bd 744fc635-8d74-914d-af27-318108d034bd}" -testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.113Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: entering leader state: leader="Node at 1568b52b-331a-882f-74a0-d70b659fb2e4 [Leader]" - testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:12.113Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: aborting pipeline replication: peer="{Voter server-744fc635-8d74-914d-af27-318108d034bd 744fc635-8d74-914d-af27-318108d034bd}" -testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:12.114Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: aborting pipeline replication: peer="{Voter server-95d157c1-a440-c46a-0d3e-40707d4563ce 95d157c1-a440-c46a-0d3e-40707d4563ce}" - testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.113Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: entering leader state: leader="Node at 1568b52b-331a-882f-74a0-d70b659fb2e4 [Leader]" -testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: added peer, starting replication: peer=server-ace9227e-5165-dd6e-c019-1d2527aa2089 - testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: added peer, starting replication: peer=server-ace9227e-5165-dd6e-c019-1d2527aa2089 -testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: added peer, starting replication: peer=server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 - testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:12.114Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: aborting pipeline replication: peer="{Voter server-95d157c1-a440-c46a-0d3e-40707d4563ce 95d157c1-a440-c46a-0d3e-40707d4563ce}" - testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: added peer, starting replication: peer=server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 -testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: added peer, starting replication: peer=server-95d157c1-a440-c46a-0d3e-40707d4563ce -testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:12.114Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: aborting pipeline replication: peer="{Voter server-20ce517a-3433-6266-3126-63cfd16a1a27 20ce517a-3433-6266-3126-63cfd16a1a27}" - testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: added peer, starting replication: peer=server-95d157c1-a440-c46a-0d3e-40707d4563ce -testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: added peer, starting replication: peer=server-20ce517a-3433-6266-3126-63cfd16a1a27 - testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: added peer, starting replication: peer=server-20ce517a-3433-6266-3126-63cfd16a1a27 -testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: added peer, starting replication: peer=server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 - testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: added peer, starting replication: peer=server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 - testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:12.114Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: aborting pipeline replication: peer="{Voter server-20ce517a-3433-6266-3126-63cfd16a1a27 20ce517a-3433-6266-3126-63cfd16a1a27}" -testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: pipelining replication: peer="{Voter server-ace9227e-5165-dd6e-c019-1d2527aa2089 ace9227e-5165-dd6e-c019-1d2527aa2089}" -testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:12.114Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: aborting pipeline replication: peer="{Voter server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 f55d7e90-36a2-ee2a-f65d-95c1e28cdb58}" - testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: pipelining replication: peer="{Voter server-ace9227e-5165-dd6e-c019-1d2527aa2089 ace9227e-5165-dd6e-c019-1d2527aa2089}" - testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:12.114Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: aborting pipeline replication: peer="{Voter server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 f55d7e90-36a2-ee2a-f65d-95c1e28cdb58}" -testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: pipelining replication: peer="{Voter server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 f55d7e90-36a2-ee2a-f65d-95c1e28cdb58}" - testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: pipelining replication: peer="{Voter server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 f55d7e90-36a2-ee2a-f65d-95c1e28cdb58}" -testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:12.114Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: aborting pipeline replication: peer="{Voter server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 a33a7d16-94dc-ff82-86ab-a4333f4b91d3}" -testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: pipelining replication: peer="{Voter server-95d157c1-a440-c46a-0d3e-40707d4563ce 95d157c1-a440-c46a-0d3e-40707d4563ce}" - testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:12.114Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: aborting pipeline replication: peer="{Voter server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 a33a7d16-94dc-ff82-86ab-a4333f4b91d3}" - testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: pipelining replication: peer="{Voter server-95d157c1-a440-c46a-0d3e-40707d4563ce 95d157c1-a440-c46a-0d3e-40707d4563ce}" -testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:12.114Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: aborting pipeline replication: peer="{Voter server-1568b52b-331a-882f-74a0-d70b659fb2e4 1568b52b-331a-882f-74a0-d70b659fb2e4}" - testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:12.114Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: aborting pipeline replication: peer="{Voter server-1568b52b-331a-882f-74a0-d70b659fb2e4 1568b52b-331a-882f-74a0-d70b659fb2e4}" -testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: added peer, starting replication: peer=server-744fc635-8d74-914d-af27-318108d034bd - testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: added peer, starting replication: peer=server-744fc635-8d74-914d-af27-318108d034bd -testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: pipelining replication: peer="{Voter server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 a33a7d16-94dc-ff82-86ab-a4333f4b91d3}" - testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: pipelining replication: peer="{Voter server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 a33a7d16-94dc-ff82-86ab-a4333f4b91d3}" -testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: pipelining replication: peer="{Voter server-20ce517a-3433-6266-3126-63cfd16a1a27 20ce517a-3433-6266-3126-63cfd16a1a27}" - testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: pipelining replication: peer="{Voter server-20ce517a-3433-6266-3126-63cfd16a1a27 20ce517a-3433-6266-3126-63cfd16a1a27}" -testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.115Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: pipelining replication: peer="{Voter server-744fc635-8d74-914d-af27-318108d034bd 744fc635-8d74-914d-af27-318108d034bd}" - testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.115Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: pipelining replication: peer="{Voter server-744fc635-8d74-914d-af27-318108d034bd 744fc635-8d74-914d-af27-318108d034bd}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:12.219Z [INFO] stable state for Leader reached at 2021-07-10 01:05:12.114044125 +0000 UTC m=+17.502202075 (1 nodes), 198.922µs from start of poll, 378.42297ms from cluster start. Timeout at 2021-07-10 01:05:12.219131533 +0000 UTC m=+17.607289523, 105.087448ms after stability - testing.go:149: cluster: 2021-07-10T01:05:12.219Z [INFO] stable state for Leader reached at 2021-07-10 01:05:12.114044125 +0000 UTC m=+17.502202075 (1 nodes), 198.922µs from start of poll, 378.42297ms from cluster start. Timeout at 2021-07-10 01:05:12.219131533 +0000 UTC m=+17.607289523, 105.087448ms after stability -testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.219Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: aborting pipeline replication: peer="{Voter server-744fc635-8d74-914d-af27-318108d034bd 744fc635-8d74-914d-af27-318108d034bd}" - testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.219Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: aborting pipeline replication: peer="{Voter server-744fc635-8d74-914d-af27-318108d034bd 744fc635-8d74-914d-af27-318108d034bd}" -testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.219Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: aborting pipeline replication: peer="{Voter server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 f55d7e90-36a2-ee2a-f65d-95c1e28cdb58}" - testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.219Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: aborting pipeline replication: peer="{Voter server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 f55d7e90-36a2-ee2a-f65d-95c1e28cdb58}" -testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.219Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: aborting pipeline replication: peer="{Voter server-20ce517a-3433-6266-3126-63cfd16a1a27 20ce517a-3433-6266-3126-63cfd16a1a27}" - testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.219Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: aborting pipeline replication: peer="{Voter server-20ce517a-3433-6266-3126-63cfd16a1a27 20ce517a-3433-6266-3126-63cfd16a1a27}" -testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.219Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: aborting pipeline replication: peer="{Voter server-95d157c1-a440-c46a-0d3e-40707d4563ce 95d157c1-a440-c46a-0d3e-40707d4563ce}" - testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.219Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: aborting pipeline replication: peer="{Voter server-95d157c1-a440-c46a-0d3e-40707d4563ce 95d157c1-a440-c46a-0d3e-40707d4563ce}" -testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.219Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: aborting pipeline replication: peer="{Voter server-ace9227e-5165-dd6e-c019-1d2527aa2089 ace9227e-5165-dd6e-c019-1d2527aa2089}" - testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.219Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: aborting pipeline replication: peer="{Voter server-ace9227e-5165-dd6e-c019-1d2527aa2089 ace9227e-5165-dd6e-c019-1d2527aa2089}" -testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.219Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: aborting pipeline replication: peer="{Voter server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 a33a7d16-94dc-ff82-86ab-a4333f4b91d3}" - testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.219Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: aborting pipeline replication: peer="{Voter server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 a33a7d16-94dc-ff82-86ab-a4333f4b91d3}" ---- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.49s) +testLoggerAdapter verbose: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.304Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1 Address:8e971f64-dc8b-a54b-2f89-68d8dd1a47d1} {Suffrage:Voter ID:server-b52f85b0-5bd7-31d4-3c29-7bc6a396394e Address:b52f85b0-5bd7-31d4-3c29-7bc6a396394e} {Suffrage:Voter ID:server-d05667eb-6bbc-8a36-30b9-cef813c37ace Address:d05667eb-6bbc-8a36-30b9-cef813c37ace} {Suffrage:Voter ID:server-7ce62693-b3eb-604e-f602-033dbfeba6cf Address:7ce62693-b3eb-604e-f602-033dbfeba6cf} {Suffrage:Voter ID:server-42117aa1-e11d-0566-89a5-775c4e02549b Address:42117aa1-e11d-0566-89a5-775c4e02549b} {Suffrage:Voter ID:server-e03b2c08-8230-f155-5dec-d07c132b9733 Address:e03b2c08-8230-f155-5dec-d07c132b9733} {Suffrage:Voter ID:server-33df0ac0-3898-14dd-d251-17a716819550 Address:33df0ac0-3898-14dd-d251-17a716819550}]" + testing.go:149: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.304Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1 Address:8e971f64-dc8b-a54b-2f89-68d8dd1a47d1} {Suffrage:Voter ID:server-b52f85b0-5bd7-31d4-3c29-7bc6a396394e Address:b52f85b0-5bd7-31d4-3c29-7bc6a396394e} {Suffrage:Voter ID:server-d05667eb-6bbc-8a36-30b9-cef813c37ace Address:d05667eb-6bbc-8a36-30b9-cef813c37ace} {Suffrage:Voter ID:server-7ce62693-b3eb-604e-f602-033dbfeba6cf Address:7ce62693-b3eb-604e-f602-033dbfeba6cf} {Suffrage:Voter ID:server-42117aa1-e11d-0566-89a5-775c4e02549b Address:42117aa1-e11d-0566-89a5-775c4e02549b} {Suffrage:Voter ID:server-e03b2c08-8230-f155-5dec-d07c132b9733 Address:e03b2c08-8230-f155-5dec-d07c132b9733} {Suffrage:Voter ID:server-33df0ac0-3898-14dd-d251-17a716819550 Address:33df0ac0-3898-14dd-d251-17a716819550}]" +testLoggerAdapter verbose: server-b52f85b0-5bd7-31d4-3c29-7bc6a396394e: 2022-08-12T07:30:16.304Z [INFO] server-b52f85b0-5bd7-31d4-3c29-7bc6a396394e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1 Address:8e971f64-dc8b-a54b-2f89-68d8dd1a47d1} {Suffrage:Voter ID:server-b52f85b0-5bd7-31d4-3c29-7bc6a396394e Address:b52f85b0-5bd7-31d4-3c29-7bc6a396394e} {Suffrage:Voter ID:server-d05667eb-6bbc-8a36-30b9-cef813c37ace Address:d05667eb-6bbc-8a36-30b9-cef813c37ace} {Suffrage:Voter ID:server-7ce62693-b3eb-604e-f602-033dbfeba6cf Address:7ce62693-b3eb-604e-f602-033dbfeba6cf} {Suffrage:Voter ID:server-42117aa1-e11d-0566-89a5-775c4e02549b Address:42117aa1-e11d-0566-89a5-775c4e02549b} {Suffrage:Voter ID:server-e03b2c08-8230-f155-5dec-d07c132b9733 Address:e03b2c08-8230-f155-5dec-d07c132b9733} {Suffrage:Voter ID:server-33df0ac0-3898-14dd-d251-17a716819550 Address:33df0ac0-3898-14dd-d251-17a716819550}]" + testing.go:149: server-b52f85b0-5bd7-31d4-3c29-7bc6a396394e: 2022-08-12T07:30:16.304Z [INFO] server-b52f85b0-5bd7-31d4-3c29-7bc6a396394e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1 Address:8e971f64-dc8b-a54b-2f89-68d8dd1a47d1} {Suffrage:Voter ID:server-b52f85b0-5bd7-31d4-3c29-7bc6a396394e Address:b52f85b0-5bd7-31d4-3c29-7bc6a396394e} {Suffrage:Voter ID:server-d05667eb-6bbc-8a36-30b9-cef813c37ace Address:d05667eb-6bbc-8a36-30b9-cef813c37ace} {Suffrage:Voter ID:server-7ce62693-b3eb-604e-f602-033dbfeba6cf Address:7ce62693-b3eb-604e-f602-033dbfeba6cf} {Suffrage:Voter ID:server-42117aa1-e11d-0566-89a5-775c4e02549b Address:42117aa1-e11d-0566-89a5-775c4e02549b} {Suffrage:Voter ID:server-e03b2c08-8230-f155-5dec-d07c132b9733 Address:e03b2c08-8230-f155-5dec-d07c132b9733} {Suffrage:Voter ID:server-33df0ac0-3898-14dd-d251-17a716819550 Address:33df0ac0-3898-14dd-d251-17a716819550}]" +testLoggerAdapter verbose: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.304Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1 Address:8e971f64-dc8b-a54b-2f89-68d8dd1a47d1} {Suffrage:Voter ID:server-b52f85b0-5bd7-31d4-3c29-7bc6a396394e Address:b52f85b0-5bd7-31d4-3c29-7bc6a396394e} {Suffrage:Voter ID:server-d05667eb-6bbc-8a36-30b9-cef813c37ace Address:d05667eb-6bbc-8a36-30b9-cef813c37ace} {Suffrage:Voter ID:server-7ce62693-b3eb-604e-f602-033dbfeba6cf Address:7ce62693-b3eb-604e-f602-033dbfeba6cf} {Suffrage:Voter ID:server-42117aa1-e11d-0566-89a5-775c4e02549b Address:42117aa1-e11d-0566-89a5-775c4e02549b} {Suffrage:Voter ID:server-e03b2c08-8230-f155-5dec-d07c132b9733 Address:e03b2c08-8230-f155-5dec-d07c132b9733} {Suffrage:Voter ID:server-33df0ac0-3898-14dd-d251-17a716819550 Address:33df0ac0-3898-14dd-d251-17a716819550}]" + testing.go:149: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.304Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1 Address:8e971f64-dc8b-a54b-2f89-68d8dd1a47d1} {Suffrage:Voter ID:server-b52f85b0-5bd7-31d4-3c29-7bc6a396394e Address:b52f85b0-5bd7-31d4-3c29-7bc6a396394e} {Suffrage:Voter ID:server-d05667eb-6bbc-8a36-30b9-cef813c37ace Address:d05667eb-6bbc-8a36-30b9-cef813c37ace} {Suffrage:Voter ID:server-7ce62693-b3eb-604e-f602-033dbfeba6cf Address:7ce62693-b3eb-604e-f602-033dbfeba6cf} {Suffrage:Voter ID:server-42117aa1-e11d-0566-89a5-775c4e02549b Address:42117aa1-e11d-0566-89a5-775c4e02549b} {Suffrage:Voter ID:server-e03b2c08-8230-f155-5dec-d07c132b9733 Address:e03b2c08-8230-f155-5dec-d07c132b9733} {Suffrage:Voter ID:server-33df0ac0-3898-14dd-d251-17a716819550 Address:33df0ac0-3898-14dd-d251-17a716819550}]" +testLoggerAdapter verbose: server-7ce62693-b3eb-604e-f602-033dbfeba6cf: 2022-08-12T07:30:16.304Z [INFO] server-7ce62693-b3eb-604e-f602-033dbfeba6cf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1 Address:8e971f64-dc8b-a54b-2f89-68d8dd1a47d1} {Suffrage:Voter ID:server-b52f85b0-5bd7-31d4-3c29-7bc6a396394e Address:b52f85b0-5bd7-31d4-3c29-7bc6a396394e} {Suffrage:Voter ID:server-d05667eb-6bbc-8a36-30b9-cef813c37ace Address:d05667eb-6bbc-8a36-30b9-cef813c37ace} {Suffrage:Voter ID:server-7ce62693-b3eb-604e-f602-033dbfeba6cf Address:7ce62693-b3eb-604e-f602-033dbfeba6cf} {Suffrage:Voter ID:server-42117aa1-e11d-0566-89a5-775c4e02549b Address:42117aa1-e11d-0566-89a5-775c4e02549b} {Suffrage:Voter ID:server-e03b2c08-8230-f155-5dec-d07c132b9733 Address:e03b2c08-8230-f155-5dec-d07c132b9733} {Suffrage:Voter ID:server-33df0ac0-3898-14dd-d251-17a716819550 Address:33df0ac0-3898-14dd-d251-17a716819550}]" + testing.go:149: server-7ce62693-b3eb-604e-f602-033dbfeba6cf: 2022-08-12T07:30:16.304Z [INFO] server-7ce62693-b3eb-604e-f602-033dbfeba6cf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1 Address:8e971f64-dc8b-a54b-2f89-68d8dd1a47d1} {Suffrage:Voter ID:server-b52f85b0-5bd7-31d4-3c29-7bc6a396394e Address:b52f85b0-5bd7-31d4-3c29-7bc6a396394e} {Suffrage:Voter ID:server-d05667eb-6bbc-8a36-30b9-cef813c37ace Address:d05667eb-6bbc-8a36-30b9-cef813c37ace} {Suffrage:Voter ID:server-7ce62693-b3eb-604e-f602-033dbfeba6cf Address:7ce62693-b3eb-604e-f602-033dbfeba6cf} {Suffrage:Voter ID:server-42117aa1-e11d-0566-89a5-775c4e02549b Address:42117aa1-e11d-0566-89a5-775c4e02549b} {Suffrage:Voter ID:server-e03b2c08-8230-f155-5dec-d07c132b9733 Address:e03b2c08-8230-f155-5dec-d07c132b9733} {Suffrage:Voter ID:server-33df0ac0-3898-14dd-d251-17a716819550 Address:33df0ac0-3898-14dd-d251-17a716819550}]" +testLoggerAdapter verbose: server-42117aa1-e11d-0566-89a5-775c4e02549b: 2022-08-12T07:30:16.305Z [INFO] server-42117aa1-e11d-0566-89a5-775c4e02549b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1 Address:8e971f64-dc8b-a54b-2f89-68d8dd1a47d1} {Suffrage:Voter ID:server-b52f85b0-5bd7-31d4-3c29-7bc6a396394e Address:b52f85b0-5bd7-31d4-3c29-7bc6a396394e} {Suffrage:Voter ID:server-d05667eb-6bbc-8a36-30b9-cef813c37ace Address:d05667eb-6bbc-8a36-30b9-cef813c37ace} {Suffrage:Voter ID:server-7ce62693-b3eb-604e-f602-033dbfeba6cf Address:7ce62693-b3eb-604e-f602-033dbfeba6cf} {Suffrage:Voter ID:server-42117aa1-e11d-0566-89a5-775c4e02549b Address:42117aa1-e11d-0566-89a5-775c4e02549b} {Suffrage:Voter ID:server-e03b2c08-8230-f155-5dec-d07c132b9733 Address:e03b2c08-8230-f155-5dec-d07c132b9733} {Suffrage:Voter ID:server-33df0ac0-3898-14dd-d251-17a716819550 Address:33df0ac0-3898-14dd-d251-17a716819550}]" + testing.go:149: server-42117aa1-e11d-0566-89a5-775c4e02549b: 2022-08-12T07:30:16.305Z [INFO] server-42117aa1-e11d-0566-89a5-775c4e02549b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1 Address:8e971f64-dc8b-a54b-2f89-68d8dd1a47d1} {Suffrage:Voter ID:server-b52f85b0-5bd7-31d4-3c29-7bc6a396394e Address:b52f85b0-5bd7-31d4-3c29-7bc6a396394e} {Suffrage:Voter ID:server-d05667eb-6bbc-8a36-30b9-cef813c37ace Address:d05667eb-6bbc-8a36-30b9-cef813c37ace} {Suffrage:Voter ID:server-7ce62693-b3eb-604e-f602-033dbfeba6cf Address:7ce62693-b3eb-604e-f602-033dbfeba6cf} {Suffrage:Voter ID:server-42117aa1-e11d-0566-89a5-775c4e02549b Address:42117aa1-e11d-0566-89a5-775c4e02549b} {Suffrage:Voter ID:server-e03b2c08-8230-f155-5dec-d07c132b9733 Address:e03b2c08-8230-f155-5dec-d07c132b9733} {Suffrage:Voter ID:server-33df0ac0-3898-14dd-d251-17a716819550 Address:33df0ac0-3898-14dd-d251-17a716819550}]" +testLoggerAdapter verbose: server-e03b2c08-8230-f155-5dec-d07c132b9733: 2022-08-12T07:30:16.305Z [INFO] server-e03b2c08-8230-f155-5dec-d07c132b9733: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1 Address:8e971f64-dc8b-a54b-2f89-68d8dd1a47d1} {Suffrage:Voter ID:server-b52f85b0-5bd7-31d4-3c29-7bc6a396394e Address:b52f85b0-5bd7-31d4-3c29-7bc6a396394e} {Suffrage:Voter ID:server-d05667eb-6bbc-8a36-30b9-cef813c37ace Address:d05667eb-6bbc-8a36-30b9-cef813c37ace} {Suffrage:Voter ID:server-7ce62693-b3eb-604e-f602-033dbfeba6cf Address:7ce62693-b3eb-604e-f602-033dbfeba6cf} {Suffrage:Voter ID:server-42117aa1-e11d-0566-89a5-775c4e02549b Address:42117aa1-e11d-0566-89a5-775c4e02549b} {Suffrage:Voter ID:server-e03b2c08-8230-f155-5dec-d07c132b9733 Address:e03b2c08-8230-f155-5dec-d07c132b9733} {Suffrage:Voter ID:server-33df0ac0-3898-14dd-d251-17a716819550 Address:33df0ac0-3898-14dd-d251-17a716819550}]" + testing.go:149: server-e03b2c08-8230-f155-5dec-d07c132b9733: 2022-08-12T07:30:16.305Z [INFO] server-e03b2c08-8230-f155-5dec-d07c132b9733: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1 Address:8e971f64-dc8b-a54b-2f89-68d8dd1a47d1} {Suffrage:Voter ID:server-b52f85b0-5bd7-31d4-3c29-7bc6a396394e Address:b52f85b0-5bd7-31d4-3c29-7bc6a396394e} {Suffrage:Voter ID:server-d05667eb-6bbc-8a36-30b9-cef813c37ace Address:d05667eb-6bbc-8a36-30b9-cef813c37ace} {Suffrage:Voter ID:server-7ce62693-b3eb-604e-f602-033dbfeba6cf Address:7ce62693-b3eb-604e-f602-033dbfeba6cf} {Suffrage:Voter ID:server-42117aa1-e11d-0566-89a5-775c4e02549b Address:42117aa1-e11d-0566-89a5-775c4e02549b} {Suffrage:Voter ID:server-e03b2c08-8230-f155-5dec-d07c132b9733 Address:e03b2c08-8230-f155-5dec-d07c132b9733} {Suffrage:Voter ID:server-33df0ac0-3898-14dd-d251-17a716819550 Address:33df0ac0-3898-14dd-d251-17a716819550}]" +testLoggerAdapter verbose: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.305Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: entering follower state: follower="Node at d05667eb-6bbc-8a36-30b9-cef813c37ace [Follower]" leader= + testing.go:149: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.305Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: entering follower state: follower="Node at d05667eb-6bbc-8a36-30b9-cef813c37ace [Follower]" leader= +testLoggerAdapter verbose: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.305Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: entering follower state: follower="Node at 8e971f64-dc8b-a54b-2f89-68d8dd1a47d1 [Follower]" leader= + testing.go:149: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.305Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: entering follower state: follower="Node at 8e971f64-dc8b-a54b-2f89-68d8dd1a47d1 [Follower]" leader= +testLoggerAdapter verbose: server-b52f85b0-5bd7-31d4-3c29-7bc6a396394e: 2022-08-12T07:30:16.305Z [INFO] server-b52f85b0-5bd7-31d4-3c29-7bc6a396394e: entering follower state: follower="Node at b52f85b0-5bd7-31d4-3c29-7bc6a396394e [Follower]" leader= + testing.go:149: server-b52f85b0-5bd7-31d4-3c29-7bc6a396394e: 2022-08-12T07:30:16.305Z [INFO] server-b52f85b0-5bd7-31d4-3c29-7bc6a396394e: entering follower state: follower="Node at b52f85b0-5bd7-31d4-3c29-7bc6a396394e [Follower]" leader= +testLoggerAdapter verbose: server-e03b2c08-8230-f155-5dec-d07c132b9733: 2022-08-12T07:30:16.305Z [INFO] server-e03b2c08-8230-f155-5dec-d07c132b9733: entering follower state: follower="Node at e03b2c08-8230-f155-5dec-d07c132b9733 [Follower]" leader= + testing.go:149: server-e03b2c08-8230-f155-5dec-d07c132b9733: 2022-08-12T07:30:16.305Z [INFO] server-e03b2c08-8230-f155-5dec-d07c132b9733: entering follower state: follower="Node at e03b2c08-8230-f155-5dec-d07c132b9733 [Follower]" leader= +testLoggerAdapter verbose: server-42117aa1-e11d-0566-89a5-775c4e02549b: 2022-08-12T07:30:16.305Z [INFO] server-42117aa1-e11d-0566-89a5-775c4e02549b: entering follower state: follower="Node at 42117aa1-e11d-0566-89a5-775c4e02549b [Follower]" leader= + testing.go:149: server-42117aa1-e11d-0566-89a5-775c4e02549b: 2022-08-12T07:30:16.305Z [INFO] server-42117aa1-e11d-0566-89a5-775c4e02549b: entering follower state: follower="Node at 42117aa1-e11d-0566-89a5-775c4e02549b [Follower]" leader= +testLoggerAdapter verbose: server-7ce62693-b3eb-604e-f602-033dbfeba6cf: 2022-08-12T07:30:16.305Z [INFO] server-7ce62693-b3eb-604e-f602-033dbfeba6cf: entering follower state: follower="Node at 7ce62693-b3eb-604e-f602-033dbfeba6cf [Follower]" leader= + testing.go:149: server-7ce62693-b3eb-604e-f602-033dbfeba6cf: 2022-08-12T07:30:16.305Z [INFO] server-7ce62693-b3eb-604e-f602-033dbfeba6cf: entering follower state: follower="Node at 7ce62693-b3eb-604e-f602-033dbfeba6cf [Follower]" leader= +testLoggerAdapter verbose: server-33df0ac0-3898-14dd-d251-17a716819550: 2022-08-12T07:30:16.306Z [INFO] server-33df0ac0-3898-14dd-d251-17a716819550: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1 Address:8e971f64-dc8b-a54b-2f89-68d8dd1a47d1} {Suffrage:Voter ID:server-b52f85b0-5bd7-31d4-3c29-7bc6a396394e Address:b52f85b0-5bd7-31d4-3c29-7bc6a396394e} {Suffrage:Voter ID:server-d05667eb-6bbc-8a36-30b9-cef813c37ace Address:d05667eb-6bbc-8a36-30b9-cef813c37ace} {Suffrage:Voter ID:server-7ce62693-b3eb-604e-f602-033dbfeba6cf Address:7ce62693-b3eb-604e-f602-033dbfeba6cf} {Suffrage:Voter ID:server-42117aa1-e11d-0566-89a5-775c4e02549b Address:42117aa1-e11d-0566-89a5-775c4e02549b} {Suffrage:Voter ID:server-e03b2c08-8230-f155-5dec-d07c132b9733 Address:e03b2c08-8230-f155-5dec-d07c132b9733} {Suffrage:Voter ID:server-33df0ac0-3898-14dd-d251-17a716819550 Address:33df0ac0-3898-14dd-d251-17a716819550}]" + testing.go:149: server-33df0ac0-3898-14dd-d251-17a716819550: 2022-08-12T07:30:16.306Z [INFO] server-33df0ac0-3898-14dd-d251-17a716819550: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1 Address:8e971f64-dc8b-a54b-2f89-68d8dd1a47d1} {Suffrage:Voter ID:server-b52f85b0-5bd7-31d4-3c29-7bc6a396394e Address:b52f85b0-5bd7-31d4-3c29-7bc6a396394e} {Suffrage:Voter ID:server-d05667eb-6bbc-8a36-30b9-cef813c37ace Address:d05667eb-6bbc-8a36-30b9-cef813c37ace} {Suffrage:Voter ID:server-7ce62693-b3eb-604e-f602-033dbfeba6cf Address:7ce62693-b3eb-604e-f602-033dbfeba6cf} {Suffrage:Voter ID:server-42117aa1-e11d-0566-89a5-775c4e02549b Address:42117aa1-e11d-0566-89a5-775c4e02549b} {Suffrage:Voter ID:server-e03b2c08-8230-f155-5dec-d07c132b9733 Address:e03b2c08-8230-f155-5dec-d07c132b9733} {Suffrage:Voter ID:server-33df0ac0-3898-14dd-d251-17a716819550 Address:33df0ac0-3898-14dd-d251-17a716819550}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:16.306Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:16.306Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-33df0ac0-3898-14dd-d251-17a716819550: 2022-08-12T07:30:16.306Z [INFO] server-33df0ac0-3898-14dd-d251-17a716819550: entering follower state: follower="Node at 33df0ac0-3898-14dd-d251-17a716819550 [Follower]" leader= + testing.go:149: server-33df0ac0-3898-14dd-d251-17a716819550: 2022-08-12T07:30:16.306Z [INFO] server-33df0ac0-3898-14dd-d251-17a716819550: entering follower state: follower="Node at 33df0ac0-3898-14dd-d251-17a716819550 [Follower]" leader= +testLoggerAdapter verbose: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.357Z [WARN] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.357Z [WARN] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.357Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: entering candidate state: node="Node at d05667eb-6bbc-8a36-30b9-cef813c37ace [Candidate]" term=2 + testing.go:149: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.357Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: entering candidate state: node="Node at d05667eb-6bbc-8a36-30b9-cef813c37ace [Candidate]" term=2 +testLoggerAdapter verbose: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.358Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: election won: tally=4 + testing.go:149: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.358Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: election won: tally=4 +testLoggerAdapter verbose: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.358Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: entering leader state: leader="Node at d05667eb-6bbc-8a36-30b9-cef813c37ace [Leader]" + testing.go:149: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.358Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: entering leader state: leader="Node at d05667eb-6bbc-8a36-30b9-cef813c37ace [Leader]" +testLoggerAdapter verbose: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.358Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: added peer, starting replication: peer=server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1 + testing.go:149: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.358Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: added peer, starting replication: peer=server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1 +testLoggerAdapter verbose: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.358Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: added peer, starting replication: peer=server-b52f85b0-5bd7-31d4-3c29-7bc6a396394e + testing.go:149: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.358Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: added peer, starting replication: peer=server-b52f85b0-5bd7-31d4-3c29-7bc6a396394e +testLoggerAdapter verbose: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.358Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: added peer, starting replication: peer=server-7ce62693-b3eb-604e-f602-033dbfeba6cf + testing.go:149: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.358Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: added peer, starting replication: peer=server-7ce62693-b3eb-604e-f602-033dbfeba6cf +testLoggerAdapter verbose: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.358Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: added peer, starting replication: peer=server-42117aa1-e11d-0566-89a5-775c4e02549b + testing.go:149: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.358Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: added peer, starting replication: peer=server-42117aa1-e11d-0566-89a5-775c4e02549b +testLoggerAdapter verbose: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.358Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: added peer, starting replication: peer=server-e03b2c08-8230-f155-5dec-d07c132b9733 + testing.go:149: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.358Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: added peer, starting replication: peer=server-e03b2c08-8230-f155-5dec-d07c132b9733 +testLoggerAdapter verbose: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.358Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: added peer, starting replication: peer=server-33df0ac0-3898-14dd-d251-17a716819550 + testing.go:149: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.358Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: added peer, starting replication: peer=server-33df0ac0-3898-14dd-d251-17a716819550 +testLoggerAdapter verbose: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.358Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: pipelining replication: peer="{Voter server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1 8e971f64-dc8b-a54b-2f89-68d8dd1a47d1}" + testing.go:149: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.358Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: pipelining replication: peer="{Voter server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1 8e971f64-dc8b-a54b-2f89-68d8dd1a47d1}" +testLoggerAdapter verbose: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.358Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: pipelining replication: peer="{Voter server-e03b2c08-8230-f155-5dec-d07c132b9733 e03b2c08-8230-f155-5dec-d07c132b9733}" + testing.go:149: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.358Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: pipelining replication: peer="{Voter server-e03b2c08-8230-f155-5dec-d07c132b9733 e03b2c08-8230-f155-5dec-d07c132b9733}" +testLoggerAdapter verbose: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.358Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: pipelining replication: peer="{Voter server-7ce62693-b3eb-604e-f602-033dbfeba6cf 7ce62693-b3eb-604e-f602-033dbfeba6cf}" + testing.go:149: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.358Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: pipelining replication: peer="{Voter server-7ce62693-b3eb-604e-f602-033dbfeba6cf 7ce62693-b3eb-604e-f602-033dbfeba6cf}" +testLoggerAdapter verbose: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.358Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: pipelining replication: peer="{Voter server-42117aa1-e11d-0566-89a5-775c4e02549b 42117aa1-e11d-0566-89a5-775c4e02549b}" + testing.go:149: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.358Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: pipelining replication: peer="{Voter server-42117aa1-e11d-0566-89a5-775c4e02549b 42117aa1-e11d-0566-89a5-775c4e02549b}" +testLoggerAdapter verbose: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.358Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: pipelining replication: peer="{Voter server-b52f85b0-5bd7-31d4-3c29-7bc6a396394e b52f85b0-5bd7-31d4-3c29-7bc6a396394e}" + testing.go:149: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.358Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: pipelining replication: peer="{Voter server-b52f85b0-5bd7-31d4-3c29-7bc6a396394e b52f85b0-5bd7-31d4-3c29-7bc6a396394e}" +testLoggerAdapter verbose: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.358Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: pipelining replication: peer="{Voter server-33df0ac0-3898-14dd-d251-17a716819550 33df0ac0-3898-14dd-d251-17a716819550}" + testing.go:149: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.358Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: pipelining replication: peer="{Voter server-33df0ac0-3898-14dd-d251-17a716819550 33df0ac0-3898-14dd-d251-17a716819550}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:16.463Z [INFO] stable state for Leader reached at 2022-08-12 07:30:16.35843776 +0000 UTC m=+17.315124241 (1 nodes), 51.6543ms from start of poll, 54.69686ms from cluster start. Timeout at 2022-08-12 07:30:16.46351444 +0000 UTC m=+17.420200941, 105.0767ms after stability + testing.go:149: cluster: 2022-08-12T07:30:16.463Z [INFO] stable state for Leader reached at 2022-08-12 07:30:16.35843776 +0000 UTC m=+17.315124241 (1 nodes), 51.6543ms from start of poll, 54.69686ms from cluster start. Timeout at 2022-08-12 07:30:16.46351444 +0000 UTC m=+17.420200941, 105.0767ms after stability +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:16.463Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2022-08-12T07:30:16.463Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:16.568Z [INFO] stable state for Follower reached at 2022-08-12 07:30:16.46376776 +0000 UTC m=+17.420454241 (6 nodes), 4.02µs from start of poll, 160.02686ms from cluster start. Timeout at 2022-08-12 07:30:16.56883764 +0000 UTC m=+17.525524121, 105.06988ms after stability + testing.go:149: cluster: 2022-08-12T07:30:16.568Z [INFO] stable state for Follower reached at 2022-08-12 07:30:16.46376776 +0000 UTC m=+17.420454241 (6 nodes), 4.02µs from start of poll, 160.02686ms from cluster start. Timeout at 2022-08-12 07:30:16.56883764 +0000 UTC m=+17.525524121, 105.06988ms after stability +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:16.569Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:16.569Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:16.674Z [INFO] stable state for Leader reached at 2022-08-12 07:30:16.56907732 +0000 UTC m=+17.525763781 (1 nodes), 3.8µs from start of poll, 265.3364ms from cluster start. Timeout at 2022-08-12 07:30:16.67414556 +0000 UTC m=+17.630832061, 105.06828ms after stability + testing.go:149: cluster: 2022-08-12T07:30:16.674Z [INFO] stable state for Leader reached at 2022-08-12 07:30:16.56907732 +0000 UTC m=+17.525763781 (1 nodes), 3.8µs from start of poll, 265.3364ms from cluster start. Timeout at 2022-08-12 07:30:16.67414556 +0000 UTC m=+17.630832061, 105.06828ms after stability +testLoggerAdapter verbose: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.674Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: entering candidate state: node="Node at 8e971f64-dc8b-a54b-2f89-68d8dd1a47d1 [Candidate]" term=3 +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:16.674Z [INFO] starting stability test: raft-state=Leader + testing.go:149: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.674Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: entering candidate state: node="Node at 8e971f64-dc8b-a54b-2f89-68d8dd1a47d1 [Candidate]" term=3 +testLoggerAdapter verbose: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.674Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: entering follower state: follower="Node at d05667eb-6bbc-8a36-30b9-cef813c37ace [Follower]" leader= + testing.go:149: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.674Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: entering follower state: follower="Node at d05667eb-6bbc-8a36-30b9-cef813c37ace [Follower]" leader= +testLoggerAdapter verbose: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.674Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: election won: tally=4 + testing.go:149: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.674Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: election won: tally=4 +testLoggerAdapter verbose: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.674Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: entering leader state: leader="Node at 8e971f64-dc8b-a54b-2f89-68d8dd1a47d1 [Leader]" +testLoggerAdapter verbose: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.674Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: aborting pipeline replication: peer="{Voter server-42117aa1-e11d-0566-89a5-775c4e02549b 42117aa1-e11d-0566-89a5-775c4e02549b}" + testing.go:149: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.674Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: entering leader state: leader="Node at 8e971f64-dc8b-a54b-2f89-68d8dd1a47d1 [Leader]" +testLoggerAdapter verbose: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.674Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: added peer, starting replication: peer=server-b52f85b0-5bd7-31d4-3c29-7bc6a396394e + testing.go:149: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.674Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: aborting pipeline replication: peer="{Voter server-42117aa1-e11d-0566-89a5-775c4e02549b 42117aa1-e11d-0566-89a5-775c4e02549b}" +testLoggerAdapter verbose: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.674Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: aborting pipeline replication: peer="{Voter server-e03b2c08-8230-f155-5dec-d07c132b9733 e03b2c08-8230-f155-5dec-d07c132b9733}" + testing.go:149: cluster: 2022-08-12T07:30:16.674Z [INFO] starting stability test: raft-state=Leader + testing.go:149: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.674Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: added peer, starting replication: peer=server-b52f85b0-5bd7-31d4-3c29-7bc6a396394e +testLoggerAdapter verbose: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.675Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: added peer, starting replication: peer=server-d05667eb-6bbc-8a36-30b9-cef813c37ace + testing.go:149: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.675Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: added peer, starting replication: peer=server-d05667eb-6bbc-8a36-30b9-cef813c37ace +testLoggerAdapter verbose: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.675Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: added peer, starting replication: peer=server-7ce62693-b3eb-604e-f602-033dbfeba6cf + testing.go:149: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.674Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: aborting pipeline replication: peer="{Voter server-e03b2c08-8230-f155-5dec-d07c132b9733 e03b2c08-8230-f155-5dec-d07c132b9733}" + testing.go:149: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.675Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: added peer, starting replication: peer=server-7ce62693-b3eb-604e-f602-033dbfeba6cf +testLoggerAdapter verbose: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.674Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: aborting pipeline replication: peer="{Voter server-b52f85b0-5bd7-31d4-3c29-7bc6a396394e b52f85b0-5bd7-31d4-3c29-7bc6a396394e}" + testing.go:149: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.674Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: aborting pipeline replication: peer="{Voter server-b52f85b0-5bd7-31d4-3c29-7bc6a396394e b52f85b0-5bd7-31d4-3c29-7bc6a396394e}" +testLoggerAdapter verbose: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.675Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: added peer, starting replication: peer=server-42117aa1-e11d-0566-89a5-775c4e02549b + testing.go:149: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.675Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: added peer, starting replication: peer=server-42117aa1-e11d-0566-89a5-775c4e02549b +testLoggerAdapter verbose: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.674Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: aborting pipeline replication: peer="{Voter server-7ce62693-b3eb-604e-f602-033dbfeba6cf 7ce62693-b3eb-604e-f602-033dbfeba6cf}" +testLoggerAdapter verbose: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.675Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: added peer, starting replication: peer=server-e03b2c08-8230-f155-5dec-d07c132b9733 + testing.go:149: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.674Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: aborting pipeline replication: peer="{Voter server-7ce62693-b3eb-604e-f602-033dbfeba6cf 7ce62693-b3eb-604e-f602-033dbfeba6cf}" + testing.go:149: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.675Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: added peer, starting replication: peer=server-e03b2c08-8230-f155-5dec-d07c132b9733 +testLoggerAdapter verbose: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.674Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: aborting pipeline replication: peer="{Voter server-33df0ac0-3898-14dd-d251-17a716819550 33df0ac0-3898-14dd-d251-17a716819550}" + testing.go:149: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.674Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: aborting pipeline replication: peer="{Voter server-33df0ac0-3898-14dd-d251-17a716819550 33df0ac0-3898-14dd-d251-17a716819550}" +testLoggerAdapter verbose: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.674Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: aborting pipeline replication: peer="{Voter server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1 8e971f64-dc8b-a54b-2f89-68d8dd1a47d1}" + testing.go:149: server-d05667eb-6bbc-8a36-30b9-cef813c37ace: 2022-08-12T07:30:16.674Z [INFO] server-d05667eb-6bbc-8a36-30b9-cef813c37ace: aborting pipeline replication: peer="{Voter server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1 8e971f64-dc8b-a54b-2f89-68d8dd1a47d1}" +testLoggerAdapter verbose: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.675Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: pipelining replication: peer="{Voter server-d05667eb-6bbc-8a36-30b9-cef813c37ace d05667eb-6bbc-8a36-30b9-cef813c37ace}" + testing.go:149: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.675Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: pipelining replication: peer="{Voter server-d05667eb-6bbc-8a36-30b9-cef813c37ace d05667eb-6bbc-8a36-30b9-cef813c37ace}" +testLoggerAdapter verbose: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.675Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: pipelining replication: peer="{Voter server-b52f85b0-5bd7-31d4-3c29-7bc6a396394e b52f85b0-5bd7-31d4-3c29-7bc6a396394e}" + testing.go:149: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.675Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: pipelining replication: peer="{Voter server-b52f85b0-5bd7-31d4-3c29-7bc6a396394e b52f85b0-5bd7-31d4-3c29-7bc6a396394e}" +testLoggerAdapter verbose: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.675Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: pipelining replication: peer="{Voter server-7ce62693-b3eb-604e-f602-033dbfeba6cf 7ce62693-b3eb-604e-f602-033dbfeba6cf}" + testing.go:149: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.675Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: pipelining replication: peer="{Voter server-7ce62693-b3eb-604e-f602-033dbfeba6cf 7ce62693-b3eb-604e-f602-033dbfeba6cf}" +testLoggerAdapter verbose: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.675Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: pipelining replication: peer="{Voter server-42117aa1-e11d-0566-89a5-775c4e02549b 42117aa1-e11d-0566-89a5-775c4e02549b}" + testing.go:149: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.675Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: pipelining replication: peer="{Voter server-42117aa1-e11d-0566-89a5-775c4e02549b 42117aa1-e11d-0566-89a5-775c4e02549b}" +testLoggerAdapter verbose: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.675Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: added peer, starting replication: peer=server-33df0ac0-3898-14dd-d251-17a716819550 + testing.go:149: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.675Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: added peer, starting replication: peer=server-33df0ac0-3898-14dd-d251-17a716819550 +testLoggerAdapter verbose: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.675Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: pipelining replication: peer="{Voter server-e03b2c08-8230-f155-5dec-d07c132b9733 e03b2c08-8230-f155-5dec-d07c132b9733}" + testing.go:149: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.675Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: pipelining replication: peer="{Voter server-e03b2c08-8230-f155-5dec-d07c132b9733 e03b2c08-8230-f155-5dec-d07c132b9733}" +testLoggerAdapter verbose: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.676Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: pipelining replication: peer="{Voter server-33df0ac0-3898-14dd-d251-17a716819550 33df0ac0-3898-14dd-d251-17a716819550}" + testing.go:149: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.676Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: pipelining replication: peer="{Voter server-33df0ac0-3898-14dd-d251-17a716819550 33df0ac0-3898-14dd-d251-17a716819550}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:16.780Z [INFO] stable state for Leader reached at 2022-08-12 07:30:16.67510532 +0000 UTC m=+17.631791781 (1 nodes), 44.34µs from start of poll, 371.3644ms from cluster start. Timeout at 2022-08-12 07:30:16.78017342 +0000 UTC m=+17.736859921, 105.06814ms after stability + testing.go:149: cluster: 2022-08-12T07:30:16.780Z [INFO] stable state for Leader reached at 2022-08-12 07:30:16.67510532 +0000 UTC m=+17.631791781 (1 nodes), 44.34µs from start of poll, 371.3644ms from cluster start. Timeout at 2022-08-12 07:30:16.78017342 +0000 UTC m=+17.736859921, 105.06814ms after stability +testLoggerAdapter verbose: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.780Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: aborting pipeline replication: peer="{Voter server-b52f85b0-5bd7-31d4-3c29-7bc6a396394e b52f85b0-5bd7-31d4-3c29-7bc6a396394e}" + testing.go:149: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.780Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: aborting pipeline replication: peer="{Voter server-b52f85b0-5bd7-31d4-3c29-7bc6a396394e b52f85b0-5bd7-31d4-3c29-7bc6a396394e}" +testLoggerAdapter verbose: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.780Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: aborting pipeline replication: peer="{Voter server-42117aa1-e11d-0566-89a5-775c4e02549b 42117aa1-e11d-0566-89a5-775c4e02549b}" + testing.go:149: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.780Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: aborting pipeline replication: peer="{Voter server-42117aa1-e11d-0566-89a5-775c4e02549b 42117aa1-e11d-0566-89a5-775c4e02549b}" +testLoggerAdapter verbose: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.780Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: aborting pipeline replication: peer="{Voter server-d05667eb-6bbc-8a36-30b9-cef813c37ace d05667eb-6bbc-8a36-30b9-cef813c37ace}" + testing.go:149: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.780Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: aborting pipeline replication: peer="{Voter server-d05667eb-6bbc-8a36-30b9-cef813c37ace d05667eb-6bbc-8a36-30b9-cef813c37ace}" +testLoggerAdapter verbose: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.780Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: aborting pipeline replication: peer="{Voter server-7ce62693-b3eb-604e-f602-033dbfeba6cf 7ce62693-b3eb-604e-f602-033dbfeba6cf}" + testing.go:149: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.780Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: aborting pipeline replication: peer="{Voter server-7ce62693-b3eb-604e-f602-033dbfeba6cf 7ce62693-b3eb-604e-f602-033dbfeba6cf}" +testLoggerAdapter verbose: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.780Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: aborting pipeline replication: peer="{Voter server-33df0ac0-3898-14dd-d251-17a716819550 33df0ac0-3898-14dd-d251-17a716819550}" + testing.go:149: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.780Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: aborting pipeline replication: peer="{Voter server-33df0ac0-3898-14dd-d251-17a716819550 33df0ac0-3898-14dd-d251-17a716819550}" +testLoggerAdapter verbose: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.780Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: aborting pipeline replication: peer="{Voter server-e03b2c08-8230-f155-5dec-d07c132b9733 e03b2c08-8230-f155-5dec-d07c132b9733}" + testing.go:149: server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: 2022-08-12T07:30:16.780Z [INFO] server-8e971f64-dc8b-a54b-2f89-68d8dd1a47d1: aborting pipeline replication: peer="{Voter server-e03b2c08-8230-f155-5dec-d07c132b9733 e03b2c08-8230-f155-5dec-d07c132b9733}" +--- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.48s) === RUN TestRaft_LeadershipTransferToInvalidID -testLoggerAdapter verbose: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.222Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c1bf599e-fdef-6653-c143-6898145d4e80 Address:c1bf599e-fdef-6653-c143-6898145d4e80} {Suffrage:Voter ID:server-fe706442-30ee-16de-924a-a53418948471 Address:fe706442-30ee-16de-924a-a53418948471} {Suffrage:Voter ID:server-6b9e4268-2f95-1adc-e18f-a6e74f441426 Address:6b9e4268-2f95-1adc-e18f-a6e74f441426}]" - testing.go:149: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.222Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c1bf599e-fdef-6653-c143-6898145d4e80 Address:c1bf599e-fdef-6653-c143-6898145d4e80} {Suffrage:Voter ID:server-fe706442-30ee-16de-924a-a53418948471 Address:fe706442-30ee-16de-924a-a53418948471} {Suffrage:Voter ID:server-6b9e4268-2f95-1adc-e18f-a6e74f441426 Address:6b9e4268-2f95-1adc-e18f-a6e74f441426}]" -testLoggerAdapter verbose: server-fe706442-30ee-16de-924a-a53418948471: 2021-07-10T01:05:12.222Z [INFO] server-fe706442-30ee-16de-924a-a53418948471: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c1bf599e-fdef-6653-c143-6898145d4e80 Address:c1bf599e-fdef-6653-c143-6898145d4e80} {Suffrage:Voter ID:server-fe706442-30ee-16de-924a-a53418948471 Address:fe706442-30ee-16de-924a-a53418948471} {Suffrage:Voter ID:server-6b9e4268-2f95-1adc-e18f-a6e74f441426 Address:6b9e4268-2f95-1adc-e18f-a6e74f441426}]" - testing.go:149: server-fe706442-30ee-16de-924a-a53418948471: 2021-07-10T01:05:12.222Z [INFO] server-fe706442-30ee-16de-924a-a53418948471: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c1bf599e-fdef-6653-c143-6898145d4e80 Address:c1bf599e-fdef-6653-c143-6898145d4e80} {Suffrage:Voter ID:server-fe706442-30ee-16de-924a-a53418948471 Address:fe706442-30ee-16de-924a-a53418948471} {Suffrage:Voter ID:server-6b9e4268-2f95-1adc-e18f-a6e74f441426 Address:6b9e4268-2f95-1adc-e18f-a6e74f441426}]" -testLoggerAdapter verbose: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.222Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: entering follower state: follower="Node at c1bf599e-fdef-6653-c143-6898145d4e80 [Follower]" leader= -testLoggerAdapter verbose: server-6b9e4268-2f95-1adc-e18f-a6e74f441426: 2021-07-10T01:05:12.222Z [INFO] server-6b9e4268-2f95-1adc-e18f-a6e74f441426: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c1bf599e-fdef-6653-c143-6898145d4e80 Address:c1bf599e-fdef-6653-c143-6898145d4e80} {Suffrage:Voter ID:server-fe706442-30ee-16de-924a-a53418948471 Address:fe706442-30ee-16de-924a-a53418948471} {Suffrage:Voter ID:server-6b9e4268-2f95-1adc-e18f-a6e74f441426 Address:6b9e4268-2f95-1adc-e18f-a6e74f441426}]" - testing.go:149: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.222Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: entering follower state: follower="Node at c1bf599e-fdef-6653-c143-6898145d4e80 [Follower]" leader= - testing.go:149: server-6b9e4268-2f95-1adc-e18f-a6e74f441426: 2021-07-10T01:05:12.222Z [INFO] server-6b9e4268-2f95-1adc-e18f-a6e74f441426: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c1bf599e-fdef-6653-c143-6898145d4e80 Address:c1bf599e-fdef-6653-c143-6898145d4e80} {Suffrage:Voter ID:server-fe706442-30ee-16de-924a-a53418948471 Address:fe706442-30ee-16de-924a-a53418948471} {Suffrage:Voter ID:server-6b9e4268-2f95-1adc-e18f-a6e74f441426 Address:6b9e4268-2f95-1adc-e18f-a6e74f441426}]" -testLoggerAdapter verbose: server-fe706442-30ee-16de-924a-a53418948471: 2021-07-10T01:05:12.223Z [INFO] server-fe706442-30ee-16de-924a-a53418948471: entering follower state: follower="Node at fe706442-30ee-16de-924a-a53418948471 [Follower]" leader= - testing.go:149: server-fe706442-30ee-16de-924a-a53418948471: 2021-07-10T01:05:12.223Z [INFO] server-fe706442-30ee-16de-924a-a53418948471: entering follower state: follower="Node at fe706442-30ee-16de-924a-a53418948471 [Follower]" leader= -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:12.223Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:12.223Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-6b9e4268-2f95-1adc-e18f-a6e74f441426: 2021-07-10T01:05:12.223Z [INFO] server-6b9e4268-2f95-1adc-e18f-a6e74f441426: entering follower state: follower="Node at 6b9e4268-2f95-1adc-e18f-a6e74f441426 [Follower]" leader= - testing.go:149: server-6b9e4268-2f95-1adc-e18f-a6e74f441426: 2021-07-10T01:05:12.223Z [INFO] server-6b9e4268-2f95-1adc-e18f-a6e74f441426: entering follower state: follower="Node at 6b9e4268-2f95-1adc-e18f-a6e74f441426 [Follower]" leader= -testLoggerAdapter verbose: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.273Z [WARN] server-c1bf599e-fdef-6653-c143-6898145d4e80: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.273Z [WARN] server-c1bf599e-fdef-6653-c143-6898145d4e80: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.274Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: entering candidate state: node="Node at c1bf599e-fdef-6653-c143-6898145d4e80 [Candidate]" term=2 - testing.go:149: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.274Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: entering candidate state: node="Node at c1bf599e-fdef-6653-c143-6898145d4e80 [Candidate]" term=2 -testLoggerAdapter verbose: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.274Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: election won: tally=2 - testing.go:149: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.274Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: election won: tally=2 -testLoggerAdapter verbose: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.274Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: entering leader state: leader="Node at c1bf599e-fdef-6653-c143-6898145d4e80 [Leader]" - testing.go:149: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.274Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: entering leader state: leader="Node at c1bf599e-fdef-6653-c143-6898145d4e80 [Leader]" -testLoggerAdapter verbose: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.274Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: added peer, starting replication: peer=server-fe706442-30ee-16de-924a-a53418948471 - testing.go:149: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.274Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: added peer, starting replication: peer=server-fe706442-30ee-16de-924a-a53418948471 -testLoggerAdapter verbose: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.274Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: added peer, starting replication: peer=server-6b9e4268-2f95-1adc-e18f-a6e74f441426 - testing.go:149: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.274Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: added peer, starting replication: peer=server-6b9e4268-2f95-1adc-e18f-a6e74f441426 -testLoggerAdapter verbose: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.274Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: pipelining replication: peer="{Voter server-fe706442-30ee-16de-924a-a53418948471 fe706442-30ee-16de-924a-a53418948471}" - testing.go:149: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.274Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: pipelining replication: peer="{Voter server-fe706442-30ee-16de-924a-a53418948471 fe706442-30ee-16de-924a-a53418948471}" -testLoggerAdapter verbose: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.274Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: pipelining replication: peer="{Voter server-6b9e4268-2f95-1adc-e18f-a6e74f441426 6b9e4268-2f95-1adc-e18f-a6e74f441426}" - testing.go:149: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.274Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: pipelining replication: peer="{Voter server-6b9e4268-2f95-1adc-e18f-a6e74f441426 6b9e4268-2f95-1adc-e18f-a6e74f441426}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:12.379Z [INFO] stable state for Leader reached at 2021-07-10 01:05:12.274413376 +0000 UTC m=+17.662571366 (1 nodes), 51.287941ms from start of poll, 52.409333ms from cluster start. Timeout at 2021-07-10 01:05:12.379420583 +0000 UTC m=+17.767578533, 105.007167ms after stability - testing.go:149: cluster: 2021-07-10T01:05:12.379Z [INFO] stable state for Leader reached at 2021-07-10 01:05:12.274413376 +0000 UTC m=+17.662571366 (1 nodes), 51.287941ms from start of poll, 52.409333ms from cluster start. Timeout at 2021-07-10 01:05:12.379420583 +0000 UTC m=+17.767578533, 105.007167ms after stability -testLoggerAdapter verbose: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.379Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: aborting pipeline replication: peer="{Voter server-6b9e4268-2f95-1adc-e18f-a6e74f441426 6b9e4268-2f95-1adc-e18f-a6e74f441426}" - testing.go:149: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.379Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: aborting pipeline replication: peer="{Voter server-6b9e4268-2f95-1adc-e18f-a6e74f441426 6b9e4268-2f95-1adc-e18f-a6e74f441426}" -testLoggerAdapter verbose: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.379Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: aborting pipeline replication: peer="{Voter server-fe706442-30ee-16de-924a-a53418948471 fe706442-30ee-16de-924a-a53418948471}" - testing.go:149: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.379Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: aborting pipeline replication: peer="{Voter server-fe706442-30ee-16de-924a-a53418948471 fe706442-30ee-16de-924a-a53418948471}" ---- PASS: TestRaft_LeadershipTransferToInvalidID (0.16s) +testLoggerAdapter verbose: server-3b91fa6a-c9a4-f0c2-9e36-0b49544831ca: 2022-08-12T07:30:16.783Z [INFO] server-3b91fa6a-c9a4-f0c2-9e36-0b49544831ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3b91fa6a-c9a4-f0c2-9e36-0b49544831ca Address:3b91fa6a-c9a4-f0c2-9e36-0b49544831ca} {Suffrage:Voter ID:server-29445b52-566c-bb79-7a1a-4dfea326bf2f Address:29445b52-566c-bb79-7a1a-4dfea326bf2f} {Suffrage:Voter ID:server-27dab4f6-87ed-859b-716c-0bc8e5f26df7 Address:27dab4f6-87ed-859b-716c-0bc8e5f26df7}]" + testing.go:149: server-3b91fa6a-c9a4-f0c2-9e36-0b49544831ca: 2022-08-12T07:30:16.783Z [INFO] server-3b91fa6a-c9a4-f0c2-9e36-0b49544831ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3b91fa6a-c9a4-f0c2-9e36-0b49544831ca Address:3b91fa6a-c9a4-f0c2-9e36-0b49544831ca} {Suffrage:Voter ID:server-29445b52-566c-bb79-7a1a-4dfea326bf2f Address:29445b52-566c-bb79-7a1a-4dfea326bf2f} {Suffrage:Voter ID:server-27dab4f6-87ed-859b-716c-0bc8e5f26df7 Address:27dab4f6-87ed-859b-716c-0bc8e5f26df7}]" +testLoggerAdapter verbose: server-29445b52-566c-bb79-7a1a-4dfea326bf2f: 2022-08-12T07:30:16.783Z [INFO] server-29445b52-566c-bb79-7a1a-4dfea326bf2f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3b91fa6a-c9a4-f0c2-9e36-0b49544831ca Address:3b91fa6a-c9a4-f0c2-9e36-0b49544831ca} {Suffrage:Voter ID:server-29445b52-566c-bb79-7a1a-4dfea326bf2f Address:29445b52-566c-bb79-7a1a-4dfea326bf2f} {Suffrage:Voter ID:server-27dab4f6-87ed-859b-716c-0bc8e5f26df7 Address:27dab4f6-87ed-859b-716c-0bc8e5f26df7}]" + testing.go:149: server-29445b52-566c-bb79-7a1a-4dfea326bf2f: 2022-08-12T07:30:16.783Z [INFO] server-29445b52-566c-bb79-7a1a-4dfea326bf2f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3b91fa6a-c9a4-f0c2-9e36-0b49544831ca Address:3b91fa6a-c9a4-f0c2-9e36-0b49544831ca} {Suffrage:Voter ID:server-29445b52-566c-bb79-7a1a-4dfea326bf2f Address:29445b52-566c-bb79-7a1a-4dfea326bf2f} {Suffrage:Voter ID:server-27dab4f6-87ed-859b-716c-0bc8e5f26df7 Address:27dab4f6-87ed-859b-716c-0bc8e5f26df7}]" +testLoggerAdapter verbose: server-3b91fa6a-c9a4-f0c2-9e36-0b49544831ca: 2022-08-12T07:30:16.783Z [INFO] server-3b91fa6a-c9a4-f0c2-9e36-0b49544831ca: entering follower state: follower="Node at 3b91fa6a-c9a4-f0c2-9e36-0b49544831ca [Follower]" leader= + testing.go:149: server-3b91fa6a-c9a4-f0c2-9e36-0b49544831ca: 2022-08-12T07:30:16.783Z [INFO] server-3b91fa6a-c9a4-f0c2-9e36-0b49544831ca: entering follower state: follower="Node at 3b91fa6a-c9a4-f0c2-9e36-0b49544831ca [Follower]" leader= +testLoggerAdapter verbose: server-29445b52-566c-bb79-7a1a-4dfea326bf2f: 2022-08-12T07:30:16.783Z [INFO] server-29445b52-566c-bb79-7a1a-4dfea326bf2f: entering follower state: follower="Node at 29445b52-566c-bb79-7a1a-4dfea326bf2f [Follower]" leader= + testing.go:149: server-29445b52-566c-bb79-7a1a-4dfea326bf2f: 2022-08-12T07:30:16.783Z [INFO] server-29445b52-566c-bb79-7a1a-4dfea326bf2f: entering follower state: follower="Node at 29445b52-566c-bb79-7a1a-4dfea326bf2f [Follower]" leader= +testLoggerAdapter verbose: server-27dab4f6-87ed-859b-716c-0bc8e5f26df7: 2022-08-12T07:30:16.783Z [INFO] server-27dab4f6-87ed-859b-716c-0bc8e5f26df7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3b91fa6a-c9a4-f0c2-9e36-0b49544831ca Address:3b91fa6a-c9a4-f0c2-9e36-0b49544831ca} {Suffrage:Voter ID:server-29445b52-566c-bb79-7a1a-4dfea326bf2f Address:29445b52-566c-bb79-7a1a-4dfea326bf2f} {Suffrage:Voter ID:server-27dab4f6-87ed-859b-716c-0bc8e5f26df7 Address:27dab4f6-87ed-859b-716c-0bc8e5f26df7}]" + testing.go:149: server-27dab4f6-87ed-859b-716c-0bc8e5f26df7: 2022-08-12T07:30:16.783Z [INFO] server-27dab4f6-87ed-859b-716c-0bc8e5f26df7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3b91fa6a-c9a4-f0c2-9e36-0b49544831ca Address:3b91fa6a-c9a4-f0c2-9e36-0b49544831ca} {Suffrage:Voter ID:server-29445b52-566c-bb79-7a1a-4dfea326bf2f Address:29445b52-566c-bb79-7a1a-4dfea326bf2f} {Suffrage:Voter ID:server-27dab4f6-87ed-859b-716c-0bc8e5f26df7 Address:27dab4f6-87ed-859b-716c-0bc8e5f26df7}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:16.783Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:16.783Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-27dab4f6-87ed-859b-716c-0bc8e5f26df7: 2022-08-12T07:30:16.784Z [INFO] server-27dab4f6-87ed-859b-716c-0bc8e5f26df7: entering follower state: follower="Node at 27dab4f6-87ed-859b-716c-0bc8e5f26df7 [Follower]" leader= + testing.go:149: server-27dab4f6-87ed-859b-716c-0bc8e5f26df7: 2022-08-12T07:30:16.784Z [INFO] server-27dab4f6-87ed-859b-716c-0bc8e5f26df7: entering follower state: follower="Node at 27dab4f6-87ed-859b-716c-0bc8e5f26df7 [Follower]" leader= +testLoggerAdapter verbose: server-29445b52-566c-bb79-7a1a-4dfea326bf2f: 2022-08-12T07:30:16.849Z [WARN] server-29445b52-566c-bb79-7a1a-4dfea326bf2f: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-29445b52-566c-bb79-7a1a-4dfea326bf2f: 2022-08-12T07:30:16.849Z [WARN] server-29445b52-566c-bb79-7a1a-4dfea326bf2f: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-29445b52-566c-bb79-7a1a-4dfea326bf2f: 2022-08-12T07:30:16.850Z [INFO] server-29445b52-566c-bb79-7a1a-4dfea326bf2f: entering candidate state: node="Node at 29445b52-566c-bb79-7a1a-4dfea326bf2f [Candidate]" term=2 + testing.go:149: server-29445b52-566c-bb79-7a1a-4dfea326bf2f: 2022-08-12T07:30:16.850Z [INFO] server-29445b52-566c-bb79-7a1a-4dfea326bf2f: entering candidate state: node="Node at 29445b52-566c-bb79-7a1a-4dfea326bf2f [Candidate]" term=2 +testLoggerAdapter verbose: server-29445b52-566c-bb79-7a1a-4dfea326bf2f: 2022-08-12T07:30:16.850Z [INFO] server-29445b52-566c-bb79-7a1a-4dfea326bf2f: election won: tally=2 + testing.go:149: server-29445b52-566c-bb79-7a1a-4dfea326bf2f: 2022-08-12T07:30:16.850Z [INFO] server-29445b52-566c-bb79-7a1a-4dfea326bf2f: election won: tally=2 +testLoggerAdapter verbose: server-29445b52-566c-bb79-7a1a-4dfea326bf2f: 2022-08-12T07:30:16.850Z [INFO] server-29445b52-566c-bb79-7a1a-4dfea326bf2f: entering leader state: leader="Node at 29445b52-566c-bb79-7a1a-4dfea326bf2f [Leader]" + testing.go:149: server-29445b52-566c-bb79-7a1a-4dfea326bf2f: 2022-08-12T07:30:16.850Z [INFO] server-29445b52-566c-bb79-7a1a-4dfea326bf2f: entering leader state: leader="Node at 29445b52-566c-bb79-7a1a-4dfea326bf2f [Leader]" +testLoggerAdapter verbose: server-29445b52-566c-bb79-7a1a-4dfea326bf2f: 2022-08-12T07:30:16.850Z [INFO] server-29445b52-566c-bb79-7a1a-4dfea326bf2f: added peer, starting replication: peer=server-3b91fa6a-c9a4-f0c2-9e36-0b49544831ca + testing.go:149: server-29445b52-566c-bb79-7a1a-4dfea326bf2f: 2022-08-12T07:30:16.850Z [INFO] server-29445b52-566c-bb79-7a1a-4dfea326bf2f: added peer, starting replication: peer=server-3b91fa6a-c9a4-f0c2-9e36-0b49544831ca +testLoggerAdapter verbose: server-29445b52-566c-bb79-7a1a-4dfea326bf2f: 2022-08-12T07:30:16.850Z [INFO] server-29445b52-566c-bb79-7a1a-4dfea326bf2f: added peer, starting replication: peer=server-27dab4f6-87ed-859b-716c-0bc8e5f26df7 + testing.go:149: server-29445b52-566c-bb79-7a1a-4dfea326bf2f: 2022-08-12T07:30:16.850Z [INFO] server-29445b52-566c-bb79-7a1a-4dfea326bf2f: added peer, starting replication: peer=server-27dab4f6-87ed-859b-716c-0bc8e5f26df7 +testLoggerAdapter verbose: server-29445b52-566c-bb79-7a1a-4dfea326bf2f: 2022-08-12T07:30:16.850Z [INFO] server-29445b52-566c-bb79-7a1a-4dfea326bf2f: pipelining replication: peer="{Voter server-3b91fa6a-c9a4-f0c2-9e36-0b49544831ca 3b91fa6a-c9a4-f0c2-9e36-0b49544831ca}" + testing.go:149: server-29445b52-566c-bb79-7a1a-4dfea326bf2f: 2022-08-12T07:30:16.850Z [INFO] server-29445b52-566c-bb79-7a1a-4dfea326bf2f: pipelining replication: peer="{Voter server-3b91fa6a-c9a4-f0c2-9e36-0b49544831ca 3b91fa6a-c9a4-f0c2-9e36-0b49544831ca}" +testLoggerAdapter verbose: server-29445b52-566c-bb79-7a1a-4dfea326bf2f: 2022-08-12T07:30:16.850Z [INFO] server-29445b52-566c-bb79-7a1a-4dfea326bf2f: pipelining replication: peer="{Voter server-27dab4f6-87ed-859b-716c-0bc8e5f26df7 27dab4f6-87ed-859b-716c-0bc8e5f26df7}" + testing.go:149: server-29445b52-566c-bb79-7a1a-4dfea326bf2f: 2022-08-12T07:30:16.850Z [INFO] server-29445b52-566c-bb79-7a1a-4dfea326bf2f: pipelining replication: peer="{Voter server-27dab4f6-87ed-859b-716c-0bc8e5f26df7 27dab4f6-87ed-859b-716c-0bc8e5f26df7}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:16.955Z [INFO] stable state for Leader reached at 2022-08-12 07:30:16.85048438 +0000 UTC m=+17.807170881 (1 nodes), 66.43174ms from start of poll, 67.52852ms from cluster start. Timeout at 2022-08-12 07:30:16.95551272 +0000 UTC m=+17.912199201, 105.02832ms after stability + testing.go:149: cluster: 2022-08-12T07:30:16.955Z [INFO] stable state for Leader reached at 2022-08-12 07:30:16.85048438 +0000 UTC m=+17.807170881 (1 nodes), 66.43174ms from start of poll, 67.52852ms from cluster start. Timeout at 2022-08-12 07:30:16.95551272 +0000 UTC m=+17.912199201, 105.02832ms after stability +testLoggerAdapter verbose: server-29445b52-566c-bb79-7a1a-4dfea326bf2f: 2022-08-12T07:30:16.955Z [INFO] server-29445b52-566c-bb79-7a1a-4dfea326bf2f: aborting pipeline replication: peer="{Voter server-27dab4f6-87ed-859b-716c-0bc8e5f26df7 27dab4f6-87ed-859b-716c-0bc8e5f26df7}" + testing.go:149: server-29445b52-566c-bb79-7a1a-4dfea326bf2f: 2022-08-12T07:30:16.955Z [INFO] server-29445b52-566c-bb79-7a1a-4dfea326bf2f: aborting pipeline replication: peer="{Voter server-27dab4f6-87ed-859b-716c-0bc8e5f26df7 27dab4f6-87ed-859b-716c-0bc8e5f26df7}" +testLoggerAdapter verbose: server-29445b52-566c-bb79-7a1a-4dfea326bf2f: 2022-08-12T07:30:16.955Z [INFO] server-29445b52-566c-bb79-7a1a-4dfea326bf2f: aborting pipeline replication: peer="{Voter server-3b91fa6a-c9a4-f0c2-9e36-0b49544831ca 3b91fa6a-c9a4-f0c2-9e36-0b49544831ca}" + testing.go:149: server-29445b52-566c-bb79-7a1a-4dfea326bf2f: 2022-08-12T07:30:16.955Z [INFO] server-29445b52-566c-bb79-7a1a-4dfea326bf2f: aborting pipeline replication: peer="{Voter server-3b91fa6a-c9a4-f0c2-9e36-0b49544831ca 3b91fa6a-c9a4-f0c2-9e36-0b49544831ca}" +--- PASS: TestRaft_LeadershipTransferToInvalidID (0.17s) === RUN TestRaft_LeadershipTransferToInvalidAddress -testLoggerAdapter verbose: server-b1772040-3133-b00b-7350-12856ccfada7: 2021-07-10T01:05:12.381Z [INFO] server-b1772040-3133-b00b-7350-12856ccfada7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b1772040-3133-b00b-7350-12856ccfada7 Address:b1772040-3133-b00b-7350-12856ccfada7} {Suffrage:Voter ID:server-d339639b-0c34-3129-0da2-ac4a7d72b2e9 Address:d339639b-0c34-3129-0da2-ac4a7d72b2e9} {Suffrage:Voter ID:server-5ee30373-c40a-9687-1f27-9b105599e38a Address:5ee30373-c40a-9687-1f27-9b105599e38a}]" - testing.go:149: server-b1772040-3133-b00b-7350-12856ccfada7: 2021-07-10T01:05:12.381Z [INFO] server-b1772040-3133-b00b-7350-12856ccfada7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b1772040-3133-b00b-7350-12856ccfada7 Address:b1772040-3133-b00b-7350-12856ccfada7} {Suffrage:Voter ID:server-d339639b-0c34-3129-0da2-ac4a7d72b2e9 Address:d339639b-0c34-3129-0da2-ac4a7d72b2e9} {Suffrage:Voter ID:server-5ee30373-c40a-9687-1f27-9b105599e38a Address:5ee30373-c40a-9687-1f27-9b105599e38a}]" -testLoggerAdapter verbose: server-b1772040-3133-b00b-7350-12856ccfada7: 2021-07-10T01:05:12.382Z [INFO] server-b1772040-3133-b00b-7350-12856ccfada7: entering follower state: follower="Node at b1772040-3133-b00b-7350-12856ccfada7 [Follower]" leader= - testing.go:149: server-b1772040-3133-b00b-7350-12856ccfada7: 2021-07-10T01:05:12.382Z [INFO] server-b1772040-3133-b00b-7350-12856ccfada7: entering follower state: follower="Node at b1772040-3133-b00b-7350-12856ccfada7 [Follower]" leader= -testLoggerAdapter verbose: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.382Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b1772040-3133-b00b-7350-12856ccfada7 Address:b1772040-3133-b00b-7350-12856ccfada7} {Suffrage:Voter ID:server-d339639b-0c34-3129-0da2-ac4a7d72b2e9 Address:d339639b-0c34-3129-0da2-ac4a7d72b2e9} {Suffrage:Voter ID:server-5ee30373-c40a-9687-1f27-9b105599e38a Address:5ee30373-c40a-9687-1f27-9b105599e38a}]" - testing.go:149: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.382Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b1772040-3133-b00b-7350-12856ccfada7 Address:b1772040-3133-b00b-7350-12856ccfada7} {Suffrage:Voter ID:server-d339639b-0c34-3129-0da2-ac4a7d72b2e9 Address:d339639b-0c34-3129-0da2-ac4a7d72b2e9} {Suffrage:Voter ID:server-5ee30373-c40a-9687-1f27-9b105599e38a Address:5ee30373-c40a-9687-1f27-9b105599e38a}]" -testLoggerAdapter verbose: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.382Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: entering follower state: follower="Node at d339639b-0c34-3129-0da2-ac4a7d72b2e9 [Follower]" leader= - testing.go:149: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.382Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: entering follower state: follower="Node at d339639b-0c34-3129-0da2-ac4a7d72b2e9 [Follower]" leader= -testLoggerAdapter verbose: server-5ee30373-c40a-9687-1f27-9b105599e38a: 2021-07-10T01:05:12.382Z [INFO] server-5ee30373-c40a-9687-1f27-9b105599e38a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b1772040-3133-b00b-7350-12856ccfada7 Address:b1772040-3133-b00b-7350-12856ccfada7} {Suffrage:Voter ID:server-d339639b-0c34-3129-0da2-ac4a7d72b2e9 Address:d339639b-0c34-3129-0da2-ac4a7d72b2e9} {Suffrage:Voter ID:server-5ee30373-c40a-9687-1f27-9b105599e38a Address:5ee30373-c40a-9687-1f27-9b105599e38a}]" - testing.go:149: server-5ee30373-c40a-9687-1f27-9b105599e38a: 2021-07-10T01:05:12.382Z [INFO] server-5ee30373-c40a-9687-1f27-9b105599e38a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b1772040-3133-b00b-7350-12856ccfada7 Address:b1772040-3133-b00b-7350-12856ccfada7} {Suffrage:Voter ID:server-d339639b-0c34-3129-0da2-ac4a7d72b2e9 Address:d339639b-0c34-3129-0da2-ac4a7d72b2e9} {Suffrage:Voter ID:server-5ee30373-c40a-9687-1f27-9b105599e38a Address:5ee30373-c40a-9687-1f27-9b105599e38a}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:12.382Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-07-10T01:05:12.382Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-5ee30373-c40a-9687-1f27-9b105599e38a: 2021-07-10T01:05:12.382Z [INFO] server-5ee30373-c40a-9687-1f27-9b105599e38a: entering follower state: follower="Node at 5ee30373-c40a-9687-1f27-9b105599e38a [Follower]" leader= - testing.go:149: server-5ee30373-c40a-9687-1f27-9b105599e38a: 2021-07-10T01:05:12.382Z [INFO] server-5ee30373-c40a-9687-1f27-9b105599e38a: entering follower state: follower="Node at 5ee30373-c40a-9687-1f27-9b105599e38a [Follower]" leader= -testLoggerAdapter verbose: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.460Z [WARN] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.460Z [WARN] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.460Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: entering candidate state: node="Node at d339639b-0c34-3129-0da2-ac4a7d72b2e9 [Candidate]" term=2 - testing.go:149: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.460Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: entering candidate state: node="Node at d339639b-0c34-3129-0da2-ac4a7d72b2e9 [Candidate]" term=2 -testLoggerAdapter verbose: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.460Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: election won: tally=2 - testing.go:149: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.460Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: election won: tally=2 -testLoggerAdapter verbose: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.460Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: entering leader state: leader="Node at d339639b-0c34-3129-0da2-ac4a7d72b2e9 [Leader]" - testing.go:149: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.460Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: entering leader state: leader="Node at d339639b-0c34-3129-0da2-ac4a7d72b2e9 [Leader]" -testLoggerAdapter verbose: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.460Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: added peer, starting replication: peer=server-b1772040-3133-b00b-7350-12856ccfada7 - testing.go:149: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.460Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: added peer, starting replication: peer=server-b1772040-3133-b00b-7350-12856ccfada7 -testLoggerAdapter verbose: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.460Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: added peer, starting replication: peer=server-5ee30373-c40a-9687-1f27-9b105599e38a - testing.go:149: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.460Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: added peer, starting replication: peer=server-5ee30373-c40a-9687-1f27-9b105599e38a -testLoggerAdapter verbose: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.460Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: pipelining replication: peer="{Voter server-b1772040-3133-b00b-7350-12856ccfada7 b1772040-3133-b00b-7350-12856ccfada7}" - testing.go:149: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.460Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: pipelining replication: peer="{Voter server-b1772040-3133-b00b-7350-12856ccfada7 b1772040-3133-b00b-7350-12856ccfada7}" -testLoggerAdapter verbose: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.460Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: pipelining replication: peer="{Voter server-5ee30373-c40a-9687-1f27-9b105599e38a 5ee30373-c40a-9687-1f27-9b105599e38a}" - testing.go:149: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.460Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: pipelining replication: peer="{Voter server-5ee30373-c40a-9687-1f27-9b105599e38a 5ee30373-c40a-9687-1f27-9b105599e38a}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:12.565Z [INFO] stable state for Follower reached at 2021-07-10 01:05:12.4607527 +0000 UTC m=+17.848910650 (2 nodes), 77.884761ms from start of poll, 79.077434ms from cluster start. Timeout at 2021-07-10 01:05:12.565821748 +0000 UTC m=+17.953979718, 105.069068ms after stability - testing.go:149: cluster: 2021-07-10T01:05:12.565Z [INFO] stable state for Follower reached at 2021-07-10 01:05:12.4607527 +0000 UTC m=+17.848910650 (2 nodes), 77.884761ms from start of poll, 79.077434ms from cluster start. Timeout at 2021-07-10 01:05:12.565821748 +0000 UTC m=+17.953979718, 105.069068ms after stability -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:12.566Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:12.566Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:12.671Z [INFO] stable state for Leader reached at 2021-07-10 01:05:12.56606609 +0000 UTC m=+17.954224060 (1 nodes), 3.3µs from start of poll, 184.390844ms from cluster start. Timeout at 2021-07-10 01:05:12.671101018 +0000 UTC m=+18.059259008, 105.034948ms after stability - testing.go:149: cluster: 2021-07-10T01:05:12.671Z [INFO] stable state for Leader reached at 2021-07-10 01:05:12.56606609 +0000 UTC m=+17.954224060 (1 nodes), 3.3µs from start of poll, 184.390844ms from cluster start. Timeout at 2021-07-10 01:05:12.671101018 +0000 UTC m=+18.059259008, 105.034948ms after stability -testLoggerAdapter verbose: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.671Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: aborting pipeline replication: peer="{Voter server-5ee30373-c40a-9687-1f27-9b105599e38a 5ee30373-c40a-9687-1f27-9b105599e38a}" - testing.go:149: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.671Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: aborting pipeline replication: peer="{Voter server-5ee30373-c40a-9687-1f27-9b105599e38a 5ee30373-c40a-9687-1f27-9b105599e38a}" -testLoggerAdapter verbose: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.671Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: aborting pipeline replication: peer="{Voter server-b1772040-3133-b00b-7350-12856ccfada7 b1772040-3133-b00b-7350-12856ccfada7}" - testing.go:149: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.671Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: aborting pipeline replication: peer="{Voter server-b1772040-3133-b00b-7350-12856ccfada7 b1772040-3133-b00b-7350-12856ccfada7}" ---- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.29s) +testLoggerAdapter verbose: server-86e667fa-9261-57d1-ebd1-739d59b4a5a5: 2022-08-12T07:30:16.957Z [INFO] server-86e667fa-9261-57d1-ebd1-739d59b4a5a5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-86e667fa-9261-57d1-ebd1-739d59b4a5a5 Address:86e667fa-9261-57d1-ebd1-739d59b4a5a5} {Suffrage:Voter ID:server-21db8f52-9522-8d03-a396-3199f4ee5e96 Address:21db8f52-9522-8d03-a396-3199f4ee5e96} {Suffrage:Voter ID:server-30e3d08e-ef4f-a401-ecf2-04d92fb79747 Address:30e3d08e-ef4f-a401-ecf2-04d92fb79747}]" + testing.go:149: server-86e667fa-9261-57d1-ebd1-739d59b4a5a5: 2022-08-12T07:30:16.957Z [INFO] server-86e667fa-9261-57d1-ebd1-739d59b4a5a5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-86e667fa-9261-57d1-ebd1-739d59b4a5a5 Address:86e667fa-9261-57d1-ebd1-739d59b4a5a5} {Suffrage:Voter ID:server-21db8f52-9522-8d03-a396-3199f4ee5e96 Address:21db8f52-9522-8d03-a396-3199f4ee5e96} {Suffrage:Voter ID:server-30e3d08e-ef4f-a401-ecf2-04d92fb79747 Address:30e3d08e-ef4f-a401-ecf2-04d92fb79747}]" +testLoggerAdapter verbose: server-86e667fa-9261-57d1-ebd1-739d59b4a5a5: 2022-08-12T07:30:16.957Z [INFO] server-86e667fa-9261-57d1-ebd1-739d59b4a5a5: entering follower state: follower="Node at 86e667fa-9261-57d1-ebd1-739d59b4a5a5 [Follower]" leader= +testLoggerAdapter verbose: server-21db8f52-9522-8d03-a396-3199f4ee5e96: 2022-08-12T07:30:16.957Z [INFO] server-21db8f52-9522-8d03-a396-3199f4ee5e96: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-86e667fa-9261-57d1-ebd1-739d59b4a5a5 Address:86e667fa-9261-57d1-ebd1-739d59b4a5a5} {Suffrage:Voter ID:server-21db8f52-9522-8d03-a396-3199f4ee5e96 Address:21db8f52-9522-8d03-a396-3199f4ee5e96} {Suffrage:Voter ID:server-30e3d08e-ef4f-a401-ecf2-04d92fb79747 Address:30e3d08e-ef4f-a401-ecf2-04d92fb79747}]" + testing.go:149: server-86e667fa-9261-57d1-ebd1-739d59b4a5a5: 2022-08-12T07:30:16.957Z [INFO] server-86e667fa-9261-57d1-ebd1-739d59b4a5a5: entering follower state: follower="Node at 86e667fa-9261-57d1-ebd1-739d59b4a5a5 [Follower]" leader= + testing.go:149: server-21db8f52-9522-8d03-a396-3199f4ee5e96: 2022-08-12T07:30:16.957Z [INFO] server-21db8f52-9522-8d03-a396-3199f4ee5e96: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-86e667fa-9261-57d1-ebd1-739d59b4a5a5 Address:86e667fa-9261-57d1-ebd1-739d59b4a5a5} {Suffrage:Voter ID:server-21db8f52-9522-8d03-a396-3199f4ee5e96 Address:21db8f52-9522-8d03-a396-3199f4ee5e96} {Suffrage:Voter ID:server-30e3d08e-ef4f-a401-ecf2-04d92fb79747 Address:30e3d08e-ef4f-a401-ecf2-04d92fb79747}]" +testLoggerAdapter verbose: server-21db8f52-9522-8d03-a396-3199f4ee5e96: 2022-08-12T07:30:16.957Z [INFO] server-21db8f52-9522-8d03-a396-3199f4ee5e96: entering follower state: follower="Node at 21db8f52-9522-8d03-a396-3199f4ee5e96 [Follower]" leader= + testing.go:149: server-21db8f52-9522-8d03-a396-3199f4ee5e96: 2022-08-12T07:30:16.957Z [INFO] server-21db8f52-9522-8d03-a396-3199f4ee5e96: entering follower state: follower="Node at 21db8f52-9522-8d03-a396-3199f4ee5e96 [Follower]" leader= +testLoggerAdapter verbose: server-30e3d08e-ef4f-a401-ecf2-04d92fb79747: 2022-08-12T07:30:16.958Z [INFO] server-30e3d08e-ef4f-a401-ecf2-04d92fb79747: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-86e667fa-9261-57d1-ebd1-739d59b4a5a5 Address:86e667fa-9261-57d1-ebd1-739d59b4a5a5} {Suffrage:Voter ID:server-21db8f52-9522-8d03-a396-3199f4ee5e96 Address:21db8f52-9522-8d03-a396-3199f4ee5e96} {Suffrage:Voter ID:server-30e3d08e-ef4f-a401-ecf2-04d92fb79747 Address:30e3d08e-ef4f-a401-ecf2-04d92fb79747}]" + testing.go:149: server-30e3d08e-ef4f-a401-ecf2-04d92fb79747: 2022-08-12T07:30:16.958Z [INFO] server-30e3d08e-ef4f-a401-ecf2-04d92fb79747: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-86e667fa-9261-57d1-ebd1-739d59b4a5a5 Address:86e667fa-9261-57d1-ebd1-739d59b4a5a5} {Suffrage:Voter ID:server-21db8f52-9522-8d03-a396-3199f4ee5e96 Address:21db8f52-9522-8d03-a396-3199f4ee5e96} {Suffrage:Voter ID:server-30e3d08e-ef4f-a401-ecf2-04d92fb79747 Address:30e3d08e-ef4f-a401-ecf2-04d92fb79747}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:16.958Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2022-08-12T07:30:16.958Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-30e3d08e-ef4f-a401-ecf2-04d92fb79747: 2022-08-12T07:30:16.958Z [INFO] server-30e3d08e-ef4f-a401-ecf2-04d92fb79747: entering follower state: follower="Node at 30e3d08e-ef4f-a401-ecf2-04d92fb79747 [Follower]" leader= + testing.go:149: server-30e3d08e-ef4f-a401-ecf2-04d92fb79747: 2022-08-12T07:30:16.958Z [INFO] server-30e3d08e-ef4f-a401-ecf2-04d92fb79747: entering follower state: follower="Node at 30e3d08e-ef4f-a401-ecf2-04d92fb79747 [Follower]" leader= +testLoggerAdapter verbose: server-21db8f52-9522-8d03-a396-3199f4ee5e96: 2022-08-12T07:30:17.021Z [WARN] server-21db8f52-9522-8d03-a396-3199f4ee5e96: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-21db8f52-9522-8d03-a396-3199f4ee5e96: 2022-08-12T07:30:17.021Z [WARN] server-21db8f52-9522-8d03-a396-3199f4ee5e96: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-21db8f52-9522-8d03-a396-3199f4ee5e96: 2022-08-12T07:30:17.021Z [INFO] server-21db8f52-9522-8d03-a396-3199f4ee5e96: entering candidate state: node="Node at 21db8f52-9522-8d03-a396-3199f4ee5e96 [Candidate]" term=2 + testing.go:149: server-21db8f52-9522-8d03-a396-3199f4ee5e96: 2022-08-12T07:30:17.021Z [INFO] server-21db8f52-9522-8d03-a396-3199f4ee5e96: entering candidate state: node="Node at 21db8f52-9522-8d03-a396-3199f4ee5e96 [Candidate]" term=2 +testLoggerAdapter verbose: server-21db8f52-9522-8d03-a396-3199f4ee5e96: 2022-08-12T07:30:17.022Z [INFO] server-21db8f52-9522-8d03-a396-3199f4ee5e96: election won: tally=2 + testing.go:149: server-21db8f52-9522-8d03-a396-3199f4ee5e96: 2022-08-12T07:30:17.022Z [INFO] server-21db8f52-9522-8d03-a396-3199f4ee5e96: election won: tally=2 +testLoggerAdapter verbose: server-21db8f52-9522-8d03-a396-3199f4ee5e96: 2022-08-12T07:30:17.022Z [INFO] server-21db8f52-9522-8d03-a396-3199f4ee5e96: entering leader state: leader="Node at 21db8f52-9522-8d03-a396-3199f4ee5e96 [Leader]" + testing.go:149: server-21db8f52-9522-8d03-a396-3199f4ee5e96: 2022-08-12T07:30:17.022Z [INFO] server-21db8f52-9522-8d03-a396-3199f4ee5e96: entering leader state: leader="Node at 21db8f52-9522-8d03-a396-3199f4ee5e96 [Leader]" +testLoggerAdapter verbose: server-21db8f52-9522-8d03-a396-3199f4ee5e96: 2022-08-12T07:30:17.022Z [INFO] server-21db8f52-9522-8d03-a396-3199f4ee5e96: added peer, starting replication: peer=server-86e667fa-9261-57d1-ebd1-739d59b4a5a5 + testing.go:149: server-21db8f52-9522-8d03-a396-3199f4ee5e96: 2022-08-12T07:30:17.022Z [INFO] server-21db8f52-9522-8d03-a396-3199f4ee5e96: added peer, starting replication: peer=server-86e667fa-9261-57d1-ebd1-739d59b4a5a5 +testLoggerAdapter verbose: server-21db8f52-9522-8d03-a396-3199f4ee5e96: 2022-08-12T07:30:17.022Z [INFO] server-21db8f52-9522-8d03-a396-3199f4ee5e96: added peer, starting replication: peer=server-30e3d08e-ef4f-a401-ecf2-04d92fb79747 + testing.go:149: server-21db8f52-9522-8d03-a396-3199f4ee5e96: 2022-08-12T07:30:17.022Z [INFO] server-21db8f52-9522-8d03-a396-3199f4ee5e96: added peer, starting replication: peer=server-30e3d08e-ef4f-a401-ecf2-04d92fb79747 +testLoggerAdapter verbose: server-21db8f52-9522-8d03-a396-3199f4ee5e96: 2022-08-12T07:30:17.022Z [INFO] server-21db8f52-9522-8d03-a396-3199f4ee5e96: pipelining replication: peer="{Voter server-86e667fa-9261-57d1-ebd1-739d59b4a5a5 86e667fa-9261-57d1-ebd1-739d59b4a5a5}" + testing.go:149: server-21db8f52-9522-8d03-a396-3199f4ee5e96: 2022-08-12T07:30:17.022Z [INFO] server-21db8f52-9522-8d03-a396-3199f4ee5e96: pipelining replication: peer="{Voter server-86e667fa-9261-57d1-ebd1-739d59b4a5a5 86e667fa-9261-57d1-ebd1-739d59b4a5a5}" +testLoggerAdapter verbose: server-21db8f52-9522-8d03-a396-3199f4ee5e96: 2022-08-12T07:30:17.023Z [INFO] server-21db8f52-9522-8d03-a396-3199f4ee5e96: pipelining replication: peer="{Voter server-30e3d08e-ef4f-a401-ecf2-04d92fb79747 30e3d08e-ef4f-a401-ecf2-04d92fb79747}" + testing.go:149: server-21db8f52-9522-8d03-a396-3199f4ee5e96: 2022-08-12T07:30:17.023Z [INFO] server-21db8f52-9522-8d03-a396-3199f4ee5e96: pipelining replication: peer="{Voter server-30e3d08e-ef4f-a401-ecf2-04d92fb79747 30e3d08e-ef4f-a401-ecf2-04d92fb79747}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:17.127Z [INFO] stable state for Follower reached at 2022-08-12 07:30:17.02269502 +0000 UTC m=+17.979381501 (2 nodes), 64.50264ms from start of poll, 65.5277ms from cluster start. Timeout at 2022-08-12 07:30:17.12775906 +0000 UTC m=+18.084445541, 105.06404ms after stability + testing.go:149: cluster: 2022-08-12T07:30:17.127Z [INFO] stable state for Follower reached at 2022-08-12 07:30:17.02269502 +0000 UTC m=+17.979381501 (2 nodes), 64.50264ms from start of poll, 65.5277ms from cluster start. Timeout at 2022-08-12 07:30:17.12775906 +0000 UTC m=+18.084445541, 105.06404ms after stability +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:17.127Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:17.127Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:17.233Z [INFO] stable state for Leader reached at 2022-08-12 07:30:17.1280685 +0000 UTC m=+18.084754981 (1 nodes), 3.94µs from start of poll, 170.90118ms from cluster start. Timeout at 2022-08-12 07:30:17.23314894 +0000 UTC m=+18.189835421, 105.08044ms after stability + testing.go:149: cluster: 2022-08-12T07:30:17.233Z [INFO] stable state for Leader reached at 2022-08-12 07:30:17.1280685 +0000 UTC m=+18.084754981 (1 nodes), 3.94µs from start of poll, 170.90118ms from cluster start. Timeout at 2022-08-12 07:30:17.23314894 +0000 UTC m=+18.189835421, 105.08044ms after stability +testLoggerAdapter verbose: server-21db8f52-9522-8d03-a396-3199f4ee5e96: 2022-08-12T07:30:17.233Z [INFO] server-21db8f52-9522-8d03-a396-3199f4ee5e96: aborting pipeline replication: peer="{Voter server-30e3d08e-ef4f-a401-ecf2-04d92fb79747 30e3d08e-ef4f-a401-ecf2-04d92fb79747}" + testing.go:149: server-21db8f52-9522-8d03-a396-3199f4ee5e96: 2022-08-12T07:30:17.233Z [INFO] server-21db8f52-9522-8d03-a396-3199f4ee5e96: aborting pipeline replication: peer="{Voter server-30e3d08e-ef4f-a401-ecf2-04d92fb79747 30e3d08e-ef4f-a401-ecf2-04d92fb79747}" +testLoggerAdapter verbose: server-21db8f52-9522-8d03-a396-3199f4ee5e96: 2022-08-12T07:30:17.233Z [INFO] server-21db8f52-9522-8d03-a396-3199f4ee5e96: aborting pipeline replication: peer="{Voter server-86e667fa-9261-57d1-ebd1-739d59b4a5a5 86e667fa-9261-57d1-ebd1-739d59b4a5a5}" + testing.go:149: server-21db8f52-9522-8d03-a396-3199f4ee5e96: 2022-08-12T07:30:17.233Z [INFO] server-21db8f52-9522-8d03-a396-3199f4ee5e96: aborting pipeline replication: peer="{Voter server-86e667fa-9261-57d1-ebd1-739d59b4a5a5 86e667fa-9261-57d1-ebd1-739d59b4a5a5}" +--- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.28s) === RUN TestRaft_LeadershipTransferToBehindServer -testLoggerAdapter verbose: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.673Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1dd37b1b-6d57-d208-7177-96f51751af2c Address:1dd37b1b-6d57-d208-7177-96f51751af2c} {Suffrage:Voter ID:server-5be7c512-6ae8-11d8-6ad2-be45b2bec328 Address:5be7c512-6ae8-11d8-6ad2-be45b2bec328} {Suffrage:Voter ID:server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f Address:c5a447e6-f091-7f22-7c30-c48d1dbbf37f}]" - testing.go:149: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.673Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1dd37b1b-6d57-d208-7177-96f51751af2c Address:1dd37b1b-6d57-d208-7177-96f51751af2c} {Suffrage:Voter ID:server-5be7c512-6ae8-11d8-6ad2-be45b2bec328 Address:5be7c512-6ae8-11d8-6ad2-be45b2bec328} {Suffrage:Voter ID:server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f Address:c5a447e6-f091-7f22-7c30-c48d1dbbf37f}]" -testLoggerAdapter verbose: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.673Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: entering follower state: follower="Node at 1dd37b1b-6d57-d208-7177-96f51751af2c [Follower]" leader= - testing.go:149: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.673Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: entering follower state: follower="Node at 1dd37b1b-6d57-d208-7177-96f51751af2c [Follower]" leader= -testLoggerAdapter verbose: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:12.673Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1dd37b1b-6d57-d208-7177-96f51751af2c Address:1dd37b1b-6d57-d208-7177-96f51751af2c} {Suffrage:Voter ID:server-5be7c512-6ae8-11d8-6ad2-be45b2bec328 Address:5be7c512-6ae8-11d8-6ad2-be45b2bec328} {Suffrage:Voter ID:server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f Address:c5a447e6-f091-7f22-7c30-c48d1dbbf37f}]" - testing.go:149: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:12.673Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1dd37b1b-6d57-d208-7177-96f51751af2c Address:1dd37b1b-6d57-d208-7177-96f51751af2c} {Suffrage:Voter ID:server-5be7c512-6ae8-11d8-6ad2-be45b2bec328 Address:5be7c512-6ae8-11d8-6ad2-be45b2bec328} {Suffrage:Voter ID:server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f Address:c5a447e6-f091-7f22-7c30-c48d1dbbf37f}]" -testLoggerAdapter verbose: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:12.673Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: entering follower state: follower="Node at 5be7c512-6ae8-11d8-6ad2-be45b2bec328 [Follower]" leader= - testing.go:149: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:12.673Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: entering follower state: follower="Node at 5be7c512-6ae8-11d8-6ad2-be45b2bec328 [Follower]" leader= -testLoggerAdapter verbose: server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f: 2021-07-10T01:05:12.673Z [INFO] server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1dd37b1b-6d57-d208-7177-96f51751af2c Address:1dd37b1b-6d57-d208-7177-96f51751af2c} {Suffrage:Voter ID:server-5be7c512-6ae8-11d8-6ad2-be45b2bec328 Address:5be7c512-6ae8-11d8-6ad2-be45b2bec328} {Suffrage:Voter ID:server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f Address:c5a447e6-f091-7f22-7c30-c48d1dbbf37f}]" - testing.go:149: server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f: 2021-07-10T01:05:12.673Z [INFO] server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1dd37b1b-6d57-d208-7177-96f51751af2c Address:1dd37b1b-6d57-d208-7177-96f51751af2c} {Suffrage:Voter ID:server-5be7c512-6ae8-11d8-6ad2-be45b2bec328 Address:5be7c512-6ae8-11d8-6ad2-be45b2bec328} {Suffrage:Voter ID:server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f Address:c5a447e6-f091-7f22-7c30-c48d1dbbf37f}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:12.673Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:12.673Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f: 2021-07-10T01:05:12.673Z [INFO] server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f: entering follower state: follower="Node at c5a447e6-f091-7f22-7c30-c48d1dbbf37f [Follower]" leader= - testing.go:149: server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f: 2021-07-10T01:05:12.673Z [INFO] server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f: entering follower state: follower="Node at c5a447e6-f091-7f22-7c30-c48d1dbbf37f [Follower]" leader= -testLoggerAdapter verbose: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.748Z [WARN] server-1dd37b1b-6d57-d208-7177-96f51751af2c: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.748Z [WARN] server-1dd37b1b-6d57-d208-7177-96f51751af2c: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.748Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: entering candidate state: node="Node at 1dd37b1b-6d57-d208-7177-96f51751af2c [Candidate]" term=2 - testing.go:149: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.748Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: entering candidate state: node="Node at 1dd37b1b-6d57-d208-7177-96f51751af2c [Candidate]" term=2 -testLoggerAdapter verbose: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.749Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: election won: tally=2 - testing.go:149: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.749Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: election won: tally=2 -testLoggerAdapter verbose: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.749Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: entering leader state: leader="Node at 1dd37b1b-6d57-d208-7177-96f51751af2c [Leader]" - testing.go:149: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.749Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: entering leader state: leader="Node at 1dd37b1b-6d57-d208-7177-96f51751af2c [Leader]" -testLoggerAdapter verbose: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.749Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: added peer, starting replication: peer=server-5be7c512-6ae8-11d8-6ad2-be45b2bec328 - testing.go:149: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.749Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: added peer, starting replication: peer=server-5be7c512-6ae8-11d8-6ad2-be45b2bec328 -testLoggerAdapter verbose: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.749Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: added peer, starting replication: peer=server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f - testing.go:149: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.749Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: added peer, starting replication: peer=server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f -testLoggerAdapter verbose: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.749Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: pipelining replication: peer="{Voter server-5be7c512-6ae8-11d8-6ad2-be45b2bec328 5be7c512-6ae8-11d8-6ad2-be45b2bec328}" - testing.go:149: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.749Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: pipelining replication: peer="{Voter server-5be7c512-6ae8-11d8-6ad2-be45b2bec328 5be7c512-6ae8-11d8-6ad2-be45b2bec328}" -testLoggerAdapter verbose: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.749Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: pipelining replication: peer="{Voter server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f c5a447e6-f091-7f22-7c30-c48d1dbbf37f}" - testing.go:149: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.749Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: pipelining replication: peer="{Voter server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f c5a447e6-f091-7f22-7c30-c48d1dbbf37f}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:12.854Z [INFO] stable state for Leader reached at 2021-07-10 01:05:12.74914728 +0000 UTC m=+18.137305230 (1 nodes), 75.171892ms from start of poll, 76.190763ms from cluster start. Timeout at 2021-07-10 01:05:12.854235728 +0000 UTC m=+18.242393698, 105.088468ms after stability - testing.go:149: cluster: 2021-07-10T01:05:12.854Z [INFO] stable state for Leader reached at 2021-07-10 01:05:12.74914728 +0000 UTC m=+18.137305230 (1 nodes), 75.171892ms from start of poll, 76.190763ms from cluster start. Timeout at 2021-07-10 01:05:12.854235728 +0000 UTC m=+18.242393698, 105.088468ms after stability -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:12.854Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-07-10T01:05:12.854Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:12.959Z [INFO] stable state for Follower reached at 2021-07-10 01:05:12.854476171 +0000 UTC m=+18.242634161 (2 nodes), 3.04µs from start of poll, 181.519694ms from cluster start. Timeout at 2021-07-10 01:05:12.959553339 +0000 UTC m=+18.347711309, 105.077148ms after stability - testing.go:149: cluster: 2021-07-10T01:05:12.959Z [INFO] stable state for Follower reached at 2021-07-10 01:05:12.854476171 +0000 UTC m=+18.242634161 (2 nodes), 3.04µs from start of poll, 181.519694ms from cluster start. Timeout at 2021-07-10 01:05:12.959553339 +0000 UTC m=+18.347711309, 105.077148ms after stability -testLoggerAdapter verbose: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:12.995Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: entering candidate state: node="Node at 5be7c512-6ae8-11d8-6ad2-be45b2bec328 [Candidate]" term=3 -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:12.996Z [INFO] starting stability test: raft-state=Leader - testing.go:149: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:12.995Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: entering candidate state: node="Node at 5be7c512-6ae8-11d8-6ad2-be45b2bec328 [Candidate]" term=3 - testing.go:149: cluster: 2021-07-10T01:05:12.996Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.996Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: entering follower state: follower="Node at 1dd37b1b-6d57-d208-7177-96f51751af2c [Follower]" leader= - testing.go:149: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.996Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: entering follower state: follower="Node at 1dd37b1b-6d57-d208-7177-96f51751af2c [Follower]" leader= -testLoggerAdapter verbose: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:12.996Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: election won: tally=2 -testLoggerAdapter verbose: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.996Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: aborting pipeline replication: peer="{Voter server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f c5a447e6-f091-7f22-7c30-c48d1dbbf37f}" - testing.go:149: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.996Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: aborting pipeline replication: peer="{Voter server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f c5a447e6-f091-7f22-7c30-c48d1dbbf37f}" - testing.go:149: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:12.996Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: election won: tally=2 -testLoggerAdapter verbose: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:12.996Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: entering leader state: leader="Node at 5be7c512-6ae8-11d8-6ad2-be45b2bec328 [Leader]" -testLoggerAdapter verbose: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.996Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: aborting pipeline replication: peer="{Voter server-5be7c512-6ae8-11d8-6ad2-be45b2bec328 5be7c512-6ae8-11d8-6ad2-be45b2bec328}" - testing.go:149: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:12.996Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: entering leader state: leader="Node at 5be7c512-6ae8-11d8-6ad2-be45b2bec328 [Leader]" -testLoggerAdapter verbose: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:12.997Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: added peer, starting replication: peer=server-1dd37b1b-6d57-d208-7177-96f51751af2c - testing.go:149: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.996Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: aborting pipeline replication: peer="{Voter server-5be7c512-6ae8-11d8-6ad2-be45b2bec328 5be7c512-6ae8-11d8-6ad2-be45b2bec328}" - testing.go:149: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:12.997Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: added peer, starting replication: peer=server-1dd37b1b-6d57-d208-7177-96f51751af2c -testLoggerAdapter verbose: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:12.997Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: added peer, starting replication: peer=server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f - testing.go:149: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:12.997Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: added peer, starting replication: peer=server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f -testLoggerAdapter verbose: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:12.997Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: pipelining replication: peer="{Voter server-1dd37b1b-6d57-d208-7177-96f51751af2c 1dd37b1b-6d57-d208-7177-96f51751af2c}" - testing.go:149: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:12.997Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: pipelining replication: peer="{Voter server-1dd37b1b-6d57-d208-7177-96f51751af2c 1dd37b1b-6d57-d208-7177-96f51751af2c}" -testLoggerAdapter verbose: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:12.997Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: pipelining replication: peer="{Voter server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f c5a447e6-f091-7f22-7c30-c48d1dbbf37f}" - testing.go:149: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:12.997Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: pipelining replication: peer="{Voter server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f c5a447e6-f091-7f22-7c30-c48d1dbbf37f}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:13.101Z [INFO] stable state for Leader reached at 2021-07-10 01:05:12.99670791 +0000 UTC m=+18.384865880 (1 nodes), 430.184µs from start of poll, 323.751413ms from cluster start. Timeout at 2021-07-10 01:05:13.101790738 +0000 UTC m=+18.489948728, 105.082848ms after stability - testing.go:149: cluster: 2021-07-10T01:05:13.101Z [INFO] stable state for Leader reached at 2021-07-10 01:05:12.99670791 +0000 UTC m=+18.384865880 (1 nodes), 430.184µs from start of poll, 323.751413ms from cluster start. Timeout at 2021-07-10 01:05:13.101790738 +0000 UTC m=+18.489948728, 105.082848ms after stability -testLoggerAdapter verbose: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:13.102Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: aborting pipeline replication: peer="{Voter server-1dd37b1b-6d57-d208-7177-96f51751af2c 1dd37b1b-6d57-d208-7177-96f51751af2c}" - testing.go:149: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:13.102Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: aborting pipeline replication: peer="{Voter server-1dd37b1b-6d57-d208-7177-96f51751af2c 1dd37b1b-6d57-d208-7177-96f51751af2c}" -testLoggerAdapter verbose: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:13.102Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: aborting pipeline replication: peer="{Voter server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f c5a447e6-f091-7f22-7c30-c48d1dbbf37f}" - testing.go:149: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:13.102Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: aborting pipeline replication: peer="{Voter server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f c5a447e6-f091-7f22-7c30-c48d1dbbf37f}" ---- PASS: TestRaft_LeadershipTransferToBehindServer (0.43s) +testLoggerAdapter verbose: server-b54abe8b-730f-1012-e4bc-0d54e9161592: 2022-08-12T07:30:17.235Z [INFO] server-b54abe8b-730f-1012-e4bc-0d54e9161592: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b54abe8b-730f-1012-e4bc-0d54e9161592 Address:b54abe8b-730f-1012-e4bc-0d54e9161592} {Suffrage:Voter ID:server-94715de0-1898-0279-571b-589005d0b04f Address:94715de0-1898-0279-571b-589005d0b04f} {Suffrage:Voter ID:server-602dc88e-2518-f89e-330b-ddf5451e2b7f Address:602dc88e-2518-f89e-330b-ddf5451e2b7f}]" + testing.go:149: server-b54abe8b-730f-1012-e4bc-0d54e9161592: 2022-08-12T07:30:17.235Z [INFO] server-b54abe8b-730f-1012-e4bc-0d54e9161592: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b54abe8b-730f-1012-e4bc-0d54e9161592 Address:b54abe8b-730f-1012-e4bc-0d54e9161592} {Suffrage:Voter ID:server-94715de0-1898-0279-571b-589005d0b04f Address:94715de0-1898-0279-571b-589005d0b04f} {Suffrage:Voter ID:server-602dc88e-2518-f89e-330b-ddf5451e2b7f Address:602dc88e-2518-f89e-330b-ddf5451e2b7f}]" +testLoggerAdapter verbose: server-94715de0-1898-0279-571b-589005d0b04f: 2022-08-12T07:30:17.235Z [INFO] server-94715de0-1898-0279-571b-589005d0b04f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b54abe8b-730f-1012-e4bc-0d54e9161592 Address:b54abe8b-730f-1012-e4bc-0d54e9161592} {Suffrage:Voter ID:server-94715de0-1898-0279-571b-589005d0b04f Address:94715de0-1898-0279-571b-589005d0b04f} {Suffrage:Voter ID:server-602dc88e-2518-f89e-330b-ddf5451e2b7f Address:602dc88e-2518-f89e-330b-ddf5451e2b7f}]" + testing.go:149: server-94715de0-1898-0279-571b-589005d0b04f: 2022-08-12T07:30:17.235Z [INFO] server-94715de0-1898-0279-571b-589005d0b04f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b54abe8b-730f-1012-e4bc-0d54e9161592 Address:b54abe8b-730f-1012-e4bc-0d54e9161592} {Suffrage:Voter ID:server-94715de0-1898-0279-571b-589005d0b04f Address:94715de0-1898-0279-571b-589005d0b04f} {Suffrage:Voter ID:server-602dc88e-2518-f89e-330b-ddf5451e2b7f Address:602dc88e-2518-f89e-330b-ddf5451e2b7f}]" +testLoggerAdapter verbose: server-602dc88e-2518-f89e-330b-ddf5451e2b7f: 2022-08-12T07:30:17.235Z [INFO] server-602dc88e-2518-f89e-330b-ddf5451e2b7f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b54abe8b-730f-1012-e4bc-0d54e9161592 Address:b54abe8b-730f-1012-e4bc-0d54e9161592} {Suffrage:Voter ID:server-94715de0-1898-0279-571b-589005d0b04f Address:94715de0-1898-0279-571b-589005d0b04f} {Suffrage:Voter ID:server-602dc88e-2518-f89e-330b-ddf5451e2b7f Address:602dc88e-2518-f89e-330b-ddf5451e2b7f}]" + testing.go:149: server-602dc88e-2518-f89e-330b-ddf5451e2b7f: 2022-08-12T07:30:17.235Z [INFO] server-602dc88e-2518-f89e-330b-ddf5451e2b7f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b54abe8b-730f-1012-e4bc-0d54e9161592 Address:b54abe8b-730f-1012-e4bc-0d54e9161592} {Suffrage:Voter ID:server-94715de0-1898-0279-571b-589005d0b04f Address:94715de0-1898-0279-571b-589005d0b04f} {Suffrage:Voter ID:server-602dc88e-2518-f89e-330b-ddf5451e2b7f Address:602dc88e-2518-f89e-330b-ddf5451e2b7f}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:17.235Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:17.235Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-b54abe8b-730f-1012-e4bc-0d54e9161592: 2022-08-12T07:30:17.235Z [INFO] server-b54abe8b-730f-1012-e4bc-0d54e9161592: entering follower state: follower="Node at b54abe8b-730f-1012-e4bc-0d54e9161592 [Follower]" leader= + testing.go:149: server-b54abe8b-730f-1012-e4bc-0d54e9161592: 2022-08-12T07:30:17.235Z [INFO] server-b54abe8b-730f-1012-e4bc-0d54e9161592: entering follower state: follower="Node at b54abe8b-730f-1012-e4bc-0d54e9161592 [Follower]" leader= +testLoggerAdapter verbose: server-94715de0-1898-0279-571b-589005d0b04f: 2022-08-12T07:30:17.235Z [INFO] server-94715de0-1898-0279-571b-589005d0b04f: entering follower state: follower="Node at 94715de0-1898-0279-571b-589005d0b04f [Follower]" leader= +testLoggerAdapter verbose: server-602dc88e-2518-f89e-330b-ddf5451e2b7f: 2022-08-12T07:30:17.235Z [INFO] server-602dc88e-2518-f89e-330b-ddf5451e2b7f: entering follower state: follower="Node at 602dc88e-2518-f89e-330b-ddf5451e2b7f [Follower]" leader= + testing.go:149: server-94715de0-1898-0279-571b-589005d0b04f: 2022-08-12T07:30:17.235Z [INFO] server-94715de0-1898-0279-571b-589005d0b04f: entering follower state: follower="Node at 94715de0-1898-0279-571b-589005d0b04f [Follower]" leader= + testing.go:149: server-602dc88e-2518-f89e-330b-ddf5451e2b7f: 2022-08-12T07:30:17.235Z [INFO] server-602dc88e-2518-f89e-330b-ddf5451e2b7f: entering follower state: follower="Node at 602dc88e-2518-f89e-330b-ddf5451e2b7f [Follower]" leader= +testLoggerAdapter verbose: server-b54abe8b-730f-1012-e4bc-0d54e9161592: 2022-08-12T07:30:17.313Z [WARN] server-b54abe8b-730f-1012-e4bc-0d54e9161592: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-b54abe8b-730f-1012-e4bc-0d54e9161592: 2022-08-12T07:30:17.313Z [WARN] server-b54abe8b-730f-1012-e4bc-0d54e9161592: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-b54abe8b-730f-1012-e4bc-0d54e9161592: 2022-08-12T07:30:17.313Z [INFO] server-b54abe8b-730f-1012-e4bc-0d54e9161592: entering candidate state: node="Node at b54abe8b-730f-1012-e4bc-0d54e9161592 [Candidate]" term=2 + testing.go:149: server-b54abe8b-730f-1012-e4bc-0d54e9161592: 2022-08-12T07:30:17.313Z [INFO] server-b54abe8b-730f-1012-e4bc-0d54e9161592: entering candidate state: node="Node at b54abe8b-730f-1012-e4bc-0d54e9161592 [Candidate]" term=2 +testLoggerAdapter verbose: server-b54abe8b-730f-1012-e4bc-0d54e9161592: 2022-08-12T07:30:17.314Z [INFO] server-b54abe8b-730f-1012-e4bc-0d54e9161592: election won: tally=2 + testing.go:149: server-b54abe8b-730f-1012-e4bc-0d54e9161592: 2022-08-12T07:30:17.314Z [INFO] server-b54abe8b-730f-1012-e4bc-0d54e9161592: election won: tally=2 +testLoggerAdapter verbose: server-b54abe8b-730f-1012-e4bc-0d54e9161592: 2022-08-12T07:30:17.314Z [INFO] server-b54abe8b-730f-1012-e4bc-0d54e9161592: entering leader state: leader="Node at b54abe8b-730f-1012-e4bc-0d54e9161592 [Leader]" + testing.go:149: server-b54abe8b-730f-1012-e4bc-0d54e9161592: 2022-08-12T07:30:17.314Z [INFO] server-b54abe8b-730f-1012-e4bc-0d54e9161592: entering leader state: leader="Node at b54abe8b-730f-1012-e4bc-0d54e9161592 [Leader]" +testLoggerAdapter verbose: server-b54abe8b-730f-1012-e4bc-0d54e9161592: 2022-08-12T07:30:17.314Z [INFO] server-b54abe8b-730f-1012-e4bc-0d54e9161592: added peer, starting replication: peer=server-94715de0-1898-0279-571b-589005d0b04f + testing.go:149: server-b54abe8b-730f-1012-e4bc-0d54e9161592: 2022-08-12T07:30:17.314Z [INFO] server-b54abe8b-730f-1012-e4bc-0d54e9161592: added peer, starting replication: peer=server-94715de0-1898-0279-571b-589005d0b04f +testLoggerAdapter verbose: server-b54abe8b-730f-1012-e4bc-0d54e9161592: 2022-08-12T07:30:17.314Z [INFO] server-b54abe8b-730f-1012-e4bc-0d54e9161592: added peer, starting replication: peer=server-602dc88e-2518-f89e-330b-ddf5451e2b7f + testing.go:149: server-b54abe8b-730f-1012-e4bc-0d54e9161592: 2022-08-12T07:30:17.314Z [INFO] server-b54abe8b-730f-1012-e4bc-0d54e9161592: added peer, starting replication: peer=server-602dc88e-2518-f89e-330b-ddf5451e2b7f +testLoggerAdapter verbose: server-b54abe8b-730f-1012-e4bc-0d54e9161592: 2022-08-12T07:30:17.314Z [INFO] server-b54abe8b-730f-1012-e4bc-0d54e9161592: pipelining replication: peer="{Voter server-602dc88e-2518-f89e-330b-ddf5451e2b7f 602dc88e-2518-f89e-330b-ddf5451e2b7f}" + testing.go:149: server-b54abe8b-730f-1012-e4bc-0d54e9161592: 2022-08-12T07:30:17.314Z [INFO] server-b54abe8b-730f-1012-e4bc-0d54e9161592: pipelining replication: peer="{Voter server-602dc88e-2518-f89e-330b-ddf5451e2b7f 602dc88e-2518-f89e-330b-ddf5451e2b7f}" +testLoggerAdapter verbose: server-b54abe8b-730f-1012-e4bc-0d54e9161592: 2022-08-12T07:30:17.314Z [INFO] server-b54abe8b-730f-1012-e4bc-0d54e9161592: pipelining replication: peer="{Voter server-94715de0-1898-0279-571b-589005d0b04f 94715de0-1898-0279-571b-589005d0b04f}" + testing.go:149: server-b54abe8b-730f-1012-e4bc-0d54e9161592: 2022-08-12T07:30:17.314Z [INFO] server-b54abe8b-730f-1012-e4bc-0d54e9161592: pipelining replication: peer="{Voter server-94715de0-1898-0279-571b-589005d0b04f 94715de0-1898-0279-571b-589005d0b04f}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:17.419Z [INFO] stable state for Leader reached at 2022-08-12 07:30:17.31428692 +0000 UTC m=+18.270973401 (1 nodes), 78.57116ms from start of poll, 79.50658ms from cluster start. Timeout at 2022-08-12 07:30:17.41936276 +0000 UTC m=+18.376049241, 105.07584ms after stability + testing.go:149: cluster: 2022-08-12T07:30:17.419Z [INFO] stable state for Leader reached at 2022-08-12 07:30:17.31428692 +0000 UTC m=+18.270973401 (1 nodes), 78.57116ms from start of poll, 79.50658ms from cluster start. Timeout at 2022-08-12 07:30:17.41936276 +0000 UTC m=+18.376049241, 105.07584ms after stability +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:17.419Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2022-08-12T07:30:17.419Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:17.524Z [INFO] stable state for Follower reached at 2022-08-12 07:30:17.41960536 +0000 UTC m=+18.376291841 (2 nodes), 2.58µs from start of poll, 184.82502ms from cluster start. Timeout at 2022-08-12 07:30:17.5246842 +0000 UTC m=+18.481370681, 105.07884ms after stability + testing.go:149: cluster: 2022-08-12T07:30:17.524Z [INFO] stable state for Follower reached at 2022-08-12 07:30:17.41960536 +0000 UTC m=+18.376291841 (2 nodes), 2.58µs from start of poll, 184.82502ms from cluster start. Timeout at 2022-08-12 07:30:17.5246842 +0000 UTC m=+18.481370681, 105.07884ms after stability +testLoggerAdapter verbose: server-94715de0-1898-0279-571b-589005d0b04f: 2022-08-12T07:30:17.552Z [INFO] server-94715de0-1898-0279-571b-589005d0b04f: entering candidate state: node="Node at 94715de0-1898-0279-571b-589005d0b04f [Candidate]" term=3 + testing.go:149: server-94715de0-1898-0279-571b-589005d0b04f: 2022-08-12T07:30:17.552Z [INFO] server-94715de0-1898-0279-571b-589005d0b04f: entering candidate state: node="Node at 94715de0-1898-0279-571b-589005d0b04f [Candidate]" term=3 +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:17.552Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-94715de0-1898-0279-571b-589005d0b04f: 2022-08-12T07:30:17.552Z [INFO] server-94715de0-1898-0279-571b-589005d0b04f: election won: tally=2 + testing.go:149: cluster: 2022-08-12T07:30:17.552Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-b54abe8b-730f-1012-e4bc-0d54e9161592: 2022-08-12T07:30:17.552Z [INFO] server-b54abe8b-730f-1012-e4bc-0d54e9161592: entering follower state: follower="Node at b54abe8b-730f-1012-e4bc-0d54e9161592 [Follower]" leader= + testing.go:149: server-94715de0-1898-0279-571b-589005d0b04f: 2022-08-12T07:30:17.552Z [INFO] server-94715de0-1898-0279-571b-589005d0b04f: election won: tally=2 +testLoggerAdapter verbose: server-94715de0-1898-0279-571b-589005d0b04f: 2022-08-12T07:30:17.552Z [INFO] server-94715de0-1898-0279-571b-589005d0b04f: entering leader state: leader="Node at 94715de0-1898-0279-571b-589005d0b04f [Leader]" + testing.go:149: server-b54abe8b-730f-1012-e4bc-0d54e9161592: 2022-08-12T07:30:17.552Z [INFO] server-b54abe8b-730f-1012-e4bc-0d54e9161592: entering follower state: follower="Node at b54abe8b-730f-1012-e4bc-0d54e9161592 [Follower]" leader= +testLoggerAdapter verbose: server-b54abe8b-730f-1012-e4bc-0d54e9161592: 2022-08-12T07:30:17.552Z [INFO] server-b54abe8b-730f-1012-e4bc-0d54e9161592: aborting pipeline replication: peer="{Voter server-602dc88e-2518-f89e-330b-ddf5451e2b7f 602dc88e-2518-f89e-330b-ddf5451e2b7f}" + testing.go:149: server-b54abe8b-730f-1012-e4bc-0d54e9161592: 2022-08-12T07:30:17.552Z [INFO] server-b54abe8b-730f-1012-e4bc-0d54e9161592: aborting pipeline replication: peer="{Voter server-602dc88e-2518-f89e-330b-ddf5451e2b7f 602dc88e-2518-f89e-330b-ddf5451e2b7f}" +testLoggerAdapter verbose: server-b54abe8b-730f-1012-e4bc-0d54e9161592: 2022-08-12T07:30:17.552Z [INFO] server-b54abe8b-730f-1012-e4bc-0d54e9161592: aborting pipeline replication: peer="{Voter server-94715de0-1898-0279-571b-589005d0b04f 94715de0-1898-0279-571b-589005d0b04f}" + testing.go:149: server-94715de0-1898-0279-571b-589005d0b04f: 2022-08-12T07:30:17.552Z [INFO] server-94715de0-1898-0279-571b-589005d0b04f: entering leader state: leader="Node at 94715de0-1898-0279-571b-589005d0b04f [Leader]" +testLoggerAdapter verbose: server-94715de0-1898-0279-571b-589005d0b04f: 2022-08-12T07:30:17.552Z [INFO] server-94715de0-1898-0279-571b-589005d0b04f: added peer, starting replication: peer=server-b54abe8b-730f-1012-e4bc-0d54e9161592 + testing.go:149: server-94715de0-1898-0279-571b-589005d0b04f: 2022-08-12T07:30:17.552Z [INFO] server-94715de0-1898-0279-571b-589005d0b04f: added peer, starting replication: peer=server-b54abe8b-730f-1012-e4bc-0d54e9161592 +testLoggerAdapter verbose: server-94715de0-1898-0279-571b-589005d0b04f: 2022-08-12T07:30:17.552Z [INFO] server-94715de0-1898-0279-571b-589005d0b04f: added peer, starting replication: peer=server-602dc88e-2518-f89e-330b-ddf5451e2b7f + testing.go:149: server-94715de0-1898-0279-571b-589005d0b04f: 2022-08-12T07:30:17.552Z [INFO] server-94715de0-1898-0279-571b-589005d0b04f: added peer, starting replication: peer=server-602dc88e-2518-f89e-330b-ddf5451e2b7f +testLoggerAdapter verbose: server-602dc88e-2518-f89e-330b-ddf5451e2b7f: 2022-08-12T07:30:17.553Z [WARN] server-602dc88e-2518-f89e-330b-ddf5451e2b7f: failed to get previous log: previous-index=1002 last-index=798 error="log not found" + testing.go:149: server-602dc88e-2518-f89e-330b-ddf5451e2b7f: 2022-08-12T07:30:17.553Z [WARN] server-602dc88e-2518-f89e-330b-ddf5451e2b7f: failed to get previous log: previous-index=1002 last-index=798 error="log not found" +testLoggerAdapter verbose: server-94715de0-1898-0279-571b-589005d0b04f: 2022-08-12T07:30:17.553Z [WARN] server-94715de0-1898-0279-571b-589005d0b04f: appendEntries rejected, sending older logs: peer="{Voter server-602dc88e-2518-f89e-330b-ddf5451e2b7f 602dc88e-2518-f89e-330b-ddf5451e2b7f}" next=799 + testing.go:149: server-94715de0-1898-0279-571b-589005d0b04f: 2022-08-12T07:30:17.553Z [WARN] server-94715de0-1898-0279-571b-589005d0b04f: appendEntries rejected, sending older logs: peer="{Voter server-602dc88e-2518-f89e-330b-ddf5451e2b7f 602dc88e-2518-f89e-330b-ddf5451e2b7f}" next=799 +testLoggerAdapter verbose: server-94715de0-1898-0279-571b-589005d0b04f: 2022-08-12T07:30:17.553Z [INFO] server-94715de0-1898-0279-571b-589005d0b04f: pipelining replication: peer="{Voter server-602dc88e-2518-f89e-330b-ddf5451e2b7f 602dc88e-2518-f89e-330b-ddf5451e2b7f}" + testing.go:149: server-94715de0-1898-0279-571b-589005d0b04f: 2022-08-12T07:30:17.553Z [INFO] server-94715de0-1898-0279-571b-589005d0b04f: pipelining replication: peer="{Voter server-602dc88e-2518-f89e-330b-ddf5451e2b7f 602dc88e-2518-f89e-330b-ddf5451e2b7f}" + testing.go:149: server-b54abe8b-730f-1012-e4bc-0d54e9161592: 2022-08-12T07:30:17.552Z [INFO] server-b54abe8b-730f-1012-e4bc-0d54e9161592: aborting pipeline replication: peer="{Voter server-94715de0-1898-0279-571b-589005d0b04f 94715de0-1898-0279-571b-589005d0b04f}" +testLoggerAdapter verbose: server-94715de0-1898-0279-571b-589005d0b04f: 2022-08-12T07:30:17.553Z [INFO] server-94715de0-1898-0279-571b-589005d0b04f: pipelining replication: peer="{Voter server-b54abe8b-730f-1012-e4bc-0d54e9161592 b54abe8b-730f-1012-e4bc-0d54e9161592}" + testing.go:149: server-94715de0-1898-0279-571b-589005d0b04f: 2022-08-12T07:30:17.553Z [INFO] server-94715de0-1898-0279-571b-589005d0b04f: pipelining replication: peer="{Voter server-b54abe8b-730f-1012-e4bc-0d54e9161592 b54abe8b-730f-1012-e4bc-0d54e9161592}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:17.657Z [INFO] stable state for Leader reached at 2022-08-12 07:30:17.55274514 +0000 UTC m=+18.509431621 (1 nodes), 107.32µs from start of poll, 317.9648ms from cluster start. Timeout at 2022-08-12 07:30:17.6577597 +0000 UTC m=+18.614446201, 105.01458ms after stability + testing.go:149: cluster: 2022-08-12T07:30:17.657Z [INFO] stable state for Leader reached at 2022-08-12 07:30:17.55274514 +0000 UTC m=+18.509431621 (1 nodes), 107.32µs from start of poll, 317.9648ms from cluster start. Timeout at 2022-08-12 07:30:17.6577597 +0000 UTC m=+18.614446201, 105.01458ms after stability +testLoggerAdapter verbose: server-94715de0-1898-0279-571b-589005d0b04f: 2022-08-12T07:30:17.658Z [INFO] server-94715de0-1898-0279-571b-589005d0b04f: aborting pipeline replication: peer="{Voter server-602dc88e-2518-f89e-330b-ddf5451e2b7f 602dc88e-2518-f89e-330b-ddf5451e2b7f}" + testing.go:149: server-94715de0-1898-0279-571b-589005d0b04f: 2022-08-12T07:30:17.658Z [INFO] server-94715de0-1898-0279-571b-589005d0b04f: aborting pipeline replication: peer="{Voter server-602dc88e-2518-f89e-330b-ddf5451e2b7f 602dc88e-2518-f89e-330b-ddf5451e2b7f}" +testLoggerAdapter verbose: server-94715de0-1898-0279-571b-589005d0b04f: 2022-08-12T07:30:17.658Z [INFO] server-94715de0-1898-0279-571b-589005d0b04f: aborting pipeline replication: peer="{Voter server-b54abe8b-730f-1012-e4bc-0d54e9161592 b54abe8b-730f-1012-e4bc-0d54e9161592}" + testing.go:149: server-94715de0-1898-0279-571b-589005d0b04f: 2022-08-12T07:30:17.658Z [INFO] server-94715de0-1898-0279-571b-589005d0b04f: aborting pipeline replication: peer="{Voter server-b54abe8b-730f-1012-e4bc-0d54e9161592 b54abe8b-730f-1012-e4bc-0d54e9161592}" +--- PASS: TestRaft_LeadershipTransferToBehindServer (0.42s) === RUN TestRaft_LeadershipTransferToItself -testLoggerAdapter verbose: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.103Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2d5700b-f740-36ed-03c4-d6cd984af401 Address:c2d5700b-f740-36ed-03c4-d6cd984af401} {Suffrage:Voter ID:server-489b32cf-416b-083b-fbd8-739b930ddf7f Address:489b32cf-416b-083b-fbd8-739b930ddf7f} {Suffrage:Voter ID:server-c3c7356b-3f6f-c121-b973-8e4356c8576a Address:c3c7356b-3f6f-c121-b973-8e4356c8576a}]" - testing.go:149: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.103Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2d5700b-f740-36ed-03c4-d6cd984af401 Address:c2d5700b-f740-36ed-03c4-d6cd984af401} {Suffrage:Voter ID:server-489b32cf-416b-083b-fbd8-739b930ddf7f Address:489b32cf-416b-083b-fbd8-739b930ddf7f} {Suffrage:Voter ID:server-c3c7356b-3f6f-c121-b973-8e4356c8576a Address:c3c7356b-3f6f-c121-b973-8e4356c8576a}]" -testLoggerAdapter verbose: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.103Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: entering follower state: follower="Node at c2d5700b-f740-36ed-03c4-d6cd984af401 [Follower]" leader= - testing.go:149: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.103Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: entering follower state: follower="Node at c2d5700b-f740-36ed-03c4-d6cd984af401 [Follower]" leader= -testLoggerAdapter verbose: server-489b32cf-416b-083b-fbd8-739b930ddf7f: 2021-07-10T01:05:13.103Z [INFO] server-489b32cf-416b-083b-fbd8-739b930ddf7f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2d5700b-f740-36ed-03c4-d6cd984af401 Address:c2d5700b-f740-36ed-03c4-d6cd984af401} {Suffrage:Voter ID:server-489b32cf-416b-083b-fbd8-739b930ddf7f Address:489b32cf-416b-083b-fbd8-739b930ddf7f} {Suffrage:Voter ID:server-c3c7356b-3f6f-c121-b973-8e4356c8576a Address:c3c7356b-3f6f-c121-b973-8e4356c8576a}]" - testing.go:149: server-489b32cf-416b-083b-fbd8-739b930ddf7f: 2021-07-10T01:05:13.103Z [INFO] server-489b32cf-416b-083b-fbd8-739b930ddf7f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2d5700b-f740-36ed-03c4-d6cd984af401 Address:c2d5700b-f740-36ed-03c4-d6cd984af401} {Suffrage:Voter ID:server-489b32cf-416b-083b-fbd8-739b930ddf7f Address:489b32cf-416b-083b-fbd8-739b930ddf7f} {Suffrage:Voter ID:server-c3c7356b-3f6f-c121-b973-8e4356c8576a Address:c3c7356b-3f6f-c121-b973-8e4356c8576a}]" -testLoggerAdapter verbose: server-489b32cf-416b-083b-fbd8-739b930ddf7f: 2021-07-10T01:05:13.103Z [INFO] server-489b32cf-416b-083b-fbd8-739b930ddf7f: entering follower state: follower="Node at 489b32cf-416b-083b-fbd8-739b930ddf7f [Follower]" leader= - testing.go:149: server-489b32cf-416b-083b-fbd8-739b930ddf7f: 2021-07-10T01:05:13.103Z [INFO] server-489b32cf-416b-083b-fbd8-739b930ddf7f: entering follower state: follower="Node at 489b32cf-416b-083b-fbd8-739b930ddf7f [Follower]" leader= -testLoggerAdapter verbose: server-c3c7356b-3f6f-c121-b973-8e4356c8576a: 2021-07-10T01:05:13.104Z [INFO] server-c3c7356b-3f6f-c121-b973-8e4356c8576a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2d5700b-f740-36ed-03c4-d6cd984af401 Address:c2d5700b-f740-36ed-03c4-d6cd984af401} {Suffrage:Voter ID:server-489b32cf-416b-083b-fbd8-739b930ddf7f Address:489b32cf-416b-083b-fbd8-739b930ddf7f} {Suffrage:Voter ID:server-c3c7356b-3f6f-c121-b973-8e4356c8576a Address:c3c7356b-3f6f-c121-b973-8e4356c8576a}]" - testing.go:149: server-c3c7356b-3f6f-c121-b973-8e4356c8576a: 2021-07-10T01:05:13.104Z [INFO] server-c3c7356b-3f6f-c121-b973-8e4356c8576a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2d5700b-f740-36ed-03c4-d6cd984af401 Address:c2d5700b-f740-36ed-03c4-d6cd984af401} {Suffrage:Voter ID:server-489b32cf-416b-083b-fbd8-739b930ddf7f Address:489b32cf-416b-083b-fbd8-739b930ddf7f} {Suffrage:Voter ID:server-c3c7356b-3f6f-c121-b973-8e4356c8576a Address:c3c7356b-3f6f-c121-b973-8e4356c8576a}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:13.104Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:13.104Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-c3c7356b-3f6f-c121-b973-8e4356c8576a: 2021-07-10T01:05:13.104Z [INFO] server-c3c7356b-3f6f-c121-b973-8e4356c8576a: entering follower state: follower="Node at c3c7356b-3f6f-c121-b973-8e4356c8576a [Follower]" leader= - testing.go:149: server-c3c7356b-3f6f-c121-b973-8e4356c8576a: 2021-07-10T01:05:13.104Z [INFO] server-c3c7356b-3f6f-c121-b973-8e4356c8576a: entering follower state: follower="Node at c3c7356b-3f6f-c121-b973-8e4356c8576a [Follower]" leader= -testLoggerAdapter verbose: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.162Z [WARN] server-c2d5700b-f740-36ed-03c4-d6cd984af401: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.162Z [WARN] server-c2d5700b-f740-36ed-03c4-d6cd984af401: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.163Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: entering candidate state: node="Node at c2d5700b-f740-36ed-03c4-d6cd984af401 [Candidate]" term=2 - testing.go:149: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.163Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: entering candidate state: node="Node at c2d5700b-f740-36ed-03c4-d6cd984af401 [Candidate]" term=2 -testLoggerAdapter verbose: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.163Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: election won: tally=2 - testing.go:149: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.163Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: election won: tally=2 -testLoggerAdapter verbose: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.163Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: entering leader state: leader="Node at c2d5700b-f740-36ed-03c4-d6cd984af401 [Leader]" - testing.go:149: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.163Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: entering leader state: leader="Node at c2d5700b-f740-36ed-03c4-d6cd984af401 [Leader]" -testLoggerAdapter verbose: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.163Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: added peer, starting replication: peer=server-489b32cf-416b-083b-fbd8-739b930ddf7f - testing.go:149: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.163Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: added peer, starting replication: peer=server-489b32cf-416b-083b-fbd8-739b930ddf7f -testLoggerAdapter verbose: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.163Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: added peer, starting replication: peer=server-c3c7356b-3f6f-c121-b973-8e4356c8576a - testing.go:149: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.163Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: added peer, starting replication: peer=server-c3c7356b-3f6f-c121-b973-8e4356c8576a -testLoggerAdapter verbose: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.163Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: pipelining replication: peer="{Voter server-489b32cf-416b-083b-fbd8-739b930ddf7f 489b32cf-416b-083b-fbd8-739b930ddf7f}" - testing.go:149: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.163Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: pipelining replication: peer="{Voter server-489b32cf-416b-083b-fbd8-739b930ddf7f 489b32cf-416b-083b-fbd8-739b930ddf7f}" -testLoggerAdapter verbose: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.163Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: pipelining replication: peer="{Voter server-c3c7356b-3f6f-c121-b973-8e4356c8576a c3c7356b-3f6f-c121-b973-8e4356c8576a}" - testing.go:149: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.163Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: pipelining replication: peer="{Voter server-c3c7356b-3f6f-c121-b973-8e4356c8576a c3c7356b-3f6f-c121-b973-8e4356c8576a}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:13.268Z [INFO] stable state for Leader reached at 2021-07-10 01:05:13.163538589 +0000 UTC m=+18.551696579 (1 nodes), 59.264925ms from start of poll, 60.266435ms from cluster start. Timeout at 2021-07-10 01:05:13.268617797 +0000 UTC m=+18.656775767, 105.079188ms after stability - testing.go:149: cluster: 2021-07-10T01:05:13.268Z [INFO] stable state for Leader reached at 2021-07-10 01:05:13.163538589 +0000 UTC m=+18.551696579 (1 nodes), 59.264925ms from start of poll, 60.266435ms from cluster start. Timeout at 2021-07-10 01:05:13.268617797 +0000 UTC m=+18.656775767, 105.079188ms after stability -testLoggerAdapter verbose: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.268Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: cannot transfer leadership to itself - testing.go:149: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.268Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: cannot transfer leadership to itself -testLoggerAdapter verbose: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.268Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: aborting pipeline replication: peer="{Voter server-c3c7356b-3f6f-c121-b973-8e4356c8576a c3c7356b-3f6f-c121-b973-8e4356c8576a}" - testing.go:149: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.268Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: aborting pipeline replication: peer="{Voter server-c3c7356b-3f6f-c121-b973-8e4356c8576a c3c7356b-3f6f-c121-b973-8e4356c8576a}" -testLoggerAdapter verbose: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.268Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: aborting pipeline replication: peer="{Voter server-489b32cf-416b-083b-fbd8-739b930ddf7f 489b32cf-416b-083b-fbd8-739b930ddf7f}" - testing.go:149: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.268Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: aborting pipeline replication: peer="{Voter server-489b32cf-416b-083b-fbd8-739b930ddf7f 489b32cf-416b-083b-fbd8-739b930ddf7f}" +testLoggerAdapter verbose: server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: 2022-08-12T07:30:17.659Z [INFO] server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0 Address:de821e9d-e16f-b07f-8bd2-b0cc3a1139c0} {Suffrage:Voter ID:server-147d159c-b1e0-60e1-990d-8fcac8db9b12 Address:147d159c-b1e0-60e1-990d-8fcac8db9b12} {Suffrage:Voter ID:server-3abee526-d504-8c41-0355-6ccafb461aeb Address:3abee526-d504-8c41-0355-6ccafb461aeb}]" + testing.go:149: server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: 2022-08-12T07:30:17.659Z [INFO] server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0 Address:de821e9d-e16f-b07f-8bd2-b0cc3a1139c0} {Suffrage:Voter ID:server-147d159c-b1e0-60e1-990d-8fcac8db9b12 Address:147d159c-b1e0-60e1-990d-8fcac8db9b12} {Suffrage:Voter ID:server-3abee526-d504-8c41-0355-6ccafb461aeb Address:3abee526-d504-8c41-0355-6ccafb461aeb}]" +testLoggerAdapter verbose: server-147d159c-b1e0-60e1-990d-8fcac8db9b12: 2022-08-12T07:30:17.659Z [INFO] server-147d159c-b1e0-60e1-990d-8fcac8db9b12: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0 Address:de821e9d-e16f-b07f-8bd2-b0cc3a1139c0} {Suffrage:Voter ID:server-147d159c-b1e0-60e1-990d-8fcac8db9b12 Address:147d159c-b1e0-60e1-990d-8fcac8db9b12} {Suffrage:Voter ID:server-3abee526-d504-8c41-0355-6ccafb461aeb Address:3abee526-d504-8c41-0355-6ccafb461aeb}]" + testing.go:149: server-147d159c-b1e0-60e1-990d-8fcac8db9b12: 2022-08-12T07:30:17.659Z [INFO] server-147d159c-b1e0-60e1-990d-8fcac8db9b12: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0 Address:de821e9d-e16f-b07f-8bd2-b0cc3a1139c0} {Suffrage:Voter ID:server-147d159c-b1e0-60e1-990d-8fcac8db9b12 Address:147d159c-b1e0-60e1-990d-8fcac8db9b12} {Suffrage:Voter ID:server-3abee526-d504-8c41-0355-6ccafb461aeb Address:3abee526-d504-8c41-0355-6ccafb461aeb}]" +testLoggerAdapter verbose: server-3abee526-d504-8c41-0355-6ccafb461aeb: 2022-08-12T07:30:17.660Z [INFO] server-3abee526-d504-8c41-0355-6ccafb461aeb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0 Address:de821e9d-e16f-b07f-8bd2-b0cc3a1139c0} {Suffrage:Voter ID:server-147d159c-b1e0-60e1-990d-8fcac8db9b12 Address:147d159c-b1e0-60e1-990d-8fcac8db9b12} {Suffrage:Voter ID:server-3abee526-d504-8c41-0355-6ccafb461aeb Address:3abee526-d504-8c41-0355-6ccafb461aeb}]" + testing.go:149: server-3abee526-d504-8c41-0355-6ccafb461aeb: 2022-08-12T07:30:17.660Z [INFO] server-3abee526-d504-8c41-0355-6ccafb461aeb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0 Address:de821e9d-e16f-b07f-8bd2-b0cc3a1139c0} {Suffrage:Voter ID:server-147d159c-b1e0-60e1-990d-8fcac8db9b12 Address:147d159c-b1e0-60e1-990d-8fcac8db9b12} {Suffrage:Voter ID:server-3abee526-d504-8c41-0355-6ccafb461aeb Address:3abee526-d504-8c41-0355-6ccafb461aeb}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:17.660Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-147d159c-b1e0-60e1-990d-8fcac8db9b12: 2022-08-12T07:30:17.660Z [INFO] server-147d159c-b1e0-60e1-990d-8fcac8db9b12: entering follower state: follower="Node at 147d159c-b1e0-60e1-990d-8fcac8db9b12 [Follower]" leader= + testing.go:149: cluster: 2022-08-12T07:30:17.660Z [INFO] starting stability test: raft-state=Leader + testing.go:149: server-147d159c-b1e0-60e1-990d-8fcac8db9b12: 2022-08-12T07:30:17.660Z [INFO] server-147d159c-b1e0-60e1-990d-8fcac8db9b12: entering follower state: follower="Node at 147d159c-b1e0-60e1-990d-8fcac8db9b12 [Follower]" leader= +testLoggerAdapter verbose: server-3abee526-d504-8c41-0355-6ccafb461aeb: 2022-08-12T07:30:17.660Z [INFO] server-3abee526-d504-8c41-0355-6ccafb461aeb: entering follower state: follower="Node at 3abee526-d504-8c41-0355-6ccafb461aeb [Follower]" leader= + testing.go:149: server-3abee526-d504-8c41-0355-6ccafb461aeb: 2022-08-12T07:30:17.660Z [INFO] server-3abee526-d504-8c41-0355-6ccafb461aeb: entering follower state: follower="Node at 3abee526-d504-8c41-0355-6ccafb461aeb [Follower]" leader= +testLoggerAdapter verbose: server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: 2022-08-12T07:30:17.661Z [INFO] server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: entering follower state: follower="Node at de821e9d-e16f-b07f-8bd2-b0cc3a1139c0 [Follower]" leader= + testing.go:149: server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: 2022-08-12T07:30:17.661Z [INFO] server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: entering follower state: follower="Node at de821e9d-e16f-b07f-8bd2-b0cc3a1139c0 [Follower]" leader= +testLoggerAdapter verbose: server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: 2022-08-12T07:30:17.719Z [WARN] server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: 2022-08-12T07:30:17.719Z [WARN] server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: 2022-08-12T07:30:17.720Z [INFO] server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: entering candidate state: node="Node at de821e9d-e16f-b07f-8bd2-b0cc3a1139c0 [Candidate]" term=2 + testing.go:149: server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: 2022-08-12T07:30:17.720Z [INFO] server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: entering candidate state: node="Node at de821e9d-e16f-b07f-8bd2-b0cc3a1139c0 [Candidate]" term=2 +testLoggerAdapter verbose: server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: 2022-08-12T07:30:17.720Z [INFO] server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: election won: tally=2 + testing.go:149: server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: 2022-08-12T07:30:17.720Z [INFO] server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: election won: tally=2 +testLoggerAdapter verbose: server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: 2022-08-12T07:30:17.720Z [INFO] server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: entering leader state: leader="Node at de821e9d-e16f-b07f-8bd2-b0cc3a1139c0 [Leader]" + testing.go:149: server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: 2022-08-12T07:30:17.720Z [INFO] server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: entering leader state: leader="Node at de821e9d-e16f-b07f-8bd2-b0cc3a1139c0 [Leader]" +testLoggerAdapter verbose: server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: 2022-08-12T07:30:17.720Z [INFO] server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: added peer, starting replication: peer=server-147d159c-b1e0-60e1-990d-8fcac8db9b12 + testing.go:149: server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: 2022-08-12T07:30:17.720Z [INFO] server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: added peer, starting replication: peer=server-147d159c-b1e0-60e1-990d-8fcac8db9b12 +testLoggerAdapter verbose: server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: 2022-08-12T07:30:17.720Z [INFO] server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: added peer, starting replication: peer=server-3abee526-d504-8c41-0355-6ccafb461aeb + testing.go:149: server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: 2022-08-12T07:30:17.720Z [INFO] server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: added peer, starting replication: peer=server-3abee526-d504-8c41-0355-6ccafb461aeb +testLoggerAdapter verbose: server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: 2022-08-12T07:30:17.720Z [INFO] server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: pipelining replication: peer="{Voter server-147d159c-b1e0-60e1-990d-8fcac8db9b12 147d159c-b1e0-60e1-990d-8fcac8db9b12}" + testing.go:149: server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: 2022-08-12T07:30:17.720Z [INFO] server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: pipelining replication: peer="{Voter server-147d159c-b1e0-60e1-990d-8fcac8db9b12 147d159c-b1e0-60e1-990d-8fcac8db9b12}" +testLoggerAdapter verbose: server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: 2022-08-12T07:30:17.720Z [INFO] server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: pipelining replication: peer="{Voter server-3abee526-d504-8c41-0355-6ccafb461aeb 3abee526-d504-8c41-0355-6ccafb461aeb}" + testing.go:149: server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: 2022-08-12T07:30:17.720Z [INFO] server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: pipelining replication: peer="{Voter server-3abee526-d504-8c41-0355-6ccafb461aeb 3abee526-d504-8c41-0355-6ccafb461aeb}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:17.825Z [INFO] stable state for Leader reached at 2022-08-12 07:30:17.72050418 +0000 UTC m=+18.677190681 (1 nodes), 59.53086ms from start of poll, 61.22592ms from cluster start. Timeout at 2022-08-12 07:30:17.82559988 +0000 UTC m=+18.782286341, 105.09566ms after stability + testing.go:149: cluster: 2022-08-12T07:30:17.825Z [INFO] stable state for Leader reached at 2022-08-12 07:30:17.72050418 +0000 UTC m=+18.677190681 (1 nodes), 59.53086ms from start of poll, 61.22592ms from cluster start. Timeout at 2022-08-12 07:30:17.82559988 +0000 UTC m=+18.782286341, 105.09566ms after stability +testLoggerAdapter verbose: server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: 2022-08-12T07:30:17.825Z [INFO] server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: cannot transfer leadership to itself + testing.go:149: server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: 2022-08-12T07:30:17.825Z [INFO] server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: cannot transfer leadership to itself +testLoggerAdapter verbose: server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: 2022-08-12T07:30:17.825Z [INFO] server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: aborting pipeline replication: peer="{Voter server-3abee526-d504-8c41-0355-6ccafb461aeb 3abee526-d504-8c41-0355-6ccafb461aeb}" + testing.go:149: server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: 2022-08-12T07:30:17.825Z [INFO] server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: aborting pipeline replication: peer="{Voter server-3abee526-d504-8c41-0355-6ccafb461aeb 3abee526-d504-8c41-0355-6ccafb461aeb}" +testLoggerAdapter verbose: server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: 2022-08-12T07:30:17.826Z [INFO] server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: aborting pipeline replication: peer="{Voter server-147d159c-b1e0-60e1-990d-8fcac8db9b12 147d159c-b1e0-60e1-990d-8fcac8db9b12}" + testing.go:149: server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: 2022-08-12T07:30:17.826Z [INFO] server-de821e9d-e16f-b07f-8bd2-b0cc3a1139c0: aborting pipeline replication: peer="{Voter server-147d159c-b1e0-60e1-990d-8fcac8db9b12 147d159c-b1e0-60e1-990d-8fcac8db9b12}" --- PASS: TestRaft_LeadershipTransferToItself (0.17s) === RUN TestRaft_LeadershipTransferLeaderRejectsClientRequests raft_test.go:2064: Disabled unreliable test --- SKIP: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.00s) === RUN TestRaft_LeadershipTransferLeaderReplicationTimeout -testLoggerAdapter verbose: server-e9bae396-860a-aac1-bfb4-a462c2172ee0: 2021-07-10T01:05:13.270Z [INFO] server-e9bae396-860a-aac1-bfb4-a462c2172ee0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e9bae396-860a-aac1-bfb4-a462c2172ee0 Address:e9bae396-860a-aac1-bfb4-a462c2172ee0} {Suffrage:Voter ID:server-6d4f2788-2b13-a8b9-e435-a0774f871ab8 Address:6d4f2788-2b13-a8b9-e435-a0774f871ab8} {Suffrage:Voter ID:server-57b5d042-8833-3178-5c09-e3f830dfd7b4 Address:57b5d042-8833-3178-5c09-e3f830dfd7b4}]" - testing.go:149: server-e9bae396-860a-aac1-bfb4-a462c2172ee0: 2021-07-10T01:05:13.270Z [INFO] server-e9bae396-860a-aac1-bfb4-a462c2172ee0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e9bae396-860a-aac1-bfb4-a462c2172ee0 Address:e9bae396-860a-aac1-bfb4-a462c2172ee0} {Suffrage:Voter ID:server-6d4f2788-2b13-a8b9-e435-a0774f871ab8 Address:6d4f2788-2b13-a8b9-e435-a0774f871ab8} {Suffrage:Voter ID:server-57b5d042-8833-3178-5c09-e3f830dfd7b4 Address:57b5d042-8833-3178-5c09-e3f830dfd7b4}]" -testLoggerAdapter verbose: server-e9bae396-860a-aac1-bfb4-a462c2172ee0: 2021-07-10T01:05:13.271Z [INFO] server-e9bae396-860a-aac1-bfb4-a462c2172ee0: entering follower state: follower="Node at e9bae396-860a-aac1-bfb4-a462c2172ee0 [Follower]" leader= - testing.go:149: server-e9bae396-860a-aac1-bfb4-a462c2172ee0: 2021-07-10T01:05:13.271Z [INFO] server-e9bae396-860a-aac1-bfb4-a462c2172ee0: entering follower state: follower="Node at e9bae396-860a-aac1-bfb4-a462c2172ee0 [Follower]" leader= -testLoggerAdapter verbose: server-6d4f2788-2b13-a8b9-e435-a0774f871ab8: 2021-07-10T01:05:13.271Z [INFO] server-6d4f2788-2b13-a8b9-e435-a0774f871ab8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e9bae396-860a-aac1-bfb4-a462c2172ee0 Address:e9bae396-860a-aac1-bfb4-a462c2172ee0} {Suffrage:Voter ID:server-6d4f2788-2b13-a8b9-e435-a0774f871ab8 Address:6d4f2788-2b13-a8b9-e435-a0774f871ab8} {Suffrage:Voter ID:server-57b5d042-8833-3178-5c09-e3f830dfd7b4 Address:57b5d042-8833-3178-5c09-e3f830dfd7b4}]" - testing.go:149: server-6d4f2788-2b13-a8b9-e435-a0774f871ab8: 2021-07-10T01:05:13.271Z [INFO] server-6d4f2788-2b13-a8b9-e435-a0774f871ab8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e9bae396-860a-aac1-bfb4-a462c2172ee0 Address:e9bae396-860a-aac1-bfb4-a462c2172ee0} {Suffrage:Voter ID:server-6d4f2788-2b13-a8b9-e435-a0774f871ab8 Address:6d4f2788-2b13-a8b9-e435-a0774f871ab8} {Suffrage:Voter ID:server-57b5d042-8833-3178-5c09-e3f830dfd7b4 Address:57b5d042-8833-3178-5c09-e3f830dfd7b4}]" -testLoggerAdapter verbose: server-6d4f2788-2b13-a8b9-e435-a0774f871ab8: 2021-07-10T01:05:13.271Z [INFO] server-6d4f2788-2b13-a8b9-e435-a0774f871ab8: entering follower state: follower="Node at 6d4f2788-2b13-a8b9-e435-a0774f871ab8 [Follower]" leader= - testing.go:149: server-6d4f2788-2b13-a8b9-e435-a0774f871ab8: 2021-07-10T01:05:13.271Z [INFO] server-6d4f2788-2b13-a8b9-e435-a0774f871ab8: entering follower state: follower="Node at 6d4f2788-2b13-a8b9-e435-a0774f871ab8 [Follower]" leader= -testLoggerAdapter verbose: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.271Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e9bae396-860a-aac1-bfb4-a462c2172ee0 Address:e9bae396-860a-aac1-bfb4-a462c2172ee0} {Suffrage:Voter ID:server-6d4f2788-2b13-a8b9-e435-a0774f871ab8 Address:6d4f2788-2b13-a8b9-e435-a0774f871ab8} {Suffrage:Voter ID:server-57b5d042-8833-3178-5c09-e3f830dfd7b4 Address:57b5d042-8833-3178-5c09-e3f830dfd7b4}]" - testing.go:149: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.271Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e9bae396-860a-aac1-bfb4-a462c2172ee0 Address:e9bae396-860a-aac1-bfb4-a462c2172ee0} {Suffrage:Voter ID:server-6d4f2788-2b13-a8b9-e435-a0774f871ab8 Address:6d4f2788-2b13-a8b9-e435-a0774f871ab8} {Suffrage:Voter ID:server-57b5d042-8833-3178-5c09-e3f830dfd7b4 Address:57b5d042-8833-3178-5c09-e3f830dfd7b4}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:13.271Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:13.271Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.271Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: entering follower state: follower="Node at 57b5d042-8833-3178-5c09-e3f830dfd7b4 [Follower]" leader= - testing.go:149: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.271Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: entering follower state: follower="Node at 57b5d042-8833-3178-5c09-e3f830dfd7b4 [Follower]" leader= -testLoggerAdapter verbose: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.350Z [WARN] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.350Z [WARN] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.350Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: entering candidate state: node="Node at 57b5d042-8833-3178-5c09-e3f830dfd7b4 [Candidate]" term=2 - testing.go:149: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.350Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: entering candidate state: node="Node at 57b5d042-8833-3178-5c09-e3f830dfd7b4 [Candidate]" term=2 -testLoggerAdapter verbose: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.350Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: election won: tally=2 - testing.go:149: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.350Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: election won: tally=2 -testLoggerAdapter verbose: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.350Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: entering leader state: leader="Node at 57b5d042-8833-3178-5c09-e3f830dfd7b4 [Leader]" - testing.go:149: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.350Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: entering leader state: leader="Node at 57b5d042-8833-3178-5c09-e3f830dfd7b4 [Leader]" -testLoggerAdapter verbose: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.350Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: added peer, starting replication: peer=server-e9bae396-860a-aac1-bfb4-a462c2172ee0 - testing.go:149: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.350Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: added peer, starting replication: peer=server-e9bae396-860a-aac1-bfb4-a462c2172ee0 -testLoggerAdapter verbose: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.350Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: added peer, starting replication: peer=server-6d4f2788-2b13-a8b9-e435-a0774f871ab8 - testing.go:149: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.350Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: added peer, starting replication: peer=server-6d4f2788-2b13-a8b9-e435-a0774f871ab8 -testLoggerAdapter verbose: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.351Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: pipelining replication: peer="{Voter server-e9bae396-860a-aac1-bfb4-a462c2172ee0 e9bae396-860a-aac1-bfb4-a462c2172ee0}" - testing.go:149: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.351Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: pipelining replication: peer="{Voter server-e9bae396-860a-aac1-bfb4-a462c2172ee0 e9bae396-860a-aac1-bfb4-a462c2172ee0}" -testLoggerAdapter verbose: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.351Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: pipelining replication: peer="{Voter server-6d4f2788-2b13-a8b9-e435-a0774f871ab8 6d4f2788-2b13-a8b9-e435-a0774f871ab8}" - testing.go:149: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.351Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: pipelining replication: peer="{Voter server-6d4f2788-2b13-a8b9-e435-a0774f871ab8 6d4f2788-2b13-a8b9-e435-a0774f871ab8}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:13.455Z [INFO] stable state for Leader reached at 2021-07-10 01:05:13.350814004 +0000 UTC m=+18.738971994 (1 nodes), 78.927913ms from start of poll, 80.119405ms from cluster start. Timeout at 2021-07-10 01:05:13.455822931 +0000 UTC m=+18.843980921, 105.008927ms after stability - testing.go:149: cluster: 2021-07-10T01:05:13.455Z [INFO] stable state for Leader reached at 2021-07-10 01:05:13.350814004 +0000 UTC m=+18.738971994 (1 nodes), 78.927913ms from start of poll, 80.119405ms from cluster start. Timeout at 2021-07-10 01:05:13.455822931 +0000 UTC m=+18.843980921, 105.008927ms after stability -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:13.456Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-07-10T01:05:13.456Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:13.561Z [INFO] stable state for Follower reached at 2021-07-10 01:05:13.456101314 +0000 UTC m=+18.844259304 (2 nodes), 3.321µs from start of poll, 185.406715ms from cluster start. Timeout at 2021-07-10 01:05:13.561180501 +0000 UTC m=+18.949338491, 105.079187ms after stability - testing.go:149: cluster: 2021-07-10T01:05:13.561Z [INFO] stable state for Follower reached at 2021-07-10 01:05:13.456101314 +0000 UTC m=+18.844259304 (2 nodes), 3.321µs from start of poll, 185.406715ms from cluster start. Timeout at 2021-07-10 01:05:13.561180501 +0000 UTC m=+18.949338491, 105.079187ms after stability -testLoggerAdapter verbose: server-e9bae396-860a-aac1-bfb4-a462c2172ee0: 2021-07-10T01:05:13.934Z [INFO] server-e9bae396-860a-aac1-bfb4-a462c2172ee0: entering candidate state: node="Node at e9bae396-860a-aac1-bfb4-a462c2172ee0 [Shutdown]" term=3 -testLoggerAdapter verbose: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.934Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: aborting pipeline replication: peer="{Voter server-6d4f2788-2b13-a8b9-e435-a0774f871ab8 6d4f2788-2b13-a8b9-e435-a0774f871ab8}" - testing.go:149: server-e9bae396-860a-aac1-bfb4-a462c2172ee0: 2021-07-10T01:05:13.934Z [INFO] server-e9bae396-860a-aac1-bfb4-a462c2172ee0: entering candidate state: node="Node at e9bae396-860a-aac1-bfb4-a462c2172ee0 [Shutdown]" term=3 - testing.go:149: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.934Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: aborting pipeline replication: peer="{Voter server-6d4f2788-2b13-a8b9-e435-a0774f871ab8 6d4f2788-2b13-a8b9-e435-a0774f871ab8}" -testLoggerAdapter verbose: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.934Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: aborting pipeline replication: peer="{Voter server-e9bae396-860a-aac1-bfb4-a462c2172ee0 e9bae396-860a-aac1-bfb4-a462c2172ee0}" - testing.go:149: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.934Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: aborting pipeline replication: peer="{Voter server-e9bae396-860a-aac1-bfb4-a462c2172ee0 e9bae396-860a-aac1-bfb4-a462c2172ee0}" -testLoggerAdapter verbose: server-e9bae396-860a-aac1-bfb4-a462c2172ee0: 2021-07-10T01:05:13.986Z [ERROR] server-e9bae396-860a-aac1-bfb4-a462c2172ee0: failed to make requestVote RPC: target="{Voter server-6d4f2788-2b13-a8b9-e435-a0774f871ab8 6d4f2788-2b13-a8b9-e435-a0774f871ab8}" error="command timed out" - testing.go:149: server-e9bae396-860a-aac1-bfb4-a462c2172ee0: 2021-07-10T01:05:13.986Z [ERROR] server-e9bae396-860a-aac1-bfb4-a462c2172ee0: failed to make requestVote RPC: target="{Voter server-6d4f2788-2b13-a8b9-e435-a0774f871ab8 6d4f2788-2b13-a8b9-e435-a0774f871ab8}" error="command timed out" -testLoggerAdapter verbose: server-e9bae396-860a-aac1-bfb4-a462c2172ee0: 2021-07-10T01:05:13.986Z [ERROR] server-e9bae396-860a-aac1-bfb4-a462c2172ee0: failed to make requestVote RPC: target="{Voter server-57b5d042-8833-3178-5c09-e3f830dfd7b4 57b5d042-8833-3178-5c09-e3f830dfd7b4}" error="command timed out" - testing.go:149: server-e9bae396-860a-aac1-bfb4-a462c2172ee0: 2021-07-10T01:05:13.986Z [ERROR] server-e9bae396-860a-aac1-bfb4-a462c2172ee0: failed to make requestVote RPC: target="{Voter server-57b5d042-8833-3178-5c09-e3f830dfd7b4 57b5d042-8833-3178-5c09-e3f830dfd7b4}" error="command timed out" -testLoggerAdapter verbose: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.989Z [ERROR] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: failed to heartbeat to: peer=6d4f2788-2b13-a8b9-e435-a0774f871ab8 error="command timed out" - testing.go:149: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.989Z [ERROR] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: failed to heartbeat to: peer=6d4f2788-2b13-a8b9-e435-a0774f871ab8 error="command timed out" -testLoggerAdapter verbose: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.989Z [ERROR] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: failed to heartbeat to: peer=e9bae396-860a-aac1-bfb4-a462c2172ee0 error="command timed out" - testing.go:149: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.989Z [ERROR] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: failed to heartbeat to: peer=e9bae396-860a-aac1-bfb4-a462c2172ee0 error="command timed out" ---- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (0.72s) +testLoggerAdapter verbose: server-5ef7b7ba-2ec2-3ba8-233d-e8d96f21d332: 2022-08-12T07:30:17.827Z [INFO] server-5ef7b7ba-2ec2-3ba8-233d-e8d96f21d332: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5ef7b7ba-2ec2-3ba8-233d-e8d96f21d332 Address:5ef7b7ba-2ec2-3ba8-233d-e8d96f21d332} {Suffrage:Voter ID:server-a4b627c2-635b-ad9e-e2ba-d4faace10854 Address:a4b627c2-635b-ad9e-e2ba-d4faace10854} {Suffrage:Voter ID:server-c49fbdfa-de9c-9fb4-d68a-4a1392ca97f6 Address:c49fbdfa-de9c-9fb4-d68a-4a1392ca97f6}]" + testing.go:149: server-5ef7b7ba-2ec2-3ba8-233d-e8d96f21d332: 2022-08-12T07:30:17.827Z [INFO] server-5ef7b7ba-2ec2-3ba8-233d-e8d96f21d332: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5ef7b7ba-2ec2-3ba8-233d-e8d96f21d332 Address:5ef7b7ba-2ec2-3ba8-233d-e8d96f21d332} {Suffrage:Voter ID:server-a4b627c2-635b-ad9e-e2ba-d4faace10854 Address:a4b627c2-635b-ad9e-e2ba-d4faace10854} {Suffrage:Voter ID:server-c49fbdfa-de9c-9fb4-d68a-4a1392ca97f6 Address:c49fbdfa-de9c-9fb4-d68a-4a1392ca97f6}]" +testLoggerAdapter verbose: server-a4b627c2-635b-ad9e-e2ba-d4faace10854: 2022-08-12T07:30:17.827Z [INFO] server-a4b627c2-635b-ad9e-e2ba-d4faace10854: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5ef7b7ba-2ec2-3ba8-233d-e8d96f21d332 Address:5ef7b7ba-2ec2-3ba8-233d-e8d96f21d332} {Suffrage:Voter ID:server-a4b627c2-635b-ad9e-e2ba-d4faace10854 Address:a4b627c2-635b-ad9e-e2ba-d4faace10854} {Suffrage:Voter ID:server-c49fbdfa-de9c-9fb4-d68a-4a1392ca97f6 Address:c49fbdfa-de9c-9fb4-d68a-4a1392ca97f6}]" + testing.go:149: server-a4b627c2-635b-ad9e-e2ba-d4faace10854: 2022-08-12T07:30:17.827Z [INFO] server-a4b627c2-635b-ad9e-e2ba-d4faace10854: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5ef7b7ba-2ec2-3ba8-233d-e8d96f21d332 Address:5ef7b7ba-2ec2-3ba8-233d-e8d96f21d332} {Suffrage:Voter ID:server-a4b627c2-635b-ad9e-e2ba-d4faace10854 Address:a4b627c2-635b-ad9e-e2ba-d4faace10854} {Suffrage:Voter ID:server-c49fbdfa-de9c-9fb4-d68a-4a1392ca97f6 Address:c49fbdfa-de9c-9fb4-d68a-4a1392ca97f6}]" +testLoggerAdapter verbose: server-c49fbdfa-de9c-9fb4-d68a-4a1392ca97f6: 2022-08-12T07:30:17.828Z [INFO] server-c49fbdfa-de9c-9fb4-d68a-4a1392ca97f6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5ef7b7ba-2ec2-3ba8-233d-e8d96f21d332 Address:5ef7b7ba-2ec2-3ba8-233d-e8d96f21d332} {Suffrage:Voter ID:server-a4b627c2-635b-ad9e-e2ba-d4faace10854 Address:a4b627c2-635b-ad9e-e2ba-d4faace10854} {Suffrage:Voter ID:server-c49fbdfa-de9c-9fb4-d68a-4a1392ca97f6 Address:c49fbdfa-de9c-9fb4-d68a-4a1392ca97f6}]" + testing.go:149: server-c49fbdfa-de9c-9fb4-d68a-4a1392ca97f6: 2022-08-12T07:30:17.828Z [INFO] server-c49fbdfa-de9c-9fb4-d68a-4a1392ca97f6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5ef7b7ba-2ec2-3ba8-233d-e8d96f21d332 Address:5ef7b7ba-2ec2-3ba8-233d-e8d96f21d332} {Suffrage:Voter ID:server-a4b627c2-635b-ad9e-e2ba-d4faace10854 Address:a4b627c2-635b-ad9e-e2ba-d4faace10854} {Suffrage:Voter ID:server-c49fbdfa-de9c-9fb4-d68a-4a1392ca97f6 Address:c49fbdfa-de9c-9fb4-d68a-4a1392ca97f6}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:17.828Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:17.828Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: server-5ef7b7ba-2ec2-3ba8-233d-e8d96f21d332: 2022-08-12T07:30:17.828Z [INFO] server-5ef7b7ba-2ec2-3ba8-233d-e8d96f21d332: entering follower state: follower="Node at 5ef7b7ba-2ec2-3ba8-233d-e8d96f21d332 [Follower]" leader= + testing.go:149: server-5ef7b7ba-2ec2-3ba8-233d-e8d96f21d332: 2022-08-12T07:30:17.828Z [INFO] server-5ef7b7ba-2ec2-3ba8-233d-e8d96f21d332: entering follower state: follower="Node at 5ef7b7ba-2ec2-3ba8-233d-e8d96f21d332 [Follower]" leader= +testLoggerAdapter verbose: server-a4b627c2-635b-ad9e-e2ba-d4faace10854: 2022-08-12T07:30:17.828Z [INFO] server-a4b627c2-635b-ad9e-e2ba-d4faace10854: entering follower state: follower="Node at a4b627c2-635b-ad9e-e2ba-d4faace10854 [Follower]" leader= + testing.go:149: server-a4b627c2-635b-ad9e-e2ba-d4faace10854: 2022-08-12T07:30:17.828Z [INFO] server-a4b627c2-635b-ad9e-e2ba-d4faace10854: entering follower state: follower="Node at a4b627c2-635b-ad9e-e2ba-d4faace10854 [Follower]" leader= +testLoggerAdapter verbose: server-c49fbdfa-de9c-9fb4-d68a-4a1392ca97f6: 2022-08-12T07:30:17.828Z [INFO] server-c49fbdfa-de9c-9fb4-d68a-4a1392ca97f6: entering follower state: follower="Node at c49fbdfa-de9c-9fb4-d68a-4a1392ca97f6 [Follower]" leader= + testing.go:149: server-c49fbdfa-de9c-9fb4-d68a-4a1392ca97f6: 2022-08-12T07:30:17.828Z [INFO] server-c49fbdfa-de9c-9fb4-d68a-4a1392ca97f6: entering follower state: follower="Node at c49fbdfa-de9c-9fb4-d68a-4a1392ca97f6 [Follower]" leader= +testLoggerAdapter verbose: server-a4b627c2-635b-ad9e-e2ba-d4faace10854: 2022-08-12T07:30:17.897Z [WARN] server-a4b627c2-635b-ad9e-e2ba-d4faace10854: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-a4b627c2-635b-ad9e-e2ba-d4faace10854: 2022-08-12T07:30:17.897Z [WARN] server-a4b627c2-635b-ad9e-e2ba-d4faace10854: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-a4b627c2-635b-ad9e-e2ba-d4faace10854: 2022-08-12T07:30:17.898Z [INFO] server-a4b627c2-635b-ad9e-e2ba-d4faace10854: entering candidate state: node="Node at a4b627c2-635b-ad9e-e2ba-d4faace10854 [Candidate]" term=2 + testing.go:149: server-a4b627c2-635b-ad9e-e2ba-d4faace10854: 2022-08-12T07:30:17.898Z [INFO] server-a4b627c2-635b-ad9e-e2ba-d4faace10854: entering candidate state: node="Node at a4b627c2-635b-ad9e-e2ba-d4faace10854 [Candidate]" term=2 +testLoggerAdapter verbose: server-a4b627c2-635b-ad9e-e2ba-d4faace10854: 2022-08-12T07:30:17.898Z [INFO] server-a4b627c2-635b-ad9e-e2ba-d4faace10854: election won: tally=2 + testing.go:149: server-a4b627c2-635b-ad9e-e2ba-d4faace10854: 2022-08-12T07:30:17.898Z [INFO] server-a4b627c2-635b-ad9e-e2ba-d4faace10854: election won: tally=2 +testLoggerAdapter verbose: server-a4b627c2-635b-ad9e-e2ba-d4faace10854: 2022-08-12T07:30:17.898Z [INFO] server-a4b627c2-635b-ad9e-e2ba-d4faace10854: entering leader state: leader="Node at a4b627c2-635b-ad9e-e2ba-d4faace10854 [Leader]" + testing.go:149: server-a4b627c2-635b-ad9e-e2ba-d4faace10854: 2022-08-12T07:30:17.898Z [INFO] server-a4b627c2-635b-ad9e-e2ba-d4faace10854: entering leader state: leader="Node at a4b627c2-635b-ad9e-e2ba-d4faace10854 [Leader]" +testLoggerAdapter verbose: server-a4b627c2-635b-ad9e-e2ba-d4faace10854: 2022-08-12T07:30:17.898Z [INFO] server-a4b627c2-635b-ad9e-e2ba-d4faace10854: added peer, starting replication: peer=server-5ef7b7ba-2ec2-3ba8-233d-e8d96f21d332 + testing.go:149: server-a4b627c2-635b-ad9e-e2ba-d4faace10854: 2022-08-12T07:30:17.898Z [INFO] server-a4b627c2-635b-ad9e-e2ba-d4faace10854: added peer, starting replication: peer=server-5ef7b7ba-2ec2-3ba8-233d-e8d96f21d332 +testLoggerAdapter verbose: server-a4b627c2-635b-ad9e-e2ba-d4faace10854: 2022-08-12T07:30:17.898Z [INFO] server-a4b627c2-635b-ad9e-e2ba-d4faace10854: added peer, starting replication: peer=server-c49fbdfa-de9c-9fb4-d68a-4a1392ca97f6 + testing.go:149: server-a4b627c2-635b-ad9e-e2ba-d4faace10854: 2022-08-12T07:30:17.898Z [INFO] server-a4b627c2-635b-ad9e-e2ba-d4faace10854: added peer, starting replication: peer=server-c49fbdfa-de9c-9fb4-d68a-4a1392ca97f6 +testLoggerAdapter verbose: server-a4b627c2-635b-ad9e-e2ba-d4faace10854: 2022-08-12T07:30:17.898Z [INFO] server-a4b627c2-635b-ad9e-e2ba-d4faace10854: pipelining replication: peer="{Voter server-5ef7b7ba-2ec2-3ba8-233d-e8d96f21d332 5ef7b7ba-2ec2-3ba8-233d-e8d96f21d332}" + testing.go:149: server-a4b627c2-635b-ad9e-e2ba-d4faace10854: 2022-08-12T07:30:17.898Z [INFO] server-a4b627c2-635b-ad9e-e2ba-d4faace10854: pipelining replication: peer="{Voter server-5ef7b7ba-2ec2-3ba8-233d-e8d96f21d332 5ef7b7ba-2ec2-3ba8-233d-e8d96f21d332}" +testLoggerAdapter verbose: server-a4b627c2-635b-ad9e-e2ba-d4faace10854: 2022-08-12T07:30:17.898Z [INFO] server-a4b627c2-635b-ad9e-e2ba-d4faace10854: pipelining replication: peer="{Voter server-c49fbdfa-de9c-9fb4-d68a-4a1392ca97f6 c49fbdfa-de9c-9fb4-d68a-4a1392ca97f6}" + testing.go:149: server-a4b627c2-635b-ad9e-e2ba-d4faace10854: 2022-08-12T07:30:17.898Z [INFO] server-a4b627c2-635b-ad9e-e2ba-d4faace10854: pipelining replication: peer="{Voter server-c49fbdfa-de9c-9fb4-d68a-4a1392ca97f6 c49fbdfa-de9c-9fb4-d68a-4a1392ca97f6}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:18.003Z [INFO] stable state for Leader reached at 2022-08-12 07:30:17.89841024 +0000 UTC m=+18.855096721 (1 nodes), 70.04556ms from start of poll, 70.99364ms from cluster start. Timeout at 2022-08-12 07:30:18.0034936 +0000 UTC m=+18.960180061, 105.08334ms after stability + testing.go:149: cluster: 2022-08-12T07:30:18.003Z [INFO] stable state for Leader reached at 2022-08-12 07:30:17.89841024 +0000 UTC m=+18.855096721 (1 nodes), 70.04556ms from start of poll, 70.99364ms from cluster start. Timeout at 2022-08-12 07:30:18.0034936 +0000 UTC m=+18.960180061, 105.08334ms after stability +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:18.003Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2022-08-12T07:30:18.003Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:18.108Z [INFO] stable state for Follower reached at 2022-08-12 07:30:18.00377448 +0000 UTC m=+18.960460941 (2 nodes), 2.52µs from start of poll, 176.35786ms from cluster start. Timeout at 2022-08-12 07:30:18.10885844 +0000 UTC m=+19.065544901, 105.08396ms after stability + testing.go:149: cluster: 2022-08-12T07:30:18.108Z [INFO] stable state for Follower reached at 2022-08-12 07:30:18.00377448 +0000 UTC m=+18.960460941 (2 nodes), 2.52µs from start of poll, 176.35786ms from cluster start. Timeout at 2022-08-12 07:30:18.10885844 +0000 UTC m=+19.065544901, 105.08396ms after stability +testLoggerAdapter verbose: server-a4b627c2-635b-ad9e-e2ba-d4faace10854: 2022-08-12T07:30:18.488Z [INFO] server-a4b627c2-635b-ad9e-e2ba-d4faace10854: aborting pipeline replication: peer="{Voter server-5ef7b7ba-2ec2-3ba8-233d-e8d96f21d332 5ef7b7ba-2ec2-3ba8-233d-e8d96f21d332}" + testing.go:149: server-a4b627c2-635b-ad9e-e2ba-d4faace10854: 2022-08-12T07:30:18.488Z [INFO] server-a4b627c2-635b-ad9e-e2ba-d4faace10854: aborting pipeline replication: peer="{Voter server-5ef7b7ba-2ec2-3ba8-233d-e8d96f21d332 5ef7b7ba-2ec2-3ba8-233d-e8d96f21d332}" +testLoggerAdapter verbose: server-a4b627c2-635b-ad9e-e2ba-d4faace10854: 2022-08-12T07:30:18.488Z [INFO] server-a4b627c2-635b-ad9e-e2ba-d4faace10854: aborting pipeline replication: peer="{Voter server-c49fbdfa-de9c-9fb4-d68a-4a1392ca97f6 c49fbdfa-de9c-9fb4-d68a-4a1392ca97f6}" + testing.go:149: server-a4b627c2-635b-ad9e-e2ba-d4faace10854: 2022-08-12T07:30:18.488Z [INFO] server-a4b627c2-635b-ad9e-e2ba-d4faace10854: aborting pipeline replication: peer="{Voter server-c49fbdfa-de9c-9fb4-d68a-4a1392ca97f6 c49fbdfa-de9c-9fb4-d68a-4a1392ca97f6}" +--- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (0.66s) === RUN TestRaft_LeadershipTransferStopRightAway --- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s) === RUN TestRaft_GetConfigurationNoBootstrap -testLoggerAdapter verbose: server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42: 2021-07-10T01:05:13.991Z [INFO] server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42 Address:6acaff73-6bcc-8805-4ba3-6fcb9e528f42} {Suffrage:Voter ID:server-7546905d-7105-f2b6-e4b1-acbde4cf5589 Address:7546905d-7105-f2b6-e4b1-acbde4cf5589}]" - testing.go:149: server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42: 2021-07-10T01:05:13.991Z [INFO] server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42 Address:6acaff73-6bcc-8805-4ba3-6fcb9e528f42} {Suffrage:Voter ID:server-7546905d-7105-f2b6-e4b1-acbde4cf5589 Address:7546905d-7105-f2b6-e4b1-acbde4cf5589}]" -testLoggerAdapter verbose: server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42: 2021-07-10T01:05:13.991Z [INFO] server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42: entering follower state: follower="Node at 6acaff73-6bcc-8805-4ba3-6fcb9e528f42 [Follower]" leader= - testing.go:149: server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42: 2021-07-10T01:05:13.991Z [INFO] server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42: entering follower state: follower="Node at 6acaff73-6bcc-8805-4ba3-6fcb9e528f42 [Follower]" leader= -testLoggerAdapter verbose: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:13.991Z [INFO] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42 Address:6acaff73-6bcc-8805-4ba3-6fcb9e528f42} {Suffrage:Voter ID:server-7546905d-7105-f2b6-e4b1-acbde4cf5589 Address:7546905d-7105-f2b6-e4b1-acbde4cf5589}]" - testing.go:149: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:13.991Z [INFO] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42 Address:6acaff73-6bcc-8805-4ba3-6fcb9e528f42} {Suffrage:Voter ID:server-7546905d-7105-f2b6-e4b1-acbde4cf5589 Address:7546905d-7105-f2b6-e4b1-acbde4cf5589}]" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:13.991Z [INFO] starting stability test: raft-state=Follower - testing.go:149: cluster: 2021-07-10T01:05:13.991Z [INFO] starting stability test: raft-state=Follower -testLoggerAdapter verbose: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:13.991Z [INFO] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: entering follower state: follower="Node at 7546905d-7105-f2b6-e4b1-acbde4cf5589 [Follower]" leader= - testing.go:149: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:13.991Z [INFO] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: entering follower state: follower="Node at 7546905d-7105-f2b6-e4b1-acbde4cf5589 [Follower]" leader= -testLoggerAdapter verbose: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:14.067Z [WARN] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: heartbeat timeout reached, starting election: last-leader= - testing.go:149: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:14.067Z [WARN] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: heartbeat timeout reached, starting election: last-leader= -testLoggerAdapter verbose: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:14.067Z [INFO] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: entering candidate state: node="Node at 7546905d-7105-f2b6-e4b1-acbde4cf5589 [Candidate]" term=2 - testing.go:149: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:14.067Z [INFO] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: entering candidate state: node="Node at 7546905d-7105-f2b6-e4b1-acbde4cf5589 [Candidate]" term=2 -testLoggerAdapter verbose: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:14.067Z [INFO] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: election won: tally=2 - testing.go:149: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:14.067Z [INFO] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: election won: tally=2 -testLoggerAdapter verbose: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:14.067Z [INFO] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: entering leader state: leader="Node at 7546905d-7105-f2b6-e4b1-acbde4cf5589 [Leader]" - testing.go:149: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:14.067Z [INFO] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: entering leader state: leader="Node at 7546905d-7105-f2b6-e4b1-acbde4cf5589 [Leader]" -testLoggerAdapter verbose: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:14.067Z [INFO] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: added peer, starting replication: peer=server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42 - testing.go:149: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:14.067Z [INFO] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: added peer, starting replication: peer=server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42 -testLoggerAdapter verbose: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:14.068Z [INFO] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: pipelining replication: peer="{Voter server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42 6acaff73-6bcc-8805-4ba3-6fcb9e528f42}" - testing.go:149: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:14.068Z [INFO] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: pipelining replication: peer="{Voter server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42 6acaff73-6bcc-8805-4ba3-6fcb9e528f42}" -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:14.173Z [INFO] stable state for Follower reached at 2021-07-10 01:05:14.067887503 +0000 UTC m=+19.456045573 (1 nodes), 76.211883ms from start of poll, 76.924631ms from cluster start. Timeout at 2021-07-10 01:05:14.172974631 +0000 UTC m=+19.561132621, 105.087048ms after stability - testing.go:149: cluster: 2021-07-10T01:05:14.173Z [INFO] stable state for Follower reached at 2021-07-10 01:05:14.067887503 +0000 UTC m=+19.456045573 (1 nodes), 76.211883ms from start of poll, 76.924631ms from cluster start. Timeout at 2021-07-10 01:05:14.172974631 +0000 UTC m=+19.561132621, 105.087048ms after stability -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:14.173Z [INFO] starting stability test: raft-state=Leader - testing.go:149: cluster: 2021-07-10T01:05:14.173Z [INFO] starting stability test: raft-state=Leader -testLoggerAdapter verbose: cluster: 2021-07-10T01:05:14.278Z [INFO] stable state for Leader reached at 2021-07-10 01:05:14.173273334 +0000 UTC m=+19.561431304 (1 nodes), 2.28µs from start of poll, 182.310362ms from cluster start. Timeout at 2021-07-10 01:05:14.278350662 +0000 UTC m=+19.666508632, 105.077328ms after stability - testing.go:149: cluster: 2021-07-10T01:05:14.278Z [INFO] stable state for Leader reached at 2021-07-10 01:05:14.173273334 +0000 UTC m=+19.561431304 (1 nodes), 2.28µs from start of poll, 182.310362ms from cluster start. Timeout at 2021-07-10 01:05:14.278350662 +0000 UTC m=+19.666508632, 105.077328ms after stability -testLoggerAdapter verbose: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:14.289Z [INFO] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42 Address:6acaff73-6bcc-8805-4ba3-6fcb9e528f42} {Suffrage:Voter ID:server-7546905d-7105-f2b6-e4b1-acbde4cf5589 Address:7546905d-7105-f2b6-e4b1-acbde4cf5589}]" - testing.go:149: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:14.289Z [INFO] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42 Address:6acaff73-6bcc-8805-4ba3-6fcb9e528f42} {Suffrage:Voter ID:server-7546905d-7105-f2b6-e4b1-acbde4cf5589 Address:7546905d-7105-f2b6-e4b1-acbde4cf5589}]" -testLoggerAdapter verbose: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:14.289Z [INFO] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: aborting pipeline replication: peer="{Voter server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42 6acaff73-6bcc-8805-4ba3-6fcb9e528f42}" - testing.go:149: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:14.289Z [INFO] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: aborting pipeline replication: peer="{Voter server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42 6acaff73-6bcc-8805-4ba3-6fcb9e528f42}" ---- PASS: TestRaft_GetConfigurationNoBootstrap (0.30s) +testLoggerAdapter verbose: server-62058fec-11d8-52fc-54fa-a7ab05de69f3: 2022-08-12T07:30:18.490Z [INFO] server-62058fec-11d8-52fc-54fa-a7ab05de69f3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62058fec-11d8-52fc-54fa-a7ab05de69f3 Address:62058fec-11d8-52fc-54fa-a7ab05de69f3} {Suffrage:Voter ID:server-cb73cf3e-5150-6faa-d48a-b407a501caaf Address:cb73cf3e-5150-6faa-d48a-b407a501caaf}]" + testing.go:149: server-62058fec-11d8-52fc-54fa-a7ab05de69f3: 2022-08-12T07:30:18.490Z [INFO] server-62058fec-11d8-52fc-54fa-a7ab05de69f3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62058fec-11d8-52fc-54fa-a7ab05de69f3 Address:62058fec-11d8-52fc-54fa-a7ab05de69f3} {Suffrage:Voter ID:server-cb73cf3e-5150-6faa-d48a-b407a501caaf Address:cb73cf3e-5150-6faa-d48a-b407a501caaf}]" +testLoggerAdapter verbose: server-cb73cf3e-5150-6faa-d48a-b407a501caaf: 2022-08-12T07:30:18.491Z [INFO] server-cb73cf3e-5150-6faa-d48a-b407a501caaf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62058fec-11d8-52fc-54fa-a7ab05de69f3 Address:62058fec-11d8-52fc-54fa-a7ab05de69f3} {Suffrage:Voter ID:server-cb73cf3e-5150-6faa-d48a-b407a501caaf Address:cb73cf3e-5150-6faa-d48a-b407a501caaf}]" + testing.go:149: server-cb73cf3e-5150-6faa-d48a-b407a501caaf: 2022-08-12T07:30:18.491Z [INFO] server-cb73cf3e-5150-6faa-d48a-b407a501caaf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62058fec-11d8-52fc-54fa-a7ab05de69f3 Address:62058fec-11d8-52fc-54fa-a7ab05de69f3} {Suffrage:Voter ID:server-cb73cf3e-5150-6faa-d48a-b407a501caaf Address:cb73cf3e-5150-6faa-d48a-b407a501caaf}]" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:18.491Z [INFO] starting stability test: raft-state=Follower + testing.go:149: cluster: 2022-08-12T07:30:18.491Z [INFO] starting stability test: raft-state=Follower +testLoggerAdapter verbose: server-62058fec-11d8-52fc-54fa-a7ab05de69f3: 2022-08-12T07:30:18.491Z [INFO] server-62058fec-11d8-52fc-54fa-a7ab05de69f3: entering follower state: follower="Node at 62058fec-11d8-52fc-54fa-a7ab05de69f3 [Follower]" leader= + testing.go:149: server-62058fec-11d8-52fc-54fa-a7ab05de69f3: 2022-08-12T07:30:18.491Z [INFO] server-62058fec-11d8-52fc-54fa-a7ab05de69f3: entering follower state: follower="Node at 62058fec-11d8-52fc-54fa-a7ab05de69f3 [Follower]" leader= +testLoggerAdapter verbose: server-cb73cf3e-5150-6faa-d48a-b407a501caaf: 2022-08-12T07:30:18.491Z [INFO] server-cb73cf3e-5150-6faa-d48a-b407a501caaf: entering follower state: follower="Node at cb73cf3e-5150-6faa-d48a-b407a501caaf [Follower]" leader= + testing.go:149: server-cb73cf3e-5150-6faa-d48a-b407a501caaf: 2022-08-12T07:30:18.491Z [INFO] server-cb73cf3e-5150-6faa-d48a-b407a501caaf: entering follower state: follower="Node at cb73cf3e-5150-6faa-d48a-b407a501caaf [Follower]" leader= +testLoggerAdapter verbose: server-cb73cf3e-5150-6faa-d48a-b407a501caaf: 2022-08-12T07:30:18.579Z [WARN] server-cb73cf3e-5150-6faa-d48a-b407a501caaf: heartbeat timeout reached, starting election: last-leader= + testing.go:149: server-cb73cf3e-5150-6faa-d48a-b407a501caaf: 2022-08-12T07:30:18.579Z [WARN] server-cb73cf3e-5150-6faa-d48a-b407a501caaf: heartbeat timeout reached, starting election: last-leader= +testLoggerAdapter verbose: server-cb73cf3e-5150-6faa-d48a-b407a501caaf: 2022-08-12T07:30:18.579Z [INFO] server-cb73cf3e-5150-6faa-d48a-b407a501caaf: entering candidate state: node="Node at cb73cf3e-5150-6faa-d48a-b407a501caaf [Candidate]" term=2 + testing.go:149: server-cb73cf3e-5150-6faa-d48a-b407a501caaf: 2022-08-12T07:30:18.579Z [INFO] server-cb73cf3e-5150-6faa-d48a-b407a501caaf: entering candidate state: node="Node at cb73cf3e-5150-6faa-d48a-b407a501caaf [Candidate]" term=2 +testLoggerAdapter verbose: server-cb73cf3e-5150-6faa-d48a-b407a501caaf: 2022-08-12T07:30:18.579Z [INFO] server-cb73cf3e-5150-6faa-d48a-b407a501caaf: election won: tally=2 + testing.go:149: server-cb73cf3e-5150-6faa-d48a-b407a501caaf: 2022-08-12T07:30:18.579Z [INFO] server-cb73cf3e-5150-6faa-d48a-b407a501caaf: election won: tally=2 +testLoggerAdapter verbose: server-cb73cf3e-5150-6faa-d48a-b407a501caaf: 2022-08-12T07:30:18.579Z [INFO] server-cb73cf3e-5150-6faa-d48a-b407a501caaf: entering leader state: leader="Node at cb73cf3e-5150-6faa-d48a-b407a501caaf [Leader]" + testing.go:149: server-cb73cf3e-5150-6faa-d48a-b407a501caaf: 2022-08-12T07:30:18.579Z [INFO] server-cb73cf3e-5150-6faa-d48a-b407a501caaf: entering leader state: leader="Node at cb73cf3e-5150-6faa-d48a-b407a501caaf [Leader]" +testLoggerAdapter verbose: server-cb73cf3e-5150-6faa-d48a-b407a501caaf: 2022-08-12T07:30:18.579Z [INFO] server-cb73cf3e-5150-6faa-d48a-b407a501caaf: added peer, starting replication: peer=server-62058fec-11d8-52fc-54fa-a7ab05de69f3 + testing.go:149: server-cb73cf3e-5150-6faa-d48a-b407a501caaf: 2022-08-12T07:30:18.579Z [INFO] server-cb73cf3e-5150-6faa-d48a-b407a501caaf: added peer, starting replication: peer=server-62058fec-11d8-52fc-54fa-a7ab05de69f3 +testLoggerAdapter verbose: server-cb73cf3e-5150-6faa-d48a-b407a501caaf: 2022-08-12T07:30:18.579Z [INFO] server-cb73cf3e-5150-6faa-d48a-b407a501caaf: pipelining replication: peer="{Voter server-62058fec-11d8-52fc-54fa-a7ab05de69f3 62058fec-11d8-52fc-54fa-a7ab05de69f3}" + testing.go:149: server-cb73cf3e-5150-6faa-d48a-b407a501caaf: 2022-08-12T07:30:18.579Z [INFO] server-cb73cf3e-5150-6faa-d48a-b407a501caaf: pipelining replication: peer="{Voter server-62058fec-11d8-52fc-54fa-a7ab05de69f3 62058fec-11d8-52fc-54fa-a7ab05de69f3}" +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:18.684Z [INFO] stable state for Follower reached at 2022-08-12 07:30:18.57956186 +0000 UTC m=+19.536248341 (1 nodes), 88.34098ms from start of poll, 89.10646ms from cluster start. Timeout at 2022-08-12 07:30:18.68464258 +0000 UTC m=+19.641329041, 105.0807ms after stability + testing.go:149: cluster: 2022-08-12T07:30:18.684Z [INFO] stable state for Follower reached at 2022-08-12 07:30:18.57956186 +0000 UTC m=+19.536248341 (1 nodes), 88.34098ms from start of poll, 89.10646ms from cluster start. Timeout at 2022-08-12 07:30:18.68464258 +0000 UTC m=+19.641329041, 105.0807ms after stability +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:18.684Z [INFO] starting stability test: raft-state=Leader + testing.go:149: cluster: 2022-08-12T07:30:18.684Z [INFO] starting stability test: raft-state=Leader +testLoggerAdapter verbose: cluster: 2022-08-12T07:30:18.790Z [INFO] stable state for Leader reached at 2022-08-12 07:30:18.68488586 +0000 UTC m=+19.641572341 (1 nodes), 2.14µs from start of poll, 194.43046ms from cluster start. Timeout at 2022-08-12 07:30:18.78993142 +0000 UTC m=+19.746618021, 105.04568ms after stability + testing.go:149: cluster: 2022-08-12T07:30:18.790Z [INFO] stable state for Leader reached at 2022-08-12 07:30:18.68488586 +0000 UTC m=+19.641572341 (1 nodes), 2.14µs from start of poll, 194.43046ms from cluster start. Timeout at 2022-08-12 07:30:18.78993142 +0000 UTC m=+19.746618021, 105.04568ms after stability +testLoggerAdapter verbose: server-cb73cf3e-5150-6faa-d48a-b407a501caaf: 2022-08-12T07:30:18.804Z [INFO] server-cb73cf3e-5150-6faa-d48a-b407a501caaf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62058fec-11d8-52fc-54fa-a7ab05de69f3 Address:62058fec-11d8-52fc-54fa-a7ab05de69f3} {Suffrage:Voter ID:server-cb73cf3e-5150-6faa-d48a-b407a501caaf Address:cb73cf3e-5150-6faa-d48a-b407a501caaf}]" + testing.go:149: server-cb73cf3e-5150-6faa-d48a-b407a501caaf: 2022-08-12T07:30:18.804Z [INFO] server-cb73cf3e-5150-6faa-d48a-b407a501caaf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62058fec-11d8-52fc-54fa-a7ab05de69f3 Address:62058fec-11d8-52fc-54fa-a7ab05de69f3} {Suffrage:Voter ID:server-cb73cf3e-5150-6faa-d48a-b407a501caaf Address:cb73cf3e-5150-6faa-d48a-b407a501caaf}]" +testLoggerAdapter verbose: server-cb73cf3e-5150-6faa-d48a-b407a501caaf: 2022-08-12T07:30:18.804Z [INFO] server-cb73cf3e-5150-6faa-d48a-b407a501caaf: aborting pipeline replication: peer="{Voter server-62058fec-11d8-52fc-54fa-a7ab05de69f3 62058fec-11d8-52fc-54fa-a7ab05de69f3}" + testing.go:149: server-cb73cf3e-5150-6faa-d48a-b407a501caaf: 2022-08-12T07:30:18.804Z [INFO] server-cb73cf3e-5150-6faa-d48a-b407a501caaf: aborting pipeline replication: peer="{Voter server-62058fec-11d8-52fc-54fa-a7ab05de69f3 62058fec-11d8-52fc-54fa-a7ab05de69f3}" +--- PASS: TestRaft_GetConfigurationNoBootstrap (0.31s) === RUN TestTCPTransport_BadAddr --- PASS: TestTCPTransport_BadAddr (0.00s) === RUN TestTCPTransport_WithAdvertise @@ -3935,7 +4066,7 @@ === RUN TestBackoff --- PASS: TestBackoff (0.00s) PASS -ok github.com/hashicorp/raft 19.697s +ok github.com/hashicorp/raft 19.780s ? github.com/hashicorp/raft/bench [no test files] create-stamp debian/debhelper-build-stamp fakeroot debian/rules binary @@ -3967,12 +4098,14 @@ dpkg-buildpackage: info: binary-only upload (no source included) dpkg-genchanges: info: including full source code in upload I: copying local configuration +I: user script /srv/workspace/pbuilder/22862/tmp/hooks/B01_cleanup starting +I: user script /srv/workspace/pbuilder/22862/tmp/hooks/B01_cleanup finished I: unmounting dev/ptmx filesystem I: unmounting dev/pts filesystem I: unmounting dev/shm filesystem I: unmounting proc filesystem I: unmounting sys filesystem I: cleaning the build env -I: removing directory /srv/workspace/pbuilder/15724 and its subdirectories -I: Current time: Fri Jul 9 13:05:22 -12 2021 -I: pbuilder-time-stamp: 1625879122 +I: removing directory /srv/workspace/pbuilder/22862 and its subdirectories +I: Current time: Fri Aug 12 21:30:27 +14 2022 +I: pbuilder-time-stamp: 1660289427