I: pbuilder: network access will be disabled during build I: Current time: Thu Aug 3 05:11:44 +14 2023 I: pbuilder-time-stamp: 1690989104 I: Building the build Environment I: extracting base tarball [/var/cache/pbuilder/bullseye-reproducible-base.tgz] I: copying local configuration W: --override-config is not set; not updating apt.conf Read the manpage for details. I: mounting /proc filesystem I: mounting /sys filesystem I: creating /{dev,run}/shm I: mounting /dev/pts filesystem I: redirecting /dev/ptmx to /dev/pts/ptmx I: policy-rc.d already exists I: Copying source file I: copying [golang-github-hashicorp-raft_1.1.2-1.dsc] I: copying [./golang-github-hashicorp-raft_1.1.2.orig.tar.xz] I: copying [./golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz] I: Extracting source gpgv: unknown type of key resource 'trustedkeys.kbx' gpgv: keyblock resource '/tmp/dpkg-verify-sig.zybKIOlR/trustedkeys.kbx': General error gpgv: Signature made Sun Jan 19 21:24:28 2020 +14 gpgv: using RSA key 50BC7CF939D20C272A6B065652B6BBD953968D1B gpgv: Can't check signature: No public key dpkg-source: warning: failed to verify signature on ./golang-github-hashicorp-raft_1.1.2-1.dsc dpkg-source: info: extracting golang-github-hashicorp-raft in golang-github-hashicorp-raft-1.1.2 dpkg-source: info: unpacking golang-github-hashicorp-raft_1.1.2.orig.tar.xz dpkg-source: info: unpacking golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz dpkg-source: info: using patch list from debian/patches/series dpkg-source: info: applying t-skip--TestNetworkTransport_AppendEntriesPipeline_CloseStreams.patch dpkg-source: info: applying t-skip--TestRaft_AutoSnapshot.patch dpkg-source: info: applying t-skip--TestRaft_LeadershipTransferLeaderRejectsClientRequests.patch dpkg-source: info: applying t-skip--TestRaft_RecoverCluster.patch dpkg-source: info: applying t-skip--TestRaft_SnapshotRestore_PeerChange.patch I: using fakeroot in build. I: Installing the build-deps I: user script /srv/workspace/pbuilder/2531752/tmp/hooks/D01_modify_environment starting debug: Running on ionos11-amd64. I: Changing host+domainname to test build reproducibility I: Adding a custom variable just for the fun of it... I: Changing /bin/sh to bash '/bin/sh' -> '/bin/bash' lrwxrwxrwx 1 root root 9 Aug 3 05:11 /bin/sh -> /bin/bash I: Setting pbuilder2's login shell to /bin/bash I: Setting pbuilder2's GECOS to second user,second room,second work-phone,second home-phone,second other I: user script /srv/workspace/pbuilder/2531752/tmp/hooks/D01_modify_environment finished I: user script /srv/workspace/pbuilder/2531752/tmp/hooks/D02_print_environment starting I: set BASH=/bin/sh BASHOPTS=checkwinsize:cmdhist:complete_fullquote:extquote:force_fignore:globasciiranges:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath BASH_ALIASES=() BASH_ARGC=() BASH_ARGV=() BASH_CMDS=() BASH_LINENO=([0]="12" [1]="0") BASH_SOURCE=([0]="/tmp/hooks/D02_print_environment" [1]="/tmp/hooks/D02_print_environment") BASH_VERSINFO=([0]="5" [1]="1" [2]="4" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu") BASH_VERSION='5.1.4(1)-release' BUILDDIR=/build BUILDUSERGECOS='second user,second room,second work-phone,second home-phone,second other' BUILDUSERNAME=pbuilder2 BUILD_ARCH=amd64 DEBIAN_FRONTEND=noninteractive DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all,-fixfilepath parallel=15 ' DIRSTACK=() DISTRIBUTION=bullseye EUID=0 FUNCNAME=([0]="Echo" [1]="main") GROUPS=() HOME=/root HOSTNAME=i-capture-the-hostname HOSTTYPE=x86_64 HOST_ARCH=amd64 IFS=' ' INVOCATION_ID=86b54b7b40ac49509f413555e39cf35d LANG=C LANGUAGE=et_EE:et LC_ALL=C MACHTYPE=x86_64-pc-linux-gnu MAIL=/var/mail/root OPTERR=1 OPTIND=1 OSTYPE=linux-gnu PATH=/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path PBCURRENTCOMMANDLINEOPERATION=build PBUILDER_OPERATION=build PBUILDER_PKGDATADIR=/usr/share/pbuilder PBUILDER_PKGLIBDIR=/usr/lib/pbuilder PBUILDER_SYSCONFDIR=/etc PIPESTATUS=([0]="0") POSIXLY_CORRECT=y PPID=2531752 PS4='+ ' PWD=/ SHELL=/bin/bash SHELLOPTS=braceexpand:errexit:hashall:interactive-comments:posix SHLVL=3 SUDO_COMMAND='/usr/bin/timeout -k 24.1h 24h /usr/bin/ionice -c 3 /usr/bin/nice -n 11 /usr/bin/unshare --uts -- /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.zfXk6ywP/pbuilderrc_RcYU --distribution bullseye --hookdir /etc/pbuilder/rebuild-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/bullseye-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.zfXk6ywP/b2 --logfile b2/build.log golang-github-hashicorp-raft_1.1.2-1.dsc' SUDO_GID=111 SUDO_UID=106 SUDO_USER=jenkins TERM=unknown TZ=/usr/share/zoneinfo/Etc/GMT-14 UID=0 USER=root _='I: set' http_proxy=http://78.137.99.97:3128 I: uname -a Linux i-capture-the-hostname 6.1.0-10-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.38-2 (2023-07-27) x86_64 GNU/Linux I: ls -l /bin total 5476 -rwxr-xr-x 1 root root 1234376 Mar 28 2022 bash -rwxr-xr-x 3 root root 38984 Jul 21 2020 bunzip2 -rwxr-xr-x 3 root root 38984 Jul 21 2020 bzcat lrwxrwxrwx 1 root root 6 Jul 21 2020 bzcmp -> bzdiff -rwxr-xr-x 1 root root 2225 Jul 21 2020 bzdiff lrwxrwxrwx 1 root root 6 Jul 21 2020 bzegrep -> bzgrep -rwxr-xr-x 1 root root 4877 Sep 5 2019 bzexe lrwxrwxrwx 1 root root 6 Jul 21 2020 bzfgrep -> bzgrep -rwxr-xr-x 1 root root 3775 Jul 21 2020 bzgrep -rwxr-xr-x 3 root root 38984 Jul 21 2020 bzip2 -rwxr-xr-x 1 root root 18424 Jul 21 2020 bzip2recover lrwxrwxrwx 1 root root 6 Jul 21 2020 bzless -> bzmore -rwxr-xr-x 1 root root 1297 Jul 21 2020 bzmore -rwxr-xr-x 1 root root 43936 Sep 24 2020 cat -rwxr-xr-x 1 root root 72672 Sep 24 2020 chgrp -rwxr-xr-x 1 root root 64448 Sep 24 2020 chmod -rwxr-xr-x 1 root root 72672 Sep 24 2020 chown -rwxr-xr-x 1 root root 151168 Sep 24 2020 cp -rwxr-xr-x 1 root root 125560 Dec 11 2020 dash -rwxr-xr-x 1 root root 113664 Sep 24 2020 date -rwxr-xr-x 1 root root 80968 Sep 24 2020 dd -rwxr-xr-x 1 root root 93936 Sep 24 2020 df -rwxr-xr-x 1 root root 147176 Sep 24 2020 dir -rwxr-xr-x 1 root root 84440 Jan 21 2022 dmesg lrwxrwxrwx 1 root root 8 Nov 8 2019 dnsdomainname -> hostname lrwxrwxrwx 1 root root 8 Nov 8 2019 domainname -> hostname -rwxr-xr-x 1 root root 39712 Sep 24 2020 echo -rwxr-xr-x 1 root root 28 Jan 25 2023 egrep -rwxr-xr-x 1 root root 39680 Sep 24 2020 false -rwxr-xr-x 1 root root 28 Jan 25 2023 fgrep -rwxr-xr-x 1 root root 69032 Jan 21 2022 findmnt -rwsr-xr-x 1 root root 34896 Feb 27 2021 fusermount -rwxr-xr-x 1 root root 203072 Jan 25 2023 grep -rwxr-xr-x 2 root root 2346 Apr 10 2022 gunzip -rwxr-xr-x 1 root root 6447 Apr 10 2022 gzexe -rwxr-xr-x 1 root root 98048 Apr 10 2022 gzip -rwxr-xr-x 1 root root 22600 Nov 8 2019 hostname -rwxr-xr-x 1 root root 72840 Sep 24 2020 ln -rwxr-xr-x 1 root root 56952 Feb 8 2020 login -rwxr-xr-x 1 root root 147176 Sep 24 2020 ls -rwxr-xr-x 1 root root 149736 Jan 21 2022 lsblk -rwxr-xr-x 1 root root 85184 Sep 24 2020 mkdir -rwxr-xr-x 1 root root 76896 Sep 24 2020 mknod -rwxr-xr-x 1 root root 48064 Sep 24 2020 mktemp -rwxr-xr-x 1 root root 59632 Jan 21 2022 more -rwsr-xr-x 1 root root 55528 Jan 21 2022 mount -rwxr-xr-x 1 root root 18664 Jan 21 2022 mountpoint -rwxr-xr-x 1 root root 147080 Sep 24 2020 mv lrwxrwxrwx 1 root root 8 Nov 8 2019 nisdomainname -> hostname lrwxrwxrwx 1 root root 14 Dec 17 2021 pidof -> /sbin/killall5 -rwxr-xr-x 1 root root 43872 Sep 24 2020 pwd lrwxrwxrwx 1 root root 4 Mar 28 2022 rbash -> bash -rwxr-xr-x 1 root root 52032 Sep 24 2020 readlink -rwxr-xr-x 1 root root 72704 Sep 24 2020 rm -rwxr-xr-x 1 root root 52032 Sep 24 2020 rmdir -rwxr-xr-x 1 root root 27472 Sep 28 2020 run-parts -rwxr-xr-x 1 root root 122224 Dec 23 2018 sed lrwxrwxrwx 1 root root 9 Aug 3 05:11 sh -> /bin/bash -rwxr-xr-x 1 root root 43808 Sep 24 2020 sleep -rwxr-xr-x 1 root root 84928 Sep 24 2020 stty -rwsr-xr-x 1 root root 71912 Jan 21 2022 su -rwxr-xr-x 1 root root 39744 Sep 24 2020 sync -rwxr-xr-x 1 root root 531928 Feb 17 2021 tar -rwxr-xr-x 1 root root 14456 Sep 28 2020 tempfile -rwxr-xr-x 1 root root 101408 Sep 24 2020 touch -rwxr-xr-x 1 root root 39680 Sep 24 2020 true -rwxr-xr-x 1 root root 14328 Feb 27 2021 ulockmgr_server -rwsr-xr-x 1 root root 35040 Jan 21 2022 umount -rwxr-xr-x 1 root root 39744 Sep 24 2020 uname -rwxr-xr-x 2 root root 2346 Apr 10 2022 uncompress -rwxr-xr-x 1 root root 147176 Sep 24 2020 vdir -rwxr-xr-x 1 root root 63744 Jan 21 2022 wdctl lrwxrwxrwx 1 root root 8 Nov 8 2019 ypdomainname -> hostname -rwxr-xr-x 1 root root 1984 Apr 10 2022 zcat -rwxr-xr-x 1 root root 1678 Apr 10 2022 zcmp -rwxr-xr-x 1 root root 5898 Apr 10 2022 zdiff -rwxr-xr-x 1 root root 29 Apr 10 2022 zegrep -rwxr-xr-x 1 root root 29 Apr 10 2022 zfgrep -rwxr-xr-x 1 root root 2081 Apr 10 2022 zforce -rwxr-xr-x 1 root root 8049 Apr 10 2022 zgrep -rwxr-xr-x 1 root root 2206 Apr 10 2022 zless -rwxr-xr-x 1 root root 1842 Apr 10 2022 zmore -rwxr-xr-x 1 root root 4577 Apr 10 2022 znew I: user script /srv/workspace/pbuilder/2531752/tmp/hooks/D02_print_environment finished -> Attempting to satisfy build-dependencies -> Creating pbuilder-satisfydepends-dummy package Package: pbuilder-satisfydepends-dummy Version: 0.invalid.0 Architecture: amd64 Maintainer: Debian Pbuilder Team Description: Dummy package to satisfy dependencies with aptitude - created by pbuilder This package was created automatically by pbuilder to satisfy the build-dependencies of the package being currently built. Depends: debhelper (>= 12~), dh-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~) dpkg-deb: building package 'pbuilder-satisfydepends-dummy' in '/tmp/satisfydepends-aptitude/pbuilder-satisfydepends-dummy.deb'. Selecting previously unselected package pbuilder-satisfydepends-dummy. (Reading database ... 19707 files and directories currently installed.) Preparing to unpack .../pbuilder-satisfydepends-dummy.deb ... Unpacking pbuilder-satisfydepends-dummy (0.invalid.0) ... dpkg: pbuilder-satisfydepends-dummy: dependency problems, but configuring anyway as you requested: pbuilder-satisfydepends-dummy depends on debhelper (>= 12~); however: Package debhelper is not installed. pbuilder-satisfydepends-dummy depends on dh-golang; however: Package dh-golang is not installed. pbuilder-satisfydepends-dummy depends on golang-any; however: Package golang-any is not installed. pbuilder-satisfydepends-dummy depends on golang-github-armon-go-metrics-dev; however: Package golang-github-armon-go-metrics-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-hclog-dev (>= 0.9.2~); however: Package golang-github-hashicorp-go-hclog-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~); however: Package golang-github-hashicorp-go-msgpack-dev is not installed. Setting up pbuilder-satisfydepends-dummy (0.invalid.0) ... Reading package lists... Building dependency tree... Reading state information... Initializing package states... Writing extended state information... Building tag database... pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0) pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0) The following NEW packages will be installed: autoconf{a} automake{a} autopoint{a} autotools-dev{a} bsdextrautils{a} debhelper{a} dh-autoreconf{a} dh-golang{a} dh-strip-nondeterminism{a} dwz{a} file{a} gettext{a} gettext-base{a} golang-1.15-go{a} golang-1.15-src{a} golang-any{a} golang-github-alecthomas-units-dev{a} golang-github-armon-go-metrics-dev{a} golang-github-beorn7-perks-dev{a} golang-github-cespare-xxhash-dev{a} golang-github-circonus-labs-circonus-gometrics-dev{a} golang-github-circonus-labs-circonusllhist-dev{a} golang-github-creack-pty-dev{a} golang-github-datadog-datadog-go-dev{a} golang-github-davecgh-go-spew-dev{a} golang-github-dgrijalva-jwt-go-dev{a} golang-github-fatih-color-dev{a} golang-github-go-kit-kit-dev{a} golang-github-go-logfmt-logfmt-dev{a} golang-github-go-stack-stack-dev{a} golang-github-golang-mock-dev{a} golang-github-google-go-cmp-dev{a} golang-github-hashicorp-go-cleanhttp-dev{a} golang-github-hashicorp-go-hclog-dev{a} golang-github-hashicorp-go-immutable-radix-dev{a} golang-github-hashicorp-go-msgpack-dev{a} golang-github-hashicorp-go-retryablehttp-dev{a} golang-github-hashicorp-go-uuid-dev{a} golang-github-hashicorp-golang-lru-dev{a} golang-github-jpillora-backoff-dev{a} golang-github-json-iterator-go-dev{a} golang-github-julienschmidt-httprouter-dev{a} golang-github-kr-pretty-dev{a} golang-github-kr-text-dev{a} golang-github-mattn-go-colorable-dev{a} golang-github-mattn-go-isatty-dev{a} golang-github-modern-go-concurrent-dev{a} golang-github-modern-go-reflect2-dev{a} golang-github-mwitkow-go-conntrack-dev{a} golang-github-opentracing-opentracing-go-dev{a} golang-github-pascaldekloe-goe-dev{a} golang-github-pkg-errors-dev{a} golang-github-pmezard-go-difflib-dev{a} golang-github-prometheus-client-golang-dev{a} golang-github-prometheus-client-model-dev{a} golang-github-prometheus-common-dev{a} golang-github-prometheus-procfs-dev{a} golang-github-sirupsen-logrus-dev{a} golang-github-stretchr-objx-dev{a} golang-github-stretchr-testify-dev{a} golang-github-tv42-httpunix-dev{a} golang-github-vmihailenco-tagparser-dev{a} golang-glog-dev{a} golang-go{a} golang-golang-x-net-dev{a} golang-golang-x-oauth2-dev{a} golang-golang-x-oauth2-google-dev{a} golang-golang-x-sync-dev{a} golang-golang-x-sys-dev{a} golang-golang-x-text-dev{a} golang-golang-x-time-dev{a} golang-golang-x-xerrors-dev{a} golang-google-cloud-compute-metadata-dev{a} golang-google-genproto-dev{a} golang-google-grpc-dev{a} golang-gopkg-alecthomas-kingpin.v2-dev{a} golang-gopkg-check.v1-dev{a} golang-gopkg-mgo.v2-dev{a} golang-gopkg-tomb.v2-dev{a} golang-gopkg-vmihailenco-msgpack.v2-dev{a} golang-gopkg-yaml.v2-dev{a} golang-gopkg-yaml.v3-dev{a} golang-goprotobuf-dev{a} golang-protobuf-extensions-dev{a} golang-src{a} groff-base{a} intltool-debian{a} libarchive-zip-perl{a} libdebhelper-perl{a} libelf1{a} libfile-stripnondeterminism-perl{a} libicu67{a} libmagic-mgc{a} libmagic1{a} libpipeline1{a} libprotobuf-dev{a} libprotobuf-lite23{a} libprotobuf23{a} libprotoc23{a} libsasl2-2{a} libsasl2-dev{a} libsasl2-modules-db{a} libsigsegv2{a} libsub-override-perl{a} libtool{a} libuchardet0{a} libxml2{a} m4{a} man-db{a} po-debconf{a} protobuf-compiler{a} sensible-utils{a} zlib1g-dev{a} The following packages are RECOMMENDED but will NOT be installed: curl libarchive-cpio-perl libltdl-dev libmail-sendmail-perl libsasl2-modules lynx pkg-config pkgconf wget 0 packages upgraded, 113 newly installed, 0 to remove and 0 not upgraded. Need to get 96.4 MB of archives. After unpacking 566 MB will be used. Writing extended state information... Get: 1 http://deb.debian.org/debian bullseye/main amd64 bsdextrautils amd64 2.36.1-8+deb11u1 [145 kB] Get: 2 http://deb.debian.org/debian bullseye/main amd64 libuchardet0 amd64 0.0.7-1 [67.8 kB] Get: 3 http://deb.debian.org/debian bullseye/main amd64 groff-base amd64 1.22.4-6 [936 kB] Get: 4 http://deb.debian.org/debian bullseye/main amd64 libpipeline1 amd64 1.5.3-1 [34.3 kB] Get: 5 http://deb.debian.org/debian bullseye/main amd64 man-db amd64 2.9.4-2 [1354 kB] Get: 6 http://deb.debian.org/debian bullseye/main amd64 golang-golang-x-sys-dev all 0.0~git20210124.22da62e-1 [308 kB] Get: 7 http://deb.debian.org/debian bullseye/main amd64 golang-github-sirupsen-logrus-dev all 1.7.0-2 [45.3 kB] Get: 8 http://deb.debian.org/debian bullseye/main amd64 sensible-utils all 0.0.14 [14.8 kB] Get: 9 http://deb.debian.org/debian bullseye/main amd64 libmagic-mgc amd64 1:5.39-3 [273 kB] Get: 10 http://deb.debian.org/debian bullseye/main amd64 libmagic1 amd64 1:5.39-3 [126 kB] Get: 11 http://deb.debian.org/debian bullseye/main amd64 file amd64 1:5.39-3 [69.1 kB] Get: 12 http://deb.debian.org/debian bullseye/main amd64 gettext-base amd64 0.21-4 [175 kB] Get: 13 http://deb.debian.org/debian bullseye/main amd64 libsigsegv2 amd64 2.13-1 [34.8 kB] Get: 14 http://deb.debian.org/debian bullseye/main amd64 m4 amd64 1.4.18-5 [204 kB] Get: 15 http://deb.debian.org/debian bullseye/main amd64 autoconf all 2.69-14 [313 kB] Get: 16 http://deb.debian.org/debian bullseye/main amd64 autotools-dev all 20180224.1+nmu1 [77.1 kB] Get: 17 http://deb.debian.org/debian bullseye/main amd64 automake all 1:1.16.3-2 [814 kB] Get: 18 http://deb.debian.org/debian bullseye/main amd64 autopoint all 0.21-4 [510 kB] Get: 19 http://deb.debian.org/debian bullseye/main amd64 libdebhelper-perl all 13.3.4 [189 kB] Get: 20 http://deb.debian.org/debian bullseye/main amd64 libtool all 2.4.6-15 [513 kB] Get: 21 http://deb.debian.org/debian bullseye/main amd64 dh-autoreconf all 20 [17.1 kB] Get: 22 http://deb.debian.org/debian bullseye/main amd64 libarchive-zip-perl all 1.68-1 [104 kB] Get: 23 http://deb.debian.org/debian bullseye/main amd64 libsub-override-perl all 0.09-2 [10.2 kB] Get: 24 http://deb.debian.org/debian bullseye/main amd64 libfile-stripnondeterminism-perl all 1.12.0-1 [26.3 kB] Get: 25 http://deb.debian.org/debian bullseye/main amd64 dh-strip-nondeterminism all 1.12.0-1 [15.4 kB] Get: 26 http://deb.debian.org/debian bullseye/main amd64 libelf1 amd64 0.183-1 [165 kB] Get: 27 http://deb.debian.org/debian bullseye/main amd64 dwz amd64 0.13+20210201-1 [175 kB] Get: 28 http://deb.debian.org/debian bullseye/main amd64 libicu67 amd64 67.1-7 [8622 kB] Get: 29 http://deb.debian.org/debian bullseye/main amd64 libxml2 amd64 2.9.10+dfsg-6.7+deb11u4 [693 kB] Get: 30 http://deb.debian.org/debian bullseye/main amd64 gettext amd64 0.21-4 [1311 kB] Get: 31 http://deb.debian.org/debian bullseye/main amd64 intltool-debian all 0.35.0+20060710.5 [26.8 kB] Get: 32 http://deb.debian.org/debian bullseye/main amd64 po-debconf all 1.0.21+nmu1 [248 kB] Get: 33 http://deb.debian.org/debian bullseye/main amd64 debhelper all 13.3.4 [1049 kB] Get: 34 http://deb.debian.org/debian bullseye/main amd64 dh-golang all 1.51 [24.6 kB] Get: 35 http://deb.debian.org/debian bullseye/main amd64 golang-1.15-src amd64 1.15.15-1~deb11u4 [13.9 MB] Get: 36 http://deb.debian.org/debian bullseye/main amd64 golang-1.15-go amd64 1.15.15-1~deb11u4 [47.4 MB] Get: 37 http://deb.debian.org/debian bullseye/main amd64 golang-src amd64 2:1.15~1 [4856 B] Get: 38 http://deb.debian.org/debian bullseye/main amd64 golang-go amd64 2:1.15~1 [23.9 kB] Get: 39 http://deb.debian.org/debian bullseye/main amd64 golang-any amd64 2:1.15~1 [4976 B] Get: 40 http://deb.debian.org/debian bullseye/main amd64 golang-github-alecthomas-units-dev all 0.0~git20201120.1786d5e-1 [7504 B] Get: 41 http://deb.debian.org/debian bullseye/main amd64 golang-github-circonus-labs-circonusllhist-dev all 0.0~git20191022.ec08cde-1 [11.1 kB] Get: 42 http://deb.debian.org/debian bullseye/main amd64 golang-github-hashicorp-go-cleanhttp-dev all 0.5.2-1 [10.8 kB] Get: 43 http://deb.debian.org/debian bullseye/main amd64 golang-github-mattn-go-isatty-dev all 0.0.12-1 [6472 B] Get: 44 http://deb.debian.org/debian bullseye/main amd64 golang-github-mattn-go-colorable-dev all 0.1.7-1 [9936 B] Get: 45 http://deb.debian.org/debian bullseye/main amd64 golang-github-fatih-color-dev all 1.7.0-1 [11.4 kB] Get: 46 http://deb.debian.org/debian bullseye/main amd64 golang-github-davecgh-go-spew-dev all 1.1.1-2 [29.7 kB] Get: 47 http://deb.debian.org/debian bullseye/main amd64 golang-github-pmezard-go-difflib-dev all 1.0.0-3 [12.3 kB] Get: 48 http://deb.debian.org/debian bullseye/main amd64 golang-github-stretchr-objx-dev all 0.3.0-1 [25.4 kB] Get: 49 http://deb.debian.org/debian bullseye/main amd64 golang-gopkg-yaml.v3-dev all 3.0.0~git20200121.a6ecf24-3 [70.5 kB] Get: 50 http://deb.debian.org/debian bullseye/main amd64 golang-github-stretchr-testify-dev all 1.6.1-2 [60.4 kB] Get: 51 http://deb.debian.org/debian bullseye/main amd64 golang-github-hashicorp-go-hclog-dev all 0.11.0-1 [17.9 kB] Get: 52 http://deb.debian.org/debian bullseye/main amd64 golang-github-hashicorp-go-retryablehttp-dev all 0.6.4-2 [17.5 kB] Get: 53 http://deb.debian.org/debian bullseye/main amd64 golang-github-pkg-errors-dev all 0.9.1-1 [13.0 kB] Get: 54 http://deb.debian.org/debian bullseye/main amd64 golang-github-tv42-httpunix-dev all 0.0~git20150427.b75d861-2 [3744 B] Get: 55 http://deb.debian.org/debian bullseye/main amd64 golang-github-circonus-labs-circonus-gometrics-dev all 2.3.1-3 [65.9 kB] Get: 56 http://deb.debian.org/debian bullseye/main amd64 golang-github-datadog-datadog-go-dev all 2.1.0-3 [14.9 kB] Get: 57 http://deb.debian.org/debian bullseye/main amd64 golang-github-hashicorp-go-uuid-dev all 1.0.2-1 [8764 B] Get: 58 http://deb.debian.org/debian bullseye/main amd64 golang-github-hashicorp-golang-lru-dev all 0.5.4-2 [14.7 kB] Get: 59 http://deb.debian.org/debian bullseye/main amd64 golang-github-hashicorp-go-immutable-radix-dev all 1.3.0-1 [25.0 kB] Get: 60 http://deb.debian.org/debian bullseye/main amd64 golang-github-pascaldekloe-goe-dev all 0.1.0-2 [21.7 kB] Get: 61 http://deb.debian.org/debian bullseye/main amd64 golang-github-beorn7-perks-dev all 1.0.1-1 [12.1 kB] Get: 62 http://deb.debian.org/debian bullseye/main amd64 golang-github-cespare-xxhash-dev all 2.1.1-1 [8748 B] Get: 63 http://deb.debian.org/debian bullseye/main amd64 golang-github-modern-go-concurrent-dev all 1.0.3-1.1 [4668 B] Get: 64 http://deb.debian.org/debian bullseye/main amd64 golang-github-modern-go-reflect2-dev all 1.0.1-1 [10.6 kB] Get: 65 http://deb.debian.org/debian bullseye/main amd64 golang-github-json-iterator-go-dev all 1.1.10-2 [69.4 kB] Get: 66 http://deb.debian.org/debian bullseye/main amd64 zlib1g-dev amd64 1:1.2.11.dfsg-2+deb11u2 [191 kB] Get: 67 http://deb.debian.org/debian bullseye/main amd64 libprotobuf23 amd64 3.12.4-1 [892 kB] Get: 68 http://deb.debian.org/debian bullseye/main amd64 libprotobuf-lite23 amd64 3.12.4-1 [241 kB] Get: 69 http://deb.debian.org/debian bullseye/main amd64 libprotobuf-dev amd64 3.12.4-1 [1232 kB] Get: 70 http://deb.debian.org/debian bullseye/main amd64 libprotoc23 amd64 3.12.4-1 [803 kB] Get: 71 http://deb.debian.org/debian bullseye/main amd64 protobuf-compiler amd64 3.12.4-1 [75.1 kB] Get: 72 http://deb.debian.org/debian bullseye/main amd64 golang-goprotobuf-dev amd64 1.3.4-2+b6 [1509 kB] Get: 73 http://deb.debian.org/debian bullseye/main amd64 golang-github-prometheus-client-model-dev all 0.2.0-2 [8476 B] Get: 74 http://deb.debian.org/debian bullseye/main amd64 golang-github-dgrijalva-jwt-go-dev all 3.2.0-3 [33.6 kB] Get: 75 http://deb.debian.org/debian bullseye/main amd64 golang-github-go-logfmt-logfmt-dev all 0.5.0-2 [13.4 kB] Get: 76 http://deb.debian.org/debian bullseye/main amd64 golang-github-go-stack-stack-dev all 1.8.0-1 [8696 B] Get: 77 http://deb.debian.org/debian bullseye/main amd64 golang-github-opentracing-opentracing-go-dev all 1.2.0-1 [29.4 kB] Get: 78 http://deb.debian.org/debian bullseye/main amd64 golang-golang-x-text-dev all 0.3.6-1 [3857 kB] Get: 79 http://deb.debian.org/debian bullseye/main amd64 golang-golang-x-net-dev all 1:0.0+git20210119.5f4716e+dfsg-4 [659 kB] Get: 80 http://deb.debian.org/debian bullseye/main amd64 golang-golang-x-time-dev all 0.0+git20200630.3af7569-1 [10.1 kB] Get: 81 http://deb.debian.org/debian bullseye/main amd64 golang-github-golang-mock-dev all 1.3.1-2 [35.1 kB] Get: 82 http://deb.debian.org/debian bullseye/main amd64 golang-golang-x-xerrors-dev all 0.0~git20191204.9bdfabe-1 [14.2 kB] Get: 83 http://deb.debian.org/debian bullseye/main amd64 golang-github-google-go-cmp-dev all 0.5.4-1 [83.7 kB] Get: 84 http://deb.debian.org/debian bullseye/main amd64 golang-glog-dev all 0.0~git20160126.23def4e-3 [17.3 kB] Get: 85 http://deb.debian.org/debian bullseye/main amd64 golang-golang-x-oauth2-dev all 0.0~git20190604.0f29369-2 [31.9 kB] Get: 86 http://deb.debian.org/debian bullseye/main amd64 golang-google-cloud-compute-metadata-dev all 0.56.0-1 [34.0 kB] Get: 87 http://deb.debian.org/debian bullseye/main amd64 golang-golang-x-oauth2-google-dev all 0.0~git20190604.0f29369-2 [13.2 kB] Get: 88 http://deb.debian.org/debian bullseye/main amd64 golang-google-genproto-dev all 0.0~git20200413.b5235f6-1 [4037 kB] Get: 89 http://deb.debian.org/debian bullseye/main amd64 golang-google-grpc-dev all 1.27.1-1 [478 kB] Get: 90 http://deb.debian.org/debian bullseye/main amd64 golang-github-go-kit-kit-dev all 0.6.0-3 [103 kB] Get: 91 http://deb.debian.org/debian bullseye/main amd64 golang-github-julienschmidt-httprouter-dev all 1.3.0-1 [19.3 kB] Get: 92 http://deb.debian.org/debian bullseye/main amd64 golang-github-jpillora-backoff-dev all 1.0.0-1.1 [3712 B] Get: 93 http://deb.debian.org/debian bullseye/main amd64 golang-github-mwitkow-go-conntrack-dev all 0.0~git20190716.2f06839-2 [14.6 kB] Get: 94 http://deb.debian.org/debian bullseye/main amd64 golang-gopkg-alecthomas-kingpin.v2-dev all 2.2.6-3 [42.8 kB] Get: 95 http://deb.debian.org/debian bullseye/main amd64 golang-gopkg-yaml.v2-dev all 2.4.0-1 [61.3 kB] Get: 96 http://deb.debian.org/debian bullseye/main amd64 golang-protobuf-extensions-dev all 1.0.1-1 [29.6 kB] Get: 97 http://deb.debian.org/debian bullseye/main amd64 golang-github-prometheus-common-dev all 0.15.0-2 [102 kB] Get: 98 http://deb.debian.org/debian bullseye/main amd64 golang-golang-x-sync-dev all 0.0~git20210220.036812b-1 [19.1 kB] Get: 99 http://deb.debian.org/debian bullseye/main amd64 golang-github-prometheus-procfs-dev all 0.3.0-2 [110 kB] Get: 100 http://deb.debian.org/debian bullseye/main amd64 golang-github-prometheus-client-golang-dev all 1.9.0-2 [134 kB] Get: 101 http://deb.debian.org/debian bullseye/main amd64 golang-github-armon-go-metrics-dev all 0.3.4-1 [30.5 kB] Get: 102 http://deb.debian.org/debian bullseye/main amd64 golang-github-creack-pty-dev all 1.1.11-1 [8380 B] Get: 103 http://deb.debian.org/debian bullseye/main amd64 golang-github-vmihailenco-tagparser-dev all 0.1.1-2 [4440 B] Get: 104 http://deb.debian.org/debian bullseye/main amd64 golang-gopkg-vmihailenco-msgpack.v2-dev all 4.3.1-2 [27.5 kB] Get: 105 http://deb.debian.org/debian bullseye/main amd64 golang-gopkg-tomb.v2-dev all 0.0~git20161208.d5d1b58-3 [6840 B] Get: 106 http://deb.debian.org/debian bullseye/main amd64 libsasl2-modules-db amd64 2.1.27+dfsg-2.1+deb11u1 [69.1 kB] Get: 107 http://deb.debian.org/debian bullseye/main amd64 libsasl2-2 amd64 2.1.27+dfsg-2.1+deb11u1 [106 kB] Get: 108 http://deb.debian.org/debian bullseye/main amd64 libsasl2-dev amd64 2.1.27+dfsg-2.1+deb11u1 [281 kB] Get: 109 http://deb.debian.org/debian bullseye/main amd64 golang-github-kr-text-dev all 0.2.0-1 [11.1 kB] Get: 110 http://deb.debian.org/debian bullseye/main amd64 golang-github-kr-pretty-dev all 0.2.1+git20200831.59b4212-1 [14.6 kB] Get: 111 http://deb.debian.org/debian bullseye/main amd64 golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-1 [32.1 kB] Get: 112 http://deb.debian.org/debian bullseye/main amd64 golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB] Get: 113 http://deb.debian.org/debian bullseye/main amd64 golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB] Fetched 96.4 MB in 2s (42.1 MB/s) debconf: delaying package configuration, since apt-utils is not installed Selecting previously unselected package bsdextrautils. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 19707 files and directories currently installed.) Preparing to unpack .../000-bsdextrautils_2.36.1-8+deb11u1_amd64.deb ... Unpacking bsdextrautils (2.36.1-8+deb11u1) ... Selecting previously unselected package libuchardet0:amd64. Preparing to unpack .../001-libuchardet0_0.0.7-1_amd64.deb ... Unpacking libuchardet0:amd64 (0.0.7-1) ... Selecting previously unselected package groff-base. Preparing to unpack .../002-groff-base_1.22.4-6_amd64.deb ... Unpacking groff-base (1.22.4-6) ... Selecting previously unselected package libpipeline1:amd64. Preparing to unpack .../003-libpipeline1_1.5.3-1_amd64.deb ... Unpacking libpipeline1:amd64 (1.5.3-1) ... Selecting previously unselected package man-db. Preparing to unpack .../004-man-db_2.9.4-2_amd64.deb ... Unpacking man-db (2.9.4-2) ... Selecting previously unselected package golang-golang-x-sys-dev. Preparing to unpack .../005-golang-golang-x-sys-dev_0.0~git20210124.22da62e-1_all.deb ... Unpacking golang-golang-x-sys-dev (0.0~git20210124.22da62e-1) ... Selecting previously unselected package golang-github-sirupsen-logrus-dev. Preparing to unpack .../006-golang-github-sirupsen-logrus-dev_1.7.0-2_all.deb ... Unpacking golang-github-sirupsen-logrus-dev (1.7.0-2) ... Selecting previously unselected package sensible-utils. Preparing to unpack .../007-sensible-utils_0.0.14_all.deb ... Unpacking sensible-utils (0.0.14) ... Selecting previously unselected package libmagic-mgc. Preparing to unpack .../008-libmagic-mgc_1%3a5.39-3_amd64.deb ... Unpacking libmagic-mgc (1:5.39-3) ... Selecting previously unselected package libmagic1:amd64. Preparing to unpack .../009-libmagic1_1%3a5.39-3_amd64.deb ... Unpacking libmagic1:amd64 (1:5.39-3) ... Selecting previously unselected package file. Preparing to unpack .../010-file_1%3a5.39-3_amd64.deb ... Unpacking file (1:5.39-3) ... Selecting previously unselected package gettext-base. Preparing to unpack .../011-gettext-base_0.21-4_amd64.deb ... Unpacking gettext-base (0.21-4) ... Selecting previously unselected package libsigsegv2:amd64. Preparing to unpack .../012-libsigsegv2_2.13-1_amd64.deb ... Unpacking libsigsegv2:amd64 (2.13-1) ... Selecting previously unselected package m4. Preparing to unpack .../013-m4_1.4.18-5_amd64.deb ... Unpacking m4 (1.4.18-5) ... Selecting previously unselected package autoconf. Preparing to unpack .../014-autoconf_2.69-14_all.deb ... Unpacking autoconf (2.69-14) ... Selecting previously unselected package autotools-dev. Preparing to unpack .../015-autotools-dev_20180224.1+nmu1_all.deb ... Unpacking autotools-dev (20180224.1+nmu1) ... Selecting previously unselected package automake. Preparing to unpack .../016-automake_1%3a1.16.3-2_all.deb ... Unpacking automake (1:1.16.3-2) ... Selecting previously unselected package autopoint. Preparing to unpack .../017-autopoint_0.21-4_all.deb ... Unpacking autopoint (0.21-4) ... Selecting previously unselected package libdebhelper-perl. Preparing to unpack .../018-libdebhelper-perl_13.3.4_all.deb ... Unpacking libdebhelper-perl (13.3.4) ... Selecting previously unselected package libtool. Preparing to unpack .../019-libtool_2.4.6-15_all.deb ... Unpacking libtool (2.4.6-15) ... Selecting previously unselected package dh-autoreconf. Preparing to unpack .../020-dh-autoreconf_20_all.deb ... Unpacking dh-autoreconf (20) ... Selecting previously unselected package libarchive-zip-perl. Preparing to unpack .../021-libarchive-zip-perl_1.68-1_all.deb ... Unpacking libarchive-zip-perl (1.68-1) ... Selecting previously unselected package libsub-override-perl. Preparing to unpack .../022-libsub-override-perl_0.09-2_all.deb ... Unpacking libsub-override-perl (0.09-2) ... Selecting previously unselected package libfile-stripnondeterminism-perl. Preparing to unpack .../023-libfile-stripnondeterminism-perl_1.12.0-1_all.deb ... Unpacking libfile-stripnondeterminism-perl (1.12.0-1) ... Selecting previously unselected package dh-strip-nondeterminism. Preparing to unpack .../024-dh-strip-nondeterminism_1.12.0-1_all.deb ... Unpacking dh-strip-nondeterminism (1.12.0-1) ... Selecting previously unselected package libelf1:amd64. Preparing to unpack .../025-libelf1_0.183-1_amd64.deb ... Unpacking libelf1:amd64 (0.183-1) ... Selecting previously unselected package dwz. Preparing to unpack .../026-dwz_0.13+20210201-1_amd64.deb ... Unpacking dwz (0.13+20210201-1) ... Selecting previously unselected package libicu67:amd64. Preparing to unpack .../027-libicu67_67.1-7_amd64.deb ... Unpacking libicu67:amd64 (67.1-7) ... Selecting previously unselected package libxml2:amd64. Preparing to unpack .../028-libxml2_2.9.10+dfsg-6.7+deb11u4_amd64.deb ... Unpacking libxml2:amd64 (2.9.10+dfsg-6.7+deb11u4) ... Selecting previously unselected package gettext. Preparing to unpack .../029-gettext_0.21-4_amd64.deb ... Unpacking gettext (0.21-4) ... Selecting previously unselected package intltool-debian. Preparing to unpack .../030-intltool-debian_0.35.0+20060710.5_all.deb ... Unpacking intltool-debian (0.35.0+20060710.5) ... Selecting previously unselected package po-debconf. Preparing to unpack .../031-po-debconf_1.0.21+nmu1_all.deb ... Unpacking po-debconf (1.0.21+nmu1) ... Selecting previously unselected package debhelper. Preparing to unpack .../032-debhelper_13.3.4_all.deb ... Unpacking debhelper (13.3.4) ... Selecting previously unselected package dh-golang. Preparing to unpack .../033-dh-golang_1.51_all.deb ... Unpacking dh-golang (1.51) ... Selecting previously unselected package golang-1.15-src. Preparing to unpack .../034-golang-1.15-src_1.15.15-1~deb11u4_amd64.deb ... Unpacking golang-1.15-src (1.15.15-1~deb11u4) ... Selecting previously unselected package golang-1.15-go. Preparing to unpack .../035-golang-1.15-go_1.15.15-1~deb11u4_amd64.deb ... Unpacking golang-1.15-go (1.15.15-1~deb11u4) ... Selecting previously unselected package golang-src:amd64. Preparing to unpack .../036-golang-src_2%3a1.15~1_amd64.deb ... Unpacking golang-src:amd64 (2:1.15~1) ... Selecting previously unselected package golang-go. Preparing to unpack .../037-golang-go_2%3a1.15~1_amd64.deb ... Unpacking golang-go (2:1.15~1) ... Selecting previously unselected package golang-any:amd64. Preparing to unpack .../038-golang-any_2%3a1.15~1_amd64.deb ... Unpacking golang-any:amd64 (2:1.15~1) ... Selecting previously unselected package golang-github-alecthomas-units-dev. Preparing to unpack .../039-golang-github-alecthomas-units-dev_0.0~git20201120.1786d5e-1_all.deb ... Unpacking golang-github-alecthomas-units-dev (0.0~git20201120.1786d5e-1) ... Selecting previously unselected package golang-github-circonus-labs-circonusllhist-dev. Preparing to unpack .../040-golang-github-circonus-labs-circonusllhist-dev_0.0~git20191022.ec08cde-1_all.deb ... Unpacking golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ... Selecting previously unselected package golang-github-hashicorp-go-cleanhttp-dev. Preparing to unpack .../041-golang-github-hashicorp-go-cleanhttp-dev_0.5.2-1_all.deb ... Unpacking golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ... Selecting previously unselected package golang-github-mattn-go-isatty-dev. Preparing to unpack .../042-golang-github-mattn-go-isatty-dev_0.0.12-1_all.deb ... Unpacking golang-github-mattn-go-isatty-dev (0.0.12-1) ... Selecting previously unselected package golang-github-mattn-go-colorable-dev. Preparing to unpack .../043-golang-github-mattn-go-colorable-dev_0.1.7-1_all.deb ... Unpacking golang-github-mattn-go-colorable-dev (0.1.7-1) ... Selecting previously unselected package golang-github-fatih-color-dev. Preparing to unpack .../044-golang-github-fatih-color-dev_1.7.0-1_all.deb ... Unpacking golang-github-fatih-color-dev (1.7.0-1) ... Selecting previously unselected package golang-github-davecgh-go-spew-dev. Preparing to unpack .../045-golang-github-davecgh-go-spew-dev_1.1.1-2_all.deb ... Unpacking golang-github-davecgh-go-spew-dev (1.1.1-2) ... Selecting previously unselected package golang-github-pmezard-go-difflib-dev. Preparing to unpack .../046-golang-github-pmezard-go-difflib-dev_1.0.0-3_all.deb ... Unpacking golang-github-pmezard-go-difflib-dev (1.0.0-3) ... Selecting previously unselected package golang-github-stretchr-objx-dev. Preparing to unpack .../047-golang-github-stretchr-objx-dev_0.3.0-1_all.deb ... Unpacking golang-github-stretchr-objx-dev (0.3.0-1) ... Selecting previously unselected package golang-gopkg-yaml.v3-dev. Preparing to unpack .../048-golang-gopkg-yaml.v3-dev_3.0.0~git20200121.a6ecf24-3_all.deb ... Unpacking golang-gopkg-yaml.v3-dev (3.0.0~git20200121.a6ecf24-3) ... Selecting previously unselected package golang-github-stretchr-testify-dev. Preparing to unpack .../049-golang-github-stretchr-testify-dev_1.6.1-2_all.deb ... Unpacking golang-github-stretchr-testify-dev (1.6.1-2) ... Selecting previously unselected package golang-github-hashicorp-go-hclog-dev. Preparing to unpack .../050-golang-github-hashicorp-go-hclog-dev_0.11.0-1_all.deb ... Unpacking golang-github-hashicorp-go-hclog-dev (0.11.0-1) ... Selecting previously unselected package golang-github-hashicorp-go-retryablehttp-dev. Preparing to unpack .../051-golang-github-hashicorp-go-retryablehttp-dev_0.6.4-2_all.deb ... Unpacking golang-github-hashicorp-go-retryablehttp-dev (0.6.4-2) ... Selecting previously unselected package golang-github-pkg-errors-dev. Preparing to unpack .../052-golang-github-pkg-errors-dev_0.9.1-1_all.deb ... Unpacking golang-github-pkg-errors-dev (0.9.1-1) ... Selecting previously unselected package golang-github-tv42-httpunix-dev. Preparing to unpack .../053-golang-github-tv42-httpunix-dev_0.0~git20150427.b75d861-2_all.deb ... Unpacking golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-2) ... Selecting previously unselected package golang-github-circonus-labs-circonus-gometrics-dev. Preparing to unpack .../054-golang-github-circonus-labs-circonus-gometrics-dev_2.3.1-3_all.deb ... Unpacking golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-3) ... Selecting previously unselected package golang-github-datadog-datadog-go-dev. Preparing to unpack .../055-golang-github-datadog-datadog-go-dev_2.1.0-3_all.deb ... Unpacking golang-github-datadog-datadog-go-dev (2.1.0-3) ... Selecting previously unselected package golang-github-hashicorp-go-uuid-dev. Preparing to unpack .../056-golang-github-hashicorp-go-uuid-dev_1.0.2-1_all.deb ... Unpacking golang-github-hashicorp-go-uuid-dev (1.0.2-1) ... Selecting previously unselected package golang-github-hashicorp-golang-lru-dev. Preparing to unpack .../057-golang-github-hashicorp-golang-lru-dev_0.5.4-2_all.deb ... Unpacking golang-github-hashicorp-golang-lru-dev (0.5.4-2) ... Selecting previously unselected package golang-github-hashicorp-go-immutable-radix-dev. Preparing to unpack .../058-golang-github-hashicorp-go-immutable-radix-dev_1.3.0-1_all.deb ... Unpacking golang-github-hashicorp-go-immutable-radix-dev (1.3.0-1) ... Selecting previously unselected package golang-github-pascaldekloe-goe-dev. Preparing to unpack .../059-golang-github-pascaldekloe-goe-dev_0.1.0-2_all.deb ... Unpacking golang-github-pascaldekloe-goe-dev (0.1.0-2) ... Selecting previously unselected package golang-github-beorn7-perks-dev. Preparing to unpack .../060-golang-github-beorn7-perks-dev_1.0.1-1_all.deb ... Unpacking golang-github-beorn7-perks-dev (1.0.1-1) ... Selecting previously unselected package golang-github-cespare-xxhash-dev. Preparing to unpack .../061-golang-github-cespare-xxhash-dev_2.1.1-1_all.deb ... Unpacking golang-github-cespare-xxhash-dev (2.1.1-1) ... Selecting previously unselected package golang-github-modern-go-concurrent-dev. Preparing to unpack .../062-golang-github-modern-go-concurrent-dev_1.0.3-1.1_all.deb ... Unpacking golang-github-modern-go-concurrent-dev (1.0.3-1.1) ... Selecting previously unselected package golang-github-modern-go-reflect2-dev. Preparing to unpack .../063-golang-github-modern-go-reflect2-dev_1.0.1-1_all.deb ... Unpacking golang-github-modern-go-reflect2-dev (1.0.1-1) ... Selecting previously unselected package golang-github-json-iterator-go-dev. Preparing to unpack .../064-golang-github-json-iterator-go-dev_1.1.10-2_all.deb ... Unpacking golang-github-json-iterator-go-dev (1.1.10-2) ... Selecting previously unselected package zlib1g-dev:amd64. Preparing to unpack .../065-zlib1g-dev_1%3a1.2.11.dfsg-2+deb11u2_amd64.deb ... Unpacking zlib1g-dev:amd64 (1:1.2.11.dfsg-2+deb11u2) ... Selecting previously unselected package libprotobuf23:amd64. Preparing to unpack .../066-libprotobuf23_3.12.4-1_amd64.deb ... Unpacking libprotobuf23:amd64 (3.12.4-1) ... Selecting previously unselected package libprotobuf-lite23:amd64. Preparing to unpack .../067-libprotobuf-lite23_3.12.4-1_amd64.deb ... Unpacking libprotobuf-lite23:amd64 (3.12.4-1) ... Selecting previously unselected package libprotobuf-dev:amd64. Preparing to unpack .../068-libprotobuf-dev_3.12.4-1_amd64.deb ... Unpacking libprotobuf-dev:amd64 (3.12.4-1) ... Selecting previously unselected package libprotoc23:amd64. Preparing to unpack .../069-libprotoc23_3.12.4-1_amd64.deb ... Unpacking libprotoc23:amd64 (3.12.4-1) ... Selecting previously unselected package protobuf-compiler. Preparing to unpack .../070-protobuf-compiler_3.12.4-1_amd64.deb ... Unpacking protobuf-compiler (3.12.4-1) ... Selecting previously unselected package golang-goprotobuf-dev. Preparing to unpack .../071-golang-goprotobuf-dev_1.3.4-2+b6_amd64.deb ... Unpacking golang-goprotobuf-dev (1.3.4-2+b6) ... Selecting previously unselected package golang-github-prometheus-client-model-dev. Preparing to unpack .../072-golang-github-prometheus-client-model-dev_0.2.0-2_all.deb ... Unpacking golang-github-prometheus-client-model-dev (0.2.0-2) ... Selecting previously unselected package golang-github-dgrijalva-jwt-go-dev. Preparing to unpack .../073-golang-github-dgrijalva-jwt-go-dev_3.2.0-3_all.deb ... Unpacking golang-github-dgrijalva-jwt-go-dev (3.2.0-3) ... Selecting previously unselected package golang-github-go-logfmt-logfmt-dev. Preparing to unpack .../074-golang-github-go-logfmt-logfmt-dev_0.5.0-2_all.deb ... Unpacking golang-github-go-logfmt-logfmt-dev (0.5.0-2) ... Selecting previously unselected package golang-github-go-stack-stack-dev. Preparing to unpack .../075-golang-github-go-stack-stack-dev_1.8.0-1_all.deb ... Unpacking golang-github-go-stack-stack-dev (1.8.0-1) ... Selecting previously unselected package golang-github-opentracing-opentracing-go-dev. Preparing to unpack .../076-golang-github-opentracing-opentracing-go-dev_1.2.0-1_all.deb ... Unpacking golang-github-opentracing-opentracing-go-dev (1.2.0-1) ... Selecting previously unselected package golang-golang-x-text-dev. Preparing to unpack .../077-golang-golang-x-text-dev_0.3.6-1_all.deb ... Unpacking golang-golang-x-text-dev (0.3.6-1) ... Selecting previously unselected package golang-golang-x-net-dev. Preparing to unpack .../078-golang-golang-x-net-dev_1%3a0.0+git20210119.5f4716e+dfsg-4_all.deb ... Unpacking golang-golang-x-net-dev (1:0.0+git20210119.5f4716e+dfsg-4) ... Selecting previously unselected package golang-golang-x-time-dev. Preparing to unpack .../079-golang-golang-x-time-dev_0.0+git20200630.3af7569-1_all.deb ... Unpacking golang-golang-x-time-dev (0.0+git20200630.3af7569-1) ... Selecting previously unselected package golang-github-golang-mock-dev. Preparing to unpack .../080-golang-github-golang-mock-dev_1.3.1-2_all.deb ... Unpacking golang-github-golang-mock-dev (1.3.1-2) ... Selecting previously unselected package golang-golang-x-xerrors-dev. Preparing to unpack .../081-golang-golang-x-xerrors-dev_0.0~git20191204.9bdfabe-1_all.deb ... Unpacking golang-golang-x-xerrors-dev (0.0~git20191204.9bdfabe-1) ... Selecting previously unselected package golang-github-google-go-cmp-dev. Preparing to unpack .../082-golang-github-google-go-cmp-dev_0.5.4-1_all.deb ... Unpacking golang-github-google-go-cmp-dev (0.5.4-1) ... Selecting previously unselected package golang-glog-dev. Preparing to unpack .../083-golang-glog-dev_0.0~git20160126.23def4e-3_all.deb ... Unpacking golang-glog-dev (0.0~git20160126.23def4e-3) ... Selecting previously unselected package golang-golang-x-oauth2-dev. Preparing to unpack .../084-golang-golang-x-oauth2-dev_0.0~git20190604.0f29369-2_all.deb ... Unpacking golang-golang-x-oauth2-dev (0.0~git20190604.0f29369-2) ... Selecting previously unselected package golang-google-cloud-compute-metadata-dev. Preparing to unpack .../085-golang-google-cloud-compute-metadata-dev_0.56.0-1_all.deb ... Unpacking golang-google-cloud-compute-metadata-dev (0.56.0-1) ... Selecting previously unselected package golang-golang-x-oauth2-google-dev. Preparing to unpack .../086-golang-golang-x-oauth2-google-dev_0.0~git20190604.0f29369-2_all.deb ... Unpacking golang-golang-x-oauth2-google-dev (0.0~git20190604.0f29369-2) ... Selecting previously unselected package golang-google-genproto-dev. Preparing to unpack .../087-golang-google-genproto-dev_0.0~git20200413.b5235f6-1_all.deb ... Unpacking golang-google-genproto-dev (0.0~git20200413.b5235f6-1) ... Selecting previously unselected package golang-google-grpc-dev. Preparing to unpack .../088-golang-google-grpc-dev_1.27.1-1_all.deb ... Unpacking golang-google-grpc-dev (1.27.1-1) ... Selecting previously unselected package golang-github-go-kit-kit-dev. Preparing to unpack .../089-golang-github-go-kit-kit-dev_0.6.0-3_all.deb ... Unpacking golang-github-go-kit-kit-dev (0.6.0-3) ... Selecting previously unselected package golang-github-julienschmidt-httprouter-dev. Preparing to unpack .../090-golang-github-julienschmidt-httprouter-dev_1.3.0-1_all.deb ... Unpacking golang-github-julienschmidt-httprouter-dev (1.3.0-1) ... Selecting previously unselected package golang-github-jpillora-backoff-dev. Preparing to unpack .../091-golang-github-jpillora-backoff-dev_1.0.0-1.1_all.deb ... Unpacking golang-github-jpillora-backoff-dev (1.0.0-1.1) ... Selecting previously unselected package golang-github-mwitkow-go-conntrack-dev. Preparing to unpack .../092-golang-github-mwitkow-go-conntrack-dev_0.0~git20190716.2f06839-2_all.deb ... Unpacking golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-2) ... Selecting previously unselected package golang-gopkg-alecthomas-kingpin.v2-dev. Preparing to unpack .../093-golang-gopkg-alecthomas-kingpin.v2-dev_2.2.6-3_all.deb ... Unpacking golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-3) ... Selecting previously unselected package golang-gopkg-yaml.v2-dev. Preparing to unpack .../094-golang-gopkg-yaml.v2-dev_2.4.0-1_all.deb ... Unpacking golang-gopkg-yaml.v2-dev (2.4.0-1) ... Selecting previously unselected package golang-protobuf-extensions-dev. Preparing to unpack .../095-golang-protobuf-extensions-dev_1.0.1-1_all.deb ... Unpacking golang-protobuf-extensions-dev (1.0.1-1) ... Selecting previously unselected package golang-github-prometheus-common-dev. Preparing to unpack .../096-golang-github-prometheus-common-dev_0.15.0-2_all.deb ... Unpacking golang-github-prometheus-common-dev (0.15.0-2) ... Selecting previously unselected package golang-golang-x-sync-dev. Preparing to unpack .../097-golang-golang-x-sync-dev_0.0~git20210220.036812b-1_all.deb ... Unpacking golang-golang-x-sync-dev (0.0~git20210220.036812b-1) ... Selecting previously unselected package golang-github-prometheus-procfs-dev. Preparing to unpack .../098-golang-github-prometheus-procfs-dev_0.3.0-2_all.deb ... Unpacking golang-github-prometheus-procfs-dev (0.3.0-2) ... Selecting previously unselected package golang-github-prometheus-client-golang-dev. Preparing to unpack .../099-golang-github-prometheus-client-golang-dev_1.9.0-2_all.deb ... Unpacking golang-github-prometheus-client-golang-dev (1.9.0-2) ... Selecting previously unselected package golang-github-armon-go-metrics-dev. Preparing to unpack .../100-golang-github-armon-go-metrics-dev_0.3.4-1_all.deb ... Unpacking golang-github-armon-go-metrics-dev (0.3.4-1) ... Selecting previously unselected package golang-github-creack-pty-dev. Preparing to unpack .../101-golang-github-creack-pty-dev_1.1.11-1_all.deb ... Unpacking golang-github-creack-pty-dev (1.1.11-1) ... Selecting previously unselected package golang-github-vmihailenco-tagparser-dev. Preparing to unpack .../102-golang-github-vmihailenco-tagparser-dev_0.1.1-2_all.deb ... Unpacking golang-github-vmihailenco-tagparser-dev (0.1.1-2) ... Selecting previously unselected package golang-gopkg-vmihailenco-msgpack.v2-dev. Preparing to unpack .../103-golang-gopkg-vmihailenco-msgpack.v2-dev_4.3.1-2_all.deb ... Unpacking golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ... Selecting previously unselected package golang-gopkg-tomb.v2-dev. Preparing to unpack .../104-golang-gopkg-tomb.v2-dev_0.0~git20161208.d5d1b58-3_all.deb ... Unpacking golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ... Selecting previously unselected package libsasl2-modules-db:amd64. Preparing to unpack .../105-libsasl2-modules-db_2.1.27+dfsg-2.1+deb11u1_amd64.deb ... Unpacking libsasl2-modules-db:amd64 (2.1.27+dfsg-2.1+deb11u1) ... Selecting previously unselected package libsasl2-2:amd64. Preparing to unpack .../106-libsasl2-2_2.1.27+dfsg-2.1+deb11u1_amd64.deb ... Unpacking libsasl2-2:amd64 (2.1.27+dfsg-2.1+deb11u1) ... Selecting previously unselected package libsasl2-dev. Preparing to unpack .../107-libsasl2-dev_2.1.27+dfsg-2.1+deb11u1_amd64.deb ... Unpacking libsasl2-dev (2.1.27+dfsg-2.1+deb11u1) ... Selecting previously unselected package golang-github-kr-text-dev. Preparing to unpack .../108-golang-github-kr-text-dev_0.2.0-1_all.deb ... Unpacking golang-github-kr-text-dev (0.2.0-1) ... Selecting previously unselected package golang-github-kr-pretty-dev. Preparing to unpack .../109-golang-github-kr-pretty-dev_0.2.1+git20200831.59b4212-1_all.deb ... Unpacking golang-github-kr-pretty-dev (0.2.1+git20200831.59b4212-1) ... Selecting previously unselected package golang-gopkg-check.v1-dev. Preparing to unpack .../110-golang-gopkg-check.v1-dev_0.0+git20200902.038fdea-1_all.deb ... Unpacking golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ... Selecting previously unselected package golang-gopkg-mgo.v2-dev. Preparing to unpack .../111-golang-gopkg-mgo.v2-dev_2016.08.01-7_all.deb ... Unpacking golang-gopkg-mgo.v2-dev (2016.08.01-7) ... Selecting previously unselected package golang-github-hashicorp-go-msgpack-dev. Preparing to unpack .../112-golang-github-hashicorp-go-msgpack-dev_0.5.5-1_all.deb ... Unpacking golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ... Setting up libpipeline1:amd64 (1.5.3-1) ... Setting up golang-gopkg-yaml.v2-dev (2.4.0-1) ... Setting up golang-github-hashicorp-go-uuid-dev (1.0.2-1) ... Setting up bsdextrautils (2.36.1-8+deb11u1) ... update-alternatives: using /usr/bin/write.ul to provide /usr/bin/write (write) in auto mode Setting up golang-github-creack-pty-dev (1.1.11-1) ... Setting up libicu67:amd64 (67.1-7) ... Setting up golang-github-golang-mock-dev (1.3.1-2) ... Setting up golang-github-stretchr-objx-dev (0.3.0-1) ... Setting up libmagic-mgc (1:5.39-3) ... Setting up golang-github-pkg-errors-dev (0.9.1-1) ... Setting up golang-github-hashicorp-golang-lru-dev (0.5.4-2) ... Setting up golang-1.15-src (1.15.15-1~deb11u4) ... Setting up libarchive-zip-perl (1.68-1) ... Setting up golang-gopkg-yaml.v3-dev (3.0.0~git20200121.a6ecf24-3) ... Setting up libdebhelper-perl (13.3.4) ... Setting up golang-golang-x-sys-dev (0.0~git20210124.22da62e-1) ... Setting up libmagic1:amd64 (1:5.39-3) ... Setting up golang-github-pmezard-go-difflib-dev (1.0.0-3) ... Setting up golang-github-modern-go-concurrent-dev (1.0.3-1.1) ... Setting up gettext-base (0.21-4) ... Setting up golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ... Setting up libprotobuf23:amd64 (3.12.4-1) ... Setting up file (1:5.39-3) ... Setting up golang-github-go-stack-stack-dev (1.8.0-1) ... Setting up golang-github-beorn7-perks-dev (1.0.1-1) ... Setting up golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ... Setting up libsasl2-modules-db:amd64 (2.1.27+dfsg-2.1+deb11u1) ... Setting up golang-github-cespare-xxhash-dev (2.1.1-1) ... Setting up golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ... Setting up golang-github-jpillora-backoff-dev (1.0.0-1.1) ... Setting up golang-github-davecgh-go-spew-dev (1.1.1-2) ... Setting up libprotobuf-lite23:amd64 (3.12.4-1) ... Setting up autotools-dev (20180224.1+nmu1) ... Setting up golang-github-pascaldekloe-goe-dev (0.1.0-2) ... Setting up golang-github-go-logfmt-logfmt-dev (0.5.0-2) ... Setting up golang-github-kr-text-dev (0.2.0-1) ... Setting up golang-github-stretchr-testify-dev (1.6.1-2) ... Setting up golang-github-hashicorp-go-immutable-radix-dev (1.3.0-1) ... Setting up libprotoc23:amd64 (3.12.4-1) ... Setting up libsigsegv2:amd64 (2.13-1) ... Setting up golang-golang-x-sync-dev (0.0~git20210220.036812b-1) ... Setting up autopoint (0.21-4) ... Setting up golang-glog-dev (0.0~git20160126.23def4e-3) ... Setting up libsasl2-2:amd64 (2.1.27+dfsg-2.1+deb11u1) ... Setting up golang-github-julienschmidt-httprouter-dev (1.3.0-1) ... Setting up zlib1g-dev:amd64 (1:1.2.11.dfsg-2+deb11u2) ... Setting up golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-2) ... Setting up sensible-utils (0.0.14) ... Setting up golang-golang-x-text-dev (0.3.6-1) ... Setting up libuchardet0:amd64 (0.0.7-1) ... Setting up golang-github-prometheus-procfs-dev (0.3.0-2) ... Setting up golang-github-alecthomas-units-dev (0.0~git20201120.1786d5e-1) ... Setting up golang-github-modern-go-reflect2-dev (1.0.1-1) ... Setting up libsub-override-perl (0.09-2) ... Setting up golang-github-dgrijalva-jwt-go-dev (3.2.0-3) ... Setting up golang-github-datadog-datadog-go-dev (2.1.0-3) ... Setting up golang-golang-x-xerrors-dev (0.0~git20191204.9bdfabe-1) ... Setting up golang-src:amd64 (2:1.15~1) ... Setting up golang-1.15-go (1.15.15-1~deb11u4) ... Setting up libelf1:amd64 (0.183-1) ... Setting up golang-github-vmihailenco-tagparser-dev (0.1.1-2) ... Setting up libxml2:amd64 (2.9.10+dfsg-6.7+deb11u4) ... Setting up golang-github-kr-pretty-dev (0.2.1+git20200831.59b4212-1) ... Setting up libfile-stripnondeterminism-perl (1.12.0-1) ... Setting up golang-github-google-go-cmp-dev (0.5.4-1) ... Setting up golang-github-mattn-go-isatty-dev (0.0.12-1) ... Setting up golang-github-sirupsen-logrus-dev (1.7.0-2) ... Setting up golang-github-opentracing-opentracing-go-dev (1.2.0-1) ... Setting up gettext (0.21-4) ... Setting up protobuf-compiler (3.12.4-1) ... Setting up golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ... Setting up golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-3) ... Setting up libtool (2.4.6-15) ... Setting up golang-go (2:1.15~1) ... Setting up golang-github-mattn-go-colorable-dev (0.1.7-1) ... Setting up m4 (1.4.18-5) ... Setting up golang-github-json-iterator-go-dev (1.1.10-2) ... Setting up golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ... Setting up golang-any:amd64 (2:1.15~1) ... Setting up intltool-debian (0.35.0+20060710.5) ... Setting up libprotobuf-dev:amd64 (3.12.4-1) ... Setting up libsasl2-dev (2.1.27+dfsg-2.1+deb11u1) ... Setting up golang-gopkg-mgo.v2-dev (2016.08.01-7) ... Setting up golang-goprotobuf-dev (1.3.4-2+b6) ... Setting up golang-golang-x-net-dev (1:0.0+git20210119.5f4716e+dfsg-4) ... Setting up autoconf (2.69-14) ... Setting up dh-strip-nondeterminism (1.12.0-1) ... Setting up dwz (0.13+20210201-1) ... Setting up groff-base (1.22.4-6) ... Setting up golang-github-prometheus-client-model-dev (0.2.0-2) ... Setting up golang-golang-x-oauth2-dev (0.0~git20190604.0f29369-2) ... Setting up automake (1:1.16.3-2) ... update-alternatives: using /usr/bin/automake-1.16 to provide /usr/bin/automake (automake) in auto mode Setting up golang-golang-x-time-dev (0.0+git20200630.3af7569-1) ... Setting up golang-github-fatih-color-dev (1.7.0-1) ... Setting up golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ... Setting up po-debconf (1.0.21+nmu1) ... Setting up golang-protobuf-extensions-dev (1.0.1-1) ... Setting up golang-google-cloud-compute-metadata-dev (0.56.0-1) ... Setting up man-db (2.9.4-2) ... Not building database; man-db/auto-update is not 'true'. Setting up dh-autoreconf (20) ... Setting up golang-golang-x-oauth2-google-dev (0.0~git20190604.0f29369-2) ... Setting up golang-github-hashicorp-go-hclog-dev (0.11.0-1) ... Setting up debhelper (13.3.4) ... Setting up golang-github-hashicorp-go-retryablehttp-dev (0.6.4-2) ... Setting up dh-golang (1.51) ... Setting up golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-3) ... Setting up golang-google-genproto-dev (0.0~git20200413.b5235f6-1) ... Setting up golang-github-prometheus-client-golang-dev (1.9.0-2) ... Setting up golang-github-armon-go-metrics-dev (0.3.4-1) ... Setting up golang-google-grpc-dev (1.27.1-1) ... Setting up golang-github-go-kit-kit-dev (0.6.0-3) ... Setting up golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-2) ... Setting up golang-github-prometheus-common-dev (0.15.0-2) ... Processing triggers for libc-bin (2.31-13+deb11u6) ... Reading package lists... Building dependency tree... Reading state information... Reading extended state information... Initializing package states... Writing extended state information... Building tag database... -> Finished parsing the build-deps Reading package lists... Building dependency tree... Reading state information... fakeroot is already the newest version (1.25.3-1.1). 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. I: Building the package I: user script /srv/workspace/pbuilder/2531752/tmp/hooks/A99_set_merged_usr starting Not re-configuring usrmerge for bullseye I: user script /srv/workspace/pbuilder/2531752/tmp/hooks/A99_set_merged_usr finished hostname: Name or service not known I: Running cd /build/golang-github-hashicorp-raft-1.1.2/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-genchanges -S > ../golang-github-hashicorp-raft_1.1.2-1_source.changes dpkg-buildpackage: info: source package golang-github-hashicorp-raft dpkg-buildpackage: info: source version 1.1.2-1 dpkg-buildpackage: info: source distribution unstable dpkg-buildpackage: info: source changed by Dmitry Smirnov dpkg-source --before-build . dpkg-buildpackage: info: host architecture amd64 fakeroot debian/rules clean dh clean --buildsystem=golang --with=golang dh_auto_clean -O--buildsystem=golang dh_autoreconf_clean -O--buildsystem=golang dh_clean -O--buildsystem=golang debian/rules build dh build --buildsystem=golang --with=golang dh_update_autotools_config -O--buildsystem=golang dh_autoreconf -O--buildsystem=golang dh_auto_configure -O--buildsystem=golang dh_auto_build -O--buildsystem=golang cd obj-x86_64-linux-gnu && go install -trimpath -v -p 15 github.com/hashicorp/raft github.com/hashicorp/raft/bench crypto/subtle internal/race internal/unsafeheader encoding container/list math/bits crypto/internal/subtle runtime/internal/sys internal/nettrace sync/atomic vendor/golang.org/x/crypto/cryptobyte/asn1 unicode runtime/internal/atomic unicode/utf8 vendor/golang.org/x/crypto/internal/subtle internal/cpu golang.org/x/sys/internal/unsafeheader unicode/utf16 runtime/cgo runtime/internal/math internal/bytealg internal/testlog math runtime internal/reflectlite sync math/rand internal/singleflight errors sort internal/oserror github.com/hashicorp/golang-lru/simplelru io strconv vendor/golang.org/x/net/dns/dnsmessage syscall hash bytes strings crypto/internal/randutil hash/crc32 crypto/hmac hash/crc64 vendor/golang.org/x/crypto/hkdf crypto crypto/rc4 reflect vendor/golang.org/x/text/transform bufio github.com/hashicorp/go-immutable-radix path regexp/syntax html internal/syscall/execenv internal/syscall/unix time regexp context internal/poll internal/fmtsort encoding/binary os encoding/base64 crypto/cipher crypto/sha512 crypto/md5 crypto/ed25519/internal/edwards25519 vendor/golang.org/x/crypto/poly1305 crypto/sha256 crypto/sha1 golang.org/x/sys/unix encoding/pem fmt path/filepath runtime/debug os/signal net crypto/des crypto/aes vendor/golang.org/x/crypto/chacha20 io/ioutil vendor/golang.org/x/sys/cpu vendor/golang.org/x/crypto/chacha20poly1305 log encoding/json compress/flate mime net/url vendor/golang.org/x/crypto/curve25519 encoding/hex mime/quotedprintable math/big vendor/golang.org/x/net/http2/hpack net/http/internal vendor/golang.org/x/text/unicode/norm vendor/golang.org/x/text/unicode/bidi encoding/gob go/token text/template/parse flag runtime/trace github.com/mattn/go-isatty github.com/mattn/go-colorable github.com/fatih/color compress/gzip testing vendor/golang.org/x/text/secure/bidirule github.com/hashicorp/go-hclog text/template vendor/golang.org/x/net/idna crypto/rand crypto/dsa encoding/asn1 crypto/elliptic crypto/rsa crypto/ed25519 crypto/x509/pkix vendor/golang.org/x/crypto/cryptobyte html/template crypto/ecdsa vendor/golang.org/x/net/http/httpproxy net/textproto crypto/x509 vendor/golang.org/x/net/http/httpguts mime/multipart crypto/tls net/http/httptrace net/http github.com/armon/go-metrics net/rpc github.com/hashicorp/go-msgpack/codec github.com/hashicorp/raft github.com/hashicorp/raft/bench dh_auto_test -O--buildsystem=golang cd obj-x86_64-linux-gnu && go test -vet=off -v -p 15 github.com/hashicorp/raft github.com/hashicorp/raft/bench === RUN TestCommitment_setVoters --- PASS: TestCommitment_setVoters (0.00s) === RUN TestCommitment_match_max --- PASS: TestCommitment_match_max (0.00s) === RUN TestCommitment_match_nonVoting --- PASS: TestCommitment_match_nonVoting (0.00s) === RUN TestCommitment_recalculate --- PASS: TestCommitment_recalculate (0.00s) === RUN TestCommitment_recalculate_startIndex --- PASS: TestCommitment_recalculate_startIndex (0.00s) === RUN TestCommitment_noVoterSanity --- PASS: TestCommitment_noVoterSanity (0.00s) === RUN TestCommitment_singleVoter --- PASS: TestCommitment_singleVoter (0.00s) === RUN TestConfiguration_Configuration_Clone --- PASS: TestConfiguration_Configuration_Clone (0.00s) === RUN TestConfiguration_configurations_Clone --- PASS: TestConfiguration_configurations_Clone (0.00s) === RUN TestConfiguration_hasVote --- PASS: TestConfiguration_hasVote (0.00s) === RUN TestConfiguration_checkConfiguration --- PASS: TestConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_nextConfiguration_table --- PASS: TestConfiguration_nextConfiguration_table (0.00s) === RUN TestConfiguration_nextConfiguration_prevIndex --- PASS: TestConfiguration_nextConfiguration_prevIndex (0.00s) === RUN TestConfiguration_nextConfiguration_checkConfiguration --- PASS: TestConfiguration_nextConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_encodeDecodePeers --- PASS: TestConfiguration_encodeDecodePeers (0.00s) === RUN TestConfiguration_encodeDecodeConfiguration --- PASS: TestConfiguration_encodeDecodeConfiguration (0.00s) === RUN TestDiscardSnapshotStoreImpl --- PASS: TestDiscardSnapshotStoreImpl (0.00s) === RUN TestDiscardSnapshotSinkImpl --- PASS: TestDiscardSnapshotSinkImpl (0.00s) === RUN TestFileSnapshotStoreImpl --- PASS: TestFileSnapshotStoreImpl (0.00s) === RUN TestFileSnapshotSinkImpl --- PASS: TestFileSnapshotSinkImpl (0.00s) === RUN TestFileSS_CreateSnapshotMissingParentDir testing.go:153: 2023-08-02T15:12:47.909Z [INFO] creating new snapshot: path=/tmp/raft672931563/raft341413710/snapshots/3-10-1690989167909.tmp --- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.00s) === RUN TestFileSS_CreateSnapshot testing.go:153: 2023-08-02T15:12:47.910Z [INFO] creating new snapshot: path=/tmp/raft985338965/snapshots/3-10-1690989167910.tmp testing.go:153: 2023-08-02T15:12:47.910Z [WARN] found temporary snapshot: name=3-10-1690989167910.tmp --- PASS: TestFileSS_CreateSnapshot (0.00s) === RUN TestFileSS_CancelSnapshot testing.go:153: 2023-08-02T15:12:47.911Z [INFO] creating new snapshot: path=/tmp/raft193297328/snapshots/3-10-1690989167911.tmp --- PASS: TestFileSS_CancelSnapshot (0.00s) === RUN TestFileSS_Retention testing.go:153: 2023-08-02T15:12:47.911Z [INFO] creating new snapshot: path=/tmp/raft572282703/snapshots/3-10-1690989167911.tmp testing.go:153: 2023-08-02T15:12:47.911Z [INFO] creating new snapshot: path=/tmp/raft572282703/snapshots/3-11-1690989167911.tmp testing.go:153: 2023-08-02T15:12:47.912Z [INFO] creating new snapshot: path=/tmp/raft572282703/snapshots/3-12-1690989167912.tmp testing.go:153: 2023-08-02T15:12:47.912Z [INFO] reaping snapshot: path=/tmp/raft572282703/snapshots/3-10-1690989167911 testing.go:153: 2023-08-02T15:12:47.912Z [INFO] creating new snapshot: path=/tmp/raft572282703/snapshots/3-13-1690989167912.tmp testing.go:153: 2023-08-02T15:12:47.912Z [INFO] reaping snapshot: path=/tmp/raft572282703/snapshots/3-11-1690989167911 testing.go:153: 2023-08-02T15:12:47.912Z [INFO] creating new snapshot: path=/tmp/raft572282703/snapshots/3-14-1690989167912.tmp testing.go:153: 2023-08-02T15:12:47.913Z [INFO] reaping snapshot: path=/tmp/raft572282703/snapshots/3-12-1690989167912 --- PASS: TestFileSS_Retention (0.00s) === RUN TestFileSS_BadPerm --- PASS: TestFileSS_BadPerm (0.00s) === RUN TestFileSS_MissingParentDir --- PASS: TestFileSS_MissingParentDir (0.00s) === RUN TestFileSS_Ordering testing.go:153: 2023-08-02T15:12:47.914Z [INFO] creating new snapshot: path=/tmp/raft600400566/snapshots/5-130350-1690989167914.tmp testing.go:153: 2023-08-02T15:12:47.915Z [INFO] creating new snapshot: path=/tmp/raft600400566/snapshots/36-204917-1690989167915.tmp --- PASS: TestFileSS_Ordering (0.00s) === RUN TestDeferFutureSuccess --- PASS: TestDeferFutureSuccess (0.00s) === RUN TestDeferFutureError --- PASS: TestDeferFutureError (0.00s) === RUN TestDeferFutureConcurrent --- PASS: TestDeferFutureConcurrent (0.00s) === RUN TestInmemSnapshotStoreImpl --- PASS: TestInmemSnapshotStoreImpl (0.00s) === RUN TestInmemSnapshotSinkImpl --- PASS: TestInmemSnapshotSinkImpl (0.00s) === RUN TestInmemSS_CreateSnapshot --- PASS: TestInmemSS_CreateSnapshot (0.00s) === RUN TestInmemSS_OpenSnapshotTwice --- PASS: TestInmemSS_OpenSnapshotTwice (0.00s) === RUN TestInmemTransportImpl --- PASS: TestInmemTransportImpl (0.00s) === RUN TestInmemTransportWriteTimeout --- PASS: TestInmemTransportWriteTimeout (0.01s) === RUN TestRaft_Integ --- SKIP: TestRaft_Integ (0.00s) === RUN TestLogCache --- PASS: TestLogCache (0.00s) === RUN TestNetworkTransport_CloseStreams --- PASS: TestNetworkTransport_CloseStreams (0.00s) === RUN TestNetworkTransport_StartStop --- PASS: TestNetworkTransport_StartStop (0.00s) === RUN TestNetworkTransport_Heartbeat_FastPath --- PASS: TestNetworkTransport_Heartbeat_FastPath (0.00s) === RUN TestNetworkTransport_AppendEntries --- PASS: TestNetworkTransport_AppendEntries (0.00s) === RUN TestNetworkTransport_AppendEntriesPipeline --- PASS: TestNetworkTransport_AppendEntriesPipeline (0.00s) === RUN TestNetworkTransport_AppendEntriesPipeline_CloseStreams net_transport_test.go:339: Disabled unreliable test --- SKIP: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.00s) === RUN TestNetworkTransport_RequestVote --- PASS: TestNetworkTransport_RequestVote (0.00s) === RUN TestNetworkTransport_InstallSnapshot --- PASS: TestNetworkTransport_InstallSnapshot (0.00s) === RUN TestNetworkTransport_EncodeDecode --- PASS: TestNetworkTransport_EncodeDecode (0.00s) === RUN TestNetworkTransport_EncodeDecode_AddressProvider --- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s) === RUN TestNetworkTransport_PooledConn --- PASS: TestNetworkTransport_PooledConn (0.00s) === RUN TestNetworkTransport_ListenBackoff net_transport_test.go:718: countingWriter: 2023-08-02T15:12:47.939Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2023-08-02T15:12:47.945Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2023-08-02T15:12:47.955Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2023-08-02T15:12:47.976Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2023-08-02T15:12:48.016Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2023-08-02T15:12:48.097Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2023-08-02T15:12:48.257Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2023-08-02T15:12:48.577Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2023-08-02T15:12:49.218Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2023-08-02T15:12:50.219Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2023-08-02T15:12:51.219Z [ERROR] test: failed to accept connection: error="intentional error in test" --- PASS: TestNetworkTransport_ListenBackoff (4.00s) === RUN TestPeersJSON_BadConfiguration --- PASS: TestPeersJSON_BadConfiguration (0.00s) === RUN TestPeersJSON_ReadPeersJSON --- PASS: TestPeersJSON_ReadPeersJSON (0.00s) === RUN TestPeersJSON_ReadConfigJSON --- PASS: TestPeersJSON_ReadConfigJSON (0.00s) === RUN TestRaft_StartStop testLoggerAdapter verbose: server-69f27bdd-0b4f-2b12-601b-dc92a8ad8e42: 2023-08-02T15:12:51.948Z [INFO] server-69f27bdd-0b4f-2b12-601b-dc92a8ad8e42: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-69f27bdd-0b4f-2b12-601b-dc92a8ad8e42 Address:69f27bdd-0b4f-2b12-601b-dc92a8ad8e42}]" testing.go:149: server-69f27bdd-0b4f-2b12-601b-dc92a8ad8e42: 2023-08-02T15:12:51.948Z [INFO] server-69f27bdd-0b4f-2b12-601b-dc92a8ad8e42: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-69f27bdd-0b4f-2b12-601b-dc92a8ad8e42 Address:69f27bdd-0b4f-2b12-601b-dc92a8ad8e42}]" testLoggerAdapter verbose: server-69f27bdd-0b4f-2b12-601b-dc92a8ad8e42: 2023-08-02T15:12:51.948Z [INFO] server-69f27bdd-0b4f-2b12-601b-dc92a8ad8e42: entering follower state: follower="Node at 69f27bdd-0b4f-2b12-601b-dc92a8ad8e42 [Shutdown]" leader= testing.go:149: server-69f27bdd-0b4f-2b12-601b-dc92a8ad8e42: 2023-08-02T15:12:51.948Z [INFO] server-69f27bdd-0b4f-2b12-601b-dc92a8ad8e42: entering follower state: follower="Node at 69f27bdd-0b4f-2b12-601b-dc92a8ad8e42 [Shutdown]" leader= --- PASS: TestRaft_StartStop (0.00s) === RUN TestRaft_AfterShutdown testLoggerAdapter verbose: server-0ffb1b59-0940-86ce-355e-ea09372129af: 2023-08-02T15:12:51.950Z [INFO] server-0ffb1b59-0940-86ce-355e-ea09372129af: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0ffb1b59-0940-86ce-355e-ea09372129af Address:0ffb1b59-0940-86ce-355e-ea09372129af}]" testing.go:149: server-0ffb1b59-0940-86ce-355e-ea09372129af: 2023-08-02T15:12:51.950Z [INFO] server-0ffb1b59-0940-86ce-355e-ea09372129af: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0ffb1b59-0940-86ce-355e-ea09372129af Address:0ffb1b59-0940-86ce-355e-ea09372129af}]" --- PASS: TestRaft_AfterShutdown (0.00s) === RUN TestRaft_LiveBootstrap testLoggerAdapter verbose: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:51.950Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: initial configuration: index=0 servers=[] testing.go:149: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:51.950Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-5717c896-6e56-63cc-1905-f457616df16e: 2023-08-02T15:12:51.950Z [INFO] server-5717c896-6e56-63cc-1905-f457616df16e: initial configuration: index=0 servers=[] testing.go:149: server-5717c896-6e56-63cc-1905-f457616df16e: 2023-08-02T15:12:51.950Z [INFO] server-5717c896-6e56-63cc-1905-f457616df16e: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-66fa6e33-5c99-450c-62f9-714c39011593: 2023-08-02T15:12:51.951Z [INFO] server-66fa6e33-5c99-450c-62f9-714c39011593: initial configuration: index=0 servers=[] testing.go:149: server-66fa6e33-5c99-450c-62f9-714c39011593: 2023-08-02T15:12:51.951Z [INFO] server-66fa6e33-5c99-450c-62f9-714c39011593: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:51.951Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: entering follower state: follower="Node at a53a533d-11bd-a760-3863-f25981d83c5b [Follower]" leader= testing.go:149: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:51.951Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: entering follower state: follower="Node at a53a533d-11bd-a760-3863-f25981d83c5b [Follower]" leader= testLoggerAdapter verbose: cluster: 2023-08-02T15:12:51.951Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-08-02T15:12:51.951Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-5717c896-6e56-63cc-1905-f457616df16e: 2023-08-02T15:12:51.951Z [INFO] server-5717c896-6e56-63cc-1905-f457616df16e: entering follower state: follower="Node at 5717c896-6e56-63cc-1905-f457616df16e [Follower]" leader= testing.go:149: server-5717c896-6e56-63cc-1905-f457616df16e: 2023-08-02T15:12:51.951Z [INFO] server-5717c896-6e56-63cc-1905-f457616df16e: entering follower state: follower="Node at 5717c896-6e56-63cc-1905-f457616df16e [Follower]" leader= testLoggerAdapter verbose: server-66fa6e33-5c99-450c-62f9-714c39011593: 2023-08-02T15:12:51.951Z [INFO] server-66fa6e33-5c99-450c-62f9-714c39011593: entering follower state: follower="Node at 66fa6e33-5c99-450c-62f9-714c39011593 [Follower]" leader= testing.go:149: server-66fa6e33-5c99-450c-62f9-714c39011593: 2023-08-02T15:12:51.951Z [INFO] server-66fa6e33-5c99-450c-62f9-714c39011593: entering follower state: follower="Node at 66fa6e33-5c99-450c-62f9-714c39011593 [Follower]" leader= testLoggerAdapter verbose: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.024Z [WARN] server-a53a533d-11bd-a760-3863-f25981d83c5b: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.024Z [WARN] server-a53a533d-11bd-a760-3863-f25981d83c5b: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.024Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: entering candidate state: node="Node at a53a533d-11bd-a760-3863-f25981d83c5b [Candidate]" term=2 testing.go:149: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.024Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: entering candidate state: node="Node at a53a533d-11bd-a760-3863-f25981d83c5b [Candidate]" term=2 testLoggerAdapter verbose: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.025Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: election won: tally=2 testing.go:149: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.025Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: election won: tally=2 testLoggerAdapter verbose: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.025Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: entering leader state: leader="Node at a53a533d-11bd-a760-3863-f25981d83c5b [Leader]" testing.go:149: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.025Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: entering leader state: leader="Node at a53a533d-11bd-a760-3863-f25981d83c5b [Leader]" testLoggerAdapter verbose: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.025Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: added peer, starting replication: peer=server-5717c896-6e56-63cc-1905-f457616df16e testing.go:149: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.025Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: added peer, starting replication: peer=server-5717c896-6e56-63cc-1905-f457616df16e testLoggerAdapter verbose: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.025Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: added peer, starting replication: peer=server-66fa6e33-5c99-450c-62f9-714c39011593 testing.go:149: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.025Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: added peer, starting replication: peer=server-66fa6e33-5c99-450c-62f9-714c39011593 testLoggerAdapter verbose: server-66fa6e33-5c99-450c-62f9-714c39011593: 2023-08-02T15:12:52.025Z [WARN] server-66fa6e33-5c99-450c-62f9-714c39011593: failed to get previous log: previous-index=1 last-index=0 error="log not found" testLoggerAdapter verbose: server-5717c896-6e56-63cc-1905-f457616df16e: 2023-08-02T15:12:52.025Z [WARN] server-5717c896-6e56-63cc-1905-f457616df16e: failed to get previous log: previous-index=1 last-index=0 error="log not found" testing.go:149: server-66fa6e33-5c99-450c-62f9-714c39011593: 2023-08-02T15:12:52.025Z [WARN] server-66fa6e33-5c99-450c-62f9-714c39011593: failed to get previous log: previous-index=1 last-index=0 error="log not found" testing.go:149: server-5717c896-6e56-63cc-1905-f457616df16e: 2023-08-02T15:12:52.025Z [WARN] server-5717c896-6e56-63cc-1905-f457616df16e: failed to get previous log: previous-index=1 last-index=0 error="log not found" testLoggerAdapter verbose: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.025Z [WARN] server-a53a533d-11bd-a760-3863-f25981d83c5b: appendEntries rejected, sending older logs: peer="{Voter server-66fa6e33-5c99-450c-62f9-714c39011593 66fa6e33-5c99-450c-62f9-714c39011593}" next=1 testing.go:149: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.025Z [WARN] server-a53a533d-11bd-a760-3863-f25981d83c5b: appendEntries rejected, sending older logs: peer="{Voter server-66fa6e33-5c99-450c-62f9-714c39011593 66fa6e33-5c99-450c-62f9-714c39011593}" next=1 testLoggerAdapter verbose: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.025Z [WARN] server-a53a533d-11bd-a760-3863-f25981d83c5b: appendEntries rejected, sending older logs: peer="{Voter server-5717c896-6e56-63cc-1905-f457616df16e 5717c896-6e56-63cc-1905-f457616df16e}" next=1 testing.go:149: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.025Z [WARN] server-a53a533d-11bd-a760-3863-f25981d83c5b: appendEntries rejected, sending older logs: peer="{Voter server-5717c896-6e56-63cc-1905-f457616df16e 5717c896-6e56-63cc-1905-f457616df16e}" next=1 testLoggerAdapter verbose: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.025Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: pipelining replication: peer="{Voter server-66fa6e33-5c99-450c-62f9-714c39011593 66fa6e33-5c99-450c-62f9-714c39011593}" testing.go:149: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.025Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: pipelining replication: peer="{Voter server-66fa6e33-5c99-450c-62f9-714c39011593 66fa6e33-5c99-450c-62f9-714c39011593}" testLoggerAdapter verbose: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.025Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: pipelining replication: peer="{Voter server-5717c896-6e56-63cc-1905-f457616df16e 5717c896-6e56-63cc-1905-f457616df16e}" testing.go:149: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.025Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: pipelining replication: peer="{Voter server-5717c896-6e56-63cc-1905-f457616df16e 5717c896-6e56-63cc-1905-f457616df16e}" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:52.130Z [INFO] stable state for Follower reached at 2023-08-02 15:12:52.025283967 +0000 UTC m=+4.121220312 (2 nodes), 74.057405ms from start of poll, 74.475155ms from cluster start. Timeout at 2023-08-02 15:12:52.130663787 +0000 UTC m=+4.226600141, 105.379829ms after stability testing.go:149: cluster: 2023-08-02T15:12:52.130Z [INFO] stable state for Follower reached at 2023-08-02 15:12:52.025283967 +0000 UTC m=+4.121220312 (2 nodes), 74.057405ms from start of poll, 74.475155ms from cluster start. Timeout at 2023-08-02 15:12:52.130663787 +0000 UTC m=+4.226600141, 105.379829ms after stability testLoggerAdapter verbose: cluster: 2023-08-02T15:12:52.130Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:12:52.130Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-08-02T15:12:52.236Z [INFO] stable state for Leader reached at 2023-08-02 15:12:52.130885617 +0000 UTC m=+4.226821942 (1 nodes), 4.619µs from start of poll, 180.076785ms from cluster start. Timeout at 2023-08-02 15:12:52.236173857 +0000 UTC m=+4.332110194, 105.288252ms after stability testing.go:149: cluster: 2023-08-02T15:12:52.236Z [INFO] stable state for Leader reached at 2023-08-02 15:12:52.130885617 +0000 UTC m=+4.226821942 (1 nodes), 4.619µs from start of poll, 180.076785ms from cluster start. Timeout at 2023-08-02 15:12:52.236173857 +0000 UTC m=+4.332110194, 105.288252ms after stability testLoggerAdapter verbose: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.246Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: aborting pipeline replication: peer="{Voter server-66fa6e33-5c99-450c-62f9-714c39011593 66fa6e33-5c99-450c-62f9-714c39011593}" testing.go:149: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.246Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: aborting pipeline replication: peer="{Voter server-66fa6e33-5c99-450c-62f9-714c39011593 66fa6e33-5c99-450c-62f9-714c39011593}" testLoggerAdapter verbose: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.246Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: aborting pipeline replication: peer="{Voter server-5717c896-6e56-63cc-1905-f457616df16e 5717c896-6e56-63cc-1905-f457616df16e}" testing.go:149: server-a53a533d-11bd-a760-3863-f25981d83c5b: 2023-08-02T15:12:52.246Z [INFO] server-a53a533d-11bd-a760-3863-f25981d83c5b: aborting pipeline replication: peer="{Voter server-5717c896-6e56-63cc-1905-f457616df16e 5717c896-6e56-63cc-1905-f457616df16e}" --- PASS: TestRaft_LiveBootstrap (0.30s) === RUN TestRaft_RecoverCluster_NoState testLoggerAdapter verbose: server-d6e77d3f-4bb6-c31b-3140-27b729478fcf: 2023-08-02T15:12:52.247Z [INFO] server-d6e77d3f-4bb6-c31b-3140-27b729478fcf: initial configuration: index=0 servers=[] testing.go:149: server-d6e77d3f-4bb6-c31b-3140-27b729478fcf: 2023-08-02T15:12:52.247Z [INFO] server-d6e77d3f-4bb6-c31b-3140-27b729478fcf: initial configuration: index=0 servers=[] --- PASS: TestRaft_RecoverCluster_NoState (0.00s) === RUN TestRaft_RecoverCluster raft_test.go:119: Disabled unreliable test --- SKIP: TestRaft_RecoverCluster (0.00s) === RUN TestRaft_HasExistingState testLoggerAdapter verbose: server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9: 2023-08-02T15:12:52.248Z [INFO] server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9 Address:14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9} {Suffrage:Voter ID:server-2327d8ee-561b-c943-ccda-380026d867c5 Address:2327d8ee-561b-c943-ccda-380026d867c5}]" testing.go:149: server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9: 2023-08-02T15:12:52.248Z [INFO] server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9 Address:14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9} {Suffrage:Voter ID:server-2327d8ee-561b-c943-ccda-380026d867c5 Address:2327d8ee-561b-c943-ccda-380026d867c5}]" testLoggerAdapter verbose: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.248Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9 Address:14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9} {Suffrage:Voter ID:server-2327d8ee-561b-c943-ccda-380026d867c5 Address:2327d8ee-561b-c943-ccda-380026d867c5}]" testing.go:149: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.248Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9 Address:14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9} {Suffrage:Voter ID:server-2327d8ee-561b-c943-ccda-380026d867c5 Address:2327d8ee-561b-c943-ccda-380026d867c5}]" testLoggerAdapter verbose: server-5017f136-cbac-923b-fb86-7e0d01c79fcf: 2023-08-02T15:12:52.248Z [INFO] server-5017f136-cbac-923b-fb86-7e0d01c79fcf: initial configuration: index=0 servers=[] testing.go:149: server-5017f136-cbac-923b-fb86-7e0d01c79fcf: 2023-08-02T15:12:52.248Z [INFO] server-5017f136-cbac-923b-fb86-7e0d01c79fcf: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2023-08-02T15:12:52.248Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:12:52.248Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9: 2023-08-02T15:12:52.248Z [INFO] server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9: entering follower state: follower="Node at 14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9 [Follower]" leader= testing.go:149: server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9: 2023-08-02T15:12:52.248Z [INFO] server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9: entering follower state: follower="Node at 14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9 [Follower]" leader= testLoggerAdapter verbose: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.248Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: entering follower state: follower="Node at 2327d8ee-561b-c943-ccda-380026d867c5 [Follower]" leader= testing.go:149: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.248Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: entering follower state: follower="Node at 2327d8ee-561b-c943-ccda-380026d867c5 [Follower]" leader= testLoggerAdapter verbose: server-5017f136-cbac-923b-fb86-7e0d01c79fcf: 2023-08-02T15:12:52.248Z [INFO] server-5017f136-cbac-923b-fb86-7e0d01c79fcf: entering follower state: follower="Node at 5017f136-cbac-923b-fb86-7e0d01c79fcf [Follower]" leader= testing.go:149: server-5017f136-cbac-923b-fb86-7e0d01c79fcf: 2023-08-02T15:12:52.248Z [INFO] server-5017f136-cbac-923b-fb86-7e0d01c79fcf: entering follower state: follower="Node at 5017f136-cbac-923b-fb86-7e0d01c79fcf [Follower]" leader= testLoggerAdapter verbose: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.318Z [WARN] server-2327d8ee-561b-c943-ccda-380026d867c5: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.318Z [WARN] server-2327d8ee-561b-c943-ccda-380026d867c5: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.318Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: entering candidate state: node="Node at 2327d8ee-561b-c943-ccda-380026d867c5 [Candidate]" term=2 testing.go:149: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.318Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: entering candidate state: node="Node at 2327d8ee-561b-c943-ccda-380026d867c5 [Candidate]" term=2 testLoggerAdapter verbose: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.318Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: election won: tally=2 testing.go:149: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.318Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: election won: tally=2 testLoggerAdapter verbose: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.318Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: entering leader state: leader="Node at 2327d8ee-561b-c943-ccda-380026d867c5 [Leader]" testing.go:149: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.318Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: entering leader state: leader="Node at 2327d8ee-561b-c943-ccda-380026d867c5 [Leader]" testLoggerAdapter verbose: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.318Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: added peer, starting replication: peer=server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9 testing.go:149: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.318Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: added peer, starting replication: peer=server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9 testLoggerAdapter verbose: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.319Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: pipelining replication: peer="{Voter server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9 14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9}" testing.go:149: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.319Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: pipelining replication: peer="{Voter server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9 14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9}" testLoggerAdapter verbose: server-5017f136-cbac-923b-fb86-7e0d01c79fcf: 2023-08-02T15:12:52.346Z [WARN] server-5017f136-cbac-923b-fb86-7e0d01c79fcf: no known peers, aborting election testing.go:149: server-5017f136-cbac-923b-fb86-7e0d01c79fcf: 2023-08-02T15:12:52.346Z [WARN] server-5017f136-cbac-923b-fb86-7e0d01c79fcf: no known peers, aborting election testLoggerAdapter verbose: cluster: 2023-08-02T15:12:52.424Z [INFO] stable state for Leader reached at 2023-08-02 15:12:52.319127734 +0000 UTC m=+4.415064061 (1 nodes), 70.401473ms from start of poll, 70.93425ms from cluster start. Timeout at 2023-08-02 15:12:52.424341428 +0000 UTC m=+4.520277769, 105.213708ms after stability testing.go:149: cluster: 2023-08-02T15:12:52.424Z [INFO] stable state for Leader reached at 2023-08-02 15:12:52.319127734 +0000 UTC m=+4.415064061 (1 nodes), 70.401473ms from start of poll, 70.93425ms from cluster start. Timeout at 2023-08-02 15:12:52.424341428 +0000 UTC m=+4.520277769, 105.213708ms after stability testLoggerAdapter verbose: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.424Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: updating configuration: command=AddStaging server-id=server-5017f136-cbac-923b-fb86-7e0d01c79fcf server-addr=5017f136-cbac-923b-fb86-7e0d01c79fcf servers="[{Suffrage:Voter ID:server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9 Address:14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9} {Suffrage:Voter ID:server-2327d8ee-561b-c943-ccda-380026d867c5 Address:2327d8ee-561b-c943-ccda-380026d867c5} {Suffrage:Voter ID:server-5017f136-cbac-923b-fb86-7e0d01c79fcf Address:5017f136-cbac-923b-fb86-7e0d01c79fcf}]" testing.go:149: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.424Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: updating configuration: command=AddStaging server-id=server-5017f136-cbac-923b-fb86-7e0d01c79fcf server-addr=5017f136-cbac-923b-fb86-7e0d01c79fcf servers="[{Suffrage:Voter ID:server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9 Address:14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9} {Suffrage:Voter ID:server-2327d8ee-561b-c943-ccda-380026d867c5 Address:2327d8ee-561b-c943-ccda-380026d867c5} {Suffrage:Voter ID:server-5017f136-cbac-923b-fb86-7e0d01c79fcf Address:5017f136-cbac-923b-fb86-7e0d01c79fcf}]" testLoggerAdapter verbose: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.424Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: added peer, starting replication: peer=server-5017f136-cbac-923b-fb86-7e0d01c79fcf testing.go:149: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.424Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: added peer, starting replication: peer=server-5017f136-cbac-923b-fb86-7e0d01c79fcf testLoggerAdapter verbose: server-5017f136-cbac-923b-fb86-7e0d01c79fcf: 2023-08-02T15:12:52.424Z [WARN] server-5017f136-cbac-923b-fb86-7e0d01c79fcf: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: server-5017f136-cbac-923b-fb86-7e0d01c79fcf: 2023-08-02T15:12:52.424Z [WARN] server-5017f136-cbac-923b-fb86-7e0d01c79fcf: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.424Z [WARN] server-2327d8ee-561b-c943-ccda-380026d867c5: appendEntries rejected, sending older logs: peer="{Voter server-5017f136-cbac-923b-fb86-7e0d01c79fcf 5017f136-cbac-923b-fb86-7e0d01c79fcf}" next=1 testing.go:149: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.424Z [WARN] server-2327d8ee-561b-c943-ccda-380026d867c5: appendEntries rejected, sending older logs: peer="{Voter server-5017f136-cbac-923b-fb86-7e0d01c79fcf 5017f136-cbac-923b-fb86-7e0d01c79fcf}" next=1 testLoggerAdapter verbose: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.424Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: pipelining replication: peer="{Voter server-5017f136-cbac-923b-fb86-7e0d01c79fcf 5017f136-cbac-923b-fb86-7e0d01c79fcf}" testing.go:149: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.424Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: pipelining replication: peer="{Voter server-5017f136-cbac-923b-fb86-7e0d01c79fcf 5017f136-cbac-923b-fb86-7e0d01c79fcf}" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:52.425Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:12:52.425Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-08-02T15:12:52.530Z [INFO] stable state for Leader reached at 2023-08-02 15:12:52.425102363 +0000 UTC m=+4.521038688 (1 nodes), 1.015µs from start of poll, 176.908877ms from cluster start. Timeout at 2023-08-02 15:12:52.530191388 +0000 UTC m=+4.626127726, 105.089038ms after stability testing.go:149: cluster: 2023-08-02T15:12:52.530Z [INFO] stable state for Leader reached at 2023-08-02 15:12:52.425102363 +0000 UTC m=+4.521038688 (1 nodes), 1.015µs from start of poll, 176.908877ms from cluster start. Timeout at 2023-08-02 15:12:52.530191388 +0000 UTC m=+4.626127726, 105.089038ms after stability testLoggerAdapter verbose: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.530Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: aborting pipeline replication: peer="{Voter server-5017f136-cbac-923b-fb86-7e0d01c79fcf 5017f136-cbac-923b-fb86-7e0d01c79fcf}" testing.go:149: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.530Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: aborting pipeline replication: peer="{Voter server-5017f136-cbac-923b-fb86-7e0d01c79fcf 5017f136-cbac-923b-fb86-7e0d01c79fcf}" testLoggerAdapter verbose: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.530Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: aborting pipeline replication: peer="{Voter server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9 14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9}" testing.go:149: server-2327d8ee-561b-c943-ccda-380026d867c5: 2023-08-02T15:12:52.530Z [INFO] server-2327d8ee-561b-c943-ccda-380026d867c5: aborting pipeline replication: peer="{Voter server-14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9 14398af1-bbcf-1ae0-44dd-d16b2a2e2ba9}" --- PASS: TestRaft_HasExistingState (0.28s) === RUN TestRaft_SingleNode testLoggerAdapter verbose: server-e1e975b8-7824-3bc1-f543-1a4998307c2f: 2023-08-02T15:12:52.531Z [INFO] server-e1e975b8-7824-3bc1-f543-1a4998307c2f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e1e975b8-7824-3bc1-f543-1a4998307c2f Address:e1e975b8-7824-3bc1-f543-1a4998307c2f}]" testing.go:149: server-e1e975b8-7824-3bc1-f543-1a4998307c2f: 2023-08-02T15:12:52.531Z [INFO] server-e1e975b8-7824-3bc1-f543-1a4998307c2f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e1e975b8-7824-3bc1-f543-1a4998307c2f Address:e1e975b8-7824-3bc1-f543-1a4998307c2f}]" testLoggerAdapter verbose: server-e1e975b8-7824-3bc1-f543-1a4998307c2f: 2023-08-02T15:12:52.531Z [INFO] server-e1e975b8-7824-3bc1-f543-1a4998307c2f: entering follower state: follower="Node at e1e975b8-7824-3bc1-f543-1a4998307c2f [Follower]" leader= testing.go:149: server-e1e975b8-7824-3bc1-f543-1a4998307c2f: 2023-08-02T15:12:52.531Z [INFO] server-e1e975b8-7824-3bc1-f543-1a4998307c2f: entering follower state: follower="Node at e1e975b8-7824-3bc1-f543-1a4998307c2f [Follower]" leader= testLoggerAdapter verbose: server-e1e975b8-7824-3bc1-f543-1a4998307c2f: 2023-08-02T15:12:52.612Z [WARN] server-e1e975b8-7824-3bc1-f543-1a4998307c2f: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-e1e975b8-7824-3bc1-f543-1a4998307c2f: 2023-08-02T15:12:52.612Z [WARN] server-e1e975b8-7824-3bc1-f543-1a4998307c2f: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-e1e975b8-7824-3bc1-f543-1a4998307c2f: 2023-08-02T15:12:52.612Z [INFO] server-e1e975b8-7824-3bc1-f543-1a4998307c2f: entering candidate state: node="Node at e1e975b8-7824-3bc1-f543-1a4998307c2f [Candidate]" term=2 testing.go:149: server-e1e975b8-7824-3bc1-f543-1a4998307c2f: 2023-08-02T15:12:52.612Z [INFO] server-e1e975b8-7824-3bc1-f543-1a4998307c2f: entering candidate state: node="Node at e1e975b8-7824-3bc1-f543-1a4998307c2f [Candidate]" term=2 testLoggerAdapter verbose: server-e1e975b8-7824-3bc1-f543-1a4998307c2f: 2023-08-02T15:12:52.612Z [INFO] server-e1e975b8-7824-3bc1-f543-1a4998307c2f: election won: tally=1 testing.go:149: server-e1e975b8-7824-3bc1-f543-1a4998307c2f: 2023-08-02T15:12:52.612Z [INFO] server-e1e975b8-7824-3bc1-f543-1a4998307c2f: election won: tally=1 testLoggerAdapter verbose: server-e1e975b8-7824-3bc1-f543-1a4998307c2f: 2023-08-02T15:12:52.612Z [INFO] server-e1e975b8-7824-3bc1-f543-1a4998307c2f: entering leader state: leader="Node at e1e975b8-7824-3bc1-f543-1a4998307c2f [Leader]" testing.go:149: server-e1e975b8-7824-3bc1-f543-1a4998307c2f: 2023-08-02T15:12:52.612Z [INFO] server-e1e975b8-7824-3bc1-f543-1a4998307c2f: entering leader state: leader="Node at e1e975b8-7824-3bc1-f543-1a4998307c2f [Leader]" --- PASS: TestRaft_SingleNode (0.08s) === RUN TestRaft_TripleNode testLoggerAdapter verbose: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.613Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62 Address:6c3170fc-6ae7-6efe-0d04-2f64bc585a62} {Suffrage:Voter ID:server-2106ef7a-f83b-291a-1cf3-215ef1d30f61 Address:2106ef7a-f83b-291a-1cf3-215ef1d30f61} {Suffrage:Voter ID:server-2d685416-8549-aec5-cd16-ee788af57344 Address:2d685416-8549-aec5-cd16-ee788af57344}]" testing.go:149: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.613Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62 Address:6c3170fc-6ae7-6efe-0d04-2f64bc585a62} {Suffrage:Voter ID:server-2106ef7a-f83b-291a-1cf3-215ef1d30f61 Address:2106ef7a-f83b-291a-1cf3-215ef1d30f61} {Suffrage:Voter ID:server-2d685416-8549-aec5-cd16-ee788af57344 Address:2d685416-8549-aec5-cd16-ee788af57344}]" testLoggerAdapter verbose: server-2106ef7a-f83b-291a-1cf3-215ef1d30f61: 2023-08-02T15:12:52.613Z [INFO] server-2106ef7a-f83b-291a-1cf3-215ef1d30f61: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62 Address:6c3170fc-6ae7-6efe-0d04-2f64bc585a62} {Suffrage:Voter ID:server-2106ef7a-f83b-291a-1cf3-215ef1d30f61 Address:2106ef7a-f83b-291a-1cf3-215ef1d30f61} {Suffrage:Voter ID:server-2d685416-8549-aec5-cd16-ee788af57344 Address:2d685416-8549-aec5-cd16-ee788af57344}]" testLoggerAdapter verbose: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.613Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: entering follower state: follower="Node at 6c3170fc-6ae7-6efe-0d04-2f64bc585a62 [Follower]" leader= testing.go:149: server-2106ef7a-f83b-291a-1cf3-215ef1d30f61: 2023-08-02T15:12:52.613Z [INFO] server-2106ef7a-f83b-291a-1cf3-215ef1d30f61: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62 Address:6c3170fc-6ae7-6efe-0d04-2f64bc585a62} {Suffrage:Voter ID:server-2106ef7a-f83b-291a-1cf3-215ef1d30f61 Address:2106ef7a-f83b-291a-1cf3-215ef1d30f61} {Suffrage:Voter ID:server-2d685416-8549-aec5-cd16-ee788af57344 Address:2d685416-8549-aec5-cd16-ee788af57344}]" testing.go:149: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.613Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: entering follower state: follower="Node at 6c3170fc-6ae7-6efe-0d04-2f64bc585a62 [Follower]" leader= testLoggerAdapter verbose: server-2106ef7a-f83b-291a-1cf3-215ef1d30f61: 2023-08-02T15:12:52.613Z [INFO] server-2106ef7a-f83b-291a-1cf3-215ef1d30f61: entering follower state: follower="Node at 2106ef7a-f83b-291a-1cf3-215ef1d30f61 [Follower]" leader= testing.go:149: server-2106ef7a-f83b-291a-1cf3-215ef1d30f61: 2023-08-02T15:12:52.613Z [INFO] server-2106ef7a-f83b-291a-1cf3-215ef1d30f61: entering follower state: follower="Node at 2106ef7a-f83b-291a-1cf3-215ef1d30f61 [Follower]" leader= testLoggerAdapter verbose: server-2d685416-8549-aec5-cd16-ee788af57344: 2023-08-02T15:12:52.613Z [INFO] server-2d685416-8549-aec5-cd16-ee788af57344: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62 Address:6c3170fc-6ae7-6efe-0d04-2f64bc585a62} {Suffrage:Voter ID:server-2106ef7a-f83b-291a-1cf3-215ef1d30f61 Address:2106ef7a-f83b-291a-1cf3-215ef1d30f61} {Suffrage:Voter ID:server-2d685416-8549-aec5-cd16-ee788af57344 Address:2d685416-8549-aec5-cd16-ee788af57344}]" testing.go:149: server-2d685416-8549-aec5-cd16-ee788af57344: 2023-08-02T15:12:52.613Z [INFO] server-2d685416-8549-aec5-cd16-ee788af57344: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62 Address:6c3170fc-6ae7-6efe-0d04-2f64bc585a62} {Suffrage:Voter ID:server-2106ef7a-f83b-291a-1cf3-215ef1d30f61 Address:2106ef7a-f83b-291a-1cf3-215ef1d30f61} {Suffrage:Voter ID:server-2d685416-8549-aec5-cd16-ee788af57344 Address:2d685416-8549-aec5-cd16-ee788af57344}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:52.613Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-08-02T15:12:52.613Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-2d685416-8549-aec5-cd16-ee788af57344: 2023-08-02T15:12:52.613Z [INFO] server-2d685416-8549-aec5-cd16-ee788af57344: entering follower state: follower="Node at 2d685416-8549-aec5-cd16-ee788af57344 [Follower]" leader= testing.go:149: server-2d685416-8549-aec5-cd16-ee788af57344: 2023-08-02T15:12:52.613Z [INFO] server-2d685416-8549-aec5-cd16-ee788af57344: entering follower state: follower="Node at 2d685416-8549-aec5-cd16-ee788af57344 [Follower]" leader= testLoggerAdapter verbose: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.675Z [WARN] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.675Z [WARN] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.676Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: entering candidate state: node="Node at 6c3170fc-6ae7-6efe-0d04-2f64bc585a62 [Candidate]" term=2 testing.go:149: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.676Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: entering candidate state: node="Node at 6c3170fc-6ae7-6efe-0d04-2f64bc585a62 [Candidate]" term=2 testLoggerAdapter verbose: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.676Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: election won: tally=2 testing.go:149: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.676Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: election won: tally=2 testLoggerAdapter verbose: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.676Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: entering leader state: leader="Node at 6c3170fc-6ae7-6efe-0d04-2f64bc585a62 [Leader]" testing.go:149: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.676Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: entering leader state: leader="Node at 6c3170fc-6ae7-6efe-0d04-2f64bc585a62 [Leader]" testLoggerAdapter verbose: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.676Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: added peer, starting replication: peer=server-2106ef7a-f83b-291a-1cf3-215ef1d30f61 testing.go:149: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.676Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: added peer, starting replication: peer=server-2106ef7a-f83b-291a-1cf3-215ef1d30f61 testLoggerAdapter verbose: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.676Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: added peer, starting replication: peer=server-2d685416-8549-aec5-cd16-ee788af57344 testing.go:149: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.676Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: added peer, starting replication: peer=server-2d685416-8549-aec5-cd16-ee788af57344 testLoggerAdapter verbose: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.676Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: pipelining replication: peer="{Voter server-2d685416-8549-aec5-cd16-ee788af57344 2d685416-8549-aec5-cd16-ee788af57344}" testing.go:149: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.676Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: pipelining replication: peer="{Voter server-2d685416-8549-aec5-cd16-ee788af57344 2d685416-8549-aec5-cd16-ee788af57344}" testLoggerAdapter verbose: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.676Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: pipelining replication: peer="{Voter server-2106ef7a-f83b-291a-1cf3-215ef1d30f61 2106ef7a-f83b-291a-1cf3-215ef1d30f61}" testing.go:149: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.676Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: pipelining replication: peer="{Voter server-2106ef7a-f83b-291a-1cf3-215ef1d30f61 2106ef7a-f83b-291a-1cf3-215ef1d30f61}" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:52.781Z [INFO] stable state for Follower reached at 2023-08-02 15:12:52.676499237 +0000 UTC m=+4.772435572 (2 nodes), 62.613624ms from start of poll, 63.151441ms from cluster start. Timeout at 2023-08-02 15:12:52.781804772 +0000 UTC m=+4.877741108, 105.305536ms after stability testing.go:149: cluster: 2023-08-02T15:12:52.781Z [INFO] stable state for Follower reached at 2023-08-02 15:12:52.676499237 +0000 UTC m=+4.772435572 (2 nodes), 62.613624ms from start of poll, 63.151441ms from cluster start. Timeout at 2023-08-02 15:12:52.781804772 +0000 UTC m=+4.877741108, 105.305536ms after stability testLoggerAdapter verbose: cluster: 2023-08-02T15:12:52.781Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:12:52.781Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-08-02T15:12:52.887Z [INFO] stable state for Leader reached at 2023-08-02 15:12:52.781997923 +0000 UTC m=+4.877934250 (1 nodes), 1.687µs from start of poll, 168.650119ms from cluster start. Timeout at 2023-08-02 15:12:52.887111162 +0000 UTC m=+4.983047500, 105.11325ms after stability testing.go:149: cluster: 2023-08-02T15:12:52.887Z [INFO] stable state for Leader reached at 2023-08-02 15:12:52.781997923 +0000 UTC m=+4.877934250 (1 nodes), 1.687µs from start of poll, 168.650119ms from cluster start. Timeout at 2023-08-02 15:12:52.887111162 +0000 UTC m=+4.983047500, 105.11325ms after stability testLoggerAdapter verbose: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.897Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: aborting pipeline replication: peer="{Voter server-2d685416-8549-aec5-cd16-ee788af57344 2d685416-8549-aec5-cd16-ee788af57344}" testing.go:149: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.897Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: aborting pipeline replication: peer="{Voter server-2d685416-8549-aec5-cd16-ee788af57344 2d685416-8549-aec5-cd16-ee788af57344}" testLoggerAdapter verbose: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.897Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: aborting pipeline replication: peer="{Voter server-2106ef7a-f83b-291a-1cf3-215ef1d30f61 2106ef7a-f83b-291a-1cf3-215ef1d30f61}" testing.go:149: server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: 2023-08-02T15:12:52.897Z [INFO] server-6c3170fc-6ae7-6efe-0d04-2f64bc585a62: aborting pipeline replication: peer="{Voter server-2106ef7a-f83b-291a-1cf3-215ef1d30f61 2106ef7a-f83b-291a-1cf3-215ef1d30f61}" --- PASS: TestRaft_TripleNode (0.29s) === RUN TestRaft_LeaderFail testLoggerAdapter verbose: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:52.899Z [INFO] server-c2229225-6527-5735-0db9-d0ba6638f6ab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2229225-6527-5735-0db9-d0ba6638f6ab Address:c2229225-6527-5735-0db9-d0ba6638f6ab} {Suffrage:Voter ID:server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 Address:b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47} {Suffrage:Voter ID:server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d Address:90e7a8fc-6dd5-aad7-a258-1318a6f7815d}]" testing.go:149: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:52.899Z [INFO] server-c2229225-6527-5735-0db9-d0ba6638f6ab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2229225-6527-5735-0db9-d0ba6638f6ab Address:c2229225-6527-5735-0db9-d0ba6638f6ab} {Suffrage:Voter ID:server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 Address:b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47} {Suffrage:Voter ID:server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d Address:90e7a8fc-6dd5-aad7-a258-1318a6f7815d}]" testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.899Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2229225-6527-5735-0db9-d0ba6638f6ab Address:c2229225-6527-5735-0db9-d0ba6638f6ab} {Suffrage:Voter ID:server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 Address:b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47} {Suffrage:Voter ID:server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d Address:90e7a8fc-6dd5-aad7-a258-1318a6f7815d}]" testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.899Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2229225-6527-5735-0db9-d0ba6638f6ab Address:c2229225-6527-5735-0db9-d0ba6638f6ab} {Suffrage:Voter ID:server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 Address:b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47} {Suffrage:Voter ID:server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d Address:90e7a8fc-6dd5-aad7-a258-1318a6f7815d}]" testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:52.899Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2229225-6527-5735-0db9-d0ba6638f6ab Address:c2229225-6527-5735-0db9-d0ba6638f6ab} {Suffrage:Voter ID:server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 Address:b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47} {Suffrage:Voter ID:server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d Address:90e7a8fc-6dd5-aad7-a258-1318a6f7815d}]" testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:52.899Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2229225-6527-5735-0db9-d0ba6638f6ab Address:c2229225-6527-5735-0db9-d0ba6638f6ab} {Suffrage:Voter ID:server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 Address:b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47} {Suffrage:Voter ID:server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d Address:90e7a8fc-6dd5-aad7-a258-1318a6f7815d}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:52.899Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-08-02T15:12:52.899Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:52.899Z [INFO] server-c2229225-6527-5735-0db9-d0ba6638f6ab: entering follower state: follower="Node at c2229225-6527-5735-0db9-d0ba6638f6ab [Follower]" leader= testing.go:149: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:52.899Z [INFO] server-c2229225-6527-5735-0db9-d0ba6638f6ab: entering follower state: follower="Node at c2229225-6527-5735-0db9-d0ba6638f6ab [Follower]" leader= testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.899Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: entering follower state: follower="Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Follower]" leader= testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.899Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: entering follower state: follower="Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Follower]" leader= testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:52.899Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: entering follower state: follower="Node at 90e7a8fc-6dd5-aad7-a258-1318a6f7815d [Follower]" leader= testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:52.899Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: entering follower state: follower="Node at 90e7a8fc-6dd5-aad7-a258-1318a6f7815d [Follower]" leader= testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.985Z [WARN] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.985Z [WARN] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.985Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: entering candidate state: node="Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Candidate]" term=2 testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.985Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: entering candidate state: node="Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Candidate]" term=2 testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.986Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: election won: tally=2 testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.986Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: election won: tally=2 testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.986Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: entering leader state: leader="Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Leader]" testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.986Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: entering leader state: leader="Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Leader]" testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.986Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: added peer, starting replication: peer=server-c2229225-6527-5735-0db9-d0ba6638f6ab testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.986Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: added peer, starting replication: peer=server-c2229225-6527-5735-0db9-d0ba6638f6ab testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.986Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: added peer, starting replication: peer=server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.986Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: added peer, starting replication: peer=server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.986Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: pipelining replication: peer="{Voter server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d 90e7a8fc-6dd5-aad7-a258-1318a6f7815d}" testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.986Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: pipelining replication: peer="{Voter server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d 90e7a8fc-6dd5-aad7-a258-1318a6f7815d}" testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.986Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: pipelining replication: peer="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:52.986Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: pipelining replication: peer="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:53.091Z [INFO] stable state for Follower reached at 2023-08-02 15:12:52.986144386 +0000 UTC m=+5.082080720 (2 nodes), 86.687489ms from start of poll, 87.245575ms from cluster start. Timeout at 2023-08-02 15:12:53.091261464 +0000 UTC m=+5.187197809, 105.117089ms after stability testing.go:149: cluster: 2023-08-02T15:12:53.091Z [INFO] stable state for Follower reached at 2023-08-02 15:12:52.986144386 +0000 UTC m=+5.082080720 (2 nodes), 86.687489ms from start of poll, 87.245575ms from cluster start. Timeout at 2023-08-02 15:12:53.091261464 +0000 UTC m=+5.187197809, 105.117089ms after stability testLoggerAdapter verbose: cluster: 2023-08-02T15:12:53.091Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:12:53.091Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-08-02T15:12:53.196Z [INFO] stable state for Leader reached at 2023-08-02 15:12:53.091428116 +0000 UTC m=+5.187364444 (1 nodes), 1.345µs from start of poll, 192.529299ms from cluster start. Timeout at 2023-08-02 15:12:53.196520949 +0000 UTC m=+5.292457284, 105.09284ms after stability testing.go:149: cluster: 2023-08-02T15:12:53.196Z [INFO] stable state for Leader reached at 2023-08-02 15:12:53.091428116 +0000 UTC m=+5.187364444 (1 nodes), 1.345µs from start of poll, 192.529299ms from cluster start. Timeout at 2023-08-02 15:12:53.196520949 +0000 UTC m=+5.292457284, 105.09284ms after stability raft_test.go:336: [INFO] Disconnecting Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Leader] testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.209Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to pipeline appendEntries: peer="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" error="append pipeline closed" testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.209Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to pipeline appendEntries: peer="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" error="append pipeline closed" testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.209Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: aborting pipeline replication: peer="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.209Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: aborting pipeline replication: peer="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.210Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to heartbeat to: peer=c2229225-6527-5735-0db9-d0ba6638f6ab error="failed to connect to peer: c2229225-6527-5735-0db9-d0ba6638f6ab" testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.210Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to heartbeat to: peer=c2229225-6527-5735-0db9-d0ba6638f6ab error="failed to connect to peer: c2229225-6527-5735-0db9-d0ba6638f6ab" testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.211Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to heartbeat to: peer=90e7a8fc-6dd5-aad7-a258-1318a6f7815d error="failed to connect to peer: 90e7a8fc-6dd5-aad7-a258-1318a6f7815d" testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.211Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to heartbeat to: peer=90e7a8fc-6dd5-aad7-a258-1318a6f7815d error="failed to connect to peer: 90e7a8fc-6dd5-aad7-a258-1318a6f7815d" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:53.212Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:12:53.212Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.212Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to pipeline appendEntries: peer="{Voter server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d 90e7a8fc-6dd5-aad7-a258-1318a6f7815d}" error="append pipeline closed" testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.212Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to pipeline appendEntries: peer="{Voter server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d 90e7a8fc-6dd5-aad7-a258-1318a6f7815d}" error="append pipeline closed" testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.212Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: aborting pipeline replication: peer="{Voter server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d 90e7a8fc-6dd5-aad7-a258-1318a6f7815d}" testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.212Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: aborting pipeline replication: peer="{Voter server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d 90e7a8fc-6dd5-aad7-a258-1318a6f7815d}" testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.218Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to appendEntries to: peer="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" error="failed to connect to peer: c2229225-6527-5735-0db9-d0ba6638f6ab" testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.218Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to appendEntries to: peer="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" error="failed to connect to peer: c2229225-6527-5735-0db9-d0ba6638f6ab" testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.221Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to appendEntries to: peer="{Voter server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d 90e7a8fc-6dd5-aad7-a258-1318a6f7815d}" error="failed to connect to peer: 90e7a8fc-6dd5-aad7-a258-1318a6f7815d" testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.221Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to appendEntries to: peer="{Voter server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d 90e7a8fc-6dd5-aad7-a258-1318a6f7815d}" error="failed to connect to peer: 90e7a8fc-6dd5-aad7-a258-1318a6f7815d" testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.228Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to heartbeat to: peer=90e7a8fc-6dd5-aad7-a258-1318a6f7815d error="failed to connect to peer: 90e7a8fc-6dd5-aad7-a258-1318a6f7815d" testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.228Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to heartbeat to: peer=90e7a8fc-6dd5-aad7-a258-1318a6f7815d error="failed to connect to peer: 90e7a8fc-6dd5-aad7-a258-1318a6f7815d" testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.229Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to heartbeat to: peer=c2229225-6527-5735-0db9-d0ba6638f6ab error="failed to connect to peer: c2229225-6527-5735-0db9-d0ba6638f6ab" testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.229Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to heartbeat to: peer=c2229225-6527-5735-0db9-d0ba6638f6ab error="failed to connect to peer: c2229225-6527-5735-0db9-d0ba6638f6ab" testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.236Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to appendEntries to: peer="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" error="failed to connect to peer: c2229225-6527-5735-0db9-d0ba6638f6ab" testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.236Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to appendEntries to: peer="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" error="failed to connect to peer: c2229225-6527-5735-0db9-d0ba6638f6ab" testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.239Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to appendEntries to: peer="{Voter server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d 90e7a8fc-6dd5-aad7-a258-1318a6f7815d}" error="failed to connect to peer: 90e7a8fc-6dd5-aad7-a258-1318a6f7815d" testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.239Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to appendEntries to: peer="{Voter server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d 90e7a8fc-6dd5-aad7-a258-1318a6f7815d}" error="failed to connect to peer: 90e7a8fc-6dd5-aad7-a258-1318a6f7815d" testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.245Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to heartbeat to: peer=90e7a8fc-6dd5-aad7-a258-1318a6f7815d error="failed to connect to peer: 90e7a8fc-6dd5-aad7-a258-1318a6f7815d" testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.245Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to heartbeat to: peer=90e7a8fc-6dd5-aad7-a258-1318a6f7815d error="failed to connect to peer: 90e7a8fc-6dd5-aad7-a258-1318a6f7815d" testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.249Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to heartbeat to: peer=c2229225-6527-5735-0db9-d0ba6638f6ab error="failed to connect to peer: c2229225-6527-5735-0db9-d0ba6638f6ab" testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.249Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to heartbeat to: peer=c2229225-6527-5735-0db9-d0ba6638f6ab error="failed to connect to peer: c2229225-6527-5735-0db9-d0ba6638f6ab" testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.255Z [WARN] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to contact: server-id=server-c2229225-6527-5735-0db9-d0ba6638f6ab time=51.598509ms testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.255Z [WARN] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to contact: server-id=server-c2229225-6527-5735-0db9-d0ba6638f6ab time=51.598509ms testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.256Z [WARN] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to contact: server-id=server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d time=51.07772ms testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.256Z [WARN] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to contact: server-id=server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d time=51.07772ms testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.256Z [WARN] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to contact quorum of nodes, stepping down testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.256Z [WARN] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to contact quorum of nodes, stepping down testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.256Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: entering follower state: follower="Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Follower]" leader= testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.256Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: entering follower state: follower="Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Follower]" leader= testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.255Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to appendEntries to: peer="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" error="failed to connect to peer: c2229225-6527-5735-0db9-d0ba6638f6ab" testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.255Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to appendEntries to: peer="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" error="failed to connect to peer: c2229225-6527-5735-0db9-d0ba6638f6ab" testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.257Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to appendEntries to: peer="{Voter server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d 90e7a8fc-6dd5-aad7-a258-1318a6f7815d}" error="failed to connect to peer: 90e7a8fc-6dd5-aad7-a258-1318a6f7815d" testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.257Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to appendEntries to: peer="{Voter server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d 90e7a8fc-6dd5-aad7-a258-1318a6f7815d}" error="failed to connect to peer: 90e7a8fc-6dd5-aad7-a258-1318a6f7815d" testLoggerAdapter verbose: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.307Z [WARN] server-c2229225-6527-5735-0db9-d0ba6638f6ab: heartbeat timeout reached, starting election: last-leader=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 testing.go:149: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.307Z [WARN] server-c2229225-6527-5735-0db9-d0ba6638f6ab: heartbeat timeout reached, starting election: last-leader=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 testLoggerAdapter verbose: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.308Z [INFO] server-c2229225-6527-5735-0db9-d0ba6638f6ab: entering candidate state: node="Node at c2229225-6527-5735-0db9-d0ba6638f6ab [Candidate]" term=3 testing.go:149: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.308Z [INFO] server-c2229225-6527-5735-0db9-d0ba6638f6ab: entering candidate state: node="Node at c2229225-6527-5735-0db9-d0ba6638f6ab [Candidate]" term=3 testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.308Z [WARN] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: rejecting vote request since we have a leader: from=c2229225-6527-5735-0db9-d0ba6638f6ab leader=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.308Z [WARN] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: rejecting vote request since we have a leader: from=c2229225-6527-5735-0db9-d0ba6638f6ab leader=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 testLoggerAdapter verbose: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.308Z [ERROR] server-c2229225-6527-5735-0db9-d0ba6638f6ab: failed to make requestVote RPC: target="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" testing.go:149: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.308Z [ERROR] server-c2229225-6527-5735-0db9-d0ba6638f6ab: failed to make requestVote RPC: target="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.310Z [WARN] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: heartbeat timeout reached, starting election: last-leader=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.310Z [WARN] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: heartbeat timeout reached, starting election: last-leader=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.310Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: entering candidate state: node="Node at 90e7a8fc-6dd5-aad7-a258-1318a6f7815d [Candidate]" term=3 testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.310Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: entering candidate state: node="Node at 90e7a8fc-6dd5-aad7-a258-1318a6f7815d [Candidate]" term=3 testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.310Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to make requestVote RPC: target="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.310Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to make requestVote RPC: target="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" testLoggerAdapter verbose: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.310Z [INFO] server-c2229225-6527-5735-0db9-d0ba6638f6ab: duplicate requestVote for same term: term=3 testing.go:149: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.310Z [INFO] server-c2229225-6527-5735-0db9-d0ba6638f6ab: duplicate requestVote for same term: term=3 testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.317Z [WARN] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.317Z [WARN] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.317Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: entering candidate state: node="Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Candidate]" term=3 testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.317Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: entering candidate state: node="Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Candidate]" term=3 testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.317Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to make requestVote RPC: target="{Voter server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d 90e7a8fc-6dd5-aad7-a258-1318a6f7815d}" error="failed to connect to peer: 90e7a8fc-6dd5-aad7-a258-1318a6f7815d" testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.317Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to make requestVote RPC: target="{Voter server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d 90e7a8fc-6dd5-aad7-a258-1318a6f7815d}" error="failed to connect to peer: 90e7a8fc-6dd5-aad7-a258-1318a6f7815d" testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.317Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to make requestVote RPC: target="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" error="failed to connect to peer: c2229225-6527-5735-0db9-d0ba6638f6ab" testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.317Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to make requestVote RPC: target="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" error="failed to connect to peer: c2229225-6527-5735-0db9-d0ba6638f6ab" testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.362Z [WARN] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: Election timeout reached, restarting election testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.362Z [WARN] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: Election timeout reached, restarting election testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.362Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: entering candidate state: node="Node at 90e7a8fc-6dd5-aad7-a258-1318a6f7815d [Candidate]" term=4 testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.362Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: entering candidate state: node="Node at 90e7a8fc-6dd5-aad7-a258-1318a6f7815d [Candidate]" term=4 testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.362Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to make requestVote RPC: target="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.362Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to make requestVote RPC: target="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" testLoggerAdapter verbose: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.362Z [INFO] server-c2229225-6527-5735-0db9-d0ba6638f6ab: entering follower state: follower="Node at c2229225-6527-5735-0db9-d0ba6638f6ab [Follower]" leader= testing.go:149: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.362Z [INFO] server-c2229225-6527-5735-0db9-d0ba6638f6ab: entering follower state: follower="Node at c2229225-6527-5735-0db9-d0ba6638f6ab [Follower]" leader= testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.362Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: election won: tally=2 testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.362Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: election won: tally=2 testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.362Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: entering leader state: leader="Node at 90e7a8fc-6dd5-aad7-a258-1318a6f7815d [Leader]" testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.362Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: entering leader state: leader="Node at 90e7a8fc-6dd5-aad7-a258-1318a6f7815d [Leader]" testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.362Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: added peer, starting replication: peer=server-c2229225-6527-5735-0db9-d0ba6638f6ab testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.362Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: added peer, starting replication: peer=server-c2229225-6527-5735-0db9-d0ba6638f6ab testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.363Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: added peer, starting replication: peer=server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.363Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: added peer, starting replication: peer=server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.363Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to appendEntries to: peer="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.363Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to appendEntries to: peer="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.363Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: pipelining replication: peer="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.363Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: pipelining replication: peer="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.369Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to heartbeat to: peer=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.369Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to heartbeat to: peer=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.383Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to appendEntries to: peer="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.383Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to appendEntries to: peer="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.389Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to heartbeat to: peer=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.389Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to heartbeat to: peer=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.402Z [WARN] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: Election timeout reached, restarting election testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.402Z [WARN] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: Election timeout reached, restarting election testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.403Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: entering candidate state: node="Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Candidate]" term=4 testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.403Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: entering candidate state: node="Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Candidate]" term=4 testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.403Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to make requestVote RPC: target="{Voter server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d 90e7a8fc-6dd5-aad7-a258-1318a6f7815d}" error="failed to connect to peer: 90e7a8fc-6dd5-aad7-a258-1318a6f7815d" testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.403Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to make requestVote RPC: target="{Voter server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d 90e7a8fc-6dd5-aad7-a258-1318a6f7815d}" error="failed to connect to peer: 90e7a8fc-6dd5-aad7-a258-1318a6f7815d" testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.403Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to make requestVote RPC: target="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" error="failed to connect to peer: c2229225-6527-5735-0db9-d0ba6638f6ab" testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.403Z [ERROR] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to make requestVote RPC: target="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" error="failed to connect to peer: c2229225-6527-5735-0db9-d0ba6638f6ab" testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.404Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to appendEntries to: peer="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.404Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to appendEntries to: peer="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.409Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to heartbeat to: peer=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.409Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to heartbeat to: peer=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.414Z [WARN] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to contact: server-id=server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 time=50.992637ms testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.414Z [WARN] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to contact: server-id=server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 time=50.992637ms testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.433Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to appendEntries to: peer="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.433Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to appendEntries to: peer="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.436Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to heartbeat to: peer=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.436Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to heartbeat to: peer=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 error="failed to connect to peer: b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47" testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.460Z [WARN] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to contact: server-id=server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 time=97.797067ms testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.460Z [WARN] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to contact: server-id=server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 time=97.797067ms testLoggerAdapter verbose: cluster: 2023-08-02T15:12:53.468Z [INFO] stable state for Leader reached at 2023-08-02 15:12:53.362960679 +0000 UTC m=+5.458897005 (1 nodes), 150.682524ms from start of poll, 464.06186ms from cluster start. Timeout at 2023-08-02 15:12:53.468104224 +0000 UTC m=+5.564040554, 105.143549ms after stability testing.go:149: cluster: 2023-08-02T15:12:53.468Z [INFO] stable state for Leader reached at 2023-08-02 15:12:53.362960679 +0000 UTC m=+5.458897005 (1 nodes), 150.682524ms from start of poll, 464.06186ms from cluster start. Timeout at 2023-08-02 15:12:53.468104224 +0000 UTC m=+5.564040554, 105.143549ms after stability raft_test.go:371: [INFO] Reconnecting Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Candidate] testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.474Z [WARN] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: Election timeout reached, restarting election testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.474Z [WARN] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: Election timeout reached, restarting election testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.474Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: entering candidate state: node="Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Candidate]" term=5 testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.474Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: entering candidate state: node="Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Candidate]" term=5 testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.475Z [WARN] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: rejecting vote request since we have a leader: from=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 leader=90e7a8fc-6dd5-aad7-a258-1318a6f7815d testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.475Z [WARN] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: rejecting vote request since we have a leader: from=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 leader=90e7a8fc-6dd5-aad7-a258-1318a6f7815d testLoggerAdapter verbose: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.475Z [WARN] server-c2229225-6527-5735-0db9-d0ba6638f6ab: rejecting vote request since we have a leader: from=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 leader=90e7a8fc-6dd5-aad7-a258-1318a6f7815d testing.go:149: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.475Z [WARN] server-c2229225-6527-5735-0db9-d0ba6638f6ab: rejecting vote request since we have a leader: from=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 leader=90e7a8fc-6dd5-aad7-a258-1318a6f7815d testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.483Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: peer has newer term, stopping replication: peer="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.483Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: peer has newer term, stopping replication: peer="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.484Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: entering follower state: follower="Node at 90e7a8fc-6dd5-aad7-a258-1318a6f7815d [Follower]" leader= testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.484Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: entering follower state: follower="Node at 90e7a8fc-6dd5-aad7-a258-1318a6f7815d [Follower]" leader= testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.484Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: aborting pipeline replication: peer="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.484Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: aborting pipeline replication: peer="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.526Z [WARN] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: Election timeout reached, restarting election testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.526Z [WARN] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: Election timeout reached, restarting election testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.527Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: entering candidate state: node="Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Candidate]" term=6 testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.527Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: entering candidate state: node="Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Candidate]" term=6 testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.527Z [WARN] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: rejecting vote request since our last term is greater: candidate=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 last-term=4 last-candidate-term=2 testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.527Z [WARN] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: rejecting vote request since our last term is greater: candidate=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 last-term=4 last-candidate-term=2 testLoggerAdapter verbose: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.527Z [WARN] server-c2229225-6527-5735-0db9-d0ba6638f6ab: rejecting vote request since we have a leader: from=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 leader=90e7a8fc-6dd5-aad7-a258-1318a6f7815d testing.go:149: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.527Z [WARN] server-c2229225-6527-5735-0db9-d0ba6638f6ab: rejecting vote request since we have a leader: from=b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 leader=90e7a8fc-6dd5-aad7-a258-1318a6f7815d testLoggerAdapter verbose: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.528Z [WARN] server-c2229225-6527-5735-0db9-d0ba6638f6ab: heartbeat timeout reached, starting election: last-leader=90e7a8fc-6dd5-aad7-a258-1318a6f7815d testing.go:149: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.528Z [WARN] server-c2229225-6527-5735-0db9-d0ba6638f6ab: heartbeat timeout reached, starting election: last-leader=90e7a8fc-6dd5-aad7-a258-1318a6f7815d testLoggerAdapter verbose: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.528Z [INFO] server-c2229225-6527-5735-0db9-d0ba6638f6ab: entering candidate state: node="Node at c2229225-6527-5735-0db9-d0ba6638f6ab [Candidate]" term=5 testing.go:149: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.528Z [INFO] server-c2229225-6527-5735-0db9-d0ba6638f6ab: entering candidate state: node="Node at c2229225-6527-5735-0db9-d0ba6638f6ab [Candidate]" term=5 testLoggerAdapter verbose: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.528Z [INFO] server-c2229225-6527-5735-0db9-d0ba6638f6ab: entering follower state: follower="Node at c2229225-6527-5735-0db9-d0ba6638f6ab [Follower]" leader= testing.go:149: server-c2229225-6527-5735-0db9-d0ba6638f6ab: 2023-08-02T15:12:53.528Z [INFO] server-c2229225-6527-5735-0db9-d0ba6638f6ab: entering follower state: follower="Node at c2229225-6527-5735-0db9-d0ba6638f6ab [Follower]" leader= testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.570Z [WARN] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.570Z [WARN] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.570Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: entering candidate state: node="Node at 90e7a8fc-6dd5-aad7-a258-1318a6f7815d [Candidate]" term=7 testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.570Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: entering candidate state: node="Node at 90e7a8fc-6dd5-aad7-a258-1318a6f7815d [Candidate]" term=7 testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.570Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: entering follower state: follower="Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Follower]" leader= testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.570Z [INFO] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: entering follower state: follower="Node at b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 [Follower]" leader= testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.570Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: election won: tally=2 testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.570Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: election won: tally=2 testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.570Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: entering leader state: leader="Node at 90e7a8fc-6dd5-aad7-a258-1318a6f7815d [Leader]" testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.570Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: entering leader state: leader="Node at 90e7a8fc-6dd5-aad7-a258-1318a6f7815d [Leader]" testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.570Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: added peer, starting replication: peer=server-c2229225-6527-5735-0db9-d0ba6638f6ab testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.570Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: added peer, starting replication: peer=server-c2229225-6527-5735-0db9-d0ba6638f6ab testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.570Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: added peer, starting replication: peer=server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.570Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: added peer, starting replication: peer=server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 testLoggerAdapter verbose: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.570Z [WARN] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to get previous log: previous-index=5 last-index=3 error="log not found" testing.go:149: server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: 2023-08-02T15:12:53.570Z [WARN] server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47: failed to get previous log: previous-index=5 last-index=3 error="log not found" testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.570Z [WARN] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: appendEntries rejected, sending older logs: peer="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" next=4 testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.570Z [WARN] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: appendEntries rejected, sending older logs: peer="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" next=4 testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.570Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: pipelining replication: peer="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.570Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: pipelining replication: peer="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.570Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: aborting pipeline replication: peer="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.570Z [INFO] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: aborting pipeline replication: peer="{Voter server-b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47 b96dc8d5-cb5f-f5d4-2163-5220d1ebbd47}" testLoggerAdapter verbose: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.620Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to make requestVote RPC: target="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" error="command timed out" testing.go:149: server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: 2023-08-02T15:12:53.620Z [ERROR] server-90e7a8fc-6dd5-aad7-a258-1318a6f7815d: failed to make requestVote RPC: target="{Voter server-c2229225-6527-5735-0db9-d0ba6638f6ab c2229225-6527-5735-0db9-d0ba6638f6ab}" error="command timed out" --- PASS: TestRaft_LeaderFail (0.72s) === RUN TestRaft_BehindFollower testLoggerAdapter verbose: server-180d44c5-13c8-1722-a927-e21b06e48e82: 2023-08-02T15:12:53.621Z [INFO] server-180d44c5-13c8-1722-a927-e21b06e48e82: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-180d44c5-13c8-1722-a927-e21b06e48e82 Address:180d44c5-13c8-1722-a927-e21b06e48e82} {Suffrage:Voter ID:server-2c5e301c-60c6-626a-ce4c-88387582e874 Address:2c5e301c-60c6-626a-ce4c-88387582e874} {Suffrage:Voter ID:server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9 Address:d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9}]" testing.go:149: server-180d44c5-13c8-1722-a927-e21b06e48e82: 2023-08-02T15:12:53.621Z [INFO] server-180d44c5-13c8-1722-a927-e21b06e48e82: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-180d44c5-13c8-1722-a927-e21b06e48e82 Address:180d44c5-13c8-1722-a927-e21b06e48e82} {Suffrage:Voter ID:server-2c5e301c-60c6-626a-ce4c-88387582e874 Address:2c5e301c-60c6-626a-ce4c-88387582e874} {Suffrage:Voter ID:server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9 Address:d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9}]" testLoggerAdapter verbose: server-2c5e301c-60c6-626a-ce4c-88387582e874: 2023-08-02T15:12:53.621Z [INFO] server-2c5e301c-60c6-626a-ce4c-88387582e874: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-180d44c5-13c8-1722-a927-e21b06e48e82 Address:180d44c5-13c8-1722-a927-e21b06e48e82} {Suffrage:Voter ID:server-2c5e301c-60c6-626a-ce4c-88387582e874 Address:2c5e301c-60c6-626a-ce4c-88387582e874} {Suffrage:Voter ID:server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9 Address:d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9}]" testing.go:149: server-2c5e301c-60c6-626a-ce4c-88387582e874: 2023-08-02T15:12:53.621Z [INFO] server-2c5e301c-60c6-626a-ce4c-88387582e874: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-180d44c5-13c8-1722-a927-e21b06e48e82 Address:180d44c5-13c8-1722-a927-e21b06e48e82} {Suffrage:Voter ID:server-2c5e301c-60c6-626a-ce4c-88387582e874 Address:2c5e301c-60c6-626a-ce4c-88387582e874} {Suffrage:Voter ID:server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9 Address:d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9}]" testLoggerAdapter verbose: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.622Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-180d44c5-13c8-1722-a927-e21b06e48e82 Address:180d44c5-13c8-1722-a927-e21b06e48e82} {Suffrage:Voter ID:server-2c5e301c-60c6-626a-ce4c-88387582e874 Address:2c5e301c-60c6-626a-ce4c-88387582e874} {Suffrage:Voter ID:server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9 Address:d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9}]" testing.go:149: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.622Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-180d44c5-13c8-1722-a927-e21b06e48e82 Address:180d44c5-13c8-1722-a927-e21b06e48e82} {Suffrage:Voter ID:server-2c5e301c-60c6-626a-ce4c-88387582e874 Address:2c5e301c-60c6-626a-ce4c-88387582e874} {Suffrage:Voter ID:server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9 Address:d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:53.622Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:12:53.622Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-180d44c5-13c8-1722-a927-e21b06e48e82: 2023-08-02T15:12:53.622Z [INFO] server-180d44c5-13c8-1722-a927-e21b06e48e82: entering follower state: follower="Node at 180d44c5-13c8-1722-a927-e21b06e48e82 [Follower]" leader= testing.go:149: server-180d44c5-13c8-1722-a927-e21b06e48e82: 2023-08-02T15:12:53.622Z [INFO] server-180d44c5-13c8-1722-a927-e21b06e48e82: entering follower state: follower="Node at 180d44c5-13c8-1722-a927-e21b06e48e82 [Follower]" leader= testLoggerAdapter verbose: server-2c5e301c-60c6-626a-ce4c-88387582e874: 2023-08-02T15:12:53.622Z [INFO] server-2c5e301c-60c6-626a-ce4c-88387582e874: entering follower state: follower="Node at 2c5e301c-60c6-626a-ce4c-88387582e874 [Follower]" leader= testing.go:149: server-2c5e301c-60c6-626a-ce4c-88387582e874: 2023-08-02T15:12:53.622Z [INFO] server-2c5e301c-60c6-626a-ce4c-88387582e874: entering follower state: follower="Node at 2c5e301c-60c6-626a-ce4c-88387582e874 [Follower]" leader= testLoggerAdapter verbose: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.622Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: entering follower state: follower="Node at d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9 [Follower]" leader= testing.go:149: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.622Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: entering follower state: follower="Node at d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9 [Follower]" leader= testLoggerAdapter verbose: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.678Z [WARN] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.678Z [WARN] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.678Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: entering candidate state: node="Node at d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9 [Candidate]" term=2 testing.go:149: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.678Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: entering candidate state: node="Node at d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9 [Candidate]" term=2 testLoggerAdapter verbose: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.678Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: election won: tally=2 testing.go:149: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.678Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: election won: tally=2 testLoggerAdapter verbose: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.678Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: entering leader state: leader="Node at d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9 [Leader]" testing.go:149: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.678Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: entering leader state: leader="Node at d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9 [Leader]" testLoggerAdapter verbose: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.678Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: added peer, starting replication: peer=server-180d44c5-13c8-1722-a927-e21b06e48e82 testing.go:149: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.678Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: added peer, starting replication: peer=server-180d44c5-13c8-1722-a927-e21b06e48e82 testLoggerAdapter verbose: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.678Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: added peer, starting replication: peer=server-2c5e301c-60c6-626a-ce4c-88387582e874 testing.go:149: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.678Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: added peer, starting replication: peer=server-2c5e301c-60c6-626a-ce4c-88387582e874 testLoggerAdapter verbose: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.678Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: pipelining replication: peer="{Voter server-2c5e301c-60c6-626a-ce4c-88387582e874 2c5e301c-60c6-626a-ce4c-88387582e874}" testing.go:149: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.678Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: pipelining replication: peer="{Voter server-2c5e301c-60c6-626a-ce4c-88387582e874 2c5e301c-60c6-626a-ce4c-88387582e874}" testLoggerAdapter verbose: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.678Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: pipelining replication: peer="{Voter server-180d44c5-13c8-1722-a927-e21b06e48e82 180d44c5-13c8-1722-a927-e21b06e48e82}" testing.go:149: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.678Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: pipelining replication: peer="{Voter server-180d44c5-13c8-1722-a927-e21b06e48e82 180d44c5-13c8-1722-a927-e21b06e48e82}" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:53.784Z [INFO] stable state for Leader reached at 2023-08-02 15:12:53.678509592 +0000 UTC m=+5.774445927 (1 nodes), 56.300257ms from start of poll, 56.860054ms from cluster start. Timeout at 2023-08-02 15:12:53.784065388 +0000 UTC m=+5.880001758, 105.555831ms after stability testing.go:149: cluster: 2023-08-02T15:12:53.784Z [INFO] stable state for Leader reached at 2023-08-02 15:12:53.678509592 +0000 UTC m=+5.774445927 (1 nodes), 56.300257ms from start of poll, 56.860054ms from cluster start. Timeout at 2023-08-02 15:12:53.784065388 +0000 UTC m=+5.880001758, 105.555831ms after stability testLoggerAdapter verbose: cluster: 2023-08-02T15:12:53.784Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-08-02T15:12:53.784Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2023-08-02T15:12:53.889Z [INFO] stable state for Follower reached at 2023-08-02 15:12:53.78451474 +0000 UTC m=+5.880451076 (2 nodes), 1.627µs from start of poll, 162.865203ms from cluster start. Timeout at 2023-08-02 15:12:53.889579491 +0000 UTC m=+5.985515846, 105.06477ms after stability testing.go:149: cluster: 2023-08-02T15:12:53.889Z [INFO] stable state for Follower reached at 2023-08-02 15:12:53.78451474 +0000 UTC m=+5.880451076 (2 nodes), 1.627µs from start of poll, 162.865203ms from cluster start. Timeout at 2023-08-02 15:12:53.889579491 +0000 UTC m=+5.985515846, 105.06477ms after stability testLoggerAdapter verbose: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.889Z [ERROR] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: failed to pipeline appendEntries: peer="{Voter server-180d44c5-13c8-1722-a927-e21b06e48e82 180d44c5-13c8-1722-a927-e21b06e48e82}" error="append pipeline closed" testing.go:149: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.889Z [ERROR] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: failed to pipeline appendEntries: peer="{Voter server-180d44c5-13c8-1722-a927-e21b06e48e82 180d44c5-13c8-1722-a927-e21b06e48e82}" error="append pipeline closed" testLoggerAdapter verbose: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.889Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: aborting pipeline replication: peer="{Voter server-180d44c5-13c8-1722-a927-e21b06e48e82 180d44c5-13c8-1722-a927-e21b06e48e82}" testing.go:149: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.889Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: aborting pipeline replication: peer="{Voter server-180d44c5-13c8-1722-a927-e21b06e48e82 180d44c5-13c8-1722-a927-e21b06e48e82}" testLoggerAdapter verbose: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.889Z [ERROR] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: failed to appendEntries to: peer="{Voter server-180d44c5-13c8-1722-a927-e21b06e48e82 180d44c5-13c8-1722-a927-e21b06e48e82}" error="failed to connect to peer: 180d44c5-13c8-1722-a927-e21b06e48e82" testing.go:149: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.889Z [ERROR] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: failed to appendEntries to: peer="{Voter server-180d44c5-13c8-1722-a927-e21b06e48e82 180d44c5-13c8-1722-a927-e21b06e48e82}" error="failed to connect to peer: 180d44c5-13c8-1722-a927-e21b06e48e82" raft_test.go:420: [INFO] Finished apply without behind follower testLoggerAdapter verbose: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.900Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: pipelining replication: peer="{Voter server-180d44c5-13c8-1722-a927-e21b06e48e82 180d44c5-13c8-1722-a927-e21b06e48e82}" testing.go:149: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:53.900Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: pipelining replication: peer="{Voter server-180d44c5-13c8-1722-a927-e21b06e48e82 180d44c5-13c8-1722-a927-e21b06e48e82}" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:53.908Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:12:53.908Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-08-02T15:12:54.014Z [INFO] stable state for Leader reached at 2023-08-02 15:12:53.908642736 +0000 UTC m=+6.004579063 (1 nodes), 980ns from start of poll, 286.99319ms from cluster start. Timeout at 2023-08-02 15:12:54.01426461 +0000 UTC m=+6.110200937, 105.621874ms after stability testing.go:149: cluster: 2023-08-02T15:12:54.014Z [INFO] stable state for Leader reached at 2023-08-02 15:12:53.908642736 +0000 UTC m=+6.004579063 (1 nodes), 980ns from start of poll, 286.99319ms from cluster start. Timeout at 2023-08-02 15:12:54.01426461 +0000 UTC m=+6.110200937, 105.621874ms after stability testLoggerAdapter verbose: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:54.014Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: aborting pipeline replication: peer="{Voter server-2c5e301c-60c6-626a-ce4c-88387582e874 2c5e301c-60c6-626a-ce4c-88387582e874}" testing.go:149: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:54.014Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: aborting pipeline replication: peer="{Voter server-2c5e301c-60c6-626a-ce4c-88387582e874 2c5e301c-60c6-626a-ce4c-88387582e874}" testLoggerAdapter verbose: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:54.014Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: aborting pipeline replication: peer="{Voter server-180d44c5-13c8-1722-a927-e21b06e48e82 180d44c5-13c8-1722-a927-e21b06e48e82}" testing.go:149: server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: 2023-08-02T15:12:54.014Z [INFO] server-d656cb3e-f4ae-cb5a-5fe2-ec7e00855dd9: aborting pipeline replication: peer="{Voter server-180d44c5-13c8-1722-a927-e21b06e48e82 180d44c5-13c8-1722-a927-e21b06e48e82}" --- PASS: TestRaft_BehindFollower (0.40s) === RUN TestRaft_ApplyNonLeader testLoggerAdapter verbose: server-a5eb522d-6f8d-744e-6cbc-1743e3136578: 2023-08-02T15:12:54.018Z [INFO] server-a5eb522d-6f8d-744e-6cbc-1743e3136578: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a5eb522d-6f8d-744e-6cbc-1743e3136578 Address:a5eb522d-6f8d-744e-6cbc-1743e3136578} {Suffrage:Voter ID:server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4 Address:24fa25a3-0bd1-da1e-da8f-2325522ee3f4} {Suffrage:Voter ID:server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1 Address:cf2e5c9d-5e3e-7ece-b910-4964c0c364e1}]" testing.go:149: server-a5eb522d-6f8d-744e-6cbc-1743e3136578: 2023-08-02T15:12:54.018Z [INFO] server-a5eb522d-6f8d-744e-6cbc-1743e3136578: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a5eb522d-6f8d-744e-6cbc-1743e3136578 Address:a5eb522d-6f8d-744e-6cbc-1743e3136578} {Suffrage:Voter ID:server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4 Address:24fa25a3-0bd1-da1e-da8f-2325522ee3f4} {Suffrage:Voter ID:server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1 Address:cf2e5c9d-5e3e-7ece-b910-4964c0c364e1}]" testLoggerAdapter verbose: server-a5eb522d-6f8d-744e-6cbc-1743e3136578: 2023-08-02T15:12:54.018Z [INFO] server-a5eb522d-6f8d-744e-6cbc-1743e3136578: entering follower state: follower="Node at a5eb522d-6f8d-744e-6cbc-1743e3136578 [Follower]" leader= testing.go:149: server-a5eb522d-6f8d-744e-6cbc-1743e3136578: 2023-08-02T15:12:54.018Z [INFO] server-a5eb522d-6f8d-744e-6cbc-1743e3136578: entering follower state: follower="Node at a5eb522d-6f8d-744e-6cbc-1743e3136578 [Follower]" leader= testLoggerAdapter verbose: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.018Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a5eb522d-6f8d-744e-6cbc-1743e3136578 Address:a5eb522d-6f8d-744e-6cbc-1743e3136578} {Suffrage:Voter ID:server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4 Address:24fa25a3-0bd1-da1e-da8f-2325522ee3f4} {Suffrage:Voter ID:server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1 Address:cf2e5c9d-5e3e-7ece-b910-4964c0c364e1}]" testing.go:149: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.018Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a5eb522d-6f8d-744e-6cbc-1743e3136578 Address:a5eb522d-6f8d-744e-6cbc-1743e3136578} {Suffrage:Voter ID:server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4 Address:24fa25a3-0bd1-da1e-da8f-2325522ee3f4} {Suffrage:Voter ID:server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1 Address:cf2e5c9d-5e3e-7ece-b910-4964c0c364e1}]" testLoggerAdapter verbose: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.018Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: entering follower state: follower="Node at 24fa25a3-0bd1-da1e-da8f-2325522ee3f4 [Follower]" leader= testing.go:149: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.018Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: entering follower state: follower="Node at 24fa25a3-0bd1-da1e-da8f-2325522ee3f4 [Follower]" leader= testLoggerAdapter verbose: server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1: 2023-08-02T15:12:54.018Z [INFO] server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a5eb522d-6f8d-744e-6cbc-1743e3136578 Address:a5eb522d-6f8d-744e-6cbc-1743e3136578} {Suffrage:Voter ID:server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4 Address:24fa25a3-0bd1-da1e-da8f-2325522ee3f4} {Suffrage:Voter ID:server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1 Address:cf2e5c9d-5e3e-7ece-b910-4964c0c364e1}]" testing.go:149: server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1: 2023-08-02T15:12:54.018Z [INFO] server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a5eb522d-6f8d-744e-6cbc-1743e3136578 Address:a5eb522d-6f8d-744e-6cbc-1743e3136578} {Suffrage:Voter ID:server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4 Address:24fa25a3-0bd1-da1e-da8f-2325522ee3f4} {Suffrage:Voter ID:server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1 Address:cf2e5c9d-5e3e-7ece-b910-4964c0c364e1}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:54.018Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:12:54.018Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1: 2023-08-02T15:12:54.018Z [INFO] server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1: entering follower state: follower="Node at cf2e5c9d-5e3e-7ece-b910-4964c0c364e1 [Follower]" leader= testing.go:149: server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1: 2023-08-02T15:12:54.018Z [INFO] server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1: entering follower state: follower="Node at cf2e5c9d-5e3e-7ece-b910-4964c0c364e1 [Follower]" leader= testLoggerAdapter verbose: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.080Z [WARN] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.080Z [WARN] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.080Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: entering candidate state: node="Node at 24fa25a3-0bd1-da1e-da8f-2325522ee3f4 [Candidate]" term=2 testing.go:149: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.080Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: entering candidate state: node="Node at 24fa25a3-0bd1-da1e-da8f-2325522ee3f4 [Candidate]" term=2 testLoggerAdapter verbose: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.080Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: election won: tally=2 testing.go:149: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.080Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: election won: tally=2 testLoggerAdapter verbose: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.080Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: entering leader state: leader="Node at 24fa25a3-0bd1-da1e-da8f-2325522ee3f4 [Leader]" testing.go:149: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.080Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: entering leader state: leader="Node at 24fa25a3-0bd1-da1e-da8f-2325522ee3f4 [Leader]" testLoggerAdapter verbose: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.080Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: added peer, starting replication: peer=server-a5eb522d-6f8d-744e-6cbc-1743e3136578 testing.go:149: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.080Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: added peer, starting replication: peer=server-a5eb522d-6f8d-744e-6cbc-1743e3136578 testLoggerAdapter verbose: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.080Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: added peer, starting replication: peer=server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1 testing.go:149: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.080Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: added peer, starting replication: peer=server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1 testLoggerAdapter verbose: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.080Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: pipelining replication: peer="{Voter server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1 cf2e5c9d-5e3e-7ece-b910-4964c0c364e1}" testing.go:149: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.080Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: pipelining replication: peer="{Voter server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1 cf2e5c9d-5e3e-7ece-b910-4964c0c364e1}" testLoggerAdapter verbose: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.080Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: pipelining replication: peer="{Voter server-a5eb522d-6f8d-744e-6cbc-1743e3136578 a5eb522d-6f8d-744e-6cbc-1743e3136578}" testing.go:149: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.080Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: pipelining replication: peer="{Voter server-a5eb522d-6f8d-744e-6cbc-1743e3136578 a5eb522d-6f8d-744e-6cbc-1743e3136578}" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:54.185Z [INFO] stable state for Leader reached at 2023-08-02 15:12:54.080785978 +0000 UTC m=+6.176722325 (1 nodes), 61.9597ms from start of poll, 62.977816ms from cluster start. Timeout at 2023-08-02 15:12:54.185876214 +0000 UTC m=+6.281812539, 105.090214ms after stability testing.go:149: cluster: 2023-08-02T15:12:54.185Z [INFO] stable state for Leader reached at 2023-08-02 15:12:54.080785978 +0000 UTC m=+6.176722325 (1 nodes), 61.9597ms from start of poll, 62.977816ms from cluster start. Timeout at 2023-08-02 15:12:54.185876214 +0000 UTC m=+6.281812539, 105.090214ms after stability testLoggerAdapter verbose: cluster: 2023-08-02T15:12:54.186Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-08-02T15:12:54.186Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2023-08-02T15:12:54.291Z [INFO] stable state for Follower reached at 2023-08-02 15:12:54.186048148 +0000 UTC m=+6.281984476 (2 nodes), 2.491µs from start of poll, 168.239967ms from cluster start. Timeout at 2023-08-02 15:12:54.291147839 +0000 UTC m=+6.387084175, 105.099699ms after stability testing.go:149: cluster: 2023-08-02T15:12:54.291Z [INFO] stable state for Follower reached at 2023-08-02 15:12:54.186048148 +0000 UTC m=+6.281984476 (2 nodes), 2.491µs from start of poll, 168.239967ms from cluster start. Timeout at 2023-08-02 15:12:54.291147839 +0000 UTC m=+6.387084175, 105.099699ms after stability testLoggerAdapter verbose: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.291Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: aborting pipeline replication: peer="{Voter server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1 cf2e5c9d-5e3e-7ece-b910-4964c0c364e1}" testing.go:149: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.291Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: aborting pipeline replication: peer="{Voter server-cf2e5c9d-5e3e-7ece-b910-4964c0c364e1 cf2e5c9d-5e3e-7ece-b910-4964c0c364e1}" testLoggerAdapter verbose: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.291Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: aborting pipeline replication: peer="{Voter server-a5eb522d-6f8d-744e-6cbc-1743e3136578 a5eb522d-6f8d-744e-6cbc-1743e3136578}" testing.go:149: server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: 2023-08-02T15:12:54.291Z [INFO] server-24fa25a3-0bd1-da1e-da8f-2325522ee3f4: aborting pipeline replication: peer="{Voter server-a5eb522d-6f8d-744e-6cbc-1743e3136578 a5eb522d-6f8d-744e-6cbc-1743e3136578}" --- PASS: TestRaft_ApplyNonLeader (0.27s) === RUN TestRaft_ApplyConcurrent testLoggerAdapter verbose: server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67: 2023-08-02T15:12:54.292Z [INFO] server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67 Address:e6f973fe-395d-dd10-ecc1-65aa2c5e7d67} {Suffrage:Voter ID:server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e Address:dc4de7ca-3447-cbf4-6782-8826c8c93c9e} {Suffrage:Voter ID:server-e3107dc3-d161-48d5-9aa6-82092b13434f Address:e3107dc3-d161-48d5-9aa6-82092b13434f}]" testing.go:149: server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67: 2023-08-02T15:12:54.292Z [INFO] server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67 Address:e6f973fe-395d-dd10-ecc1-65aa2c5e7d67} {Suffrage:Voter ID:server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e Address:dc4de7ca-3447-cbf4-6782-8826c8c93c9e} {Suffrage:Voter ID:server-e3107dc3-d161-48d5-9aa6-82092b13434f Address:e3107dc3-d161-48d5-9aa6-82092b13434f}]" testLoggerAdapter verbose: server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e: 2023-08-02T15:12:54.292Z [INFO] server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67 Address:e6f973fe-395d-dd10-ecc1-65aa2c5e7d67} {Suffrage:Voter ID:server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e Address:dc4de7ca-3447-cbf4-6782-8826c8c93c9e} {Suffrage:Voter ID:server-e3107dc3-d161-48d5-9aa6-82092b13434f Address:e3107dc3-d161-48d5-9aa6-82092b13434f}]" testing.go:149: server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e: 2023-08-02T15:12:54.292Z [INFO] server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67 Address:e6f973fe-395d-dd10-ecc1-65aa2c5e7d67} {Suffrage:Voter ID:server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e Address:dc4de7ca-3447-cbf4-6782-8826c8c93c9e} {Suffrage:Voter ID:server-e3107dc3-d161-48d5-9aa6-82092b13434f Address:e3107dc3-d161-48d5-9aa6-82092b13434f}]" testLoggerAdapter verbose: server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67: 2023-08-02T15:12:54.292Z [INFO] server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67: entering follower state: follower="Node at e6f973fe-395d-dd10-ecc1-65aa2c5e7d67 [Follower]" leader= testLoggerAdapter verbose: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.292Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67 Address:e6f973fe-395d-dd10-ecc1-65aa2c5e7d67} {Suffrage:Voter ID:server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e Address:dc4de7ca-3447-cbf4-6782-8826c8c93c9e} {Suffrage:Voter ID:server-e3107dc3-d161-48d5-9aa6-82092b13434f Address:e3107dc3-d161-48d5-9aa6-82092b13434f}]" testing.go:149: server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67: 2023-08-02T15:12:54.292Z [INFO] server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67: entering follower state: follower="Node at e6f973fe-395d-dd10-ecc1-65aa2c5e7d67 [Follower]" leader= testing.go:149: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.292Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67 Address:e6f973fe-395d-dd10-ecc1-65aa2c5e7d67} {Suffrage:Voter ID:server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e Address:dc4de7ca-3447-cbf4-6782-8826c8c93c9e} {Suffrage:Voter ID:server-e3107dc3-d161-48d5-9aa6-82092b13434f Address:e3107dc3-d161-48d5-9aa6-82092b13434f}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:54.292Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:12:54.292Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.292Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: entering follower state: follower="Node at e3107dc3-d161-48d5-9aa6-82092b13434f [Follower]" leader= testing.go:149: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.292Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: entering follower state: follower="Node at e3107dc3-d161-48d5-9aa6-82092b13434f [Follower]" leader= testLoggerAdapter verbose: server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e: 2023-08-02T15:12:54.292Z [INFO] server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e: entering follower state: follower="Node at dc4de7ca-3447-cbf4-6782-8826c8c93c9e [Follower]" leader= testing.go:149: server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e: 2023-08-02T15:12:54.292Z [INFO] server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e: entering follower state: follower="Node at dc4de7ca-3447-cbf4-6782-8826c8c93c9e [Follower]" leader= testLoggerAdapter verbose: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.437Z [WARN] server-e3107dc3-d161-48d5-9aa6-82092b13434f: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.437Z [WARN] server-e3107dc3-d161-48d5-9aa6-82092b13434f: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.437Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: entering candidate state: node="Node at e3107dc3-d161-48d5-9aa6-82092b13434f [Candidate]" term=2 testing.go:149: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.437Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: entering candidate state: node="Node at e3107dc3-d161-48d5-9aa6-82092b13434f [Candidate]" term=2 testLoggerAdapter verbose: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.437Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: election won: tally=2 testing.go:149: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.437Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: election won: tally=2 testLoggerAdapter verbose: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.437Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: entering leader state: leader="Node at e3107dc3-d161-48d5-9aa6-82092b13434f [Leader]" testing.go:149: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.437Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: entering leader state: leader="Node at e3107dc3-d161-48d5-9aa6-82092b13434f [Leader]" testLoggerAdapter verbose: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.437Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: added peer, starting replication: peer=server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67 testing.go:149: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.437Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: added peer, starting replication: peer=server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67 testLoggerAdapter verbose: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.437Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: added peer, starting replication: peer=server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e testing.go:149: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.437Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: added peer, starting replication: peer=server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e testLoggerAdapter verbose: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.437Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: pipelining replication: peer="{Voter server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e dc4de7ca-3447-cbf4-6782-8826c8c93c9e}" testing.go:149: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.437Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: pipelining replication: peer="{Voter server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e dc4de7ca-3447-cbf4-6782-8826c8c93c9e}" testLoggerAdapter verbose: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.437Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: pipelining replication: peer="{Voter server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67 e6f973fe-395d-dd10-ecc1-65aa2c5e7d67}" testing.go:149: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.437Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: pipelining replication: peer="{Voter server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67 e6f973fe-395d-dd10-ecc1-65aa2c5e7d67}" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:54.642Z [INFO] stable state for Leader reached at 2023-08-02 15:12:54.437844916 +0000 UTC m=+6.533781249 (1 nodes), 144.904933ms from start of poll, 145.444755ms from cluster start. Timeout at 2023-08-02 15:12:54.642921318 +0000 UTC m=+6.738857645, 205.076396ms after stability testing.go:149: cluster: 2023-08-02T15:12:54.642Z [INFO] stable state for Leader reached at 2023-08-02 15:12:54.437844916 +0000 UTC m=+6.533781249 (1 nodes), 144.904933ms from start of poll, 145.444755ms from cluster start. Timeout at 2023-08-02 15:12:54.642921318 +0000 UTC m=+6.738857645, 205.076396ms after stability testLoggerAdapter verbose: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.656Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: aborting pipeline replication: peer="{Voter server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e dc4de7ca-3447-cbf4-6782-8826c8c93c9e}" testing.go:149: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.656Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: aborting pipeline replication: peer="{Voter server-dc4de7ca-3447-cbf4-6782-8826c8c93c9e dc4de7ca-3447-cbf4-6782-8826c8c93c9e}" testLoggerAdapter verbose: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.656Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: aborting pipeline replication: peer="{Voter server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67 e6f973fe-395d-dd10-ecc1-65aa2c5e7d67}" testing.go:149: server-e3107dc3-d161-48d5-9aa6-82092b13434f: 2023-08-02T15:12:54.656Z [INFO] server-e3107dc3-d161-48d5-9aa6-82092b13434f: aborting pipeline replication: peer="{Voter server-e6f973fe-395d-dd10-ecc1-65aa2c5e7d67 e6f973fe-395d-dd10-ecc1-65aa2c5e7d67}" --- PASS: TestRaft_ApplyConcurrent (0.36s) === RUN TestRaft_ApplyConcurrent_Timeout testLoggerAdapter verbose: server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: 2023-08-02T15:12:54.657Z [INFO] server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b Address:f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b}]" testing.go:149: server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: 2023-08-02T15:12:54.657Z [INFO] server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b Address:f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:54.657Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:12:54.657Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: 2023-08-02T15:12:54.657Z [INFO] server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: entering follower state: follower="Node at f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b [Follower]" leader= testing.go:149: server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: 2023-08-02T15:12:54.657Z [INFO] server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: entering follower state: follower="Node at f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b [Follower]" leader= testLoggerAdapter verbose: server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: 2023-08-02T15:12:54.846Z [WARN] server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: 2023-08-02T15:12:54.846Z [WARN] server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: 2023-08-02T15:12:54.846Z [INFO] server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: entering candidate state: node="Node at f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b [Candidate]" term=2 testing.go:149: server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: 2023-08-02T15:12:54.846Z [INFO] server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: entering candidate state: node="Node at f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b [Candidate]" term=2 testLoggerAdapter verbose: server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: 2023-08-02T15:12:54.846Z [INFO] server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: election won: tally=1 testing.go:149: server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: 2023-08-02T15:12:54.846Z [INFO] server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: election won: tally=1 testLoggerAdapter verbose: server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: 2023-08-02T15:12:54.846Z [INFO] server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: entering leader state: leader="Node at f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b [Leader]" testing.go:149: server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: 2023-08-02T15:12:54.846Z [INFO] server-f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b: entering leader state: leader="Node at f4cc8cf4-8763-ab25-0f4a-fd87d111ee1b [Leader]" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:55.048Z [INFO] stable state for Leader reached at 2023-08-02 15:12:54.846906416 +0000 UTC m=+6.942842751 (1 nodes), 189.72564ms from start of poll, 189.94733ms from cluster start. Timeout at 2023-08-02 15:12:55.048355248 +0000 UTC m=+7.144291579, 201.448828ms after stability testing.go:149: cluster: 2023-08-02T15:12:55.048Z [INFO] stable state for Leader reached at 2023-08-02 15:12:54.846906416 +0000 UTC m=+6.942842751 (1 nodes), 189.72564ms from start of poll, 189.94733ms from cluster start. Timeout at 2023-08-02 15:12:55.048355248 +0000 UTC m=+7.144291579, 201.448828ms after stability --- PASS: TestRaft_ApplyConcurrent_Timeout (0.39s) === RUN TestRaft_JoinNode testLoggerAdapter verbose: server-403d52cd-d42b-d401-2e45-a7e7a20a02d3: 2023-08-02T15:12:55.051Z [INFO] server-403d52cd-d42b-d401-2e45-a7e7a20a02d3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-403d52cd-d42b-d401-2e45-a7e7a20a02d3 Address:403d52cd-d42b-d401-2e45-a7e7a20a02d3} {Suffrage:Voter ID:server-f311d765-c353-90cb-94a0-c34bfa0aa721 Address:f311d765-c353-90cb-94a0-c34bfa0aa721}]" testing.go:149: server-403d52cd-d42b-d401-2e45-a7e7a20a02d3: 2023-08-02T15:12:55.051Z [INFO] server-403d52cd-d42b-d401-2e45-a7e7a20a02d3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-403d52cd-d42b-d401-2e45-a7e7a20a02d3 Address:403d52cd-d42b-d401-2e45-a7e7a20a02d3} {Suffrage:Voter ID:server-f311d765-c353-90cb-94a0-c34bfa0aa721 Address:f311d765-c353-90cb-94a0-c34bfa0aa721}]" testLoggerAdapter verbose: server-403d52cd-d42b-d401-2e45-a7e7a20a02d3: 2023-08-02T15:12:55.051Z [INFO] server-403d52cd-d42b-d401-2e45-a7e7a20a02d3: entering follower state: follower="Node at 403d52cd-d42b-d401-2e45-a7e7a20a02d3 [Follower]" leader= testLoggerAdapter verbose: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.051Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-403d52cd-d42b-d401-2e45-a7e7a20a02d3 Address:403d52cd-d42b-d401-2e45-a7e7a20a02d3} {Suffrage:Voter ID:server-f311d765-c353-90cb-94a0-c34bfa0aa721 Address:f311d765-c353-90cb-94a0-c34bfa0aa721}]" testing.go:149: server-403d52cd-d42b-d401-2e45-a7e7a20a02d3: 2023-08-02T15:12:55.051Z [INFO] server-403d52cd-d42b-d401-2e45-a7e7a20a02d3: entering follower state: follower="Node at 403d52cd-d42b-d401-2e45-a7e7a20a02d3 [Follower]" leader= testing.go:149: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.051Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-403d52cd-d42b-d401-2e45-a7e7a20a02d3 Address:403d52cd-d42b-d401-2e45-a7e7a20a02d3} {Suffrage:Voter ID:server-f311d765-c353-90cb-94a0-c34bfa0aa721 Address:f311d765-c353-90cb-94a0-c34bfa0aa721}]" testLoggerAdapter verbose: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.051Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: entering follower state: follower="Node at f311d765-c353-90cb-94a0-c34bfa0aa721 [Follower]" leader= testing.go:149: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.051Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: entering follower state: follower="Node at f311d765-c353-90cb-94a0-c34bfa0aa721 [Follower]" leader= testLoggerAdapter verbose: server-69de86e2-46d2-260c-3434-5955a9711433: 2023-08-02T15:12:55.051Z [INFO] server-69de86e2-46d2-260c-3434-5955a9711433: initial configuration: index=0 servers=[] testing.go:149: server-69de86e2-46d2-260c-3434-5955a9711433: 2023-08-02T15:12:55.051Z [INFO] server-69de86e2-46d2-260c-3434-5955a9711433: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2023-08-02T15:12:55.051Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:12:55.051Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-69de86e2-46d2-260c-3434-5955a9711433: 2023-08-02T15:12:55.051Z [INFO] server-69de86e2-46d2-260c-3434-5955a9711433: entering follower state: follower="Node at 69de86e2-46d2-260c-3434-5955a9711433 [Follower]" leader= testing.go:149: server-69de86e2-46d2-260c-3434-5955a9711433: 2023-08-02T15:12:55.051Z [INFO] server-69de86e2-46d2-260c-3434-5955a9711433: entering follower state: follower="Node at 69de86e2-46d2-260c-3434-5955a9711433 [Follower]" leader= testLoggerAdapter verbose: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.106Z [WARN] server-f311d765-c353-90cb-94a0-c34bfa0aa721: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.106Z [WARN] server-f311d765-c353-90cb-94a0-c34bfa0aa721: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.106Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: entering candidate state: node="Node at f311d765-c353-90cb-94a0-c34bfa0aa721 [Candidate]" term=2 testing.go:149: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.106Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: entering candidate state: node="Node at f311d765-c353-90cb-94a0-c34bfa0aa721 [Candidate]" term=2 testLoggerAdapter verbose: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.111Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: election won: tally=2 testing.go:149: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.111Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: election won: tally=2 testLoggerAdapter verbose: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.111Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: entering leader state: leader="Node at f311d765-c353-90cb-94a0-c34bfa0aa721 [Leader]" testing.go:149: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.111Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: entering leader state: leader="Node at f311d765-c353-90cb-94a0-c34bfa0aa721 [Leader]" testLoggerAdapter verbose: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.111Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: added peer, starting replication: peer=server-403d52cd-d42b-d401-2e45-a7e7a20a02d3 testing.go:149: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.111Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: added peer, starting replication: peer=server-403d52cd-d42b-d401-2e45-a7e7a20a02d3 testLoggerAdapter verbose: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.112Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: pipelining replication: peer="{Voter server-403d52cd-d42b-d401-2e45-a7e7a20a02d3 403d52cd-d42b-d401-2e45-a7e7a20a02d3}" testing.go:149: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.112Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: pipelining replication: peer="{Voter server-403d52cd-d42b-d401-2e45-a7e7a20a02d3 403d52cd-d42b-d401-2e45-a7e7a20a02d3}" testLoggerAdapter verbose: server-69de86e2-46d2-260c-3434-5955a9711433: 2023-08-02T15:12:55.118Z [WARN] server-69de86e2-46d2-260c-3434-5955a9711433: no known peers, aborting election testing.go:149: server-69de86e2-46d2-260c-3434-5955a9711433: 2023-08-02T15:12:55.118Z [WARN] server-69de86e2-46d2-260c-3434-5955a9711433: no known peers, aborting election testLoggerAdapter verbose: cluster: 2023-08-02T15:12:55.217Z [INFO] stable state for Leader reached at 2023-08-02 15:12:55.111869237 +0000 UTC m=+7.207805596 (1 nodes), 60.159062ms from start of poll, 60.826267ms from cluster start. Timeout at 2023-08-02 15:12:55.216979364 +0000 UTC m=+7.312915735, 105.110139ms after stability testing.go:149: cluster: 2023-08-02T15:12:55.217Z [INFO] stable state for Leader reached at 2023-08-02 15:12:55.111869237 +0000 UTC m=+7.207805596 (1 nodes), 60.159062ms from start of poll, 60.826267ms from cluster start. Timeout at 2023-08-02 15:12:55.216979364 +0000 UTC m=+7.312915735, 105.110139ms after stability testLoggerAdapter verbose: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.217Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: updating configuration: command=AddStaging server-id=server-69de86e2-46d2-260c-3434-5955a9711433 server-addr=69de86e2-46d2-260c-3434-5955a9711433 servers="[{Suffrage:Voter ID:server-403d52cd-d42b-d401-2e45-a7e7a20a02d3 Address:403d52cd-d42b-d401-2e45-a7e7a20a02d3} {Suffrage:Voter ID:server-f311d765-c353-90cb-94a0-c34bfa0aa721 Address:f311d765-c353-90cb-94a0-c34bfa0aa721} {Suffrage:Voter ID:server-69de86e2-46d2-260c-3434-5955a9711433 Address:69de86e2-46d2-260c-3434-5955a9711433}]" testing.go:149: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.217Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: updating configuration: command=AddStaging server-id=server-69de86e2-46d2-260c-3434-5955a9711433 server-addr=69de86e2-46d2-260c-3434-5955a9711433 servers="[{Suffrage:Voter ID:server-403d52cd-d42b-d401-2e45-a7e7a20a02d3 Address:403d52cd-d42b-d401-2e45-a7e7a20a02d3} {Suffrage:Voter ID:server-f311d765-c353-90cb-94a0-c34bfa0aa721 Address:f311d765-c353-90cb-94a0-c34bfa0aa721} {Suffrage:Voter ID:server-69de86e2-46d2-260c-3434-5955a9711433 Address:69de86e2-46d2-260c-3434-5955a9711433}]" testLoggerAdapter verbose: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.217Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: added peer, starting replication: peer=server-69de86e2-46d2-260c-3434-5955a9711433 testing.go:149: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.217Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: added peer, starting replication: peer=server-69de86e2-46d2-260c-3434-5955a9711433 testLoggerAdapter verbose: server-69de86e2-46d2-260c-3434-5955a9711433: 2023-08-02T15:12:55.217Z [WARN] server-69de86e2-46d2-260c-3434-5955a9711433: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: server-69de86e2-46d2-260c-3434-5955a9711433: 2023-08-02T15:12:55.217Z [WARN] server-69de86e2-46d2-260c-3434-5955a9711433: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.217Z [WARN] server-f311d765-c353-90cb-94a0-c34bfa0aa721: appendEntries rejected, sending older logs: peer="{Voter server-69de86e2-46d2-260c-3434-5955a9711433 69de86e2-46d2-260c-3434-5955a9711433}" next=1 testing.go:149: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.217Z [WARN] server-f311d765-c353-90cb-94a0-c34bfa0aa721: appendEntries rejected, sending older logs: peer="{Voter server-69de86e2-46d2-260c-3434-5955a9711433 69de86e2-46d2-260c-3434-5955a9711433}" next=1 testLoggerAdapter verbose: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.217Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: pipelining replication: peer="{Voter server-69de86e2-46d2-260c-3434-5955a9711433 69de86e2-46d2-260c-3434-5955a9711433}" testing.go:149: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.217Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: pipelining replication: peer="{Voter server-69de86e2-46d2-260c-3434-5955a9711433 69de86e2-46d2-260c-3434-5955a9711433}" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:55.217Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:12:55.217Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-08-02T15:12:55.323Z [INFO] stable state for Leader reached at 2023-08-02 15:12:55.217828545 +0000 UTC m=+7.313764871 (1 nodes), 907ns from start of poll, 166.785542ms from cluster start. Timeout at 2023-08-02 15:12:55.323024326 +0000 UTC m=+7.418960672, 105.195801ms after stability testing.go:149: cluster: 2023-08-02T15:12:55.323Z [INFO] stable state for Leader reached at 2023-08-02 15:12:55.217828545 +0000 UTC m=+7.313764871 (1 nodes), 907ns from start of poll, 166.785542ms from cluster start. Timeout at 2023-08-02 15:12:55.323024326 +0000 UTC m=+7.418960672, 105.195801ms after stability testLoggerAdapter verbose: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.323Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: aborting pipeline replication: peer="{Voter server-69de86e2-46d2-260c-3434-5955a9711433 69de86e2-46d2-260c-3434-5955a9711433}" testing.go:149: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.323Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: aborting pipeline replication: peer="{Voter server-69de86e2-46d2-260c-3434-5955a9711433 69de86e2-46d2-260c-3434-5955a9711433}" testLoggerAdapter verbose: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.323Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: aborting pipeline replication: peer="{Voter server-403d52cd-d42b-d401-2e45-a7e7a20a02d3 403d52cd-d42b-d401-2e45-a7e7a20a02d3}" testing.go:149: server-f311d765-c353-90cb-94a0-c34bfa0aa721: 2023-08-02T15:12:55.323Z [INFO] server-f311d765-c353-90cb-94a0-c34bfa0aa721: aborting pipeline replication: peer="{Voter server-403d52cd-d42b-d401-2e45-a7e7a20a02d3 403d52cd-d42b-d401-2e45-a7e7a20a02d3}" --- PASS: TestRaft_JoinNode (0.27s) === RUN TestRaft_JoinNode_ConfigStore testLoggerAdapter verbose: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.324Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d Address:9a6571b0-ba9b-6e3e-6646-3a21e92ff67d}]" testing.go:149: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.324Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d Address:9a6571b0-ba9b-6e3e-6646-3a21e92ff67d}]" testLoggerAdapter verbose: server-bf17074b-abfd-043e-0171-489e92d0ccb3: 2023-08-02T15:12:55.324Z [INFO] server-bf17074b-abfd-043e-0171-489e92d0ccb3: initial configuration: index=0 servers=[] testing.go:149: server-bf17074b-abfd-043e-0171-489e92d0ccb3: 2023-08-02T15:12:55.324Z [INFO] server-bf17074b-abfd-043e-0171-489e92d0ccb3: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91: 2023-08-02T15:12:55.324Z [INFO] server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91: initial configuration: index=0 servers=[] testing.go:149: server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91: 2023-08-02T15:12:55.324Z [INFO] server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2023-08-02T15:12:55.324Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:12:55.324Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.324Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: entering follower state: follower="Node at 9a6571b0-ba9b-6e3e-6646-3a21e92ff67d [Follower]" leader= testing.go:149: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.324Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: entering follower state: follower="Node at 9a6571b0-ba9b-6e3e-6646-3a21e92ff67d [Follower]" leader= testLoggerAdapter verbose: server-bf17074b-abfd-043e-0171-489e92d0ccb3: 2023-08-02T15:12:55.324Z [INFO] server-bf17074b-abfd-043e-0171-489e92d0ccb3: entering follower state: follower="Node at bf17074b-abfd-043e-0171-489e92d0ccb3 [Follower]" leader= testing.go:149: server-bf17074b-abfd-043e-0171-489e92d0ccb3: 2023-08-02T15:12:55.324Z [INFO] server-bf17074b-abfd-043e-0171-489e92d0ccb3: entering follower state: follower="Node at bf17074b-abfd-043e-0171-489e92d0ccb3 [Follower]" leader= testLoggerAdapter verbose: server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91: 2023-08-02T15:12:55.325Z [INFO] server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91: entering follower state: follower="Node at 157a3fd6-85d6-fb89-488c-d7d46c6dbb91 [Follower]" leader= testing.go:149: server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91: 2023-08-02T15:12:55.325Z [INFO] server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91: entering follower state: follower="Node at 157a3fd6-85d6-fb89-488c-d7d46c6dbb91 [Follower]" leader= testLoggerAdapter verbose: server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91: 2023-08-02T15:12:55.376Z [WARN] server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91: no known peers, aborting election testing.go:149: server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91: 2023-08-02T15:12:55.376Z [WARN] server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91: no known peers, aborting election testLoggerAdapter verbose: server-bf17074b-abfd-043e-0171-489e92d0ccb3: 2023-08-02T15:12:55.399Z [WARN] server-bf17074b-abfd-043e-0171-489e92d0ccb3: no known peers, aborting election testing.go:149: server-bf17074b-abfd-043e-0171-489e92d0ccb3: 2023-08-02T15:12:55.399Z [WARN] server-bf17074b-abfd-043e-0171-489e92d0ccb3: no known peers, aborting election testLoggerAdapter verbose: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.420Z [WARN] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.420Z [WARN] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.420Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: entering candidate state: node="Node at 9a6571b0-ba9b-6e3e-6646-3a21e92ff67d [Candidate]" term=2 testing.go:149: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.420Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: entering candidate state: node="Node at 9a6571b0-ba9b-6e3e-6646-3a21e92ff67d [Candidate]" term=2 testLoggerAdapter verbose: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.420Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: election won: tally=1 testing.go:149: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.420Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: election won: tally=1 testLoggerAdapter verbose: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.420Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: entering leader state: leader="Node at 9a6571b0-ba9b-6e3e-6646-3a21e92ff67d [Leader]" testing.go:149: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.420Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: entering leader state: leader="Node at 9a6571b0-ba9b-6e3e-6646-3a21e92ff67d [Leader]" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:55.526Z [INFO] stable state for Leader reached at 2023-08-02 15:12:55.42059634 +0000 UTC m=+7.516532684 (1 nodes), 95.657028ms from start of poll, 96.334108ms from cluster start. Timeout at 2023-08-02 15:12:55.526351416 +0000 UTC m=+7.622287751, 105.755067ms after stability testing.go:149: cluster: 2023-08-02T15:12:55.526Z [INFO] stable state for Leader reached at 2023-08-02 15:12:55.42059634 +0000 UTC m=+7.516532684 (1 nodes), 95.657028ms from start of poll, 96.334108ms from cluster start. Timeout at 2023-08-02 15:12:55.526351416 +0000 UTC m=+7.622287751, 105.755067ms after stability testLoggerAdapter verbose: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.526Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: updating configuration: command=AddStaging server-id=server-bf17074b-abfd-043e-0171-489e92d0ccb3 server-addr=bf17074b-abfd-043e-0171-489e92d0ccb3 servers="[{Suffrage:Voter ID:server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d Address:9a6571b0-ba9b-6e3e-6646-3a21e92ff67d} {Suffrage:Voter ID:server-bf17074b-abfd-043e-0171-489e92d0ccb3 Address:bf17074b-abfd-043e-0171-489e92d0ccb3}]" testing.go:149: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.526Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: updating configuration: command=AddStaging server-id=server-bf17074b-abfd-043e-0171-489e92d0ccb3 server-addr=bf17074b-abfd-043e-0171-489e92d0ccb3 servers="[{Suffrage:Voter ID:server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d Address:9a6571b0-ba9b-6e3e-6646-3a21e92ff67d} {Suffrage:Voter ID:server-bf17074b-abfd-043e-0171-489e92d0ccb3 Address:bf17074b-abfd-043e-0171-489e92d0ccb3}]" testLoggerAdapter verbose: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.526Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: added peer, starting replication: peer=server-bf17074b-abfd-043e-0171-489e92d0ccb3 testing.go:149: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.526Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: added peer, starting replication: peer=server-bf17074b-abfd-043e-0171-489e92d0ccb3 testLoggerAdapter verbose: cluster: 2023-08-02T15:12:55.526Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:12:55.526Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-bf17074b-abfd-043e-0171-489e92d0ccb3: 2023-08-02T15:12:55.526Z [WARN] server-bf17074b-abfd-043e-0171-489e92d0ccb3: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: server-bf17074b-abfd-043e-0171-489e92d0ccb3: 2023-08-02T15:12:55.526Z [WARN] server-bf17074b-abfd-043e-0171-489e92d0ccb3: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.526Z [WARN] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: appendEntries rejected, sending older logs: peer="{Voter server-bf17074b-abfd-043e-0171-489e92d0ccb3 bf17074b-abfd-043e-0171-489e92d0ccb3}" next=1 testing.go:149: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.526Z [WARN] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: appendEntries rejected, sending older logs: peer="{Voter server-bf17074b-abfd-043e-0171-489e92d0ccb3 bf17074b-abfd-043e-0171-489e92d0ccb3}" next=1 testLoggerAdapter verbose: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.526Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: pipelining replication: peer="{Voter server-bf17074b-abfd-043e-0171-489e92d0ccb3 bf17074b-abfd-043e-0171-489e92d0ccb3}" testing.go:149: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.526Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: pipelining replication: peer="{Voter server-bf17074b-abfd-043e-0171-489e92d0ccb3 bf17074b-abfd-043e-0171-489e92d0ccb3}" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:55.632Z [INFO] stable state for Leader reached at 2023-08-02 15:12:55.526786437 +0000 UTC m=+7.622722763 (1 nodes), 925ns from start of poll, 202.524187ms from cluster start. Timeout at 2023-08-02 15:12:55.632153074 +0000 UTC m=+7.728089479, 105.366716ms after stability testing.go:149: cluster: 2023-08-02T15:12:55.632Z [INFO] stable state for Leader reached at 2023-08-02 15:12:55.526786437 +0000 UTC m=+7.622722763 (1 nodes), 925ns from start of poll, 202.524187ms from cluster start. Timeout at 2023-08-02 15:12:55.632153074 +0000 UTC m=+7.728089479, 105.366716ms after stability testLoggerAdapter verbose: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.632Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: updating configuration: command=AddStaging server-id=server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91 server-addr=157a3fd6-85d6-fb89-488c-d7d46c6dbb91 servers="[{Suffrage:Voter ID:server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d Address:9a6571b0-ba9b-6e3e-6646-3a21e92ff67d} {Suffrage:Voter ID:server-bf17074b-abfd-043e-0171-489e92d0ccb3 Address:bf17074b-abfd-043e-0171-489e92d0ccb3} {Suffrage:Voter ID:server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91 Address:157a3fd6-85d6-fb89-488c-d7d46c6dbb91}]" testing.go:149: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.632Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: updating configuration: command=AddStaging server-id=server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91 server-addr=157a3fd6-85d6-fb89-488c-d7d46c6dbb91 servers="[{Suffrage:Voter ID:server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d Address:9a6571b0-ba9b-6e3e-6646-3a21e92ff67d} {Suffrage:Voter ID:server-bf17074b-abfd-043e-0171-489e92d0ccb3 Address:bf17074b-abfd-043e-0171-489e92d0ccb3} {Suffrage:Voter ID:server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91 Address:157a3fd6-85d6-fb89-488c-d7d46c6dbb91}]" testLoggerAdapter verbose: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.632Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: added peer, starting replication: peer=server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91 testing.go:149: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.632Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: added peer, starting replication: peer=server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91 testLoggerAdapter verbose: server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91: 2023-08-02T15:12:55.632Z [WARN] server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91: failed to get previous log: previous-index=4 last-index=0 error="log not found" testing.go:149: server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91: 2023-08-02T15:12:55.632Z [WARN] server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91: failed to get previous log: previous-index=4 last-index=0 error="log not found" testLoggerAdapter verbose: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.632Z [WARN] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: appendEntries rejected, sending older logs: peer="{Voter server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91 157a3fd6-85d6-fb89-488c-d7d46c6dbb91}" next=1 testing.go:149: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.632Z [WARN] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: appendEntries rejected, sending older logs: peer="{Voter server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91 157a3fd6-85d6-fb89-488c-d7d46c6dbb91}" next=1 testLoggerAdapter verbose: cluster: 2023-08-02T15:12:55.632Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.632Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: pipelining replication: peer="{Voter server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91 157a3fd6-85d6-fb89-488c-d7d46c6dbb91}" testing.go:149: cluster: 2023-08-02T15:12:55.632Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.632Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: pipelining replication: peer="{Voter server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91 157a3fd6-85d6-fb89-488c-d7d46c6dbb91}" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:55.738Z [INFO] stable state for Leader reached at 2023-08-02 15:12:55.632792751 +0000 UTC m=+7.728729079 (1 nodes), 4.907µs from start of poll, 308.530503ms from cluster start. Timeout at 2023-08-02 15:12:55.738092604 +0000 UTC m=+7.834028930, 105.299851ms after stability testing.go:149: cluster: 2023-08-02T15:12:55.738Z [INFO] stable state for Leader reached at 2023-08-02 15:12:55.632792751 +0000 UTC m=+7.728729079 (1 nodes), 4.907µs from start of poll, 308.530503ms from cluster start. Timeout at 2023-08-02 15:12:55.738092604 +0000 UTC m=+7.834028930, 105.299851ms after stability testLoggerAdapter verbose: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.738Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: aborting pipeline replication: peer="{Voter server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91 157a3fd6-85d6-fb89-488c-d7d46c6dbb91}" testing.go:149: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.738Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: aborting pipeline replication: peer="{Voter server-157a3fd6-85d6-fb89-488c-d7d46c6dbb91 157a3fd6-85d6-fb89-488c-d7d46c6dbb91}" testLoggerAdapter verbose: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.738Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: aborting pipeline replication: peer="{Voter server-bf17074b-abfd-043e-0171-489e92d0ccb3 bf17074b-abfd-043e-0171-489e92d0ccb3}" testing.go:149: server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: 2023-08-02T15:12:55.738Z [INFO] server-9a6571b0-ba9b-6e3e-6646-3a21e92ff67d: aborting pipeline replication: peer="{Voter server-bf17074b-abfd-043e-0171-489e92d0ccb3 bf17074b-abfd-043e-0171-489e92d0ccb3}" --- PASS: TestRaft_JoinNode_ConfigStore (0.41s) === RUN TestRaft_RemoveFollower testLoggerAdapter verbose: server-7fad1ebc-7dca-2b0b-423e-052f90989cfb: 2023-08-02T15:12:55.739Z [INFO] server-7fad1ebc-7dca-2b0b-423e-052f90989cfb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7fad1ebc-7dca-2b0b-423e-052f90989cfb Address:7fad1ebc-7dca-2b0b-423e-052f90989cfb} {Suffrage:Voter ID:server-5774e529-6576-711c-31bc-25c62d49a15c Address:5774e529-6576-711c-31bc-25c62d49a15c} {Suffrage:Voter ID:server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a Address:e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a}]" testing.go:149: server-7fad1ebc-7dca-2b0b-423e-052f90989cfb: 2023-08-02T15:12:55.739Z [INFO] server-7fad1ebc-7dca-2b0b-423e-052f90989cfb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7fad1ebc-7dca-2b0b-423e-052f90989cfb Address:7fad1ebc-7dca-2b0b-423e-052f90989cfb} {Suffrage:Voter ID:server-5774e529-6576-711c-31bc-25c62d49a15c Address:5774e529-6576-711c-31bc-25c62d49a15c} {Suffrage:Voter ID:server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a Address:e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a}]" testLoggerAdapter verbose: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.739Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7fad1ebc-7dca-2b0b-423e-052f90989cfb Address:7fad1ebc-7dca-2b0b-423e-052f90989cfb} {Suffrage:Voter ID:server-5774e529-6576-711c-31bc-25c62d49a15c Address:5774e529-6576-711c-31bc-25c62d49a15c} {Suffrage:Voter ID:server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a Address:e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a}]" testing.go:149: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.739Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7fad1ebc-7dca-2b0b-423e-052f90989cfb Address:7fad1ebc-7dca-2b0b-423e-052f90989cfb} {Suffrage:Voter ID:server-5774e529-6576-711c-31bc-25c62d49a15c Address:5774e529-6576-711c-31bc-25c62d49a15c} {Suffrage:Voter ID:server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a Address:e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a}]" testLoggerAdapter verbose: server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a: 2023-08-02T15:12:55.739Z [INFO] server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7fad1ebc-7dca-2b0b-423e-052f90989cfb Address:7fad1ebc-7dca-2b0b-423e-052f90989cfb} {Suffrage:Voter ID:server-5774e529-6576-711c-31bc-25c62d49a15c Address:5774e529-6576-711c-31bc-25c62d49a15c} {Suffrage:Voter ID:server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a Address:e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a}]" testing.go:149: server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a: 2023-08-02T15:12:55.739Z [INFO] server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7fad1ebc-7dca-2b0b-423e-052f90989cfb Address:7fad1ebc-7dca-2b0b-423e-052f90989cfb} {Suffrage:Voter ID:server-5774e529-6576-711c-31bc-25c62d49a15c Address:5774e529-6576-711c-31bc-25c62d49a15c} {Suffrage:Voter ID:server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a Address:e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:55.739Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:12:55.739Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-7fad1ebc-7dca-2b0b-423e-052f90989cfb: 2023-08-02T15:12:55.739Z [INFO] server-7fad1ebc-7dca-2b0b-423e-052f90989cfb: entering follower state: follower="Node at 7fad1ebc-7dca-2b0b-423e-052f90989cfb [Follower]" leader= testing.go:149: server-7fad1ebc-7dca-2b0b-423e-052f90989cfb: 2023-08-02T15:12:55.739Z [INFO] server-7fad1ebc-7dca-2b0b-423e-052f90989cfb: entering follower state: follower="Node at 7fad1ebc-7dca-2b0b-423e-052f90989cfb [Follower]" leader= testLoggerAdapter verbose: server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a: 2023-08-02T15:12:55.739Z [INFO] server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a: entering follower state: follower="Node at e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a [Follower]" leader= testing.go:149: server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a: 2023-08-02T15:12:55.739Z [INFO] server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a: entering follower state: follower="Node at e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a [Follower]" leader= testLoggerAdapter verbose: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.739Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: entering follower state: follower="Node at 5774e529-6576-711c-31bc-25c62d49a15c [Follower]" leader= testing.go:149: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.739Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: entering follower state: follower="Node at 5774e529-6576-711c-31bc-25c62d49a15c [Follower]" leader= testLoggerAdapter verbose: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.810Z [WARN] server-5774e529-6576-711c-31bc-25c62d49a15c: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.810Z [WARN] server-5774e529-6576-711c-31bc-25c62d49a15c: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.810Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: entering candidate state: node="Node at 5774e529-6576-711c-31bc-25c62d49a15c [Candidate]" term=2 testing.go:149: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.810Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: entering candidate state: node="Node at 5774e529-6576-711c-31bc-25c62d49a15c [Candidate]" term=2 testLoggerAdapter verbose: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.810Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: election won: tally=2 testing.go:149: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.810Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: election won: tally=2 testLoggerAdapter verbose: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.810Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: entering leader state: leader="Node at 5774e529-6576-711c-31bc-25c62d49a15c [Leader]" testing.go:149: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.810Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: entering leader state: leader="Node at 5774e529-6576-711c-31bc-25c62d49a15c [Leader]" testLoggerAdapter verbose: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.810Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: added peer, starting replication: peer=server-7fad1ebc-7dca-2b0b-423e-052f90989cfb testing.go:149: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.810Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: added peer, starting replication: peer=server-7fad1ebc-7dca-2b0b-423e-052f90989cfb testLoggerAdapter verbose: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.810Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: added peer, starting replication: peer=server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a testing.go:149: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.810Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: added peer, starting replication: peer=server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a testLoggerAdapter verbose: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.810Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: pipelining replication: peer="{Voter server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a}" testing.go:149: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.810Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: pipelining replication: peer="{Voter server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a}" testLoggerAdapter verbose: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.810Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: pipelining replication: peer="{Voter server-7fad1ebc-7dca-2b0b-423e-052f90989cfb 7fad1ebc-7dca-2b0b-423e-052f90989cfb}" testing.go:149: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:55.810Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: pipelining replication: peer="{Voter server-7fad1ebc-7dca-2b0b-423e-052f90989cfb 7fad1ebc-7dca-2b0b-423e-052f90989cfb}" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:55.915Z [INFO] stable state for Leader reached at 2023-08-02 15:12:55.810624041 +0000 UTC m=+7.906560375 (1 nodes), 70.90321ms from start of poll, 71.384558ms from cluster start. Timeout at 2023-08-02 15:12:55.91569773 +0000 UTC m=+8.011634065, 105.07369ms after stability testing.go:149: cluster: 2023-08-02T15:12:55.915Z [INFO] stable state for Leader reached at 2023-08-02 15:12:55.810624041 +0000 UTC m=+7.906560375 (1 nodes), 70.90321ms from start of poll, 71.384558ms from cluster start. Timeout at 2023-08-02 15:12:55.91569773 +0000 UTC m=+8.011634065, 105.07369ms after stability testLoggerAdapter verbose: cluster: 2023-08-02T15:12:55.922Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-08-02T15:12:55.922Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2023-08-02T15:12:56.027Z [INFO] stable state for Follower reached at 2023-08-02 15:12:55.92270497 +0000 UTC m=+8.018641307 (2 nodes), 2.03µs from start of poll, 183.46549ms from cluster start. Timeout at 2023-08-02 15:12:56.027732181 +0000 UTC m=+8.123668508, 105.027201ms after stability testing.go:149: cluster: 2023-08-02T15:12:56.027Z [INFO] stable state for Follower reached at 2023-08-02 15:12:55.92270497 +0000 UTC m=+8.018641307 (2 nodes), 2.03µs from start of poll, 183.46549ms from cluster start. Timeout at 2023-08-02 15:12:56.027732181 +0000 UTC m=+8.123668508, 105.027201ms after stability testLoggerAdapter verbose: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:56.027Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: updating configuration: command=RemoveServer server-id=server-7fad1ebc-7dca-2b0b-423e-052f90989cfb server-addr= servers="[{Suffrage:Voter ID:server-5774e529-6576-711c-31bc-25c62d49a15c Address:5774e529-6576-711c-31bc-25c62d49a15c} {Suffrage:Voter ID:server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a Address:e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a}]" testing.go:149: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:56.027Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: updating configuration: command=RemoveServer server-id=server-7fad1ebc-7dca-2b0b-423e-052f90989cfb server-addr= servers="[{Suffrage:Voter ID:server-5774e529-6576-711c-31bc-25c62d49a15c Address:5774e529-6576-711c-31bc-25c62d49a15c} {Suffrage:Voter ID:server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a Address:e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a}]" testLoggerAdapter verbose: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:56.028Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: removed peer, stopping replication: peer=server-7fad1ebc-7dca-2b0b-423e-052f90989cfb last-index=3 testing.go:149: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:56.028Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: removed peer, stopping replication: peer=server-7fad1ebc-7dca-2b0b-423e-052f90989cfb last-index=3 testLoggerAdapter verbose: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:56.028Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: aborting pipeline replication: peer="{Voter server-7fad1ebc-7dca-2b0b-423e-052f90989cfb 7fad1ebc-7dca-2b0b-423e-052f90989cfb}" testing.go:149: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:56.028Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: aborting pipeline replication: peer="{Voter server-7fad1ebc-7dca-2b0b-423e-052f90989cfb 7fad1ebc-7dca-2b0b-423e-052f90989cfb}" testLoggerAdapter verbose: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:56.133Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: aborting pipeline replication: peer="{Voter server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a}" testing.go:149: server-5774e529-6576-711c-31bc-25c62d49a15c: 2023-08-02T15:12:56.133Z [INFO] server-5774e529-6576-711c-31bc-25c62d49a15c: aborting pipeline replication: peer="{Voter server-e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a e98deb3f-5b54-8dce-9dad-e90a3cbc3f7a}" --- PASS: TestRaft_RemoveFollower (0.39s) === RUN TestRaft_RemoveLeader testLoggerAdapter verbose: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.134Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b Address:8a122bf6-ae8e-d64b-47cf-5e59c182a21b} {Suffrage:Voter ID:server-f090f725-515f-71ea-a7db-f4ed53d23adf Address:f090f725-515f-71ea-a7db-f4ed53d23adf} {Suffrage:Voter ID:server-08f55ce9-54f6-1660-4710-cb41b5e07326 Address:08f55ce9-54f6-1660-4710-cb41b5e07326}]" testing.go:149: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.134Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b Address:8a122bf6-ae8e-d64b-47cf-5e59c182a21b} {Suffrage:Voter ID:server-f090f725-515f-71ea-a7db-f4ed53d23adf Address:f090f725-515f-71ea-a7db-f4ed53d23adf} {Suffrage:Voter ID:server-08f55ce9-54f6-1660-4710-cb41b5e07326 Address:08f55ce9-54f6-1660-4710-cb41b5e07326}]" testLoggerAdapter verbose: server-f090f725-515f-71ea-a7db-f4ed53d23adf: 2023-08-02T15:12:56.134Z [INFO] server-f090f725-515f-71ea-a7db-f4ed53d23adf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b Address:8a122bf6-ae8e-d64b-47cf-5e59c182a21b} {Suffrage:Voter ID:server-f090f725-515f-71ea-a7db-f4ed53d23adf Address:f090f725-515f-71ea-a7db-f4ed53d23adf} {Suffrage:Voter ID:server-08f55ce9-54f6-1660-4710-cb41b5e07326 Address:08f55ce9-54f6-1660-4710-cb41b5e07326}]" testing.go:149: server-f090f725-515f-71ea-a7db-f4ed53d23adf: 2023-08-02T15:12:56.134Z [INFO] server-f090f725-515f-71ea-a7db-f4ed53d23adf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b Address:8a122bf6-ae8e-d64b-47cf-5e59c182a21b} {Suffrage:Voter ID:server-f090f725-515f-71ea-a7db-f4ed53d23adf Address:f090f725-515f-71ea-a7db-f4ed53d23adf} {Suffrage:Voter ID:server-08f55ce9-54f6-1660-4710-cb41b5e07326 Address:08f55ce9-54f6-1660-4710-cb41b5e07326}]" testLoggerAdapter verbose: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.134Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b Address:8a122bf6-ae8e-d64b-47cf-5e59c182a21b} {Suffrage:Voter ID:server-f090f725-515f-71ea-a7db-f4ed53d23adf Address:f090f725-515f-71ea-a7db-f4ed53d23adf} {Suffrage:Voter ID:server-08f55ce9-54f6-1660-4710-cb41b5e07326 Address:08f55ce9-54f6-1660-4710-cb41b5e07326}]" testing.go:149: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.134Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b Address:8a122bf6-ae8e-d64b-47cf-5e59c182a21b} {Suffrage:Voter ID:server-f090f725-515f-71ea-a7db-f4ed53d23adf Address:f090f725-515f-71ea-a7db-f4ed53d23adf} {Suffrage:Voter ID:server-08f55ce9-54f6-1660-4710-cb41b5e07326 Address:08f55ce9-54f6-1660-4710-cb41b5e07326}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:56.134Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:12:56.134Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.134Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: entering follower state: follower="Node at 8a122bf6-ae8e-d64b-47cf-5e59c182a21b [Follower]" leader= testing.go:149: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.134Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: entering follower state: follower="Node at 8a122bf6-ae8e-d64b-47cf-5e59c182a21b [Follower]" leader= testLoggerAdapter verbose: server-f090f725-515f-71ea-a7db-f4ed53d23adf: 2023-08-02T15:12:56.134Z [INFO] server-f090f725-515f-71ea-a7db-f4ed53d23adf: entering follower state: follower="Node at f090f725-515f-71ea-a7db-f4ed53d23adf [Follower]" leader= testing.go:149: server-f090f725-515f-71ea-a7db-f4ed53d23adf: 2023-08-02T15:12:56.134Z [INFO] server-f090f725-515f-71ea-a7db-f4ed53d23adf: entering follower state: follower="Node at f090f725-515f-71ea-a7db-f4ed53d23adf [Follower]" leader= testLoggerAdapter verbose: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.134Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: entering follower state: follower="Node at 08f55ce9-54f6-1660-4710-cb41b5e07326 [Follower]" leader= testing.go:149: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.134Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: entering follower state: follower="Node at 08f55ce9-54f6-1660-4710-cb41b5e07326 [Follower]" leader= testLoggerAdapter verbose: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.186Z [WARN] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.186Z [WARN] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.186Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: entering candidate state: node="Node at 8a122bf6-ae8e-d64b-47cf-5e59c182a21b [Candidate]" term=2 testing.go:149: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.186Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: entering candidate state: node="Node at 8a122bf6-ae8e-d64b-47cf-5e59c182a21b [Candidate]" term=2 testLoggerAdapter verbose: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.186Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: election won: tally=2 testing.go:149: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.186Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: election won: tally=2 testLoggerAdapter verbose: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.186Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: entering leader state: leader="Node at 8a122bf6-ae8e-d64b-47cf-5e59c182a21b [Leader]" testing.go:149: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.186Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: entering leader state: leader="Node at 8a122bf6-ae8e-d64b-47cf-5e59c182a21b [Leader]" testLoggerAdapter verbose: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.186Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: added peer, starting replication: peer=server-f090f725-515f-71ea-a7db-f4ed53d23adf testing.go:149: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.186Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: added peer, starting replication: peer=server-f090f725-515f-71ea-a7db-f4ed53d23adf testLoggerAdapter verbose: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.187Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: added peer, starting replication: peer=server-08f55ce9-54f6-1660-4710-cb41b5e07326 testing.go:149: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.187Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: added peer, starting replication: peer=server-08f55ce9-54f6-1660-4710-cb41b5e07326 testLoggerAdapter verbose: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.187Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: pipelining replication: peer="{Voter server-08f55ce9-54f6-1660-4710-cb41b5e07326 08f55ce9-54f6-1660-4710-cb41b5e07326}" testing.go:149: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.187Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: pipelining replication: peer="{Voter server-08f55ce9-54f6-1660-4710-cb41b5e07326 08f55ce9-54f6-1660-4710-cb41b5e07326}" testLoggerAdapter verbose: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.187Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: pipelining replication: peer="{Voter server-f090f725-515f-71ea-a7db-f4ed53d23adf f090f725-515f-71ea-a7db-f4ed53d23adf}" testing.go:149: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.187Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: pipelining replication: peer="{Voter server-f090f725-515f-71ea-a7db-f4ed53d23adf f090f725-515f-71ea-a7db-f4ed53d23adf}" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:56.292Z [INFO] stable state for Leader reached at 2023-08-02 15:12:56.187298041 +0000 UTC m=+8.283234380 (1 nodes), 52.450177ms from start of poll, 53.053767ms from cluster start. Timeout at 2023-08-02 15:12:56.292551678 +0000 UTC m=+8.388488026, 105.253646ms after stability testing.go:149: cluster: 2023-08-02T15:12:56.292Z [INFO] stable state for Leader reached at 2023-08-02 15:12:56.187298041 +0000 UTC m=+8.283234380 (1 nodes), 52.450177ms from start of poll, 53.053767ms from cluster start. Timeout at 2023-08-02 15:12:56.292551678 +0000 UTC m=+8.388488026, 105.253646ms after stability testLoggerAdapter verbose: cluster: 2023-08-02T15:12:56.298Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-08-02T15:12:56.298Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2023-08-02T15:12:56.404Z [INFO] stable state for Follower reached at 2023-08-02 15:12:56.2987994 +0000 UTC m=+8.394735728 (2 nodes), 1.734µs from start of poll, 164.555115ms from cluster start. Timeout at 2023-08-02 15:12:56.403998394 +0000 UTC m=+8.499934977, 105.199249ms after stability testing.go:149: cluster: 2023-08-02T15:12:56.404Z [INFO] stable state for Follower reached at 2023-08-02 15:12:56.2987994 +0000 UTC m=+8.394735728 (2 nodes), 1.734µs from start of poll, 164.555115ms from cluster start. Timeout at 2023-08-02 15:12:56.403998394 +0000 UTC m=+8.499934977, 105.199249ms after stability testLoggerAdapter verbose: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.404Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: updating configuration: command=RemoveServer server-id=server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b server-addr= servers="[{Suffrage:Voter ID:server-f090f725-515f-71ea-a7db-f4ed53d23adf Address:f090f725-515f-71ea-a7db-f4ed53d23adf} {Suffrage:Voter ID:server-08f55ce9-54f6-1660-4710-cb41b5e07326 Address:08f55ce9-54f6-1660-4710-cb41b5e07326}]" testing.go:149: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.404Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: updating configuration: command=RemoveServer server-id=server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b server-addr= servers="[{Suffrage:Voter ID:server-f090f725-515f-71ea-a7db-f4ed53d23adf Address:f090f725-515f-71ea-a7db-f4ed53d23adf} {Suffrage:Voter ID:server-08f55ce9-54f6-1660-4710-cb41b5e07326 Address:08f55ce9-54f6-1660-4710-cb41b5e07326}]" testLoggerAdapter verbose: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.404Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: removed ourself, shutting down testing.go:149: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.404Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: removed ourself, shutting down testLoggerAdapter verbose: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.404Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: aborting pipeline replication: peer="{Voter server-08f55ce9-54f6-1660-4710-cb41b5e07326 08f55ce9-54f6-1660-4710-cb41b5e07326}" testing.go:149: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.404Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: aborting pipeline replication: peer="{Voter server-08f55ce9-54f6-1660-4710-cb41b5e07326 08f55ce9-54f6-1660-4710-cb41b5e07326}" testLoggerAdapter verbose: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.404Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: aborting pipeline replication: peer="{Voter server-f090f725-515f-71ea-a7db-f4ed53d23adf f090f725-515f-71ea-a7db-f4ed53d23adf}" testing.go:149: server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: 2023-08-02T15:12:56.404Z [INFO] server-8a122bf6-ae8e-d64b-47cf-5e59c182a21b: aborting pipeline replication: peer="{Voter server-f090f725-515f-71ea-a7db-f4ed53d23adf f090f725-515f-71ea-a7db-f4ed53d23adf}" testLoggerAdapter verbose: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.484Z [WARN] server-08f55ce9-54f6-1660-4710-cb41b5e07326: heartbeat timeout reached, starting election: last-leader=8a122bf6-ae8e-d64b-47cf-5e59c182a21b testing.go:149: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.484Z [WARN] server-08f55ce9-54f6-1660-4710-cb41b5e07326: heartbeat timeout reached, starting election: last-leader=8a122bf6-ae8e-d64b-47cf-5e59c182a21b testLoggerAdapter verbose: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.484Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: entering candidate state: node="Node at 08f55ce9-54f6-1660-4710-cb41b5e07326 [Candidate]" term=3 testing.go:149: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.484Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: entering candidate state: node="Node at 08f55ce9-54f6-1660-4710-cb41b5e07326 [Candidate]" term=3 testLoggerAdapter verbose: server-f090f725-515f-71ea-a7db-f4ed53d23adf: 2023-08-02T15:12:56.484Z [WARN] server-f090f725-515f-71ea-a7db-f4ed53d23adf: rejecting vote request since we have a leader: from=08f55ce9-54f6-1660-4710-cb41b5e07326 leader=8a122bf6-ae8e-d64b-47cf-5e59c182a21b testing.go:149: server-f090f725-515f-71ea-a7db-f4ed53d23adf: 2023-08-02T15:12:56.484Z [WARN] server-f090f725-515f-71ea-a7db-f4ed53d23adf: rejecting vote request since we have a leader: from=08f55ce9-54f6-1660-4710-cb41b5e07326 leader=8a122bf6-ae8e-d64b-47cf-5e59c182a21b testLoggerAdapter verbose: server-f090f725-515f-71ea-a7db-f4ed53d23adf: 2023-08-02T15:12:56.484Z [WARN] server-f090f725-515f-71ea-a7db-f4ed53d23adf: heartbeat timeout reached, starting election: last-leader=8a122bf6-ae8e-d64b-47cf-5e59c182a21b testing.go:149: server-f090f725-515f-71ea-a7db-f4ed53d23adf: 2023-08-02T15:12:56.484Z [WARN] server-f090f725-515f-71ea-a7db-f4ed53d23adf: heartbeat timeout reached, starting election: last-leader=8a122bf6-ae8e-d64b-47cf-5e59c182a21b testLoggerAdapter verbose: server-f090f725-515f-71ea-a7db-f4ed53d23adf: 2023-08-02T15:12:56.484Z [INFO] server-f090f725-515f-71ea-a7db-f4ed53d23adf: entering candidate state: node="Node at f090f725-515f-71ea-a7db-f4ed53d23adf [Candidate]" term=3 testing.go:149: server-f090f725-515f-71ea-a7db-f4ed53d23adf: 2023-08-02T15:12:56.484Z [INFO] server-f090f725-515f-71ea-a7db-f4ed53d23adf: entering candidate state: node="Node at f090f725-515f-71ea-a7db-f4ed53d23adf [Candidate]" term=3 testLoggerAdapter verbose: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.484Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: duplicate requestVote for same term: term=3 testing.go:149: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.484Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: duplicate requestVote for same term: term=3 testLoggerAdapter verbose: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.546Z [WARN] server-08f55ce9-54f6-1660-4710-cb41b5e07326: Election timeout reached, restarting election testing.go:149: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.546Z [WARN] server-08f55ce9-54f6-1660-4710-cb41b5e07326: Election timeout reached, restarting election testLoggerAdapter verbose: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.546Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: entering candidate state: node="Node at 08f55ce9-54f6-1660-4710-cb41b5e07326 [Candidate]" term=4 testing.go:149: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.546Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: entering candidate state: node="Node at 08f55ce9-54f6-1660-4710-cb41b5e07326 [Candidate]" term=4 testLoggerAdapter verbose: server-f090f725-515f-71ea-a7db-f4ed53d23adf: 2023-08-02T15:12:56.546Z [INFO] server-f090f725-515f-71ea-a7db-f4ed53d23adf: entering follower state: follower="Node at f090f725-515f-71ea-a7db-f4ed53d23adf [Follower]" leader= testing.go:149: server-f090f725-515f-71ea-a7db-f4ed53d23adf: 2023-08-02T15:12:56.546Z [INFO] server-f090f725-515f-71ea-a7db-f4ed53d23adf: entering follower state: follower="Node at f090f725-515f-71ea-a7db-f4ed53d23adf [Follower]" leader= testLoggerAdapter verbose: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.546Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: election won: tally=2 testing.go:149: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.546Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: election won: tally=2 testLoggerAdapter verbose: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.546Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: entering leader state: leader="Node at 08f55ce9-54f6-1660-4710-cb41b5e07326 [Leader]" testing.go:149: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.546Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: entering leader state: leader="Node at 08f55ce9-54f6-1660-4710-cb41b5e07326 [Leader]" testLoggerAdapter verbose: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.546Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: added peer, starting replication: peer=server-f090f725-515f-71ea-a7db-f4ed53d23adf testing.go:149: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.546Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: added peer, starting replication: peer=server-f090f725-515f-71ea-a7db-f4ed53d23adf testLoggerAdapter verbose: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.546Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: pipelining replication: peer="{Voter server-f090f725-515f-71ea-a7db-f4ed53d23adf f090f725-515f-71ea-a7db-f4ed53d23adf}" testing.go:149: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.546Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: pipelining replication: peer="{Voter server-f090f725-515f-71ea-a7db-f4ed53d23adf f090f725-515f-71ea-a7db-f4ed53d23adf}" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:56.615Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:12:56.615Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-08-02T15:12:56.720Z [INFO] stable state for Leader reached at 2023-08-02 15:12:56.615217861 +0000 UTC m=+8.711154208 (1 nodes), 35.438µs from start of poll, 480.973595ms from cluster start. Timeout at 2023-08-02 15:12:56.720502669 +0000 UTC m=+8.816439011, 105.284803ms after stability testing.go:149: cluster: 2023-08-02T15:12:56.720Z [INFO] stable state for Leader reached at 2023-08-02 15:12:56.615217861 +0000 UTC m=+8.711154208 (1 nodes), 35.438µs from start of poll, 480.973595ms from cluster start. Timeout at 2023-08-02 15:12:56.720502669 +0000 UTC m=+8.816439011, 105.284803ms after stability testLoggerAdapter verbose: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.720Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: aborting pipeline replication: peer="{Voter server-f090f725-515f-71ea-a7db-f4ed53d23adf f090f725-515f-71ea-a7db-f4ed53d23adf}" testing.go:149: server-08f55ce9-54f6-1660-4710-cb41b5e07326: 2023-08-02T15:12:56.720Z [INFO] server-08f55ce9-54f6-1660-4710-cb41b5e07326: aborting pipeline replication: peer="{Voter server-f090f725-515f-71ea-a7db-f4ed53d23adf f090f725-515f-71ea-a7db-f4ed53d23adf}" --- PASS: TestRaft_RemoveLeader (0.59s) === RUN TestRaft_RemoveLeader_NoShutdown testLoggerAdapter verbose: server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: 2023-08-02T15:12:56.724Z [INFO] server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c Address:bf8e15c9-b2a9-d822-6159-523a1e97ec9c} {Suffrage:Voter ID:server-064a8bac-5a48-2c3c-6153-da94e6d050c7 Address:064a8bac-5a48-2c3c-6153-da94e6d050c7} {Suffrage:Voter ID:server-559cfffc-7a69-fb43-6ca4-99a924e98ca9 Address:559cfffc-7a69-fb43-6ca4-99a924e98ca9}]" testing.go:149: server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: 2023-08-02T15:12:56.724Z [INFO] server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c Address:bf8e15c9-b2a9-d822-6159-523a1e97ec9c} {Suffrage:Voter ID:server-064a8bac-5a48-2c3c-6153-da94e6d050c7 Address:064a8bac-5a48-2c3c-6153-da94e6d050c7} {Suffrage:Voter ID:server-559cfffc-7a69-fb43-6ca4-99a924e98ca9 Address:559cfffc-7a69-fb43-6ca4-99a924e98ca9}]" testLoggerAdapter verbose: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:56.724Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c Address:bf8e15c9-b2a9-d822-6159-523a1e97ec9c} {Suffrage:Voter ID:server-064a8bac-5a48-2c3c-6153-da94e6d050c7 Address:064a8bac-5a48-2c3c-6153-da94e6d050c7} {Suffrage:Voter ID:server-559cfffc-7a69-fb43-6ca4-99a924e98ca9 Address:559cfffc-7a69-fb43-6ca4-99a924e98ca9}]" testing.go:149: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:56.724Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c Address:bf8e15c9-b2a9-d822-6159-523a1e97ec9c} {Suffrage:Voter ID:server-064a8bac-5a48-2c3c-6153-da94e6d050c7 Address:064a8bac-5a48-2c3c-6153-da94e6d050c7} {Suffrage:Voter ID:server-559cfffc-7a69-fb43-6ca4-99a924e98ca9 Address:559cfffc-7a69-fb43-6ca4-99a924e98ca9}]" testLoggerAdapter verbose: server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: 2023-08-02T15:12:56.724Z [INFO] server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: entering follower state: follower="Node at bf8e15c9-b2a9-d822-6159-523a1e97ec9c [Follower]" leader= testing.go:149: server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: 2023-08-02T15:12:56.724Z [INFO] server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: entering follower state: follower="Node at bf8e15c9-b2a9-d822-6159-523a1e97ec9c [Follower]" leader= testLoggerAdapter verbose: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.724Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c Address:bf8e15c9-b2a9-d822-6159-523a1e97ec9c} {Suffrage:Voter ID:server-064a8bac-5a48-2c3c-6153-da94e6d050c7 Address:064a8bac-5a48-2c3c-6153-da94e6d050c7} {Suffrage:Voter ID:server-559cfffc-7a69-fb43-6ca4-99a924e98ca9 Address:559cfffc-7a69-fb43-6ca4-99a924e98ca9}]" testing.go:149: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.724Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c Address:bf8e15c9-b2a9-d822-6159-523a1e97ec9c} {Suffrage:Voter ID:server-064a8bac-5a48-2c3c-6153-da94e6d050c7 Address:064a8bac-5a48-2c3c-6153-da94e6d050c7} {Suffrage:Voter ID:server-559cfffc-7a69-fb43-6ca4-99a924e98ca9 Address:559cfffc-7a69-fb43-6ca4-99a924e98ca9}]" testLoggerAdapter verbose: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:56.724Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: entering follower state: follower="Node at 064a8bac-5a48-2c3c-6153-da94e6d050c7 [Follower]" leader= testing.go:149: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:56.724Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: entering follower state: follower="Node at 064a8bac-5a48-2c3c-6153-da94e6d050c7 [Follower]" leader= testLoggerAdapter verbose: cluster: 2023-08-02T15:12:56.724Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-08-02T15:12:56.724Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.724Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: entering follower state: follower="Node at 559cfffc-7a69-fb43-6ca4-99a924e98ca9 [Follower]" leader= testing.go:149: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.724Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: entering follower state: follower="Node at 559cfffc-7a69-fb43-6ca4-99a924e98ca9 [Follower]" leader= testLoggerAdapter verbose: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.787Z [WARN] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.787Z [WARN] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.788Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: entering candidate state: node="Node at 559cfffc-7a69-fb43-6ca4-99a924e98ca9 [Candidate]" term=2 testing.go:149: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.788Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: entering candidate state: node="Node at 559cfffc-7a69-fb43-6ca4-99a924e98ca9 [Candidate]" term=2 testLoggerAdapter verbose: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.788Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: election won: tally=2 testing.go:149: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.788Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: election won: tally=2 testLoggerAdapter verbose: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.788Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: entering leader state: leader="Node at 559cfffc-7a69-fb43-6ca4-99a924e98ca9 [Leader]" testing.go:149: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.788Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: entering leader state: leader="Node at 559cfffc-7a69-fb43-6ca4-99a924e98ca9 [Leader]" testLoggerAdapter verbose: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.788Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: added peer, starting replication: peer=server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c testing.go:149: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.788Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: added peer, starting replication: peer=server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c testLoggerAdapter verbose: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.788Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: added peer, starting replication: peer=server-064a8bac-5a48-2c3c-6153-da94e6d050c7 testing.go:149: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.788Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: added peer, starting replication: peer=server-064a8bac-5a48-2c3c-6153-da94e6d050c7 testLoggerAdapter verbose: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.788Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: pipelining replication: peer="{Voter server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c bf8e15c9-b2a9-d822-6159-523a1e97ec9c}" testing.go:149: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.788Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: pipelining replication: peer="{Voter server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c bf8e15c9-b2a9-d822-6159-523a1e97ec9c}" testLoggerAdapter verbose: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.788Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: pipelining replication: peer="{Voter server-064a8bac-5a48-2c3c-6153-da94e6d050c7 064a8bac-5a48-2c3c-6153-da94e6d050c7}" testing.go:149: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:56.788Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: pipelining replication: peer="{Voter server-064a8bac-5a48-2c3c-6153-da94e6d050c7 064a8bac-5a48-2c3c-6153-da94e6d050c7}" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:56.893Z [INFO] stable state for Follower reached at 2023-08-02 15:12:56.788303194 +0000 UTC m=+8.884239528 (2 nodes), 63.321073ms from start of poll, 64.096025ms from cluster start. Timeout at 2023-08-02 15:12:56.893618173 +0000 UTC m=+8.989554743, 105.315215ms after stability testing.go:149: cluster: 2023-08-02T15:12:56.893Z [INFO] stable state for Follower reached at 2023-08-02 15:12:56.788303194 +0000 UTC m=+8.884239528 (2 nodes), 63.321073ms from start of poll, 64.096025ms from cluster start. Timeout at 2023-08-02 15:12:56.893618173 +0000 UTC m=+8.989554743, 105.315215ms after stability testLoggerAdapter verbose: cluster: 2023-08-02T15:12:56.893Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:12:56.893Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-08-02T15:12:56.998Z [INFO] stable state for Leader reached at 2023-08-02 15:12:56.893802243 +0000 UTC m=+8.989738570 (1 nodes), 1.547µs from start of poll, 169.595067ms from cluster start. Timeout at 2023-08-02 15:12:56.998912236 +0000 UTC m=+9.094848580, 105.11001ms after stability testing.go:149: cluster: 2023-08-02T15:12:56.998Z [INFO] stable state for Leader reached at 2023-08-02 15:12:56.893802243 +0000 UTC m=+8.989738570 (1 nodes), 1.547µs from start of poll, 169.595067ms from cluster start. Timeout at 2023-08-02 15:12:56.998912236 +0000 UTC m=+9.094848580, 105.11001ms after stability testLoggerAdapter verbose: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:57.001Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: updating configuration: command=RemoveServer server-id=server-559cfffc-7a69-fb43-6ca4-99a924e98ca9 server-addr= servers="[{Suffrage:Voter ID:server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c Address:bf8e15c9-b2a9-d822-6159-523a1e97ec9c} {Suffrage:Voter ID:server-064a8bac-5a48-2c3c-6153-da94e6d050c7 Address:064a8bac-5a48-2c3c-6153-da94e6d050c7}]" testing.go:149: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:57.001Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: updating configuration: command=RemoveServer server-id=server-559cfffc-7a69-fb43-6ca4-99a924e98ca9 server-addr= servers="[{Suffrage:Voter ID:server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c Address:bf8e15c9-b2a9-d822-6159-523a1e97ec9c} {Suffrage:Voter ID:server-064a8bac-5a48-2c3c-6153-da94e6d050c7 Address:064a8bac-5a48-2c3c-6153-da94e6d050c7}]" testLoggerAdapter verbose: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:57.001Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: removed ourself, transitioning to follower testing.go:149: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:57.001Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: removed ourself, transitioning to follower testLoggerAdapter verbose: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:57.001Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: entering follower state: follower="Node at 559cfffc-7a69-fb43-6ca4-99a924e98ca9 [Follower]" leader= testing.go:149: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:57.001Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: entering follower state: follower="Node at 559cfffc-7a69-fb43-6ca4-99a924e98ca9 [Follower]" leader= testLoggerAdapter verbose: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:57.001Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: aborting pipeline replication: peer="{Voter server-064a8bac-5a48-2c3c-6153-da94e6d050c7 064a8bac-5a48-2c3c-6153-da94e6d050c7}" testing.go:149: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:57.001Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: aborting pipeline replication: peer="{Voter server-064a8bac-5a48-2c3c-6153-da94e6d050c7 064a8bac-5a48-2c3c-6153-da94e6d050c7}" testLoggerAdapter verbose: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:57.001Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: aborting pipeline replication: peer="{Voter server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c bf8e15c9-b2a9-d822-6159-523a1e97ec9c}" testing.go:149: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:57.001Z [INFO] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: aborting pipeline replication: peer="{Voter server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c bf8e15c9-b2a9-d822-6159-523a1e97ec9c}" testLoggerAdapter verbose: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.070Z [WARN] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: heartbeat timeout reached, starting election: last-leader=559cfffc-7a69-fb43-6ca4-99a924e98ca9 testing.go:149: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.070Z [WARN] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: heartbeat timeout reached, starting election: last-leader=559cfffc-7a69-fb43-6ca4-99a924e98ca9 testLoggerAdapter verbose: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.070Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: entering candidate state: node="Node at 064a8bac-5a48-2c3c-6153-da94e6d050c7 [Candidate]" term=3 testing.go:149: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.070Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: entering candidate state: node="Node at 064a8bac-5a48-2c3c-6153-da94e6d050c7 [Candidate]" term=3 testLoggerAdapter verbose: server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: 2023-08-02T15:12:57.070Z [WARN] server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: rejecting vote request since we have a leader: from=064a8bac-5a48-2c3c-6153-da94e6d050c7 leader=559cfffc-7a69-fb43-6ca4-99a924e98ca9 testing.go:149: server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: 2023-08-02T15:12:57.070Z [WARN] server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: rejecting vote request since we have a leader: from=064a8bac-5a48-2c3c-6153-da94e6d050c7 leader=559cfffc-7a69-fb43-6ca4-99a924e98ca9 testLoggerAdapter verbose: server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: 2023-08-02T15:12:57.078Z [WARN] server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: heartbeat timeout reached, starting election: last-leader=559cfffc-7a69-fb43-6ca4-99a924e98ca9 testing.go:149: server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: 2023-08-02T15:12:57.078Z [WARN] server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: heartbeat timeout reached, starting election: last-leader=559cfffc-7a69-fb43-6ca4-99a924e98ca9 testLoggerAdapter verbose: server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: 2023-08-02T15:12:57.078Z [INFO] server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: entering candidate state: node="Node at bf8e15c9-b2a9-d822-6159-523a1e97ec9c [Candidate]" term=3 testing.go:149: server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: 2023-08-02T15:12:57.078Z [INFO] server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: entering candidate state: node="Node at bf8e15c9-b2a9-d822-6159-523a1e97ec9c [Candidate]" term=3 testLoggerAdapter verbose: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.078Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: duplicate requestVote for same term: term=3 testing.go:149: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.078Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: duplicate requestVote for same term: term=3 testLoggerAdapter verbose: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:57.088Z [WARN] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: not part of stable configuration, aborting election testing.go:149: server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: 2023-08-02T15:12:57.088Z [WARN] server-559cfffc-7a69-fb43-6ca4-99a924e98ca9: not part of stable configuration, aborting election testLoggerAdapter verbose: cluster: 2023-08-02T15:12:57.107Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:12:57.107Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.132Z [WARN] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: Election timeout reached, restarting election testing.go:149: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.132Z [WARN] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: Election timeout reached, restarting election testLoggerAdapter verbose: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.132Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: entering candidate state: node="Node at 064a8bac-5a48-2c3c-6153-da94e6d050c7 [Candidate]" term=4 testing.go:149: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.132Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: entering candidate state: node="Node at 064a8bac-5a48-2c3c-6153-da94e6d050c7 [Candidate]" term=4 testLoggerAdapter verbose: server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: 2023-08-02T15:12:57.132Z [INFO] server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: entering follower state: follower="Node at bf8e15c9-b2a9-d822-6159-523a1e97ec9c [Follower]" leader= testing.go:149: server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: 2023-08-02T15:12:57.132Z [INFO] server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c: entering follower state: follower="Node at bf8e15c9-b2a9-d822-6159-523a1e97ec9c [Follower]" leader= testLoggerAdapter verbose: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.132Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: election won: tally=2 testing.go:149: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.132Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: election won: tally=2 testLoggerAdapter verbose: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.132Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: entering leader state: leader="Node at 064a8bac-5a48-2c3c-6153-da94e6d050c7 [Leader]" testing.go:149: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.132Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: entering leader state: leader="Node at 064a8bac-5a48-2c3c-6153-da94e6d050c7 [Leader]" testLoggerAdapter verbose: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.132Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: added peer, starting replication: peer=server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c testing.go:149: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.132Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: added peer, starting replication: peer=server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c testLoggerAdapter verbose: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.132Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: pipelining replication: peer="{Voter server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c bf8e15c9-b2a9-d822-6159-523a1e97ec9c}" testing.go:149: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.132Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: pipelining replication: peer="{Voter server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c bf8e15c9-b2a9-d822-6159-523a1e97ec9c}" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:57.237Z [INFO] stable state for Leader reached at 2023-08-02 15:12:57.132797661 +0000 UTC m=+9.228734012 (1 nodes), 25.669304ms from start of poll, 408.590509ms from cluster start. Timeout at 2023-08-02 15:12:57.237804592 +0000 UTC m=+9.333740928, 105.006916ms after stability testing.go:149: cluster: 2023-08-02T15:12:57.237Z [INFO] stable state for Leader reached at 2023-08-02 15:12:57.132797661 +0000 UTC m=+9.228734012 (1 nodes), 25.669304ms from start of poll, 408.590509ms from cluster start. Timeout at 2023-08-02 15:12:57.237804592 +0000 UTC m=+9.333740928, 105.006916ms after stability testLoggerAdapter verbose: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.343Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: aborting pipeline replication: peer="{Voter server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c bf8e15c9-b2a9-d822-6159-523a1e97ec9c}" testing.go:149: server-064a8bac-5a48-2c3c-6153-da94e6d050c7: 2023-08-02T15:12:57.343Z [INFO] server-064a8bac-5a48-2c3c-6153-da94e6d050c7: aborting pipeline replication: peer="{Voter server-bf8e15c9-b2a9-d822-6159-523a1e97ec9c bf8e15c9-b2a9-d822-6159-523a1e97ec9c}" --- PASS: TestRaft_RemoveLeader_NoShutdown (0.62s) === RUN TestRaft_RemoveFollower_SplitCluster testLoggerAdapter verbose: server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: 2023-08-02T15:12:57.345Z [INFO] server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 Address:2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32} {Suffrage:Voter ID:server-72d5cbac-9805-20ef-6943-1ce879c34793 Address:72d5cbac-9805-20ef-6943-1ce879c34793} {Suffrage:Voter ID:server-c352a243-aaff-1c2a-41f5-7d01e4e3980f Address:c352a243-aaff-1c2a-41f5-7d01e4e3980f} {Suffrage:Voter ID:server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 Address:af9bc9da-291e-572a-d18b-aa9cc6c426a8}]" testing.go:149: server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: 2023-08-02T15:12:57.345Z [INFO] server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 Address:2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32} {Suffrage:Voter ID:server-72d5cbac-9805-20ef-6943-1ce879c34793 Address:72d5cbac-9805-20ef-6943-1ce879c34793} {Suffrage:Voter ID:server-c352a243-aaff-1c2a-41f5-7d01e4e3980f Address:c352a243-aaff-1c2a-41f5-7d01e4e3980f} {Suffrage:Voter ID:server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 Address:af9bc9da-291e-572a-d18b-aa9cc6c426a8}]" testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.345Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 Address:2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32} {Suffrage:Voter ID:server-72d5cbac-9805-20ef-6943-1ce879c34793 Address:72d5cbac-9805-20ef-6943-1ce879c34793} {Suffrage:Voter ID:server-c352a243-aaff-1c2a-41f5-7d01e4e3980f Address:c352a243-aaff-1c2a-41f5-7d01e4e3980f} {Suffrage:Voter ID:server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 Address:af9bc9da-291e-572a-d18b-aa9cc6c426a8}]" testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.345Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 Address:2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32} {Suffrage:Voter ID:server-72d5cbac-9805-20ef-6943-1ce879c34793 Address:72d5cbac-9805-20ef-6943-1ce879c34793} {Suffrage:Voter ID:server-c352a243-aaff-1c2a-41f5-7d01e4e3980f Address:c352a243-aaff-1c2a-41f5-7d01e4e3980f} {Suffrage:Voter ID:server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 Address:af9bc9da-291e-572a-d18b-aa9cc6c426a8}]" testLoggerAdapter verbose: server-c352a243-aaff-1c2a-41f5-7d01e4e3980f: 2023-08-02T15:12:57.345Z [INFO] server-c352a243-aaff-1c2a-41f5-7d01e4e3980f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 Address:2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32} {Suffrage:Voter ID:server-72d5cbac-9805-20ef-6943-1ce879c34793 Address:72d5cbac-9805-20ef-6943-1ce879c34793} {Suffrage:Voter ID:server-c352a243-aaff-1c2a-41f5-7d01e4e3980f Address:c352a243-aaff-1c2a-41f5-7d01e4e3980f} {Suffrage:Voter ID:server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 Address:af9bc9da-291e-572a-d18b-aa9cc6c426a8}]" testing.go:149: server-c352a243-aaff-1c2a-41f5-7d01e4e3980f: 2023-08-02T15:12:57.345Z [INFO] server-c352a243-aaff-1c2a-41f5-7d01e4e3980f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 Address:2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32} {Suffrage:Voter ID:server-72d5cbac-9805-20ef-6943-1ce879c34793 Address:72d5cbac-9805-20ef-6943-1ce879c34793} {Suffrage:Voter ID:server-c352a243-aaff-1c2a-41f5-7d01e4e3980f Address:c352a243-aaff-1c2a-41f5-7d01e4e3980f} {Suffrage:Voter ID:server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 Address:af9bc9da-291e-572a-d18b-aa9cc6c426a8}]" testLoggerAdapter verbose: server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: 2023-08-02T15:12:57.345Z [INFO] server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: entering follower state: follower="Node at 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 [Follower]" leader= testing.go:149: server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: 2023-08-02T15:12:57.345Z [INFO] server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: entering follower state: follower="Node at 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 [Follower]" leader= testLoggerAdapter verbose: server-af9bc9da-291e-572a-d18b-aa9cc6c426a8: 2023-08-02T15:12:57.345Z [INFO] server-af9bc9da-291e-572a-d18b-aa9cc6c426a8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 Address:2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32} {Suffrage:Voter ID:server-72d5cbac-9805-20ef-6943-1ce879c34793 Address:72d5cbac-9805-20ef-6943-1ce879c34793} {Suffrage:Voter ID:server-c352a243-aaff-1c2a-41f5-7d01e4e3980f Address:c352a243-aaff-1c2a-41f5-7d01e4e3980f} {Suffrage:Voter ID:server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 Address:af9bc9da-291e-572a-d18b-aa9cc6c426a8}]" testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.345Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: entering follower state: follower="Node at 72d5cbac-9805-20ef-6943-1ce879c34793 [Follower]" leader= testing.go:149: server-af9bc9da-291e-572a-d18b-aa9cc6c426a8: 2023-08-02T15:12:57.345Z [INFO] server-af9bc9da-291e-572a-d18b-aa9cc6c426a8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 Address:2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32} {Suffrage:Voter ID:server-72d5cbac-9805-20ef-6943-1ce879c34793 Address:72d5cbac-9805-20ef-6943-1ce879c34793} {Suffrage:Voter ID:server-c352a243-aaff-1c2a-41f5-7d01e4e3980f Address:c352a243-aaff-1c2a-41f5-7d01e4e3980f} {Suffrage:Voter ID:server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 Address:af9bc9da-291e-572a-d18b-aa9cc6c426a8}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:57.345Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:12:57.345Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-c352a243-aaff-1c2a-41f5-7d01e4e3980f: 2023-08-02T15:12:57.345Z [INFO] server-c352a243-aaff-1c2a-41f5-7d01e4e3980f: entering follower state: follower="Node at c352a243-aaff-1c2a-41f5-7d01e4e3980f [Follower]" leader= testing.go:149: server-c352a243-aaff-1c2a-41f5-7d01e4e3980f: 2023-08-02T15:12:57.345Z [INFO] server-c352a243-aaff-1c2a-41f5-7d01e4e3980f: entering follower state: follower="Node at c352a243-aaff-1c2a-41f5-7d01e4e3980f [Follower]" leader= testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.345Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: entering follower state: follower="Node at 72d5cbac-9805-20ef-6943-1ce879c34793 [Follower]" leader= testLoggerAdapter verbose: server-af9bc9da-291e-572a-d18b-aa9cc6c426a8: 2023-08-02T15:12:57.345Z [INFO] server-af9bc9da-291e-572a-d18b-aa9cc6c426a8: entering follower state: follower="Node at af9bc9da-291e-572a-d18b-aa9cc6c426a8 [Follower]" leader= testing.go:149: server-af9bc9da-291e-572a-d18b-aa9cc6c426a8: 2023-08-02T15:12:57.345Z [INFO] server-af9bc9da-291e-572a-d18b-aa9cc6c426a8: entering follower state: follower="Node at af9bc9da-291e-572a-d18b-aa9cc6c426a8 [Follower]" leader= testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.401Z [WARN] server-72d5cbac-9805-20ef-6943-1ce879c34793: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.401Z [WARN] server-72d5cbac-9805-20ef-6943-1ce879c34793: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.402Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: entering candidate state: node="Node at 72d5cbac-9805-20ef-6943-1ce879c34793 [Candidate]" term=2 testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.402Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: entering candidate state: node="Node at 72d5cbac-9805-20ef-6943-1ce879c34793 [Candidate]" term=2 testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.402Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: election won: tally=3 testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.402Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: election won: tally=3 testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.402Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: entering leader state: leader="Node at 72d5cbac-9805-20ef-6943-1ce879c34793 [Leader]" testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.402Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: entering leader state: leader="Node at 72d5cbac-9805-20ef-6943-1ce879c34793 [Leader]" testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.402Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: added peer, starting replication: peer=server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.402Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: added peer, starting replication: peer=server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.402Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: added peer, starting replication: peer=server-c352a243-aaff-1c2a-41f5-7d01e4e3980f testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.402Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: added peer, starting replication: peer=server-c352a243-aaff-1c2a-41f5-7d01e4e3980f testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.402Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: added peer, starting replication: peer=server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.402Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: added peer, starting replication: peer=server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.402Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: pipelining replication: peer="{Voter server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 af9bc9da-291e-572a-d18b-aa9cc6c426a8}" testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.402Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: pipelining replication: peer="{Voter server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 af9bc9da-291e-572a-d18b-aa9cc6c426a8}" testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.402Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: pipelining replication: peer="{Voter server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32}" testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.402Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: pipelining replication: peer="{Voter server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32}" testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.402Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: pipelining replication: peer="{Voter server-c352a243-aaff-1c2a-41f5-7d01e4e3980f c352a243-aaff-1c2a-41f5-7d01e4e3980f}" testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.402Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: pipelining replication: peer="{Voter server-c352a243-aaff-1c2a-41f5-7d01e4e3980f c352a243-aaff-1c2a-41f5-7d01e4e3980f}" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:57.507Z [INFO] stable state for Leader reached at 2023-08-02 15:12:57.402357675 +0000 UTC m=+9.498294029 (1 nodes), 56.742829ms from start of poll, 57.45646ms from cluster start. Timeout at 2023-08-02 15:12:57.507559901 +0000 UTC m=+9.603496236, 105.202207ms after stability testing.go:149: cluster: 2023-08-02T15:12:57.507Z [INFO] stable state for Leader reached at 2023-08-02 15:12:57.402357675 +0000 UTC m=+9.498294029 (1 nodes), 56.742829ms from start of poll, 57.45646ms from cluster start. Timeout at 2023-08-02 15:12:57.507559901 +0000 UTC m=+9.603496236, 105.202207ms after stability testLoggerAdapter verbose: cluster: 2023-08-02T15:12:57.613Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-08-02T15:12:57.613Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2023-08-02T15:12:57.718Z [INFO] stable state for Follower reached at 2023-08-02 15:12:57.613119085 +0000 UTC m=+9.709055424 (3 nodes), 1.97µs from start of poll, 268.217855ms from cluster start. Timeout at 2023-08-02 15:12:57.718250306 +0000 UTC m=+9.814186639, 105.131215ms after stability testing.go:149: cluster: 2023-08-02T15:12:57.718Z [INFO] stable state for Follower reached at 2023-08-02 15:12:57.613119085 +0000 UTC m=+9.709055424 (3 nodes), 1.97µs from start of poll, 268.217855ms from cluster start. Timeout at 2023-08-02 15:12:57.718250306 +0000 UTC m=+9.814186639, 105.131215ms after stability testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.718Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: updating configuration: command=RemoveServer server-id=server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 server-addr= servers="[{Suffrage:Voter ID:server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 Address:2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32} {Suffrage:Voter ID:server-72d5cbac-9805-20ef-6943-1ce879c34793 Address:72d5cbac-9805-20ef-6943-1ce879c34793} {Suffrage:Voter ID:server-c352a243-aaff-1c2a-41f5-7d01e4e3980f Address:c352a243-aaff-1c2a-41f5-7d01e4e3980f}]" testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.718Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: updating configuration: command=RemoveServer server-id=server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 server-addr= servers="[{Suffrage:Voter ID:server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 Address:2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32} {Suffrage:Voter ID:server-72d5cbac-9805-20ef-6943-1ce879c34793 Address:72d5cbac-9805-20ef-6943-1ce879c34793} {Suffrage:Voter ID:server-c352a243-aaff-1c2a-41f5-7d01e4e3980f Address:c352a243-aaff-1c2a-41f5-7d01e4e3980f}]" testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.718Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: removed peer, stopping replication: peer=server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 last-index=3 testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.718Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: removed peer, stopping replication: peer=server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 last-index=3 testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.718Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to pipeline appendEntries: peer="{Voter server-c352a243-aaff-1c2a-41f5-7d01e4e3980f c352a243-aaff-1c2a-41f5-7d01e4e3980f}" error="append pipeline closed" testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.718Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to pipeline appendEntries: peer="{Voter server-c352a243-aaff-1c2a-41f5-7d01e4e3980f c352a243-aaff-1c2a-41f5-7d01e4e3980f}" error="append pipeline closed" testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.718Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: aborting pipeline replication: peer="{Voter server-c352a243-aaff-1c2a-41f5-7d01e4e3980f c352a243-aaff-1c2a-41f5-7d01e4e3980f}" testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.718Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: aborting pipeline replication: peer="{Voter server-c352a243-aaff-1c2a-41f5-7d01e4e3980f c352a243-aaff-1c2a-41f5-7d01e4e3980f}" testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.718Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to pipeline appendEntries: peer="{Voter server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32}" error="append pipeline closed" testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.718Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to pipeline appendEntries: peer="{Voter server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32}" error="append pipeline closed" testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.718Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: aborting pipeline replication: peer="{Voter server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32}" testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.718Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: aborting pipeline replication: peer="{Voter server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32}" testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.718Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to heartbeat to: peer=c352a243-aaff-1c2a-41f5-7d01e4e3980f error="failed to connect to peer: c352a243-aaff-1c2a-41f5-7d01e4e3980f" testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.718Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to heartbeat to: peer=c352a243-aaff-1c2a-41f5-7d01e4e3980f error="failed to connect to peer: c352a243-aaff-1c2a-41f5-7d01e4e3980f" testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.718Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: aborting pipeline replication: peer="{Voter server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 af9bc9da-291e-572a-d18b-aa9cc6c426a8}" testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.718Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: aborting pipeline replication: peer="{Voter server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 af9bc9da-291e-572a-d18b-aa9cc6c426a8}" testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.718Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to heartbeat to: peer=2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 error="failed to connect to peer: 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32" testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.718Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to heartbeat to: peer=2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 error="failed to connect to peer: 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32" testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.725Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to appendEntries to: peer="{Voter server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32}" error="failed to connect to peer: 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32" testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.725Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to appendEntries to: peer="{Voter server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32}" error="failed to connect to peer: 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32" testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.726Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to appendEntries to: peer="{Voter server-c352a243-aaff-1c2a-41f5-7d01e4e3980f c352a243-aaff-1c2a-41f5-7d01e4e3980f}" error="failed to connect to peer: c352a243-aaff-1c2a-41f5-7d01e4e3980f" testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.726Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to appendEntries to: peer="{Voter server-c352a243-aaff-1c2a-41f5-7d01e4e3980f c352a243-aaff-1c2a-41f5-7d01e4e3980f}" error="failed to connect to peer: c352a243-aaff-1c2a-41f5-7d01e4e3980f" testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.735Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to heartbeat to: peer=c352a243-aaff-1c2a-41f5-7d01e4e3980f error="failed to connect to peer: c352a243-aaff-1c2a-41f5-7d01e4e3980f" testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.735Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to heartbeat to: peer=c352a243-aaff-1c2a-41f5-7d01e4e3980f error="failed to connect to peer: c352a243-aaff-1c2a-41f5-7d01e4e3980f" testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.737Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to heartbeat to: peer=2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 error="failed to connect to peer: 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32" testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.737Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to heartbeat to: peer=2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 error="failed to connect to peer: 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32" testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.745Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to appendEntries to: peer="{Voter server-c352a243-aaff-1c2a-41f5-7d01e4e3980f c352a243-aaff-1c2a-41f5-7d01e4e3980f}" error="failed to connect to peer: c352a243-aaff-1c2a-41f5-7d01e4e3980f" testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.745Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to appendEntries to: peer="{Voter server-c352a243-aaff-1c2a-41f5-7d01e4e3980f c352a243-aaff-1c2a-41f5-7d01e4e3980f}" error="failed to connect to peer: c352a243-aaff-1c2a-41f5-7d01e4e3980f" testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.745Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to appendEntries to: peer="{Voter server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32}" error="failed to connect to peer: 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32" testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.745Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to appendEntries to: peer="{Voter server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32}" error="failed to connect to peer: 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32" testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.755Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to heartbeat to: peer=c352a243-aaff-1c2a-41f5-7d01e4e3980f error="failed to connect to peer: c352a243-aaff-1c2a-41f5-7d01e4e3980f" testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.755Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to heartbeat to: peer=c352a243-aaff-1c2a-41f5-7d01e4e3980f error="failed to connect to peer: c352a243-aaff-1c2a-41f5-7d01e4e3980f" testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.755Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to heartbeat to: peer=2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 error="failed to connect to peer: 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32" testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.755Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to heartbeat to: peer=2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 error="failed to connect to peer: 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32" testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.762Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to appendEntries to: peer="{Voter server-c352a243-aaff-1c2a-41f5-7d01e4e3980f c352a243-aaff-1c2a-41f5-7d01e4e3980f}" error="failed to connect to peer: c352a243-aaff-1c2a-41f5-7d01e4e3980f" testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.762Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to appendEntries to: peer="{Voter server-c352a243-aaff-1c2a-41f5-7d01e4e3980f c352a243-aaff-1c2a-41f5-7d01e4e3980f}" error="failed to connect to peer: c352a243-aaff-1c2a-41f5-7d01e4e3980f" testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.763Z [WARN] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to contact: server-id=server-c352a243-aaff-1c2a-41f5-7d01e4e3980f time=50.078705ms testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.763Z [WARN] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to contact: server-id=server-c352a243-aaff-1c2a-41f5-7d01e4e3980f time=50.078705ms testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.765Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to appendEntries to: peer="{Voter server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32}" error="failed to connect to peer: 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32" testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.765Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to appendEntries to: peer="{Voter server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32}" error="failed to connect to peer: 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32" testLoggerAdapter verbose: server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: 2023-08-02T15:12:57.772Z [WARN] server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: heartbeat timeout reached, starting election: last-leader=72d5cbac-9805-20ef-6943-1ce879c34793 testing.go:149: server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: 2023-08-02T15:12:57.772Z [WARN] server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: heartbeat timeout reached, starting election: last-leader=72d5cbac-9805-20ef-6943-1ce879c34793 testLoggerAdapter verbose: server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: 2023-08-02T15:12:57.772Z [INFO] server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: entering candidate state: node="Node at 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 [Candidate]" term=3 testing.go:149: server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: 2023-08-02T15:12:57.772Z [INFO] server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: entering candidate state: node="Node at 2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 [Candidate]" term=3 testLoggerAdapter verbose: server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: 2023-08-02T15:12:57.772Z [ERROR] server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: failed to make requestVote RPC: target="{Voter server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 af9bc9da-291e-572a-d18b-aa9cc6c426a8}" error="failed to connect to peer: af9bc9da-291e-572a-d18b-aa9cc6c426a8" testing.go:149: server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: 2023-08-02T15:12:57.772Z [ERROR] server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: failed to make requestVote RPC: target="{Voter server-af9bc9da-291e-572a-d18b-aa9cc6c426a8 af9bc9da-291e-572a-d18b-aa9cc6c426a8}" error="failed to connect to peer: af9bc9da-291e-572a-d18b-aa9cc6c426a8" testLoggerAdapter verbose: server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: 2023-08-02T15:12:57.772Z [ERROR] server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: failed to make requestVote RPC: target="{Voter server-72d5cbac-9805-20ef-6943-1ce879c34793 72d5cbac-9805-20ef-6943-1ce879c34793}" error="failed to connect to peer: 72d5cbac-9805-20ef-6943-1ce879c34793" testing.go:149: server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: 2023-08-02T15:12:57.772Z [ERROR] server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32: failed to make requestVote RPC: target="{Voter server-72d5cbac-9805-20ef-6943-1ce879c34793 72d5cbac-9805-20ef-6943-1ce879c34793}" error="failed to connect to peer: 72d5cbac-9805-20ef-6943-1ce879c34793" testLoggerAdapter verbose: server-c352a243-aaff-1c2a-41f5-7d01e4e3980f: 2023-08-02T15:12:57.772Z [WARN] server-c352a243-aaff-1c2a-41f5-7d01e4e3980f: rejecting vote request since we have a leader: from=2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 leader=72d5cbac-9805-20ef-6943-1ce879c34793 testing.go:149: server-c352a243-aaff-1c2a-41f5-7d01e4e3980f: 2023-08-02T15:12:57.772Z [WARN] server-c352a243-aaff-1c2a-41f5-7d01e4e3980f: rejecting vote request since we have a leader: from=2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 leader=72d5cbac-9805-20ef-6943-1ce879c34793 testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.773Z [WARN] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to contact: server-id=server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 time=55.818744ms testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.773Z [WARN] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to contact: server-id=server-2d12ae8f-3e1b-ce79-e8b0-2e4cccb68a32 time=55.818744ms testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.773Z [WARN] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to contact: server-id=server-c352a243-aaff-1c2a-41f5-7d01e4e3980f time=60.547117ms testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.773Z [WARN] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to contact: server-id=server-c352a243-aaff-1c2a-41f5-7d01e4e3980f time=60.547117ms testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.773Z [WARN] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to contact quorum of nodes, stepping down testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.773Z [WARN] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to contact quorum of nodes, stepping down testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.773Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: entering follower state: follower="Node at 72d5cbac-9805-20ef-6943-1ce879c34793 [Follower]" leader= testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.773Z [INFO] server-72d5cbac-9805-20ef-6943-1ce879c34793: entering follower state: follower="Node at 72d5cbac-9805-20ef-6943-1ce879c34793 [Follower]" leader= testLoggerAdapter verbose: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.773Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to appendEntries to: peer="{Voter server-c352a243-aaff-1c2a-41f5-7d01e4e3980f c352a243-aaff-1c2a-41f5-7d01e4e3980f}" error="failed to connect to peer: c352a243-aaff-1c2a-41f5-7d01e4e3980f" testing.go:149: server-72d5cbac-9805-20ef-6943-1ce879c34793: 2023-08-02T15:12:57.773Z [ERROR] server-72d5cbac-9805-20ef-6943-1ce879c34793: failed to appendEntries to: peer="{Voter server-c352a243-aaff-1c2a-41f5-7d01e4e3980f c352a243-aaff-1c2a-41f5-7d01e4e3980f}" error="failed to connect to peer: c352a243-aaff-1c2a-41f5-7d01e4e3980f" --- PASS: TestRaft_RemoveFollower_SplitCluster (0.43s) === RUN TestRaft_AddKnownPeer testLoggerAdapter verbose: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.775Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968 Address:edc63f06-c6c2-f61c-4a9a-b0fb889b2968} {Suffrage:Voter ID:server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd Address:13ecbf72-8fd5-c6d3-559b-ee06ae55cddd} {Suffrage:Voter ID:server-2393508c-a264-c68d-6a01-4620c05d8d00 Address:2393508c-a264-c68d-6a01-4620c05d8d00}]" testing.go:149: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.775Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968 Address:edc63f06-c6c2-f61c-4a9a-b0fb889b2968} {Suffrage:Voter ID:server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd Address:13ecbf72-8fd5-c6d3-559b-ee06ae55cddd} {Suffrage:Voter ID:server-2393508c-a264-c68d-6a01-4620c05d8d00 Address:2393508c-a264-c68d-6a01-4620c05d8d00}]" testLoggerAdapter verbose: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.775Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: entering follower state: follower="Node at edc63f06-c6c2-f61c-4a9a-b0fb889b2968 [Follower]" leader= testLoggerAdapter verbose: server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd: 2023-08-02T15:12:57.775Z [INFO] server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968 Address:edc63f06-c6c2-f61c-4a9a-b0fb889b2968} {Suffrage:Voter ID:server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd Address:13ecbf72-8fd5-c6d3-559b-ee06ae55cddd} {Suffrage:Voter ID:server-2393508c-a264-c68d-6a01-4620c05d8d00 Address:2393508c-a264-c68d-6a01-4620c05d8d00}]" testing.go:149: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.775Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: entering follower state: follower="Node at edc63f06-c6c2-f61c-4a9a-b0fb889b2968 [Follower]" leader= testing.go:149: server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd: 2023-08-02T15:12:57.775Z [INFO] server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968 Address:edc63f06-c6c2-f61c-4a9a-b0fb889b2968} {Suffrage:Voter ID:server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd Address:13ecbf72-8fd5-c6d3-559b-ee06ae55cddd} {Suffrage:Voter ID:server-2393508c-a264-c68d-6a01-4620c05d8d00 Address:2393508c-a264-c68d-6a01-4620c05d8d00}]" testLoggerAdapter verbose: server-2393508c-a264-c68d-6a01-4620c05d8d00: 2023-08-02T15:12:57.776Z [INFO] server-2393508c-a264-c68d-6a01-4620c05d8d00: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968 Address:edc63f06-c6c2-f61c-4a9a-b0fb889b2968} {Suffrage:Voter ID:server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd Address:13ecbf72-8fd5-c6d3-559b-ee06ae55cddd} {Suffrage:Voter ID:server-2393508c-a264-c68d-6a01-4620c05d8d00 Address:2393508c-a264-c68d-6a01-4620c05d8d00}]" testing.go:149: server-2393508c-a264-c68d-6a01-4620c05d8d00: 2023-08-02T15:12:57.776Z [INFO] server-2393508c-a264-c68d-6a01-4620c05d8d00: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968 Address:edc63f06-c6c2-f61c-4a9a-b0fb889b2968} {Suffrage:Voter ID:server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd Address:13ecbf72-8fd5-c6d3-559b-ee06ae55cddd} {Suffrage:Voter ID:server-2393508c-a264-c68d-6a01-4620c05d8d00 Address:2393508c-a264-c68d-6a01-4620c05d8d00}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:57.776Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:12:57.776Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd: 2023-08-02T15:12:57.776Z [INFO] server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd: entering follower state: follower="Node at 13ecbf72-8fd5-c6d3-559b-ee06ae55cddd [Follower]" leader= testing.go:149: server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd: 2023-08-02T15:12:57.776Z [INFO] server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd: entering follower state: follower="Node at 13ecbf72-8fd5-c6d3-559b-ee06ae55cddd [Follower]" leader= testLoggerAdapter verbose: server-2393508c-a264-c68d-6a01-4620c05d8d00: 2023-08-02T15:12:57.776Z [INFO] server-2393508c-a264-c68d-6a01-4620c05d8d00: entering follower state: follower="Node at 2393508c-a264-c68d-6a01-4620c05d8d00 [Follower]" leader= testing.go:149: server-2393508c-a264-c68d-6a01-4620c05d8d00: 2023-08-02T15:12:57.776Z [INFO] server-2393508c-a264-c68d-6a01-4620c05d8d00: entering follower state: follower="Node at 2393508c-a264-c68d-6a01-4620c05d8d00 [Follower]" leader= testLoggerAdapter verbose: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.835Z [WARN] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.835Z [WARN] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.835Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: entering candidate state: node="Node at edc63f06-c6c2-f61c-4a9a-b0fb889b2968 [Candidate]" term=2 testing.go:149: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.835Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: entering candidate state: node="Node at edc63f06-c6c2-f61c-4a9a-b0fb889b2968 [Candidate]" term=2 testLoggerAdapter verbose: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.835Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: election won: tally=2 testing.go:149: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.835Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: election won: tally=2 testLoggerAdapter verbose: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.835Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: entering leader state: leader="Node at edc63f06-c6c2-f61c-4a9a-b0fb889b2968 [Leader]" testing.go:149: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.835Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: entering leader state: leader="Node at edc63f06-c6c2-f61c-4a9a-b0fb889b2968 [Leader]" testLoggerAdapter verbose: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.835Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: added peer, starting replication: peer=server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd testing.go:149: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.835Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: added peer, starting replication: peer=server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd testLoggerAdapter verbose: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.835Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: added peer, starting replication: peer=server-2393508c-a264-c68d-6a01-4620c05d8d00 testing.go:149: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.835Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: added peer, starting replication: peer=server-2393508c-a264-c68d-6a01-4620c05d8d00 testLoggerAdapter verbose: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.835Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: pipelining replication: peer="{Voter server-2393508c-a264-c68d-6a01-4620c05d8d00 2393508c-a264-c68d-6a01-4620c05d8d00}" testing.go:149: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.835Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: pipelining replication: peer="{Voter server-2393508c-a264-c68d-6a01-4620c05d8d00 2393508c-a264-c68d-6a01-4620c05d8d00}" testLoggerAdapter verbose: server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd: 2023-08-02T15:12:57.835Z [WARN] server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd: rejecting vote request since our last term is greater: candidate=edc63f06-c6c2-f61c-4a9a-b0fb889b2968 last-term=2 last-candidate-term=1 testing.go:149: server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd: 2023-08-02T15:12:57.835Z [WARN] server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd: rejecting vote request since our last term is greater: candidate=edc63f06-c6c2-f61c-4a9a-b0fb889b2968 last-term=2 last-candidate-term=1 testLoggerAdapter verbose: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.835Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: pipelining replication: peer="{Voter server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd 13ecbf72-8fd5-c6d3-559b-ee06ae55cddd}" testing.go:149: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:57.835Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: pipelining replication: peer="{Voter server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd 13ecbf72-8fd5-c6d3-559b-ee06ae55cddd}" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:57.940Z [INFO] stable state for Leader reached at 2023-08-02 15:12:57.835642048 +0000 UTC m=+9.931578382 (1 nodes), 59.407415ms from start of poll, 60.744835ms from cluster start. Timeout at 2023-08-02 15:12:57.94072152 +0000 UTC m=+10.036657888, 105.079506ms after stability testing.go:149: cluster: 2023-08-02T15:12:57.940Z [INFO] stable state for Leader reached at 2023-08-02 15:12:57.835642048 +0000 UTC m=+9.931578382 (1 nodes), 59.407415ms from start of poll, 60.744835ms from cluster start. Timeout at 2023-08-02 15:12:57.94072152 +0000 UTC m=+10.036657888, 105.079506ms after stability testLoggerAdapter verbose: cluster: 2023-08-02T15:12:57.940Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-08-02T15:12:57.940Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2023-08-02T15:12:58.046Z [INFO] stable state for Follower reached at 2023-08-02 15:12:57.940942397 +0000 UTC m=+10.036878734 (2 nodes), 1.85µs from start of poll, 166.045187ms from cluster start. Timeout at 2023-08-02 15:12:58.045974526 +0000 UTC m=+10.141910851, 105.032117ms after stability testing.go:149: cluster: 2023-08-02T15:12:58.046Z [INFO] stable state for Follower reached at 2023-08-02 15:12:57.940942397 +0000 UTC m=+10.036878734 (2 nodes), 1.85µs from start of poll, 166.045187ms from cluster start. Timeout at 2023-08-02 15:12:58.045974526 +0000 UTC m=+10.141910851, 105.032117ms after stability testLoggerAdapter verbose: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:58.046Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: updating configuration: command=AddStaging server-id=server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd server-addr=13ecbf72-8fd5-c6d3-559b-ee06ae55cddd servers="[{Suffrage:Voter ID:server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968 Address:edc63f06-c6c2-f61c-4a9a-b0fb889b2968} {Suffrage:Voter ID:server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd Address:13ecbf72-8fd5-c6d3-559b-ee06ae55cddd} {Suffrage:Voter ID:server-2393508c-a264-c68d-6a01-4620c05d8d00 Address:2393508c-a264-c68d-6a01-4620c05d8d00}]" testing.go:149: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:58.046Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: updating configuration: command=AddStaging server-id=server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd server-addr=13ecbf72-8fd5-c6d3-559b-ee06ae55cddd servers="[{Suffrage:Voter ID:server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968 Address:edc63f06-c6c2-f61c-4a9a-b0fb889b2968} {Suffrage:Voter ID:server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd Address:13ecbf72-8fd5-c6d3-559b-ee06ae55cddd} {Suffrage:Voter ID:server-2393508c-a264-c68d-6a01-4620c05d8d00 Address:2393508c-a264-c68d-6a01-4620c05d8d00}]" testLoggerAdapter verbose: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:58.046Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: aborting pipeline replication: peer="{Voter server-2393508c-a264-c68d-6a01-4620c05d8d00 2393508c-a264-c68d-6a01-4620c05d8d00}" testing.go:149: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:58.046Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: aborting pipeline replication: peer="{Voter server-2393508c-a264-c68d-6a01-4620c05d8d00 2393508c-a264-c68d-6a01-4620c05d8d00}" testLoggerAdapter verbose: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:58.046Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: aborting pipeline replication: peer="{Voter server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd 13ecbf72-8fd5-c6d3-559b-ee06ae55cddd}" testing.go:149: server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: 2023-08-02T15:12:58.046Z [INFO] server-edc63f06-c6c2-f61c-4a9a-b0fb889b2968: aborting pipeline replication: peer="{Voter server-13ecbf72-8fd5-c6d3-559b-ee06ae55cddd 13ecbf72-8fd5-c6d3-559b-ee06ae55cddd}" --- PASS: TestRaft_AddKnownPeer (0.27s) === RUN TestRaft_RemoveUnknownPeer testLoggerAdapter verbose: server-fbc18987-2172-6726-425e-580f0ca1aabf: 2023-08-02T15:12:58.047Z [INFO] server-fbc18987-2172-6726-425e-580f0ca1aabf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fbc18987-2172-6726-425e-580f0ca1aabf Address:fbc18987-2172-6726-425e-580f0ca1aabf} {Suffrage:Voter ID:server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697 Address:ba99bbaf-1217-e8ac-c80d-3d3ff83a4697} {Suffrage:Voter ID:server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b Address:9cc529c2-6e9e-abe0-0f91-27f748f76e3b}]" testing.go:149: server-fbc18987-2172-6726-425e-580f0ca1aabf: 2023-08-02T15:12:58.047Z [INFO] server-fbc18987-2172-6726-425e-580f0ca1aabf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fbc18987-2172-6726-425e-580f0ca1aabf Address:fbc18987-2172-6726-425e-580f0ca1aabf} {Suffrage:Voter ID:server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697 Address:ba99bbaf-1217-e8ac-c80d-3d3ff83a4697} {Suffrage:Voter ID:server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b Address:9cc529c2-6e9e-abe0-0f91-27f748f76e3b}]" testLoggerAdapter verbose: server-fbc18987-2172-6726-425e-580f0ca1aabf: 2023-08-02T15:12:58.047Z [INFO] server-fbc18987-2172-6726-425e-580f0ca1aabf: entering follower state: follower="Node at fbc18987-2172-6726-425e-580f0ca1aabf [Follower]" leader= testing.go:149: server-fbc18987-2172-6726-425e-580f0ca1aabf: 2023-08-02T15:12:58.047Z [INFO] server-fbc18987-2172-6726-425e-580f0ca1aabf: entering follower state: follower="Node at fbc18987-2172-6726-425e-580f0ca1aabf [Follower]" leader= testLoggerAdapter verbose: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.047Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fbc18987-2172-6726-425e-580f0ca1aabf Address:fbc18987-2172-6726-425e-580f0ca1aabf} {Suffrage:Voter ID:server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697 Address:ba99bbaf-1217-e8ac-c80d-3d3ff83a4697} {Suffrage:Voter ID:server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b Address:9cc529c2-6e9e-abe0-0f91-27f748f76e3b}]" testing.go:149: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.047Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fbc18987-2172-6726-425e-580f0ca1aabf Address:fbc18987-2172-6726-425e-580f0ca1aabf} {Suffrage:Voter ID:server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697 Address:ba99bbaf-1217-e8ac-c80d-3d3ff83a4697} {Suffrage:Voter ID:server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b Address:9cc529c2-6e9e-abe0-0f91-27f748f76e3b}]" testLoggerAdapter verbose: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.048Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: entering follower state: follower="Node at ba99bbaf-1217-e8ac-c80d-3d3ff83a4697 [Follower]" leader= testing.go:149: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.048Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: entering follower state: follower="Node at ba99bbaf-1217-e8ac-c80d-3d3ff83a4697 [Follower]" leader= testLoggerAdapter verbose: server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b: 2023-08-02T15:12:58.048Z [INFO] server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fbc18987-2172-6726-425e-580f0ca1aabf Address:fbc18987-2172-6726-425e-580f0ca1aabf} {Suffrage:Voter ID:server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697 Address:ba99bbaf-1217-e8ac-c80d-3d3ff83a4697} {Suffrage:Voter ID:server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b Address:9cc529c2-6e9e-abe0-0f91-27f748f76e3b}]" testing.go:149: server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b: 2023-08-02T15:12:58.048Z [INFO] server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fbc18987-2172-6726-425e-580f0ca1aabf Address:fbc18987-2172-6726-425e-580f0ca1aabf} {Suffrage:Voter ID:server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697 Address:ba99bbaf-1217-e8ac-c80d-3d3ff83a4697} {Suffrage:Voter ID:server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b Address:9cc529c2-6e9e-abe0-0f91-27f748f76e3b}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:58.048Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:12:58.048Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b: 2023-08-02T15:12:58.048Z [INFO] server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b: entering follower state: follower="Node at 9cc529c2-6e9e-abe0-0f91-27f748f76e3b [Follower]" leader= testing.go:149: server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b: 2023-08-02T15:12:58.048Z [INFO] server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b: entering follower state: follower="Node at 9cc529c2-6e9e-abe0-0f91-27f748f76e3b [Follower]" leader= testLoggerAdapter verbose: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.116Z [WARN] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.116Z [WARN] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.116Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: entering candidate state: node="Node at ba99bbaf-1217-e8ac-c80d-3d3ff83a4697 [Candidate]" term=2 testing.go:149: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.116Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: entering candidate state: node="Node at ba99bbaf-1217-e8ac-c80d-3d3ff83a4697 [Candidate]" term=2 testLoggerAdapter verbose: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.116Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: election won: tally=2 testing.go:149: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.116Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: election won: tally=2 testLoggerAdapter verbose: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.117Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: entering leader state: leader="Node at ba99bbaf-1217-e8ac-c80d-3d3ff83a4697 [Leader]" testing.go:149: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.117Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: entering leader state: leader="Node at ba99bbaf-1217-e8ac-c80d-3d3ff83a4697 [Leader]" testLoggerAdapter verbose: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.117Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: added peer, starting replication: peer=server-fbc18987-2172-6726-425e-580f0ca1aabf testing.go:149: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.117Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: added peer, starting replication: peer=server-fbc18987-2172-6726-425e-580f0ca1aabf testLoggerAdapter verbose: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.117Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: added peer, starting replication: peer=server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b testing.go:149: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.117Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: added peer, starting replication: peer=server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b testLoggerAdapter verbose: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.117Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: pipelining replication: peer="{Voter server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b 9cc529c2-6e9e-abe0-0f91-27f748f76e3b}" testing.go:149: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.117Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: pipelining replication: peer="{Voter server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b 9cc529c2-6e9e-abe0-0f91-27f748f76e3b}" testLoggerAdapter verbose: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.117Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: pipelining replication: peer="{Voter server-fbc18987-2172-6726-425e-580f0ca1aabf fbc18987-2172-6726-425e-580f0ca1aabf}" testLoggerAdapter verbose: server-fbc18987-2172-6726-425e-580f0ca1aabf: 2023-08-02T15:12:58.117Z [WARN] server-fbc18987-2172-6726-425e-580f0ca1aabf: rejecting vote request since our last term is greater: candidate=ba99bbaf-1217-e8ac-c80d-3d3ff83a4697 last-term=2 last-candidate-term=1 testing.go:149: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.117Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: pipelining replication: peer="{Voter server-fbc18987-2172-6726-425e-580f0ca1aabf fbc18987-2172-6726-425e-580f0ca1aabf}" testing.go:149: server-fbc18987-2172-6726-425e-580f0ca1aabf: 2023-08-02T15:12:58.117Z [WARN] server-fbc18987-2172-6726-425e-580f0ca1aabf: rejecting vote request since our last term is greater: candidate=ba99bbaf-1217-e8ac-c80d-3d3ff83a4697 last-term=2 last-candidate-term=1 testLoggerAdapter verbose: cluster: 2023-08-02T15:12:58.222Z [INFO] stable state for Leader reached at 2023-08-02 15:12:58.117392465 +0000 UTC m=+10.213328802 (1 nodes), 69.096338ms from start of poll, 69.875016ms from cluster start. Timeout at 2023-08-02 15:12:58.222523436 +0000 UTC m=+10.318459770, 105.130968ms after stability testing.go:149: cluster: 2023-08-02T15:12:58.222Z [INFO] stable state for Leader reached at 2023-08-02 15:12:58.117392465 +0000 UTC m=+10.213328802 (1 nodes), 69.096338ms from start of poll, 69.875016ms from cluster start. Timeout at 2023-08-02 15:12:58.222523436 +0000 UTC m=+10.318459770, 105.130968ms after stability testLoggerAdapter verbose: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.222Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: updating configuration: command=RemoveServer server-id=9d9e9ce2-d4a5-fc88-1aaf-97f96cb011f2 server-addr= servers="[{Suffrage:Voter ID:server-fbc18987-2172-6726-425e-580f0ca1aabf Address:fbc18987-2172-6726-425e-580f0ca1aabf} {Suffrage:Voter ID:server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697 Address:ba99bbaf-1217-e8ac-c80d-3d3ff83a4697} {Suffrage:Voter ID:server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b Address:9cc529c2-6e9e-abe0-0f91-27f748f76e3b}]" testing.go:149: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.222Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: updating configuration: command=RemoveServer server-id=9d9e9ce2-d4a5-fc88-1aaf-97f96cb011f2 server-addr= servers="[{Suffrage:Voter ID:server-fbc18987-2172-6726-425e-580f0ca1aabf Address:fbc18987-2172-6726-425e-580f0ca1aabf} {Suffrage:Voter ID:server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697 Address:ba99bbaf-1217-e8ac-c80d-3d3ff83a4697} {Suffrage:Voter ID:server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b Address:9cc529c2-6e9e-abe0-0f91-27f748f76e3b}]" testLoggerAdapter verbose: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.223Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: aborting pipeline replication: peer="{Voter server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b 9cc529c2-6e9e-abe0-0f91-27f748f76e3b}" testing.go:149: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.223Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: aborting pipeline replication: peer="{Voter server-9cc529c2-6e9e-abe0-0f91-27f748f76e3b 9cc529c2-6e9e-abe0-0f91-27f748f76e3b}" testLoggerAdapter verbose: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.223Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: aborting pipeline replication: peer="{Voter server-fbc18987-2172-6726-425e-580f0ca1aabf fbc18987-2172-6726-425e-580f0ca1aabf}" testing.go:149: server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: 2023-08-02T15:12:58.223Z [INFO] server-ba99bbaf-1217-e8ac-c80d-3d3ff83a4697: aborting pipeline replication: peer="{Voter server-fbc18987-2172-6726-425e-580f0ca1aabf fbc18987-2172-6726-425e-580f0ca1aabf}" --- PASS: TestRaft_RemoveUnknownPeer (0.18s) === RUN TestRaft_SnapshotRestore testLoggerAdapter verbose: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.223Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e Address:b6cdb39d-6a85-bda7-38a7-abcb262e1d8e}]" testing.go:149: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.223Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e Address:b6cdb39d-6a85-bda7-38a7-abcb262e1d8e}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:58.223Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:12:58.223Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.223Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: entering follower state: follower="Node at b6cdb39d-6a85-bda7-38a7-abcb262e1d8e [Follower]" leader= testing.go:149: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.223Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: entering follower state: follower="Node at b6cdb39d-6a85-bda7-38a7-abcb262e1d8e [Follower]" leader= testLoggerAdapter verbose: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.290Z [WARN] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.290Z [WARN] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.290Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: entering candidate state: node="Node at b6cdb39d-6a85-bda7-38a7-abcb262e1d8e [Candidate]" term=2 testing.go:149: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.290Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: entering candidate state: node="Node at b6cdb39d-6a85-bda7-38a7-abcb262e1d8e [Candidate]" term=2 testLoggerAdapter verbose: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.290Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: election won: tally=1 testing.go:149: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.290Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: election won: tally=1 testLoggerAdapter verbose: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.290Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: entering leader state: leader="Node at b6cdb39d-6a85-bda7-38a7-abcb262e1d8e [Leader]" testing.go:149: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.290Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: entering leader state: leader="Node at b6cdb39d-6a85-bda7-38a7-abcb262e1d8e [Leader]" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:58.395Z [INFO] stable state for Leader reached at 2023-08-02 15:12:58.290594284 +0000 UTC m=+10.386530617 (1 nodes), 66.688321ms from start of poll, 66.932408ms from cluster start. Timeout at 2023-08-02 15:12:58.395850695 +0000 UTC m=+10.491787030, 105.256413ms after stability testing.go:149: cluster: 2023-08-02T15:12:58.395Z [INFO] stable state for Leader reached at 2023-08-02 15:12:58.290594284 +0000 UTC m=+10.386530617 (1 nodes), 66.688321ms from start of poll, 66.932408ms from cluster start. Timeout at 2023-08-02 15:12:58.395850695 +0000 UTC m=+10.491787030, 105.256413ms after stability testLoggerAdapter verbose: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.397Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: starting snapshot up to: index=102 testing.go:149: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.397Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: starting snapshot up to: index=102 testing.go:153: 2023-08-02T15:12:58.397Z [INFO] creating new snapshot: path=/tmp/raft356815295/snapshots/2-102-1690989178397.tmp testLoggerAdapter verbose: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.400Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: compacting logs: from=1 to=92 testing.go:149: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.400Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.400Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: snapshot complete up to: index=102 testing.go:149: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.400Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: snapshot complete up to: index=102 testLoggerAdapter verbose: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.400Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: restored from snapshot: id=2-102-1690989178397 testing.go:149: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.400Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: restored from snapshot: id=2-102-1690989178397 testLoggerAdapter verbose: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.400Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e Address:b6cdb39d-6a85-bda7-38a7-abcb262e1d8e}]" testing.go:149: server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: 2023-08-02T15:12:58.400Z [INFO] server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6cdb39d-6a85-bda7-38a7-abcb262e1d8e Address:b6cdb39d-6a85-bda7-38a7-abcb262e1d8e}]" --- PASS: TestRaft_SnapshotRestore (0.18s) === RUN TestRaft_NoRestoreOnStart testLoggerAdapter verbose: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.402Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424 Address:b4f9dcb1-6c91-d98c-2202-bf5a3a26d424}]" testing.go:149: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.402Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424 Address:b4f9dcb1-6c91-d98c-2202-bf5a3a26d424}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:58.402Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:12:58.402Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.402Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: entering follower state: follower="Node at b4f9dcb1-6c91-d98c-2202-bf5a3a26d424 [Follower]" leader= testing.go:149: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.402Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: entering follower state: follower="Node at b4f9dcb1-6c91-d98c-2202-bf5a3a26d424 [Follower]" leader= testLoggerAdapter verbose: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.476Z [WARN] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.476Z [WARN] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.476Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: entering candidate state: node="Node at b4f9dcb1-6c91-d98c-2202-bf5a3a26d424 [Candidate]" term=2 testing.go:149: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.476Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: entering candidate state: node="Node at b4f9dcb1-6c91-d98c-2202-bf5a3a26d424 [Candidate]" term=2 testLoggerAdapter verbose: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.476Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: election won: tally=1 testing.go:149: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.476Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: election won: tally=1 testLoggerAdapter verbose: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.476Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: entering leader state: leader="Node at b4f9dcb1-6c91-d98c-2202-bf5a3a26d424 [Leader]" testing.go:149: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.476Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: entering leader state: leader="Node at b4f9dcb1-6c91-d98c-2202-bf5a3a26d424 [Leader]" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:58.582Z [INFO] stable state for Leader reached at 2023-08-02 15:12:58.476486081 +0000 UTC m=+10.572422420 (1 nodes), 74.199832ms from start of poll, 74.73178ms from cluster start. Timeout at 2023-08-02 15:12:58.582165972 +0000 UTC m=+10.678102323, 105.679903ms after stability testing.go:149: cluster: 2023-08-02T15:12:58.582Z [INFO] stable state for Leader reached at 2023-08-02 15:12:58.476486081 +0000 UTC m=+10.572422420 (1 nodes), 74.199832ms from start of poll, 74.73178ms from cluster start. Timeout at 2023-08-02 15:12:58.582165972 +0000 UTC m=+10.678102323, 105.679903ms after stability testLoggerAdapter verbose: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.583Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: starting snapshot up to: index=102 testing.go:149: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.583Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: starting snapshot up to: index=102 testing.go:153: 2023-08-02T15:12:58.584Z [INFO] creating new snapshot: path=/tmp/raft012387401/snapshots/2-102-1690989178583.tmp testLoggerAdapter verbose: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.584Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: compacting logs: from=1 to=92 testing.go:149: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.584Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.584Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: snapshot complete up to: index=102 testing.go:149: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.584Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: snapshot complete up to: index=102 testLoggerAdapter verbose: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.584Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424 Address:b4f9dcb1-6c91-d98c-2202-bf5a3a26d424}]" testing.go:149: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.584Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424 Address:b4f9dcb1-6c91-d98c-2202-bf5a3a26d424}]" --- PASS: TestRaft_NoRestoreOnStart (0.18s) === RUN TestRaft_SnapshotRestore_PeerChange raft_test.go:1026: Disabled unreliable test --- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s) === RUN TestRaft_AutoSnapshot raft_test.go:1132: Disabled unreliable test --- SKIP: TestRaft_AutoSnapshot (0.00s) === RUN TestRaft_UserSnapshot testLoggerAdapter verbose: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.585Z [INFO] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-902566b9-8890-c51f-12ae-ef78a06fa7e3 Address:902566b9-8890-c51f-12ae-ef78a06fa7e3}]" testing.go:149: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.585Z [INFO] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-902566b9-8890-c51f-12ae-ef78a06fa7e3 Address:902566b9-8890-c51f-12ae-ef78a06fa7e3}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:58.585Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:12:58.585Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.585Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: entering follower state: follower="Node at b4f9dcb1-6c91-d98c-2202-bf5a3a26d424 [Follower]" leader= testLoggerAdapter verbose: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.585Z [INFO] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: entering follower state: follower="Node at 902566b9-8890-c51f-12ae-ef78a06fa7e3 [Follower]" leader= testing.go:149: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.585Z [INFO] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: entering follower state: follower="Node at 902566b9-8890-c51f-12ae-ef78a06fa7e3 [Follower]" leader= testLoggerAdapter verbose: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.667Z [WARN] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.667Z [WARN] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.668Z [INFO] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: entering candidate state: node="Node at 902566b9-8890-c51f-12ae-ef78a06fa7e3 [Candidate]" term=2 testing.go:149: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.668Z [INFO] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: entering candidate state: node="Node at 902566b9-8890-c51f-12ae-ef78a06fa7e3 [Candidate]" term=2 testLoggerAdapter verbose: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.668Z [INFO] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: election won: tally=1 testing.go:149: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.668Z [INFO] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: election won: tally=1 testLoggerAdapter verbose: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.668Z [INFO] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: entering leader state: leader="Node at 902566b9-8890-c51f-12ae-ef78a06fa7e3 [Leader]" testing.go:149: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.668Z [INFO] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: entering leader state: leader="Node at 902566b9-8890-c51f-12ae-ef78a06fa7e3 [Leader]" testLoggerAdapter verbose: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.679Z [WARN] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.679Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: entering candidate state: node="Node at b4f9dcb1-6c91-d98c-2202-bf5a3a26d424 [Candidate]" term=3 testLoggerAdapter verbose: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.679Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: election won: tally=1 testLoggerAdapter verbose: server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: 2023-08-02T15:12:58.679Z [INFO] server-b4f9dcb1-6c91-d98c-2202-bf5a3a26d424: entering leader state: leader="Node at b4f9dcb1-6c91-d98c-2202-bf5a3a26d424 [Leader]" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:58.773Z [INFO] stable state for Leader reached at 2023-08-02 15:12:58.668285644 +0000 UTC m=+10.764222010 (1 nodes), 83.123386ms from start of poll, 83.288678ms from cluster start. Timeout at 2023-08-02 15:12:58.77373504 +0000 UTC m=+10.869671410, 105.4494ms after stability testing.go:149: cluster: 2023-08-02T15:12:58.773Z [INFO] stable state for Leader reached at 2023-08-02 15:12:58.668285644 +0000 UTC m=+10.764222010 (1 nodes), 83.123386ms from start of poll, 83.288678ms from cluster start. Timeout at 2023-08-02 15:12:58.77373504 +0000 UTC m=+10.869671410, 105.4494ms after stability testLoggerAdapter verbose: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.773Z [ERROR] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: failed to take snapshot: error="nothing new to snapshot" testing.go:149: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.773Z [ERROR] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: failed to take snapshot: error="nothing new to snapshot" testLoggerAdapter verbose: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.774Z [INFO] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: starting snapshot up to: index=12 testing.go:149: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.774Z [INFO] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: starting snapshot up to: index=12 testing.go:153: 2023-08-02T15:12:58.774Z [INFO] creating new snapshot: path=/tmp/raft815413603/snapshots/2-12-1690989178774.tmp testLoggerAdapter verbose: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.774Z [INFO] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: compacting logs: from=1 to=2 testing.go:149: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.774Z [INFO] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: compacting logs: from=1 to=2 testLoggerAdapter verbose: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.774Z [INFO] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: snapshot complete up to: index=12 testing.go:149: server-902566b9-8890-c51f-12ae-ef78a06fa7e3: 2023-08-02T15:12:58.774Z [INFO] server-902566b9-8890-c51f-12ae-ef78a06fa7e3: snapshot complete up to: index=12 --- PASS: TestRaft_UserSnapshot (0.19s) === RUN TestRaft_UserRestore testLoggerAdapter verbose: server-f8811921-2735-d284-f23e-ce90eafcb10a: 2023-08-02T15:12:58.775Z [INFO] server-f8811921-2735-d284-f23e-ce90eafcb10a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f8811921-2735-d284-f23e-ce90eafcb10a Address:f8811921-2735-d284-f23e-ce90eafcb10a} {Suffrage:Voter ID:server-f25432d1-1355-b829-f9d5-1c675504ab50 Address:f25432d1-1355-b829-f9d5-1c675504ab50} {Suffrage:Voter ID:server-11a0d01c-4a46-e06a-e67b-844791818f99 Address:11a0d01c-4a46-e06a-e67b-844791818f99}]" testing.go:149: server-f8811921-2735-d284-f23e-ce90eafcb10a: 2023-08-02T15:12:58.775Z [INFO] server-f8811921-2735-d284-f23e-ce90eafcb10a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f8811921-2735-d284-f23e-ce90eafcb10a Address:f8811921-2735-d284-f23e-ce90eafcb10a} {Suffrage:Voter ID:server-f25432d1-1355-b829-f9d5-1c675504ab50 Address:f25432d1-1355-b829-f9d5-1c675504ab50} {Suffrage:Voter ID:server-11a0d01c-4a46-e06a-e67b-844791818f99 Address:11a0d01c-4a46-e06a-e67b-844791818f99}]" testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.775Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f8811921-2735-d284-f23e-ce90eafcb10a Address:f8811921-2735-d284-f23e-ce90eafcb10a} {Suffrage:Voter ID:server-f25432d1-1355-b829-f9d5-1c675504ab50 Address:f25432d1-1355-b829-f9d5-1c675504ab50} {Suffrage:Voter ID:server-11a0d01c-4a46-e06a-e67b-844791818f99 Address:11a0d01c-4a46-e06a-e67b-844791818f99}]" testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.775Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f8811921-2735-d284-f23e-ce90eafcb10a Address:f8811921-2735-d284-f23e-ce90eafcb10a} {Suffrage:Voter ID:server-f25432d1-1355-b829-f9d5-1c675504ab50 Address:f25432d1-1355-b829-f9d5-1c675504ab50} {Suffrage:Voter ID:server-11a0d01c-4a46-e06a-e67b-844791818f99 Address:11a0d01c-4a46-e06a-e67b-844791818f99}]" testLoggerAdapter verbose: server-11a0d01c-4a46-e06a-e67b-844791818f99: 2023-08-02T15:12:58.776Z [INFO] server-11a0d01c-4a46-e06a-e67b-844791818f99: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f8811921-2735-d284-f23e-ce90eafcb10a Address:f8811921-2735-d284-f23e-ce90eafcb10a} {Suffrage:Voter ID:server-f25432d1-1355-b829-f9d5-1c675504ab50 Address:f25432d1-1355-b829-f9d5-1c675504ab50} {Suffrage:Voter ID:server-11a0d01c-4a46-e06a-e67b-844791818f99 Address:11a0d01c-4a46-e06a-e67b-844791818f99}]" testing.go:149: server-11a0d01c-4a46-e06a-e67b-844791818f99: 2023-08-02T15:12:58.776Z [INFO] server-11a0d01c-4a46-e06a-e67b-844791818f99: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f8811921-2735-d284-f23e-ce90eafcb10a Address:f8811921-2735-d284-f23e-ce90eafcb10a} {Suffrage:Voter ID:server-f25432d1-1355-b829-f9d5-1c675504ab50 Address:f25432d1-1355-b829-f9d5-1c675504ab50} {Suffrage:Voter ID:server-11a0d01c-4a46-e06a-e67b-844791818f99 Address:11a0d01c-4a46-e06a-e67b-844791818f99}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:58.776Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:12:58.776Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-f8811921-2735-d284-f23e-ce90eafcb10a: 2023-08-02T15:12:58.776Z [INFO] server-f8811921-2735-d284-f23e-ce90eafcb10a: entering follower state: follower="Node at f8811921-2735-d284-f23e-ce90eafcb10a [Follower]" leader= testing.go:149: server-f8811921-2735-d284-f23e-ce90eafcb10a: 2023-08-02T15:12:58.776Z [INFO] server-f8811921-2735-d284-f23e-ce90eafcb10a: entering follower state: follower="Node at f8811921-2735-d284-f23e-ce90eafcb10a [Follower]" leader= testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.776Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: entering follower state: follower="Node at f25432d1-1355-b829-f9d5-1c675504ab50 [Follower]" leader= testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.776Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: entering follower state: follower="Node at f25432d1-1355-b829-f9d5-1c675504ab50 [Follower]" leader= testLoggerAdapter verbose: server-11a0d01c-4a46-e06a-e67b-844791818f99: 2023-08-02T15:12:58.776Z [INFO] server-11a0d01c-4a46-e06a-e67b-844791818f99: entering follower state: follower="Node at 11a0d01c-4a46-e06a-e67b-844791818f99 [Follower]" leader= testing.go:149: server-11a0d01c-4a46-e06a-e67b-844791818f99: 2023-08-02T15:12:58.776Z [INFO] server-11a0d01c-4a46-e06a-e67b-844791818f99: entering follower state: follower="Node at 11a0d01c-4a46-e06a-e67b-844791818f99 [Follower]" leader= testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.834Z [WARN] server-f25432d1-1355-b829-f9d5-1c675504ab50: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.834Z [WARN] server-f25432d1-1355-b829-f9d5-1c675504ab50: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.834Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: entering candidate state: node="Node at f25432d1-1355-b829-f9d5-1c675504ab50 [Candidate]" term=2 testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.834Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: entering candidate state: node="Node at f25432d1-1355-b829-f9d5-1c675504ab50 [Candidate]" term=2 testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.835Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: election won: tally=2 testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.835Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: election won: tally=2 testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.835Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: entering leader state: leader="Node at f25432d1-1355-b829-f9d5-1c675504ab50 [Leader]" testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.835Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: entering leader state: leader="Node at f25432d1-1355-b829-f9d5-1c675504ab50 [Leader]" testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.835Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: added peer, starting replication: peer=server-f8811921-2735-d284-f23e-ce90eafcb10a testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.835Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: added peer, starting replication: peer=server-f8811921-2735-d284-f23e-ce90eafcb10a testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.835Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: added peer, starting replication: peer=server-11a0d01c-4a46-e06a-e67b-844791818f99 testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.835Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: added peer, starting replication: peer=server-11a0d01c-4a46-e06a-e67b-844791818f99 testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.835Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: pipelining replication: peer="{Voter server-f8811921-2735-d284-f23e-ce90eafcb10a f8811921-2735-d284-f23e-ce90eafcb10a}" testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.835Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: pipelining replication: peer="{Voter server-f8811921-2735-d284-f23e-ce90eafcb10a f8811921-2735-d284-f23e-ce90eafcb10a}" testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.835Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: pipelining replication: peer="{Voter server-11a0d01c-4a46-e06a-e67b-844791818f99 11a0d01c-4a46-e06a-e67b-844791818f99}" testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.835Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: pipelining replication: peer="{Voter server-11a0d01c-4a46-e06a-e67b-844791818f99 11a0d01c-4a46-e06a-e67b-844791818f99}" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:58.940Z [INFO] stable state for Leader reached at 2023-08-02 15:12:58.835315302 +0000 UTC m=+10.931251660 (1 nodes), 59.064405ms from start of poll, 59.752137ms from cluster start. Timeout at 2023-08-02 15:12:58.940436599 +0000 UTC m=+11.036372988, 105.121328ms after stability testing.go:149: cluster: 2023-08-02T15:12:58.940Z [INFO] stable state for Leader reached at 2023-08-02 15:12:58.835315302 +0000 UTC m=+10.931251660 (1 nodes), 59.064405ms from start of poll, 59.752137ms from cluster start. Timeout at 2023-08-02 15:12:58.940436599 +0000 UTC m=+11.036372988, 105.121328ms after stability testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.941Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: starting snapshot up to: index=12 testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.941Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: starting snapshot up to: index=12 testing.go:153: 2023-08-02T15:12:58.941Z [INFO] creating new snapshot: path=/tmp/raft142344519/snapshots/2-12-1690989178941.tmp testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.941Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: snapshot complete up to: index=12 testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.941Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: snapshot complete up to: index=12 testing.go:153: 2023-08-02T15:12:58.942Z [INFO] creating new snapshot: path=/tmp/raft142344519/snapshots/2-23-1690989178942.tmp testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.942Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: copied to local snapshot: bytes=71 testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.942Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.942Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: restored user snapshot: index=1 testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.942Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: restored user snapshot: index=1 testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.942Z [ERROR] server-f25432d1-1355-b829-f9d5-1c675504ab50: failed to get log: index=23 error="log not found" testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.942Z [ERROR] server-f25432d1-1355-b829-f9d5-1c675504ab50: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.942Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: aborting pipeline replication: peer="{Voter server-11a0d01c-4a46-e06a-e67b-844791818f99 11a0d01c-4a46-e06a-e67b-844791818f99}" testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.942Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: aborting pipeline replication: peer="{Voter server-11a0d01c-4a46-e06a-e67b-844791818f99 11a0d01c-4a46-e06a-e67b-844791818f99}" testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.942Z [ERROR] server-f25432d1-1355-b829-f9d5-1c675504ab50: failed to get log: index=23 error="log not found" testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.942Z [ERROR] server-f25432d1-1355-b829-f9d5-1c675504ab50: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.942Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: aborting pipeline replication: peer="{Voter server-f8811921-2735-d284-f23e-ce90eafcb10a f8811921-2735-d284-f23e-ce90eafcb10a}" testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.942Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: aborting pipeline replication: peer="{Voter server-f8811921-2735-d284-f23e-ce90eafcb10a f8811921-2735-d284-f23e-ce90eafcb10a}" testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.942Z [ERROR] server-f25432d1-1355-b829-f9d5-1c675504ab50: failed to get log: index=23 error="log not found" testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.942Z [ERROR] server-f25432d1-1355-b829-f9d5-1c675504ab50: failed to get log: index=23 error="log not found" testing.go:153: 2023-08-02T15:12:58.943Z [INFO] creating new snapshot: path=/tmp/raft013424013/snapshots/2-23-1690989178943.tmp testLoggerAdapter verbose: server-f8811921-2735-d284-f23e-ce90eafcb10a: 2023-08-02T15:12:58.943Z [INFO] server-f8811921-2735-d284-f23e-ce90eafcb10a: copied to local snapshot: bytes=71 testing.go:149: server-f8811921-2735-d284-f23e-ce90eafcb10a: 2023-08-02T15:12:58.943Z [INFO] server-f8811921-2735-d284-f23e-ce90eafcb10a: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-f8811921-2735-d284-f23e-ce90eafcb10a: 2023-08-02T15:12:58.944Z [INFO] server-f8811921-2735-d284-f23e-ce90eafcb10a: Installed remote snapshot testing.go:149: server-f8811921-2735-d284-f23e-ce90eafcb10a: 2023-08-02T15:12:58.944Z [INFO] server-f8811921-2735-d284-f23e-ce90eafcb10a: Installed remote snapshot testLoggerAdapter verbose: server-f8811921-2735-d284-f23e-ce90eafcb10a: 2023-08-02T15:12:58.944Z [WARN] server-f8811921-2735-d284-f23e-ce90eafcb10a: skipping application of old log: index=22 testing.go:149: server-f8811921-2735-d284-f23e-ce90eafcb10a: 2023-08-02T15:12:58.944Z [WARN] server-f8811921-2735-d284-f23e-ce90eafcb10a: skipping application of old log: index=22 testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.944Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: pipelining replication: peer="{Voter server-f8811921-2735-d284-f23e-ce90eafcb10a f8811921-2735-d284-f23e-ce90eafcb10a}" testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.944Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: pipelining replication: peer="{Voter server-f8811921-2735-d284-f23e-ce90eafcb10a f8811921-2735-d284-f23e-ce90eafcb10a}" testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.952Z [ERROR] server-f25432d1-1355-b829-f9d5-1c675504ab50: failed to get log: index=23 error="log not found" testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.952Z [ERROR] server-f25432d1-1355-b829-f9d5-1c675504ab50: failed to get log: index=23 error="log not found" testing.go:153: 2023-08-02T15:12:58.952Z [INFO] creating new snapshot: path=/tmp/raft242716433/snapshots/2-23-1690989178952.tmp testLoggerAdapter verbose: server-11a0d01c-4a46-e06a-e67b-844791818f99: 2023-08-02T15:12:58.953Z [INFO] server-11a0d01c-4a46-e06a-e67b-844791818f99: copied to local snapshot: bytes=71 testing.go:149: server-11a0d01c-4a46-e06a-e67b-844791818f99: 2023-08-02T15:12:58.953Z [INFO] server-11a0d01c-4a46-e06a-e67b-844791818f99: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-11a0d01c-4a46-e06a-e67b-844791818f99: 2023-08-02T15:12:58.953Z [INFO] server-11a0d01c-4a46-e06a-e67b-844791818f99: Installed remote snapshot testing.go:149: server-11a0d01c-4a46-e06a-e67b-844791818f99: 2023-08-02T15:12:58.953Z [INFO] server-11a0d01c-4a46-e06a-e67b-844791818f99: Installed remote snapshot testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.953Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: pipelining replication: peer="{Voter server-11a0d01c-4a46-e06a-e67b-844791818f99 11a0d01c-4a46-e06a-e67b-844791818f99}" testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.953Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: pipelining replication: peer="{Voter server-11a0d01c-4a46-e06a-e67b-844791818f99 11a0d01c-4a46-e06a-e67b-844791818f99}" testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.965Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: aborting pipeline replication: peer="{Voter server-11a0d01c-4a46-e06a-e67b-844791818f99 11a0d01c-4a46-e06a-e67b-844791818f99}" testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.965Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: aborting pipeline replication: peer="{Voter server-11a0d01c-4a46-e06a-e67b-844791818f99 11a0d01c-4a46-e06a-e67b-844791818f99}" testLoggerAdapter verbose: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.965Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: aborting pipeline replication: peer="{Voter server-f8811921-2735-d284-f23e-ce90eafcb10a f8811921-2735-d284-f23e-ce90eafcb10a}" testing.go:149: server-f25432d1-1355-b829-f9d5-1c675504ab50: 2023-08-02T15:12:58.965Z [INFO] server-f25432d1-1355-b829-f9d5-1c675504ab50: aborting pipeline replication: peer="{Voter server-f8811921-2735-d284-f23e-ce90eafcb10a f8811921-2735-d284-f23e-ce90eafcb10a}" testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:58.966Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b137adde-4720-880c-9538-d85b60b2a59b Address:b137adde-4720-880c-9538-d85b60b2a59b} {Suffrage:Voter ID:server-cff6bc78-1346-3f29-6e01-90cee53fe183 Address:cff6bc78-1346-3f29-6e01-90cee53fe183} {Suffrage:Voter ID:server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d Address:5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d}]" testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:58.966Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b137adde-4720-880c-9538-d85b60b2a59b Address:b137adde-4720-880c-9538-d85b60b2a59b} {Suffrage:Voter ID:server-cff6bc78-1346-3f29-6e01-90cee53fe183 Address:cff6bc78-1346-3f29-6e01-90cee53fe183} {Suffrage:Voter ID:server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d Address:5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d}]" testLoggerAdapter verbose: server-cff6bc78-1346-3f29-6e01-90cee53fe183: 2023-08-02T15:12:58.966Z [INFO] server-cff6bc78-1346-3f29-6e01-90cee53fe183: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b137adde-4720-880c-9538-d85b60b2a59b Address:b137adde-4720-880c-9538-d85b60b2a59b} {Suffrage:Voter ID:server-cff6bc78-1346-3f29-6e01-90cee53fe183 Address:cff6bc78-1346-3f29-6e01-90cee53fe183} {Suffrage:Voter ID:server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d Address:5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d}]" testing.go:149: server-cff6bc78-1346-3f29-6e01-90cee53fe183: 2023-08-02T15:12:58.966Z [INFO] server-cff6bc78-1346-3f29-6e01-90cee53fe183: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b137adde-4720-880c-9538-d85b60b2a59b Address:b137adde-4720-880c-9538-d85b60b2a59b} {Suffrage:Voter ID:server-cff6bc78-1346-3f29-6e01-90cee53fe183 Address:cff6bc78-1346-3f29-6e01-90cee53fe183} {Suffrage:Voter ID:server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d Address:5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d}]" testLoggerAdapter verbose: server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d: 2023-08-02T15:12:58.966Z [INFO] server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b137adde-4720-880c-9538-d85b60b2a59b Address:b137adde-4720-880c-9538-d85b60b2a59b} {Suffrage:Voter ID:server-cff6bc78-1346-3f29-6e01-90cee53fe183 Address:cff6bc78-1346-3f29-6e01-90cee53fe183} {Suffrage:Voter ID:server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d Address:5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d}]" testing.go:149: server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d: 2023-08-02T15:12:58.966Z [INFO] server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b137adde-4720-880c-9538-d85b60b2a59b Address:b137adde-4720-880c-9538-d85b60b2a59b} {Suffrage:Voter ID:server-cff6bc78-1346-3f29-6e01-90cee53fe183 Address:cff6bc78-1346-3f29-6e01-90cee53fe183} {Suffrage:Voter ID:server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d Address:5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:58.967Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:12:58.967Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:58.967Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: entering follower state: follower="Node at b137adde-4720-880c-9538-d85b60b2a59b [Follower]" leader= testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:58.967Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: entering follower state: follower="Node at b137adde-4720-880c-9538-d85b60b2a59b [Follower]" leader= testLoggerAdapter verbose: server-cff6bc78-1346-3f29-6e01-90cee53fe183: 2023-08-02T15:12:58.967Z [INFO] server-cff6bc78-1346-3f29-6e01-90cee53fe183: entering follower state: follower="Node at cff6bc78-1346-3f29-6e01-90cee53fe183 [Follower]" leader= testing.go:149: server-cff6bc78-1346-3f29-6e01-90cee53fe183: 2023-08-02T15:12:58.967Z [INFO] server-cff6bc78-1346-3f29-6e01-90cee53fe183: entering follower state: follower="Node at cff6bc78-1346-3f29-6e01-90cee53fe183 [Follower]" leader= testLoggerAdapter verbose: server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d: 2023-08-02T15:12:58.967Z [INFO] server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d: entering follower state: follower="Node at 5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d [Follower]" leader= testing.go:149: server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d: 2023-08-02T15:12:58.967Z [INFO] server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d: entering follower state: follower="Node at 5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d [Follower]" leader= testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.021Z [WARN] server-b137adde-4720-880c-9538-d85b60b2a59b: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.021Z [WARN] server-b137adde-4720-880c-9538-d85b60b2a59b: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.021Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: entering candidate state: node="Node at b137adde-4720-880c-9538-d85b60b2a59b [Candidate]" term=2 testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.021Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: entering candidate state: node="Node at b137adde-4720-880c-9538-d85b60b2a59b [Candidate]" term=2 testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.021Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: election won: tally=2 testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.021Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: election won: tally=2 testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.022Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: entering leader state: leader="Node at b137adde-4720-880c-9538-d85b60b2a59b [Leader]" testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.022Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: entering leader state: leader="Node at b137adde-4720-880c-9538-d85b60b2a59b [Leader]" testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.022Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: added peer, starting replication: peer=server-cff6bc78-1346-3f29-6e01-90cee53fe183 testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.022Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: added peer, starting replication: peer=server-cff6bc78-1346-3f29-6e01-90cee53fe183 testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.022Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: added peer, starting replication: peer=server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.022Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: added peer, starting replication: peer=server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.022Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: pipelining replication: peer="{Voter server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d 5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d}" testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.022Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: pipelining replication: peer="{Voter server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d 5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d}" testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.022Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: pipelining replication: peer="{Voter server-cff6bc78-1346-3f29-6e01-90cee53fe183 cff6bc78-1346-3f29-6e01-90cee53fe183}" testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.022Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: pipelining replication: peer="{Voter server-cff6bc78-1346-3f29-6e01-90cee53fe183 cff6bc78-1346-3f29-6e01-90cee53fe183}" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:59.127Z [INFO] stable state for Leader reached at 2023-08-02 15:12:59.022308266 +0000 UTC m=+11.118244598 (1 nodes), 55.256349ms from start of poll, 55.776229ms from cluster start. Timeout at 2023-08-02 15:12:59.127443147 +0000 UTC m=+11.223379516, 105.134918ms after stability testing.go:149: cluster: 2023-08-02T15:12:59.127Z [INFO] stable state for Leader reached at 2023-08-02 15:12:59.022308266 +0000 UTC m=+11.118244598 (1 nodes), 55.256349ms from start of poll, 55.776229ms from cluster start. Timeout at 2023-08-02 15:12:59.127443147 +0000 UTC m=+11.223379516, 105.134918ms after stability testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.128Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: starting snapshot up to: index=12 testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.128Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: starting snapshot up to: index=12 testing.go:153: 2023-08-02T15:12:59.128Z [INFO] creating new snapshot: path=/tmp/raft464897387/snapshots/2-12-1690989179128.tmp testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.128Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: snapshot complete up to: index=12 testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.128Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: snapshot complete up to: index=12 testing.go:153: 2023-08-02T15:12:59.129Z [INFO] creating new snapshot: path=/tmp/raft464897387/snapshots/2-23-1690989179129.tmp testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.129Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: copied to local snapshot: bytes=71 testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.129Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.129Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: restored user snapshot: index=1 testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.129Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: restored user snapshot: index=1 testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.129Z [ERROR] server-b137adde-4720-880c-9538-d85b60b2a59b: failed to get log: index=23 error="log not found" testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.129Z [ERROR] server-b137adde-4720-880c-9538-d85b60b2a59b: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.129Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: aborting pipeline replication: peer="{Voter server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d 5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d}" testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.129Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: aborting pipeline replication: peer="{Voter server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d 5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d}" testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.129Z [ERROR] server-b137adde-4720-880c-9538-d85b60b2a59b: failed to get log: index=23 error="log not found" testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.129Z [ERROR] server-b137adde-4720-880c-9538-d85b60b2a59b: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.129Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: aborting pipeline replication: peer="{Voter server-cff6bc78-1346-3f29-6e01-90cee53fe183 cff6bc78-1346-3f29-6e01-90cee53fe183}" testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.129Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: aborting pipeline replication: peer="{Voter server-cff6bc78-1346-3f29-6e01-90cee53fe183 cff6bc78-1346-3f29-6e01-90cee53fe183}" testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.137Z [ERROR] server-b137adde-4720-880c-9538-d85b60b2a59b: failed to get log: index=23 error="log not found" testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.137Z [ERROR] server-b137adde-4720-880c-9538-d85b60b2a59b: failed to get log: index=23 error="log not found" testing.go:153: 2023-08-02T15:12:59.138Z [INFO] creating new snapshot: path=/tmp/raft980704469/snapshots/2-23-1690989179138.tmp testLoggerAdapter verbose: server-cff6bc78-1346-3f29-6e01-90cee53fe183: 2023-08-02T15:12:59.138Z [INFO] server-cff6bc78-1346-3f29-6e01-90cee53fe183: copied to local snapshot: bytes=71 testing.go:149: server-cff6bc78-1346-3f29-6e01-90cee53fe183: 2023-08-02T15:12:59.138Z [INFO] server-cff6bc78-1346-3f29-6e01-90cee53fe183: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-cff6bc78-1346-3f29-6e01-90cee53fe183: 2023-08-02T15:12:59.139Z [INFO] server-cff6bc78-1346-3f29-6e01-90cee53fe183: Installed remote snapshot testing.go:149: server-cff6bc78-1346-3f29-6e01-90cee53fe183: 2023-08-02T15:12:59.139Z [INFO] server-cff6bc78-1346-3f29-6e01-90cee53fe183: Installed remote snapshot testLoggerAdapter verbose: server-cff6bc78-1346-3f29-6e01-90cee53fe183: 2023-08-02T15:12:59.139Z [WARN] server-cff6bc78-1346-3f29-6e01-90cee53fe183: skipping application of old log: index=22 testing.go:149: server-cff6bc78-1346-3f29-6e01-90cee53fe183: 2023-08-02T15:12:59.139Z [WARN] server-cff6bc78-1346-3f29-6e01-90cee53fe183: skipping application of old log: index=22 testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.139Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: pipelining replication: peer="{Voter server-cff6bc78-1346-3f29-6e01-90cee53fe183 cff6bc78-1346-3f29-6e01-90cee53fe183}" testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.139Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: pipelining replication: peer="{Voter server-cff6bc78-1346-3f29-6e01-90cee53fe183 cff6bc78-1346-3f29-6e01-90cee53fe183}" testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.139Z [ERROR] server-b137adde-4720-880c-9538-d85b60b2a59b: failed to get log: index=23 error="log not found" testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.139Z [ERROR] server-b137adde-4720-880c-9538-d85b60b2a59b: failed to get log: index=23 error="log not found" testing.go:153: 2023-08-02T15:12:59.142Z [INFO] creating new snapshot: path=/tmp/raft956950479/snapshots/2-23-1690989179142.tmp testLoggerAdapter verbose: server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d: 2023-08-02T15:12:59.143Z [INFO] server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d: copied to local snapshot: bytes=71 testing.go:149: server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d: 2023-08-02T15:12:59.143Z [INFO] server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d: 2023-08-02T15:12:59.143Z [INFO] server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d: Installed remote snapshot testing.go:149: server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d: 2023-08-02T15:12:59.143Z [INFO] server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d: Installed remote snapshot testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.143Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: pipelining replication: peer="{Voter server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d 5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d}" testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.143Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: pipelining replication: peer="{Voter server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d 5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d}" testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.155Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: aborting pipeline replication: peer="{Voter server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d 5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d}" testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.155Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: aborting pipeline replication: peer="{Voter server-5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d 5c94dc9b-76c5-64c7-cd9d-0a0eb1be413d}" testLoggerAdapter verbose: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.155Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: aborting pipeline replication: peer="{Voter server-cff6bc78-1346-3f29-6e01-90cee53fe183 cff6bc78-1346-3f29-6e01-90cee53fe183}" testing.go:149: server-b137adde-4720-880c-9538-d85b60b2a59b: 2023-08-02T15:12:59.155Z [INFO] server-b137adde-4720-880c-9538-d85b60b2a59b: aborting pipeline replication: peer="{Voter server-cff6bc78-1346-3f29-6e01-90cee53fe183 cff6bc78-1346-3f29-6e01-90cee53fe183}" testLoggerAdapter verbose: server-97b6deac-af23-d668-a303-38e9fd8a9ad7: 2023-08-02T15:12:59.156Z [INFO] server-97b6deac-af23-d668-a303-38e9fd8a9ad7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-97b6deac-af23-d668-a303-38e9fd8a9ad7 Address:97b6deac-af23-d668-a303-38e9fd8a9ad7} {Suffrage:Voter ID:server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9 Address:52f0d966-03d3-98ea-7f23-a0c6d2a743a9} {Suffrage:Voter ID:server-2d4e71af-f3b4-de58-aecf-42726d85262c Address:2d4e71af-f3b4-de58-aecf-42726d85262c}]" testing.go:149: server-97b6deac-af23-d668-a303-38e9fd8a9ad7: 2023-08-02T15:12:59.156Z [INFO] server-97b6deac-af23-d668-a303-38e9fd8a9ad7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-97b6deac-af23-d668-a303-38e9fd8a9ad7 Address:97b6deac-af23-d668-a303-38e9fd8a9ad7} {Suffrage:Voter ID:server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9 Address:52f0d966-03d3-98ea-7f23-a0c6d2a743a9} {Suffrage:Voter ID:server-2d4e71af-f3b4-de58-aecf-42726d85262c Address:2d4e71af-f3b4-de58-aecf-42726d85262c}]" testLoggerAdapter verbose: server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: 2023-08-02T15:12:59.156Z [INFO] server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-97b6deac-af23-d668-a303-38e9fd8a9ad7 Address:97b6deac-af23-d668-a303-38e9fd8a9ad7} {Suffrage:Voter ID:server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9 Address:52f0d966-03d3-98ea-7f23-a0c6d2a743a9} {Suffrage:Voter ID:server-2d4e71af-f3b4-de58-aecf-42726d85262c Address:2d4e71af-f3b4-de58-aecf-42726d85262c}]" testing.go:149: server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: 2023-08-02T15:12:59.156Z [INFO] server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-97b6deac-af23-d668-a303-38e9fd8a9ad7 Address:97b6deac-af23-d668-a303-38e9fd8a9ad7} {Suffrage:Voter ID:server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9 Address:52f0d966-03d3-98ea-7f23-a0c6d2a743a9} {Suffrage:Voter ID:server-2d4e71af-f3b4-de58-aecf-42726d85262c Address:2d4e71af-f3b4-de58-aecf-42726d85262c}]" testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.156Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-97b6deac-af23-d668-a303-38e9fd8a9ad7 Address:97b6deac-af23-d668-a303-38e9fd8a9ad7} {Suffrage:Voter ID:server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9 Address:52f0d966-03d3-98ea-7f23-a0c6d2a743a9} {Suffrage:Voter ID:server-2d4e71af-f3b4-de58-aecf-42726d85262c Address:2d4e71af-f3b4-de58-aecf-42726d85262c}]" testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.156Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-97b6deac-af23-d668-a303-38e9fd8a9ad7 Address:97b6deac-af23-d668-a303-38e9fd8a9ad7} {Suffrage:Voter ID:server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9 Address:52f0d966-03d3-98ea-7f23-a0c6d2a743a9} {Suffrage:Voter ID:server-2d4e71af-f3b4-de58-aecf-42726d85262c Address:2d4e71af-f3b4-de58-aecf-42726d85262c}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:59.157Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:12:59.157Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-97b6deac-af23-d668-a303-38e9fd8a9ad7: 2023-08-02T15:12:59.157Z [INFO] server-97b6deac-af23-d668-a303-38e9fd8a9ad7: entering follower state: follower="Node at 97b6deac-af23-d668-a303-38e9fd8a9ad7 [Follower]" leader= testing.go:149: server-97b6deac-af23-d668-a303-38e9fd8a9ad7: 2023-08-02T15:12:59.157Z [INFO] server-97b6deac-af23-d668-a303-38e9fd8a9ad7: entering follower state: follower="Node at 97b6deac-af23-d668-a303-38e9fd8a9ad7 [Follower]" leader= testLoggerAdapter verbose: server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: 2023-08-02T15:12:59.157Z [INFO] server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: entering follower state: follower="Node at 52f0d966-03d3-98ea-7f23-a0c6d2a743a9 [Follower]" leader= testing.go:149: server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: 2023-08-02T15:12:59.157Z [INFO] server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: entering follower state: follower="Node at 52f0d966-03d3-98ea-7f23-a0c6d2a743a9 [Follower]" leader= testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.157Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: entering follower state: follower="Node at 2d4e71af-f3b4-de58-aecf-42726d85262c [Follower]" leader= testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.157Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: entering follower state: follower="Node at 2d4e71af-f3b4-de58-aecf-42726d85262c [Follower]" leader= testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.210Z [WARN] server-2d4e71af-f3b4-de58-aecf-42726d85262c: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.210Z [WARN] server-2d4e71af-f3b4-de58-aecf-42726d85262c: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.211Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: entering candidate state: node="Node at 2d4e71af-f3b4-de58-aecf-42726d85262c [Candidate]" term=2 testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.211Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: entering candidate state: node="Node at 2d4e71af-f3b4-de58-aecf-42726d85262c [Candidate]" term=2 testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.211Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: election won: tally=2 testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.211Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: election won: tally=2 testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.211Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: entering leader state: leader="Node at 2d4e71af-f3b4-de58-aecf-42726d85262c [Leader]" testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.211Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: entering leader state: leader="Node at 2d4e71af-f3b4-de58-aecf-42726d85262c [Leader]" testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.211Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: added peer, starting replication: peer=server-97b6deac-af23-d668-a303-38e9fd8a9ad7 testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.211Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: added peer, starting replication: peer=server-97b6deac-af23-d668-a303-38e9fd8a9ad7 testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.211Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: added peer, starting replication: peer=server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9 testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.211Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: added peer, starting replication: peer=server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9 testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.211Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: pipelining replication: peer="{Voter server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9 52f0d966-03d3-98ea-7f23-a0c6d2a743a9}" testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.211Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: pipelining replication: peer="{Voter server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9 52f0d966-03d3-98ea-7f23-a0c6d2a743a9}" testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.211Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: pipelining replication: peer="{Voter server-97b6deac-af23-d668-a303-38e9fd8a9ad7 97b6deac-af23-d668-a303-38e9fd8a9ad7}" testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.211Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: pipelining replication: peer="{Voter server-97b6deac-af23-d668-a303-38e9fd8a9ad7 97b6deac-af23-d668-a303-38e9fd8a9ad7}" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:59.316Z [INFO] stable state for Leader reached at 2023-08-02 15:12:59.211589405 +0000 UTC m=+11.307525733 (1 nodes), 54.532045ms from start of poll, 55.12247ms from cluster start. Timeout at 2023-08-02 15:12:59.316686956 +0000 UTC m=+11.412623283, 105.09755ms after stability testing.go:149: cluster: 2023-08-02T15:12:59.316Z [INFO] stable state for Leader reached at 2023-08-02 15:12:59.211589405 +0000 UTC m=+11.307525733 (1 nodes), 54.532045ms from start of poll, 55.12247ms from cluster start. Timeout at 2023-08-02 15:12:59.316686956 +0000 UTC m=+11.412623283, 105.09755ms after stability testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.317Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: starting snapshot up to: index=12 testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.317Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: starting snapshot up to: index=12 testing.go:153: 2023-08-02T15:12:59.317Z [INFO] creating new snapshot: path=/tmp/raft515795741/snapshots/2-12-1690989179317.tmp testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.317Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: snapshot complete up to: index=12 testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.317Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: snapshot complete up to: index=12 testing.go:153: 2023-08-02T15:12:59.318Z [INFO] creating new snapshot: path=/tmp/raft515795741/snapshots/2-23-1690989179318.tmp testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.318Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: copied to local snapshot: bytes=71 testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.318Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.318Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: restored user snapshot: index=1 testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.318Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: restored user snapshot: index=1 testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.318Z [ERROR] server-2d4e71af-f3b4-de58-aecf-42726d85262c: failed to get log: index=23 error="log not found" testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.318Z [ERROR] server-2d4e71af-f3b4-de58-aecf-42726d85262c: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.319Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: aborting pipeline replication: peer="{Voter server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9 52f0d966-03d3-98ea-7f23-a0c6d2a743a9}" testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.319Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: aborting pipeline replication: peer="{Voter server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9 52f0d966-03d3-98ea-7f23-a0c6d2a743a9}" testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.318Z [ERROR] server-2d4e71af-f3b4-de58-aecf-42726d85262c: failed to get log: index=23 error="log not found" testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.318Z [ERROR] server-2d4e71af-f3b4-de58-aecf-42726d85262c: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.319Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: aborting pipeline replication: peer="{Voter server-97b6deac-af23-d668-a303-38e9fd8a9ad7 97b6deac-af23-d668-a303-38e9fd8a9ad7}" testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.319Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: aborting pipeline replication: peer="{Voter server-97b6deac-af23-d668-a303-38e9fd8a9ad7 97b6deac-af23-d668-a303-38e9fd8a9ad7}" testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.326Z [ERROR] server-2d4e71af-f3b4-de58-aecf-42726d85262c: failed to get log: index=23 error="log not found" testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.326Z [ERROR] server-2d4e71af-f3b4-de58-aecf-42726d85262c: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.326Z [ERROR] server-2d4e71af-f3b4-de58-aecf-42726d85262c: failed to get log: index=23 error="log not found" testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.326Z [ERROR] server-2d4e71af-f3b4-de58-aecf-42726d85262c: failed to get log: index=23 error="log not found" testing.go:153: 2023-08-02T15:12:59.326Z [INFO] creating new snapshot: path=/tmp/raft130860761/snapshots/2-23-1690989179326.tmp testing.go:153: 2023-08-02T15:12:59.326Z [INFO] creating new snapshot: path=/tmp/raft861224563/snapshots/2-23-1690989179326.tmp testLoggerAdapter verbose: server-97b6deac-af23-d668-a303-38e9fd8a9ad7: 2023-08-02T15:12:59.327Z [INFO] server-97b6deac-af23-d668-a303-38e9fd8a9ad7: copied to local snapshot: bytes=71 testing.go:149: server-97b6deac-af23-d668-a303-38e9fd8a9ad7: 2023-08-02T15:12:59.327Z [INFO] server-97b6deac-af23-d668-a303-38e9fd8a9ad7: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: 2023-08-02T15:12:59.327Z [INFO] server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: copied to local snapshot: bytes=71 testing.go:149: server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: 2023-08-02T15:12:59.327Z [INFO] server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-97b6deac-af23-d668-a303-38e9fd8a9ad7: 2023-08-02T15:12:59.327Z [INFO] server-97b6deac-af23-d668-a303-38e9fd8a9ad7: Installed remote snapshot testing.go:149: server-97b6deac-af23-d668-a303-38e9fd8a9ad7: 2023-08-02T15:12:59.327Z [INFO] server-97b6deac-af23-d668-a303-38e9fd8a9ad7: Installed remote snapshot testLoggerAdapter verbose: server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: 2023-08-02T15:12:59.327Z [INFO] server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: Installed remote snapshot testLoggerAdapter verbose: server-97b6deac-af23-d668-a303-38e9fd8a9ad7: 2023-08-02T15:12:59.327Z [WARN] server-97b6deac-af23-d668-a303-38e9fd8a9ad7: skipping application of old log: index=22 testing.go:149: server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: 2023-08-02T15:12:59.327Z [INFO] server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: Installed remote snapshot testing.go:149: server-97b6deac-af23-d668-a303-38e9fd8a9ad7: 2023-08-02T15:12:59.327Z [WARN] server-97b6deac-af23-d668-a303-38e9fd8a9ad7: skipping application of old log: index=22 testLoggerAdapter verbose: server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: 2023-08-02T15:12:59.327Z [WARN] server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: skipping application of old log: index=23 testing.go:149: server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: 2023-08-02T15:12:59.327Z [WARN] server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9: skipping application of old log: index=23 testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.327Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: pipelining replication: peer="{Voter server-97b6deac-af23-d668-a303-38e9fd8a9ad7 97b6deac-af23-d668-a303-38e9fd8a9ad7}" testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.327Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: pipelining replication: peer="{Voter server-97b6deac-af23-d668-a303-38e9fd8a9ad7 97b6deac-af23-d668-a303-38e9fd8a9ad7}" testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.327Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: pipelining replication: peer="{Voter server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9 52f0d966-03d3-98ea-7f23-a0c6d2a743a9}" testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.327Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: pipelining replication: peer="{Voter server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9 52f0d966-03d3-98ea-7f23-a0c6d2a743a9}" testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.338Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: aborting pipeline replication: peer="{Voter server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9 52f0d966-03d3-98ea-7f23-a0c6d2a743a9}" testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.338Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: aborting pipeline replication: peer="{Voter server-52f0d966-03d3-98ea-7f23-a0c6d2a743a9 52f0d966-03d3-98ea-7f23-a0c6d2a743a9}" testLoggerAdapter verbose: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.338Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: aborting pipeline replication: peer="{Voter server-97b6deac-af23-d668-a303-38e9fd8a9ad7 97b6deac-af23-d668-a303-38e9fd8a9ad7}" testing.go:149: server-2d4e71af-f3b4-de58-aecf-42726d85262c: 2023-08-02T15:12:59.338Z [INFO] server-2d4e71af-f3b4-de58-aecf-42726d85262c: aborting pipeline replication: peer="{Voter server-97b6deac-af23-d668-a303-38e9fd8a9ad7 97b6deac-af23-d668-a303-38e9fd8a9ad7}" testLoggerAdapter verbose: server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2: 2023-08-02T15:12:59.339Z [INFO] server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2 Address:abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2} {Suffrage:Voter ID:server-009a1e34-c01e-0280-4511-05dabf8f55db Address:009a1e34-c01e-0280-4511-05dabf8f55db} {Suffrage:Voter ID:server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9 Address:5afb8d1a-261a-64dc-db92-bc1cbc2525b9}]" testing.go:149: server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2: 2023-08-02T15:12:59.339Z [INFO] server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2 Address:abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2} {Suffrage:Voter ID:server-009a1e34-c01e-0280-4511-05dabf8f55db Address:009a1e34-c01e-0280-4511-05dabf8f55db} {Suffrage:Voter ID:server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9 Address:5afb8d1a-261a-64dc-db92-bc1cbc2525b9}]" testLoggerAdapter verbose: server-009a1e34-c01e-0280-4511-05dabf8f55db: 2023-08-02T15:12:59.339Z [INFO] server-009a1e34-c01e-0280-4511-05dabf8f55db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2 Address:abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2} {Suffrage:Voter ID:server-009a1e34-c01e-0280-4511-05dabf8f55db Address:009a1e34-c01e-0280-4511-05dabf8f55db} {Suffrage:Voter ID:server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9 Address:5afb8d1a-261a-64dc-db92-bc1cbc2525b9}]" testing.go:149: server-009a1e34-c01e-0280-4511-05dabf8f55db: 2023-08-02T15:12:59.339Z [INFO] server-009a1e34-c01e-0280-4511-05dabf8f55db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2 Address:abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2} {Suffrage:Voter ID:server-009a1e34-c01e-0280-4511-05dabf8f55db Address:009a1e34-c01e-0280-4511-05dabf8f55db} {Suffrage:Voter ID:server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9 Address:5afb8d1a-261a-64dc-db92-bc1cbc2525b9}]" testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.339Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2 Address:abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2} {Suffrage:Voter ID:server-009a1e34-c01e-0280-4511-05dabf8f55db Address:009a1e34-c01e-0280-4511-05dabf8f55db} {Suffrage:Voter ID:server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9 Address:5afb8d1a-261a-64dc-db92-bc1cbc2525b9}]" testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.339Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2 Address:abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2} {Suffrage:Voter ID:server-009a1e34-c01e-0280-4511-05dabf8f55db Address:009a1e34-c01e-0280-4511-05dabf8f55db} {Suffrage:Voter ID:server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9 Address:5afb8d1a-261a-64dc-db92-bc1cbc2525b9}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:59.339Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:12:59.339Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2: 2023-08-02T15:12:59.339Z [INFO] server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2: entering follower state: follower="Node at abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2 [Follower]" leader= testing.go:149: server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2: 2023-08-02T15:12:59.339Z [INFO] server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2: entering follower state: follower="Node at abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2 [Follower]" leader= testLoggerAdapter verbose: server-009a1e34-c01e-0280-4511-05dabf8f55db: 2023-08-02T15:12:59.340Z [INFO] server-009a1e34-c01e-0280-4511-05dabf8f55db: entering follower state: follower="Node at 009a1e34-c01e-0280-4511-05dabf8f55db [Follower]" leader= testing.go:149: server-009a1e34-c01e-0280-4511-05dabf8f55db: 2023-08-02T15:12:59.340Z [INFO] server-009a1e34-c01e-0280-4511-05dabf8f55db: entering follower state: follower="Node at 009a1e34-c01e-0280-4511-05dabf8f55db [Follower]" leader= testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.340Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: entering follower state: follower="Node at 5afb8d1a-261a-64dc-db92-bc1cbc2525b9 [Follower]" leader= testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.340Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: entering follower state: follower="Node at 5afb8d1a-261a-64dc-db92-bc1cbc2525b9 [Follower]" leader= testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.406Z [WARN] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.406Z [WARN] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.407Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: entering candidate state: node="Node at 5afb8d1a-261a-64dc-db92-bc1cbc2525b9 [Candidate]" term=2 testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.407Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: entering candidate state: node="Node at 5afb8d1a-261a-64dc-db92-bc1cbc2525b9 [Candidate]" term=2 testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.407Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: election won: tally=2 testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.407Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: election won: tally=2 testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.407Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: entering leader state: leader="Node at 5afb8d1a-261a-64dc-db92-bc1cbc2525b9 [Leader]" testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.407Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: entering leader state: leader="Node at 5afb8d1a-261a-64dc-db92-bc1cbc2525b9 [Leader]" testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.407Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: added peer, starting replication: peer=server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2 testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.407Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: added peer, starting replication: peer=server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2 testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.407Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: added peer, starting replication: peer=server-009a1e34-c01e-0280-4511-05dabf8f55db testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.407Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: added peer, starting replication: peer=server-009a1e34-c01e-0280-4511-05dabf8f55db testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.408Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: pipelining replication: peer="{Voter server-009a1e34-c01e-0280-4511-05dabf8f55db 009a1e34-c01e-0280-4511-05dabf8f55db}" testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.408Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: pipelining replication: peer="{Voter server-009a1e34-c01e-0280-4511-05dabf8f55db 009a1e34-c01e-0280-4511-05dabf8f55db}" testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.408Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: pipelining replication: peer="{Voter server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2 abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2}" testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.408Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: pipelining replication: peer="{Voter server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2 abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2}" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:59.513Z [INFO] stable state for Leader reached at 2023-08-02 15:12:59.408227435 +0000 UTC m=+11.504163780 (1 nodes), 68.284665ms from start of poll, 68.787649ms from cluster start. Timeout at 2023-08-02 15:12:59.513295383 +0000 UTC m=+11.609231960, 105.06818ms after stability testing.go:149: cluster: 2023-08-02T15:12:59.513Z [INFO] stable state for Leader reached at 2023-08-02 15:12:59.408227435 +0000 UTC m=+11.504163780 (1 nodes), 68.284665ms from start of poll, 68.787649ms from cluster start. Timeout at 2023-08-02 15:12:59.513295383 +0000 UTC m=+11.609231960, 105.06818ms after stability testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.514Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: starting snapshot up to: index=12 testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.514Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: starting snapshot up to: index=12 testing.go:153: 2023-08-02T15:12:59.514Z [INFO] creating new snapshot: path=/tmp/raft565098107/snapshots/2-12-1690989179514.tmp testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.514Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: snapshot complete up to: index=12 testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.514Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: snapshot complete up to: index=12 testing.go:153: 2023-08-02T15:12:59.515Z [INFO] creating new snapshot: path=/tmp/raft565098107/snapshots/2-113-1690989179515.tmp testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.516Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: copied to local snapshot: bytes=71 testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.516Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.516Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: restored user snapshot: index=1 testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.516Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: restored user snapshot: index=1 testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.516Z [ERROR] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: failed to get log: index=23 error="log not found" testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.516Z [ERROR] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.516Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: aborting pipeline replication: peer="{Voter server-009a1e34-c01e-0280-4511-05dabf8f55db 009a1e34-c01e-0280-4511-05dabf8f55db}" testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.516Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: aborting pipeline replication: peer="{Voter server-009a1e34-c01e-0280-4511-05dabf8f55db 009a1e34-c01e-0280-4511-05dabf8f55db}" testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.516Z [ERROR] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: failed to get log: index=23 error="log not found" testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.516Z [ERROR] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.516Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: aborting pipeline replication: peer="{Voter server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2 abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2}" testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.516Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: aborting pipeline replication: peer="{Voter server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2 abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2}" testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.523Z [ERROR] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: failed to get log: index=23 error="log not found" testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.523Z [ERROR] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.523Z [ERROR] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: failed to get log: index=23 error="log not found" testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.523Z [ERROR] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: failed to get log: index=23 error="log not found" testing.go:153: 2023-08-02T15:12:59.524Z [INFO] creating new snapshot: path=/tmp/raft368743841/snapshots/2-113-1690989179524.tmp testing.go:153: 2023-08-02T15:12:59.524Z [INFO] creating new snapshot: path=/tmp/raft182143831/snapshots/2-113-1690989179524.tmp testLoggerAdapter verbose: server-009a1e34-c01e-0280-4511-05dabf8f55db: 2023-08-02T15:12:59.525Z [INFO] server-009a1e34-c01e-0280-4511-05dabf8f55db: copied to local snapshot: bytes=71 testing.go:149: server-009a1e34-c01e-0280-4511-05dabf8f55db: 2023-08-02T15:12:59.525Z [INFO] server-009a1e34-c01e-0280-4511-05dabf8f55db: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2: 2023-08-02T15:12:59.525Z [INFO] server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2: copied to local snapshot: bytes=71 testing.go:149: server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2: 2023-08-02T15:12:59.525Z [INFO] server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-009a1e34-c01e-0280-4511-05dabf8f55db: 2023-08-02T15:12:59.525Z [INFO] server-009a1e34-c01e-0280-4511-05dabf8f55db: Installed remote snapshot testing.go:149: server-009a1e34-c01e-0280-4511-05dabf8f55db: 2023-08-02T15:12:59.525Z [INFO] server-009a1e34-c01e-0280-4511-05dabf8f55db: Installed remote snapshot testLoggerAdapter verbose: server-009a1e34-c01e-0280-4511-05dabf8f55db: 2023-08-02T15:12:59.525Z [WARN] server-009a1e34-c01e-0280-4511-05dabf8f55db: skipping application of old log: index=22 testing.go:149: server-009a1e34-c01e-0280-4511-05dabf8f55db: 2023-08-02T15:12:59.525Z [WARN] server-009a1e34-c01e-0280-4511-05dabf8f55db: skipping application of old log: index=22 testLoggerAdapter verbose: server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2: 2023-08-02T15:12:59.525Z [INFO] server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2: Installed remote snapshot testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.525Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: pipelining replication: peer="{Voter server-009a1e34-c01e-0280-4511-05dabf8f55db 009a1e34-c01e-0280-4511-05dabf8f55db}" testing.go:149: server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2: 2023-08-02T15:12:59.525Z [INFO] server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2: Installed remote snapshot testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.525Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: pipelining replication: peer="{Voter server-009a1e34-c01e-0280-4511-05dabf8f55db 009a1e34-c01e-0280-4511-05dabf8f55db}" testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.525Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: pipelining replication: peer="{Voter server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2 abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2}" testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.525Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: pipelining replication: peer="{Voter server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2 abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2}" testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.536Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: aborting pipeline replication: peer="{Voter server-009a1e34-c01e-0280-4511-05dabf8f55db 009a1e34-c01e-0280-4511-05dabf8f55db}" testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.536Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: aborting pipeline replication: peer="{Voter server-009a1e34-c01e-0280-4511-05dabf8f55db 009a1e34-c01e-0280-4511-05dabf8f55db}" testLoggerAdapter verbose: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.536Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: aborting pipeline replication: peer="{Voter server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2 abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2}" testing.go:149: server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: 2023-08-02T15:12:59.536Z [INFO] server-5afb8d1a-261a-64dc-db92-bc1cbc2525b9: aborting pipeline replication: peer="{Voter server-abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2 abbb55f1-df1c-b6e2-2eb7-5c9fa8e644d2}" testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.551Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced Address:70ef33c8-8204-4afb-87b1-6fabdcbe3ced} {Suffrage:Voter ID:server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9 Address:0dad452b-68e0-c2e4-90d2-3f90baeab6c9} {Suffrage:Voter ID:server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4 Address:2b5444f2-c2c5-d465-f6d5-c4f4551af8d4}]" testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.551Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced Address:70ef33c8-8204-4afb-87b1-6fabdcbe3ced} {Suffrage:Voter ID:server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9 Address:0dad452b-68e0-c2e4-90d2-3f90baeab6c9} {Suffrage:Voter ID:server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4 Address:2b5444f2-c2c5-d465-f6d5-c4f4551af8d4}]" testLoggerAdapter verbose: server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9: 2023-08-02T15:12:59.551Z [INFO] server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced Address:70ef33c8-8204-4afb-87b1-6fabdcbe3ced} {Suffrage:Voter ID:server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9 Address:0dad452b-68e0-c2e4-90d2-3f90baeab6c9} {Suffrage:Voter ID:server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4 Address:2b5444f2-c2c5-d465-f6d5-c4f4551af8d4}]" testing.go:149: server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9: 2023-08-02T15:12:59.551Z [INFO] server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced Address:70ef33c8-8204-4afb-87b1-6fabdcbe3ced} {Suffrage:Voter ID:server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9 Address:0dad452b-68e0-c2e4-90d2-3f90baeab6c9} {Suffrage:Voter ID:server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4 Address:2b5444f2-c2c5-d465-f6d5-c4f4551af8d4}]" testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.551Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: entering follower state: follower="Node at 70ef33c8-8204-4afb-87b1-6fabdcbe3ced [Follower]" leader= testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.551Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: entering follower state: follower="Node at 70ef33c8-8204-4afb-87b1-6fabdcbe3ced [Follower]" leader= testLoggerAdapter verbose: server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: 2023-08-02T15:12:59.551Z [INFO] server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced Address:70ef33c8-8204-4afb-87b1-6fabdcbe3ced} {Suffrage:Voter ID:server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9 Address:0dad452b-68e0-c2e4-90d2-3f90baeab6c9} {Suffrage:Voter ID:server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4 Address:2b5444f2-c2c5-d465-f6d5-c4f4551af8d4}]" testLoggerAdapter verbose: server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9: 2023-08-02T15:12:59.551Z [INFO] server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9: entering follower state: follower="Node at 0dad452b-68e0-c2e4-90d2-3f90baeab6c9 [Follower]" leader= testing.go:149: server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: 2023-08-02T15:12:59.551Z [INFO] server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced Address:70ef33c8-8204-4afb-87b1-6fabdcbe3ced} {Suffrage:Voter ID:server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9 Address:0dad452b-68e0-c2e4-90d2-3f90baeab6c9} {Suffrage:Voter ID:server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4 Address:2b5444f2-c2c5-d465-f6d5-c4f4551af8d4}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:59.551Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:12:59.551Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9: 2023-08-02T15:12:59.551Z [INFO] server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9: entering follower state: follower="Node at 0dad452b-68e0-c2e4-90d2-3f90baeab6c9 [Follower]" leader= testLoggerAdapter verbose: server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: 2023-08-02T15:12:59.551Z [INFO] server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: entering follower state: follower="Node at 2b5444f2-c2c5-d465-f6d5-c4f4551af8d4 [Follower]" leader= testing.go:149: server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: 2023-08-02T15:12:59.551Z [INFO] server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: entering follower state: follower="Node at 2b5444f2-c2c5-d465-f6d5-c4f4551af8d4 [Follower]" leader= testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.607Z [WARN] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.607Z [WARN] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.607Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: entering candidate state: node="Node at 70ef33c8-8204-4afb-87b1-6fabdcbe3ced [Candidate]" term=2 testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.607Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: entering candidate state: node="Node at 70ef33c8-8204-4afb-87b1-6fabdcbe3ced [Candidate]" term=2 testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.608Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: election won: tally=2 testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.608Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: election won: tally=2 testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.608Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: entering leader state: leader="Node at 70ef33c8-8204-4afb-87b1-6fabdcbe3ced [Leader]" testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.608Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: entering leader state: leader="Node at 70ef33c8-8204-4afb-87b1-6fabdcbe3ced [Leader]" testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.608Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: added peer, starting replication: peer=server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9 testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.608Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: added peer, starting replication: peer=server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9 testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.608Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: added peer, starting replication: peer=server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4 testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.608Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: added peer, starting replication: peer=server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4 testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.608Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: pipelining replication: peer="{Voter server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4 2b5444f2-c2c5-d465-f6d5-c4f4551af8d4}" testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.608Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: pipelining replication: peer="{Voter server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4 2b5444f2-c2c5-d465-f6d5-c4f4551af8d4}" testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.608Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: pipelining replication: peer="{Voter server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9 0dad452b-68e0-c2e4-90d2-3f90baeab6c9}" testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.608Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: pipelining replication: peer="{Voter server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9 0dad452b-68e0-c2e4-90d2-3f90baeab6c9}" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:59.713Z [INFO] stable state for Leader reached at 2023-08-02 15:12:59.608344187 +0000 UTC m=+11.704280554 (1 nodes), 56.54741ms from start of poll, 57.359673ms from cluster start. Timeout at 2023-08-02 15:12:59.713425547 +0000 UTC m=+11.809361898, 105.081344ms after stability testing.go:149: cluster: 2023-08-02T15:12:59.713Z [INFO] stable state for Leader reached at 2023-08-02 15:12:59.608344187 +0000 UTC m=+11.704280554 (1 nodes), 56.54741ms from start of poll, 57.359673ms from cluster start. Timeout at 2023-08-02 15:12:59.713425547 +0000 UTC m=+11.809361898, 105.081344ms after stability testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.714Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: starting snapshot up to: index=12 testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.714Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: starting snapshot up to: index=12 testing.go:153: 2023-08-02T15:12:59.714Z [INFO] creating new snapshot: path=/tmp/raft860841061/snapshots/2-12-1690989179714.tmp testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.714Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: snapshot complete up to: index=12 testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.714Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: snapshot complete up to: index=12 testing.go:153: 2023-08-02T15:12:59.715Z [INFO] creating new snapshot: path=/tmp/raft860841061/snapshots/2-1013-1690989179715.tmp testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.715Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: copied to local snapshot: bytes=71 testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.715Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.720Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: restored user snapshot: index=1 testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.720Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: restored user snapshot: index=1 testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.720Z [ERROR] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: failed to get log: index=23 error="log not found" testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.720Z [ERROR] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.720Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: aborting pipeline replication: peer="{Voter server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4 2b5444f2-c2c5-d465-f6d5-c4f4551af8d4}" testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.720Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: aborting pipeline replication: peer="{Voter server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4 2b5444f2-c2c5-d465-f6d5-c4f4551af8d4}" testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.720Z [ERROR] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: failed to get log: index=23 error="log not found" testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.720Z [ERROR] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.721Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: aborting pipeline replication: peer="{Voter server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9 0dad452b-68e0-c2e4-90d2-3f90baeab6c9}" testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.721Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: aborting pipeline replication: peer="{Voter server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9 0dad452b-68e0-c2e4-90d2-3f90baeab6c9}" testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.727Z [ERROR] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: failed to get log: index=23 error="log not found" testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.727Z [ERROR] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: failed to get log: index=23 error="log not found" testing.go:153: 2023-08-02T15:12:59.728Z [INFO] creating new snapshot: path=/tmp/raft211612521/snapshots/2-1013-1690989179728.tmp testLoggerAdapter verbose: server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: 2023-08-02T15:12:59.729Z [INFO] server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: copied to local snapshot: bytes=71 testing.go:149: server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: 2023-08-02T15:12:59.729Z [INFO] server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: 2023-08-02T15:12:59.729Z [INFO] server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: Installed remote snapshot testing.go:149: server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: 2023-08-02T15:12:59.729Z [INFO] server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: Installed remote snapshot testLoggerAdapter verbose: server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: 2023-08-02T15:12:59.730Z [WARN] server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: skipping application of old log: index=22 testing.go:149: server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: 2023-08-02T15:12:59.730Z [WARN] server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4: skipping application of old log: index=22 testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.730Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: pipelining replication: peer="{Voter server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4 2b5444f2-c2c5-d465-f6d5-c4f4551af8d4}" testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.730Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: pipelining replication: peer="{Voter server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4 2b5444f2-c2c5-d465-f6d5-c4f4551af8d4}" testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.730Z [ERROR] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: failed to get log: index=23 error="log not found" testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.730Z [ERROR] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: failed to get log: index=23 error="log not found" testing.go:153: 2023-08-02T15:12:59.730Z [INFO] creating new snapshot: path=/tmp/raft786216927/snapshots/2-1013-1690989179730.tmp testLoggerAdapter verbose: server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9: 2023-08-02T15:12:59.731Z [INFO] server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9: copied to local snapshot: bytes=71 testing.go:149: server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9: 2023-08-02T15:12:59.731Z [INFO] server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9: 2023-08-02T15:12:59.731Z [INFO] server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9: Installed remote snapshot testing.go:149: server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9: 2023-08-02T15:12:59.731Z [INFO] server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9: Installed remote snapshot testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.731Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: pipelining replication: peer="{Voter server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9 0dad452b-68e0-c2e4-90d2-3f90baeab6c9}" testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.731Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: pipelining replication: peer="{Voter server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9 0dad452b-68e0-c2e4-90d2-3f90baeab6c9}" testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.745Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: aborting pipeline replication: peer="{Voter server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4 2b5444f2-c2c5-d465-f6d5-c4f4551af8d4}" testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.745Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: aborting pipeline replication: peer="{Voter server-2b5444f2-c2c5-d465-f6d5-c4f4551af8d4 2b5444f2-c2c5-d465-f6d5-c4f4551af8d4}" testLoggerAdapter verbose: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.745Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: aborting pipeline replication: peer="{Voter server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9 0dad452b-68e0-c2e4-90d2-3f90baeab6c9}" testing.go:149: server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: 2023-08-02T15:12:59.745Z [INFO] server-70ef33c8-8204-4afb-87b1-6fabdcbe3ced: aborting pipeline replication: peer="{Voter server-0dad452b-68e0-c2e4-90d2-3f90baeab6c9 0dad452b-68e0-c2e4-90d2-3f90baeab6c9}" testLoggerAdapter verbose: server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee: 2023-08-02T15:12:59.747Z [INFO] server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee Address:a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee} {Suffrage:Voter ID:server-fcba1207-da01-9c09-257d-4ce22241f775 Address:fcba1207-da01-9c09-257d-4ce22241f775} {Suffrage:Voter ID:server-7f10c0ec-9758-2914-9b53-04ccadf5b75c Address:7f10c0ec-9758-2914-9b53-04ccadf5b75c}]" testing.go:149: server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee: 2023-08-02T15:12:59.747Z [INFO] server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee Address:a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee} {Suffrage:Voter ID:server-fcba1207-da01-9c09-257d-4ce22241f775 Address:fcba1207-da01-9c09-257d-4ce22241f775} {Suffrage:Voter ID:server-7f10c0ec-9758-2914-9b53-04ccadf5b75c Address:7f10c0ec-9758-2914-9b53-04ccadf5b75c}]" testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.747Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee Address:a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee} {Suffrage:Voter ID:server-fcba1207-da01-9c09-257d-4ce22241f775 Address:fcba1207-da01-9c09-257d-4ce22241f775} {Suffrage:Voter ID:server-7f10c0ec-9758-2914-9b53-04ccadf5b75c Address:7f10c0ec-9758-2914-9b53-04ccadf5b75c}]" testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.747Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee Address:a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee} {Suffrage:Voter ID:server-fcba1207-da01-9c09-257d-4ce22241f775 Address:fcba1207-da01-9c09-257d-4ce22241f775} {Suffrage:Voter ID:server-7f10c0ec-9758-2914-9b53-04ccadf5b75c Address:7f10c0ec-9758-2914-9b53-04ccadf5b75c}]" testLoggerAdapter verbose: server-7f10c0ec-9758-2914-9b53-04ccadf5b75c: 2023-08-02T15:12:59.747Z [INFO] server-7f10c0ec-9758-2914-9b53-04ccadf5b75c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee Address:a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee} {Suffrage:Voter ID:server-fcba1207-da01-9c09-257d-4ce22241f775 Address:fcba1207-da01-9c09-257d-4ce22241f775} {Suffrage:Voter ID:server-7f10c0ec-9758-2914-9b53-04ccadf5b75c Address:7f10c0ec-9758-2914-9b53-04ccadf5b75c}]" testing.go:149: server-7f10c0ec-9758-2914-9b53-04ccadf5b75c: 2023-08-02T15:12:59.747Z [INFO] server-7f10c0ec-9758-2914-9b53-04ccadf5b75c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee Address:a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee} {Suffrage:Voter ID:server-fcba1207-da01-9c09-257d-4ce22241f775 Address:fcba1207-da01-9c09-257d-4ce22241f775} {Suffrage:Voter ID:server-7f10c0ec-9758-2914-9b53-04ccadf5b75c Address:7f10c0ec-9758-2914-9b53-04ccadf5b75c}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:59.747Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:12:59.747Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.747Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: entering follower state: follower="Node at fcba1207-da01-9c09-257d-4ce22241f775 [Follower]" leader= testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.747Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: entering follower state: follower="Node at fcba1207-da01-9c09-257d-4ce22241f775 [Follower]" leader= testLoggerAdapter verbose: server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee: 2023-08-02T15:12:59.747Z [INFO] server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee: entering follower state: follower="Node at a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee [Follower]" leader= testing.go:149: server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee: 2023-08-02T15:12:59.747Z [INFO] server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee: entering follower state: follower="Node at a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee [Follower]" leader= testLoggerAdapter verbose: server-7f10c0ec-9758-2914-9b53-04ccadf5b75c: 2023-08-02T15:12:59.747Z [INFO] server-7f10c0ec-9758-2914-9b53-04ccadf5b75c: entering follower state: follower="Node at 7f10c0ec-9758-2914-9b53-04ccadf5b75c [Follower]" leader= testing.go:149: server-7f10c0ec-9758-2914-9b53-04ccadf5b75c: 2023-08-02T15:12:59.747Z [INFO] server-7f10c0ec-9758-2914-9b53-04ccadf5b75c: entering follower state: follower="Node at 7f10c0ec-9758-2914-9b53-04ccadf5b75c [Follower]" leader= testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.820Z [WARN] server-fcba1207-da01-9c09-257d-4ce22241f775: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.820Z [WARN] server-fcba1207-da01-9c09-257d-4ce22241f775: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.821Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: entering candidate state: node="Node at fcba1207-da01-9c09-257d-4ce22241f775 [Candidate]" term=2 testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.821Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: entering candidate state: node="Node at fcba1207-da01-9c09-257d-4ce22241f775 [Candidate]" term=2 testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.821Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: election won: tally=2 testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.821Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: election won: tally=2 testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.821Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: entering leader state: leader="Node at fcba1207-da01-9c09-257d-4ce22241f775 [Leader]" testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.821Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: entering leader state: leader="Node at fcba1207-da01-9c09-257d-4ce22241f775 [Leader]" testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.821Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: added peer, starting replication: peer=server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.821Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: added peer, starting replication: peer=server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.821Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: added peer, starting replication: peer=server-7f10c0ec-9758-2914-9b53-04ccadf5b75c testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.821Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: added peer, starting replication: peer=server-7f10c0ec-9758-2914-9b53-04ccadf5b75c testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.821Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: pipelining replication: peer="{Voter server-7f10c0ec-9758-2914-9b53-04ccadf5b75c 7f10c0ec-9758-2914-9b53-04ccadf5b75c}" testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.821Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: pipelining replication: peer="{Voter server-7f10c0ec-9758-2914-9b53-04ccadf5b75c 7f10c0ec-9758-2914-9b53-04ccadf5b75c}" testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.821Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: pipelining replication: peer="{Voter server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee}" testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.821Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: pipelining replication: peer="{Voter server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee}" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:59.926Z [INFO] stable state for Leader reached at 2023-08-02 15:12:59.821294997 +0000 UTC m=+11.917231331 (1 nodes), 73.794891ms from start of poll, 74.205928ms from cluster start. Timeout at 2023-08-02 15:12:59.926323397 +0000 UTC m=+12.022259732, 105.028401ms after stability testing.go:149: cluster: 2023-08-02T15:12:59.926Z [INFO] stable state for Leader reached at 2023-08-02 15:12:59.821294997 +0000 UTC m=+11.917231331 (1 nodes), 73.794891ms from start of poll, 74.205928ms from cluster start. Timeout at 2023-08-02 15:12:59.926323397 +0000 UTC m=+12.022259732, 105.028401ms after stability testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.926Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: starting snapshot up to: index=12 testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.926Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: starting snapshot up to: index=12 testing.go:153: 2023-08-02T15:12:59.926Z [INFO] creating new snapshot: path=/tmp/raft370563757/snapshots/2-12-1690989179926.tmp testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.927Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: snapshot complete up to: index=12 testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.927Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: snapshot complete up to: index=12 testing.go:153: 2023-08-02T15:12:59.927Z [INFO] creating new snapshot: path=/tmp/raft370563757/snapshots/2-10013-1690989179927.tmp testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.928Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: copied to local snapshot: bytes=71 testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.928Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.928Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: restored user snapshot: index=1 testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.928Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: restored user snapshot: index=1 testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.928Z [ERROR] server-fcba1207-da01-9c09-257d-4ce22241f775: failed to get log: index=23 error="log not found" testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.928Z [ERROR] server-fcba1207-da01-9c09-257d-4ce22241f775: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.928Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: aborting pipeline replication: peer="{Voter server-7f10c0ec-9758-2914-9b53-04ccadf5b75c 7f10c0ec-9758-2914-9b53-04ccadf5b75c}" testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.928Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: aborting pipeline replication: peer="{Voter server-7f10c0ec-9758-2914-9b53-04ccadf5b75c 7f10c0ec-9758-2914-9b53-04ccadf5b75c}" testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.928Z [ERROR] server-fcba1207-da01-9c09-257d-4ce22241f775: failed to get log: index=23 error="log not found" testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.928Z [ERROR] server-fcba1207-da01-9c09-257d-4ce22241f775: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.928Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: aborting pipeline replication: peer="{Voter server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee}" testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.928Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: aborting pipeline replication: peer="{Voter server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee}" testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.936Z [ERROR] server-fcba1207-da01-9c09-257d-4ce22241f775: failed to get log: index=23 error="log not found" testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.936Z [ERROR] server-fcba1207-da01-9c09-257d-4ce22241f775: failed to get log: index=23 error="log not found" testing.go:153: 2023-08-02T15:12:59.937Z [INFO] creating new snapshot: path=/tmp/raft659919235/snapshots/2-10013-1690989179937.tmp testLoggerAdapter verbose: server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee: 2023-08-02T15:12:59.937Z [INFO] server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee: copied to local snapshot: bytes=71 testing.go:149: server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee: 2023-08-02T15:12:59.937Z [INFO] server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee: 2023-08-02T15:12:59.937Z [INFO] server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee: Installed remote snapshot testing.go:149: server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee: 2023-08-02T15:12:59.937Z [INFO] server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee: Installed remote snapshot testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.937Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: pipelining replication: peer="{Voter server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee}" testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.937Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: pipelining replication: peer="{Voter server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee}" testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.938Z [ERROR] server-fcba1207-da01-9c09-257d-4ce22241f775: failed to get log: index=23 error="log not found" testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.938Z [ERROR] server-fcba1207-da01-9c09-257d-4ce22241f775: failed to get log: index=23 error="log not found" testing.go:153: 2023-08-02T15:12:59.939Z [INFO] creating new snapshot: path=/tmp/raft166029159/snapshots/2-10013-1690989179939.tmp testLoggerAdapter verbose: server-7f10c0ec-9758-2914-9b53-04ccadf5b75c: 2023-08-02T15:12:59.939Z [INFO] server-7f10c0ec-9758-2914-9b53-04ccadf5b75c: copied to local snapshot: bytes=71 testing.go:149: server-7f10c0ec-9758-2914-9b53-04ccadf5b75c: 2023-08-02T15:12:59.939Z [INFO] server-7f10c0ec-9758-2914-9b53-04ccadf5b75c: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-7f10c0ec-9758-2914-9b53-04ccadf5b75c: 2023-08-02T15:12:59.939Z [INFO] server-7f10c0ec-9758-2914-9b53-04ccadf5b75c: Installed remote snapshot testing.go:149: server-7f10c0ec-9758-2914-9b53-04ccadf5b75c: 2023-08-02T15:12:59.939Z [INFO] server-7f10c0ec-9758-2914-9b53-04ccadf5b75c: Installed remote snapshot testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.939Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: pipelining replication: peer="{Voter server-7f10c0ec-9758-2914-9b53-04ccadf5b75c 7f10c0ec-9758-2914-9b53-04ccadf5b75c}" testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.939Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: pipelining replication: peer="{Voter server-7f10c0ec-9758-2914-9b53-04ccadf5b75c 7f10c0ec-9758-2914-9b53-04ccadf5b75c}" testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.954Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: aborting pipeline replication: peer="{Voter server-7f10c0ec-9758-2914-9b53-04ccadf5b75c 7f10c0ec-9758-2914-9b53-04ccadf5b75c}" testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.954Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: aborting pipeline replication: peer="{Voter server-7f10c0ec-9758-2914-9b53-04ccadf5b75c 7f10c0ec-9758-2914-9b53-04ccadf5b75c}" testLoggerAdapter verbose: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.954Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: aborting pipeline replication: peer="{Voter server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee}" testing.go:149: server-fcba1207-da01-9c09-257d-4ce22241f775: 2023-08-02T15:12:59.954Z [INFO] server-fcba1207-da01-9c09-257d-4ce22241f775: aborting pipeline replication: peer="{Voter server-a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee a689aa5b-5594-ffab-bcf9-8dbbd4d7b9ee}" --- PASS: TestRaft_UserRestore (1.18s) === RUN TestRaft_SendSnapshotFollower testLoggerAdapter verbose: server-90180b34-5b90-ebe3-c828-5fe59d8e596a: 2023-08-02T15:12:59.956Z [INFO] server-90180b34-5b90-ebe3-c828-5fe59d8e596a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90180b34-5b90-ebe3-c828-5fe59d8e596a Address:90180b34-5b90-ebe3-c828-5fe59d8e596a} {Suffrage:Voter ID:server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680 Address:5e1a394b-e983-b1a6-50ae-99f0a0ab8680} {Suffrage:Voter ID:server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4 Address:4dd7916e-4fcc-1112-eaed-08c8cc08fef4}]" testing.go:149: server-90180b34-5b90-ebe3-c828-5fe59d8e596a: 2023-08-02T15:12:59.956Z [INFO] server-90180b34-5b90-ebe3-c828-5fe59d8e596a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90180b34-5b90-ebe3-c828-5fe59d8e596a Address:90180b34-5b90-ebe3-c828-5fe59d8e596a} {Suffrage:Voter ID:server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680 Address:5e1a394b-e983-b1a6-50ae-99f0a0ab8680} {Suffrage:Voter ID:server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4 Address:4dd7916e-4fcc-1112-eaed-08c8cc08fef4}]" testLoggerAdapter verbose: server-90180b34-5b90-ebe3-c828-5fe59d8e596a: 2023-08-02T15:12:59.957Z [INFO] server-90180b34-5b90-ebe3-c828-5fe59d8e596a: entering follower state: follower="Node at 90180b34-5b90-ebe3-c828-5fe59d8e596a [Follower]" leader= testing.go:149: server-90180b34-5b90-ebe3-c828-5fe59d8e596a: 2023-08-02T15:12:59.957Z [INFO] server-90180b34-5b90-ebe3-c828-5fe59d8e596a: entering follower state: follower="Node at 90180b34-5b90-ebe3-c828-5fe59d8e596a [Follower]" leader= testLoggerAdapter verbose: server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: 2023-08-02T15:12:59.957Z [INFO] server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90180b34-5b90-ebe3-c828-5fe59d8e596a Address:90180b34-5b90-ebe3-c828-5fe59d8e596a} {Suffrage:Voter ID:server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680 Address:5e1a394b-e983-b1a6-50ae-99f0a0ab8680} {Suffrage:Voter ID:server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4 Address:4dd7916e-4fcc-1112-eaed-08c8cc08fef4}]" testing.go:149: server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: 2023-08-02T15:12:59.957Z [INFO] server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90180b34-5b90-ebe3-c828-5fe59d8e596a Address:90180b34-5b90-ebe3-c828-5fe59d8e596a} {Suffrage:Voter ID:server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680 Address:5e1a394b-e983-b1a6-50ae-99f0a0ab8680} {Suffrage:Voter ID:server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4 Address:4dd7916e-4fcc-1112-eaed-08c8cc08fef4}]" testLoggerAdapter verbose: server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: 2023-08-02T15:12:59.957Z [INFO] server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: entering follower state: follower="Node at 5e1a394b-e983-b1a6-50ae-99f0a0ab8680 [Follower]" leader= testing.go:149: server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: 2023-08-02T15:12:59.957Z [INFO] server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: entering follower state: follower="Node at 5e1a394b-e983-b1a6-50ae-99f0a0ab8680 [Follower]" leader= testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:12:59.957Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90180b34-5b90-ebe3-c828-5fe59d8e596a Address:90180b34-5b90-ebe3-c828-5fe59d8e596a} {Suffrage:Voter ID:server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680 Address:5e1a394b-e983-b1a6-50ae-99f0a0ab8680} {Suffrage:Voter ID:server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4 Address:4dd7916e-4fcc-1112-eaed-08c8cc08fef4}]" testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:12:59.957Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90180b34-5b90-ebe3-c828-5fe59d8e596a Address:90180b34-5b90-ebe3-c828-5fe59d8e596a} {Suffrage:Voter ID:server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680 Address:5e1a394b-e983-b1a6-50ae-99f0a0ab8680} {Suffrage:Voter ID:server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4 Address:4dd7916e-4fcc-1112-eaed-08c8cc08fef4}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:12:59.957Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-08-02T15:12:59.957Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:12:59.957Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: entering follower state: follower="Node at 4dd7916e-4fcc-1112-eaed-08c8cc08fef4 [Follower]" leader= testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:12:59.957Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: entering follower state: follower="Node at 4dd7916e-4fcc-1112-eaed-08c8cc08fef4 [Follower]" leader= testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.042Z [WARN] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.042Z [WARN] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.042Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: entering candidate state: node="Node at 4dd7916e-4fcc-1112-eaed-08c8cc08fef4 [Candidate]" term=2 testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.042Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: entering candidate state: node="Node at 4dd7916e-4fcc-1112-eaed-08c8cc08fef4 [Candidate]" term=2 testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.042Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: election won: tally=2 testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.042Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: election won: tally=2 testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.042Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: entering leader state: leader="Node at 4dd7916e-4fcc-1112-eaed-08c8cc08fef4 [Leader]" testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.042Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: entering leader state: leader="Node at 4dd7916e-4fcc-1112-eaed-08c8cc08fef4 [Leader]" testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.042Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: added peer, starting replication: peer=server-90180b34-5b90-ebe3-c828-5fe59d8e596a testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.042Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: added peer, starting replication: peer=server-90180b34-5b90-ebe3-c828-5fe59d8e596a testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.042Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: added peer, starting replication: peer=server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680 testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.042Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: added peer, starting replication: peer=server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680 testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.042Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: pipelining replication: peer="{Voter server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680 5e1a394b-e983-b1a6-50ae-99f0a0ab8680}" testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.042Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: pipelining replication: peer="{Voter server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680 5e1a394b-e983-b1a6-50ae-99f0a0ab8680}" testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.043Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: pipelining replication: peer="{Voter server-90180b34-5b90-ebe3-c828-5fe59d8e596a 90180b34-5b90-ebe3-c828-5fe59d8e596a}" testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.043Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: pipelining replication: peer="{Voter server-90180b34-5b90-ebe3-c828-5fe59d8e596a 90180b34-5b90-ebe3-c828-5fe59d8e596a}" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:00.149Z [INFO] stable state for Follower reached at 2023-08-02 15:13:00.043136359 +0000 UTC m=+12.139072697 (2 nodes), 85.47898ms from start of poll, 86.773127ms from cluster start. Timeout at 2023-08-02 15:13:00.149135193 +0000 UTC m=+12.245071708, 105.999011ms after stability testing.go:149: cluster: 2023-08-02T15:13:00.149Z [INFO] stable state for Follower reached at 2023-08-02 15:13:00.043136359 +0000 UTC m=+12.139072697 (2 nodes), 85.47898ms from start of poll, 86.773127ms from cluster start. Timeout at 2023-08-02 15:13:00.149135193 +0000 UTC m=+12.245071708, 105.999011ms after stability testLoggerAdapter verbose: cluster: 2023-08-02T15:13:00.149Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:13:00.149Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-08-02T15:13:00.254Z [INFO] stable state for Leader reached at 2023-08-02 15:13:00.149356647 +0000 UTC m=+12.245292975 (1 nodes), 2.1µs from start of poll, 192.993405ms from cluster start. Timeout at 2023-08-02 15:13:00.25444801 +0000 UTC m=+12.350384354, 105.091379ms after stability testing.go:149: cluster: 2023-08-02T15:13:00.254Z [INFO] stable state for Leader reached at 2023-08-02 15:13:00.149356647 +0000 UTC m=+12.245292975 (1 nodes), 2.1µs from start of poll, 192.993405ms from cluster start. Timeout at 2023-08-02 15:13:00.25444801 +0000 UTC m=+12.350384354, 105.091379ms after stability testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.254Z [ERROR] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: failed to pipeline appendEntries: peer="{Voter server-90180b34-5b90-ebe3-c828-5fe59d8e596a 90180b34-5b90-ebe3-c828-5fe59d8e596a}" error="append pipeline closed" testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.254Z [ERROR] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: failed to pipeline appendEntries: peer="{Voter server-90180b34-5b90-ebe3-c828-5fe59d8e596a 90180b34-5b90-ebe3-c828-5fe59d8e596a}" error="append pipeline closed" testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.254Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: aborting pipeline replication: peer="{Voter server-90180b34-5b90-ebe3-c828-5fe59d8e596a 90180b34-5b90-ebe3-c828-5fe59d8e596a}" testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.254Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: aborting pipeline replication: peer="{Voter server-90180b34-5b90-ebe3-c828-5fe59d8e596a 90180b34-5b90-ebe3-c828-5fe59d8e596a}" testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.254Z [ERROR] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: failed to appendEntries to: peer="{Voter server-90180b34-5b90-ebe3-c828-5fe59d8e596a 90180b34-5b90-ebe3-c828-5fe59d8e596a}" error="failed to connect to peer: 90180b34-5b90-ebe3-c828-5fe59d8e596a" testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.254Z [ERROR] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: failed to appendEntries to: peer="{Voter server-90180b34-5b90-ebe3-c828-5fe59d8e596a 90180b34-5b90-ebe3-c828-5fe59d8e596a}" error="failed to connect to peer: 90180b34-5b90-ebe3-c828-5fe59d8e596a" testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.255Z [ERROR] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: failed to heartbeat to: peer=90180b34-5b90-ebe3-c828-5fe59d8e596a error="failed to connect to peer: 90180b34-5b90-ebe3-c828-5fe59d8e596a" testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.255Z [ERROR] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: failed to heartbeat to: peer=90180b34-5b90-ebe3-c828-5fe59d8e596a error="failed to connect to peer: 90180b34-5b90-ebe3-c828-5fe59d8e596a" raft_test.go:1320: [INFO] Finished apply without behind follower testLoggerAdapter verbose: server-90180b34-5b90-ebe3-c828-5fe59d8e596a: 2023-08-02T15:13:00.257Z [ERROR] server-90180b34-5b90-ebe3-c828-5fe59d8e596a: failed to take snapshot: error="nothing new to snapshot" testing.go:149: server-90180b34-5b90-ebe3-c828-5fe59d8e596a: 2023-08-02T15:13:00.257Z [ERROR] server-90180b34-5b90-ebe3-c828-5fe59d8e596a: failed to take snapshot: error="nothing new to snapshot" testLoggerAdapter verbose: server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: 2023-08-02T15:13:00.257Z [INFO] server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: starting snapshot up to: index=98 testing.go:149: server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: 2023-08-02T15:13:00.257Z [INFO] server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: starting snapshot up to: index=98 testing.go:153: 2023-08-02T15:13:00.257Z [INFO] creating new snapshot: path=/tmp/raft916630411/snapshots/2-98-1690989180257.tmp testLoggerAdapter verbose: server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: 2023-08-02T15:13:00.258Z [INFO] server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: compacting logs: from=1 to=92 testing.go:149: server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: 2023-08-02T15:13:00.258Z [INFO] server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: 2023-08-02T15:13:00.258Z [INFO] server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: snapshot complete up to: index=98 testing.go:149: server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: 2023-08-02T15:13:00.258Z [INFO] server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680: snapshot complete up to: index=98 testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.258Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: starting snapshot up to: index=102 testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.258Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: starting snapshot up to: index=102 testing.go:153: 2023-08-02T15:13:00.258Z [INFO] creating new snapshot: path=/tmp/raft626003957/snapshots/2-102-1690989180258.tmp testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.258Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: compacting logs: from=1 to=92 testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.258Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.258Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: snapshot complete up to: index=102 testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.258Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: snapshot complete up to: index=102 testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.264Z [ERROR] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: failed to get log: index=2 error="log not found" testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.264Z [ERROR] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: failed to get log: index=2 error="log not found" testing.go:153: 2023-08-02T15:13:00.265Z [INFO] creating new snapshot: path=/tmp/raft245301297/snapshots/2-102-1690989180265.tmp testLoggerAdapter verbose: server-90180b34-5b90-ebe3-c828-5fe59d8e596a: 2023-08-02T15:13:00.265Z [INFO] server-90180b34-5b90-ebe3-c828-5fe59d8e596a: copied to local snapshot: bytes=693 testing.go:149: server-90180b34-5b90-ebe3-c828-5fe59d8e596a: 2023-08-02T15:13:00.265Z [INFO] server-90180b34-5b90-ebe3-c828-5fe59d8e596a: copied to local snapshot: bytes=693 testLoggerAdapter verbose: server-90180b34-5b90-ebe3-c828-5fe59d8e596a: 2023-08-02T15:13:00.266Z [INFO] server-90180b34-5b90-ebe3-c828-5fe59d8e596a: Installed remote snapshot testing.go:149: server-90180b34-5b90-ebe3-c828-5fe59d8e596a: 2023-08-02T15:13:00.266Z [INFO] server-90180b34-5b90-ebe3-c828-5fe59d8e596a: Installed remote snapshot testLoggerAdapter verbose: server-90180b34-5b90-ebe3-c828-5fe59d8e596a: 2023-08-02T15:13:00.266Z [WARN] server-90180b34-5b90-ebe3-c828-5fe59d8e596a: skipping application of old log: index=102 testing.go:149: server-90180b34-5b90-ebe3-c828-5fe59d8e596a: 2023-08-02T15:13:00.266Z [WARN] server-90180b34-5b90-ebe3-c828-5fe59d8e596a: skipping application of old log: index=102 testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.266Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: pipelining replication: peer="{Voter server-90180b34-5b90-ebe3-c828-5fe59d8e596a 90180b34-5b90-ebe3-c828-5fe59d8e596a}" testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.266Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: pipelining replication: peer="{Voter server-90180b34-5b90-ebe3-c828-5fe59d8e596a 90180b34-5b90-ebe3-c828-5fe59d8e596a}" testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.269Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: aborting pipeline replication: peer="{Voter server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680 5e1a394b-e983-b1a6-50ae-99f0a0ab8680}" testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.269Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: aborting pipeline replication: peer="{Voter server-5e1a394b-e983-b1a6-50ae-99f0a0ab8680 5e1a394b-e983-b1a6-50ae-99f0a0ab8680}" testLoggerAdapter verbose: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.269Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: aborting pipeline replication: peer="{Voter server-90180b34-5b90-ebe3-c828-5fe59d8e596a 90180b34-5b90-ebe3-c828-5fe59d8e596a}" testing.go:149: server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: 2023-08-02T15:13:00.269Z [INFO] server-4dd7916e-4fcc-1112-eaed-08c8cc08fef4: aborting pipeline replication: peer="{Voter server-90180b34-5b90-ebe3-c828-5fe59d8e596a 90180b34-5b90-ebe3-c828-5fe59d8e596a}" --- PASS: TestRaft_SendSnapshotFollower (0.31s) === RUN TestRaft_SendSnapshotAndLogsFollower testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.270Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-781112b6-6781-e0d3-2e7a-fb39add31117 Address:781112b6-6781-e0d3-2e7a-fb39add31117} {Suffrage:Voter ID:server-80b68dcd-3486-a063-a2f4-474aebdf95c1 Address:80b68dcd-3486-a063-a2f4-474aebdf95c1} {Suffrage:Voter ID:server-6a85796e-de9a-002e-9695-01b9be4344c6 Address:6a85796e-de9a-002e-9695-01b9be4344c6}]" testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.270Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-781112b6-6781-e0d3-2e7a-fb39add31117 Address:781112b6-6781-e0d3-2e7a-fb39add31117} {Suffrage:Voter ID:server-80b68dcd-3486-a063-a2f4-474aebdf95c1 Address:80b68dcd-3486-a063-a2f4-474aebdf95c1} {Suffrage:Voter ID:server-6a85796e-de9a-002e-9695-01b9be4344c6 Address:6a85796e-de9a-002e-9695-01b9be4344c6}]" testLoggerAdapter verbose: server-80b68dcd-3486-a063-a2f4-474aebdf95c1: 2023-08-02T15:13:00.271Z [INFO] server-80b68dcd-3486-a063-a2f4-474aebdf95c1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-781112b6-6781-e0d3-2e7a-fb39add31117 Address:781112b6-6781-e0d3-2e7a-fb39add31117} {Suffrage:Voter ID:server-80b68dcd-3486-a063-a2f4-474aebdf95c1 Address:80b68dcd-3486-a063-a2f4-474aebdf95c1} {Suffrage:Voter ID:server-6a85796e-de9a-002e-9695-01b9be4344c6 Address:6a85796e-de9a-002e-9695-01b9be4344c6}]" testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.271Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: entering follower state: follower="Node at 781112b6-6781-e0d3-2e7a-fb39add31117 [Follower]" leader= testing.go:149: server-80b68dcd-3486-a063-a2f4-474aebdf95c1: 2023-08-02T15:13:00.271Z [INFO] server-80b68dcd-3486-a063-a2f4-474aebdf95c1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-781112b6-6781-e0d3-2e7a-fb39add31117 Address:781112b6-6781-e0d3-2e7a-fb39add31117} {Suffrage:Voter ID:server-80b68dcd-3486-a063-a2f4-474aebdf95c1 Address:80b68dcd-3486-a063-a2f4-474aebdf95c1} {Suffrage:Voter ID:server-6a85796e-de9a-002e-9695-01b9be4344c6 Address:6a85796e-de9a-002e-9695-01b9be4344c6}]" testLoggerAdapter verbose: server-80b68dcd-3486-a063-a2f4-474aebdf95c1: 2023-08-02T15:13:00.271Z [INFO] server-80b68dcd-3486-a063-a2f4-474aebdf95c1: entering follower state: follower="Node at 80b68dcd-3486-a063-a2f4-474aebdf95c1 [Follower]" leader= testing.go:149: server-80b68dcd-3486-a063-a2f4-474aebdf95c1: 2023-08-02T15:13:00.271Z [INFO] server-80b68dcd-3486-a063-a2f4-474aebdf95c1: entering follower state: follower="Node at 80b68dcd-3486-a063-a2f4-474aebdf95c1 [Follower]" leader= testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.271Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: entering follower state: follower="Node at 781112b6-6781-e0d3-2e7a-fb39add31117 [Follower]" leader= testLoggerAdapter verbose: server-6a85796e-de9a-002e-9695-01b9be4344c6: 2023-08-02T15:13:00.271Z [INFO] server-6a85796e-de9a-002e-9695-01b9be4344c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-781112b6-6781-e0d3-2e7a-fb39add31117 Address:781112b6-6781-e0d3-2e7a-fb39add31117} {Suffrage:Voter ID:server-80b68dcd-3486-a063-a2f4-474aebdf95c1 Address:80b68dcd-3486-a063-a2f4-474aebdf95c1} {Suffrage:Voter ID:server-6a85796e-de9a-002e-9695-01b9be4344c6 Address:6a85796e-de9a-002e-9695-01b9be4344c6}]" testing.go:149: server-6a85796e-de9a-002e-9695-01b9be4344c6: 2023-08-02T15:13:00.271Z [INFO] server-6a85796e-de9a-002e-9695-01b9be4344c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-781112b6-6781-e0d3-2e7a-fb39add31117 Address:781112b6-6781-e0d3-2e7a-fb39add31117} {Suffrage:Voter ID:server-80b68dcd-3486-a063-a2f4-474aebdf95c1 Address:80b68dcd-3486-a063-a2f4-474aebdf95c1} {Suffrage:Voter ID:server-6a85796e-de9a-002e-9695-01b9be4344c6 Address:6a85796e-de9a-002e-9695-01b9be4344c6}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:00.271Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-08-02T15:13:00.271Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-6a85796e-de9a-002e-9695-01b9be4344c6: 2023-08-02T15:13:00.271Z [INFO] server-6a85796e-de9a-002e-9695-01b9be4344c6: entering follower state: follower="Node at 6a85796e-de9a-002e-9695-01b9be4344c6 [Follower]" leader= testing.go:149: server-6a85796e-de9a-002e-9695-01b9be4344c6: 2023-08-02T15:13:00.271Z [INFO] server-6a85796e-de9a-002e-9695-01b9be4344c6: entering follower state: follower="Node at 6a85796e-de9a-002e-9695-01b9be4344c6 [Follower]" leader= testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.324Z [WARN] server-781112b6-6781-e0d3-2e7a-fb39add31117: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.324Z [WARN] server-781112b6-6781-e0d3-2e7a-fb39add31117: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.324Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: entering candidate state: node="Node at 781112b6-6781-e0d3-2e7a-fb39add31117 [Candidate]" term=2 testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.324Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: entering candidate state: node="Node at 781112b6-6781-e0d3-2e7a-fb39add31117 [Candidate]" term=2 testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.324Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: election won: tally=2 testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.324Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: election won: tally=2 testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.325Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: entering leader state: leader="Node at 781112b6-6781-e0d3-2e7a-fb39add31117 [Leader]" testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.325Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: entering leader state: leader="Node at 781112b6-6781-e0d3-2e7a-fb39add31117 [Leader]" testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.325Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: added peer, starting replication: peer=server-80b68dcd-3486-a063-a2f4-474aebdf95c1 testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.325Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: added peer, starting replication: peer=server-80b68dcd-3486-a063-a2f4-474aebdf95c1 testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.325Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: added peer, starting replication: peer=server-6a85796e-de9a-002e-9695-01b9be4344c6 testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.325Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: added peer, starting replication: peer=server-6a85796e-de9a-002e-9695-01b9be4344c6 testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.325Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: pipelining replication: peer="{Voter server-6a85796e-de9a-002e-9695-01b9be4344c6 6a85796e-de9a-002e-9695-01b9be4344c6}" testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.325Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: pipelining replication: peer="{Voter server-6a85796e-de9a-002e-9695-01b9be4344c6 6a85796e-de9a-002e-9695-01b9be4344c6}" testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.325Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: pipelining replication: peer="{Voter server-80b68dcd-3486-a063-a2f4-474aebdf95c1 80b68dcd-3486-a063-a2f4-474aebdf95c1}" testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.325Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: pipelining replication: peer="{Voter server-80b68dcd-3486-a063-a2f4-474aebdf95c1 80b68dcd-3486-a063-a2f4-474aebdf95c1}" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:00.430Z [INFO] stable state for Follower reached at 2023-08-02 15:13:00.325446619 +0000 UTC m=+12.421382958 (2 nodes), 53.969814ms from start of poll, 54.752205ms from cluster start. Timeout at 2023-08-02 15:13:00.430498896 +0000 UTC m=+12.526435241, 105.052283ms after stability testing.go:149: cluster: 2023-08-02T15:13:00.430Z [INFO] stable state for Follower reached at 2023-08-02 15:13:00.325446619 +0000 UTC m=+12.421382958 (2 nodes), 53.969814ms from start of poll, 54.752205ms from cluster start. Timeout at 2023-08-02 15:13:00.430498896 +0000 UTC m=+12.526435241, 105.052283ms after stability testLoggerAdapter verbose: cluster: 2023-08-02T15:13:00.430Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:13:00.430Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-08-02T15:13:00.536Z [INFO] stable state for Leader reached at 2023-08-02 15:13:00.430685202 +0000 UTC m=+12.526621546 (1 nodes), 1.747µs from start of poll, 159.990793ms from cluster start. Timeout at 2023-08-02 15:13:00.536009117 +0000 UTC m=+12.631945456, 105.32391ms after stability testing.go:149: cluster: 2023-08-02T15:13:00.536Z [INFO] stable state for Leader reached at 2023-08-02 15:13:00.430685202 +0000 UTC m=+12.526621546 (1 nodes), 1.747µs from start of poll, 159.990793ms from cluster start. Timeout at 2023-08-02 15:13:00.536009117 +0000 UTC m=+12.631945456, 105.32391ms after stability testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.536Z [ERROR] server-781112b6-6781-e0d3-2e7a-fb39add31117: failed to pipeline appendEntries: peer="{Voter server-80b68dcd-3486-a063-a2f4-474aebdf95c1 80b68dcd-3486-a063-a2f4-474aebdf95c1}" error="append pipeline closed" testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.536Z [ERROR] server-781112b6-6781-e0d3-2e7a-fb39add31117: failed to pipeline appendEntries: peer="{Voter server-80b68dcd-3486-a063-a2f4-474aebdf95c1 80b68dcd-3486-a063-a2f4-474aebdf95c1}" error="append pipeline closed" testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.536Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: aborting pipeline replication: peer="{Voter server-80b68dcd-3486-a063-a2f4-474aebdf95c1 80b68dcd-3486-a063-a2f4-474aebdf95c1}" testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.536Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: aborting pipeline replication: peer="{Voter server-80b68dcd-3486-a063-a2f4-474aebdf95c1 80b68dcd-3486-a063-a2f4-474aebdf95c1}" testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.536Z [ERROR] server-781112b6-6781-e0d3-2e7a-fb39add31117: failed to appendEntries to: peer="{Voter server-80b68dcd-3486-a063-a2f4-474aebdf95c1 80b68dcd-3486-a063-a2f4-474aebdf95c1}" error="failed to connect to peer: 80b68dcd-3486-a063-a2f4-474aebdf95c1" testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.536Z [ERROR] server-781112b6-6781-e0d3-2e7a-fb39add31117: failed to appendEntries to: peer="{Voter server-80b68dcd-3486-a063-a2f4-474aebdf95c1 80b68dcd-3486-a063-a2f4-474aebdf95c1}" error="failed to connect to peer: 80b68dcd-3486-a063-a2f4-474aebdf95c1" raft_test.go:1362: [INFO] Finished apply without behind follower testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.538Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: starting snapshot up to: index=102 testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.538Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: starting snapshot up to: index=102 testing.go:153: 2023-08-02T15:13:00.539Z [INFO] creating new snapshot: path=/tmp/raft796835311/snapshots/2-102-1690989180539.tmp testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.539Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: compacting logs: from=1 to=92 testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.539Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.539Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: snapshot complete up to: index=102 testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.539Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: snapshot complete up to: index=102 testLoggerAdapter verbose: server-80b68dcd-3486-a063-a2f4-474aebdf95c1: 2023-08-02T15:13:00.539Z [ERROR] server-80b68dcd-3486-a063-a2f4-474aebdf95c1: failed to take snapshot: error="nothing new to snapshot" testing.go:149: server-80b68dcd-3486-a063-a2f4-474aebdf95c1: 2023-08-02T15:13:00.539Z [ERROR] server-80b68dcd-3486-a063-a2f4-474aebdf95c1: failed to take snapshot: error="nothing new to snapshot" testLoggerAdapter verbose: server-6a85796e-de9a-002e-9695-01b9be4344c6: 2023-08-02T15:13:00.539Z [INFO] server-6a85796e-de9a-002e-9695-01b9be4344c6: starting snapshot up to: index=88 testing.go:149: server-6a85796e-de9a-002e-9695-01b9be4344c6: 2023-08-02T15:13:00.539Z [INFO] server-6a85796e-de9a-002e-9695-01b9be4344c6: starting snapshot up to: index=88 testing.go:153: 2023-08-02T15:13:00.539Z [INFO] creating new snapshot: path=/tmp/raft471634067/snapshots/2-88-1690989180539.tmp testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.540Z [ERROR] server-781112b6-6781-e0d3-2e7a-fb39add31117: failed to heartbeat to: peer=80b68dcd-3486-a063-a2f4-474aebdf95c1 error="failed to connect to peer: 80b68dcd-3486-a063-a2f4-474aebdf95c1" testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.540Z [ERROR] server-781112b6-6781-e0d3-2e7a-fb39add31117: failed to heartbeat to: peer=80b68dcd-3486-a063-a2f4-474aebdf95c1 error="failed to connect to peer: 80b68dcd-3486-a063-a2f4-474aebdf95c1" testLoggerAdapter verbose: server-6a85796e-de9a-002e-9695-01b9be4344c6: 2023-08-02T15:13:00.541Z [INFO] server-6a85796e-de9a-002e-9695-01b9be4344c6: compacting logs: from=1 to=88 testing.go:149: server-6a85796e-de9a-002e-9695-01b9be4344c6: 2023-08-02T15:13:00.541Z [INFO] server-6a85796e-de9a-002e-9695-01b9be4344c6: compacting logs: from=1 to=88 testLoggerAdapter verbose: server-6a85796e-de9a-002e-9695-01b9be4344c6: 2023-08-02T15:13:00.541Z [INFO] server-6a85796e-de9a-002e-9695-01b9be4344c6: snapshot complete up to: index=88 testing.go:149: server-6a85796e-de9a-002e-9695-01b9be4344c6: 2023-08-02T15:13:00.541Z [INFO] server-6a85796e-de9a-002e-9695-01b9be4344c6: snapshot complete up to: index=88 raft_test.go:1383: [INFO] Finished apply without behind follower testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.550Z [ERROR] server-781112b6-6781-e0d3-2e7a-fb39add31117: failed to get log: index=2 error="log not found" testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.550Z [ERROR] server-781112b6-6781-e0d3-2e7a-fb39add31117: failed to get log: index=2 error="log not found" testing.go:153: 2023-08-02T15:13:00.551Z [INFO] creating new snapshot: path=/tmp/raft864382969/snapshots/2-102-1690989180551.tmp testLoggerAdapter verbose: server-80b68dcd-3486-a063-a2f4-474aebdf95c1: 2023-08-02T15:13:00.551Z [INFO] server-80b68dcd-3486-a063-a2f4-474aebdf95c1: copied to local snapshot: bytes=693 testing.go:149: server-80b68dcd-3486-a063-a2f4-474aebdf95c1: 2023-08-02T15:13:00.551Z [INFO] server-80b68dcd-3486-a063-a2f4-474aebdf95c1: copied to local snapshot: bytes=693 testLoggerAdapter verbose: server-80b68dcd-3486-a063-a2f4-474aebdf95c1: 2023-08-02T15:13:00.551Z [INFO] server-80b68dcd-3486-a063-a2f4-474aebdf95c1: Installed remote snapshot testing.go:149: server-80b68dcd-3486-a063-a2f4-474aebdf95c1: 2023-08-02T15:13:00.551Z [INFO] server-80b68dcd-3486-a063-a2f4-474aebdf95c1: Installed remote snapshot testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.552Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: pipelining replication: peer="{Voter server-80b68dcd-3486-a063-a2f4-474aebdf95c1 80b68dcd-3486-a063-a2f4-474aebdf95c1}" testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.552Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: pipelining replication: peer="{Voter server-80b68dcd-3486-a063-a2f4-474aebdf95c1 80b68dcd-3486-a063-a2f4-474aebdf95c1}" testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.561Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: aborting pipeline replication: peer="{Voter server-6a85796e-de9a-002e-9695-01b9be4344c6 6a85796e-de9a-002e-9695-01b9be4344c6}" testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.561Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: aborting pipeline replication: peer="{Voter server-6a85796e-de9a-002e-9695-01b9be4344c6 6a85796e-de9a-002e-9695-01b9be4344c6}" testLoggerAdapter verbose: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.561Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: aborting pipeline replication: peer="{Voter server-80b68dcd-3486-a063-a2f4-474aebdf95c1 80b68dcd-3486-a063-a2f4-474aebdf95c1}" testing.go:149: server-781112b6-6781-e0d3-2e7a-fb39add31117: 2023-08-02T15:13:00.561Z [INFO] server-781112b6-6781-e0d3-2e7a-fb39add31117: aborting pipeline replication: peer="{Voter server-80b68dcd-3486-a063-a2f4-474aebdf95c1 80b68dcd-3486-a063-a2f4-474aebdf95c1}" --- PASS: TestRaft_SendSnapshotAndLogsFollower (0.29s) === RUN TestRaft_ReJoinFollower testLoggerAdapter verbose: server-2eb76803-14cb-629a-acf5-13a64eb4363d: 2023-08-02T15:13:00.562Z [INFO] server-2eb76803-14cb-629a-acf5-13a64eb4363d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2eb76803-14cb-629a-acf5-13a64eb4363d Address:2eb76803-14cb-629a-acf5-13a64eb4363d} {Suffrage:Voter ID:server-52547b59-7eae-c603-5b0b-827290183393 Address:52547b59-7eae-c603-5b0b-827290183393} {Suffrage:Voter ID:server-121f24e6-86e6-79bf-62ed-7d3216c32cfb Address:121f24e6-86e6-79bf-62ed-7d3216c32cfb}]" testing.go:149: server-2eb76803-14cb-629a-acf5-13a64eb4363d: 2023-08-02T15:13:00.562Z [INFO] server-2eb76803-14cb-629a-acf5-13a64eb4363d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2eb76803-14cb-629a-acf5-13a64eb4363d Address:2eb76803-14cb-629a-acf5-13a64eb4363d} {Suffrage:Voter ID:server-52547b59-7eae-c603-5b0b-827290183393 Address:52547b59-7eae-c603-5b0b-827290183393} {Suffrage:Voter ID:server-121f24e6-86e6-79bf-62ed-7d3216c32cfb Address:121f24e6-86e6-79bf-62ed-7d3216c32cfb}]" testLoggerAdapter verbose: server-52547b59-7eae-c603-5b0b-827290183393: 2023-08-02T15:13:00.562Z [INFO] server-52547b59-7eae-c603-5b0b-827290183393: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2eb76803-14cb-629a-acf5-13a64eb4363d Address:2eb76803-14cb-629a-acf5-13a64eb4363d} {Suffrage:Voter ID:server-52547b59-7eae-c603-5b0b-827290183393 Address:52547b59-7eae-c603-5b0b-827290183393} {Suffrage:Voter ID:server-121f24e6-86e6-79bf-62ed-7d3216c32cfb Address:121f24e6-86e6-79bf-62ed-7d3216c32cfb}]" testing.go:149: server-52547b59-7eae-c603-5b0b-827290183393: 2023-08-02T15:13:00.562Z [INFO] server-52547b59-7eae-c603-5b0b-827290183393: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2eb76803-14cb-629a-acf5-13a64eb4363d Address:2eb76803-14cb-629a-acf5-13a64eb4363d} {Suffrage:Voter ID:server-52547b59-7eae-c603-5b0b-827290183393 Address:52547b59-7eae-c603-5b0b-827290183393} {Suffrage:Voter ID:server-121f24e6-86e6-79bf-62ed-7d3216c32cfb Address:121f24e6-86e6-79bf-62ed-7d3216c32cfb}]" testLoggerAdapter verbose: server-2eb76803-14cb-629a-acf5-13a64eb4363d: 2023-08-02T15:13:00.562Z [INFO] server-2eb76803-14cb-629a-acf5-13a64eb4363d: entering follower state: follower="Node at 2eb76803-14cb-629a-acf5-13a64eb4363d [Follower]" leader= testing.go:149: server-2eb76803-14cb-629a-acf5-13a64eb4363d: 2023-08-02T15:13:00.562Z [INFO] server-2eb76803-14cb-629a-acf5-13a64eb4363d: entering follower state: follower="Node at 2eb76803-14cb-629a-acf5-13a64eb4363d [Follower]" leader= testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.563Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2eb76803-14cb-629a-acf5-13a64eb4363d Address:2eb76803-14cb-629a-acf5-13a64eb4363d} {Suffrage:Voter ID:server-52547b59-7eae-c603-5b0b-827290183393 Address:52547b59-7eae-c603-5b0b-827290183393} {Suffrage:Voter ID:server-121f24e6-86e6-79bf-62ed-7d3216c32cfb Address:121f24e6-86e6-79bf-62ed-7d3216c32cfb}]" testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.563Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2eb76803-14cb-629a-acf5-13a64eb4363d Address:2eb76803-14cb-629a-acf5-13a64eb4363d} {Suffrage:Voter ID:server-52547b59-7eae-c603-5b0b-827290183393 Address:52547b59-7eae-c603-5b0b-827290183393} {Suffrage:Voter ID:server-121f24e6-86e6-79bf-62ed-7d3216c32cfb Address:121f24e6-86e6-79bf-62ed-7d3216c32cfb}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:00.563Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:13:00.563Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-52547b59-7eae-c603-5b0b-827290183393: 2023-08-02T15:13:00.563Z [INFO] server-52547b59-7eae-c603-5b0b-827290183393: entering follower state: follower="Node at 52547b59-7eae-c603-5b0b-827290183393 [Follower]" leader= testing.go:149: server-52547b59-7eae-c603-5b0b-827290183393: 2023-08-02T15:13:00.563Z [INFO] server-52547b59-7eae-c603-5b0b-827290183393: entering follower state: follower="Node at 52547b59-7eae-c603-5b0b-827290183393 [Follower]" leader= testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.563Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: entering follower state: follower="Node at 121f24e6-86e6-79bf-62ed-7d3216c32cfb [Follower]" leader= testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.563Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: entering follower state: follower="Node at 121f24e6-86e6-79bf-62ed-7d3216c32cfb [Follower]" leader= testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.619Z [WARN] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.619Z [WARN] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.619Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: entering candidate state: node="Node at 121f24e6-86e6-79bf-62ed-7d3216c32cfb [Candidate]" term=2 testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.619Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: entering candidate state: node="Node at 121f24e6-86e6-79bf-62ed-7d3216c32cfb [Candidate]" term=2 testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.619Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: election won: tally=2 testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.619Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: election won: tally=2 testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.619Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: entering leader state: leader="Node at 121f24e6-86e6-79bf-62ed-7d3216c32cfb [Leader]" testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.619Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: entering leader state: leader="Node at 121f24e6-86e6-79bf-62ed-7d3216c32cfb [Leader]" testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.619Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: added peer, starting replication: peer=server-2eb76803-14cb-629a-acf5-13a64eb4363d testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.619Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: added peer, starting replication: peer=server-2eb76803-14cb-629a-acf5-13a64eb4363d testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.619Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: added peer, starting replication: peer=server-52547b59-7eae-c603-5b0b-827290183393 testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.619Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: added peer, starting replication: peer=server-52547b59-7eae-c603-5b0b-827290183393 testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.619Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: pipelining replication: peer="{Voter server-52547b59-7eae-c603-5b0b-827290183393 52547b59-7eae-c603-5b0b-827290183393}" testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.619Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: pipelining replication: peer="{Voter server-52547b59-7eae-c603-5b0b-827290183393 52547b59-7eae-c603-5b0b-827290183393}" testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.620Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: pipelining replication: peer="{Voter server-2eb76803-14cb-629a-acf5-13a64eb4363d 2eb76803-14cb-629a-acf5-13a64eb4363d}" testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.620Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: pipelining replication: peer="{Voter server-2eb76803-14cb-629a-acf5-13a64eb4363d 2eb76803-14cb-629a-acf5-13a64eb4363d}" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:00.725Z [INFO] stable state for Leader reached at 2023-08-02 15:13:00.620109742 +0000 UTC m=+12.716046069 (1 nodes), 56.967285ms from start of poll, 57.408444ms from cluster start. Timeout at 2023-08-02 15:13:00.72521219 +0000 UTC m=+12.821148523, 105.102454ms after stability testing.go:149: cluster: 2023-08-02T15:13:00.725Z [INFO] stable state for Leader reached at 2023-08-02 15:13:00.620109742 +0000 UTC m=+12.716046069 (1 nodes), 56.967285ms from start of poll, 57.408444ms from cluster start. Timeout at 2023-08-02 15:13:00.72521219 +0000 UTC m=+12.821148523, 105.102454ms after stability testLoggerAdapter verbose: cluster: 2023-08-02T15:13:00.730Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-08-02T15:13:00.730Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2023-08-02T15:13:00.835Z [INFO] stable state for Follower reached at 2023-08-02 15:13:00.730583341 +0000 UTC m=+12.826519669 (2 nodes), 2.917µs from start of poll, 167.882044ms from cluster start. Timeout at 2023-08-02 15:13:00.835626419 +0000 UTC m=+12.931562753, 105.043084ms after stability testing.go:149: cluster: 2023-08-02T15:13:00.835Z [INFO] stable state for Follower reached at 2023-08-02 15:13:00.730583341 +0000 UTC m=+12.826519669 (2 nodes), 2.917µs from start of poll, 167.882044ms from cluster start. Timeout at 2023-08-02 15:13:00.835626419 +0000 UTC m=+12.931562753, 105.043084ms after stability testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.835Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: updating configuration: command=RemoveServer server-id=server-2eb76803-14cb-629a-acf5-13a64eb4363d server-addr= servers="[{Suffrage:Voter ID:server-52547b59-7eae-c603-5b0b-827290183393 Address:52547b59-7eae-c603-5b0b-827290183393} {Suffrage:Voter ID:server-121f24e6-86e6-79bf-62ed-7d3216c32cfb Address:121f24e6-86e6-79bf-62ed-7d3216c32cfb}]" testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.835Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: updating configuration: command=RemoveServer server-id=server-2eb76803-14cb-629a-acf5-13a64eb4363d server-addr= servers="[{Suffrage:Voter ID:server-52547b59-7eae-c603-5b0b-827290183393 Address:52547b59-7eae-c603-5b0b-827290183393} {Suffrage:Voter ID:server-121f24e6-86e6-79bf-62ed-7d3216c32cfb Address:121f24e6-86e6-79bf-62ed-7d3216c32cfb}]" testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.835Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: removed peer, stopping replication: peer=server-2eb76803-14cb-629a-acf5-13a64eb4363d last-index=3 testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.835Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: removed peer, stopping replication: peer=server-2eb76803-14cb-629a-acf5-13a64eb4363d last-index=3 testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.835Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: aborting pipeline replication: peer="{Voter server-2eb76803-14cb-629a-acf5-13a64eb4363d 2eb76803-14cb-629a-acf5-13a64eb4363d}" testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.835Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: aborting pipeline replication: peer="{Voter server-2eb76803-14cb-629a-acf5-13a64eb4363d 2eb76803-14cb-629a-acf5-13a64eb4363d}" testLoggerAdapter verbose: server-2eb76803-14cb-629a-acf5-13a64eb4363d: 2023-08-02T15:13:00.921Z [WARN] server-2eb76803-14cb-629a-acf5-13a64eb4363d: heartbeat timeout reached, starting election: last-leader=121f24e6-86e6-79bf-62ed-7d3216c32cfb testing.go:149: server-2eb76803-14cb-629a-acf5-13a64eb4363d: 2023-08-02T15:13:00.921Z [WARN] server-2eb76803-14cb-629a-acf5-13a64eb4363d: heartbeat timeout reached, starting election: last-leader=121f24e6-86e6-79bf-62ed-7d3216c32cfb testLoggerAdapter verbose: server-2eb76803-14cb-629a-acf5-13a64eb4363d: 2023-08-02T15:13:00.922Z [INFO] server-2eb76803-14cb-629a-acf5-13a64eb4363d: entering candidate state: node="Node at 2eb76803-14cb-629a-acf5-13a64eb4363d [Candidate]" term=3 testing.go:149: server-2eb76803-14cb-629a-acf5-13a64eb4363d: 2023-08-02T15:13:00.922Z [INFO] server-2eb76803-14cb-629a-acf5-13a64eb4363d: entering candidate state: node="Node at 2eb76803-14cb-629a-acf5-13a64eb4363d [Candidate]" term=3 testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.922Z [WARN] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: rejecting vote request since we have a leader: from=2eb76803-14cb-629a-acf5-13a64eb4363d leader=121f24e6-86e6-79bf-62ed-7d3216c32cfb testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.922Z [WARN] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: rejecting vote request since we have a leader: from=2eb76803-14cb-629a-acf5-13a64eb4363d leader=121f24e6-86e6-79bf-62ed-7d3216c32cfb testLoggerAdapter verbose: server-52547b59-7eae-c603-5b0b-827290183393: 2023-08-02T15:13:00.922Z [WARN] server-52547b59-7eae-c603-5b0b-827290183393: rejecting vote request since we have a leader: from=2eb76803-14cb-629a-acf5-13a64eb4363d leader=121f24e6-86e6-79bf-62ed-7d3216c32cfb testing.go:149: server-52547b59-7eae-c603-5b0b-827290183393: 2023-08-02T15:13:00.922Z [WARN] server-52547b59-7eae-c603-5b0b-827290183393: rejecting vote request since we have a leader: from=2eb76803-14cb-629a-acf5-13a64eb4363d leader=121f24e6-86e6-79bf-62ed-7d3216c32cfb testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.941Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: updating configuration: command=AddStaging server-id=server-2eb76803-14cb-629a-acf5-13a64eb4363d server-addr=2eb76803-14cb-629a-acf5-13a64eb4363d servers="[{Suffrage:Voter ID:server-52547b59-7eae-c603-5b0b-827290183393 Address:52547b59-7eae-c603-5b0b-827290183393} {Suffrage:Voter ID:server-121f24e6-86e6-79bf-62ed-7d3216c32cfb Address:121f24e6-86e6-79bf-62ed-7d3216c32cfb} {Suffrage:Voter ID:server-2eb76803-14cb-629a-acf5-13a64eb4363d Address:2eb76803-14cb-629a-acf5-13a64eb4363d}]" testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.941Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: updating configuration: command=AddStaging server-id=server-2eb76803-14cb-629a-acf5-13a64eb4363d server-addr=2eb76803-14cb-629a-acf5-13a64eb4363d servers="[{Suffrage:Voter ID:server-52547b59-7eae-c603-5b0b-827290183393 Address:52547b59-7eae-c603-5b0b-827290183393} {Suffrage:Voter ID:server-121f24e6-86e6-79bf-62ed-7d3216c32cfb Address:121f24e6-86e6-79bf-62ed-7d3216c32cfb} {Suffrage:Voter ID:server-2eb76803-14cb-629a-acf5-13a64eb4363d Address:2eb76803-14cb-629a-acf5-13a64eb4363d}]" testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.941Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: added peer, starting replication: peer=server-2eb76803-14cb-629a-acf5-13a64eb4363d testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.941Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: added peer, starting replication: peer=server-2eb76803-14cb-629a-acf5-13a64eb4363d testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.941Z [ERROR] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: peer has newer term, stopping replication: peer="{Voter server-2eb76803-14cb-629a-acf5-13a64eb4363d 2eb76803-14cb-629a-acf5-13a64eb4363d}" testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.941Z [ERROR] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: peer has newer term, stopping replication: peer="{Voter server-2eb76803-14cb-629a-acf5-13a64eb4363d 2eb76803-14cb-629a-acf5-13a64eb4363d}" testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.941Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: entering follower state: follower="Node at 121f24e6-86e6-79bf-62ed-7d3216c32cfb [Follower]" leader= testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.941Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: entering follower state: follower="Node at 121f24e6-86e6-79bf-62ed-7d3216c32cfb [Follower]" leader= testLoggerAdapter verbose: cluster: 2023-08-02T15:13:00.941Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:13:00.941Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.941Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: aborting pipeline replication: peer="{Voter server-52547b59-7eae-c603-5b0b-827290183393 52547b59-7eae-c603-5b0b-827290183393}" testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:00.941Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: aborting pipeline replication: peer="{Voter server-52547b59-7eae-c603-5b0b-827290183393 52547b59-7eae-c603-5b0b-827290183393}" testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.006Z [WARN] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.006Z [WARN] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.006Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: entering candidate state: node="Node at 121f24e6-86e6-79bf-62ed-7d3216c32cfb [Candidate]" term=3 testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.006Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: entering candidate state: node="Node at 121f24e6-86e6-79bf-62ed-7d3216c32cfb [Candidate]" term=3 testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.006Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: election won: tally=2 testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.006Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: election won: tally=2 testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.006Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: entering leader state: leader="Node at 121f24e6-86e6-79bf-62ed-7d3216c32cfb [Leader]" testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.006Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: entering leader state: leader="Node at 121f24e6-86e6-79bf-62ed-7d3216c32cfb [Leader]" testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.006Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: added peer, starting replication: peer=server-52547b59-7eae-c603-5b0b-827290183393 testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.006Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: added peer, starting replication: peer=server-52547b59-7eae-c603-5b0b-827290183393 testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.006Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: added peer, starting replication: peer=server-2eb76803-14cb-629a-acf5-13a64eb4363d testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.006Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: added peer, starting replication: peer=server-2eb76803-14cb-629a-acf5-13a64eb4363d testLoggerAdapter verbose: server-2eb76803-14cb-629a-acf5-13a64eb4363d: 2023-08-02T15:13:01.007Z [WARN] server-2eb76803-14cb-629a-acf5-13a64eb4363d: failed to get previous log: previous-index=4 last-index=3 error="log not found" testing.go:149: server-2eb76803-14cb-629a-acf5-13a64eb4363d: 2023-08-02T15:13:01.007Z [WARN] server-2eb76803-14cb-629a-acf5-13a64eb4363d: failed to get previous log: previous-index=4 last-index=3 error="log not found" testLoggerAdapter verbose: server-2eb76803-14cb-629a-acf5-13a64eb4363d: 2023-08-02T15:13:01.007Z [INFO] server-2eb76803-14cb-629a-acf5-13a64eb4363d: entering follower state: follower="Node at 2eb76803-14cb-629a-acf5-13a64eb4363d [Follower]" leader=121f24e6-86e6-79bf-62ed-7d3216c32cfb testing.go:149: server-2eb76803-14cb-629a-acf5-13a64eb4363d: 2023-08-02T15:13:01.007Z [INFO] server-2eb76803-14cb-629a-acf5-13a64eb4363d: entering follower state: follower="Node at 2eb76803-14cb-629a-acf5-13a64eb4363d [Follower]" leader=121f24e6-86e6-79bf-62ed-7d3216c32cfb testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.007Z [WARN] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: appendEntries rejected, sending older logs: peer="{Voter server-2eb76803-14cb-629a-acf5-13a64eb4363d 2eb76803-14cb-629a-acf5-13a64eb4363d}" next=4 testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.007Z [WARN] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: appendEntries rejected, sending older logs: peer="{Voter server-2eb76803-14cb-629a-acf5-13a64eb4363d 2eb76803-14cb-629a-acf5-13a64eb4363d}" next=4 testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.007Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: pipelining replication: peer="{Voter server-52547b59-7eae-c603-5b0b-827290183393 52547b59-7eae-c603-5b0b-827290183393}" testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.007Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: pipelining replication: peer="{Voter server-52547b59-7eae-c603-5b0b-827290183393 52547b59-7eae-c603-5b0b-827290183393}" testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.007Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: pipelining replication: peer="{Voter server-2eb76803-14cb-629a-acf5-13a64eb4363d 2eb76803-14cb-629a-acf5-13a64eb4363d}" testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.007Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: pipelining replication: peer="{Voter server-2eb76803-14cb-629a-acf5-13a64eb4363d 2eb76803-14cb-629a-acf5-13a64eb4363d}" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:01.112Z [INFO] stable state for Leader reached at 2023-08-02 15:13:01.007107292 +0000 UTC m=+13.103043620 (1 nodes), 65.281287ms from start of poll, 444.405995ms from cluster start. Timeout at 2023-08-02 15:13:01.112207411 +0000 UTC m=+13.208143750, 105.10013ms after stability testing.go:149: cluster: 2023-08-02T15:13:01.112Z [INFO] stable state for Leader reached at 2023-08-02 15:13:01.007107292 +0000 UTC m=+13.103043620 (1 nodes), 65.281287ms from start of poll, 444.405995ms from cluster start. Timeout at 2023-08-02 15:13:01.112207411 +0000 UTC m=+13.208143750, 105.10013ms after stability testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.112Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: aborting pipeline replication: peer="{Voter server-2eb76803-14cb-629a-acf5-13a64eb4363d 2eb76803-14cb-629a-acf5-13a64eb4363d}" testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.112Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: aborting pipeline replication: peer="{Voter server-2eb76803-14cb-629a-acf5-13a64eb4363d 2eb76803-14cb-629a-acf5-13a64eb4363d}" testLoggerAdapter verbose: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.112Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: aborting pipeline replication: peer="{Voter server-52547b59-7eae-c603-5b0b-827290183393 52547b59-7eae-c603-5b0b-827290183393}" testing.go:149: server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: 2023-08-02T15:13:01.112Z [INFO] server-121f24e6-86e6-79bf-62ed-7d3216c32cfb: aborting pipeline replication: peer="{Voter server-52547b59-7eae-c603-5b0b-827290183393 52547b59-7eae-c603-5b0b-827290183393}" --- PASS: TestRaft_ReJoinFollower (0.55s) === RUN TestRaft_LeaderLeaseExpire testLoggerAdapter verbose: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.113Z [INFO] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-412e45c2-b294-66dc-33a9-f062c8ae3eea Address:412e45c2-b294-66dc-33a9-f062c8ae3eea} {Suffrage:Voter ID:server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f Address:f153d14b-dc13-123e-9a82-a2ac10ac8f3f}]" testing.go:149: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.113Z [INFO] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-412e45c2-b294-66dc-33a9-f062c8ae3eea Address:412e45c2-b294-66dc-33a9-f062c8ae3eea} {Suffrage:Voter ID:server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f Address:f153d14b-dc13-123e-9a82-a2ac10ac8f3f}]" testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.113Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-412e45c2-b294-66dc-33a9-f062c8ae3eea Address:412e45c2-b294-66dc-33a9-f062c8ae3eea} {Suffrage:Voter ID:server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f Address:f153d14b-dc13-123e-9a82-a2ac10ac8f3f}]" testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.113Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-412e45c2-b294-66dc-33a9-f062c8ae3eea Address:412e45c2-b294-66dc-33a9-f062c8ae3eea} {Suffrage:Voter ID:server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f Address:f153d14b-dc13-123e-9a82-a2ac10ac8f3f}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:01.113Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:13:01.113Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.113Z [INFO] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: entering follower state: follower="Node at 412e45c2-b294-66dc-33a9-f062c8ae3eea [Follower]" leader= testing.go:149: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.113Z [INFO] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: entering follower state: follower="Node at 412e45c2-b294-66dc-33a9-f062c8ae3eea [Follower]" leader= testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.113Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: entering follower state: follower="Node at f153d14b-dc13-123e-9a82-a2ac10ac8f3f [Follower]" leader= testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.113Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: entering follower state: follower="Node at f153d14b-dc13-123e-9a82-a2ac10ac8f3f [Follower]" leader= testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.183Z [WARN] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.183Z [WARN] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.183Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: entering candidate state: node="Node at f153d14b-dc13-123e-9a82-a2ac10ac8f3f [Candidate]" term=2 testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.183Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: entering candidate state: node="Node at f153d14b-dc13-123e-9a82-a2ac10ac8f3f [Candidate]" term=2 testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.183Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: election won: tally=2 testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.183Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: election won: tally=2 testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.183Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: entering leader state: leader="Node at f153d14b-dc13-123e-9a82-a2ac10ac8f3f [Leader]" testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.183Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: entering leader state: leader="Node at f153d14b-dc13-123e-9a82-a2ac10ac8f3f [Leader]" testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.183Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: added peer, starting replication: peer=server-412e45c2-b294-66dc-33a9-f062c8ae3eea testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.183Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: added peer, starting replication: peer=server-412e45c2-b294-66dc-33a9-f062c8ae3eea testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.183Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: pipelining replication: peer="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.183Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: pipelining replication: peer="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:01.288Z [INFO] stable state for Leader reached at 2023-08-02 15:13:01.18362594 +0000 UTC m=+13.279562275 (1 nodes), 69.965836ms from start of poll, 70.345877ms from cluster start. Timeout at 2023-08-02 15:13:01.288766989 +0000 UTC m=+13.384703343, 105.141068ms after stability testing.go:149: cluster: 2023-08-02T15:13:01.288Z [INFO] stable state for Leader reached at 2023-08-02 15:13:01.18362594 +0000 UTC m=+13.279562275 (1 nodes), 69.965836ms from start of poll, 70.345877ms from cluster start. Timeout at 2023-08-02 15:13:01.288766989 +0000 UTC m=+13.384703343, 105.141068ms after stability testLoggerAdapter verbose: cluster: 2023-08-02T15:13:01.294Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-08-02T15:13:01.294Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2023-08-02T15:13:01.399Z [INFO] stable state for Follower reached at 2023-08-02 15:13:01.294225947 +0000 UTC m=+13.390162282 (1 nodes), 1.592µs from start of poll, 180.945884ms from cluster start. Timeout at 2023-08-02 15:13:01.399333228 +0000 UTC m=+13.495269575, 105.107293ms after stability testing.go:149: cluster: 2023-08-02T15:13:01.399Z [INFO] stable state for Follower reached at 2023-08-02 15:13:01.294225947 +0000 UTC m=+13.390162282 (1 nodes), 1.592µs from start of poll, 180.945884ms from cluster start. Timeout at 2023-08-02 15:13:01.399333228 +0000 UTC m=+13.495269575, 105.107293ms after stability raft_test.go:1492: [INFO] Disconnecting Node at 412e45c2-b294-66dc-33a9-f062c8ae3eea [Follower] testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.400Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to pipeline appendEntries: peer="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" error="append pipeline closed" testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.400Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to pipeline appendEntries: peer="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" error="append pipeline closed" testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.400Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: aborting pipeline replication: peer="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.400Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: aborting pipeline replication: peer="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.403Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to heartbeat to: peer=412e45c2-b294-66dc-33a9-f062c8ae3eea error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.403Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to heartbeat to: peer=412e45c2-b294-66dc-33a9-f062c8ae3eea error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.409Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to appendEntries to: peer="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.409Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to appendEntries to: peer="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.423Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to heartbeat to: peer=412e45c2-b294-66dc-33a9-f062c8ae3eea error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.423Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to heartbeat to: peer=412e45c2-b294-66dc-33a9-f062c8ae3eea error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.430Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to appendEntries to: peer="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.430Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to appendEntries to: peer="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.439Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to heartbeat to: peer=412e45c2-b294-66dc-33a9-f062c8ae3eea error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.439Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to heartbeat to: peer=412e45c2-b294-66dc-33a9-f062c8ae3eea error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.444Z [WARN] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to contact: server-id=server-412e45c2-b294-66dc-33a9-f062c8ae3eea time=50.11397ms testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.444Z [WARN] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to contact: server-id=server-412e45c2-b294-66dc-33a9-f062c8ae3eea time=50.11397ms testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.444Z [WARN] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to contact quorum of nodes, stepping down testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.444Z [WARN] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to contact quorum of nodes, stepping down testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.445Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: entering follower state: follower="Node at f153d14b-dc13-123e-9a82-a2ac10ac8f3f [Follower]" leader= testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.445Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: entering follower state: follower="Node at f153d14b-dc13-123e-9a82-a2ac10ac8f3f [Follower]" leader= testLoggerAdapter verbose: cluster: 2023-08-02T15:13:01.445Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:13:01.445Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.451Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to appendEntries to: peer="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.451Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to appendEntries to: peer="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" testLoggerAdapter verbose: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.510Z [WARN] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: heartbeat timeout reached, starting election: last-leader=f153d14b-dc13-123e-9a82-a2ac10ac8f3f testing.go:149: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.510Z [WARN] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: heartbeat timeout reached, starting election: last-leader=f153d14b-dc13-123e-9a82-a2ac10ac8f3f testLoggerAdapter verbose: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.510Z [INFO] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: entering candidate state: node="Node at 412e45c2-b294-66dc-33a9-f062c8ae3eea [Candidate]" term=3 testing.go:149: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.510Z [INFO] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: entering candidate state: node="Node at 412e45c2-b294-66dc-33a9-f062c8ae3eea [Candidate]" term=3 testLoggerAdapter verbose: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.510Z [ERROR] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: failed to make requestVote RPC: target="{Voter server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f f153d14b-dc13-123e-9a82-a2ac10ac8f3f}" error="failed to connect to peer: f153d14b-dc13-123e-9a82-a2ac10ac8f3f" testing.go:149: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.510Z [ERROR] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: failed to make requestVote RPC: target="{Voter server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f f153d14b-dc13-123e-9a82-a2ac10ac8f3f}" error="failed to connect to peer: f153d14b-dc13-123e-9a82-a2ac10ac8f3f" testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.541Z [WARN] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.541Z [WARN] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.542Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: entering candidate state: node="Node at f153d14b-dc13-123e-9a82-a2ac10ac8f3f [Candidate]" term=3 testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.542Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: entering candidate state: node="Node at f153d14b-dc13-123e-9a82-a2ac10ac8f3f [Candidate]" term=3 testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.542Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to make requestVote RPC: target="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.542Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to make requestVote RPC: target="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" testLoggerAdapter verbose: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.598Z [WARN] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: Election timeout reached, restarting election testing.go:149: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.598Z [WARN] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: Election timeout reached, restarting election testLoggerAdapter verbose: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.599Z [INFO] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: entering candidate state: node="Node at 412e45c2-b294-66dc-33a9-f062c8ae3eea [Candidate]" term=4 testing.go:149: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.599Z [INFO] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: entering candidate state: node="Node at 412e45c2-b294-66dc-33a9-f062c8ae3eea [Candidate]" term=4 testLoggerAdapter verbose: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.599Z [ERROR] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: failed to make requestVote RPC: target="{Voter server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f f153d14b-dc13-123e-9a82-a2ac10ac8f3f}" error="failed to connect to peer: f153d14b-dc13-123e-9a82-a2ac10ac8f3f" testing.go:149: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.599Z [ERROR] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: failed to make requestVote RPC: target="{Voter server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f f153d14b-dc13-123e-9a82-a2ac10ac8f3f}" error="failed to connect to peer: f153d14b-dc13-123e-9a82-a2ac10ac8f3f" testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.612Z [WARN] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: Election timeout reached, restarting election testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.612Z [WARN] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: Election timeout reached, restarting election testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.612Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: entering candidate state: node="Node at f153d14b-dc13-123e-9a82-a2ac10ac8f3f [Candidate]" term=4 testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.612Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: entering candidate state: node="Node at f153d14b-dc13-123e-9a82-a2ac10ac8f3f [Candidate]" term=4 testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.613Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to make requestVote RPC: target="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.613Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to make requestVote RPC: target="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:01.647Z [INFO] stable state for Leader reached at 2023-08-02 15:13:01.542220035 +0000 UTC m=+13.638156380 (0 nodes), 97.065321ms from start of poll, 428.939982ms from cluster start. Timeout at 2023-08-02 15:13:01.647336274 +0000 UTC m=+13.743272616, 105.116236ms after stability testing.go:149: cluster: 2023-08-02T15:13:01.647Z [INFO] stable state for Leader reached at 2023-08-02 15:13:01.542220035 +0000 UTC m=+13.638156380 (0 nodes), 97.065321ms from start of poll, 428.939982ms from cluster start. Timeout at 2023-08-02 15:13:01.647336274 +0000 UTC m=+13.743272616, 105.116236ms after stability testLoggerAdapter verbose: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.653Z [WARN] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: Election timeout reached, restarting election testing.go:149: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.653Z [WARN] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: Election timeout reached, restarting election testLoggerAdapter verbose: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.653Z [INFO] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: entering candidate state: node="Node at 412e45c2-b294-66dc-33a9-f062c8ae3eea [Candidate]" term=5 testing.go:149: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.653Z [INFO] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: entering candidate state: node="Node at 412e45c2-b294-66dc-33a9-f062c8ae3eea [Candidate]" term=5 testLoggerAdapter verbose: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.653Z [ERROR] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: failed to make requestVote RPC: target="{Voter server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f f153d14b-dc13-123e-9a82-a2ac10ac8f3f}" error="failed to connect to peer: f153d14b-dc13-123e-9a82-a2ac10ac8f3f" testing.go:149: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.653Z [ERROR] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: failed to make requestVote RPC: target="{Voter server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f f153d14b-dc13-123e-9a82-a2ac10ac8f3f}" error="failed to connect to peer: f153d14b-dc13-123e-9a82-a2ac10ac8f3f" testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.681Z [WARN] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: Election timeout reached, restarting election testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.681Z [WARN] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: Election timeout reached, restarting election testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.681Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: entering candidate state: node="Node at f153d14b-dc13-123e-9a82-a2ac10ac8f3f [Candidate]" term=5 testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.681Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: entering candidate state: node="Node at f153d14b-dc13-123e-9a82-a2ac10ac8f3f [Candidate]" term=5 testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.682Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to make requestVote RPC: target="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.682Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to make requestVote RPC: target="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" testLoggerAdapter verbose: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.720Z [WARN] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: Election timeout reached, restarting election testing.go:149: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.720Z [WARN] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: Election timeout reached, restarting election testLoggerAdapter verbose: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.720Z [INFO] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: entering candidate state: node="Node at 412e45c2-b294-66dc-33a9-f062c8ae3eea [Candidate]" term=6 testing.go:149: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.720Z [INFO] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: entering candidate state: node="Node at 412e45c2-b294-66dc-33a9-f062c8ae3eea [Candidate]" term=6 testLoggerAdapter verbose: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.720Z [ERROR] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: failed to make requestVote RPC: target="{Voter server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f f153d14b-dc13-123e-9a82-a2ac10ac8f3f}" error="failed to connect to peer: f153d14b-dc13-123e-9a82-a2ac10ac8f3f" testing.go:149: server-412e45c2-b294-66dc-33a9-f062c8ae3eea: 2023-08-02T15:13:01.720Z [ERROR] server-412e45c2-b294-66dc-33a9-f062c8ae3eea: failed to make requestVote RPC: target="{Voter server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f f153d14b-dc13-123e-9a82-a2ac10ac8f3f}" error="failed to connect to peer: f153d14b-dc13-123e-9a82-a2ac10ac8f3f" testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.738Z [WARN] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: Election timeout reached, restarting election testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.738Z [WARN] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: Election timeout reached, restarting election testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.738Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: entering candidate state: node="Node at f153d14b-dc13-123e-9a82-a2ac10ac8f3f [Candidate]" term=6 testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.738Z [INFO] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: entering candidate state: node="Node at f153d14b-dc13-123e-9a82-a2ac10ac8f3f [Candidate]" term=6 testLoggerAdapter verbose: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.738Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to make requestVote RPC: target="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" testing.go:149: server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: 2023-08-02T15:13:01.738Z [ERROR] server-f153d14b-dc13-123e-9a82-a2ac10ac8f3f: failed to make requestVote RPC: target="{Voter server-412e45c2-b294-66dc-33a9-f062c8ae3eea 412e45c2-b294-66dc-33a9-f062c8ae3eea}" error="failed to connect to peer: 412e45c2-b294-66dc-33a9-f062c8ae3eea" --- PASS: TestRaft_LeaderLeaseExpire (0.64s) === RUN TestRaft_Barrier testLoggerAdapter verbose: server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5: 2023-08-02T15:13:01.754Z [INFO] server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5 Address:fcaa246d-0e1b-6eba-1533-44e9d7ec79f5} {Suffrage:Voter ID:server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8 Address:7f9f7a1a-cb9b-484f-e407-7713638e3ab8} {Suffrage:Voter ID:server-948f5509-fc1f-3029-1799-1b3b8018197f Address:948f5509-fc1f-3029-1799-1b3b8018197f}]" testing.go:149: server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5: 2023-08-02T15:13:01.754Z [INFO] server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5 Address:fcaa246d-0e1b-6eba-1533-44e9d7ec79f5} {Suffrage:Voter ID:server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8 Address:7f9f7a1a-cb9b-484f-e407-7713638e3ab8} {Suffrage:Voter ID:server-948f5509-fc1f-3029-1799-1b3b8018197f Address:948f5509-fc1f-3029-1799-1b3b8018197f}]" testLoggerAdapter verbose: server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8: 2023-08-02T15:13:01.754Z [INFO] server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5 Address:fcaa246d-0e1b-6eba-1533-44e9d7ec79f5} {Suffrage:Voter ID:server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8 Address:7f9f7a1a-cb9b-484f-e407-7713638e3ab8} {Suffrage:Voter ID:server-948f5509-fc1f-3029-1799-1b3b8018197f Address:948f5509-fc1f-3029-1799-1b3b8018197f}]" testing.go:149: server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8: 2023-08-02T15:13:01.754Z [INFO] server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5 Address:fcaa246d-0e1b-6eba-1533-44e9d7ec79f5} {Suffrage:Voter ID:server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8 Address:7f9f7a1a-cb9b-484f-e407-7713638e3ab8} {Suffrage:Voter ID:server-948f5509-fc1f-3029-1799-1b3b8018197f Address:948f5509-fc1f-3029-1799-1b3b8018197f}]" testLoggerAdapter verbose: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.754Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5 Address:fcaa246d-0e1b-6eba-1533-44e9d7ec79f5} {Suffrage:Voter ID:server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8 Address:7f9f7a1a-cb9b-484f-e407-7713638e3ab8} {Suffrage:Voter ID:server-948f5509-fc1f-3029-1799-1b3b8018197f Address:948f5509-fc1f-3029-1799-1b3b8018197f}]" testing.go:149: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.754Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5 Address:fcaa246d-0e1b-6eba-1533-44e9d7ec79f5} {Suffrage:Voter ID:server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8 Address:7f9f7a1a-cb9b-484f-e407-7713638e3ab8} {Suffrage:Voter ID:server-948f5509-fc1f-3029-1799-1b3b8018197f Address:948f5509-fc1f-3029-1799-1b3b8018197f}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:01.754Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8: 2023-08-02T15:13:01.754Z [INFO] server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8: entering follower state: follower="Node at 7f9f7a1a-cb9b-484f-e407-7713638e3ab8 [Follower]" leader= testing.go:149: server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8: 2023-08-02T15:13:01.754Z [INFO] server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8: entering follower state: follower="Node at 7f9f7a1a-cb9b-484f-e407-7713638e3ab8 [Follower]" leader= testLoggerAdapter verbose: server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5: 2023-08-02T15:13:01.754Z [INFO] server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5: entering follower state: follower="Node at fcaa246d-0e1b-6eba-1533-44e9d7ec79f5 [Follower]" leader= testing.go:149: server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5: 2023-08-02T15:13:01.754Z [INFO] server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5: entering follower state: follower="Node at fcaa246d-0e1b-6eba-1533-44e9d7ec79f5 [Follower]" leader= testLoggerAdapter verbose: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.754Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: entering follower state: follower="Node at 948f5509-fc1f-3029-1799-1b3b8018197f [Follower]" leader= testing.go:149: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.754Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: entering follower state: follower="Node at 948f5509-fc1f-3029-1799-1b3b8018197f [Follower]" leader= testing.go:149: cluster: 2023-08-02T15:13:01.754Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.810Z [WARN] server-948f5509-fc1f-3029-1799-1b3b8018197f: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.810Z [WARN] server-948f5509-fc1f-3029-1799-1b3b8018197f: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.810Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: entering candidate state: node="Node at 948f5509-fc1f-3029-1799-1b3b8018197f [Candidate]" term=2 testing.go:149: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.810Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: entering candidate state: node="Node at 948f5509-fc1f-3029-1799-1b3b8018197f [Candidate]" term=2 testLoggerAdapter verbose: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.810Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: election won: tally=2 testing.go:149: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.810Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: election won: tally=2 testLoggerAdapter verbose: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.810Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: entering leader state: leader="Node at 948f5509-fc1f-3029-1799-1b3b8018197f [Leader]" testing.go:149: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.810Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: entering leader state: leader="Node at 948f5509-fc1f-3029-1799-1b3b8018197f [Leader]" testLoggerAdapter verbose: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.810Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: added peer, starting replication: peer=server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5 testing.go:149: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.810Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: added peer, starting replication: peer=server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5 testLoggerAdapter verbose: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.810Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: added peer, starting replication: peer=server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8 testing.go:149: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.810Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: added peer, starting replication: peer=server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8 testLoggerAdapter verbose: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.810Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: pipelining replication: peer="{Voter server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8 7f9f7a1a-cb9b-484f-e407-7713638e3ab8}" testing.go:149: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.810Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: pipelining replication: peer="{Voter server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8 7f9f7a1a-cb9b-484f-e407-7713638e3ab8}" testLoggerAdapter verbose: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.810Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: pipelining replication: peer="{Voter server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5 fcaa246d-0e1b-6eba-1533-44e9d7ec79f5}" testing.go:149: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.810Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: pipelining replication: peer="{Voter server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5 fcaa246d-0e1b-6eba-1533-44e9d7ec79f5}" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:01.915Z [INFO] stable state for Leader reached at 2023-08-02 15:13:01.810605981 +0000 UTC m=+13.906542315 (1 nodes), 55.672251ms from start of poll, 56.650316ms from cluster start. Timeout at 2023-08-02 15:13:01.915819577 +0000 UTC m=+14.011755914, 105.213599ms after stability testing.go:149: cluster: 2023-08-02T15:13:01.915Z [INFO] stable state for Leader reached at 2023-08-02 15:13:01.810605981 +0000 UTC m=+13.906542315 (1 nodes), 55.672251ms from start of poll, 56.650316ms from cluster start. Timeout at 2023-08-02 15:13:01.915819577 +0000 UTC m=+14.011755914, 105.213599ms after stability testLoggerAdapter verbose: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.929Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: aborting pipeline replication: peer="{Voter server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8 7f9f7a1a-cb9b-484f-e407-7713638e3ab8}" testing.go:149: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.929Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: aborting pipeline replication: peer="{Voter server-7f9f7a1a-cb9b-484f-e407-7713638e3ab8 7f9f7a1a-cb9b-484f-e407-7713638e3ab8}" testLoggerAdapter verbose: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.929Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: aborting pipeline replication: peer="{Voter server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5 fcaa246d-0e1b-6eba-1533-44e9d7ec79f5}" testing.go:149: server-948f5509-fc1f-3029-1799-1b3b8018197f: 2023-08-02T15:13:01.929Z [INFO] server-948f5509-fc1f-3029-1799-1b3b8018197f: aborting pipeline replication: peer="{Voter server-fcaa246d-0e1b-6eba-1533-44e9d7ec79f5 fcaa246d-0e1b-6eba-1533-44e9d7ec79f5}" --- PASS: TestRaft_Barrier (0.18s) === RUN TestRaft_VerifyLeader testLoggerAdapter verbose: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.930Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce240121-2598-81a5-cbf8-0215ef3bc5ec Address:ce240121-2598-81a5-cbf8-0215ef3bc5ec} {Suffrage:Voter ID:server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb Address:9e31aa83-a216-b7a1-1a6c-25d0da1699eb} {Suffrage:Voter ID:server-b2a27528-b3be-e9e3-6130-3a02f3c122fd Address:b2a27528-b3be-e9e3-6130-3a02f3c122fd}]" testing.go:149: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.930Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce240121-2598-81a5-cbf8-0215ef3bc5ec Address:ce240121-2598-81a5-cbf8-0215ef3bc5ec} {Suffrage:Voter ID:server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb Address:9e31aa83-a216-b7a1-1a6c-25d0da1699eb} {Suffrage:Voter ID:server-b2a27528-b3be-e9e3-6130-3a02f3c122fd Address:b2a27528-b3be-e9e3-6130-3a02f3c122fd}]" testLoggerAdapter verbose: server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: 2023-08-02T15:13:01.930Z [INFO] server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce240121-2598-81a5-cbf8-0215ef3bc5ec Address:ce240121-2598-81a5-cbf8-0215ef3bc5ec} {Suffrage:Voter ID:server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb Address:9e31aa83-a216-b7a1-1a6c-25d0da1699eb} {Suffrage:Voter ID:server-b2a27528-b3be-e9e3-6130-3a02f3c122fd Address:b2a27528-b3be-e9e3-6130-3a02f3c122fd}]" testing.go:149: server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: 2023-08-02T15:13:01.930Z [INFO] server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce240121-2598-81a5-cbf8-0215ef3bc5ec Address:ce240121-2598-81a5-cbf8-0215ef3bc5ec} {Suffrage:Voter ID:server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb Address:9e31aa83-a216-b7a1-1a6c-25d0da1699eb} {Suffrage:Voter ID:server-b2a27528-b3be-e9e3-6130-3a02f3c122fd Address:b2a27528-b3be-e9e3-6130-3a02f3c122fd}]" testLoggerAdapter verbose: server-b2a27528-b3be-e9e3-6130-3a02f3c122fd: 2023-08-02T15:13:01.931Z [INFO] server-b2a27528-b3be-e9e3-6130-3a02f3c122fd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce240121-2598-81a5-cbf8-0215ef3bc5ec Address:ce240121-2598-81a5-cbf8-0215ef3bc5ec} {Suffrage:Voter ID:server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb Address:9e31aa83-a216-b7a1-1a6c-25d0da1699eb} {Suffrage:Voter ID:server-b2a27528-b3be-e9e3-6130-3a02f3c122fd Address:b2a27528-b3be-e9e3-6130-3a02f3c122fd}]" testing.go:149: server-b2a27528-b3be-e9e3-6130-3a02f3c122fd: 2023-08-02T15:13:01.931Z [INFO] server-b2a27528-b3be-e9e3-6130-3a02f3c122fd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce240121-2598-81a5-cbf8-0215ef3bc5ec Address:ce240121-2598-81a5-cbf8-0215ef3bc5ec} {Suffrage:Voter ID:server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb Address:9e31aa83-a216-b7a1-1a6c-25d0da1699eb} {Suffrage:Voter ID:server-b2a27528-b3be-e9e3-6130-3a02f3c122fd Address:b2a27528-b3be-e9e3-6130-3a02f3c122fd}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:01.931Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:13:01.931Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.931Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: entering follower state: follower="Node at ce240121-2598-81a5-cbf8-0215ef3bc5ec [Follower]" leader= testing.go:149: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.931Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: entering follower state: follower="Node at ce240121-2598-81a5-cbf8-0215ef3bc5ec [Follower]" leader= testLoggerAdapter verbose: server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: 2023-08-02T15:13:01.931Z [INFO] server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: entering follower state: follower="Node at 9e31aa83-a216-b7a1-1a6c-25d0da1699eb [Follower]" leader= testing.go:149: server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: 2023-08-02T15:13:01.931Z [INFO] server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: entering follower state: follower="Node at 9e31aa83-a216-b7a1-1a6c-25d0da1699eb [Follower]" leader= testLoggerAdapter verbose: server-b2a27528-b3be-e9e3-6130-3a02f3c122fd: 2023-08-02T15:13:01.931Z [INFO] server-b2a27528-b3be-e9e3-6130-3a02f3c122fd: entering follower state: follower="Node at b2a27528-b3be-e9e3-6130-3a02f3c122fd [Follower]" leader= testing.go:149: server-b2a27528-b3be-e9e3-6130-3a02f3c122fd: 2023-08-02T15:13:01.931Z [INFO] server-b2a27528-b3be-e9e3-6130-3a02f3c122fd: entering follower state: follower="Node at b2a27528-b3be-e9e3-6130-3a02f3c122fd [Follower]" leader= testLoggerAdapter verbose: server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: 2023-08-02T15:13:01.985Z [WARN] server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: 2023-08-02T15:13:01.985Z [WARN] server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: 2023-08-02T15:13:01.985Z [INFO] server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: entering candidate state: node="Node at 9e31aa83-a216-b7a1-1a6c-25d0da1699eb [Candidate]" term=2 testing.go:149: server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: 2023-08-02T15:13:01.985Z [INFO] server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: entering candidate state: node="Node at 9e31aa83-a216-b7a1-1a6c-25d0da1699eb [Candidate]" term=2 testLoggerAdapter verbose: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.985Z [WARN] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.985Z [WARN] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.985Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: entering candidate state: node="Node at ce240121-2598-81a5-cbf8-0215ef3bc5ec [Candidate]" term=2 testing.go:149: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.985Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: entering candidate state: node="Node at ce240121-2598-81a5-cbf8-0215ef3bc5ec [Candidate]" term=2 testLoggerAdapter verbose: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.986Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: election won: tally=2 testing.go:149: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.986Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: election won: tally=2 testLoggerAdapter verbose: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.986Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: entering leader state: leader="Node at ce240121-2598-81a5-cbf8-0215ef3bc5ec [Leader]" testing.go:149: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.986Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: entering leader state: leader="Node at ce240121-2598-81a5-cbf8-0215ef3bc5ec [Leader]" testLoggerAdapter verbose: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.986Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: added peer, starting replication: peer=server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb testing.go:149: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.986Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: added peer, starting replication: peer=server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb testLoggerAdapter verbose: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.986Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: added peer, starting replication: peer=server-b2a27528-b3be-e9e3-6130-3a02f3c122fd testing.go:149: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.986Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: added peer, starting replication: peer=server-b2a27528-b3be-e9e3-6130-3a02f3c122fd testLoggerAdapter verbose: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.986Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: pipelining replication: peer="{Voter server-b2a27528-b3be-e9e3-6130-3a02f3c122fd b2a27528-b3be-e9e3-6130-3a02f3c122fd}" testing.go:149: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.986Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: pipelining replication: peer="{Voter server-b2a27528-b3be-e9e3-6130-3a02f3c122fd b2a27528-b3be-e9e3-6130-3a02f3c122fd}" testLoggerAdapter verbose: server-b2a27528-b3be-e9e3-6130-3a02f3c122fd: 2023-08-02T15:13:01.986Z [WARN] server-b2a27528-b3be-e9e3-6130-3a02f3c122fd: rejecting vote request since we have a leader: from=9e31aa83-a216-b7a1-1a6c-25d0da1699eb leader=ce240121-2598-81a5-cbf8-0215ef3bc5ec testing.go:149: server-b2a27528-b3be-e9e3-6130-3a02f3c122fd: 2023-08-02T15:13:01.986Z [WARN] server-b2a27528-b3be-e9e3-6130-3a02f3c122fd: rejecting vote request since we have a leader: from=9e31aa83-a216-b7a1-1a6c-25d0da1699eb leader=ce240121-2598-81a5-cbf8-0215ef3bc5ec testLoggerAdapter verbose: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.986Z [WARN] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: rejecting vote request since we have a leader: from=9e31aa83-a216-b7a1-1a6c-25d0da1699eb leader=ce240121-2598-81a5-cbf8-0215ef3bc5ec testing.go:149: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.986Z [WARN] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: rejecting vote request since we have a leader: from=9e31aa83-a216-b7a1-1a6c-25d0da1699eb leader=ce240121-2598-81a5-cbf8-0215ef3bc5ec testLoggerAdapter verbose: server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: 2023-08-02T15:13:01.986Z [INFO] server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: duplicate requestVote for same term: term=2 testing.go:149: server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: 2023-08-02T15:13:01.986Z [INFO] server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: 2023-08-02T15:13:01.986Z [INFO] server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: entering follower state: follower="Node at 9e31aa83-a216-b7a1-1a6c-25d0da1699eb [Follower]" leader=ce240121-2598-81a5-cbf8-0215ef3bc5ec testing.go:149: server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: 2023-08-02T15:13:01.986Z [INFO] server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb: entering follower state: follower="Node at 9e31aa83-a216-b7a1-1a6c-25d0da1699eb [Follower]" leader=ce240121-2598-81a5-cbf8-0215ef3bc5ec testLoggerAdapter verbose: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.986Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: pipelining replication: peer="{Voter server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb 9e31aa83-a216-b7a1-1a6c-25d0da1699eb}" testing.go:149: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:01.986Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: pipelining replication: peer="{Voter server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb 9e31aa83-a216-b7a1-1a6c-25d0da1699eb}" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:02.092Z [INFO] stable state for Leader reached at 2023-08-02 15:13:01.986783382 +0000 UTC m=+14.082719733 (1 nodes), 55.632441ms from start of poll, 56.281044ms from cluster start. Timeout at 2023-08-02 15:13:02.092067611 +0000 UTC m=+14.188003951, 105.284218ms after stability testing.go:149: cluster: 2023-08-02T15:13:02.092Z [INFO] stable state for Leader reached at 2023-08-02 15:13:01.986783382 +0000 UTC m=+14.082719733 (1 nodes), 55.632441ms from start of poll, 56.281044ms from cluster start. Timeout at 2023-08-02 15:13:02.092067611 +0000 UTC m=+14.188003951, 105.284218ms after stability testLoggerAdapter verbose: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:02.092Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: aborting pipeline replication: peer="{Voter server-b2a27528-b3be-e9e3-6130-3a02f3c122fd b2a27528-b3be-e9e3-6130-3a02f3c122fd}" testing.go:149: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:02.092Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: aborting pipeline replication: peer="{Voter server-b2a27528-b3be-e9e3-6130-3a02f3c122fd b2a27528-b3be-e9e3-6130-3a02f3c122fd}" testLoggerAdapter verbose: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:02.092Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: aborting pipeline replication: peer="{Voter server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb 9e31aa83-a216-b7a1-1a6c-25d0da1699eb}" testing.go:149: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:02.092Z [INFO] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: aborting pipeline replication: peer="{Voter server-9e31aa83-a216-b7a1-1a6c-25d0da1699eb 9e31aa83-a216-b7a1-1a6c-25d0da1699eb}" testLoggerAdapter verbose: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:02.142Z [ERROR] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: failed to heartbeat to: peer=b2a27528-b3be-e9e3-6130-3a02f3c122fd error="command timed out" testing.go:149: server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: 2023-08-02T15:13:02.142Z [ERROR] server-ce240121-2598-81a5-cbf8-0215ef3bc5ec: failed to heartbeat to: peer=b2a27528-b3be-e9e3-6130-3a02f3c122fd error="command timed out" --- PASS: TestRaft_VerifyLeader (0.21s) === RUN TestRaft_VerifyLeader_Single testLoggerAdapter verbose: server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: 2023-08-02T15:13:02.143Z [INFO] server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f Address:f4221d0f-3e37-6c31-8f2b-efef8b0bc38f}]" testing.go:149: server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: 2023-08-02T15:13:02.143Z [INFO] server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f Address:f4221d0f-3e37-6c31-8f2b-efef8b0bc38f}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:02.143Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:13:02.143Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: 2023-08-02T15:13:02.143Z [INFO] server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: entering follower state: follower="Node at f4221d0f-3e37-6c31-8f2b-efef8b0bc38f [Follower]" leader= testing.go:149: server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: 2023-08-02T15:13:02.143Z [INFO] server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: entering follower state: follower="Node at f4221d0f-3e37-6c31-8f2b-efef8b0bc38f [Follower]" leader= testLoggerAdapter verbose: server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: 2023-08-02T15:13:02.193Z [WARN] server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: 2023-08-02T15:13:02.193Z [WARN] server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: 2023-08-02T15:13:02.193Z [INFO] server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: entering candidate state: node="Node at f4221d0f-3e37-6c31-8f2b-efef8b0bc38f [Candidate]" term=2 testing.go:149: server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: 2023-08-02T15:13:02.193Z [INFO] server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: entering candidate state: node="Node at f4221d0f-3e37-6c31-8f2b-efef8b0bc38f [Candidate]" term=2 testLoggerAdapter verbose: server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: 2023-08-02T15:13:02.194Z [INFO] server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: election won: tally=1 testing.go:149: server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: 2023-08-02T15:13:02.194Z [INFO] server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: election won: tally=1 testLoggerAdapter verbose: server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: 2023-08-02T15:13:02.194Z [INFO] server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: entering leader state: leader="Node at f4221d0f-3e37-6c31-8f2b-efef8b0bc38f [Leader]" testing.go:149: server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: 2023-08-02T15:13:02.194Z [INFO] server-f4221d0f-3e37-6c31-8f2b-efef8b0bc38f: entering leader state: leader="Node at f4221d0f-3e37-6c31-8f2b-efef8b0bc38f [Leader]" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:02.302Z [INFO] stable state for Leader reached at 2023-08-02 15:13:02.194081154 +0000 UTC m=+14.290017500 (1 nodes), 50.656475ms from start of poll, 50.903098ms from cluster start. Timeout at 2023-08-02 15:13:02.302631008 +0000 UTC m=+14.398567421, 108.549921ms after stability testing.go:149: cluster: 2023-08-02T15:13:02.302Z [INFO] stable state for Leader reached at 2023-08-02 15:13:02.194081154 +0000 UTC m=+14.290017500 (1 nodes), 50.656475ms from start of poll, 50.903098ms from cluster start. Timeout at 2023-08-02 15:13:02.302631008 +0000 UTC m=+14.398567421, 108.549921ms after stability --- PASS: TestRaft_VerifyLeader_Single (0.16s) === RUN TestRaft_VerifyLeader_Fail testLoggerAdapter verbose: server-7c654af0-419b-3eb9-1d7d-8b0ee8d572f0: 2023-08-02T15:13:02.303Z [INFO] server-7c654af0-419b-3eb9-1d7d-8b0ee8d572f0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7c654af0-419b-3eb9-1d7d-8b0ee8d572f0 Address:7c654af0-419b-3eb9-1d7d-8b0ee8d572f0} {Suffrage:Voter ID:server-a68448d6-81fd-6fdd-9450-70ba17177931 Address:a68448d6-81fd-6fdd-9450-70ba17177931}]" testing.go:149: server-7c654af0-419b-3eb9-1d7d-8b0ee8d572f0: 2023-08-02T15:13:02.303Z [INFO] server-7c654af0-419b-3eb9-1d7d-8b0ee8d572f0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7c654af0-419b-3eb9-1d7d-8b0ee8d572f0 Address:7c654af0-419b-3eb9-1d7d-8b0ee8d572f0} {Suffrage:Voter ID:server-a68448d6-81fd-6fdd-9450-70ba17177931 Address:a68448d6-81fd-6fdd-9450-70ba17177931}]" testLoggerAdapter verbose: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.304Z [INFO] server-a68448d6-81fd-6fdd-9450-70ba17177931: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7c654af0-419b-3eb9-1d7d-8b0ee8d572f0 Address:7c654af0-419b-3eb9-1d7d-8b0ee8d572f0} {Suffrage:Voter ID:server-a68448d6-81fd-6fdd-9450-70ba17177931 Address:a68448d6-81fd-6fdd-9450-70ba17177931}]" testing.go:149: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.304Z [INFO] server-a68448d6-81fd-6fdd-9450-70ba17177931: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7c654af0-419b-3eb9-1d7d-8b0ee8d572f0 Address:7c654af0-419b-3eb9-1d7d-8b0ee8d572f0} {Suffrage:Voter ID:server-a68448d6-81fd-6fdd-9450-70ba17177931 Address:a68448d6-81fd-6fdd-9450-70ba17177931}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:02.304Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:13:02.304Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-7c654af0-419b-3eb9-1d7d-8b0ee8d572f0: 2023-08-02T15:13:02.304Z [INFO] server-7c654af0-419b-3eb9-1d7d-8b0ee8d572f0: entering follower state: follower="Node at 7c654af0-419b-3eb9-1d7d-8b0ee8d572f0 [Follower]" leader= testing.go:149: server-7c654af0-419b-3eb9-1d7d-8b0ee8d572f0: 2023-08-02T15:13:02.304Z [INFO] server-7c654af0-419b-3eb9-1d7d-8b0ee8d572f0: entering follower state: follower="Node at 7c654af0-419b-3eb9-1d7d-8b0ee8d572f0 [Follower]" leader= testLoggerAdapter verbose: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.304Z [INFO] server-a68448d6-81fd-6fdd-9450-70ba17177931: entering follower state: follower="Node at a68448d6-81fd-6fdd-9450-70ba17177931 [Follower]" leader= testing.go:149: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.304Z [INFO] server-a68448d6-81fd-6fdd-9450-70ba17177931: entering follower state: follower="Node at a68448d6-81fd-6fdd-9450-70ba17177931 [Follower]" leader= testLoggerAdapter verbose: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.372Z [WARN] server-a68448d6-81fd-6fdd-9450-70ba17177931: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.372Z [WARN] server-a68448d6-81fd-6fdd-9450-70ba17177931: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.372Z [INFO] server-a68448d6-81fd-6fdd-9450-70ba17177931: entering candidate state: node="Node at a68448d6-81fd-6fdd-9450-70ba17177931 [Candidate]" term=2 testing.go:149: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.372Z [INFO] server-a68448d6-81fd-6fdd-9450-70ba17177931: entering candidate state: node="Node at a68448d6-81fd-6fdd-9450-70ba17177931 [Candidate]" term=2 testLoggerAdapter verbose: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.372Z [INFO] server-a68448d6-81fd-6fdd-9450-70ba17177931: election won: tally=2 testing.go:149: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.372Z [INFO] server-a68448d6-81fd-6fdd-9450-70ba17177931: election won: tally=2 testLoggerAdapter verbose: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.372Z [INFO] server-a68448d6-81fd-6fdd-9450-70ba17177931: entering leader state: leader="Node at a68448d6-81fd-6fdd-9450-70ba17177931 [Leader]" testing.go:149: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.372Z [INFO] server-a68448d6-81fd-6fdd-9450-70ba17177931: entering leader state: leader="Node at a68448d6-81fd-6fdd-9450-70ba17177931 [Leader]" testLoggerAdapter verbose: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.372Z [INFO] server-a68448d6-81fd-6fdd-9450-70ba17177931: added peer, starting replication: peer=server-7c654af0-419b-3eb9-1d7d-8b0ee8d572f0 testing.go:149: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.372Z [INFO] server-a68448d6-81fd-6fdd-9450-70ba17177931: added peer, starting replication: peer=server-7c654af0-419b-3eb9-1d7d-8b0ee8d572f0 testLoggerAdapter verbose: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.372Z [INFO] server-a68448d6-81fd-6fdd-9450-70ba17177931: pipelining replication: peer="{Voter server-7c654af0-419b-3eb9-1d7d-8b0ee8d572f0 7c654af0-419b-3eb9-1d7d-8b0ee8d572f0}" testing.go:149: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.372Z [INFO] server-a68448d6-81fd-6fdd-9450-70ba17177931: pipelining replication: peer="{Voter server-7c654af0-419b-3eb9-1d7d-8b0ee8d572f0 7c654af0-419b-3eb9-1d7d-8b0ee8d572f0}" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:02.477Z [INFO] stable state for Leader reached at 2023-08-02 15:13:02.372485667 +0000 UTC m=+14.468421996 (1 nodes), 68.251859ms from start of poll, 68.848709ms from cluster start. Timeout at 2023-08-02 15:13:02.477602556 +0000 UTC m=+14.573538891, 105.116895ms after stability testing.go:149: cluster: 2023-08-02T15:13:02.477Z [INFO] stable state for Leader reached at 2023-08-02 15:13:02.372485667 +0000 UTC m=+14.468421996 (1 nodes), 68.251859ms from start of poll, 68.848709ms from cluster start. Timeout at 2023-08-02 15:13:02.477602556 +0000 UTC m=+14.573538891, 105.116895ms after stability testLoggerAdapter verbose: cluster: 2023-08-02T15:13:02.477Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-08-02T15:13:02.477Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2023-08-02T15:13:02.582Z [INFO] stable state for Follower reached at 2023-08-02 15:13:02.477833216 +0000 UTC m=+14.573769554 (1 nodes), 4.277µs from start of poll, 174.196267ms from cluster start. Timeout at 2023-08-02 15:13:02.582944698 +0000 UTC m=+14.678881026, 105.111472ms after stability testing.go:149: cluster: 2023-08-02T15:13:02.582Z [INFO] stable state for Follower reached at 2023-08-02 15:13:02.477833216 +0000 UTC m=+14.573769554 (1 nodes), 4.277µs from start of poll, 174.196267ms from cluster start. Timeout at 2023-08-02 15:13:02.582944698 +0000 UTC m=+14.678881026, 105.111472ms after stability testLoggerAdapter verbose: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.583Z [WARN] server-a68448d6-81fd-6fdd-9450-70ba17177931: new leader elected, stepping down testing.go:149: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.583Z [WARN] server-a68448d6-81fd-6fdd-9450-70ba17177931: new leader elected, stepping down testLoggerAdapter verbose: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.583Z [INFO] server-a68448d6-81fd-6fdd-9450-70ba17177931: entering follower state: follower="Node at a68448d6-81fd-6fdd-9450-70ba17177931 [Follower]" leader= testing.go:149: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.583Z [INFO] server-a68448d6-81fd-6fdd-9450-70ba17177931: entering follower state: follower="Node at a68448d6-81fd-6fdd-9450-70ba17177931 [Follower]" leader= testLoggerAdapter verbose: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.583Z [INFO] server-a68448d6-81fd-6fdd-9450-70ba17177931: aborting pipeline replication: peer="{Voter server-7c654af0-419b-3eb9-1d7d-8b0ee8d572f0 7c654af0-419b-3eb9-1d7d-8b0ee8d572f0}" testing.go:149: server-a68448d6-81fd-6fdd-9450-70ba17177931: 2023-08-02T15:13:02.583Z [INFO] server-a68448d6-81fd-6fdd-9450-70ba17177931: aborting pipeline replication: peer="{Voter server-7c654af0-419b-3eb9-1d7d-8b0ee8d572f0 7c654af0-419b-3eb9-1d7d-8b0ee8d572f0}" --- PASS: TestRaft_VerifyLeader_Fail (0.28s) === RUN TestRaft_VerifyLeader_PartialConnect testLoggerAdapter verbose: server-3645a2fd-2c05-2806-6e81-ed5f4763e766: 2023-08-02T15:13:02.584Z [INFO] server-3645a2fd-2c05-2806-6e81-ed5f4763e766: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3645a2fd-2c05-2806-6e81-ed5f4763e766 Address:3645a2fd-2c05-2806-6e81-ed5f4763e766} {Suffrage:Voter ID:server-4d34c2e8-a619-4768-eded-14353e29d27b Address:4d34c2e8-a619-4768-eded-14353e29d27b} {Suffrage:Voter ID:server-b3744a4e-f1ec-98db-8b97-0efc209868ca Address:b3744a4e-f1ec-98db-8b97-0efc209868ca}]" testing.go:149: server-3645a2fd-2c05-2806-6e81-ed5f4763e766: 2023-08-02T15:13:02.584Z [INFO] server-3645a2fd-2c05-2806-6e81-ed5f4763e766: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3645a2fd-2c05-2806-6e81-ed5f4763e766 Address:3645a2fd-2c05-2806-6e81-ed5f4763e766} {Suffrage:Voter ID:server-4d34c2e8-a619-4768-eded-14353e29d27b Address:4d34c2e8-a619-4768-eded-14353e29d27b} {Suffrage:Voter ID:server-b3744a4e-f1ec-98db-8b97-0efc209868ca Address:b3744a4e-f1ec-98db-8b97-0efc209868ca}]" testLoggerAdapter verbose: server-3645a2fd-2c05-2806-6e81-ed5f4763e766: 2023-08-02T15:13:02.584Z [INFO] server-3645a2fd-2c05-2806-6e81-ed5f4763e766: entering follower state: follower="Node at 3645a2fd-2c05-2806-6e81-ed5f4763e766 [Follower]" leader= testing.go:149: server-3645a2fd-2c05-2806-6e81-ed5f4763e766: 2023-08-02T15:13:02.584Z [INFO] server-3645a2fd-2c05-2806-6e81-ed5f4763e766: entering follower state: follower="Node at 3645a2fd-2c05-2806-6e81-ed5f4763e766 [Follower]" leader= testLoggerAdapter verbose: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.584Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3645a2fd-2c05-2806-6e81-ed5f4763e766 Address:3645a2fd-2c05-2806-6e81-ed5f4763e766} {Suffrage:Voter ID:server-4d34c2e8-a619-4768-eded-14353e29d27b Address:4d34c2e8-a619-4768-eded-14353e29d27b} {Suffrage:Voter ID:server-b3744a4e-f1ec-98db-8b97-0efc209868ca Address:b3744a4e-f1ec-98db-8b97-0efc209868ca}]" testing.go:149: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.584Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3645a2fd-2c05-2806-6e81-ed5f4763e766 Address:3645a2fd-2c05-2806-6e81-ed5f4763e766} {Suffrage:Voter ID:server-4d34c2e8-a619-4768-eded-14353e29d27b Address:4d34c2e8-a619-4768-eded-14353e29d27b} {Suffrage:Voter ID:server-b3744a4e-f1ec-98db-8b97-0efc209868ca Address:b3744a4e-f1ec-98db-8b97-0efc209868ca}]" testLoggerAdapter verbose: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.584Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: entering follower state: follower="Node at 4d34c2e8-a619-4768-eded-14353e29d27b [Follower]" leader= testing.go:149: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.584Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: entering follower state: follower="Node at 4d34c2e8-a619-4768-eded-14353e29d27b [Follower]" leader= testLoggerAdapter verbose: server-b3744a4e-f1ec-98db-8b97-0efc209868ca: 2023-08-02T15:13:02.584Z [INFO] server-b3744a4e-f1ec-98db-8b97-0efc209868ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3645a2fd-2c05-2806-6e81-ed5f4763e766 Address:3645a2fd-2c05-2806-6e81-ed5f4763e766} {Suffrage:Voter ID:server-4d34c2e8-a619-4768-eded-14353e29d27b Address:4d34c2e8-a619-4768-eded-14353e29d27b} {Suffrage:Voter ID:server-b3744a4e-f1ec-98db-8b97-0efc209868ca Address:b3744a4e-f1ec-98db-8b97-0efc209868ca}]" testing.go:149: server-b3744a4e-f1ec-98db-8b97-0efc209868ca: 2023-08-02T15:13:02.584Z [INFO] server-b3744a4e-f1ec-98db-8b97-0efc209868ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3645a2fd-2c05-2806-6e81-ed5f4763e766 Address:3645a2fd-2c05-2806-6e81-ed5f4763e766} {Suffrage:Voter ID:server-4d34c2e8-a619-4768-eded-14353e29d27b Address:4d34c2e8-a619-4768-eded-14353e29d27b} {Suffrage:Voter ID:server-b3744a4e-f1ec-98db-8b97-0efc209868ca Address:b3744a4e-f1ec-98db-8b97-0efc209868ca}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:02.584Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:13:02.584Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-b3744a4e-f1ec-98db-8b97-0efc209868ca: 2023-08-02T15:13:02.584Z [INFO] server-b3744a4e-f1ec-98db-8b97-0efc209868ca: entering follower state: follower="Node at b3744a4e-f1ec-98db-8b97-0efc209868ca [Follower]" leader= testing.go:149: server-b3744a4e-f1ec-98db-8b97-0efc209868ca: 2023-08-02T15:13:02.584Z [INFO] server-b3744a4e-f1ec-98db-8b97-0efc209868ca: entering follower state: follower="Node at b3744a4e-f1ec-98db-8b97-0efc209868ca [Follower]" leader= testLoggerAdapter verbose: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.666Z [WARN] server-4d34c2e8-a619-4768-eded-14353e29d27b: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.666Z [WARN] server-4d34c2e8-a619-4768-eded-14353e29d27b: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.666Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: entering candidate state: node="Node at 4d34c2e8-a619-4768-eded-14353e29d27b [Candidate]" term=2 testing.go:149: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.666Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: entering candidate state: node="Node at 4d34c2e8-a619-4768-eded-14353e29d27b [Candidate]" term=2 testLoggerAdapter verbose: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.666Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: election won: tally=2 testing.go:149: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.666Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: election won: tally=2 testLoggerAdapter verbose: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.666Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: entering leader state: leader="Node at 4d34c2e8-a619-4768-eded-14353e29d27b [Leader]" testing.go:149: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.666Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: entering leader state: leader="Node at 4d34c2e8-a619-4768-eded-14353e29d27b [Leader]" testLoggerAdapter verbose: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.666Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: added peer, starting replication: peer=server-3645a2fd-2c05-2806-6e81-ed5f4763e766 testing.go:149: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.666Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: added peer, starting replication: peer=server-3645a2fd-2c05-2806-6e81-ed5f4763e766 testLoggerAdapter verbose: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.666Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: added peer, starting replication: peer=server-b3744a4e-f1ec-98db-8b97-0efc209868ca testing.go:149: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.666Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: added peer, starting replication: peer=server-b3744a4e-f1ec-98db-8b97-0efc209868ca testLoggerAdapter verbose: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.666Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: pipelining replication: peer="{Voter server-b3744a4e-f1ec-98db-8b97-0efc209868ca b3744a4e-f1ec-98db-8b97-0efc209868ca}" testing.go:149: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.666Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: pipelining replication: peer="{Voter server-b3744a4e-f1ec-98db-8b97-0efc209868ca b3744a4e-f1ec-98db-8b97-0efc209868ca}" testLoggerAdapter verbose: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.666Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: pipelining replication: peer="{Voter server-3645a2fd-2c05-2806-6e81-ed5f4763e766 3645a2fd-2c05-2806-6e81-ed5f4763e766}" testing.go:149: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.666Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: pipelining replication: peer="{Voter server-3645a2fd-2c05-2806-6e81-ed5f4763e766 3645a2fd-2c05-2806-6e81-ed5f4763e766}" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:02.771Z [INFO] stable state for Leader reached at 2023-08-02 15:13:02.66678072 +0000 UTC m=+14.762717072 (1 nodes), 82.250255ms from start of poll, 82.816725ms from cluster start. Timeout at 2023-08-02 15:13:02.771921304 +0000 UTC m=+14.867857637, 105.140565ms after stability testing.go:149: cluster: 2023-08-02T15:13:02.771Z [INFO] stable state for Leader reached at 2023-08-02 15:13:02.66678072 +0000 UTC m=+14.762717072 (1 nodes), 82.250255ms from start of poll, 82.816725ms from cluster start. Timeout at 2023-08-02 15:13:02.771921304 +0000 UTC m=+14.867857637, 105.140565ms after stability testLoggerAdapter verbose: cluster: 2023-08-02T15:13:02.777Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-08-02T15:13:02.777Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2023-08-02T15:13:02.883Z [INFO] stable state for Follower reached at 2023-08-02 15:13:02.777263003 +0000 UTC m=+14.873199329 (2 nodes), 1.546µs from start of poll, 193.298982ms from cluster start. Timeout at 2023-08-02 15:13:02.883318944 +0000 UTC m=+14.979255287, 106.055958ms after stability testing.go:149: cluster: 2023-08-02T15:13:02.883Z [INFO] stable state for Follower reached at 2023-08-02 15:13:02.777263003 +0000 UTC m=+14.873199329 (2 nodes), 1.546µs from start of poll, 193.298982ms from cluster start. Timeout at 2023-08-02 15:13:02.883318944 +0000 UTC m=+14.979255287, 106.055958ms after stability raft_test.go:1647: [INFO] Disconnecting Node at 3645a2fd-2c05-2806-6e81-ed5f4763e766 [Follower] testLoggerAdapter verbose: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.883Z [ERROR] server-4d34c2e8-a619-4768-eded-14353e29d27b: failed to heartbeat to: peer=3645a2fd-2c05-2806-6e81-ed5f4763e766 error="failed to connect to peer: 3645a2fd-2c05-2806-6e81-ed5f4763e766" testing.go:149: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.883Z [ERROR] server-4d34c2e8-a619-4768-eded-14353e29d27b: failed to heartbeat to: peer=3645a2fd-2c05-2806-6e81-ed5f4763e766 error="failed to connect to peer: 3645a2fd-2c05-2806-6e81-ed5f4763e766" testLoggerAdapter verbose: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.883Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: aborting pipeline replication: peer="{Voter server-b3744a4e-f1ec-98db-8b97-0efc209868ca b3744a4e-f1ec-98db-8b97-0efc209868ca}" testing.go:149: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.883Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: aborting pipeline replication: peer="{Voter server-b3744a4e-f1ec-98db-8b97-0efc209868ca b3744a4e-f1ec-98db-8b97-0efc209868ca}" testLoggerAdapter verbose: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.883Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: aborting pipeline replication: peer="{Voter server-3645a2fd-2c05-2806-6e81-ed5f4763e766 3645a2fd-2c05-2806-6e81-ed5f4763e766}" testing.go:149: server-4d34c2e8-a619-4768-eded-14353e29d27b: 2023-08-02T15:13:02.883Z [INFO] server-4d34c2e8-a619-4768-eded-14353e29d27b: aborting pipeline replication: peer="{Voter server-3645a2fd-2c05-2806-6e81-ed5f4763e766 3645a2fd-2c05-2806-6e81-ed5f4763e766}" --- PASS: TestRaft_VerifyLeader_PartialConnect (0.30s) === RUN TestRaft_StartAsLeader testLoggerAdapter verbose: server-fb46439f-cfb6-a2c1-77e9-dd5a747192ec: 2023-08-02T15:13:02.885Z [INFO] server-fb46439f-cfb6-a2c1-77e9-dd5a747192ec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fb46439f-cfb6-a2c1-77e9-dd5a747192ec Address:fb46439f-cfb6-a2c1-77e9-dd5a747192ec}]" testing.go:149: server-fb46439f-cfb6-a2c1-77e9-dd5a747192ec: 2023-08-02T15:13:02.885Z [INFO] server-fb46439f-cfb6-a2c1-77e9-dd5a747192ec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fb46439f-cfb6-a2c1-77e9-dd5a747192ec Address:fb46439f-cfb6-a2c1-77e9-dd5a747192ec}]" testLoggerAdapter verbose: server-fb46439f-cfb6-a2c1-77e9-dd5a747192ec: 2023-08-02T15:13:02.885Z [INFO] server-fb46439f-cfb6-a2c1-77e9-dd5a747192ec: entering leader state: leader="Node at fb46439f-cfb6-a2c1-77e9-dd5a747192ec [Leader]" testing.go:149: server-fb46439f-cfb6-a2c1-77e9-dd5a747192ec: 2023-08-02T15:13:02.885Z [INFO] server-fb46439f-cfb6-a2c1-77e9-dd5a747192ec: entering leader state: leader="Node at fb46439f-cfb6-a2c1-77e9-dd5a747192ec [Leader]" --- PASS: TestRaft_StartAsLeader (0.00s) === RUN TestRaft_NotifyCh testLoggerAdapter verbose: server-03feaa6a-da93-368d-6f1c-f91e4480848d: 2023-08-02T15:13:02.886Z [INFO] server-03feaa6a-da93-368d-6f1c-f91e4480848d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-03feaa6a-da93-368d-6f1c-f91e4480848d Address:03feaa6a-da93-368d-6f1c-f91e4480848d}]" testing.go:149: server-03feaa6a-da93-368d-6f1c-f91e4480848d: 2023-08-02T15:13:02.886Z [INFO] server-03feaa6a-da93-368d-6f1c-f91e4480848d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-03feaa6a-da93-368d-6f1c-f91e4480848d Address:03feaa6a-da93-368d-6f1c-f91e4480848d}]" testLoggerAdapter verbose: server-03feaa6a-da93-368d-6f1c-f91e4480848d: 2023-08-02T15:13:02.886Z [INFO] server-03feaa6a-da93-368d-6f1c-f91e4480848d: entering follower state: follower="Node at 03feaa6a-da93-368d-6f1c-f91e4480848d [Follower]" leader= testing.go:149: server-03feaa6a-da93-368d-6f1c-f91e4480848d: 2023-08-02T15:13:02.886Z [INFO] server-03feaa6a-da93-368d-6f1c-f91e4480848d: entering follower state: follower="Node at 03feaa6a-da93-368d-6f1c-f91e4480848d [Follower]" leader= testLoggerAdapter verbose: server-03feaa6a-da93-368d-6f1c-f91e4480848d: 2023-08-02T15:13:02.971Z [WARN] server-03feaa6a-da93-368d-6f1c-f91e4480848d: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-03feaa6a-da93-368d-6f1c-f91e4480848d: 2023-08-02T15:13:02.971Z [WARN] server-03feaa6a-da93-368d-6f1c-f91e4480848d: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-03feaa6a-da93-368d-6f1c-f91e4480848d: 2023-08-02T15:13:02.971Z [INFO] server-03feaa6a-da93-368d-6f1c-f91e4480848d: entering candidate state: node="Node at 03feaa6a-da93-368d-6f1c-f91e4480848d [Candidate]" term=2 testing.go:149: server-03feaa6a-da93-368d-6f1c-f91e4480848d: 2023-08-02T15:13:02.971Z [INFO] server-03feaa6a-da93-368d-6f1c-f91e4480848d: entering candidate state: node="Node at 03feaa6a-da93-368d-6f1c-f91e4480848d [Candidate]" term=2 testLoggerAdapter verbose: server-03feaa6a-da93-368d-6f1c-f91e4480848d: 2023-08-02T15:13:02.971Z [INFO] server-03feaa6a-da93-368d-6f1c-f91e4480848d: election won: tally=1 testing.go:149: server-03feaa6a-da93-368d-6f1c-f91e4480848d: 2023-08-02T15:13:02.971Z [INFO] server-03feaa6a-da93-368d-6f1c-f91e4480848d: election won: tally=1 testLoggerAdapter verbose: server-03feaa6a-da93-368d-6f1c-f91e4480848d: 2023-08-02T15:13:02.971Z [INFO] server-03feaa6a-da93-368d-6f1c-f91e4480848d: entering leader state: leader="Node at 03feaa6a-da93-368d-6f1c-f91e4480848d [Leader]" testing.go:149: server-03feaa6a-da93-368d-6f1c-f91e4480848d: 2023-08-02T15:13:02.971Z [INFO] server-03feaa6a-da93-368d-6f1c-f91e4480848d: entering leader state: leader="Node at 03feaa6a-da93-368d-6f1c-f91e4480848d [Leader]" --- PASS: TestRaft_NotifyCh (0.09s) === RUN TestRaft_Voting testLoggerAdapter verbose: server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44: 2023-08-02T15:13:02.972Z [INFO] server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44 Address:84a510be-8c77-a26a-a8ab-1b5f6cc96c44} {Suffrage:Voter ID:server-4315480e-881b-0df7-4949-932968f11d4e Address:4315480e-881b-0df7-4949-932968f11d4e} {Suffrage:Voter ID:server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535 Address:6a9bcf8f-e132-323e-d5dc-c3e34ddf8535}]" testing.go:149: server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44: 2023-08-02T15:13:02.972Z [INFO] server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44 Address:84a510be-8c77-a26a-a8ab-1b5f6cc96c44} {Suffrage:Voter ID:server-4315480e-881b-0df7-4949-932968f11d4e Address:4315480e-881b-0df7-4949-932968f11d4e} {Suffrage:Voter ID:server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535 Address:6a9bcf8f-e132-323e-d5dc-c3e34ddf8535}]" testLoggerAdapter verbose: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:02.972Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44 Address:84a510be-8c77-a26a-a8ab-1b5f6cc96c44} {Suffrage:Voter ID:server-4315480e-881b-0df7-4949-932968f11d4e Address:4315480e-881b-0df7-4949-932968f11d4e} {Suffrage:Voter ID:server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535 Address:6a9bcf8f-e132-323e-d5dc-c3e34ddf8535}]" testing.go:149: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:02.972Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44 Address:84a510be-8c77-a26a-a8ab-1b5f6cc96c44} {Suffrage:Voter ID:server-4315480e-881b-0df7-4949-932968f11d4e Address:4315480e-881b-0df7-4949-932968f11d4e} {Suffrage:Voter ID:server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535 Address:6a9bcf8f-e132-323e-d5dc-c3e34ddf8535}]" testLoggerAdapter verbose: server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535: 2023-08-02T15:13:02.972Z [INFO] server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44 Address:84a510be-8c77-a26a-a8ab-1b5f6cc96c44} {Suffrage:Voter ID:server-4315480e-881b-0df7-4949-932968f11d4e Address:4315480e-881b-0df7-4949-932968f11d4e} {Suffrage:Voter ID:server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535 Address:6a9bcf8f-e132-323e-d5dc-c3e34ddf8535}]" testing.go:149: server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535: 2023-08-02T15:13:02.972Z [INFO] server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44 Address:84a510be-8c77-a26a-a8ab-1b5f6cc96c44} {Suffrage:Voter ID:server-4315480e-881b-0df7-4949-932968f11d4e Address:4315480e-881b-0df7-4949-932968f11d4e} {Suffrage:Voter ID:server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535 Address:6a9bcf8f-e132-323e-d5dc-c3e34ddf8535}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:02.972Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-08-02T15:13:02.972Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44: 2023-08-02T15:13:02.972Z [INFO] server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44: entering follower state: follower="Node at 84a510be-8c77-a26a-a8ab-1b5f6cc96c44 [Follower]" leader= testing.go:149: server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44: 2023-08-02T15:13:02.972Z [INFO] server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44: entering follower state: follower="Node at 84a510be-8c77-a26a-a8ab-1b5f6cc96c44 [Follower]" leader= testLoggerAdapter verbose: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:02.972Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: entering follower state: follower="Node at 4315480e-881b-0df7-4949-932968f11d4e [Follower]" leader= testing.go:149: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:02.972Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: entering follower state: follower="Node at 4315480e-881b-0df7-4949-932968f11d4e [Follower]" leader= testLoggerAdapter verbose: server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535: 2023-08-02T15:13:02.972Z [INFO] server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535: entering follower state: follower="Node at 6a9bcf8f-e132-323e-d5dc-c3e34ddf8535 [Follower]" leader= testing.go:149: server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535: 2023-08-02T15:13:02.972Z [INFO] server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535: entering follower state: follower="Node at 6a9bcf8f-e132-323e-d5dc-c3e34ddf8535 [Follower]" leader= testLoggerAdapter verbose: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.023Z [WARN] server-4315480e-881b-0df7-4949-932968f11d4e: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.023Z [WARN] server-4315480e-881b-0df7-4949-932968f11d4e: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.023Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: entering candidate state: node="Node at 4315480e-881b-0df7-4949-932968f11d4e [Candidate]" term=2 testing.go:149: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.023Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: entering candidate state: node="Node at 4315480e-881b-0df7-4949-932968f11d4e [Candidate]" term=2 testLoggerAdapter verbose: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.024Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: election won: tally=2 testing.go:149: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.024Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: election won: tally=2 testLoggerAdapter verbose: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.024Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: entering leader state: leader="Node at 4315480e-881b-0df7-4949-932968f11d4e [Leader]" testing.go:149: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.024Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: entering leader state: leader="Node at 4315480e-881b-0df7-4949-932968f11d4e [Leader]" testLoggerAdapter verbose: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.024Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: added peer, starting replication: peer=server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44 testing.go:149: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.024Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: added peer, starting replication: peer=server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44 testLoggerAdapter verbose: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.024Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: added peer, starting replication: peer=server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535 testing.go:149: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.024Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: added peer, starting replication: peer=server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535 testLoggerAdapter verbose: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.024Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: pipelining replication: peer="{Voter server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44 84a510be-8c77-a26a-a8ab-1b5f6cc96c44}" testing.go:149: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.024Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: pipelining replication: peer="{Voter server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44 84a510be-8c77-a26a-a8ab-1b5f6cc96c44}" testLoggerAdapter verbose: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.024Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: pipelining replication: peer="{Voter server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535 6a9bcf8f-e132-323e-d5dc-c3e34ddf8535}" testing.go:149: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.024Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: pipelining replication: peer="{Voter server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535 6a9bcf8f-e132-323e-d5dc-c3e34ddf8535}" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:03.129Z [INFO] stable state for Follower reached at 2023-08-02 15:13:03.024198385 +0000 UTC m=+15.120134719 (2 nodes), 51.358472ms from start of poll, 51.762294ms from cluster start. Timeout at 2023-08-02 15:13:03.129289345 +0000 UTC m=+15.225225678, 105.090959ms after stability testing.go:149: cluster: 2023-08-02T15:13:03.129Z [INFO] stable state for Follower reached at 2023-08-02 15:13:03.024198385 +0000 UTC m=+15.120134719 (2 nodes), 51.358472ms from start of poll, 51.762294ms from cluster start. Timeout at 2023-08-02 15:13:03.129289345 +0000 UTC m=+15.225225678, 105.090959ms after stability testLoggerAdapter verbose: cluster: 2023-08-02T15:13:03.129Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:13:03.129Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-08-02T15:13:03.234Z [INFO] stable state for Leader reached at 2023-08-02 15:13:03.129470464 +0000 UTC m=+15.225406790 (1 nodes), 1.189µs from start of poll, 157.034365ms from cluster start. Timeout at 2023-08-02 15:13:03.234593345 +0000 UTC m=+15.330529680, 105.12289ms after stability testing.go:149: cluster: 2023-08-02T15:13:03.234Z [INFO] stable state for Leader reached at 2023-08-02 15:13:03.129470464 +0000 UTC m=+15.225406790 (1 nodes), 1.189µs from start of poll, 157.034365ms from cluster start. Timeout at 2023-08-02 15:13:03.234593345 +0000 UTC m=+15.330529680, 105.12289ms after stability testLoggerAdapter verbose: server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535: 2023-08-02T15:13:03.234Z [WARN] server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535: rejecting vote request since we have a leader: from=84a510be-8c77-a26a-a8ab-1b5f6cc96c44 leader=4315480e-881b-0df7-4949-932968f11d4e testing.go:149: server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535: 2023-08-02T15:13:03.234Z [WARN] server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535: rejecting vote request since we have a leader: from=84a510be-8c77-a26a-a8ab-1b5f6cc96c44 leader=4315480e-881b-0df7-4949-932968f11d4e testLoggerAdapter verbose: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.234Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: aborting pipeline replication: peer="{Voter server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535 6a9bcf8f-e132-323e-d5dc-c3e34ddf8535}" testing.go:149: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.234Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: aborting pipeline replication: peer="{Voter server-6a9bcf8f-e132-323e-d5dc-c3e34ddf8535 6a9bcf8f-e132-323e-d5dc-c3e34ddf8535}" testLoggerAdapter verbose: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.234Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: aborting pipeline replication: peer="{Voter server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44 84a510be-8c77-a26a-a8ab-1b5f6cc96c44}" testing.go:149: server-4315480e-881b-0df7-4949-932968f11d4e: 2023-08-02T15:13:03.234Z [INFO] server-4315480e-881b-0df7-4949-932968f11d4e: aborting pipeline replication: peer="{Voter server-84a510be-8c77-a26a-a8ab-1b5f6cc96c44 84a510be-8c77-a26a-a8ab-1b5f6cc96c44}" --- PASS: TestRaft_Voting (0.26s) === RUN TestRaft_ProtocolVersion_RejectRPC testLoggerAdapter verbose: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.236Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68d0c667-b6c6-93be-7c92-5c727698721a Address:68d0c667-b6c6-93be-7c92-5c727698721a} {Suffrage:Voter ID:server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a Address:fb9428dc-383b-ae4d-0fb8-6c45b88e459a} {Suffrage:Voter ID:server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1 Address:3e9df4c1-56f7-e878-ed42-5ba6850bafb1}]" testing.go:149: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.236Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68d0c667-b6c6-93be-7c92-5c727698721a Address:68d0c667-b6c6-93be-7c92-5c727698721a} {Suffrage:Voter ID:server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a Address:fb9428dc-383b-ae4d-0fb8-6c45b88e459a} {Suffrage:Voter ID:server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1 Address:3e9df4c1-56f7-e878-ed42-5ba6850bafb1}]" testLoggerAdapter verbose: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.236Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: entering follower state: follower="Node at 68d0c667-b6c6-93be-7c92-5c727698721a [Follower]" leader= testLoggerAdapter verbose: server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a: 2023-08-02T15:13:03.236Z [INFO] server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68d0c667-b6c6-93be-7c92-5c727698721a Address:68d0c667-b6c6-93be-7c92-5c727698721a} {Suffrage:Voter ID:server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a Address:fb9428dc-383b-ae4d-0fb8-6c45b88e459a} {Suffrage:Voter ID:server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1 Address:3e9df4c1-56f7-e878-ed42-5ba6850bafb1}]" testing.go:149: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.236Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: entering follower state: follower="Node at 68d0c667-b6c6-93be-7c92-5c727698721a [Follower]" leader= testing.go:149: server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a: 2023-08-02T15:13:03.236Z [INFO] server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68d0c667-b6c6-93be-7c92-5c727698721a Address:68d0c667-b6c6-93be-7c92-5c727698721a} {Suffrage:Voter ID:server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a Address:fb9428dc-383b-ae4d-0fb8-6c45b88e459a} {Suffrage:Voter ID:server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1 Address:3e9df4c1-56f7-e878-ed42-5ba6850bafb1}]" testLoggerAdapter verbose: server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a: 2023-08-02T15:13:03.236Z [INFO] server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a: entering follower state: follower="Node at fb9428dc-383b-ae4d-0fb8-6c45b88e459a [Follower]" leader= testing.go:149: server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a: 2023-08-02T15:13:03.236Z [INFO] server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a: entering follower state: follower="Node at fb9428dc-383b-ae4d-0fb8-6c45b88e459a [Follower]" leader= testLoggerAdapter verbose: server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1: 2023-08-02T15:13:03.236Z [INFO] server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68d0c667-b6c6-93be-7c92-5c727698721a Address:68d0c667-b6c6-93be-7c92-5c727698721a} {Suffrage:Voter ID:server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a Address:fb9428dc-383b-ae4d-0fb8-6c45b88e459a} {Suffrage:Voter ID:server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1 Address:3e9df4c1-56f7-e878-ed42-5ba6850bafb1}]" testing.go:149: server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1: 2023-08-02T15:13:03.236Z [INFO] server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68d0c667-b6c6-93be-7c92-5c727698721a Address:68d0c667-b6c6-93be-7c92-5c727698721a} {Suffrage:Voter ID:server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a Address:fb9428dc-383b-ae4d-0fb8-6c45b88e459a} {Suffrage:Voter ID:server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1 Address:3e9df4c1-56f7-e878-ed42-5ba6850bafb1}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:03.236Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-08-02T15:13:03.236Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1: 2023-08-02T15:13:03.236Z [INFO] server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1: entering follower state: follower="Node at 3e9df4c1-56f7-e878-ed42-5ba6850bafb1 [Follower]" leader= testing.go:149: server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1: 2023-08-02T15:13:03.236Z [INFO] server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1: entering follower state: follower="Node at 3e9df4c1-56f7-e878-ed42-5ba6850bafb1 [Follower]" leader= testLoggerAdapter verbose: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.291Z [WARN] server-68d0c667-b6c6-93be-7c92-5c727698721a: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.291Z [WARN] server-68d0c667-b6c6-93be-7c92-5c727698721a: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.291Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: entering candidate state: node="Node at 68d0c667-b6c6-93be-7c92-5c727698721a [Candidate]" term=2 testing.go:149: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.291Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: entering candidate state: node="Node at 68d0c667-b6c6-93be-7c92-5c727698721a [Candidate]" term=2 testLoggerAdapter verbose: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.291Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: election won: tally=2 testing.go:149: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.291Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: election won: tally=2 testLoggerAdapter verbose: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.291Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: entering leader state: leader="Node at 68d0c667-b6c6-93be-7c92-5c727698721a [Leader]" testing.go:149: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.291Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: entering leader state: leader="Node at 68d0c667-b6c6-93be-7c92-5c727698721a [Leader]" testLoggerAdapter verbose: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.291Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: added peer, starting replication: peer=server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a testing.go:149: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.291Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: added peer, starting replication: peer=server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a testLoggerAdapter verbose: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.291Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: added peer, starting replication: peer=server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1 testing.go:149: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.291Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: added peer, starting replication: peer=server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1 testLoggerAdapter verbose: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.291Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: pipelining replication: peer="{Voter server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1 3e9df4c1-56f7-e878-ed42-5ba6850bafb1}" testing.go:149: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.291Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: pipelining replication: peer="{Voter server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1 3e9df4c1-56f7-e878-ed42-5ba6850bafb1}" testLoggerAdapter verbose: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.291Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: pipelining replication: peer="{Voter server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a fb9428dc-383b-ae4d-0fb8-6c45b88e459a}" testing.go:149: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.291Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: pipelining replication: peer="{Voter server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a fb9428dc-383b-ae4d-0fb8-6c45b88e459a}" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:03.396Z [INFO] stable state for Follower reached at 2023-08-02 15:13:03.291704946 +0000 UTC m=+15.387641273 (2 nodes), 55.009122ms from start of poll, 55.796773ms from cluster start. Timeout at 2023-08-02 15:13:03.396725276 +0000 UTC m=+15.492661617, 105.020344ms after stability testing.go:149: cluster: 2023-08-02T15:13:03.396Z [INFO] stable state for Follower reached at 2023-08-02 15:13:03.291704946 +0000 UTC m=+15.387641273 (2 nodes), 55.009122ms from start of poll, 55.796773ms from cluster start. Timeout at 2023-08-02 15:13:03.396725276 +0000 UTC m=+15.492661617, 105.020344ms after stability testLoggerAdapter verbose: cluster: 2023-08-02T15:13:03.396Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:13:03.396Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-08-02T15:13:03.502Z [INFO] stable state for Leader reached at 2023-08-02 15:13:03.396885163 +0000 UTC m=+15.492821491 (1 nodes), 1.322µs from start of poll, 160.976991ms from cluster start. Timeout at 2023-08-02 15:13:03.502566897 +0000 UTC m=+15.598503230, 105.681739ms after stability testing.go:149: cluster: 2023-08-02T15:13:03.502Z [INFO] stable state for Leader reached at 2023-08-02 15:13:03.396885163 +0000 UTC m=+15.492821491 (1 nodes), 1.322µs from start of poll, 160.976991ms from cluster start. Timeout at 2023-08-02 15:13:03.502566897 +0000 UTC m=+15.598503230, 105.681739ms after stability testLoggerAdapter verbose: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.502Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: aborting pipeline replication: peer="{Voter server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1 3e9df4c1-56f7-e878-ed42-5ba6850bafb1}" testing.go:149: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.502Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: aborting pipeline replication: peer="{Voter server-3e9df4c1-56f7-e878-ed42-5ba6850bafb1 3e9df4c1-56f7-e878-ed42-5ba6850bafb1}" testLoggerAdapter verbose: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.502Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: aborting pipeline replication: peer="{Voter server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a fb9428dc-383b-ae4d-0fb8-6c45b88e459a}" testing.go:149: server-68d0c667-b6c6-93be-7c92-5c727698721a: 2023-08-02T15:13:03.502Z [INFO] server-68d0c667-b6c6-93be-7c92-5c727698721a: aborting pipeline replication: peer="{Voter server-fb9428dc-383b-ae4d-0fb8-6c45b88e459a fb9428dc-383b-ae4d-0fb8-6c45b88e459a}" --- PASS: TestRaft_ProtocolVersion_RejectRPC (0.27s) === RUN TestRaft_ProtocolVersion_Upgrade_1_2 testLoggerAdapter verbose: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.503Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: initial configuration: index=1 servers="[{Suffrage:Voter ID:ae3e7bcb-aec5-d032-06c8-d90ab73ce914 Address:ae3e7bcb-aec5-d032-06c8-d90ab73ce914} {Suffrage:Voter ID:db56337b-e877-27f1-3b4a-764bfa81b234 Address:db56337b-e877-27f1-3b4a-764bfa81b234}]" testing.go:149: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.503Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: initial configuration: index=1 servers="[{Suffrage:Voter ID:ae3e7bcb-aec5-d032-06c8-d90ab73ce914 Address:ae3e7bcb-aec5-d032-06c8-d90ab73ce914} {Suffrage:Voter ID:db56337b-e877-27f1-3b4a-764bfa81b234 Address:db56337b-e877-27f1-3b4a-764bfa81b234}]" testLoggerAdapter verbose: db56337b-e877-27f1-3b4a-764bfa81b234: 2023-08-02T15:13:03.503Z [INFO] db56337b-e877-27f1-3b4a-764bfa81b234: initial configuration: index=1 servers="[{Suffrage:Voter ID:ae3e7bcb-aec5-d032-06c8-d90ab73ce914 Address:ae3e7bcb-aec5-d032-06c8-d90ab73ce914} {Suffrage:Voter ID:db56337b-e877-27f1-3b4a-764bfa81b234 Address:db56337b-e877-27f1-3b4a-764bfa81b234}]" testing.go:149: db56337b-e877-27f1-3b4a-764bfa81b234: 2023-08-02T15:13:03.503Z [INFO] db56337b-e877-27f1-3b4a-764bfa81b234: initial configuration: index=1 servers="[{Suffrage:Voter ID:ae3e7bcb-aec5-d032-06c8-d90ab73ce914 Address:ae3e7bcb-aec5-d032-06c8-d90ab73ce914} {Suffrage:Voter ID:db56337b-e877-27f1-3b4a-764bfa81b234 Address:db56337b-e877-27f1-3b4a-764bfa81b234}]" testLoggerAdapter verbose: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.503Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: entering follower state: follower="Node at ae3e7bcb-aec5-d032-06c8-d90ab73ce914 [Follower]" leader= testing.go:149: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.503Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: entering follower state: follower="Node at ae3e7bcb-aec5-d032-06c8-d90ab73ce914 [Follower]" leader= testLoggerAdapter verbose: db56337b-e877-27f1-3b4a-764bfa81b234: 2023-08-02T15:13:03.503Z [INFO] db56337b-e877-27f1-3b4a-764bfa81b234: entering follower state: follower="Node at db56337b-e877-27f1-3b4a-764bfa81b234 [Follower]" leader= testing.go:149: db56337b-e877-27f1-3b4a-764bfa81b234: 2023-08-02T15:13:03.503Z [INFO] db56337b-e877-27f1-3b4a-764bfa81b234: entering follower state: follower="Node at db56337b-e877-27f1-3b4a-764bfa81b234 [Follower]" leader= testLoggerAdapter verbose: 39a60e0c-77b5-4ca1-ccc4-6b716f84331a: 2023-08-02T15:13:03.503Z [INFO] 39a60e0c-77b5-4ca1-ccc4-6b716f84331a: initial configuration: index=0 servers=[] testing.go:149: 39a60e0c-77b5-4ca1-ccc4-6b716f84331a: 2023-08-02T15:13:03.503Z [INFO] 39a60e0c-77b5-4ca1-ccc4-6b716f84331a: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2023-08-02T15:13:03.503Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:13:03.503Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: 39a60e0c-77b5-4ca1-ccc4-6b716f84331a: 2023-08-02T15:13:03.504Z [INFO] 39a60e0c-77b5-4ca1-ccc4-6b716f84331a: entering follower state: follower="Node at 39a60e0c-77b5-4ca1-ccc4-6b716f84331a [Follower]" leader= testing.go:149: 39a60e0c-77b5-4ca1-ccc4-6b716f84331a: 2023-08-02T15:13:03.504Z [INFO] 39a60e0c-77b5-4ca1-ccc4-6b716f84331a: entering follower state: follower="Node at 39a60e0c-77b5-4ca1-ccc4-6b716f84331a [Follower]" leader= testLoggerAdapter verbose: 39a60e0c-77b5-4ca1-ccc4-6b716f84331a: 2023-08-02T15:13:03.562Z [WARN] 39a60e0c-77b5-4ca1-ccc4-6b716f84331a: no known peers, aborting election testing.go:149: 39a60e0c-77b5-4ca1-ccc4-6b716f84331a: 2023-08-02T15:13:03.562Z [WARN] 39a60e0c-77b5-4ca1-ccc4-6b716f84331a: no known peers, aborting election testLoggerAdapter verbose: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.565Z [WARN] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: heartbeat timeout reached, starting election: last-leader= testing.go:149: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.565Z [WARN] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.565Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: entering candidate state: node="Node at ae3e7bcb-aec5-d032-06c8-d90ab73ce914 [Candidate]" term=2 testing.go:149: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.565Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: entering candidate state: node="Node at ae3e7bcb-aec5-d032-06c8-d90ab73ce914 [Candidate]" term=2 testLoggerAdapter verbose: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.566Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: election won: tally=2 testing.go:149: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.566Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: election won: tally=2 testLoggerAdapter verbose: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.566Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: entering leader state: leader="Node at ae3e7bcb-aec5-d032-06c8-d90ab73ce914 [Leader]" testing.go:149: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.566Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: entering leader state: leader="Node at ae3e7bcb-aec5-d032-06c8-d90ab73ce914 [Leader]" testLoggerAdapter verbose: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.566Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: added peer, starting replication: peer=db56337b-e877-27f1-3b4a-764bfa81b234 testing.go:149: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.566Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: added peer, starting replication: peer=db56337b-e877-27f1-3b4a-764bfa81b234 testLoggerAdapter verbose: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.566Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: pipelining replication: peer="{Voter db56337b-e877-27f1-3b4a-764bfa81b234 db56337b-e877-27f1-3b4a-764bfa81b234}" testing.go:149: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.566Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: pipelining replication: peer="{Voter db56337b-e877-27f1-3b4a-764bfa81b234 db56337b-e877-27f1-3b4a-764bfa81b234}" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:03.671Z [INFO] stable state for Leader reached at 2023-08-02 15:13:03.566250029 +0000 UTC m=+15.662186356 (1 nodes), 62.22204ms from start of poll, 62.730003ms from cluster start. Timeout at 2023-08-02 15:13:03.67135306 +0000 UTC m=+15.767289390, 105.103034ms after stability testing.go:149: cluster: 2023-08-02T15:13:03.671Z [INFO] stable state for Leader reached at 2023-08-02 15:13:03.566250029 +0000 UTC m=+15.662186356 (1 nodes), 62.22204ms from start of poll, 62.730003ms from cluster start. Timeout at 2023-08-02 15:13:03.67135306 +0000 UTC m=+15.767289390, 105.103034ms after stability testLoggerAdapter verbose: cluster: 2023-08-02T15:13:03.671Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:13:03.671Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-08-02T15:13:03.776Z [INFO] stable state for Leader reached at 2023-08-02 15:13:03.671497672 +0000 UTC m=+15.767433999 (1 nodes), 1.078µs from start of poll, 167.977646ms from cluster start. Timeout at 2023-08-02 15:13:03.776797502 +0000 UTC m=+15.872733836, 105.299837ms after stability testing.go:149: cluster: 2023-08-02T15:13:03.776Z [INFO] stable state for Leader reached at 2023-08-02 15:13:03.671497672 +0000 UTC m=+15.767433999 (1 nodes), 1.078µs from start of poll, 167.977646ms from cluster start. Timeout at 2023-08-02 15:13:03.776797502 +0000 UTC m=+15.872733836, 105.299837ms after stability testLoggerAdapter verbose: cluster: 2023-08-02T15:13:03.776Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:13:03.776Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-08-02T15:13:03.882Z [INFO] stable state for Leader reached at 2023-08-02 15:13:03.776985016 +0000 UTC m=+15.872921353 (1 nodes), 1.149µs from start of poll, 273.465ms from cluster start. Timeout at 2023-08-02 15:13:03.882067591 +0000 UTC m=+15.978003917, 105.082564ms after stability testing.go:149: cluster: 2023-08-02T15:13:03.882Z [INFO] stable state for Leader reached at 2023-08-02 15:13:03.776985016 +0000 UTC m=+15.872921353 (1 nodes), 1.149µs from start of poll, 273.465ms from cluster start. Timeout at 2023-08-02 15:13:03.882067591 +0000 UTC m=+15.978003917, 105.082564ms after stability testLoggerAdapter verbose: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.882Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: updating configuration: command=AddStaging server-id=39a60e0c-77b5-4ca1-ccc4-6b716f84331a server-addr=39a60e0c-77b5-4ca1-ccc4-6b716f84331a servers="[{Suffrage:Voter ID:ae3e7bcb-aec5-d032-06c8-d90ab73ce914 Address:ae3e7bcb-aec5-d032-06c8-d90ab73ce914} {Suffrage:Voter ID:db56337b-e877-27f1-3b4a-764bfa81b234 Address:db56337b-e877-27f1-3b4a-764bfa81b234} {Suffrage:Voter ID:39a60e0c-77b5-4ca1-ccc4-6b716f84331a Address:39a60e0c-77b5-4ca1-ccc4-6b716f84331a}]" testing.go:149: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.882Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: updating configuration: command=AddStaging server-id=39a60e0c-77b5-4ca1-ccc4-6b716f84331a server-addr=39a60e0c-77b5-4ca1-ccc4-6b716f84331a servers="[{Suffrage:Voter ID:ae3e7bcb-aec5-d032-06c8-d90ab73ce914 Address:ae3e7bcb-aec5-d032-06c8-d90ab73ce914} {Suffrage:Voter ID:db56337b-e877-27f1-3b4a-764bfa81b234 Address:db56337b-e877-27f1-3b4a-764bfa81b234} {Suffrage:Voter ID:39a60e0c-77b5-4ca1-ccc4-6b716f84331a Address:39a60e0c-77b5-4ca1-ccc4-6b716f84331a}]" testLoggerAdapter verbose: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.882Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: added peer, starting replication: peer=39a60e0c-77b5-4ca1-ccc4-6b716f84331a testing.go:149: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.882Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: added peer, starting replication: peer=39a60e0c-77b5-4ca1-ccc4-6b716f84331a testLoggerAdapter verbose: 39a60e0c-77b5-4ca1-ccc4-6b716f84331a: 2023-08-02T15:13:03.882Z [WARN] 39a60e0c-77b5-4ca1-ccc4-6b716f84331a: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: 39a60e0c-77b5-4ca1-ccc4-6b716f84331a: 2023-08-02T15:13:03.882Z [WARN] 39a60e0c-77b5-4ca1-ccc4-6b716f84331a: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.882Z [WARN] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: appendEntries rejected, sending older logs: peer="{Voter 39a60e0c-77b5-4ca1-ccc4-6b716f84331a 39a60e0c-77b5-4ca1-ccc4-6b716f84331a}" next=1 testing.go:149: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.882Z [WARN] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: appendEntries rejected, sending older logs: peer="{Voter 39a60e0c-77b5-4ca1-ccc4-6b716f84331a 39a60e0c-77b5-4ca1-ccc4-6b716f84331a}" next=1 testLoggerAdapter verbose: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.882Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: pipelining replication: peer="{Voter 39a60e0c-77b5-4ca1-ccc4-6b716f84331a 39a60e0c-77b5-4ca1-ccc4-6b716f84331a}" testing.go:149: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:03.882Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: pipelining replication: peer="{Voter 39a60e0c-77b5-4ca1-ccc4-6b716f84331a 39a60e0c-77b5-4ca1-ccc4-6b716f84331a}" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:03.882Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:13:03.882Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-08-02T15:13:03.987Z [INFO] stable state for Leader reached at 2023-08-02 15:13:03.882682506 +0000 UTC m=+15.978618839 (1 nodes), 756ns from start of poll, 379.162486ms from cluster start. Timeout at 2023-08-02 15:13:03.987735051 +0000 UTC m=+16.083671386, 105.052547ms after stability testing.go:149: cluster: 2023-08-02T15:13:03.987Z [INFO] stable state for Leader reached at 2023-08-02 15:13:03.882682506 +0000 UTC m=+15.978618839 (1 nodes), 756ns from start of poll, 379.162486ms from cluster start. Timeout at 2023-08-02 15:13:03.987735051 +0000 UTC m=+16.083671386, 105.052547ms after stability testLoggerAdapter verbose: cluster: 2023-08-02T15:13:03.987Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:13:03.987Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-08-02T15:13:04.093Z [INFO] stable state for Leader reached at 2023-08-02 15:13:03.987932063 +0000 UTC m=+16.083868389 (1 nodes), 1.09µs from start of poll, 484.412036ms from cluster start. Timeout at 2023-08-02 15:13:04.093544787 +0000 UTC m=+16.189481124, 105.612735ms after stability testing.go:149: cluster: 2023-08-02T15:13:04.093Z [INFO] stable state for Leader reached at 2023-08-02 15:13:03.987932063 +0000 UTC m=+16.083868389 (1 nodes), 1.09µs from start of poll, 484.412036ms from cluster start. Timeout at 2023-08-02 15:13:04.093544787 +0000 UTC m=+16.189481124, 105.612735ms after stability testLoggerAdapter verbose: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:04.093Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: updating configuration: command=RemoveServer server-id=39a60e0c-77b5-4ca1-ccc4-6b716f84331a server-addr= servers="[{Suffrage:Voter ID:ae3e7bcb-aec5-d032-06c8-d90ab73ce914 Address:ae3e7bcb-aec5-d032-06c8-d90ab73ce914} {Suffrage:Voter ID:db56337b-e877-27f1-3b4a-764bfa81b234 Address:db56337b-e877-27f1-3b4a-764bfa81b234}]" testing.go:149: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:04.093Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: updating configuration: command=RemoveServer server-id=39a60e0c-77b5-4ca1-ccc4-6b716f84331a server-addr= servers="[{Suffrage:Voter ID:ae3e7bcb-aec5-d032-06c8-d90ab73ce914 Address:ae3e7bcb-aec5-d032-06c8-d90ab73ce914} {Suffrage:Voter ID:db56337b-e877-27f1-3b4a-764bfa81b234 Address:db56337b-e877-27f1-3b4a-764bfa81b234}]" testLoggerAdapter verbose: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:04.093Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: removed peer, stopping replication: peer=39a60e0c-77b5-4ca1-ccc4-6b716f84331a last-index=4 testing.go:149: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:04.093Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: removed peer, stopping replication: peer=39a60e0c-77b5-4ca1-ccc4-6b716f84331a last-index=4 testLoggerAdapter verbose: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:04.093Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: aborting pipeline replication: peer="{Voter 39a60e0c-77b5-4ca1-ccc4-6b716f84331a 39a60e0c-77b5-4ca1-ccc4-6b716f84331a}" testing.go:149: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:04.093Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: aborting pipeline replication: peer="{Voter 39a60e0c-77b5-4ca1-ccc4-6b716f84331a 39a60e0c-77b5-4ca1-ccc4-6b716f84331a}" testLoggerAdapter verbose: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:04.094Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: aborting pipeline replication: peer="{Voter db56337b-e877-27f1-3b4a-764bfa81b234 db56337b-e877-27f1-3b4a-764bfa81b234}" testing.go:149: ae3e7bcb-aec5-d032-06c8-d90ab73ce914: 2023-08-02T15:13:04.094Z [INFO] ae3e7bcb-aec5-d032-06c8-d90ab73ce914: aborting pipeline replication: peer="{Voter db56337b-e877-27f1-3b4a-764bfa81b234 db56337b-e877-27f1-3b4a-764bfa81b234}" --- PASS: TestRaft_ProtocolVersion_Upgrade_1_2 (0.59s) === RUN TestRaft_ProtocolVersion_Upgrade_2_3 testLoggerAdapter verbose: 04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba: 2023-08-02T15:13:04.098Z [INFO] 04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba: initial configuration: index=1 servers="[{Suffrage:Voter ID:04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba Address:04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba} {Suffrage:Voter ID:814bc0ee-8056-34a8-4670-5bc20d0207e2 Address:814bc0ee-8056-34a8-4670-5bc20d0207e2}]" testing.go:149: 04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba: 2023-08-02T15:13:04.098Z [INFO] 04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba: initial configuration: index=1 servers="[{Suffrage:Voter ID:04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba Address:04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba} {Suffrage:Voter ID:814bc0ee-8056-34a8-4670-5bc20d0207e2 Address:814bc0ee-8056-34a8-4670-5bc20d0207e2}]" testLoggerAdapter verbose: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.098Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: initial configuration: index=1 servers="[{Suffrage:Voter ID:04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba Address:04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba} {Suffrage:Voter ID:814bc0ee-8056-34a8-4670-5bc20d0207e2 Address:814bc0ee-8056-34a8-4670-5bc20d0207e2}]" testing.go:149: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.098Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: initial configuration: index=1 servers="[{Suffrage:Voter ID:04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba Address:04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba} {Suffrage:Voter ID:814bc0ee-8056-34a8-4670-5bc20d0207e2 Address:814bc0ee-8056-34a8-4670-5bc20d0207e2}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:04.098Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-08-02T15:13:04.098Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: 04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba: 2023-08-02T15:13:04.098Z [INFO] 04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba: entering follower state: follower="Node at 04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba [Follower]" leader= testing.go:149: 04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba: 2023-08-02T15:13:04.098Z [INFO] 04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba: entering follower state: follower="Node at 04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba [Follower]" leader= testLoggerAdapter verbose: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.098Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: entering follower state: follower="Node at 814bc0ee-8056-34a8-4670-5bc20d0207e2 [Follower]" leader= testing.go:149: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.098Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: entering follower state: follower="Node at 814bc0ee-8056-34a8-4670-5bc20d0207e2 [Follower]" leader= testLoggerAdapter verbose: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.162Z [WARN] 814bc0ee-8056-34a8-4670-5bc20d0207e2: heartbeat timeout reached, starting election: last-leader= testing.go:149: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.162Z [WARN] 814bc0ee-8056-34a8-4670-5bc20d0207e2: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.162Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: entering candidate state: node="Node at 814bc0ee-8056-34a8-4670-5bc20d0207e2 [Candidate]" term=2 testing.go:149: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.162Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: entering candidate state: node="Node at 814bc0ee-8056-34a8-4670-5bc20d0207e2 [Candidate]" term=2 testLoggerAdapter verbose: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.162Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: election won: tally=2 testing.go:149: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.162Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: election won: tally=2 testLoggerAdapter verbose: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.162Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: entering leader state: leader="Node at 814bc0ee-8056-34a8-4670-5bc20d0207e2 [Leader]" testing.go:149: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.162Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: entering leader state: leader="Node at 814bc0ee-8056-34a8-4670-5bc20d0207e2 [Leader]" testLoggerAdapter verbose: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.162Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: added peer, starting replication: peer=04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba testing.go:149: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.162Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: added peer, starting replication: peer=04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba testLoggerAdapter verbose: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.163Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: pipelining replication: peer="{Voter 04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba 04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba}" testing.go:149: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.163Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: pipelining replication: peer="{Voter 04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba 04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba}" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:04.268Z [INFO] stable state for Follower reached at 2023-08-02 15:13:04.16297693 +0000 UTC m=+16.258913266 (1 nodes), 64.292947ms from start of poll, 64.767257ms from cluster start. Timeout at 2023-08-02 15:13:04.268014184 +0000 UTC m=+16.363950657, 105.037391ms after stability testing.go:149: cluster: 2023-08-02T15:13:04.268Z [INFO] stable state for Follower reached at 2023-08-02 15:13:04.16297693 +0000 UTC m=+16.258913266 (1 nodes), 64.292947ms from start of poll, 64.767257ms from cluster start. Timeout at 2023-08-02 15:13:04.268014184 +0000 UTC m=+16.363950657, 105.037391ms after stability testLoggerAdapter verbose: server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f: 2023-08-02T15:13:04.268Z [INFO] server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f: initial configuration: index=0 servers=[] testing.go:149: server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f: 2023-08-02T15:13:04.268Z [INFO] server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2023-08-02T15:13:04.268Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:13:04.268Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f: 2023-08-02T15:13:04.268Z [INFO] server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f: entering follower state: follower="Node at 5cad1438-9cdf-8bb0-9425-cab2bf1b077f [Follower]" leader= testing.go:149: server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f: 2023-08-02T15:13:04.268Z [INFO] server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f: entering follower state: follower="Node at 5cad1438-9cdf-8bb0-9425-cab2bf1b077f [Follower]" leader= testLoggerAdapter verbose: server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f: 2023-08-02T15:13:04.329Z [WARN] server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f: no known peers, aborting election testing.go:149: server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f: 2023-08-02T15:13:04.329Z [WARN] server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f: no known peers, aborting election testLoggerAdapter verbose: cluster: 2023-08-02T15:13:04.373Z [INFO] stable state for Leader reached at 2023-08-02 15:13:04.268464324 +0000 UTC m=+16.364400667 (1 nodes), 931ns from start of poll, 170.254658ms from cluster start. Timeout at 2023-08-02 15:13:04.373597537 +0000 UTC m=+16.469533875, 105.133208ms after stability testing.go:149: cluster: 2023-08-02T15:13:04.373Z [INFO] stable state for Leader reached at 2023-08-02 15:13:04.268464324 +0000 UTC m=+16.364400667 (1 nodes), 931ns from start of poll, 170.254658ms from cluster start. Timeout at 2023-08-02 15:13:04.373597537 +0000 UTC m=+16.469533875, 105.133208ms after stability testLoggerAdapter verbose: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.373Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: updating configuration: command=AddStaging server-id=server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f server-addr=5cad1438-9cdf-8bb0-9425-cab2bf1b077f servers="[{Suffrage:Voter ID:04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba Address:04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba} {Suffrage:Voter ID:814bc0ee-8056-34a8-4670-5bc20d0207e2 Address:814bc0ee-8056-34a8-4670-5bc20d0207e2} {Suffrage:Voter ID:server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f Address:5cad1438-9cdf-8bb0-9425-cab2bf1b077f}]" testing.go:149: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.373Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: updating configuration: command=AddStaging server-id=server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f server-addr=5cad1438-9cdf-8bb0-9425-cab2bf1b077f servers="[{Suffrage:Voter ID:04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba Address:04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba} {Suffrage:Voter ID:814bc0ee-8056-34a8-4670-5bc20d0207e2 Address:814bc0ee-8056-34a8-4670-5bc20d0207e2} {Suffrage:Voter ID:server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f Address:5cad1438-9cdf-8bb0-9425-cab2bf1b077f}]" testLoggerAdapter verbose: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.373Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: added peer, starting replication: peer=server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f testing.go:149: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.373Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: added peer, starting replication: peer=server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f testLoggerAdapter verbose: server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f: 2023-08-02T15:13:04.374Z [WARN] server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f: 2023-08-02T15:13:04.374Z [WARN] server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.374Z [WARN] 814bc0ee-8056-34a8-4670-5bc20d0207e2: appendEntries rejected, sending older logs: peer="{Voter server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f 5cad1438-9cdf-8bb0-9425-cab2bf1b077f}" next=1 testing.go:149: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.374Z [WARN] 814bc0ee-8056-34a8-4670-5bc20d0207e2: appendEntries rejected, sending older logs: peer="{Voter server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f 5cad1438-9cdf-8bb0-9425-cab2bf1b077f}" next=1 testLoggerAdapter verbose: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.374Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: pipelining replication: peer="{Voter server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f 5cad1438-9cdf-8bb0-9425-cab2bf1b077f}" testing.go:149: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.374Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: pipelining replication: peer="{Voter server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f 5cad1438-9cdf-8bb0-9425-cab2bf1b077f}" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:04.374Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:13:04.374Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-08-02T15:13:04.479Z [INFO] stable state for Leader reached at 2023-08-02 15:13:04.374400785 +0000 UTC m=+16.470337120 (1 nodes), 1.117µs from start of poll, 276.191111ms from cluster start. Timeout at 2023-08-02 15:13:04.479815172 +0000 UTC m=+16.575751509, 105.414389ms after stability testing.go:149: cluster: 2023-08-02T15:13:04.479Z [INFO] stable state for Leader reached at 2023-08-02 15:13:04.374400785 +0000 UTC m=+16.470337120 (1 nodes), 1.117µs from start of poll, 276.191111ms from cluster start. Timeout at 2023-08-02 15:13:04.479815172 +0000 UTC m=+16.575751509, 105.414389ms after stability testLoggerAdapter verbose: cluster: 2023-08-02T15:13:04.479Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:13:04.479Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-08-02T15:13:04.586Z [INFO] stable state for Leader reached at 2023-08-02 15:13:04.480014834 +0000 UTC m=+16.575951161 (1 nodes), 1.162µs from start of poll, 381.805152ms from cluster start. Timeout at 2023-08-02 15:13:04.586653775 +0000 UTC m=+16.682590101, 106.63894ms after stability testing.go:149: cluster: 2023-08-02T15:13:04.586Z [INFO] stable state for Leader reached at 2023-08-02 15:13:04.480014834 +0000 UTC m=+16.575951161 (1 nodes), 1.162µs from start of poll, 381.805152ms from cluster start. Timeout at 2023-08-02 15:13:04.586653775 +0000 UTC m=+16.682590101, 106.63894ms after stability testLoggerAdapter verbose: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.586Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: updating configuration: command=RemoveServer server-id=04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba server-addr= servers="[{Suffrage:Voter ID:814bc0ee-8056-34a8-4670-5bc20d0207e2 Address:814bc0ee-8056-34a8-4670-5bc20d0207e2} {Suffrage:Voter ID:server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f Address:5cad1438-9cdf-8bb0-9425-cab2bf1b077f}]" testing.go:149: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.586Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: updating configuration: command=RemoveServer server-id=04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba server-addr= servers="[{Suffrage:Voter ID:814bc0ee-8056-34a8-4670-5bc20d0207e2 Address:814bc0ee-8056-34a8-4670-5bc20d0207e2} {Suffrage:Voter ID:server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f Address:5cad1438-9cdf-8bb0-9425-cab2bf1b077f}]" testLoggerAdapter verbose: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.586Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: removed peer, stopping replication: peer=04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba last-index=4 testing.go:149: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.586Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: removed peer, stopping replication: peer=04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba last-index=4 testLoggerAdapter verbose: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.586Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: aborting pipeline replication: peer="{Voter 04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba 04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba}" testing.go:149: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.586Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: aborting pipeline replication: peer="{Voter 04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba 04cdb1b5-9a8e-c699-9cf0-a4c7ed455cba}" testLoggerAdapter verbose: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.587Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: aborting pipeline replication: peer="{Voter server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f 5cad1438-9cdf-8bb0-9425-cab2bf1b077f}" testing.go:149: 814bc0ee-8056-34a8-4670-5bc20d0207e2: 2023-08-02T15:13:04.587Z [INFO] 814bc0ee-8056-34a8-4670-5bc20d0207e2: aborting pipeline replication: peer="{Voter server-5cad1438-9cdf-8bb0-9425-cab2bf1b077f 5cad1438-9cdf-8bb0-9425-cab2bf1b077f}" --- PASS: TestRaft_ProtocolVersion_Upgrade_2_3 (0.49s) === RUN TestRaft_LeadershipTransferInProgress --- PASS: TestRaft_LeadershipTransferInProgress (0.00s) === RUN TestRaft_LeadershipTransferPickServer --- PASS: TestRaft_LeadershipTransferPickServer (0.00s) === RUN TestRaft_LeadershipTransfer testLoggerAdapter verbose: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.588Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-00c30f88-6bbe-af60-55fc-a32a9becb5d6 Address:00c30f88-6bbe-af60-55fc-a32a9becb5d6} {Suffrage:Voter ID:server-dd168698-36f7-4ac1-79e0-e76b3df6e677 Address:dd168698-36f7-4ac1-79e0-e76b3df6e677} {Suffrage:Voter ID:server-7c31cd11-c815-8385-066c-72f1002d3639 Address:7c31cd11-c815-8385-066c-72f1002d3639}]" testing.go:149: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.588Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-00c30f88-6bbe-af60-55fc-a32a9becb5d6 Address:00c30f88-6bbe-af60-55fc-a32a9becb5d6} {Suffrage:Voter ID:server-dd168698-36f7-4ac1-79e0-e76b3df6e677 Address:dd168698-36f7-4ac1-79e0-e76b3df6e677} {Suffrage:Voter ID:server-7c31cd11-c815-8385-066c-72f1002d3639 Address:7c31cd11-c815-8385-066c-72f1002d3639}]" testLoggerAdapter verbose: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.588Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-00c30f88-6bbe-af60-55fc-a32a9becb5d6 Address:00c30f88-6bbe-af60-55fc-a32a9becb5d6} {Suffrage:Voter ID:server-dd168698-36f7-4ac1-79e0-e76b3df6e677 Address:dd168698-36f7-4ac1-79e0-e76b3df6e677} {Suffrage:Voter ID:server-7c31cd11-c815-8385-066c-72f1002d3639 Address:7c31cd11-c815-8385-066c-72f1002d3639}]" testing.go:149: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.588Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-00c30f88-6bbe-af60-55fc-a32a9becb5d6 Address:00c30f88-6bbe-af60-55fc-a32a9becb5d6} {Suffrage:Voter ID:server-dd168698-36f7-4ac1-79e0-e76b3df6e677 Address:dd168698-36f7-4ac1-79e0-e76b3df6e677} {Suffrage:Voter ID:server-7c31cd11-c815-8385-066c-72f1002d3639 Address:7c31cd11-c815-8385-066c-72f1002d3639}]" testLoggerAdapter verbose: server-7c31cd11-c815-8385-066c-72f1002d3639: 2023-08-02T15:13:04.588Z [INFO] server-7c31cd11-c815-8385-066c-72f1002d3639: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-00c30f88-6bbe-af60-55fc-a32a9becb5d6 Address:00c30f88-6bbe-af60-55fc-a32a9becb5d6} {Suffrage:Voter ID:server-dd168698-36f7-4ac1-79e0-e76b3df6e677 Address:dd168698-36f7-4ac1-79e0-e76b3df6e677} {Suffrage:Voter ID:server-7c31cd11-c815-8385-066c-72f1002d3639 Address:7c31cd11-c815-8385-066c-72f1002d3639}]" testing.go:149: server-7c31cd11-c815-8385-066c-72f1002d3639: 2023-08-02T15:13:04.588Z [INFO] server-7c31cd11-c815-8385-066c-72f1002d3639: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-00c30f88-6bbe-af60-55fc-a32a9becb5d6 Address:00c30f88-6bbe-af60-55fc-a32a9becb5d6} {Suffrage:Voter ID:server-dd168698-36f7-4ac1-79e0-e76b3df6e677 Address:dd168698-36f7-4ac1-79e0-e76b3df6e677} {Suffrage:Voter ID:server-7c31cd11-c815-8385-066c-72f1002d3639 Address:7c31cd11-c815-8385-066c-72f1002d3639}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:04.588Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:13:04.588Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.588Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: entering follower state: follower="Node at 00c30f88-6bbe-af60-55fc-a32a9becb5d6 [Follower]" leader= testing.go:149: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.588Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: entering follower state: follower="Node at 00c30f88-6bbe-af60-55fc-a32a9becb5d6 [Follower]" leader= testLoggerAdapter verbose: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.588Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: entering follower state: follower="Node at dd168698-36f7-4ac1-79e0-e76b3df6e677 [Follower]" leader= testing.go:149: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.588Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: entering follower state: follower="Node at dd168698-36f7-4ac1-79e0-e76b3df6e677 [Follower]" leader= testLoggerAdapter verbose: server-7c31cd11-c815-8385-066c-72f1002d3639: 2023-08-02T15:13:04.589Z [INFO] server-7c31cd11-c815-8385-066c-72f1002d3639: entering follower state: follower="Node at 7c31cd11-c815-8385-066c-72f1002d3639 [Follower]" leader= testing.go:149: server-7c31cd11-c815-8385-066c-72f1002d3639: 2023-08-02T15:13:04.589Z [INFO] server-7c31cd11-c815-8385-066c-72f1002d3639: entering follower state: follower="Node at 7c31cd11-c815-8385-066c-72f1002d3639 [Follower]" leader= testLoggerAdapter verbose: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.656Z [WARN] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.656Z [WARN] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.657Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: entering candidate state: node="Node at dd168698-36f7-4ac1-79e0-e76b3df6e677 [Candidate]" term=2 testing.go:149: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.657Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: entering candidate state: node="Node at dd168698-36f7-4ac1-79e0-e76b3df6e677 [Candidate]" term=2 testLoggerAdapter verbose: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.657Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: election won: tally=2 testing.go:149: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.657Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: election won: tally=2 testLoggerAdapter verbose: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.657Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: entering leader state: leader="Node at dd168698-36f7-4ac1-79e0-e76b3df6e677 [Leader]" testing.go:149: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.657Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: entering leader state: leader="Node at dd168698-36f7-4ac1-79e0-e76b3df6e677 [Leader]" testLoggerAdapter verbose: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.657Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: added peer, starting replication: peer=server-00c30f88-6bbe-af60-55fc-a32a9becb5d6 testing.go:149: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.657Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: added peer, starting replication: peer=server-00c30f88-6bbe-af60-55fc-a32a9becb5d6 testLoggerAdapter verbose: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.657Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: added peer, starting replication: peer=server-7c31cd11-c815-8385-066c-72f1002d3639 testing.go:149: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.657Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: added peer, starting replication: peer=server-7c31cd11-c815-8385-066c-72f1002d3639 testLoggerAdapter verbose: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.657Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: pipelining replication: peer="{Voter server-7c31cd11-c815-8385-066c-72f1002d3639 7c31cd11-c815-8385-066c-72f1002d3639}" testing.go:149: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.657Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: pipelining replication: peer="{Voter server-7c31cd11-c815-8385-066c-72f1002d3639 7c31cd11-c815-8385-066c-72f1002d3639}" testLoggerAdapter verbose: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.657Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: pipelining replication: peer="{Voter server-00c30f88-6bbe-af60-55fc-a32a9becb5d6 00c30f88-6bbe-af60-55fc-a32a9becb5d6}" testing.go:149: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.657Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: pipelining replication: peer="{Voter server-00c30f88-6bbe-af60-55fc-a32a9becb5d6 00c30f88-6bbe-af60-55fc-a32a9becb5d6}" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:04.762Z [INFO] stable state for Leader reached at 2023-08-02 15:13:04.657614099 +0000 UTC m=+16.753550449 (1 nodes), 68.735688ms from start of poll, 69.305716ms from cluster start. Timeout at 2023-08-02 15:13:04.76272211 +0000 UTC m=+16.858658449, 105.108ms after stability testing.go:149: cluster: 2023-08-02T15:13:04.762Z [INFO] stable state for Leader reached at 2023-08-02 15:13:04.657614099 +0000 UTC m=+16.753550449 (1 nodes), 68.735688ms from start of poll, 69.305716ms from cluster start. Timeout at 2023-08-02 15:13:04.76272211 +0000 UTC m=+16.858658449, 105.108ms after stability testLoggerAdapter verbose: cluster: 2023-08-02T15:13:04.762Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:13:04.762Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-08-02T15:13:04.867Z [INFO] stable state for Leader reached at 2023-08-02 15:13:04.762905299 +0000 UTC m=+16.858841623 (1 nodes), 1.375µs from start of poll, 174.59689ms from cluster start. Timeout at 2023-08-02 15:13:04.867933101 +0000 UTC m=+16.963869428, 105.027805ms after stability testing.go:149: cluster: 2023-08-02T15:13:04.867Z [INFO] stable state for Leader reached at 2023-08-02 15:13:04.762905299 +0000 UTC m=+16.858841623 (1 nodes), 1.375µs from start of poll, 174.59689ms from cluster start. Timeout at 2023-08-02 15:13:04.867933101 +0000 UTC m=+16.963869428, 105.027805ms after stability testLoggerAdapter verbose: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.868Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: entering candidate state: node="Node at 00c30f88-6bbe-af60-55fc-a32a9becb5d6 [Candidate]" term=3 testing.go:149: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.868Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: entering candidate state: node="Node at 00c30f88-6bbe-af60-55fc-a32a9becb5d6 [Candidate]" term=3 testLoggerAdapter verbose: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.868Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: election won: tally=2 testing.go:149: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.868Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: election won: tally=2 testLoggerAdapter verbose: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.868Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: entering leader state: leader="Node at 00c30f88-6bbe-af60-55fc-a32a9becb5d6 [Leader]" testing.go:149: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.868Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: entering leader state: leader="Node at 00c30f88-6bbe-af60-55fc-a32a9becb5d6 [Leader]" testLoggerAdapter verbose: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.868Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: added peer, starting replication: peer=server-dd168698-36f7-4ac1-79e0-e76b3df6e677 testing.go:149: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.868Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: added peer, starting replication: peer=server-dd168698-36f7-4ac1-79e0-e76b3df6e677 testLoggerAdapter verbose: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.868Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: added peer, starting replication: peer=server-7c31cd11-c815-8385-066c-72f1002d3639 testing.go:149: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.868Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: added peer, starting replication: peer=server-7c31cd11-c815-8385-066c-72f1002d3639 testLoggerAdapter verbose: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.868Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: pipelining replication: peer="{Voter server-7c31cd11-c815-8385-066c-72f1002d3639 7c31cd11-c815-8385-066c-72f1002d3639}" testing.go:149: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.868Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: pipelining replication: peer="{Voter server-7c31cd11-c815-8385-066c-72f1002d3639 7c31cd11-c815-8385-066c-72f1002d3639}" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:04.868Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:13:04.868Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.868Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: entering follower state: follower="Node at dd168698-36f7-4ac1-79e0-e76b3df6e677 [Follower]" leader= testing.go:149: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.868Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: entering follower state: follower="Node at dd168698-36f7-4ac1-79e0-e76b3df6e677 [Follower]" leader= testLoggerAdapter verbose: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.868Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: aborting pipeline replication: peer="{Voter server-7c31cd11-c815-8385-066c-72f1002d3639 7c31cd11-c815-8385-066c-72f1002d3639}" testing.go:149: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.868Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: aborting pipeline replication: peer="{Voter server-7c31cd11-c815-8385-066c-72f1002d3639 7c31cd11-c815-8385-066c-72f1002d3639}" testLoggerAdapter verbose: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.868Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: pipelining replication: peer="{Voter server-dd168698-36f7-4ac1-79e0-e76b3df6e677 dd168698-36f7-4ac1-79e0-e76b3df6e677}" testing.go:149: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.868Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: pipelining replication: peer="{Voter server-dd168698-36f7-4ac1-79e0-e76b3df6e677 dd168698-36f7-4ac1-79e0-e76b3df6e677}" testLoggerAdapter verbose: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.868Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: aborting pipeline replication: peer="{Voter server-00c30f88-6bbe-af60-55fc-a32a9becb5d6 00c30f88-6bbe-af60-55fc-a32a9becb5d6}" testing.go:149: server-dd168698-36f7-4ac1-79e0-e76b3df6e677: 2023-08-02T15:13:04.868Z [INFO] server-dd168698-36f7-4ac1-79e0-e76b3df6e677: aborting pipeline replication: peer="{Voter server-00c30f88-6bbe-af60-55fc-a32a9becb5d6 00c30f88-6bbe-af60-55fc-a32a9becb5d6}" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:04.973Z [INFO] stable state for Leader reached at 2023-08-02 15:13:04.868673705 +0000 UTC m=+16.964610040 (1 nodes), 231.317µs from start of poll, 280.365307ms from cluster start. Timeout at 2023-08-02 15:13:04.973740457 +0000 UTC m=+17.069676817, 105.066777ms after stability testing.go:149: cluster: 2023-08-02T15:13:04.973Z [INFO] stable state for Leader reached at 2023-08-02 15:13:04.868673705 +0000 UTC m=+16.964610040 (1 nodes), 231.317µs from start of poll, 280.365307ms from cluster start. Timeout at 2023-08-02 15:13:04.973740457 +0000 UTC m=+17.069676817, 105.066777ms after stability testLoggerAdapter verbose: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.973Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: aborting pipeline replication: peer="{Voter server-7c31cd11-c815-8385-066c-72f1002d3639 7c31cd11-c815-8385-066c-72f1002d3639}" testing.go:149: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.973Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: aborting pipeline replication: peer="{Voter server-7c31cd11-c815-8385-066c-72f1002d3639 7c31cd11-c815-8385-066c-72f1002d3639}" testLoggerAdapter verbose: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.974Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: aborting pipeline replication: peer="{Voter server-dd168698-36f7-4ac1-79e0-e76b3df6e677 dd168698-36f7-4ac1-79e0-e76b3df6e677}" testing.go:149: server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: 2023-08-02T15:13:04.974Z [INFO] server-00c30f88-6bbe-af60-55fc-a32a9becb5d6: aborting pipeline replication: peer="{Voter server-dd168698-36f7-4ac1-79e0-e76b3df6e677 dd168698-36f7-4ac1-79e0-e76b3df6e677}" --- PASS: TestRaft_LeadershipTransfer (0.39s) === RUN TestRaft_LeadershipTransferWithOneNode testLoggerAdapter verbose: server-49667621-7452-7d38-0601-b1f5e9ec633d: 2023-08-02T15:13:04.974Z [INFO] server-49667621-7452-7d38-0601-b1f5e9ec633d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49667621-7452-7d38-0601-b1f5e9ec633d Address:49667621-7452-7d38-0601-b1f5e9ec633d}]" testing.go:149: server-49667621-7452-7d38-0601-b1f5e9ec633d: 2023-08-02T15:13:04.974Z [INFO] server-49667621-7452-7d38-0601-b1f5e9ec633d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49667621-7452-7d38-0601-b1f5e9ec633d Address:49667621-7452-7d38-0601-b1f5e9ec633d}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:04.974Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:13:04.974Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-49667621-7452-7d38-0601-b1f5e9ec633d: 2023-08-02T15:13:04.975Z [INFO] server-49667621-7452-7d38-0601-b1f5e9ec633d: entering follower state: follower="Node at 49667621-7452-7d38-0601-b1f5e9ec633d [Follower]" leader= testing.go:149: server-49667621-7452-7d38-0601-b1f5e9ec633d: 2023-08-02T15:13:04.975Z [INFO] server-49667621-7452-7d38-0601-b1f5e9ec633d: entering follower state: follower="Node at 49667621-7452-7d38-0601-b1f5e9ec633d [Follower]" leader= testLoggerAdapter verbose: server-49667621-7452-7d38-0601-b1f5e9ec633d: 2023-08-02T15:13:05.062Z [WARN] server-49667621-7452-7d38-0601-b1f5e9ec633d: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-49667621-7452-7d38-0601-b1f5e9ec633d: 2023-08-02T15:13:05.062Z [WARN] server-49667621-7452-7d38-0601-b1f5e9ec633d: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-49667621-7452-7d38-0601-b1f5e9ec633d: 2023-08-02T15:13:05.062Z [INFO] server-49667621-7452-7d38-0601-b1f5e9ec633d: entering candidate state: node="Node at 49667621-7452-7d38-0601-b1f5e9ec633d [Candidate]" term=2 testing.go:149: server-49667621-7452-7d38-0601-b1f5e9ec633d: 2023-08-02T15:13:05.062Z [INFO] server-49667621-7452-7d38-0601-b1f5e9ec633d: entering candidate state: node="Node at 49667621-7452-7d38-0601-b1f5e9ec633d [Candidate]" term=2 testLoggerAdapter verbose: server-49667621-7452-7d38-0601-b1f5e9ec633d: 2023-08-02T15:13:05.062Z [INFO] server-49667621-7452-7d38-0601-b1f5e9ec633d: election won: tally=1 testing.go:149: server-49667621-7452-7d38-0601-b1f5e9ec633d: 2023-08-02T15:13:05.062Z [INFO] server-49667621-7452-7d38-0601-b1f5e9ec633d: election won: tally=1 testLoggerAdapter verbose: server-49667621-7452-7d38-0601-b1f5e9ec633d: 2023-08-02T15:13:05.062Z [INFO] server-49667621-7452-7d38-0601-b1f5e9ec633d: entering leader state: leader="Node at 49667621-7452-7d38-0601-b1f5e9ec633d [Leader]" testing.go:149: server-49667621-7452-7d38-0601-b1f5e9ec633d: 2023-08-02T15:13:05.062Z [INFO] server-49667621-7452-7d38-0601-b1f5e9ec633d: entering leader state: leader="Node at 49667621-7452-7d38-0601-b1f5e9ec633d [Leader]" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:05.168Z [INFO] stable state for Leader reached at 2023-08-02 15:13:05.062802441 +0000 UTC m=+17.158738773 (1 nodes), 87.784567ms from start of poll, 88.086061ms from cluster start. Timeout at 2023-08-02 15:13:05.168717691 +0000 UTC m=+17.264654016, 105.915243ms after stability testing.go:149: cluster: 2023-08-02T15:13:05.168Z [INFO] stable state for Leader reached at 2023-08-02 15:13:05.062802441 +0000 UTC m=+17.158738773 (1 nodes), 87.784567ms from start of poll, 88.086061ms from cluster start. Timeout at 2023-08-02 15:13:05.168717691 +0000 UTC m=+17.264654016, 105.915243ms after stability --- PASS: TestRaft_LeadershipTransferWithOneNode (0.19s) === RUN TestRaft_LeadershipTransferWithSevenNodes testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.169Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd Address:16c4d59e-8dcd-06b6-dc57-8e1b5667abdd} {Suffrage:Voter ID:server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 Address:d2db4dc6-ece6-d891-330a-3a459dfa2bd5} {Suffrage:Voter ID:server-06e0cc33-930b-b8b4-1164-30f2237882ba Address:06e0cc33-930b-b8b4-1164-30f2237882ba} {Suffrage:Voter ID:server-130b2734-1a0a-f1c3-6a90-3fd880c1748a Address:130b2734-1a0a-f1c3-6a90-3fd880c1748a} {Suffrage:Voter ID:server-fcf65dbc-909a-19b9-2058-4b94c479035b Address:fcf65dbc-909a-19b9-2058-4b94c479035b} {Suffrage:Voter ID:server-00a5042f-6283-5ea0-b291-ddca3bab9553 Address:00a5042f-6283-5ea0-b291-ddca3bab9553} {Suffrage:Voter ID:server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd Address:74a2e60e-7ba2-6dfb-22b7-2d76da0718bd}]" testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.169Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd Address:16c4d59e-8dcd-06b6-dc57-8e1b5667abdd} {Suffrage:Voter ID:server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 Address:d2db4dc6-ece6-d891-330a-3a459dfa2bd5} {Suffrage:Voter ID:server-06e0cc33-930b-b8b4-1164-30f2237882ba Address:06e0cc33-930b-b8b4-1164-30f2237882ba} {Suffrage:Voter ID:server-130b2734-1a0a-f1c3-6a90-3fd880c1748a Address:130b2734-1a0a-f1c3-6a90-3fd880c1748a} {Suffrage:Voter ID:server-fcf65dbc-909a-19b9-2058-4b94c479035b Address:fcf65dbc-909a-19b9-2058-4b94c479035b} {Suffrage:Voter ID:server-00a5042f-6283-5ea0-b291-ddca3bab9553 Address:00a5042f-6283-5ea0-b291-ddca3bab9553} {Suffrage:Voter ID:server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd Address:74a2e60e-7ba2-6dfb-22b7-2d76da0718bd}]" testLoggerAdapter verbose: server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5: 2023-08-02T15:13:05.170Z [INFO] server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd Address:16c4d59e-8dcd-06b6-dc57-8e1b5667abdd} {Suffrage:Voter ID:server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 Address:d2db4dc6-ece6-d891-330a-3a459dfa2bd5} {Suffrage:Voter ID:server-06e0cc33-930b-b8b4-1164-30f2237882ba Address:06e0cc33-930b-b8b4-1164-30f2237882ba} {Suffrage:Voter ID:server-130b2734-1a0a-f1c3-6a90-3fd880c1748a Address:130b2734-1a0a-f1c3-6a90-3fd880c1748a} {Suffrage:Voter ID:server-fcf65dbc-909a-19b9-2058-4b94c479035b Address:fcf65dbc-909a-19b9-2058-4b94c479035b} {Suffrage:Voter ID:server-00a5042f-6283-5ea0-b291-ddca3bab9553 Address:00a5042f-6283-5ea0-b291-ddca3bab9553} {Suffrage:Voter ID:server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd Address:74a2e60e-7ba2-6dfb-22b7-2d76da0718bd}]" testing.go:149: server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5: 2023-08-02T15:13:05.170Z [INFO] server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd Address:16c4d59e-8dcd-06b6-dc57-8e1b5667abdd} {Suffrage:Voter ID:server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 Address:d2db4dc6-ece6-d891-330a-3a459dfa2bd5} {Suffrage:Voter ID:server-06e0cc33-930b-b8b4-1164-30f2237882ba Address:06e0cc33-930b-b8b4-1164-30f2237882ba} {Suffrage:Voter ID:server-130b2734-1a0a-f1c3-6a90-3fd880c1748a Address:130b2734-1a0a-f1c3-6a90-3fd880c1748a} {Suffrage:Voter ID:server-fcf65dbc-909a-19b9-2058-4b94c479035b Address:fcf65dbc-909a-19b9-2058-4b94c479035b} {Suffrage:Voter ID:server-00a5042f-6283-5ea0-b291-ddca3bab9553 Address:00a5042f-6283-5ea0-b291-ddca3bab9553} {Suffrage:Voter ID:server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd Address:74a2e60e-7ba2-6dfb-22b7-2d76da0718bd}]" testLoggerAdapter verbose: server-06e0cc33-930b-b8b4-1164-30f2237882ba: 2023-08-02T15:13:05.170Z [INFO] server-06e0cc33-930b-b8b4-1164-30f2237882ba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd Address:16c4d59e-8dcd-06b6-dc57-8e1b5667abdd} {Suffrage:Voter ID:server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 Address:d2db4dc6-ece6-d891-330a-3a459dfa2bd5} {Suffrage:Voter ID:server-06e0cc33-930b-b8b4-1164-30f2237882ba Address:06e0cc33-930b-b8b4-1164-30f2237882ba} {Suffrage:Voter ID:server-130b2734-1a0a-f1c3-6a90-3fd880c1748a Address:130b2734-1a0a-f1c3-6a90-3fd880c1748a} {Suffrage:Voter ID:server-fcf65dbc-909a-19b9-2058-4b94c479035b Address:fcf65dbc-909a-19b9-2058-4b94c479035b} {Suffrage:Voter ID:server-00a5042f-6283-5ea0-b291-ddca3bab9553 Address:00a5042f-6283-5ea0-b291-ddca3bab9553} {Suffrage:Voter ID:server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd Address:74a2e60e-7ba2-6dfb-22b7-2d76da0718bd}]" testing.go:149: server-06e0cc33-930b-b8b4-1164-30f2237882ba: 2023-08-02T15:13:05.170Z [INFO] server-06e0cc33-930b-b8b4-1164-30f2237882ba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd Address:16c4d59e-8dcd-06b6-dc57-8e1b5667abdd} {Suffrage:Voter ID:server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 Address:d2db4dc6-ece6-d891-330a-3a459dfa2bd5} {Suffrage:Voter ID:server-06e0cc33-930b-b8b4-1164-30f2237882ba Address:06e0cc33-930b-b8b4-1164-30f2237882ba} {Suffrage:Voter ID:server-130b2734-1a0a-f1c3-6a90-3fd880c1748a Address:130b2734-1a0a-f1c3-6a90-3fd880c1748a} {Suffrage:Voter ID:server-fcf65dbc-909a-19b9-2058-4b94c479035b Address:fcf65dbc-909a-19b9-2058-4b94c479035b} {Suffrage:Voter ID:server-00a5042f-6283-5ea0-b291-ddca3bab9553 Address:00a5042f-6283-5ea0-b291-ddca3bab9553} {Suffrage:Voter ID:server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd Address:74a2e60e-7ba2-6dfb-22b7-2d76da0718bd}]" testLoggerAdapter verbose: server-130b2734-1a0a-f1c3-6a90-3fd880c1748a: 2023-08-02T15:13:05.170Z [INFO] server-130b2734-1a0a-f1c3-6a90-3fd880c1748a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd Address:16c4d59e-8dcd-06b6-dc57-8e1b5667abdd} {Suffrage:Voter ID:server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 Address:d2db4dc6-ece6-d891-330a-3a459dfa2bd5} {Suffrage:Voter ID:server-06e0cc33-930b-b8b4-1164-30f2237882ba Address:06e0cc33-930b-b8b4-1164-30f2237882ba} {Suffrage:Voter ID:server-130b2734-1a0a-f1c3-6a90-3fd880c1748a Address:130b2734-1a0a-f1c3-6a90-3fd880c1748a} {Suffrage:Voter ID:server-fcf65dbc-909a-19b9-2058-4b94c479035b Address:fcf65dbc-909a-19b9-2058-4b94c479035b} {Suffrage:Voter ID:server-00a5042f-6283-5ea0-b291-ddca3bab9553 Address:00a5042f-6283-5ea0-b291-ddca3bab9553} {Suffrage:Voter ID:server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd Address:74a2e60e-7ba2-6dfb-22b7-2d76da0718bd}]" testing.go:149: server-130b2734-1a0a-f1c3-6a90-3fd880c1748a: 2023-08-02T15:13:05.170Z [INFO] server-130b2734-1a0a-f1c3-6a90-3fd880c1748a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd Address:16c4d59e-8dcd-06b6-dc57-8e1b5667abdd} {Suffrage:Voter ID:server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 Address:d2db4dc6-ece6-d891-330a-3a459dfa2bd5} {Suffrage:Voter ID:server-06e0cc33-930b-b8b4-1164-30f2237882ba Address:06e0cc33-930b-b8b4-1164-30f2237882ba} {Suffrage:Voter ID:server-130b2734-1a0a-f1c3-6a90-3fd880c1748a Address:130b2734-1a0a-f1c3-6a90-3fd880c1748a} {Suffrage:Voter ID:server-fcf65dbc-909a-19b9-2058-4b94c479035b Address:fcf65dbc-909a-19b9-2058-4b94c479035b} {Suffrage:Voter ID:server-00a5042f-6283-5ea0-b291-ddca3bab9553 Address:00a5042f-6283-5ea0-b291-ddca3bab9553} {Suffrage:Voter ID:server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd Address:74a2e60e-7ba2-6dfb-22b7-2d76da0718bd}]" testLoggerAdapter verbose: server-fcf65dbc-909a-19b9-2058-4b94c479035b: 2023-08-02T15:13:05.170Z [INFO] server-fcf65dbc-909a-19b9-2058-4b94c479035b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd Address:16c4d59e-8dcd-06b6-dc57-8e1b5667abdd} {Suffrage:Voter ID:server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 Address:d2db4dc6-ece6-d891-330a-3a459dfa2bd5} {Suffrage:Voter ID:server-06e0cc33-930b-b8b4-1164-30f2237882ba Address:06e0cc33-930b-b8b4-1164-30f2237882ba} {Suffrage:Voter ID:server-130b2734-1a0a-f1c3-6a90-3fd880c1748a Address:130b2734-1a0a-f1c3-6a90-3fd880c1748a} {Suffrage:Voter ID:server-fcf65dbc-909a-19b9-2058-4b94c479035b Address:fcf65dbc-909a-19b9-2058-4b94c479035b} {Suffrage:Voter ID:server-00a5042f-6283-5ea0-b291-ddca3bab9553 Address:00a5042f-6283-5ea0-b291-ddca3bab9553} {Suffrage:Voter ID:server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd Address:74a2e60e-7ba2-6dfb-22b7-2d76da0718bd}]" testing.go:149: server-fcf65dbc-909a-19b9-2058-4b94c479035b: 2023-08-02T15:13:05.170Z [INFO] server-fcf65dbc-909a-19b9-2058-4b94c479035b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd Address:16c4d59e-8dcd-06b6-dc57-8e1b5667abdd} {Suffrage:Voter ID:server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 Address:d2db4dc6-ece6-d891-330a-3a459dfa2bd5} {Suffrage:Voter ID:server-06e0cc33-930b-b8b4-1164-30f2237882ba Address:06e0cc33-930b-b8b4-1164-30f2237882ba} {Suffrage:Voter ID:server-130b2734-1a0a-f1c3-6a90-3fd880c1748a Address:130b2734-1a0a-f1c3-6a90-3fd880c1748a} {Suffrage:Voter ID:server-fcf65dbc-909a-19b9-2058-4b94c479035b Address:fcf65dbc-909a-19b9-2058-4b94c479035b} {Suffrage:Voter ID:server-00a5042f-6283-5ea0-b291-ddca3bab9553 Address:00a5042f-6283-5ea0-b291-ddca3bab9553} {Suffrage:Voter ID:server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd Address:74a2e60e-7ba2-6dfb-22b7-2d76da0718bd}]" testLoggerAdapter verbose: server-00a5042f-6283-5ea0-b291-ddca3bab9553: 2023-08-02T15:13:05.170Z [INFO] server-00a5042f-6283-5ea0-b291-ddca3bab9553: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd Address:16c4d59e-8dcd-06b6-dc57-8e1b5667abdd} {Suffrage:Voter ID:server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 Address:d2db4dc6-ece6-d891-330a-3a459dfa2bd5} {Suffrage:Voter ID:server-06e0cc33-930b-b8b4-1164-30f2237882ba Address:06e0cc33-930b-b8b4-1164-30f2237882ba} {Suffrage:Voter ID:server-130b2734-1a0a-f1c3-6a90-3fd880c1748a Address:130b2734-1a0a-f1c3-6a90-3fd880c1748a} {Suffrage:Voter ID:server-fcf65dbc-909a-19b9-2058-4b94c479035b Address:fcf65dbc-909a-19b9-2058-4b94c479035b} {Suffrage:Voter ID:server-00a5042f-6283-5ea0-b291-ddca3bab9553 Address:00a5042f-6283-5ea0-b291-ddca3bab9553} {Suffrage:Voter ID:server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd Address:74a2e60e-7ba2-6dfb-22b7-2d76da0718bd}]" testing.go:149: server-00a5042f-6283-5ea0-b291-ddca3bab9553: 2023-08-02T15:13:05.170Z [INFO] server-00a5042f-6283-5ea0-b291-ddca3bab9553: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd Address:16c4d59e-8dcd-06b6-dc57-8e1b5667abdd} {Suffrage:Voter ID:server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 Address:d2db4dc6-ece6-d891-330a-3a459dfa2bd5} {Suffrage:Voter ID:server-06e0cc33-930b-b8b4-1164-30f2237882ba Address:06e0cc33-930b-b8b4-1164-30f2237882ba} {Suffrage:Voter ID:server-130b2734-1a0a-f1c3-6a90-3fd880c1748a Address:130b2734-1a0a-f1c3-6a90-3fd880c1748a} {Suffrage:Voter ID:server-fcf65dbc-909a-19b9-2058-4b94c479035b Address:fcf65dbc-909a-19b9-2058-4b94c479035b} {Suffrage:Voter ID:server-00a5042f-6283-5ea0-b291-ddca3bab9553 Address:00a5042f-6283-5ea0-b291-ddca3bab9553} {Suffrage:Voter ID:server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd Address:74a2e60e-7ba2-6dfb-22b7-2d76da0718bd}]" testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.170Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd Address:16c4d59e-8dcd-06b6-dc57-8e1b5667abdd} {Suffrage:Voter ID:server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 Address:d2db4dc6-ece6-d891-330a-3a459dfa2bd5} {Suffrage:Voter ID:server-06e0cc33-930b-b8b4-1164-30f2237882ba Address:06e0cc33-930b-b8b4-1164-30f2237882ba} {Suffrage:Voter ID:server-130b2734-1a0a-f1c3-6a90-3fd880c1748a Address:130b2734-1a0a-f1c3-6a90-3fd880c1748a} {Suffrage:Voter ID:server-fcf65dbc-909a-19b9-2058-4b94c479035b Address:fcf65dbc-909a-19b9-2058-4b94c479035b} {Suffrage:Voter ID:server-00a5042f-6283-5ea0-b291-ddca3bab9553 Address:00a5042f-6283-5ea0-b291-ddca3bab9553} {Suffrage:Voter ID:server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd Address:74a2e60e-7ba2-6dfb-22b7-2d76da0718bd}]" testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.170Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd Address:16c4d59e-8dcd-06b6-dc57-8e1b5667abdd} {Suffrage:Voter ID:server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 Address:d2db4dc6-ece6-d891-330a-3a459dfa2bd5} {Suffrage:Voter ID:server-06e0cc33-930b-b8b4-1164-30f2237882ba Address:06e0cc33-930b-b8b4-1164-30f2237882ba} {Suffrage:Voter ID:server-130b2734-1a0a-f1c3-6a90-3fd880c1748a Address:130b2734-1a0a-f1c3-6a90-3fd880c1748a} {Suffrage:Voter ID:server-fcf65dbc-909a-19b9-2058-4b94c479035b Address:fcf65dbc-909a-19b9-2058-4b94c479035b} {Suffrage:Voter ID:server-00a5042f-6283-5ea0-b291-ddca3bab9553 Address:00a5042f-6283-5ea0-b291-ddca3bab9553} {Suffrage:Voter ID:server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd Address:74a2e60e-7ba2-6dfb-22b7-2d76da0718bd}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:05.171Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:13:05.171Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.171Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: entering follower state: follower="Node at 16c4d59e-8dcd-06b6-dc57-8e1b5667abdd [Follower]" leader= testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.171Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: entering follower state: follower="Node at 16c4d59e-8dcd-06b6-dc57-8e1b5667abdd [Follower]" leader= testLoggerAdapter verbose: server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5: 2023-08-02T15:13:05.171Z [INFO] server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5: entering follower state: follower="Node at d2db4dc6-ece6-d891-330a-3a459dfa2bd5 [Follower]" leader= testing.go:149: server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5: 2023-08-02T15:13:05.171Z [INFO] server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5: entering follower state: follower="Node at d2db4dc6-ece6-d891-330a-3a459dfa2bd5 [Follower]" leader= testLoggerAdapter verbose: server-06e0cc33-930b-b8b4-1164-30f2237882ba: 2023-08-02T15:13:05.171Z [INFO] server-06e0cc33-930b-b8b4-1164-30f2237882ba: entering follower state: follower="Node at 06e0cc33-930b-b8b4-1164-30f2237882ba [Follower]" leader= testing.go:149: server-06e0cc33-930b-b8b4-1164-30f2237882ba: 2023-08-02T15:13:05.171Z [INFO] server-06e0cc33-930b-b8b4-1164-30f2237882ba: entering follower state: follower="Node at 06e0cc33-930b-b8b4-1164-30f2237882ba [Follower]" leader= testLoggerAdapter verbose: server-130b2734-1a0a-f1c3-6a90-3fd880c1748a: 2023-08-02T15:13:05.171Z [INFO] server-130b2734-1a0a-f1c3-6a90-3fd880c1748a: entering follower state: follower="Node at 130b2734-1a0a-f1c3-6a90-3fd880c1748a [Follower]" leader= testing.go:149: server-130b2734-1a0a-f1c3-6a90-3fd880c1748a: 2023-08-02T15:13:05.171Z [INFO] server-130b2734-1a0a-f1c3-6a90-3fd880c1748a: entering follower state: follower="Node at 130b2734-1a0a-f1c3-6a90-3fd880c1748a [Follower]" leader= testLoggerAdapter verbose: server-fcf65dbc-909a-19b9-2058-4b94c479035b: 2023-08-02T15:13:05.171Z [INFO] server-fcf65dbc-909a-19b9-2058-4b94c479035b: entering follower state: follower="Node at fcf65dbc-909a-19b9-2058-4b94c479035b [Follower]" leader= testing.go:149: server-fcf65dbc-909a-19b9-2058-4b94c479035b: 2023-08-02T15:13:05.171Z [INFO] server-fcf65dbc-909a-19b9-2058-4b94c479035b: entering follower state: follower="Node at fcf65dbc-909a-19b9-2058-4b94c479035b [Follower]" leader= testLoggerAdapter verbose: server-00a5042f-6283-5ea0-b291-ddca3bab9553: 2023-08-02T15:13:05.171Z [INFO] server-00a5042f-6283-5ea0-b291-ddca3bab9553: entering follower state: follower="Node at 00a5042f-6283-5ea0-b291-ddca3bab9553 [Follower]" leader= testing.go:149: server-00a5042f-6283-5ea0-b291-ddca3bab9553: 2023-08-02T15:13:05.171Z [INFO] server-00a5042f-6283-5ea0-b291-ddca3bab9553: entering follower state: follower="Node at 00a5042f-6283-5ea0-b291-ddca3bab9553 [Follower]" leader= testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.171Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: entering follower state: follower="Node at 74a2e60e-7ba2-6dfb-22b7-2d76da0718bd [Follower]" leader= testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.171Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: entering follower state: follower="Node at 74a2e60e-7ba2-6dfb-22b7-2d76da0718bd [Follower]" leader= testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.222Z [WARN] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.222Z [WARN] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.222Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: entering candidate state: node="Node at 74a2e60e-7ba2-6dfb-22b7-2d76da0718bd [Candidate]" term=2 testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.222Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: entering candidate state: node="Node at 74a2e60e-7ba2-6dfb-22b7-2d76da0718bd [Candidate]" term=2 testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.222Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: election won: tally=4 testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.222Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: election won: tally=4 testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.222Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: entering leader state: leader="Node at 74a2e60e-7ba2-6dfb-22b7-2d76da0718bd [Leader]" testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.222Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: entering leader state: leader="Node at 74a2e60e-7ba2-6dfb-22b7-2d76da0718bd [Leader]" testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.222Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: added peer, starting replication: peer=server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.222Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: added peer, starting replication: peer=server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.222Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: added peer, starting replication: peer=server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.222Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: added peer, starting replication: peer=server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.230Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: pipelining replication: peer="{Voter server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd 16c4d59e-8dcd-06b6-dc57-8e1b5667abdd}" testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.230Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: pipelining replication: peer="{Voter server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd 16c4d59e-8dcd-06b6-dc57-8e1b5667abdd}" testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.230Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: added peer, starting replication: peer=server-06e0cc33-930b-b8b4-1164-30f2237882ba testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.230Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: added peer, starting replication: peer=server-06e0cc33-930b-b8b4-1164-30f2237882ba testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.235Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: added peer, starting replication: peer=server-130b2734-1a0a-f1c3-6a90-3fd880c1748a testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.235Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: added peer, starting replication: peer=server-130b2734-1a0a-f1c3-6a90-3fd880c1748a testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.235Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: added peer, starting replication: peer=server-fcf65dbc-909a-19b9-2058-4b94c479035b testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.235Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: added peer, starting replication: peer=server-fcf65dbc-909a-19b9-2058-4b94c479035b testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.235Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: added peer, starting replication: peer=server-00a5042f-6283-5ea0-b291-ddca3bab9553 testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.235Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: added peer, starting replication: peer=server-00a5042f-6283-5ea0-b291-ddca3bab9553 testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.230Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: pipelining replication: peer="{Voter server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 d2db4dc6-ece6-d891-330a-3a459dfa2bd5}" testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.230Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: pipelining replication: peer="{Voter server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 d2db4dc6-ece6-d891-330a-3a459dfa2bd5}" testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.235Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: pipelining replication: peer="{Voter server-fcf65dbc-909a-19b9-2058-4b94c479035b fcf65dbc-909a-19b9-2058-4b94c479035b}" testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.235Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: pipelining replication: peer="{Voter server-fcf65dbc-909a-19b9-2058-4b94c479035b fcf65dbc-909a-19b9-2058-4b94c479035b}" testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.236Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: pipelining replication: peer="{Voter server-130b2734-1a0a-f1c3-6a90-3fd880c1748a 130b2734-1a0a-f1c3-6a90-3fd880c1748a}" testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.236Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: pipelining replication: peer="{Voter server-130b2734-1a0a-f1c3-6a90-3fd880c1748a 130b2734-1a0a-f1c3-6a90-3fd880c1748a}" testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.236Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: pipelining replication: peer="{Voter server-00a5042f-6283-5ea0-b291-ddca3bab9553 00a5042f-6283-5ea0-b291-ddca3bab9553}" testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.236Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: pipelining replication: peer="{Voter server-00a5042f-6283-5ea0-b291-ddca3bab9553 00a5042f-6283-5ea0-b291-ddca3bab9553}" testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.236Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: pipelining replication: peer="{Voter server-06e0cc33-930b-b8b4-1164-30f2237882ba 06e0cc33-930b-b8b4-1164-30f2237882ba}" testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.236Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: pipelining replication: peer="{Voter server-06e0cc33-930b-b8b4-1164-30f2237882ba 06e0cc33-930b-b8b4-1164-30f2237882ba}" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:05.338Z [INFO] stable state for Leader reached at 2023-08-02 15:13:05.23103949 +0000 UTC m=+17.326975817 (1 nodes), 59.949266ms from start of poll, 61.29534ms from cluster start. Timeout at 2023-08-02 15:13:05.33855993 +0000 UTC m=+17.434496264, 107.520447ms after stability testing.go:149: cluster: 2023-08-02T15:13:05.338Z [INFO] stable state for Leader reached at 2023-08-02 15:13:05.23103949 +0000 UTC m=+17.326975817 (1 nodes), 59.949266ms from start of poll, 61.29534ms from cluster start. Timeout at 2023-08-02 15:13:05.33855993 +0000 UTC m=+17.434496264, 107.520447ms after stability testLoggerAdapter verbose: cluster: 2023-08-02T15:13:05.338Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-08-02T15:13:05.338Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2023-08-02T15:13:05.443Z [INFO] stable state for Follower reached at 2023-08-02 15:13:05.338740762 +0000 UTC m=+17.434677088 (6 nodes), 3.116µs from start of poll, 168.996611ms from cluster start. Timeout at 2023-08-02 15:13:05.443830348 +0000 UTC m=+17.539766675, 105.089587ms after stability testing.go:149: cluster: 2023-08-02T15:13:05.443Z [INFO] stable state for Follower reached at 2023-08-02 15:13:05.338740762 +0000 UTC m=+17.434677088 (6 nodes), 3.116µs from start of poll, 168.996611ms from cluster start. Timeout at 2023-08-02 15:13:05.443830348 +0000 UTC m=+17.539766675, 105.089587ms after stability testLoggerAdapter verbose: cluster: 2023-08-02T15:13:05.443Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:13:05.443Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-08-02T15:13:05.549Z [INFO] stable state for Leader reached at 2023-08-02 15:13:05.444018076 +0000 UTC m=+17.539954403 (1 nodes), 2.396µs from start of poll, 274.273926ms from cluster start. Timeout at 2023-08-02 15:13:05.549035874 +0000 UTC m=+17.644972204, 105.017801ms after stability testing.go:149: cluster: 2023-08-02T15:13:05.549Z [INFO] stable state for Leader reached at 2023-08-02 15:13:05.444018076 +0000 UTC m=+17.539954403 (1 nodes), 2.396µs from start of poll, 274.273926ms from cluster start. Timeout at 2023-08-02 15:13:05.549035874 +0000 UTC m=+17.644972204, 105.017801ms after stability testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: entering candidate state: node="Node at 16c4d59e-8dcd-06b6-dc57-8e1b5667abdd [Candidate]" term=3 testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: entering candidate state: node="Node at 16c4d59e-8dcd-06b6-dc57-8e1b5667abdd [Candidate]" term=3 testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.549Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: entering follower state: follower="Node at 74a2e60e-7ba2-6dfb-22b7-2d76da0718bd [Follower]" leader= testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.549Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: entering follower state: follower="Node at 74a2e60e-7ba2-6dfb-22b7-2d76da0718bd [Follower]" leader= testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.549Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: aborting pipeline replication: peer="{Voter server-00a5042f-6283-5ea0-b291-ddca3bab9553 00a5042f-6283-5ea0-b291-ddca3bab9553}" testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.549Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: aborting pipeline replication: peer="{Voter server-00a5042f-6283-5ea0-b291-ddca3bab9553 00a5042f-6283-5ea0-b291-ddca3bab9553}" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:05.549Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:13:05.549Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.549Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: aborting pipeline replication: peer="{Voter server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd 16c4d59e-8dcd-06b6-dc57-8e1b5667abdd}" testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.549Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: aborting pipeline replication: peer="{Voter server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd 16c4d59e-8dcd-06b6-dc57-8e1b5667abdd}" testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.549Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: aborting pipeline replication: peer="{Voter server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 d2db4dc6-ece6-d891-330a-3a459dfa2bd5}" testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.549Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: aborting pipeline replication: peer="{Voter server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 d2db4dc6-ece6-d891-330a-3a459dfa2bd5}" testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.549Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: aborting pipeline replication: peer="{Voter server-06e0cc33-930b-b8b4-1164-30f2237882ba 06e0cc33-930b-b8b4-1164-30f2237882ba}" testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.549Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: aborting pipeline replication: peer="{Voter server-06e0cc33-930b-b8b4-1164-30f2237882ba 06e0cc33-930b-b8b4-1164-30f2237882ba}" testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: election won: tally=4 testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.549Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: aborting pipeline replication: peer="{Voter server-130b2734-1a0a-f1c3-6a90-3fd880c1748a 130b2734-1a0a-f1c3-6a90-3fd880c1748a}" testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.549Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: aborting pipeline replication: peer="{Voter server-130b2734-1a0a-f1c3-6a90-3fd880c1748a 130b2734-1a0a-f1c3-6a90-3fd880c1748a}" testLoggerAdapter verbose: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.549Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: aborting pipeline replication: peer="{Voter server-fcf65dbc-909a-19b9-2058-4b94c479035b fcf65dbc-909a-19b9-2058-4b94c479035b}" testing.go:149: server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: 2023-08-02T15:13:05.549Z [INFO] server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd: aborting pipeline replication: peer="{Voter server-fcf65dbc-909a-19b9-2058-4b94c479035b fcf65dbc-909a-19b9-2058-4b94c479035b}" testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: election won: tally=4 testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: entering leader state: leader="Node at 16c4d59e-8dcd-06b6-dc57-8e1b5667abdd [Leader]" testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: entering leader state: leader="Node at 16c4d59e-8dcd-06b6-dc57-8e1b5667abdd [Leader]" testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: added peer, starting replication: peer=server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: added peer, starting replication: peer=server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: added peer, starting replication: peer=server-06e0cc33-930b-b8b4-1164-30f2237882ba testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: added peer, starting replication: peer=server-06e0cc33-930b-b8b4-1164-30f2237882ba testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: added peer, starting replication: peer=server-130b2734-1a0a-f1c3-6a90-3fd880c1748a testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: added peer, starting replication: peer=server-130b2734-1a0a-f1c3-6a90-3fd880c1748a testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: added peer, starting replication: peer=server-fcf65dbc-909a-19b9-2058-4b94c479035b testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: added peer, starting replication: peer=server-fcf65dbc-909a-19b9-2058-4b94c479035b testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: added peer, starting replication: peer=server-00a5042f-6283-5ea0-b291-ddca3bab9553 testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: added peer, starting replication: peer=server-00a5042f-6283-5ea0-b291-ddca3bab9553 testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: added peer, starting replication: peer=server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: added peer, starting replication: peer=server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.550Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: pipelining replication: peer="{Voter server-fcf65dbc-909a-19b9-2058-4b94c479035b fcf65dbc-909a-19b9-2058-4b94c479035b}" testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.550Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: pipelining replication: peer="{Voter server-fcf65dbc-909a-19b9-2058-4b94c479035b fcf65dbc-909a-19b9-2058-4b94c479035b}" testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.550Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: pipelining replication: peer="{Voter server-00a5042f-6283-5ea0-b291-ddca3bab9553 00a5042f-6283-5ea0-b291-ddca3bab9553}" testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.550Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: pipelining replication: peer="{Voter server-00a5042f-6283-5ea0-b291-ddca3bab9553 00a5042f-6283-5ea0-b291-ddca3bab9553}" testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.550Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: pipelining replication: peer="{Voter server-130b2734-1a0a-f1c3-6a90-3fd880c1748a 130b2734-1a0a-f1c3-6a90-3fd880c1748a}" testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.550Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: pipelining replication: peer="{Voter server-130b2734-1a0a-f1c3-6a90-3fd880c1748a 130b2734-1a0a-f1c3-6a90-3fd880c1748a}" testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.550Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: pipelining replication: peer="{Voter server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd 74a2e60e-7ba2-6dfb-22b7-2d76da0718bd}" testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.550Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: pipelining replication: peer="{Voter server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd 74a2e60e-7ba2-6dfb-22b7-2d76da0718bd}" testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: pipelining replication: peer="{Voter server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 d2db4dc6-ece6-d891-330a-3a459dfa2bd5}" testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: pipelining replication: peer="{Voter server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 d2db4dc6-ece6-d891-330a-3a459dfa2bd5}" testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: pipelining replication: peer="{Voter server-06e0cc33-930b-b8b4-1164-30f2237882ba 06e0cc33-930b-b8b4-1164-30f2237882ba}" testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.549Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: pipelining replication: peer="{Voter server-06e0cc33-930b-b8b4-1164-30f2237882ba 06e0cc33-930b-b8b4-1164-30f2237882ba}" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:05.658Z [INFO] stable state for Leader reached at 2023-08-02 15:13:05.549842462 +0000 UTC m=+17.645778805 (1 nodes), 370.663µs from start of poll, 380.098328ms from cluster start. Timeout at 2023-08-02 15:13:05.658567131 +0000 UTC m=+17.754503458, 108.724653ms after stability testing.go:149: cluster: 2023-08-02T15:13:05.658Z [INFO] stable state for Leader reached at 2023-08-02 15:13:05.549842462 +0000 UTC m=+17.645778805 (1 nodes), 370.663µs from start of poll, 380.098328ms from cluster start. Timeout at 2023-08-02 15:13:05.658567131 +0000 UTC m=+17.754503458, 108.724653ms after stability testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.658Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: aborting pipeline replication: peer="{Voter server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd 74a2e60e-7ba2-6dfb-22b7-2d76da0718bd}" testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.658Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: aborting pipeline replication: peer="{Voter server-74a2e60e-7ba2-6dfb-22b7-2d76da0718bd 74a2e60e-7ba2-6dfb-22b7-2d76da0718bd}" testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.658Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: aborting pipeline replication: peer="{Voter server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 d2db4dc6-ece6-d891-330a-3a459dfa2bd5}" testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.658Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: aborting pipeline replication: peer="{Voter server-d2db4dc6-ece6-d891-330a-3a459dfa2bd5 d2db4dc6-ece6-d891-330a-3a459dfa2bd5}" testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.658Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: aborting pipeline replication: peer="{Voter server-00a5042f-6283-5ea0-b291-ddca3bab9553 00a5042f-6283-5ea0-b291-ddca3bab9553}" testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.658Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: aborting pipeline replication: peer="{Voter server-00a5042f-6283-5ea0-b291-ddca3bab9553 00a5042f-6283-5ea0-b291-ddca3bab9553}" testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.658Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: aborting pipeline replication: peer="{Voter server-06e0cc33-930b-b8b4-1164-30f2237882ba 06e0cc33-930b-b8b4-1164-30f2237882ba}" testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.658Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: aborting pipeline replication: peer="{Voter server-06e0cc33-930b-b8b4-1164-30f2237882ba 06e0cc33-930b-b8b4-1164-30f2237882ba}" testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.658Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: aborting pipeline replication: peer="{Voter server-130b2734-1a0a-f1c3-6a90-3fd880c1748a 130b2734-1a0a-f1c3-6a90-3fd880c1748a}" testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.658Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: aborting pipeline replication: peer="{Voter server-130b2734-1a0a-f1c3-6a90-3fd880c1748a 130b2734-1a0a-f1c3-6a90-3fd880c1748a}" testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.658Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: aborting pipeline replication: peer="{Voter server-fcf65dbc-909a-19b9-2058-4b94c479035b fcf65dbc-909a-19b9-2058-4b94c479035b}" testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.658Z [INFO] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: aborting pipeline replication: peer="{Voter server-fcf65dbc-909a-19b9-2058-4b94c479035b fcf65dbc-909a-19b9-2058-4b94c479035b}" testLoggerAdapter verbose: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.709Z [ERROR] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: failed to heartbeat to: peer=06e0cc33-930b-b8b4-1164-30f2237882ba error="command timed out" testing.go:149: server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: 2023-08-02T15:13:05.709Z [ERROR] server-16c4d59e-8dcd-06b6-dc57-8e1b5667abdd: failed to heartbeat to: peer=06e0cc33-930b-b8b4-1164-30f2237882ba error="command timed out" --- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.54s) === RUN TestRaft_LeadershipTransferToInvalidID testLoggerAdapter verbose: server-7b75885a-58df-7895-aca2-8b53f27dcc4d: 2023-08-02T15:13:05.710Z [INFO] server-7b75885a-58df-7895-aca2-8b53f27dcc4d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b75885a-58df-7895-aca2-8b53f27dcc4d Address:7b75885a-58df-7895-aca2-8b53f27dcc4d} {Suffrage:Voter ID:server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93 Address:ab6bbf92-dd6c-152a-570b-7bdb8d8def93} {Suffrage:Voter ID:server-c540ccc5-51a2-2b80-4840-6b9cf369567b Address:c540ccc5-51a2-2b80-4840-6b9cf369567b}]" testing.go:149: server-7b75885a-58df-7895-aca2-8b53f27dcc4d: 2023-08-02T15:13:05.710Z [INFO] server-7b75885a-58df-7895-aca2-8b53f27dcc4d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b75885a-58df-7895-aca2-8b53f27dcc4d Address:7b75885a-58df-7895-aca2-8b53f27dcc4d} {Suffrage:Voter ID:server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93 Address:ab6bbf92-dd6c-152a-570b-7bdb8d8def93} {Suffrage:Voter ID:server-c540ccc5-51a2-2b80-4840-6b9cf369567b Address:c540ccc5-51a2-2b80-4840-6b9cf369567b}]" testLoggerAdapter verbose: server-7b75885a-58df-7895-aca2-8b53f27dcc4d: 2023-08-02T15:13:05.711Z [INFO] server-7b75885a-58df-7895-aca2-8b53f27dcc4d: entering follower state: follower="Node at 7b75885a-58df-7895-aca2-8b53f27dcc4d [Follower]" leader= testing.go:149: server-7b75885a-58df-7895-aca2-8b53f27dcc4d: 2023-08-02T15:13:05.711Z [INFO] server-7b75885a-58df-7895-aca2-8b53f27dcc4d: entering follower state: follower="Node at 7b75885a-58df-7895-aca2-8b53f27dcc4d [Follower]" leader= testLoggerAdapter verbose: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.711Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b75885a-58df-7895-aca2-8b53f27dcc4d Address:7b75885a-58df-7895-aca2-8b53f27dcc4d} {Suffrage:Voter ID:server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93 Address:ab6bbf92-dd6c-152a-570b-7bdb8d8def93} {Suffrage:Voter ID:server-c540ccc5-51a2-2b80-4840-6b9cf369567b Address:c540ccc5-51a2-2b80-4840-6b9cf369567b}]" testing.go:149: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.711Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b75885a-58df-7895-aca2-8b53f27dcc4d Address:7b75885a-58df-7895-aca2-8b53f27dcc4d} {Suffrage:Voter ID:server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93 Address:ab6bbf92-dd6c-152a-570b-7bdb8d8def93} {Suffrage:Voter ID:server-c540ccc5-51a2-2b80-4840-6b9cf369567b Address:c540ccc5-51a2-2b80-4840-6b9cf369567b}]" testLoggerAdapter verbose: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.711Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: entering follower state: follower="Node at ab6bbf92-dd6c-152a-570b-7bdb8d8def93 [Follower]" leader= testing.go:149: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.711Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: entering follower state: follower="Node at ab6bbf92-dd6c-152a-570b-7bdb8d8def93 [Follower]" leader= testLoggerAdapter verbose: server-c540ccc5-51a2-2b80-4840-6b9cf369567b: 2023-08-02T15:13:05.711Z [INFO] server-c540ccc5-51a2-2b80-4840-6b9cf369567b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b75885a-58df-7895-aca2-8b53f27dcc4d Address:7b75885a-58df-7895-aca2-8b53f27dcc4d} {Suffrage:Voter ID:server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93 Address:ab6bbf92-dd6c-152a-570b-7bdb8d8def93} {Suffrage:Voter ID:server-c540ccc5-51a2-2b80-4840-6b9cf369567b Address:c540ccc5-51a2-2b80-4840-6b9cf369567b}]" testing.go:149: server-c540ccc5-51a2-2b80-4840-6b9cf369567b: 2023-08-02T15:13:05.711Z [INFO] server-c540ccc5-51a2-2b80-4840-6b9cf369567b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b75885a-58df-7895-aca2-8b53f27dcc4d Address:7b75885a-58df-7895-aca2-8b53f27dcc4d} {Suffrage:Voter ID:server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93 Address:ab6bbf92-dd6c-152a-570b-7bdb8d8def93} {Suffrage:Voter ID:server-c540ccc5-51a2-2b80-4840-6b9cf369567b Address:c540ccc5-51a2-2b80-4840-6b9cf369567b}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:05.711Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:13:05.711Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-c540ccc5-51a2-2b80-4840-6b9cf369567b: 2023-08-02T15:13:05.711Z [INFO] server-c540ccc5-51a2-2b80-4840-6b9cf369567b: entering follower state: follower="Node at c540ccc5-51a2-2b80-4840-6b9cf369567b [Follower]" leader= testing.go:149: server-c540ccc5-51a2-2b80-4840-6b9cf369567b: 2023-08-02T15:13:05.711Z [INFO] server-c540ccc5-51a2-2b80-4840-6b9cf369567b: entering follower state: follower="Node at c540ccc5-51a2-2b80-4840-6b9cf369567b [Follower]" leader= testLoggerAdapter verbose: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.769Z [WARN] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.769Z [WARN] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.769Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: entering candidate state: node="Node at ab6bbf92-dd6c-152a-570b-7bdb8d8def93 [Candidate]" term=2 testing.go:149: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.769Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: entering candidate state: node="Node at ab6bbf92-dd6c-152a-570b-7bdb8d8def93 [Candidate]" term=2 testLoggerAdapter verbose: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.770Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: election won: tally=2 testing.go:149: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.770Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: election won: tally=2 testLoggerAdapter verbose: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.770Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: entering leader state: leader="Node at ab6bbf92-dd6c-152a-570b-7bdb8d8def93 [Leader]" testing.go:149: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.770Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: entering leader state: leader="Node at ab6bbf92-dd6c-152a-570b-7bdb8d8def93 [Leader]" testLoggerAdapter verbose: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.770Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: added peer, starting replication: peer=server-7b75885a-58df-7895-aca2-8b53f27dcc4d testing.go:149: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.770Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: added peer, starting replication: peer=server-7b75885a-58df-7895-aca2-8b53f27dcc4d testLoggerAdapter verbose: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.770Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: added peer, starting replication: peer=server-c540ccc5-51a2-2b80-4840-6b9cf369567b testing.go:149: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.770Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: added peer, starting replication: peer=server-c540ccc5-51a2-2b80-4840-6b9cf369567b testLoggerAdapter verbose: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.770Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: pipelining replication: peer="{Voter server-c540ccc5-51a2-2b80-4840-6b9cf369567b c540ccc5-51a2-2b80-4840-6b9cf369567b}" testing.go:149: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.770Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: pipelining replication: peer="{Voter server-c540ccc5-51a2-2b80-4840-6b9cf369567b c540ccc5-51a2-2b80-4840-6b9cf369567b}" testLoggerAdapter verbose: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.770Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: pipelining replication: peer="{Voter server-7b75885a-58df-7895-aca2-8b53f27dcc4d 7b75885a-58df-7895-aca2-8b53f27dcc4d}" testing.go:149: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.770Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: pipelining replication: peer="{Voter server-7b75885a-58df-7895-aca2-8b53f27dcc4d 7b75885a-58df-7895-aca2-8b53f27dcc4d}" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:05.875Z [INFO] stable state for Leader reached at 2023-08-02 15:13:05.77032997 +0000 UTC m=+17.866266298 (1 nodes), 58.695271ms from start of poll, 59.523058ms from cluster start. Timeout at 2023-08-02 15:13:05.875657213 +0000 UTC m=+17.971593547, 105.327249ms after stability testing.go:149: cluster: 2023-08-02T15:13:05.875Z [INFO] stable state for Leader reached at 2023-08-02 15:13:05.77032997 +0000 UTC m=+17.866266298 (1 nodes), 58.695271ms from start of poll, 59.523058ms from cluster start. Timeout at 2023-08-02 15:13:05.875657213 +0000 UTC m=+17.971593547, 105.327249ms after stability testLoggerAdapter verbose: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.875Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: aborting pipeline replication: peer="{Voter server-c540ccc5-51a2-2b80-4840-6b9cf369567b c540ccc5-51a2-2b80-4840-6b9cf369567b}" testing.go:149: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.875Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: aborting pipeline replication: peer="{Voter server-c540ccc5-51a2-2b80-4840-6b9cf369567b c540ccc5-51a2-2b80-4840-6b9cf369567b}" testLoggerAdapter verbose: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.875Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: aborting pipeline replication: peer="{Voter server-7b75885a-58df-7895-aca2-8b53f27dcc4d 7b75885a-58df-7895-aca2-8b53f27dcc4d}" testing.go:149: server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: 2023-08-02T15:13:05.875Z [INFO] server-ab6bbf92-dd6c-152a-570b-7bdb8d8def93: aborting pipeline replication: peer="{Voter server-7b75885a-58df-7895-aca2-8b53f27dcc4d 7b75885a-58df-7895-aca2-8b53f27dcc4d}" --- PASS: TestRaft_LeadershipTransferToInvalidID (0.17s) === RUN TestRaft_LeadershipTransferToInvalidAddress testLoggerAdapter verbose: server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb: 2023-08-02T15:13:05.879Z [INFO] server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb Address:8310b25d-94b0-b733-7d80-ee5e4ca6f0bb} {Suffrage:Voter ID:server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f Address:ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f} {Suffrage:Voter ID:server-9ace28e7-af59-7e83-274a-696f16352763 Address:9ace28e7-af59-7e83-274a-696f16352763}]" testing.go:149: server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb: 2023-08-02T15:13:05.879Z [INFO] server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb Address:8310b25d-94b0-b733-7d80-ee5e4ca6f0bb} {Suffrage:Voter ID:server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f Address:ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f} {Suffrage:Voter ID:server-9ace28e7-af59-7e83-274a-696f16352763 Address:9ace28e7-af59-7e83-274a-696f16352763}]" testLoggerAdapter verbose: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.879Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb Address:8310b25d-94b0-b733-7d80-ee5e4ca6f0bb} {Suffrage:Voter ID:server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f Address:ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f} {Suffrage:Voter ID:server-9ace28e7-af59-7e83-274a-696f16352763 Address:9ace28e7-af59-7e83-274a-696f16352763}]" testing.go:149: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.879Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb Address:8310b25d-94b0-b733-7d80-ee5e4ca6f0bb} {Suffrage:Voter ID:server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f Address:ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f} {Suffrage:Voter ID:server-9ace28e7-af59-7e83-274a-696f16352763 Address:9ace28e7-af59-7e83-274a-696f16352763}]" testLoggerAdapter verbose: server-9ace28e7-af59-7e83-274a-696f16352763: 2023-08-02T15:13:05.880Z [INFO] server-9ace28e7-af59-7e83-274a-696f16352763: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb Address:8310b25d-94b0-b733-7d80-ee5e4ca6f0bb} {Suffrage:Voter ID:server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f Address:ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f} {Suffrage:Voter ID:server-9ace28e7-af59-7e83-274a-696f16352763 Address:9ace28e7-af59-7e83-274a-696f16352763}]" testing.go:149: server-9ace28e7-af59-7e83-274a-696f16352763: 2023-08-02T15:13:05.880Z [INFO] server-9ace28e7-af59-7e83-274a-696f16352763: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb Address:8310b25d-94b0-b733-7d80-ee5e4ca6f0bb} {Suffrage:Voter ID:server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f Address:ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f} {Suffrage:Voter ID:server-9ace28e7-af59-7e83-274a-696f16352763 Address:9ace28e7-af59-7e83-274a-696f16352763}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:05.880Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-08-02T15:13:05.880Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb: 2023-08-02T15:13:05.880Z [INFO] server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb: entering follower state: follower="Node at 8310b25d-94b0-b733-7d80-ee5e4ca6f0bb [Follower]" leader= testing.go:149: server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb: 2023-08-02T15:13:05.880Z [INFO] server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb: entering follower state: follower="Node at 8310b25d-94b0-b733-7d80-ee5e4ca6f0bb [Follower]" leader= testLoggerAdapter verbose: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.880Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: entering follower state: follower="Node at ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f [Follower]" leader= testing.go:149: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.880Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: entering follower state: follower="Node at ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f [Follower]" leader= testLoggerAdapter verbose: server-9ace28e7-af59-7e83-274a-696f16352763: 2023-08-02T15:13:05.880Z [INFO] server-9ace28e7-af59-7e83-274a-696f16352763: entering follower state: follower="Node at 9ace28e7-af59-7e83-274a-696f16352763 [Follower]" leader= testing.go:149: server-9ace28e7-af59-7e83-274a-696f16352763: 2023-08-02T15:13:05.880Z [INFO] server-9ace28e7-af59-7e83-274a-696f16352763: entering follower state: follower="Node at 9ace28e7-af59-7e83-274a-696f16352763 [Follower]" leader= testLoggerAdapter verbose: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.941Z [WARN] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.941Z [WARN] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.942Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: entering candidate state: node="Node at ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f [Candidate]" term=2 testing.go:149: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.942Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: entering candidate state: node="Node at ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f [Candidate]" term=2 testLoggerAdapter verbose: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.942Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: election won: tally=2 testing.go:149: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.942Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: election won: tally=2 testLoggerAdapter verbose: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.942Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: entering leader state: leader="Node at ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f [Leader]" testing.go:149: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.942Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: entering leader state: leader="Node at ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f [Leader]" testLoggerAdapter verbose: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.942Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: added peer, starting replication: peer=server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb testing.go:149: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.942Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: added peer, starting replication: peer=server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb testLoggerAdapter verbose: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.942Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: added peer, starting replication: peer=server-9ace28e7-af59-7e83-274a-696f16352763 testing.go:149: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.942Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: added peer, starting replication: peer=server-9ace28e7-af59-7e83-274a-696f16352763 testLoggerAdapter verbose: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.942Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: pipelining replication: peer="{Voter server-9ace28e7-af59-7e83-274a-696f16352763 9ace28e7-af59-7e83-274a-696f16352763}" testing.go:149: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.942Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: pipelining replication: peer="{Voter server-9ace28e7-af59-7e83-274a-696f16352763 9ace28e7-af59-7e83-274a-696f16352763}" testLoggerAdapter verbose: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.942Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: pipelining replication: peer="{Voter server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb 8310b25d-94b0-b733-7d80-ee5e4ca6f0bb}" testing.go:149: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:05.942Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: pipelining replication: peer="{Voter server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb 8310b25d-94b0-b733-7d80-ee5e4ca6f0bb}" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:06.049Z [INFO] stable state for Follower reached at 2023-08-02 15:13:05.942635779 +0000 UTC m=+18.038572113 (2 nodes), 62.427552ms from start of poll, 63.226792ms from cluster start. Timeout at 2023-08-02 15:13:06.049021964 +0000 UTC m=+18.144958291, 106.386178ms after stability testing.go:149: cluster: 2023-08-02T15:13:06.049Z [INFO] stable state for Follower reached at 2023-08-02 15:13:05.942635779 +0000 UTC m=+18.038572113 (2 nodes), 62.427552ms from start of poll, 63.226792ms from cluster start. Timeout at 2023-08-02 15:13:06.049021964 +0000 UTC m=+18.144958291, 106.386178ms after stability testLoggerAdapter verbose: cluster: 2023-08-02T15:13:06.049Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:13:06.049Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-08-02T15:13:06.154Z [INFO] stable state for Leader reached at 2023-08-02 15:13:06.049177891 +0000 UTC m=+18.145114217 (1 nodes), 2.057µs from start of poll, 169.768896ms from cluster start. Timeout at 2023-08-02 15:13:06.154393648 +0000 UTC m=+18.250329981, 105.215764ms after stability testing.go:149: cluster: 2023-08-02T15:13:06.154Z [INFO] stable state for Leader reached at 2023-08-02 15:13:06.049177891 +0000 UTC m=+18.145114217 (1 nodes), 2.057µs from start of poll, 169.768896ms from cluster start. Timeout at 2023-08-02 15:13:06.154393648 +0000 UTC m=+18.250329981, 105.215764ms after stability testLoggerAdapter verbose: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:06.154Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: aborting pipeline replication: peer="{Voter server-9ace28e7-af59-7e83-274a-696f16352763 9ace28e7-af59-7e83-274a-696f16352763}" testing.go:149: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:06.154Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: aborting pipeline replication: peer="{Voter server-9ace28e7-af59-7e83-274a-696f16352763 9ace28e7-af59-7e83-274a-696f16352763}" testLoggerAdapter verbose: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:06.154Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: aborting pipeline replication: peer="{Voter server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb 8310b25d-94b0-b733-7d80-ee5e4ca6f0bb}" testing.go:149: server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: 2023-08-02T15:13:06.154Z [INFO] server-ddc28f8f-9c2b-6b1e-0508-c59198c0ed1f: aborting pipeline replication: peer="{Voter server-8310b25d-94b0-b733-7d80-ee5e4ca6f0bb 8310b25d-94b0-b733-7d80-ee5e4ca6f0bb}" --- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.28s) === RUN TestRaft_LeadershipTransferToBehindServer testLoggerAdapter verbose: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.155Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-361c6eda-3ec7-9fbe-2143-3add7f61b589 Address:361c6eda-3ec7-9fbe-2143-3add7f61b589} {Suffrage:Voter ID:server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad Address:692b65e7-a6eb-02c9-8cd2-b051259fa6ad} {Suffrage:Voter ID:server-16957a74-3d2b-9d6f-b9e4-51212393e041 Address:16957a74-3d2b-9d6f-b9e4-51212393e041}]" testing.go:149: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.155Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-361c6eda-3ec7-9fbe-2143-3add7f61b589 Address:361c6eda-3ec7-9fbe-2143-3add7f61b589} {Suffrage:Voter ID:server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad Address:692b65e7-a6eb-02c9-8cd2-b051259fa6ad} {Suffrage:Voter ID:server-16957a74-3d2b-9d6f-b9e4-51212393e041 Address:16957a74-3d2b-9d6f-b9e4-51212393e041}]" testLoggerAdapter verbose: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.155Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-361c6eda-3ec7-9fbe-2143-3add7f61b589 Address:361c6eda-3ec7-9fbe-2143-3add7f61b589} {Suffrage:Voter ID:server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad Address:692b65e7-a6eb-02c9-8cd2-b051259fa6ad} {Suffrage:Voter ID:server-16957a74-3d2b-9d6f-b9e4-51212393e041 Address:16957a74-3d2b-9d6f-b9e4-51212393e041}]" testing.go:149: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.155Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-361c6eda-3ec7-9fbe-2143-3add7f61b589 Address:361c6eda-3ec7-9fbe-2143-3add7f61b589} {Suffrage:Voter ID:server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad Address:692b65e7-a6eb-02c9-8cd2-b051259fa6ad} {Suffrage:Voter ID:server-16957a74-3d2b-9d6f-b9e4-51212393e041 Address:16957a74-3d2b-9d6f-b9e4-51212393e041}]" testLoggerAdapter verbose: server-16957a74-3d2b-9d6f-b9e4-51212393e041: 2023-08-02T15:13:06.155Z [INFO] server-16957a74-3d2b-9d6f-b9e4-51212393e041: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-361c6eda-3ec7-9fbe-2143-3add7f61b589 Address:361c6eda-3ec7-9fbe-2143-3add7f61b589} {Suffrage:Voter ID:server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad Address:692b65e7-a6eb-02c9-8cd2-b051259fa6ad} {Suffrage:Voter ID:server-16957a74-3d2b-9d6f-b9e4-51212393e041 Address:16957a74-3d2b-9d6f-b9e4-51212393e041}]" testing.go:149: server-16957a74-3d2b-9d6f-b9e4-51212393e041: 2023-08-02T15:13:06.155Z [INFO] server-16957a74-3d2b-9d6f-b9e4-51212393e041: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-361c6eda-3ec7-9fbe-2143-3add7f61b589 Address:361c6eda-3ec7-9fbe-2143-3add7f61b589} {Suffrage:Voter ID:server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad Address:692b65e7-a6eb-02c9-8cd2-b051259fa6ad} {Suffrage:Voter ID:server-16957a74-3d2b-9d6f-b9e4-51212393e041 Address:16957a74-3d2b-9d6f-b9e4-51212393e041}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:06.155Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:13:06.155Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.155Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: entering follower state: follower="Node at 361c6eda-3ec7-9fbe-2143-3add7f61b589 [Follower]" leader= testing.go:149: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.155Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: entering follower state: follower="Node at 361c6eda-3ec7-9fbe-2143-3add7f61b589 [Follower]" leader= testLoggerAdapter verbose: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.155Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: entering follower state: follower="Node at 692b65e7-a6eb-02c9-8cd2-b051259fa6ad [Follower]" leader= testing.go:149: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.155Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: entering follower state: follower="Node at 692b65e7-a6eb-02c9-8cd2-b051259fa6ad [Follower]" leader= testLoggerAdapter verbose: server-16957a74-3d2b-9d6f-b9e4-51212393e041: 2023-08-02T15:13:06.156Z [INFO] server-16957a74-3d2b-9d6f-b9e4-51212393e041: entering follower state: follower="Node at 16957a74-3d2b-9d6f-b9e4-51212393e041 [Follower]" leader= testing.go:149: server-16957a74-3d2b-9d6f-b9e4-51212393e041: 2023-08-02T15:13:06.156Z [INFO] server-16957a74-3d2b-9d6f-b9e4-51212393e041: entering follower state: follower="Node at 16957a74-3d2b-9d6f-b9e4-51212393e041 [Follower]" leader= testLoggerAdapter verbose: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.216Z [WARN] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.216Z [WARN] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.216Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: entering candidate state: node="Node at 361c6eda-3ec7-9fbe-2143-3add7f61b589 [Candidate]" term=2 testing.go:149: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.216Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: entering candidate state: node="Node at 361c6eda-3ec7-9fbe-2143-3add7f61b589 [Candidate]" term=2 testLoggerAdapter verbose: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.216Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: election won: tally=2 testing.go:149: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.216Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: election won: tally=2 testLoggerAdapter verbose: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.216Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: entering leader state: leader="Node at 361c6eda-3ec7-9fbe-2143-3add7f61b589 [Leader]" testing.go:149: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.216Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: entering leader state: leader="Node at 361c6eda-3ec7-9fbe-2143-3add7f61b589 [Leader]" testLoggerAdapter verbose: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.216Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: added peer, starting replication: peer=server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad testing.go:149: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.216Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: added peer, starting replication: peer=server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad testLoggerAdapter verbose: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.216Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: added peer, starting replication: peer=server-16957a74-3d2b-9d6f-b9e4-51212393e041 testing.go:149: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.216Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: added peer, starting replication: peer=server-16957a74-3d2b-9d6f-b9e4-51212393e041 testLoggerAdapter verbose: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.216Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: pipelining replication: peer="{Voter server-16957a74-3d2b-9d6f-b9e4-51212393e041 16957a74-3d2b-9d6f-b9e4-51212393e041}" testing.go:149: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.216Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: pipelining replication: peer="{Voter server-16957a74-3d2b-9d6f-b9e4-51212393e041 16957a74-3d2b-9d6f-b9e4-51212393e041}" testLoggerAdapter verbose: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.216Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: pipelining replication: peer="{Voter server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad 692b65e7-a6eb-02c9-8cd2-b051259fa6ad}" testing.go:149: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.216Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: pipelining replication: peer="{Voter server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad 692b65e7-a6eb-02c9-8cd2-b051259fa6ad}" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:06.321Z [INFO] stable state for Leader reached at 2023-08-02 15:13:06.216899543 +0000 UTC m=+18.312835877 (1 nodes), 60.997433ms from start of poll, 61.453724ms from cluster start. Timeout at 2023-08-02 15:13:06.32192285 +0000 UTC m=+18.417859176, 105.023299ms after stability testing.go:149: cluster: 2023-08-02T15:13:06.321Z [INFO] stable state for Leader reached at 2023-08-02 15:13:06.216899543 +0000 UTC m=+18.312835877 (1 nodes), 60.997433ms from start of poll, 61.453724ms from cluster start. Timeout at 2023-08-02 15:13:06.32192285 +0000 UTC m=+18.417859176, 105.023299ms after stability testLoggerAdapter verbose: cluster: 2023-08-02T15:13:06.322Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-08-02T15:13:06.322Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2023-08-02T15:13:06.428Z [INFO] stable state for Follower reached at 2023-08-02 15:13:06.322102992 +0000 UTC m=+18.418039318 (2 nodes), 2.351µs from start of poll, 166.657165ms from cluster start. Timeout at 2023-08-02 15:13:06.428071596 +0000 UTC m=+18.524007939, 105.968621ms after stability testing.go:149: cluster: 2023-08-02T15:13:06.428Z [INFO] stable state for Follower reached at 2023-08-02 15:13:06.322102992 +0000 UTC m=+18.418039318 (2 nodes), 2.351µs from start of poll, 166.657165ms from cluster start. Timeout at 2023-08-02 15:13:06.428071596 +0000 UTC m=+18.524007939, 105.968621ms after stability testLoggerAdapter verbose: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.465Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: entering candidate state: node="Node at 692b65e7-a6eb-02c9-8cd2-b051259fa6ad [Candidate]" term=3 testing.go:149: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.465Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: entering candidate state: node="Node at 692b65e7-a6eb-02c9-8cd2-b051259fa6ad [Candidate]" term=3 testLoggerAdapter verbose: cluster: 2023-08-02T15:13:06.465Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.465Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: election won: tally=2 testing.go:149: cluster: 2023-08-02T15:13:06.465Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.465Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: election won: tally=2 testLoggerAdapter verbose: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.465Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: entering follower state: follower="Node at 361c6eda-3ec7-9fbe-2143-3add7f61b589 [Follower]" leader= testing.go:149: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.465Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: entering follower state: follower="Node at 361c6eda-3ec7-9fbe-2143-3add7f61b589 [Follower]" leader= testLoggerAdapter verbose: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.465Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: entering leader state: leader="Node at 692b65e7-a6eb-02c9-8cd2-b051259fa6ad [Leader]" testLoggerAdapter verbose: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.465Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: aborting pipeline replication: peer="{Voter server-16957a74-3d2b-9d6f-b9e4-51212393e041 16957a74-3d2b-9d6f-b9e4-51212393e041}" testing.go:149: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.465Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: aborting pipeline replication: peer="{Voter server-16957a74-3d2b-9d6f-b9e4-51212393e041 16957a74-3d2b-9d6f-b9e4-51212393e041}" testLoggerAdapter verbose: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.465Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: aborting pipeline replication: peer="{Voter server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad 692b65e7-a6eb-02c9-8cd2-b051259fa6ad}" testing.go:149: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.465Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: entering leader state: leader="Node at 692b65e7-a6eb-02c9-8cd2-b051259fa6ad [Leader]" testLoggerAdapter verbose: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.465Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: added peer, starting replication: peer=server-361c6eda-3ec7-9fbe-2143-3add7f61b589 testing.go:149: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.465Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: added peer, starting replication: peer=server-361c6eda-3ec7-9fbe-2143-3add7f61b589 testLoggerAdapter verbose: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.465Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: added peer, starting replication: peer=server-16957a74-3d2b-9d6f-b9e4-51212393e041 testing.go:149: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.465Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: added peer, starting replication: peer=server-16957a74-3d2b-9d6f-b9e4-51212393e041 testLoggerAdapter verbose: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.466Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: pipelining replication: peer="{Voter server-361c6eda-3ec7-9fbe-2143-3add7f61b589 361c6eda-3ec7-9fbe-2143-3add7f61b589}" testing.go:149: server-361c6eda-3ec7-9fbe-2143-3add7f61b589: 2023-08-02T15:13:06.465Z [INFO] server-361c6eda-3ec7-9fbe-2143-3add7f61b589: aborting pipeline replication: peer="{Voter server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad 692b65e7-a6eb-02c9-8cd2-b051259fa6ad}" testing.go:149: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.466Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: pipelining replication: peer="{Voter server-361c6eda-3ec7-9fbe-2143-3add7f61b589 361c6eda-3ec7-9fbe-2143-3add7f61b589}" testLoggerAdapter verbose: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.466Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: pipelining replication: peer="{Voter server-16957a74-3d2b-9d6f-b9e4-51212393e041 16957a74-3d2b-9d6f-b9e4-51212393e041}" testing.go:149: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.466Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: pipelining replication: peer="{Voter server-16957a74-3d2b-9d6f-b9e4-51212393e041 16957a74-3d2b-9d6f-b9e4-51212393e041}" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:06.571Z [INFO] stable state for Leader reached at 2023-08-02 15:13:06.465711788 +0000 UTC m=+18.561648121 (1 nodes), 22.751µs from start of poll, 310.265968ms from cluster start. Timeout at 2023-08-02 15:13:06.571569975 +0000 UTC m=+18.667506377, 105.858256ms after stability testing.go:149: cluster: 2023-08-02T15:13:06.571Z [INFO] stable state for Leader reached at 2023-08-02 15:13:06.465711788 +0000 UTC m=+18.561648121 (1 nodes), 22.751µs from start of poll, 310.265968ms from cluster start. Timeout at 2023-08-02 15:13:06.571569975 +0000 UTC m=+18.667506377, 105.858256ms after stability testLoggerAdapter verbose: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.571Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: aborting pipeline replication: peer="{Voter server-16957a74-3d2b-9d6f-b9e4-51212393e041 16957a74-3d2b-9d6f-b9e4-51212393e041}" testing.go:149: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.571Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: aborting pipeline replication: peer="{Voter server-16957a74-3d2b-9d6f-b9e4-51212393e041 16957a74-3d2b-9d6f-b9e4-51212393e041}" testLoggerAdapter verbose: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.571Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: aborting pipeline replication: peer="{Voter server-361c6eda-3ec7-9fbe-2143-3add7f61b589 361c6eda-3ec7-9fbe-2143-3add7f61b589}" testing.go:149: server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: 2023-08-02T15:13:06.571Z [INFO] server-692b65e7-a6eb-02c9-8cd2-b051259fa6ad: aborting pipeline replication: peer="{Voter server-361c6eda-3ec7-9fbe-2143-3add7f61b589 361c6eda-3ec7-9fbe-2143-3add7f61b589}" --- PASS: TestRaft_LeadershipTransferToBehindServer (0.42s) === RUN TestRaft_LeadershipTransferToItself testLoggerAdapter verbose: server-0137d5fb-72ab-e229-35fa-8443f56e385d: 2023-08-02T15:13:06.572Z [INFO] server-0137d5fb-72ab-e229-35fa-8443f56e385d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0137d5fb-72ab-e229-35fa-8443f56e385d Address:0137d5fb-72ab-e229-35fa-8443f56e385d} {Suffrage:Voter ID:server-cb6b69c4-336f-faaa-53d5-815fb5c02720 Address:cb6b69c4-336f-faaa-53d5-815fb5c02720} {Suffrage:Voter ID:server-40bdc3a2-eba6-6266-135a-e92d125923bc Address:40bdc3a2-eba6-6266-135a-e92d125923bc}]" testing.go:149: server-0137d5fb-72ab-e229-35fa-8443f56e385d: 2023-08-02T15:13:06.572Z [INFO] server-0137d5fb-72ab-e229-35fa-8443f56e385d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0137d5fb-72ab-e229-35fa-8443f56e385d Address:0137d5fb-72ab-e229-35fa-8443f56e385d} {Suffrage:Voter ID:server-cb6b69c4-336f-faaa-53d5-815fb5c02720 Address:cb6b69c4-336f-faaa-53d5-815fb5c02720} {Suffrage:Voter ID:server-40bdc3a2-eba6-6266-135a-e92d125923bc Address:40bdc3a2-eba6-6266-135a-e92d125923bc}]" testLoggerAdapter verbose: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.573Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0137d5fb-72ab-e229-35fa-8443f56e385d Address:0137d5fb-72ab-e229-35fa-8443f56e385d} {Suffrage:Voter ID:server-cb6b69c4-336f-faaa-53d5-815fb5c02720 Address:cb6b69c4-336f-faaa-53d5-815fb5c02720} {Suffrage:Voter ID:server-40bdc3a2-eba6-6266-135a-e92d125923bc Address:40bdc3a2-eba6-6266-135a-e92d125923bc}]" testLoggerAdapter verbose: server-0137d5fb-72ab-e229-35fa-8443f56e385d: 2023-08-02T15:13:06.573Z [INFO] server-0137d5fb-72ab-e229-35fa-8443f56e385d: entering follower state: follower="Node at 0137d5fb-72ab-e229-35fa-8443f56e385d [Follower]" leader= testing.go:149: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.573Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0137d5fb-72ab-e229-35fa-8443f56e385d Address:0137d5fb-72ab-e229-35fa-8443f56e385d} {Suffrage:Voter ID:server-cb6b69c4-336f-faaa-53d5-815fb5c02720 Address:cb6b69c4-336f-faaa-53d5-815fb5c02720} {Suffrage:Voter ID:server-40bdc3a2-eba6-6266-135a-e92d125923bc Address:40bdc3a2-eba6-6266-135a-e92d125923bc}]" testing.go:149: server-0137d5fb-72ab-e229-35fa-8443f56e385d: 2023-08-02T15:13:06.573Z [INFO] server-0137d5fb-72ab-e229-35fa-8443f56e385d: entering follower state: follower="Node at 0137d5fb-72ab-e229-35fa-8443f56e385d [Follower]" leader= testLoggerAdapter verbose: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.573Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: entering follower state: follower="Node at cb6b69c4-336f-faaa-53d5-815fb5c02720 [Follower]" leader= testing.go:149: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.573Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: entering follower state: follower="Node at cb6b69c4-336f-faaa-53d5-815fb5c02720 [Follower]" leader= testLoggerAdapter verbose: server-40bdc3a2-eba6-6266-135a-e92d125923bc: 2023-08-02T15:13:06.573Z [INFO] server-40bdc3a2-eba6-6266-135a-e92d125923bc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0137d5fb-72ab-e229-35fa-8443f56e385d Address:0137d5fb-72ab-e229-35fa-8443f56e385d} {Suffrage:Voter ID:server-cb6b69c4-336f-faaa-53d5-815fb5c02720 Address:cb6b69c4-336f-faaa-53d5-815fb5c02720} {Suffrage:Voter ID:server-40bdc3a2-eba6-6266-135a-e92d125923bc Address:40bdc3a2-eba6-6266-135a-e92d125923bc}]" testing.go:149: server-40bdc3a2-eba6-6266-135a-e92d125923bc: 2023-08-02T15:13:06.573Z [INFO] server-40bdc3a2-eba6-6266-135a-e92d125923bc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0137d5fb-72ab-e229-35fa-8443f56e385d Address:0137d5fb-72ab-e229-35fa-8443f56e385d} {Suffrage:Voter ID:server-cb6b69c4-336f-faaa-53d5-815fb5c02720 Address:cb6b69c4-336f-faaa-53d5-815fb5c02720} {Suffrage:Voter ID:server-40bdc3a2-eba6-6266-135a-e92d125923bc Address:40bdc3a2-eba6-6266-135a-e92d125923bc}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:06.573Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-40bdc3a2-eba6-6266-135a-e92d125923bc: 2023-08-02T15:13:06.573Z [INFO] server-40bdc3a2-eba6-6266-135a-e92d125923bc: entering follower state: follower="Node at 40bdc3a2-eba6-6266-135a-e92d125923bc [Follower]" leader= testing.go:149: cluster: 2023-08-02T15:13:06.573Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-40bdc3a2-eba6-6266-135a-e92d125923bc: 2023-08-02T15:13:06.573Z [INFO] server-40bdc3a2-eba6-6266-135a-e92d125923bc: entering follower state: follower="Node at 40bdc3a2-eba6-6266-135a-e92d125923bc [Follower]" leader= testLoggerAdapter verbose: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.634Z [WARN] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.634Z [WARN] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.634Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: entering candidate state: node="Node at cb6b69c4-336f-faaa-53d5-815fb5c02720 [Candidate]" term=2 testing.go:149: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.634Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: entering candidate state: node="Node at cb6b69c4-336f-faaa-53d5-815fb5c02720 [Candidate]" term=2 testLoggerAdapter verbose: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.634Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: election won: tally=2 testing.go:149: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.634Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: election won: tally=2 testLoggerAdapter verbose: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.634Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: entering leader state: leader="Node at cb6b69c4-336f-faaa-53d5-815fb5c02720 [Leader]" testing.go:149: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.634Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: entering leader state: leader="Node at cb6b69c4-336f-faaa-53d5-815fb5c02720 [Leader]" testLoggerAdapter verbose: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.634Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: added peer, starting replication: peer=server-0137d5fb-72ab-e229-35fa-8443f56e385d testing.go:149: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.634Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: added peer, starting replication: peer=server-0137d5fb-72ab-e229-35fa-8443f56e385d testLoggerAdapter verbose: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.634Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: added peer, starting replication: peer=server-40bdc3a2-eba6-6266-135a-e92d125923bc testing.go:149: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.634Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: added peer, starting replication: peer=server-40bdc3a2-eba6-6266-135a-e92d125923bc testLoggerAdapter verbose: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.634Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: pipelining replication: peer="{Voter server-40bdc3a2-eba6-6266-135a-e92d125923bc 40bdc3a2-eba6-6266-135a-e92d125923bc}" testing.go:149: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.634Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: pipelining replication: peer="{Voter server-40bdc3a2-eba6-6266-135a-e92d125923bc 40bdc3a2-eba6-6266-135a-e92d125923bc}" testLoggerAdapter verbose: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.634Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: pipelining replication: peer="{Voter server-0137d5fb-72ab-e229-35fa-8443f56e385d 0137d5fb-72ab-e229-35fa-8443f56e385d}" testing.go:149: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.634Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: pipelining replication: peer="{Voter server-0137d5fb-72ab-e229-35fa-8443f56e385d 0137d5fb-72ab-e229-35fa-8443f56e385d}" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:06.739Z [INFO] stable state for Leader reached at 2023-08-02 15:13:06.634561329 +0000 UTC m=+18.730497671 (1 nodes), 61.129914ms from start of poll, 61.754533ms from cluster start. Timeout at 2023-08-02 15:13:06.739569554 +0000 UTC m=+18.835505895, 105.008224ms after stability testing.go:149: cluster: 2023-08-02T15:13:06.739Z [INFO] stable state for Leader reached at 2023-08-02 15:13:06.634561329 +0000 UTC m=+18.730497671 (1 nodes), 61.129914ms from start of poll, 61.754533ms from cluster start. Timeout at 2023-08-02 15:13:06.739569554 +0000 UTC m=+18.835505895, 105.008224ms after stability testLoggerAdapter verbose: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.739Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: cannot transfer leadership to itself testing.go:149: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.739Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: cannot transfer leadership to itself testLoggerAdapter verbose: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.739Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: aborting pipeline replication: peer="{Voter server-40bdc3a2-eba6-6266-135a-e92d125923bc 40bdc3a2-eba6-6266-135a-e92d125923bc}" testing.go:149: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.739Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: aborting pipeline replication: peer="{Voter server-40bdc3a2-eba6-6266-135a-e92d125923bc 40bdc3a2-eba6-6266-135a-e92d125923bc}" testLoggerAdapter verbose: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.739Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: aborting pipeline replication: peer="{Voter server-0137d5fb-72ab-e229-35fa-8443f56e385d 0137d5fb-72ab-e229-35fa-8443f56e385d}" testing.go:149: server-cb6b69c4-336f-faaa-53d5-815fb5c02720: 2023-08-02T15:13:06.739Z [INFO] server-cb6b69c4-336f-faaa-53d5-815fb5c02720: aborting pipeline replication: peer="{Voter server-0137d5fb-72ab-e229-35fa-8443f56e385d 0137d5fb-72ab-e229-35fa-8443f56e385d}" --- PASS: TestRaft_LeadershipTransferToItself (0.17s) === RUN TestRaft_LeadershipTransferLeaderRejectsClientRequests raft_test.go:2064: Disabled unreliable test --- SKIP: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.00s) === RUN TestRaft_LeadershipTransferLeaderReplicationTimeout testLoggerAdapter verbose: server-1e7dbaaa-cfaf-c459-0903-55ccc957901e: 2023-08-02T15:13:06.741Z [INFO] server-1e7dbaaa-cfaf-c459-0903-55ccc957901e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e7dbaaa-cfaf-c459-0903-55ccc957901e Address:1e7dbaaa-cfaf-c459-0903-55ccc957901e} {Suffrage:Voter ID:server-ad663a07-48f4-b5af-a68c-21294e0da3c6 Address:ad663a07-48f4-b5af-a68c-21294e0da3c6} {Suffrage:Voter ID:server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 Address:5acdc2a5-4065-9b1f-727c-b07fb59d6ef3}]" testing.go:149: server-1e7dbaaa-cfaf-c459-0903-55ccc957901e: 2023-08-02T15:13:06.741Z [INFO] server-1e7dbaaa-cfaf-c459-0903-55ccc957901e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e7dbaaa-cfaf-c459-0903-55ccc957901e Address:1e7dbaaa-cfaf-c459-0903-55ccc957901e} {Suffrage:Voter ID:server-ad663a07-48f4-b5af-a68c-21294e0da3c6 Address:ad663a07-48f4-b5af-a68c-21294e0da3c6} {Suffrage:Voter ID:server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 Address:5acdc2a5-4065-9b1f-727c-b07fb59d6ef3}]" testLoggerAdapter verbose: server-1e7dbaaa-cfaf-c459-0903-55ccc957901e: 2023-08-02T15:13:06.741Z [INFO] server-1e7dbaaa-cfaf-c459-0903-55ccc957901e: entering follower state: follower="Node at 1e7dbaaa-cfaf-c459-0903-55ccc957901e [Follower]" leader= testLoggerAdapter verbose: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.741Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e7dbaaa-cfaf-c459-0903-55ccc957901e Address:1e7dbaaa-cfaf-c459-0903-55ccc957901e} {Suffrage:Voter ID:server-ad663a07-48f4-b5af-a68c-21294e0da3c6 Address:ad663a07-48f4-b5af-a68c-21294e0da3c6} {Suffrage:Voter ID:server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 Address:5acdc2a5-4065-9b1f-727c-b07fb59d6ef3}]" testing.go:149: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.741Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e7dbaaa-cfaf-c459-0903-55ccc957901e Address:1e7dbaaa-cfaf-c459-0903-55ccc957901e} {Suffrage:Voter ID:server-ad663a07-48f4-b5af-a68c-21294e0da3c6 Address:ad663a07-48f4-b5af-a68c-21294e0da3c6} {Suffrage:Voter ID:server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 Address:5acdc2a5-4065-9b1f-727c-b07fb59d6ef3}]" testing.go:149: server-1e7dbaaa-cfaf-c459-0903-55ccc957901e: 2023-08-02T15:13:06.741Z [INFO] server-1e7dbaaa-cfaf-c459-0903-55ccc957901e: entering follower state: follower="Node at 1e7dbaaa-cfaf-c459-0903-55ccc957901e [Follower]" leader= testLoggerAdapter verbose: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.743Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: entering follower state: follower="Node at ad663a07-48f4-b5af-a68c-21294e0da3c6 [Follower]" leader= testing.go:149: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.743Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: entering follower state: follower="Node at ad663a07-48f4-b5af-a68c-21294e0da3c6 [Follower]" leader= testLoggerAdapter verbose: server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: 2023-08-02T15:13:06.743Z [INFO] server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e7dbaaa-cfaf-c459-0903-55ccc957901e Address:1e7dbaaa-cfaf-c459-0903-55ccc957901e} {Suffrage:Voter ID:server-ad663a07-48f4-b5af-a68c-21294e0da3c6 Address:ad663a07-48f4-b5af-a68c-21294e0da3c6} {Suffrage:Voter ID:server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 Address:5acdc2a5-4065-9b1f-727c-b07fb59d6ef3}]" testing.go:149: server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: 2023-08-02T15:13:06.743Z [INFO] server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e7dbaaa-cfaf-c459-0903-55ccc957901e Address:1e7dbaaa-cfaf-c459-0903-55ccc957901e} {Suffrage:Voter ID:server-ad663a07-48f4-b5af-a68c-21294e0da3c6 Address:ad663a07-48f4-b5af-a68c-21294e0da3c6} {Suffrage:Voter ID:server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 Address:5acdc2a5-4065-9b1f-727c-b07fb59d6ef3}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:06.744Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:13:06.744Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: 2023-08-02T15:13:06.744Z [INFO] server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: entering follower state: follower="Node at 5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 [Follower]" leader= testing.go:149: server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: 2023-08-02T15:13:06.744Z [INFO] server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: entering follower state: follower="Node at 5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 [Follower]" leader= testLoggerAdapter verbose: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.810Z [WARN] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.810Z [WARN] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.810Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: entering candidate state: node="Node at ad663a07-48f4-b5af-a68c-21294e0da3c6 [Candidate]" term=2 testing.go:149: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.810Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: entering candidate state: node="Node at ad663a07-48f4-b5af-a68c-21294e0da3c6 [Candidate]" term=2 testLoggerAdapter verbose: server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: 2023-08-02T15:13:06.810Z [WARN] server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: 2023-08-02T15:13:06.810Z [WARN] server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: 2023-08-02T15:13:06.810Z [INFO] server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: entering candidate state: node="Node at 5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 [Candidate]" term=2 testLoggerAdapter verbose: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.810Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: election won: tally=2 testing.go:149: server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: 2023-08-02T15:13:06.810Z [INFO] server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: entering candidate state: node="Node at 5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 [Candidate]" term=2 testing.go:149: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.810Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: election won: tally=2 testLoggerAdapter verbose: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.810Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: entering leader state: leader="Node at ad663a07-48f4-b5af-a68c-21294e0da3c6 [Leader]" testing.go:149: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.810Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: entering leader state: leader="Node at ad663a07-48f4-b5af-a68c-21294e0da3c6 [Leader]" testLoggerAdapter verbose: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.810Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: added peer, starting replication: peer=server-1e7dbaaa-cfaf-c459-0903-55ccc957901e testLoggerAdapter verbose: server-1e7dbaaa-cfaf-c459-0903-55ccc957901e: 2023-08-02T15:13:06.810Z [INFO] server-1e7dbaaa-cfaf-c459-0903-55ccc957901e: duplicate requestVote for same term: term=2 testing.go:149: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.810Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: added peer, starting replication: peer=server-1e7dbaaa-cfaf-c459-0903-55ccc957901e testing.go:149: server-1e7dbaaa-cfaf-c459-0903-55ccc957901e: 2023-08-02T15:13:06.810Z [INFO] server-1e7dbaaa-cfaf-c459-0903-55ccc957901e: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.810Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: added peer, starting replication: peer=server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 testing.go:149: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.810Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: added peer, starting replication: peer=server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 testLoggerAdapter verbose: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.810Z [WARN] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: rejecting vote request since we have a leader: from=5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 leader=ad663a07-48f4-b5af-a68c-21294e0da3c6 testing.go:149: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.810Z [WARN] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: rejecting vote request since we have a leader: from=5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 leader=ad663a07-48f4-b5af-a68c-21294e0da3c6 testLoggerAdapter verbose: server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: 2023-08-02T15:13:06.810Z [INFO] server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: entering follower state: follower="Node at 5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 [Follower]" leader=ad663a07-48f4-b5af-a68c-21294e0da3c6 testing.go:149: server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: 2023-08-02T15:13:06.810Z [INFO] server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: entering follower state: follower="Node at 5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 [Follower]" leader=ad663a07-48f4-b5af-a68c-21294e0da3c6 testLoggerAdapter verbose: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.811Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: pipelining replication: peer="{Voter server-1e7dbaaa-cfaf-c459-0903-55ccc957901e 1e7dbaaa-cfaf-c459-0903-55ccc957901e}" testing.go:149: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.811Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: pipelining replication: peer="{Voter server-1e7dbaaa-cfaf-c459-0903-55ccc957901e 1e7dbaaa-cfaf-c459-0903-55ccc957901e}" testLoggerAdapter verbose: server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: 2023-08-02T15:13:06.811Z [INFO] server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: duplicate requestVote for same term: term=2 testing.go:149: server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: 2023-08-02T15:13:06.811Z [INFO] server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.811Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: pipelining replication: peer="{Voter server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 5acdc2a5-4065-9b1f-727c-b07fb59d6ef3}" testing.go:149: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:06.811Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: pipelining replication: peer="{Voter server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 5acdc2a5-4065-9b1f-727c-b07fb59d6ef3}" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:06.916Z [INFO] stable state for Leader reached at 2023-08-02 15:13:06.811106675 +0000 UTC m=+18.907043026 (1 nodes), 66.943868ms from start of poll, 70.204546ms from cluster start. Timeout at 2023-08-02 15:13:06.916675089 +0000 UTC m=+19.012611425, 105.568399ms after stability testing.go:149: cluster: 2023-08-02T15:13:06.916Z [INFO] stable state for Leader reached at 2023-08-02 15:13:06.811106675 +0000 UTC m=+18.907043026 (1 nodes), 66.943868ms from start of poll, 70.204546ms from cluster start. Timeout at 2023-08-02 15:13:06.916675089 +0000 UTC m=+19.012611425, 105.568399ms after stability testLoggerAdapter verbose: cluster: 2023-08-02T15:13:06.916Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-08-02T15:13:06.916Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2023-08-02T15:13:07.022Z [INFO] stable state for Follower reached at 2023-08-02 15:13:06.916843278 +0000 UTC m=+19.012779615 (2 nodes), 1.377µs from start of poll, 175.941135ms from cluster start. Timeout at 2023-08-02 15:13:07.022916744 +0000 UTC m=+19.118853083, 106.073468ms after stability testing.go:149: cluster: 2023-08-02T15:13:07.022Z [INFO] stable state for Follower reached at 2023-08-02 15:13:06.916843278 +0000 UTC m=+19.012779615 (2 nodes), 1.377µs from start of poll, 175.941135ms from cluster start. Timeout at 2023-08-02 15:13:07.022916744 +0000 UTC m=+19.118853083, 106.073468ms after stability testLoggerAdapter verbose: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:07.389Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: aborting pipeline replication: peer="{Voter server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 5acdc2a5-4065-9b1f-727c-b07fb59d6ef3}" testing.go:149: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:07.389Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: aborting pipeline replication: peer="{Voter server-5acdc2a5-4065-9b1f-727c-b07fb59d6ef3 5acdc2a5-4065-9b1f-727c-b07fb59d6ef3}" testLoggerAdapter verbose: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:07.389Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: aborting pipeline replication: peer="{Voter server-1e7dbaaa-cfaf-c459-0903-55ccc957901e 1e7dbaaa-cfaf-c459-0903-55ccc957901e}" testing.go:149: server-ad663a07-48f4-b5af-a68c-21294e0da3c6: 2023-08-02T15:13:07.389Z [INFO] server-ad663a07-48f4-b5af-a68c-21294e0da3c6: aborting pipeline replication: peer="{Voter server-1e7dbaaa-cfaf-c459-0903-55ccc957901e 1e7dbaaa-cfaf-c459-0903-55ccc957901e}" --- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (0.65s) === RUN TestRaft_LeadershipTransferStopRightAway --- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s) === RUN TestRaft_GetConfigurationNoBootstrap testLoggerAdapter verbose: server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f: 2023-08-02T15:13:07.390Z [INFO] server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f Address:2eff0449-18f1-1803-8b89-a0cb3d2ed45f} {Suffrage:Voter ID:server-0c0b1d30-08a0-862e-86f5-15c95132833d Address:0c0b1d30-08a0-862e-86f5-15c95132833d}]" testing.go:149: server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f: 2023-08-02T15:13:07.390Z [INFO] server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f Address:2eff0449-18f1-1803-8b89-a0cb3d2ed45f} {Suffrage:Voter ID:server-0c0b1d30-08a0-862e-86f5-15c95132833d Address:0c0b1d30-08a0-862e-86f5-15c95132833d}]" testLoggerAdapter verbose: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.390Z [INFO] server-0c0b1d30-08a0-862e-86f5-15c95132833d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f Address:2eff0449-18f1-1803-8b89-a0cb3d2ed45f} {Suffrage:Voter ID:server-0c0b1d30-08a0-862e-86f5-15c95132833d Address:0c0b1d30-08a0-862e-86f5-15c95132833d}]" testing.go:149: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.390Z [INFO] server-0c0b1d30-08a0-862e-86f5-15c95132833d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f Address:2eff0449-18f1-1803-8b89-a0cb3d2ed45f} {Suffrage:Voter ID:server-0c0b1d30-08a0-862e-86f5-15c95132833d Address:0c0b1d30-08a0-862e-86f5-15c95132833d}]" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:07.390Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2023-08-02T15:13:07.390Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f: 2023-08-02T15:13:07.390Z [INFO] server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f: entering follower state: follower="Node at 2eff0449-18f1-1803-8b89-a0cb3d2ed45f [Follower]" leader= testing.go:149: server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f: 2023-08-02T15:13:07.390Z [INFO] server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f: entering follower state: follower="Node at 2eff0449-18f1-1803-8b89-a0cb3d2ed45f [Follower]" leader= testLoggerAdapter verbose: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.390Z [INFO] server-0c0b1d30-08a0-862e-86f5-15c95132833d: entering follower state: follower="Node at 0c0b1d30-08a0-862e-86f5-15c95132833d [Follower]" leader= testing.go:149: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.390Z [INFO] server-0c0b1d30-08a0-862e-86f5-15c95132833d: entering follower state: follower="Node at 0c0b1d30-08a0-862e-86f5-15c95132833d [Follower]" leader= testLoggerAdapter verbose: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.459Z [WARN] server-0c0b1d30-08a0-862e-86f5-15c95132833d: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.459Z [WARN] server-0c0b1d30-08a0-862e-86f5-15c95132833d: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.460Z [INFO] server-0c0b1d30-08a0-862e-86f5-15c95132833d: entering candidate state: node="Node at 0c0b1d30-08a0-862e-86f5-15c95132833d [Candidate]" term=2 testing.go:149: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.460Z [INFO] server-0c0b1d30-08a0-862e-86f5-15c95132833d: entering candidate state: node="Node at 0c0b1d30-08a0-862e-86f5-15c95132833d [Candidate]" term=2 testLoggerAdapter verbose: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.463Z [INFO] server-0c0b1d30-08a0-862e-86f5-15c95132833d: election won: tally=2 testing.go:149: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.463Z [INFO] server-0c0b1d30-08a0-862e-86f5-15c95132833d: election won: tally=2 testLoggerAdapter verbose: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.463Z [INFO] server-0c0b1d30-08a0-862e-86f5-15c95132833d: entering leader state: leader="Node at 0c0b1d30-08a0-862e-86f5-15c95132833d [Leader]" testing.go:149: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.463Z [INFO] server-0c0b1d30-08a0-862e-86f5-15c95132833d: entering leader state: leader="Node at 0c0b1d30-08a0-862e-86f5-15c95132833d [Leader]" testLoggerAdapter verbose: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.463Z [INFO] server-0c0b1d30-08a0-862e-86f5-15c95132833d: added peer, starting replication: peer=server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f testing.go:149: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.463Z [INFO] server-0c0b1d30-08a0-862e-86f5-15c95132833d: added peer, starting replication: peer=server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f testLoggerAdapter verbose: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.463Z [INFO] server-0c0b1d30-08a0-862e-86f5-15c95132833d: pipelining replication: peer="{Voter server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f 2eff0449-18f1-1803-8b89-a0cb3d2ed45f}" testing.go:149: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.463Z [INFO] server-0c0b1d30-08a0-862e-86f5-15c95132833d: pipelining replication: peer="{Voter server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f 2eff0449-18f1-1803-8b89-a0cb3d2ed45f}" testLoggerAdapter verbose: cluster: 2023-08-02T15:13:07.568Z [INFO] stable state for Follower reached at 2023-08-02 15:13:07.46341475 +0000 UTC m=+19.559351103 (1 nodes), 72.519126ms from start of poll, 72.882111ms from cluster start. Timeout at 2023-08-02 15:13:07.568468396 +0000 UTC m=+19.664404746, 105.053643ms after stability testing.go:149: cluster: 2023-08-02T15:13:07.568Z [INFO] stable state for Follower reached at 2023-08-02 15:13:07.46341475 +0000 UTC m=+19.559351103 (1 nodes), 72.519126ms from start of poll, 72.882111ms from cluster start. Timeout at 2023-08-02 15:13:07.568468396 +0000 UTC m=+19.664404746, 105.053643ms after stability testLoggerAdapter verbose: cluster: 2023-08-02T15:13:07.568Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2023-08-02T15:13:07.568Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2023-08-02T15:13:07.673Z [INFO] stable state for Leader reached at 2023-08-02 15:13:07.568662813 +0000 UTC m=+19.664599149 (1 nodes), 2.17µs from start of poll, 178.130157ms from cluster start. Timeout at 2023-08-02 15:13:07.67379294 +0000 UTC m=+19.769729295, 105.130146ms after stability testing.go:149: cluster: 2023-08-02T15:13:07.673Z [INFO] stable state for Leader reached at 2023-08-02 15:13:07.568662813 +0000 UTC m=+19.664599149 (1 nodes), 2.17µs from start of poll, 178.130157ms from cluster start. Timeout at 2023-08-02 15:13:07.67379294 +0000 UTC m=+19.769729295, 105.130146ms after stability testLoggerAdapter verbose: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.684Z [INFO] server-0c0b1d30-08a0-862e-86f5-15c95132833d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f Address:2eff0449-18f1-1803-8b89-a0cb3d2ed45f} {Suffrage:Voter ID:server-0c0b1d30-08a0-862e-86f5-15c95132833d Address:0c0b1d30-08a0-862e-86f5-15c95132833d}]" testing.go:149: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.684Z [INFO] server-0c0b1d30-08a0-862e-86f5-15c95132833d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f Address:2eff0449-18f1-1803-8b89-a0cb3d2ed45f} {Suffrage:Voter ID:server-0c0b1d30-08a0-862e-86f5-15c95132833d Address:0c0b1d30-08a0-862e-86f5-15c95132833d}]" testLoggerAdapter verbose: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.684Z [INFO] server-0c0b1d30-08a0-862e-86f5-15c95132833d: aborting pipeline replication: peer="{Voter server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f 2eff0449-18f1-1803-8b89-a0cb3d2ed45f}" testing.go:149: server-0c0b1d30-08a0-862e-86f5-15c95132833d: 2023-08-02T15:13:07.684Z [INFO] server-0c0b1d30-08a0-862e-86f5-15c95132833d: aborting pipeline replication: peer="{Voter server-2eff0449-18f1-1803-8b89-a0cb3d2ed45f 2eff0449-18f1-1803-8b89-a0cb3d2ed45f}" --- PASS: TestRaft_GetConfigurationNoBootstrap (0.29s) === RUN TestTCPTransport_BadAddr --- PASS: TestTCPTransport_BadAddr (0.00s) === RUN TestTCPTransport_WithAdvertise --- PASS: TestTCPTransport_WithAdvertise (0.00s) === RUN TestTransport_StartStop --- PASS: TestTransport_StartStop (0.00s) === RUN TestTransport_AppendEntries --- PASS: TestTransport_AppendEntries (0.00s) === RUN TestTransport_AppendEntriesPipeline --- PASS: TestTransport_AppendEntriesPipeline (0.00s) === RUN TestTransport_RequestVote --- PASS: TestTransport_RequestVote (0.00s) === RUN TestTransport_InstallSnapshot --- PASS: TestTransport_InstallSnapshot (0.00s) === RUN TestTransport_EncodeDecode --- PASS: TestTransport_EncodeDecode (0.00s) === RUN TestRandomTimeout --- PASS: TestRandomTimeout (0.00s) === RUN TestNewSeed --- PASS: TestNewSeed (0.00s) === RUN TestRandomTimeout_NoTime --- PASS: TestRandomTimeout_NoTime (0.00s) === RUN TestMin --- PASS: TestMin (0.00s) === RUN TestMax --- PASS: TestMax (0.00s) === RUN TestGenerateUUID --- PASS: TestGenerateUUID (0.00s) === RUN TestBackoff --- PASS: TestBackoff (0.00s) PASS ok github.com/hashicorp/raft 19.806s ? github.com/hashicorp/raft/bench [no test files] create-stamp debian/debhelper-build-stamp fakeroot debian/rules binary dh binary --buildsystem=golang --with=golang dh_testroot -O--buildsystem=golang dh_prep -O--buildsystem=golang dh_auto_install -O--buildsystem=golang dh_installdocs -O--buildsystem=golang dh_installchangelogs -O--buildsystem=golang dh_installinit -O--buildsystem=golang dh_installsystemduser -O--buildsystem=golang dh_perl -O--buildsystem=golang dh_link -O--buildsystem=golang dh_strip_nondeterminism -O--buildsystem=golang dh_compress -O--buildsystem=golang dh_fixperms -O--buildsystem=golang dh_missing -O--buildsystem=golang dh_installdeb -O--buildsystem=golang dh_golang -O--buildsystem=golang dh_gencontrol -O--buildsystem=golang dpkg-gencontrol: warning: Depends field of package golang-github-hashicorp-raft-dev: substitution variable ${shlibs:Depends} used, but is not defined dh_md5sums -O--buildsystem=golang dh_builddeb -O--buildsystem=golang dpkg-deb: building package 'golang-github-hashicorp-raft-dev' in '../golang-github-hashicorp-raft-dev_1.1.2-1_all.deb'. dpkg-genbuildinfo --build=binary dpkg-genchanges --build=binary >../golang-github-hashicorp-raft_1.1.2-1_amd64.changes dpkg-genchanges: info: binary-only upload (no source code included) dpkg-source --after-build . dpkg-buildpackage: info: binary-only upload (no source included) dpkg-genchanges: info: including full source code in upload I: copying local configuration I: user script /srv/workspace/pbuilder/2531752/tmp/hooks/B01_cleanup starting I: user script /srv/workspace/pbuilder/2531752/tmp/hooks/B01_cleanup finished I: unmounting dev/ptmx filesystem I: unmounting dev/pts filesystem I: unmounting dev/shm filesystem I: unmounting proc filesystem I: unmounting sys filesystem I: cleaning the build env I: removing directory /srv/workspace/pbuilder/2531752 and its subdirectories I: Current time: Thu Aug 3 05:13:16 +14 2023 I: pbuilder-time-stamp: 1690989196