I: pbuilder: network access will be disabled during build I: Current time: Mon Jan 8 17:08:02 -12 2024 I: pbuilder-time-stamp: 1704776882 I: Building the build Environment I: extracting base tarball [/var/cache/pbuilder/bullseye-reproducible-base.tgz] I: copying local configuration W: --override-config is not set; not updating apt.conf Read the manpage for details. I: mounting /proc filesystem I: mounting /sys filesystem I: creating /{dev,run}/shm I: mounting /dev/pts filesystem I: redirecting /dev/ptmx to /dev/pts/ptmx I: policy-rc.d already exists I: using eatmydata during job I: Copying source file I: copying [golang-github-hashicorp-raft_1.1.2-1.dsc] I: copying [./golang-github-hashicorp-raft_1.1.2.orig.tar.xz] I: copying [./golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz] I: Extracting source gpgv: unknown type of key resource 'trustedkeys.kbx' gpgv: keyblock resource '/tmp/dpkg-verify-sig.IocVDySR/trustedkeys.kbx': General error gpgv: Signature made Sun Jan 19 07:24:28 2020 gpgv: using RSA key 50BC7CF939D20C272A6B065652B6BBD953968D1B gpgv: Can't check signature: No public key dpkg-source: warning: failed to verify signature on ./golang-github-hashicorp-raft_1.1.2-1.dsc dpkg-source: info: extracting golang-github-hashicorp-raft in golang-github-hashicorp-raft-1.1.2 dpkg-source: info: unpacking golang-github-hashicorp-raft_1.1.2.orig.tar.xz dpkg-source: info: unpacking golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz dpkg-source: info: using patch list from debian/patches/series dpkg-source: info: applying t-skip--TestNetworkTransport_AppendEntriesPipeline_CloseStreams.patch dpkg-source: info: applying t-skip--TestRaft_AutoSnapshot.patch dpkg-source: info: applying t-skip--TestRaft_LeadershipTransferLeaderRejectsClientRequests.patch dpkg-source: info: applying t-skip--TestRaft_RecoverCluster.patch dpkg-source: info: applying t-skip--TestRaft_SnapshotRestore_PeerChange.patch I: using fakeroot in build. I: Installing the build-deps I: user script /srv/workspace/pbuilder/904/tmp/hooks/D02_print_environment starting I: set BUILDDIR='/build/reproducible-path' BUILDUSERGECOS='first user,first room,first work-phone,first home-phone,first other' BUILDUSERNAME='pbuilder1' BUILD_ARCH='i386' DEBIAN_FRONTEND='noninteractive' DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all,-fixfilepath parallel=8 ' DISTRIBUTION='bullseye' HOME='/root' HOST_ARCH='i386' IFS=' ' INVOCATION_ID='c7bdbcbca84f40239dfe7a5fef4cb1a5' LANG='C' LANGUAGE='en_US:en' LC_ALL='C' LD_LIBRARY_PATH='/usr/lib/libeatmydata' LD_PRELOAD='libeatmydata.so' MAIL='/var/mail/root' OPTIND='1' PATH='/usr/sbin:/usr/bin:/sbin:/bin:/usr/games' PBCURRENTCOMMANDLINEOPERATION='build' PBUILDER_OPERATION='build' PBUILDER_PKGDATADIR='/usr/share/pbuilder' PBUILDER_PKGLIBDIR='/usr/lib/pbuilder' PBUILDER_SYSCONFDIR='/etc' PPID='904' PS1='# ' PS2='> ' PS4='+ ' PWD='/' SHELL='/bin/bash' SHLVL='2' SUDO_COMMAND='/usr/bin/timeout -k 18.1h 18h /usr/bin/ionice -c 3 /usr/bin/nice /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.IdhfiL2y/pbuilderrc_H70d --distribution bullseye --hookdir /etc/pbuilder/first-build-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/bullseye-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.IdhfiL2y/b1 --logfile b1/build.log golang-github-hashicorp-raft_1.1.2-1.dsc' SUDO_GID='112' SUDO_UID='107' SUDO_USER='jenkins' TERM='unknown' TZ='/usr/share/zoneinfo/Etc/GMT+12' USER='root' _='/usr/bin/systemd-run' http_proxy='http://78.137.99.97:3128' I: uname -a Linux ionos2-i386 6.1.0-17-686-pae #1 SMP PREEMPT_DYNAMIC Debian 6.1.69-1 (2023-12-30) i686 GNU/Linux I: ls -l /bin total 5776 -rwxr-xr-x 1 root root 1367848 Mar 27 2022 bash -rwxr-xr-x 3 root root 38280 Jul 20 2020 bunzip2 -rwxr-xr-x 3 root root 38280 Jul 20 2020 bzcat lrwxrwxrwx 1 root root 6 Jul 20 2020 bzcmp -> bzdiff -rwxr-xr-x 1 root root 2225 Jul 20 2020 bzdiff lrwxrwxrwx 1 root root 6 Jul 20 2020 bzegrep -> bzgrep -rwxr-xr-x 1 root root 4877 Sep 4 2019 bzexe lrwxrwxrwx 1 root root 6 Jul 20 2020 bzfgrep -> bzgrep -rwxr-xr-x 1 root root 3775 Jul 20 2020 bzgrep -rwxr-xr-x 3 root root 38280 Jul 20 2020 bzip2 -rwxr-xr-x 1 root root 17768 Jul 20 2020 bzip2recover lrwxrwxrwx 1 root root 6 Jul 20 2020 bzless -> bzmore -rwxr-xr-x 1 root root 1297 Jul 20 2020 bzmore -rwxr-xr-x 1 root root 38824 Sep 22 2020 cat -rwxr-xr-x 1 root root 71624 Sep 22 2020 chgrp -rwxr-xr-x 1 root root 67528 Sep 22 2020 chmod -rwxr-xr-x 1 root root 75752 Sep 22 2020 chown -rwxr-xr-x 1 root root 157960 Sep 22 2020 cp -rwxr-xr-x 1 root root 128724 Dec 10 2020 dash -rwxr-xr-x 1 root root 124904 Sep 22 2020 date -rwxr-xr-x 1 root root 92172 Sep 22 2020 dd -rwxr-xr-x 1 root root 100752 Sep 22 2020 df -rwxr-xr-x 1 root root 153964 Sep 22 2020 dir -rwxr-xr-x 1 root root 83644 Jan 20 2022 dmesg lrwxrwxrwx 1 root root 8 Nov 7 2019 dnsdomainname -> hostname lrwxrwxrwx 1 root root 8 Nov 7 2019 domainname -> hostname -rwxr-xr-x 1 root root 34664 Sep 22 2020 echo -rwxr-xr-x 1 root root 28 Jan 25 2023 egrep -rwxr-xr-x 1 root root 34664 Sep 22 2020 false -rwxr-xr-x 1 root root 28 Jan 25 2023 fgrep -rwxr-xr-x 1 root root 71928 Jan 20 2022 findmnt -rwsr-xr-x 1 root root 30112 Feb 26 2021 fusermount -rwxr-xr-x 1 root root 210488 Jan 25 2023 grep -rwxr-xr-x 2 root root 2346 Apr 10 2022 gunzip -rwxr-xr-x 1 root root 6447 Apr 10 2022 gzexe -rwxr-xr-x 1 root root 100952 Apr 10 2022 gzip -rwxr-xr-x 1 root root 21916 Nov 7 2019 hostname -rwxr-xr-x 1 root root 83980 Sep 22 2020 ln -rwxr-xr-x 1 root root 55572 Feb 7 2020 login -rwxr-xr-x 1 root root 153964 Sep 22 2020 ls -rwxr-xr-x 1 root root 153124 Jan 20 2022 lsblk -rwxr-xr-x 1 root root 96328 Sep 22 2020 mkdir -rwxr-xr-x 1 root root 79912 Sep 22 2020 mknod -rwxr-xr-x 1 root root 47048 Sep 22 2020 mktemp -rwxr-xr-x 1 root root 58920 Jan 20 2022 more -rwsr-xr-x 1 root root 50720 Jan 20 2022 mount -rwxr-xr-x 1 root root 13856 Jan 20 2022 mountpoint -rwxr-xr-x 1 root root 157996 Sep 22 2020 mv lrwxrwxrwx 1 root root 8 Nov 7 2019 nisdomainname -> hostname lrwxrwxrwx 1 root root 14 Dec 16 2021 pidof -> /sbin/killall5 -rwxr-xr-x 1 root root 38824 Sep 22 2020 pwd lrwxrwxrwx 1 root root 4 Mar 27 2022 rbash -> bash -rwxr-xr-x 1 root root 46984 Sep 22 2020 readlink -rwxr-xr-x 1 root root 75720 Sep 22 2020 rm -rwxr-xr-x 1 root root 46984 Sep 22 2020 rmdir -rwxr-xr-x 1 root root 22292 Sep 27 2020 run-parts -rwxr-xr-x 1 root root 125036 Dec 22 2018 sed lrwxrwxrwx 1 root root 4 Jan 7 09:24 sh -> dash -rwxr-xr-x 1 root root 34696 Sep 22 2020 sleep -rwxr-xr-x 1 root root 83880 Sep 22 2020 stty -rwsr-xr-x 1 root root 79396 Jan 20 2022 su -rwxr-xr-x 1 root root 34696 Sep 22 2020 sync -rwxr-xr-x 1 root root 602584 Feb 17 2021 tar -rwxr-xr-x 1 root root 13860 Sep 27 2020 tempfile -rwxr-xr-x 1 root root 108520 Sep 22 2020 touch -rwxr-xr-x 1 root root 34664 Sep 22 2020 true -rwxr-xr-x 1 root root 17768 Feb 26 2021 ulockmgr_server -rwsr-xr-x 1 root root 30236 Jan 20 2022 umount -rwxr-xr-x 1 root root 34664 Sep 22 2020 uname -rwxr-xr-x 2 root root 2346 Apr 10 2022 uncompress -rwxr-xr-x 1 root root 153964 Sep 22 2020 vdir -rwxr-xr-x 1 root root 63024 Jan 20 2022 wdctl lrwxrwxrwx 1 root root 8 Nov 7 2019 ypdomainname -> hostname -rwxr-xr-x 1 root root 1984 Apr 10 2022 zcat -rwxr-xr-x 1 root root 1678 Apr 10 2022 zcmp -rwxr-xr-x 1 root root 5898 Apr 10 2022 zdiff -rwxr-xr-x 1 root root 29 Apr 10 2022 zegrep -rwxr-xr-x 1 root root 29 Apr 10 2022 zfgrep -rwxr-xr-x 1 root root 2081 Apr 10 2022 zforce -rwxr-xr-x 1 root root 8049 Apr 10 2022 zgrep -rwxr-xr-x 1 root root 2206 Apr 10 2022 zless -rwxr-xr-x 1 root root 1842 Apr 10 2022 zmore -rwxr-xr-x 1 root root 4577 Apr 10 2022 znew I: user script /srv/workspace/pbuilder/904/tmp/hooks/D02_print_environment finished -> Attempting to satisfy build-dependencies -> Creating pbuilder-satisfydepends-dummy package Package: pbuilder-satisfydepends-dummy Version: 0.invalid.0 Architecture: i386 Maintainer: Debian Pbuilder Team Description: Dummy package to satisfy dependencies with aptitude - created by pbuilder This package was created automatically by pbuilder to satisfy the build-dependencies of the package being currently built. Depends: debhelper (>= 12~), dh-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~) dpkg-deb: building package 'pbuilder-satisfydepends-dummy' in '/tmp/satisfydepends-aptitude/pbuilder-satisfydepends-dummy.deb'. Selecting previously unselected package pbuilder-satisfydepends-dummy. (Reading database ... 17763 files and directories currently installed.) Preparing to unpack .../pbuilder-satisfydepends-dummy.deb ... Unpacking pbuilder-satisfydepends-dummy (0.invalid.0) ... dpkg: pbuilder-satisfydepends-dummy: dependency problems, but configuring anyway as you requested: pbuilder-satisfydepends-dummy depends on debhelper (>= 12~); however: Package debhelper is not installed. pbuilder-satisfydepends-dummy depends on dh-golang; however: Package dh-golang is not installed. pbuilder-satisfydepends-dummy depends on golang-any; however: Package golang-any is not installed. pbuilder-satisfydepends-dummy depends on golang-github-armon-go-metrics-dev; however: Package golang-github-armon-go-metrics-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-hclog-dev (>= 0.9.2~); however: Package golang-github-hashicorp-go-hclog-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~); however: Package golang-github-hashicorp-go-msgpack-dev is not installed. Setting up pbuilder-satisfydepends-dummy (0.invalid.0) ... Reading package lists... Building dependency tree... Reading state information... Initializing package states... Writing extended state information... Building tag database... pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0) pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0) The following NEW packages will be installed: autoconf{a} automake{a} autopoint{a} autotools-dev{a} bsdextrautils{a} debhelper{a} dh-autoreconf{a} dh-golang{a} dh-strip-nondeterminism{a} dwz{a} file{a} gettext{a} gettext-base{a} golang-1.15-go{a} golang-1.15-src{a} golang-any{a} golang-github-alecthomas-units-dev{a} golang-github-armon-go-metrics-dev{a} golang-github-beorn7-perks-dev{a} golang-github-cespare-xxhash-dev{a} golang-github-circonus-labs-circonus-gometrics-dev{a} golang-github-circonus-labs-circonusllhist-dev{a} golang-github-creack-pty-dev{a} golang-github-datadog-datadog-go-dev{a} golang-github-davecgh-go-spew-dev{a} golang-github-dgrijalva-jwt-go-dev{a} golang-github-fatih-color-dev{a} golang-github-go-kit-kit-dev{a} golang-github-go-logfmt-logfmt-dev{a} golang-github-go-stack-stack-dev{a} golang-github-golang-mock-dev{a} golang-github-google-go-cmp-dev{a} golang-github-hashicorp-go-cleanhttp-dev{a} golang-github-hashicorp-go-hclog-dev{a} golang-github-hashicorp-go-immutable-radix-dev{a} golang-github-hashicorp-go-msgpack-dev{a} golang-github-hashicorp-go-retryablehttp-dev{a} golang-github-hashicorp-go-uuid-dev{a} golang-github-hashicorp-golang-lru-dev{a} golang-github-jpillora-backoff-dev{a} golang-github-json-iterator-go-dev{a} golang-github-julienschmidt-httprouter-dev{a} golang-github-kr-pretty-dev{a} golang-github-kr-text-dev{a} golang-github-mattn-go-colorable-dev{a} golang-github-mattn-go-isatty-dev{a} golang-github-modern-go-concurrent-dev{a} golang-github-modern-go-reflect2-dev{a} golang-github-mwitkow-go-conntrack-dev{a} golang-github-opentracing-opentracing-go-dev{a} golang-github-pascaldekloe-goe-dev{a} golang-github-pkg-errors-dev{a} golang-github-pmezard-go-difflib-dev{a} golang-github-prometheus-client-golang-dev{a} golang-github-prometheus-client-model-dev{a} golang-github-prometheus-common-dev{a} golang-github-prometheus-procfs-dev{a} golang-github-sirupsen-logrus-dev{a} golang-github-stretchr-objx-dev{a} golang-github-stretchr-testify-dev{a} golang-github-tv42-httpunix-dev{a} golang-github-vmihailenco-tagparser-dev{a} golang-glog-dev{a} golang-go{a} golang-golang-x-net-dev{a} golang-golang-x-oauth2-dev{a} golang-golang-x-oauth2-google-dev{a} golang-golang-x-sync-dev{a} golang-golang-x-sys-dev{a} golang-golang-x-text-dev{a} golang-golang-x-time-dev{a} golang-golang-x-xerrors-dev{a} golang-google-cloud-compute-metadata-dev{a} golang-google-genproto-dev{a} golang-google-grpc-dev{a} golang-gopkg-alecthomas-kingpin.v2-dev{a} golang-gopkg-check.v1-dev{a} golang-gopkg-mgo.v2-dev{a} golang-gopkg-tomb.v2-dev{a} golang-gopkg-vmihailenco-msgpack.v2-dev{a} golang-gopkg-yaml.v2-dev{a} golang-gopkg-yaml.v3-dev{a} golang-goprotobuf-dev{a} golang-protobuf-extensions-dev{a} golang-src{a} groff-base{a} intltool-debian{a} libarchive-zip-perl{a} libdebhelper-perl{a} libelf1{a} libfile-stripnondeterminism-perl{a} libicu67{a} libmagic-mgc{a} libmagic1{a} libpipeline1{a} libprotobuf-dev{a} libprotobuf-lite23{a} libprotobuf23{a} libprotoc23{a} libsasl2-2{a} libsasl2-dev{a} libsasl2-modules-db{a} libsigsegv2{a} libsub-override-perl{a} libtool{a} libuchardet0{a} libxml2{a} m4{a} man-db{a} po-debconf{a} protobuf-compiler{a} sensible-utils{a} zlib1g-dev{a} The following packages are RECOMMENDED but will NOT be installed: curl libarchive-cpio-perl libltdl-dev libmail-sendmail-perl libsasl2-modules lynx pkg-config pkgconf wget 0 packages upgraded, 113 newly installed, 0 to remove and 0 not upgraded. Need to get 95.8 MB of archives. After unpacking 538 MB will be used. Writing extended state information... Get: 1 http://deb.debian.org/debian bullseye/main i386 bsdextrautils i386 2.36.1-8+deb11u1 [149 kB] Get: 2 http://deb.debian.org/debian bullseye/main i386 libuchardet0 i386 0.0.7-1 [67.9 kB] Get: 3 http://deb.debian.org/debian bullseye/main i386 groff-base i386 1.22.4-6 [952 kB] Get: 4 http://deb.debian.org/debian bullseye/main i386 libpipeline1 i386 1.5.3-1 [36.8 kB] Get: 5 http://deb.debian.org/debian bullseye/main i386 man-db i386 2.9.4-2 [1367 kB] Get: 6 http://deb.debian.org/debian bullseye/main i386 golang-golang-x-sys-dev all 0.0~git20210124.22da62e-1 [308 kB] Get: 7 http://deb.debian.org/debian bullseye/main i386 golang-github-sirupsen-logrus-dev all 1.7.0-2 [45.3 kB] Get: 8 http://deb.debian.org/debian bullseye/main i386 sensible-utils all 0.0.14 [14.8 kB] Get: 9 http://deb.debian.org/debian bullseye/main i386 libmagic-mgc i386 1:5.39-3+deb11u1 [273 kB] Get: 10 http://deb.debian.org/debian bullseye/main i386 libmagic1 i386 1:5.39-3+deb11u1 [135 kB] Get: 11 http://deb.debian.org/debian bullseye/main i386 file i386 1:5.39-3+deb11u1 [69.2 kB] Get: 12 http://deb.debian.org/debian bullseye/main i386 gettext-base i386 0.21-4 [176 kB] Get: 13 http://deb.debian.org/debian bullseye/main i386 libsigsegv2 i386 2.13-1 [35.1 kB] Get: 14 http://deb.debian.org/debian bullseye/main i386 m4 i386 1.4.18-5 [206 kB] Get: 15 http://deb.debian.org/debian bullseye/main i386 autoconf all 2.69-14 [313 kB] Get: 16 http://deb.debian.org/debian bullseye/main i386 autotools-dev all 20180224.1+nmu1 [77.1 kB] Get: 17 http://deb.debian.org/debian bullseye/main i386 automake all 1:1.16.3-2 [814 kB] Get: 18 http://deb.debian.org/debian bullseye/main i386 autopoint all 0.21-4 [510 kB] Get: 19 http://deb.debian.org/debian bullseye/main i386 libdebhelper-perl all 13.3.4 [189 kB] Get: 20 http://deb.debian.org/debian bullseye/main i386 libtool all 2.4.6-15 [513 kB] Get: 21 http://deb.debian.org/debian bullseye/main i386 dh-autoreconf all 20 [17.1 kB] Get: 22 http://deb.debian.org/debian bullseye/main i386 libarchive-zip-perl all 1.68-1 [104 kB] Get: 23 http://deb.debian.org/debian bullseye/main i386 libsub-override-perl all 0.09-2 [10.2 kB] Get: 24 http://deb.debian.org/debian bullseye/main i386 libfile-stripnondeterminism-perl all 1.12.0-1 [26.3 kB] Get: 25 http://deb.debian.org/debian bullseye/main i386 dh-strip-nondeterminism all 1.12.0-1 [15.4 kB] Get: 26 http://deb.debian.org/debian bullseye/main i386 libelf1 i386 0.183-1 [171 kB] Get: 27 http://deb.debian.org/debian bullseye/main i386 dwz i386 0.13+20210201-1 [179 kB] Get: 28 http://deb.debian.org/debian bullseye/main i386 libicu67 i386 67.1-7 [8775 kB] Get: 29 http://deb.debian.org/debian bullseye/main i386 libxml2 i386 2.9.10+dfsg-6.7+deb11u4 [728 kB] Get: 30 http://deb.debian.org/debian bullseye/main i386 gettext i386 0.21-4 [1322 kB] Get: 31 http://deb.debian.org/debian bullseye/main i386 intltool-debian all 0.35.0+20060710.5 [26.8 kB] Get: 32 http://deb.debian.org/debian bullseye/main i386 po-debconf all 1.0.21+nmu1 [248 kB] Get: 33 http://deb.debian.org/debian bullseye/main i386 debhelper all 13.3.4 [1049 kB] Get: 34 http://deb.debian.org/debian bullseye/main i386 dh-golang all 1.51 [24.6 kB] Get: 35 http://deb.debian.org/debian bullseye/main i386 golang-1.15-src i386 1.15.15-1~deb11u4 [13.9 MB] Get: 36 http://deb.debian.org/debian bullseye/main i386 golang-1.15-go i386 1.15.15-1~deb11u4 [46.3 MB] Get: 37 http://deb.debian.org/debian bullseye/main i386 golang-src i386 2:1.15~1 [4856 B] Get: 38 http://deb.debian.org/debian bullseye/main i386 golang-go i386 2:1.15~1 [24.0 kB] Get: 39 http://deb.debian.org/debian bullseye/main i386 golang-any i386 2:1.15~1 [4980 B] Get: 40 http://deb.debian.org/debian bullseye/main i386 golang-github-alecthomas-units-dev all 0.0~git20201120.1786d5e-1 [7504 B] Get: 41 http://deb.debian.org/debian bullseye/main i386 golang-github-circonus-labs-circonusllhist-dev all 0.0~git20191022.ec08cde-1 [11.1 kB] Get: 42 http://deb.debian.org/debian bullseye/main i386 golang-github-hashicorp-go-cleanhttp-dev all 0.5.2-1 [10.8 kB] Get: 43 http://deb.debian.org/debian bullseye/main i386 golang-github-mattn-go-isatty-dev all 0.0.12-1 [6472 B] Get: 44 http://deb.debian.org/debian bullseye/main i386 golang-github-mattn-go-colorable-dev all 0.1.7-1 [9936 B] Get: 45 http://deb.debian.org/debian bullseye/main i386 golang-github-fatih-color-dev all 1.7.0-1 [11.4 kB] Get: 46 http://deb.debian.org/debian bullseye/main i386 golang-github-davecgh-go-spew-dev all 1.1.1-2 [29.7 kB] Get: 47 http://deb.debian.org/debian bullseye/main i386 golang-github-pmezard-go-difflib-dev all 1.0.0-3 [12.3 kB] Get: 48 http://deb.debian.org/debian bullseye/main i386 golang-github-stretchr-objx-dev all 0.3.0-1 [25.4 kB] Get: 49 http://deb.debian.org/debian bullseye/main i386 golang-gopkg-yaml.v3-dev all 3.0.0~git20200121.a6ecf24-3 [70.5 kB] Get: 50 http://deb.debian.org/debian bullseye/main i386 golang-github-stretchr-testify-dev all 1.6.1-2 [60.4 kB] Get: 51 http://deb.debian.org/debian bullseye/main i386 golang-github-hashicorp-go-hclog-dev all 0.11.0-1 [17.9 kB] Get: 52 http://deb.debian.org/debian bullseye/main i386 golang-github-hashicorp-go-retryablehttp-dev all 0.6.4-2 [17.5 kB] Get: 53 http://deb.debian.org/debian bullseye/main i386 golang-github-pkg-errors-dev all 0.9.1-1 [13.0 kB] Get: 54 http://deb.debian.org/debian bullseye/main i386 golang-github-tv42-httpunix-dev all 0.0~git20150427.b75d861-2 [3744 B] Get: 55 http://deb.debian.org/debian bullseye/main i386 golang-github-circonus-labs-circonus-gometrics-dev all 2.3.1-3 [65.9 kB] Get: 56 http://deb.debian.org/debian bullseye/main i386 golang-github-datadog-datadog-go-dev all 2.1.0-3 [14.9 kB] Get: 57 http://deb.debian.org/debian bullseye/main i386 golang-github-hashicorp-go-uuid-dev all 1.0.2-1 [8764 B] Get: 58 http://deb.debian.org/debian bullseye/main i386 golang-github-hashicorp-golang-lru-dev all 0.5.4-2 [14.7 kB] Get: 59 http://deb.debian.org/debian bullseye/main i386 golang-github-hashicorp-go-immutable-radix-dev all 1.3.0-1 [25.0 kB] Get: 60 http://deb.debian.org/debian bullseye/main i386 golang-github-pascaldekloe-goe-dev all 0.1.0-2 [21.7 kB] Get: 61 http://deb.debian.org/debian bullseye/main i386 golang-github-beorn7-perks-dev all 1.0.1-1 [12.1 kB] Get: 62 http://deb.debian.org/debian bullseye/main i386 golang-github-cespare-xxhash-dev all 2.1.1-1 [8748 B] Get: 63 http://deb.debian.org/debian bullseye/main i386 golang-github-modern-go-concurrent-dev all 1.0.3-1.1 [4668 B] Get: 64 http://deb.debian.org/debian bullseye/main i386 golang-github-modern-go-reflect2-dev all 1.0.1-1 [10.6 kB] Get: 65 http://deb.debian.org/debian bullseye/main i386 golang-github-json-iterator-go-dev all 1.1.10-2 [69.4 kB] Get: 66 http://deb.debian.org/debian bullseye/main i386 zlib1g-dev i386 1:1.2.11.dfsg-2+deb11u2 [194 kB] Get: 67 http://deb.debian.org/debian bullseye/main i386 libprotobuf23 i386 3.12.4-1+deb11u1 [976 kB] Get: 68 http://deb.debian.org/debian bullseye/main i386 libprotobuf-lite23 i386 3.12.4-1+deb11u1 [263 kB] Get: 69 http://deb.debian.org/debian bullseye/main i386 libprotobuf-dev i386 3.12.4-1+deb11u1 [1337 kB] Get: 70 http://deb.debian.org/debian bullseye/main i386 libprotoc23 i386 3.12.4-1+deb11u1 [872 kB] Get: 71 http://deb.debian.org/debian bullseye/main i386 protobuf-compiler i386 3.12.4-1+deb11u1 [75.4 kB] Get: 72 http://deb.debian.org/debian bullseye/main i386 golang-goprotobuf-dev i386 1.3.4-2+b6 [1442 kB] Get: 73 http://deb.debian.org/debian bullseye/main i386 golang-github-prometheus-client-model-dev all 0.2.0-2 [8476 B] Get: 74 http://deb.debian.org/debian bullseye/main i386 golang-github-dgrijalva-jwt-go-dev all 3.2.0-3 [33.6 kB] Get: 75 http://deb.debian.org/debian bullseye/main i386 golang-github-go-logfmt-logfmt-dev all 0.5.0-2 [13.4 kB] Get: 76 http://deb.debian.org/debian bullseye/main i386 golang-github-go-stack-stack-dev all 1.8.0-1 [8696 B] Get: 77 http://deb.debian.org/debian bullseye/main i386 golang-github-opentracing-opentracing-go-dev all 1.2.0-1 [29.4 kB] Get: 78 http://deb.debian.org/debian bullseye/main i386 golang-golang-x-text-dev all 0.3.6-1 [3857 kB] Get: 79 http://deb.debian.org/debian bullseye/main i386 golang-golang-x-net-dev all 1:0.0+git20210119.5f4716e+dfsg-4 [659 kB] Get: 80 http://deb.debian.org/debian bullseye/main i386 golang-golang-x-time-dev all 0.0+git20200630.3af7569-1 [10.1 kB] Get: 81 http://deb.debian.org/debian bullseye/main i386 golang-github-golang-mock-dev all 1.3.1-2 [35.1 kB] Get: 82 http://deb.debian.org/debian bullseye/main i386 golang-golang-x-xerrors-dev all 0.0~git20191204.9bdfabe-1 [14.2 kB] Get: 83 http://deb.debian.org/debian bullseye/main i386 golang-github-google-go-cmp-dev all 0.5.4-1 [83.7 kB] Get: 84 http://deb.debian.org/debian bullseye/main i386 golang-glog-dev all 0.0~git20160126.23def4e-3 [17.3 kB] Get: 85 http://deb.debian.org/debian bullseye/main i386 golang-golang-x-oauth2-dev all 0.0~git20190604.0f29369-2 [31.9 kB] Get: 86 http://deb.debian.org/debian bullseye/main i386 golang-google-cloud-compute-metadata-dev all 0.56.0-1 [34.0 kB] Get: 87 http://deb.debian.org/debian bullseye/main i386 golang-golang-x-oauth2-google-dev all 0.0~git20190604.0f29369-2 [13.2 kB] Get: 88 http://deb.debian.org/debian bullseye/main i386 golang-google-genproto-dev all 0.0~git20200413.b5235f6-1 [4037 kB] Get: 89 http://deb.debian.org/debian bullseye/main i386 golang-google-grpc-dev all 1.27.1-1 [478 kB] Get: 90 http://deb.debian.org/debian bullseye/main i386 golang-github-go-kit-kit-dev all 0.6.0-3 [103 kB] Get: 91 http://deb.debian.org/debian bullseye/main i386 golang-github-julienschmidt-httprouter-dev all 1.3.0-1 [19.3 kB] Get: 92 http://deb.debian.org/debian bullseye/main i386 golang-github-jpillora-backoff-dev all 1.0.0-1.1 [3712 B] Get: 93 http://deb.debian.org/debian bullseye/main i386 golang-github-mwitkow-go-conntrack-dev all 0.0~git20190716.2f06839-2 [14.6 kB] Get: 94 http://deb.debian.org/debian bullseye/main i386 golang-gopkg-alecthomas-kingpin.v2-dev all 2.2.6-3 [42.8 kB] Get: 95 http://deb.debian.org/debian bullseye/main i386 golang-gopkg-yaml.v2-dev all 2.4.0-1 [61.3 kB] Get: 96 http://deb.debian.org/debian bullseye/main i386 golang-protobuf-extensions-dev all 1.0.1-1 [29.6 kB] Get: 97 http://deb.debian.org/debian bullseye/main i386 golang-github-prometheus-common-dev all 0.15.0-2 [102 kB] Get: 98 http://deb.debian.org/debian bullseye/main i386 golang-golang-x-sync-dev all 0.0~git20210220.036812b-1 [19.1 kB] Get: 99 http://deb.debian.org/debian bullseye/main i386 golang-github-prometheus-procfs-dev all 0.3.0-2 [110 kB] Get: 100 http://deb.debian.org/debian bullseye/main i386 golang-github-prometheus-client-golang-dev all 1.9.0-2 [134 kB] Get: 101 http://deb.debian.org/debian bullseye/main i386 golang-github-armon-go-metrics-dev all 0.3.4-1 [30.5 kB] Get: 102 http://deb.debian.org/debian bullseye/main i386 golang-github-creack-pty-dev all 1.1.11-1 [8380 B] Get: 103 http://deb.debian.org/debian bullseye/main i386 golang-github-vmihailenco-tagparser-dev all 0.1.1-2 [4440 B] Get: 104 http://deb.debian.org/debian bullseye/main i386 golang-gopkg-vmihailenco-msgpack.v2-dev all 4.3.1-2 [27.5 kB] Get: 105 http://deb.debian.org/debian bullseye/main i386 golang-gopkg-tomb.v2-dev all 0.0~git20161208.d5d1b58-3 [6840 B] Get: 106 http://deb.debian.org/debian bullseye/main i386 libsasl2-modules-db i386 2.1.27+dfsg-2.1+deb11u1 [70.1 kB] Get: 107 http://deb.debian.org/debian bullseye/main i386 libsasl2-2 i386 2.1.27+dfsg-2.1+deb11u1 [110 kB] Get: 108 http://deb.debian.org/debian bullseye/main i386 libsasl2-dev i386 2.1.27+dfsg-2.1+deb11u1 [295 kB] Get: 109 http://deb.debian.org/debian bullseye/main i386 golang-github-kr-text-dev all 0.2.0-1 [11.1 kB] Get: 110 http://deb.debian.org/debian bullseye/main i386 golang-github-kr-pretty-dev all 0.2.1+git20200831.59b4212-1 [14.6 kB] Get: 111 http://deb.debian.org/debian bullseye/main i386 golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-1 [32.1 kB] Get: 112 http://deb.debian.org/debian bullseye/main i386 golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB] Get: 113 http://deb.debian.org/debian bullseye/main i386 golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB] Fetched 95.8 MB in 8s (12.5 MB/s) debconf: delaying package configuration, since apt-utils is not installed Selecting previously unselected package bsdextrautils. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 17763 files and directories currently installed.) Preparing to unpack .../000-bsdextrautils_2.36.1-8+deb11u1_i386.deb ... Unpacking bsdextrautils (2.36.1-8+deb11u1) ... Selecting previously unselected package libuchardet0:i386. Preparing to unpack .../001-libuchardet0_0.0.7-1_i386.deb ... Unpacking libuchardet0:i386 (0.0.7-1) ... Selecting previously unselected package groff-base. Preparing to unpack .../002-groff-base_1.22.4-6_i386.deb ... Unpacking groff-base (1.22.4-6) ... Selecting previously unselected package libpipeline1:i386. Preparing to unpack .../003-libpipeline1_1.5.3-1_i386.deb ... Unpacking libpipeline1:i386 (1.5.3-1) ... Selecting previously unselected package man-db. Preparing to unpack .../004-man-db_2.9.4-2_i386.deb ... Unpacking man-db (2.9.4-2) ... Selecting previously unselected package golang-golang-x-sys-dev. Preparing to unpack .../005-golang-golang-x-sys-dev_0.0~git20210124.22da62e-1_all.deb ... Unpacking golang-golang-x-sys-dev (0.0~git20210124.22da62e-1) ... Selecting previously unselected package golang-github-sirupsen-logrus-dev. Preparing to unpack .../006-golang-github-sirupsen-logrus-dev_1.7.0-2_all.deb ... Unpacking golang-github-sirupsen-logrus-dev (1.7.0-2) ... Selecting previously unselected package sensible-utils. Preparing to unpack .../007-sensible-utils_0.0.14_all.deb ... Unpacking sensible-utils (0.0.14) ... Selecting previously unselected package libmagic-mgc. Preparing to unpack .../008-libmagic-mgc_1%3a5.39-3+deb11u1_i386.deb ... Unpacking libmagic-mgc (1:5.39-3+deb11u1) ... Selecting previously unselected package libmagic1:i386. Preparing to unpack .../009-libmagic1_1%3a5.39-3+deb11u1_i386.deb ... Unpacking libmagic1:i386 (1:5.39-3+deb11u1) ... Selecting previously unselected package file. Preparing to unpack .../010-file_1%3a5.39-3+deb11u1_i386.deb ... Unpacking file (1:5.39-3+deb11u1) ... Selecting previously unselected package gettext-base. Preparing to unpack .../011-gettext-base_0.21-4_i386.deb ... Unpacking gettext-base (0.21-4) ... Selecting previously unselected package libsigsegv2:i386. Preparing to unpack .../012-libsigsegv2_2.13-1_i386.deb ... Unpacking libsigsegv2:i386 (2.13-1) ... Selecting previously unselected package m4. Preparing to unpack .../013-m4_1.4.18-5_i386.deb ... Unpacking m4 (1.4.18-5) ... Selecting previously unselected package autoconf. Preparing to unpack .../014-autoconf_2.69-14_all.deb ... Unpacking autoconf (2.69-14) ... Selecting previously unselected package autotools-dev. Preparing to unpack .../015-autotools-dev_20180224.1+nmu1_all.deb ... Unpacking autotools-dev (20180224.1+nmu1) ... Selecting previously unselected package automake. Preparing to unpack .../016-automake_1%3a1.16.3-2_all.deb ... Unpacking automake (1:1.16.3-2) ... Selecting previously unselected package autopoint. Preparing to unpack .../017-autopoint_0.21-4_all.deb ... Unpacking autopoint (0.21-4) ... Selecting previously unselected package libdebhelper-perl. Preparing to unpack .../018-libdebhelper-perl_13.3.4_all.deb ... Unpacking libdebhelper-perl (13.3.4) ... Selecting previously unselected package libtool. Preparing to unpack .../019-libtool_2.4.6-15_all.deb ... Unpacking libtool (2.4.6-15) ... Selecting previously unselected package dh-autoreconf. Preparing to unpack .../020-dh-autoreconf_20_all.deb ... Unpacking dh-autoreconf (20) ... Selecting previously unselected package libarchive-zip-perl. Preparing to unpack .../021-libarchive-zip-perl_1.68-1_all.deb ... Unpacking libarchive-zip-perl (1.68-1) ... Selecting previously unselected package libsub-override-perl. Preparing to unpack .../022-libsub-override-perl_0.09-2_all.deb ... Unpacking libsub-override-perl (0.09-2) ... Selecting previously unselected package libfile-stripnondeterminism-perl. Preparing to unpack .../023-libfile-stripnondeterminism-perl_1.12.0-1_all.deb ... Unpacking libfile-stripnondeterminism-perl (1.12.0-1) ... Selecting previously unselected package dh-strip-nondeterminism. Preparing to unpack .../024-dh-strip-nondeterminism_1.12.0-1_all.deb ... Unpacking dh-strip-nondeterminism (1.12.0-1) ... Selecting previously unselected package libelf1:i386. Preparing to unpack .../025-libelf1_0.183-1_i386.deb ... Unpacking libelf1:i386 (0.183-1) ... Selecting previously unselected package dwz. Preparing to unpack .../026-dwz_0.13+20210201-1_i386.deb ... Unpacking dwz (0.13+20210201-1) ... Selecting previously unselected package libicu67:i386. Preparing to unpack .../027-libicu67_67.1-7_i386.deb ... Unpacking libicu67:i386 (67.1-7) ... Selecting previously unselected package libxml2:i386. Preparing to unpack .../028-libxml2_2.9.10+dfsg-6.7+deb11u4_i386.deb ... Unpacking libxml2:i386 (2.9.10+dfsg-6.7+deb11u4) ... Selecting previously unselected package gettext. Preparing to unpack .../029-gettext_0.21-4_i386.deb ... Unpacking gettext (0.21-4) ... Selecting previously unselected package intltool-debian. Preparing to unpack .../030-intltool-debian_0.35.0+20060710.5_all.deb ... Unpacking intltool-debian (0.35.0+20060710.5) ... Selecting previously unselected package po-debconf. Preparing to unpack .../031-po-debconf_1.0.21+nmu1_all.deb ... Unpacking po-debconf (1.0.21+nmu1) ... Selecting previously unselected package debhelper. Preparing to unpack .../032-debhelper_13.3.4_all.deb ... Unpacking debhelper (13.3.4) ... Selecting previously unselected package dh-golang. Preparing to unpack .../033-dh-golang_1.51_all.deb ... Unpacking dh-golang (1.51) ... Selecting previously unselected package golang-1.15-src. Preparing to unpack .../034-golang-1.15-src_1.15.15-1~deb11u4_i386.deb ... Unpacking golang-1.15-src (1.15.15-1~deb11u4) ... Selecting previously unselected package golang-1.15-go. Preparing to unpack .../035-golang-1.15-go_1.15.15-1~deb11u4_i386.deb ... Unpacking golang-1.15-go (1.15.15-1~deb11u4) ... Selecting previously unselected package golang-src:i386. Preparing to unpack .../036-golang-src_2%3a1.15~1_i386.deb ... Unpacking golang-src:i386 (2:1.15~1) ... Selecting previously unselected package golang-go. Preparing to unpack .../037-golang-go_2%3a1.15~1_i386.deb ... Unpacking golang-go (2:1.15~1) ... Selecting previously unselected package golang-any:i386. Preparing to unpack .../038-golang-any_2%3a1.15~1_i386.deb ... Unpacking golang-any:i386 (2:1.15~1) ... Selecting previously unselected package golang-github-alecthomas-units-dev. Preparing to unpack .../039-golang-github-alecthomas-units-dev_0.0~git20201120.1786d5e-1_all.deb ... Unpacking golang-github-alecthomas-units-dev (0.0~git20201120.1786d5e-1) ... Selecting previously unselected package golang-github-circonus-labs-circonusllhist-dev. Preparing to unpack .../040-golang-github-circonus-labs-circonusllhist-dev_0.0~git20191022.ec08cde-1_all.deb ... Unpacking golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ... Selecting previously unselected package golang-github-hashicorp-go-cleanhttp-dev. Preparing to unpack .../041-golang-github-hashicorp-go-cleanhttp-dev_0.5.2-1_all.deb ... Unpacking golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ... Selecting previously unselected package golang-github-mattn-go-isatty-dev. Preparing to unpack .../042-golang-github-mattn-go-isatty-dev_0.0.12-1_all.deb ... Unpacking golang-github-mattn-go-isatty-dev (0.0.12-1) ... Selecting previously unselected package golang-github-mattn-go-colorable-dev. Preparing to unpack .../043-golang-github-mattn-go-colorable-dev_0.1.7-1_all.deb ... Unpacking golang-github-mattn-go-colorable-dev (0.1.7-1) ... Selecting previously unselected package golang-github-fatih-color-dev. Preparing to unpack .../044-golang-github-fatih-color-dev_1.7.0-1_all.deb ... Unpacking golang-github-fatih-color-dev (1.7.0-1) ... Selecting previously unselected package golang-github-davecgh-go-spew-dev. Preparing to unpack .../045-golang-github-davecgh-go-spew-dev_1.1.1-2_all.deb ... Unpacking golang-github-davecgh-go-spew-dev (1.1.1-2) ... Selecting previously unselected package golang-github-pmezard-go-difflib-dev. Preparing to unpack .../046-golang-github-pmezard-go-difflib-dev_1.0.0-3_all.deb ... Unpacking golang-github-pmezard-go-difflib-dev (1.0.0-3) ... Selecting previously unselected package golang-github-stretchr-objx-dev. Preparing to unpack .../047-golang-github-stretchr-objx-dev_0.3.0-1_all.deb ... Unpacking golang-github-stretchr-objx-dev (0.3.0-1) ... Selecting previously unselected package golang-gopkg-yaml.v3-dev. Preparing to unpack .../048-golang-gopkg-yaml.v3-dev_3.0.0~git20200121.a6ecf24-3_all.deb ... Unpacking golang-gopkg-yaml.v3-dev (3.0.0~git20200121.a6ecf24-3) ... Selecting previously unselected package golang-github-stretchr-testify-dev. Preparing to unpack .../049-golang-github-stretchr-testify-dev_1.6.1-2_all.deb ... Unpacking golang-github-stretchr-testify-dev (1.6.1-2) ... Selecting previously unselected package golang-github-hashicorp-go-hclog-dev. Preparing to unpack .../050-golang-github-hashicorp-go-hclog-dev_0.11.0-1_all.deb ... Unpacking golang-github-hashicorp-go-hclog-dev (0.11.0-1) ... Selecting previously unselected package golang-github-hashicorp-go-retryablehttp-dev. Preparing to unpack .../051-golang-github-hashicorp-go-retryablehttp-dev_0.6.4-2_all.deb ... Unpacking golang-github-hashicorp-go-retryablehttp-dev (0.6.4-2) ... Selecting previously unselected package golang-github-pkg-errors-dev. Preparing to unpack .../052-golang-github-pkg-errors-dev_0.9.1-1_all.deb ... Unpacking golang-github-pkg-errors-dev (0.9.1-1) ... Selecting previously unselected package golang-github-tv42-httpunix-dev. Preparing to unpack .../053-golang-github-tv42-httpunix-dev_0.0~git20150427.b75d861-2_all.deb ... Unpacking golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-2) ... Selecting previously unselected package golang-github-circonus-labs-circonus-gometrics-dev. Preparing to unpack .../054-golang-github-circonus-labs-circonus-gometrics-dev_2.3.1-3_all.deb ... Unpacking golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-3) ... Selecting previously unselected package golang-github-datadog-datadog-go-dev. Preparing to unpack .../055-golang-github-datadog-datadog-go-dev_2.1.0-3_all.deb ... Unpacking golang-github-datadog-datadog-go-dev (2.1.0-3) ... Selecting previously unselected package golang-github-hashicorp-go-uuid-dev. Preparing to unpack .../056-golang-github-hashicorp-go-uuid-dev_1.0.2-1_all.deb ... Unpacking golang-github-hashicorp-go-uuid-dev (1.0.2-1) ... Selecting previously unselected package golang-github-hashicorp-golang-lru-dev. Preparing to unpack .../057-golang-github-hashicorp-golang-lru-dev_0.5.4-2_all.deb ... Unpacking golang-github-hashicorp-golang-lru-dev (0.5.4-2) ... Selecting previously unselected package golang-github-hashicorp-go-immutable-radix-dev. Preparing to unpack .../058-golang-github-hashicorp-go-immutable-radix-dev_1.3.0-1_all.deb ... Unpacking golang-github-hashicorp-go-immutable-radix-dev (1.3.0-1) ... Selecting previously unselected package golang-github-pascaldekloe-goe-dev. Preparing to unpack .../059-golang-github-pascaldekloe-goe-dev_0.1.0-2_all.deb ... Unpacking golang-github-pascaldekloe-goe-dev (0.1.0-2) ... Selecting previously unselected package golang-github-beorn7-perks-dev. Preparing to unpack .../060-golang-github-beorn7-perks-dev_1.0.1-1_all.deb ... Unpacking golang-github-beorn7-perks-dev (1.0.1-1) ... Selecting previously unselected package golang-github-cespare-xxhash-dev. Preparing to unpack .../061-golang-github-cespare-xxhash-dev_2.1.1-1_all.deb ... Unpacking golang-github-cespare-xxhash-dev (2.1.1-1) ... Selecting previously unselected package golang-github-modern-go-concurrent-dev. Preparing to unpack .../062-golang-github-modern-go-concurrent-dev_1.0.3-1.1_all.deb ... Unpacking golang-github-modern-go-concurrent-dev (1.0.3-1.1) ... Selecting previously unselected package golang-github-modern-go-reflect2-dev. Preparing to unpack .../063-golang-github-modern-go-reflect2-dev_1.0.1-1_all.deb ... Unpacking golang-github-modern-go-reflect2-dev (1.0.1-1) ... Selecting previously unselected package golang-github-json-iterator-go-dev. Preparing to unpack .../064-golang-github-json-iterator-go-dev_1.1.10-2_all.deb ... Unpacking golang-github-json-iterator-go-dev (1.1.10-2) ... Selecting previously unselected package zlib1g-dev:i386. Preparing to unpack .../065-zlib1g-dev_1%3a1.2.11.dfsg-2+deb11u2_i386.deb ... Unpacking zlib1g-dev:i386 (1:1.2.11.dfsg-2+deb11u2) ... Selecting previously unselected package libprotobuf23:i386. Preparing to unpack .../066-libprotobuf23_3.12.4-1+deb11u1_i386.deb ... Unpacking libprotobuf23:i386 (3.12.4-1+deb11u1) ... Selecting previously unselected package libprotobuf-lite23:i386. Preparing to unpack .../067-libprotobuf-lite23_3.12.4-1+deb11u1_i386.deb ... Unpacking libprotobuf-lite23:i386 (3.12.4-1+deb11u1) ... Selecting previously unselected package libprotobuf-dev:i386. Preparing to unpack .../068-libprotobuf-dev_3.12.4-1+deb11u1_i386.deb ... Unpacking libprotobuf-dev:i386 (3.12.4-1+deb11u1) ... Selecting previously unselected package libprotoc23:i386. Preparing to unpack .../069-libprotoc23_3.12.4-1+deb11u1_i386.deb ... Unpacking libprotoc23:i386 (3.12.4-1+deb11u1) ... Selecting previously unselected package protobuf-compiler. Preparing to unpack .../070-protobuf-compiler_3.12.4-1+deb11u1_i386.deb ... Unpacking protobuf-compiler (3.12.4-1+deb11u1) ... Selecting previously unselected package golang-goprotobuf-dev. Preparing to unpack .../071-golang-goprotobuf-dev_1.3.4-2+b6_i386.deb ... Unpacking golang-goprotobuf-dev (1.3.4-2+b6) ... Selecting previously unselected package golang-github-prometheus-client-model-dev. Preparing to unpack .../072-golang-github-prometheus-client-model-dev_0.2.0-2_all.deb ... Unpacking golang-github-prometheus-client-model-dev (0.2.0-2) ... Selecting previously unselected package golang-github-dgrijalva-jwt-go-dev. Preparing to unpack .../073-golang-github-dgrijalva-jwt-go-dev_3.2.0-3_all.deb ... Unpacking golang-github-dgrijalva-jwt-go-dev (3.2.0-3) ... Selecting previously unselected package golang-github-go-logfmt-logfmt-dev. Preparing to unpack .../074-golang-github-go-logfmt-logfmt-dev_0.5.0-2_all.deb ... Unpacking golang-github-go-logfmt-logfmt-dev (0.5.0-2) ... Selecting previously unselected package golang-github-go-stack-stack-dev. Preparing to unpack .../075-golang-github-go-stack-stack-dev_1.8.0-1_all.deb ... Unpacking golang-github-go-stack-stack-dev (1.8.0-1) ... Selecting previously unselected package golang-github-opentracing-opentracing-go-dev. Preparing to unpack .../076-golang-github-opentracing-opentracing-go-dev_1.2.0-1_all.deb ... Unpacking golang-github-opentracing-opentracing-go-dev (1.2.0-1) ... Selecting previously unselected package golang-golang-x-text-dev. Preparing to unpack .../077-golang-golang-x-text-dev_0.3.6-1_all.deb ... Unpacking golang-golang-x-text-dev (0.3.6-1) ... Selecting previously unselected package golang-golang-x-net-dev. Preparing to unpack .../078-golang-golang-x-net-dev_1%3a0.0+git20210119.5f4716e+dfsg-4_all.deb ... Unpacking golang-golang-x-net-dev (1:0.0+git20210119.5f4716e+dfsg-4) ... Selecting previously unselected package golang-golang-x-time-dev. Preparing to unpack .../079-golang-golang-x-time-dev_0.0+git20200630.3af7569-1_all.deb ... Unpacking golang-golang-x-time-dev (0.0+git20200630.3af7569-1) ... Selecting previously unselected package golang-github-golang-mock-dev. Preparing to unpack .../080-golang-github-golang-mock-dev_1.3.1-2_all.deb ... Unpacking golang-github-golang-mock-dev (1.3.1-2) ... Selecting previously unselected package golang-golang-x-xerrors-dev. Preparing to unpack .../081-golang-golang-x-xerrors-dev_0.0~git20191204.9bdfabe-1_all.deb ... Unpacking golang-golang-x-xerrors-dev (0.0~git20191204.9bdfabe-1) ... Selecting previously unselected package golang-github-google-go-cmp-dev. Preparing to unpack .../082-golang-github-google-go-cmp-dev_0.5.4-1_all.deb ... Unpacking golang-github-google-go-cmp-dev (0.5.4-1) ... Selecting previously unselected package golang-glog-dev. Preparing to unpack .../083-golang-glog-dev_0.0~git20160126.23def4e-3_all.deb ... Unpacking golang-glog-dev (0.0~git20160126.23def4e-3) ... Selecting previously unselected package golang-golang-x-oauth2-dev. Preparing to unpack .../084-golang-golang-x-oauth2-dev_0.0~git20190604.0f29369-2_all.deb ... Unpacking golang-golang-x-oauth2-dev (0.0~git20190604.0f29369-2) ... Selecting previously unselected package golang-google-cloud-compute-metadata-dev. Preparing to unpack .../085-golang-google-cloud-compute-metadata-dev_0.56.0-1_all.deb ... Unpacking golang-google-cloud-compute-metadata-dev (0.56.0-1) ... Selecting previously unselected package golang-golang-x-oauth2-google-dev. Preparing to unpack .../086-golang-golang-x-oauth2-google-dev_0.0~git20190604.0f29369-2_all.deb ... Unpacking golang-golang-x-oauth2-google-dev (0.0~git20190604.0f29369-2) ... Selecting previously unselected package golang-google-genproto-dev. Preparing to unpack .../087-golang-google-genproto-dev_0.0~git20200413.b5235f6-1_all.deb ... Unpacking golang-google-genproto-dev (0.0~git20200413.b5235f6-1) ... Selecting previously unselected package golang-google-grpc-dev. Preparing to unpack .../088-golang-google-grpc-dev_1.27.1-1_all.deb ... Unpacking golang-google-grpc-dev (1.27.1-1) ... Selecting previously unselected package golang-github-go-kit-kit-dev. Preparing to unpack .../089-golang-github-go-kit-kit-dev_0.6.0-3_all.deb ... Unpacking golang-github-go-kit-kit-dev (0.6.0-3) ... Selecting previously unselected package golang-github-julienschmidt-httprouter-dev. Preparing to unpack .../090-golang-github-julienschmidt-httprouter-dev_1.3.0-1_all.deb ... Unpacking golang-github-julienschmidt-httprouter-dev (1.3.0-1) ... Selecting previously unselected package golang-github-jpillora-backoff-dev. Preparing to unpack .../091-golang-github-jpillora-backoff-dev_1.0.0-1.1_all.deb ... Unpacking golang-github-jpillora-backoff-dev (1.0.0-1.1) ... Selecting previously unselected package golang-github-mwitkow-go-conntrack-dev. Preparing to unpack .../092-golang-github-mwitkow-go-conntrack-dev_0.0~git20190716.2f06839-2_all.deb ... Unpacking golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-2) ... Selecting previously unselected package golang-gopkg-alecthomas-kingpin.v2-dev. Preparing to unpack .../093-golang-gopkg-alecthomas-kingpin.v2-dev_2.2.6-3_all.deb ... Unpacking golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-3) ... Selecting previously unselected package golang-gopkg-yaml.v2-dev. Preparing to unpack .../094-golang-gopkg-yaml.v2-dev_2.4.0-1_all.deb ... Unpacking golang-gopkg-yaml.v2-dev (2.4.0-1) ... Selecting previously unselected package golang-protobuf-extensions-dev. Preparing to unpack .../095-golang-protobuf-extensions-dev_1.0.1-1_all.deb ... Unpacking golang-protobuf-extensions-dev (1.0.1-1) ... Selecting previously unselected package golang-github-prometheus-common-dev. Preparing to unpack .../096-golang-github-prometheus-common-dev_0.15.0-2_all.deb ... Unpacking golang-github-prometheus-common-dev (0.15.0-2) ... Selecting previously unselected package golang-golang-x-sync-dev. Preparing to unpack .../097-golang-golang-x-sync-dev_0.0~git20210220.036812b-1_all.deb ... Unpacking golang-golang-x-sync-dev (0.0~git20210220.036812b-1) ... Selecting previously unselected package golang-github-prometheus-procfs-dev. Preparing to unpack .../098-golang-github-prometheus-procfs-dev_0.3.0-2_all.deb ... Unpacking golang-github-prometheus-procfs-dev (0.3.0-2) ... Selecting previously unselected package golang-github-prometheus-client-golang-dev. Preparing to unpack .../099-golang-github-prometheus-client-golang-dev_1.9.0-2_all.deb ... Unpacking golang-github-prometheus-client-golang-dev (1.9.0-2) ... Selecting previously unselected package golang-github-armon-go-metrics-dev. Preparing to unpack .../100-golang-github-armon-go-metrics-dev_0.3.4-1_all.deb ... Unpacking golang-github-armon-go-metrics-dev (0.3.4-1) ... Selecting previously unselected package golang-github-creack-pty-dev. Preparing to unpack .../101-golang-github-creack-pty-dev_1.1.11-1_all.deb ... Unpacking golang-github-creack-pty-dev (1.1.11-1) ... Selecting previously unselected package golang-github-vmihailenco-tagparser-dev. Preparing to unpack .../102-golang-github-vmihailenco-tagparser-dev_0.1.1-2_all.deb ... Unpacking golang-github-vmihailenco-tagparser-dev (0.1.1-2) ... Selecting previously unselected package golang-gopkg-vmihailenco-msgpack.v2-dev. Preparing to unpack .../103-golang-gopkg-vmihailenco-msgpack.v2-dev_4.3.1-2_all.deb ... Unpacking golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ... Selecting previously unselected package golang-gopkg-tomb.v2-dev. Preparing to unpack .../104-golang-gopkg-tomb.v2-dev_0.0~git20161208.d5d1b58-3_all.deb ... Unpacking golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ... Selecting previously unselected package libsasl2-modules-db:i386. Preparing to unpack .../105-libsasl2-modules-db_2.1.27+dfsg-2.1+deb11u1_i386.deb ... Unpacking libsasl2-modules-db:i386 (2.1.27+dfsg-2.1+deb11u1) ... Selecting previously unselected package libsasl2-2:i386. Preparing to unpack .../106-libsasl2-2_2.1.27+dfsg-2.1+deb11u1_i386.deb ... Unpacking libsasl2-2:i386 (2.1.27+dfsg-2.1+deb11u1) ... Selecting previously unselected package libsasl2-dev. Preparing to unpack .../107-libsasl2-dev_2.1.27+dfsg-2.1+deb11u1_i386.deb ... Unpacking libsasl2-dev (2.1.27+dfsg-2.1+deb11u1) ... Selecting previously unselected package golang-github-kr-text-dev. Preparing to unpack .../108-golang-github-kr-text-dev_0.2.0-1_all.deb ... Unpacking golang-github-kr-text-dev (0.2.0-1) ... Selecting previously unselected package golang-github-kr-pretty-dev. Preparing to unpack .../109-golang-github-kr-pretty-dev_0.2.1+git20200831.59b4212-1_all.deb ... Unpacking golang-github-kr-pretty-dev (0.2.1+git20200831.59b4212-1) ... Selecting previously unselected package golang-gopkg-check.v1-dev. Preparing to unpack .../110-golang-gopkg-check.v1-dev_0.0+git20200902.038fdea-1_all.deb ... Unpacking golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ... Selecting previously unselected package golang-gopkg-mgo.v2-dev. Preparing to unpack .../111-golang-gopkg-mgo.v2-dev_2016.08.01-7_all.deb ... Unpacking golang-gopkg-mgo.v2-dev (2016.08.01-7) ... Selecting previously unselected package golang-github-hashicorp-go-msgpack-dev. Preparing to unpack .../112-golang-github-hashicorp-go-msgpack-dev_0.5.5-1_all.deb ... Unpacking golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ... Setting up libpipeline1:i386 (1.5.3-1) ... Setting up golang-gopkg-yaml.v2-dev (2.4.0-1) ... Setting up golang-github-hashicorp-go-uuid-dev (1.0.2-1) ... Setting up bsdextrautils (2.36.1-8+deb11u1) ... update-alternatives: using /usr/bin/write.ul to provide /usr/bin/write (write) in auto mode Setting up golang-github-creack-pty-dev (1.1.11-1) ... Setting up libicu67:i386 (67.1-7) ... Setting up golang-github-golang-mock-dev (1.3.1-2) ... Setting up golang-github-stretchr-objx-dev (0.3.0-1) ... Setting up libmagic-mgc (1:5.39-3+deb11u1) ... Setting up golang-github-pkg-errors-dev (0.9.1-1) ... Setting up golang-github-hashicorp-golang-lru-dev (0.5.4-2) ... Setting up golang-1.15-src (1.15.15-1~deb11u4) ... Setting up libarchive-zip-perl (1.68-1) ... Setting up golang-gopkg-yaml.v3-dev (3.0.0~git20200121.a6ecf24-3) ... Setting up libdebhelper-perl (13.3.4) ... Setting up golang-golang-x-sys-dev (0.0~git20210124.22da62e-1) ... Setting up libmagic1:i386 (1:5.39-3+deb11u1) ... Setting up golang-github-pmezard-go-difflib-dev (1.0.0-3) ... Setting up golang-github-modern-go-concurrent-dev (1.0.3-1.1) ... Setting up gettext-base (0.21-4) ... Setting up golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ... Setting up libprotobuf23:i386 (3.12.4-1+deb11u1) ... Setting up file (1:5.39-3+deb11u1) ... Setting up golang-github-go-stack-stack-dev (1.8.0-1) ... Setting up golang-github-beorn7-perks-dev (1.0.1-1) ... Setting up golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ... Setting up libsasl2-modules-db:i386 (2.1.27+dfsg-2.1+deb11u1) ... Setting up golang-github-cespare-xxhash-dev (2.1.1-1) ... Setting up golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ... Setting up golang-github-jpillora-backoff-dev (1.0.0-1.1) ... Setting up golang-github-davecgh-go-spew-dev (1.1.1-2) ... Setting up libprotobuf-lite23:i386 (3.12.4-1+deb11u1) ... Setting up autotools-dev (20180224.1+nmu1) ... Setting up golang-github-pascaldekloe-goe-dev (0.1.0-2) ... Setting up golang-github-go-logfmt-logfmt-dev (0.5.0-2) ... Setting up golang-github-kr-text-dev (0.2.0-1) ... Setting up golang-github-stretchr-testify-dev (1.6.1-2) ... Setting up golang-github-hashicorp-go-immutable-radix-dev (1.3.0-1) ... Setting up libprotoc23:i386 (3.12.4-1+deb11u1) ... Setting up libsigsegv2:i386 (2.13-1) ... Setting up golang-golang-x-sync-dev (0.0~git20210220.036812b-1) ... Setting up autopoint (0.21-4) ... Setting up golang-glog-dev (0.0~git20160126.23def4e-3) ... Setting up libsasl2-2:i386 (2.1.27+dfsg-2.1+deb11u1) ... Setting up golang-github-julienschmidt-httprouter-dev (1.3.0-1) ... Setting up zlib1g-dev:i386 (1:1.2.11.dfsg-2+deb11u2) ... Setting up golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-2) ... Setting up sensible-utils (0.0.14) ... Setting up golang-golang-x-text-dev (0.3.6-1) ... Setting up libuchardet0:i386 (0.0.7-1) ... Setting up golang-github-prometheus-procfs-dev (0.3.0-2) ... Setting up golang-github-alecthomas-units-dev (0.0~git20201120.1786d5e-1) ... Setting up golang-github-modern-go-reflect2-dev (1.0.1-1) ... Setting up libsub-override-perl (0.09-2) ... Setting up golang-github-dgrijalva-jwt-go-dev (3.2.0-3) ... Setting up golang-github-datadog-datadog-go-dev (2.1.0-3) ... Setting up golang-golang-x-xerrors-dev (0.0~git20191204.9bdfabe-1) ... Setting up golang-src:i386 (2:1.15~1) ... Setting up golang-1.15-go (1.15.15-1~deb11u4) ... Setting up libelf1:i386 (0.183-1) ... Setting up golang-github-vmihailenco-tagparser-dev (0.1.1-2) ... Setting up libxml2:i386 (2.9.10+dfsg-6.7+deb11u4) ... Setting up golang-github-kr-pretty-dev (0.2.1+git20200831.59b4212-1) ... Setting up libfile-stripnondeterminism-perl (1.12.0-1) ... Setting up golang-github-google-go-cmp-dev (0.5.4-1) ... Setting up golang-github-mattn-go-isatty-dev (0.0.12-1) ... Setting up golang-github-sirupsen-logrus-dev (1.7.0-2) ... Setting up golang-github-opentracing-opentracing-go-dev (1.2.0-1) ... Setting up gettext (0.21-4) ... Setting up protobuf-compiler (3.12.4-1+deb11u1) ... Setting up golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ... Setting up golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-3) ... Setting up libtool (2.4.6-15) ... Setting up golang-go (2:1.15~1) ... Setting up golang-github-mattn-go-colorable-dev (0.1.7-1) ... Setting up m4 (1.4.18-5) ... Setting up golang-github-json-iterator-go-dev (1.1.10-2) ... Setting up golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ... Setting up golang-any:i386 (2:1.15~1) ... Setting up intltool-debian (0.35.0+20060710.5) ... Setting up libprotobuf-dev:i386 (3.12.4-1+deb11u1) ... Setting up libsasl2-dev (2.1.27+dfsg-2.1+deb11u1) ... Setting up golang-gopkg-mgo.v2-dev (2016.08.01-7) ... Setting up golang-goprotobuf-dev (1.3.4-2+b6) ... Setting up golang-golang-x-net-dev (1:0.0+git20210119.5f4716e+dfsg-4) ... Setting up autoconf (2.69-14) ... Setting up dh-strip-nondeterminism (1.12.0-1) ... Setting up dwz (0.13+20210201-1) ... Setting up groff-base (1.22.4-6) ... Setting up golang-github-prometheus-client-model-dev (0.2.0-2) ... Setting up golang-golang-x-oauth2-dev (0.0~git20190604.0f29369-2) ... Setting up automake (1:1.16.3-2) ... update-alternatives: using /usr/bin/automake-1.16 to provide /usr/bin/automake (automake) in auto mode Setting up golang-golang-x-time-dev (0.0+git20200630.3af7569-1) ... Setting up golang-github-fatih-color-dev (1.7.0-1) ... Setting up golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ... Setting up po-debconf (1.0.21+nmu1) ... Setting up golang-protobuf-extensions-dev (1.0.1-1) ... Setting up golang-google-cloud-compute-metadata-dev (0.56.0-1) ... Setting up man-db (2.9.4-2) ... Not building database; man-db/auto-update is not 'true'. Setting up dh-autoreconf (20) ... Setting up golang-golang-x-oauth2-google-dev (0.0~git20190604.0f29369-2) ... Setting up golang-github-hashicorp-go-hclog-dev (0.11.0-1) ... Setting up debhelper (13.3.4) ... Setting up golang-github-hashicorp-go-retryablehttp-dev (0.6.4-2) ... Setting up dh-golang (1.51) ... Setting up golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-3) ... Setting up golang-google-genproto-dev (0.0~git20200413.b5235f6-1) ... Setting up golang-github-prometheus-client-golang-dev (1.9.0-2) ... Setting up golang-github-armon-go-metrics-dev (0.3.4-1) ... Setting up golang-google-grpc-dev (1.27.1-1) ... Setting up golang-github-go-kit-kit-dev (0.6.0-3) ... Setting up golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-2) ... Setting up golang-github-prometheus-common-dev (0.15.0-2) ... Processing triggers for libc-bin (2.31-13+deb11u6) ... Reading package lists... Building dependency tree... Reading state information... Reading extended state information... Initializing package states... Writing extended state information... Building tag database... -> Finished parsing the build-deps Reading package lists... Building dependency tree... Reading state information... fakeroot is already the newest version (1.25.3-1.1). 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. I: Building the package I: Running cd /build/reproducible-path/golang-github-hashicorp-raft-1.1.2/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-genchanges -S > ../golang-github-hashicorp-raft_1.1.2-1_source.changes dpkg-buildpackage: info: source package golang-github-hashicorp-raft dpkg-buildpackage: info: source version 1.1.2-1 dpkg-buildpackage: info: source distribution unstable dpkg-buildpackage: info: source changed by Dmitry Smirnov dpkg-source --before-build . dpkg-buildpackage: info: host architecture i386 fakeroot debian/rules clean dh clean --buildsystem=golang --with=golang dh_auto_clean -O--buildsystem=golang dh_autoreconf_clean -O--buildsystem=golang dh_clean -O--buildsystem=golang debian/rules build dh build --buildsystem=golang --with=golang dh_update_autotools_config -O--buildsystem=golang dh_autoreconf -O--buildsystem=golang dh_auto_configure -O--buildsystem=golang dh_auto_build -O--buildsystem=golang cd obj-i686-linux-gnu && go install -trimpath -v -p 8 github.com/hashicorp/raft github.com/hashicorp/raft/bench internal/unsafeheader runtime/internal/atomic unicode/utf8 internal/cpu container/list internal/race runtime/internal/sys sync/atomic unicode crypto/internal/subtle crypto/subtle math/bits encoding internal/testlog unicode/utf16 internal/nettrace runtime/internal/math vendor/golang.org/x/crypto/cryptobyte/asn1 vendor/golang.org/x/crypto/internal/subtle golang.org/x/sys/internal/unsafeheader runtime/cgo internal/bytealg math runtime internal/reflectlite sync math/rand internal/singleflight errors sort io internal/oserror strconv github.com/hashicorp/golang-lru/simplelru vendor/golang.org/x/net/dns/dnsmessage syscall bytes strings crypto/internal/randutil hash hash/crc32 crypto/hmac hash/crc64 vendor/golang.org/x/crypto/hkdf crypto crypto/rc4 reflect vendor/golang.org/x/text/transform bufio path regexp/syntax github.com/hashicorp/go-immutable-radix html internal/syscall/execenv internal/syscall/unix time regexp context internal/poll encoding/binary internal/fmtsort os crypto/cipher crypto/sha256 encoding/base64 crypto/sha512 crypto/ed25519/internal/edwards25519 crypto/md5 crypto/sha1 vendor/golang.org/x/crypto/poly1305 encoding/pem crypto/aes crypto/des vendor/golang.org/x/crypto/chacha20 golang.org/x/sys/unix fmt os/signal path/filepath runtime/debug net vendor/golang.org/x/crypto/chacha20poly1305 io/ioutil encoding/json log encoding/hex net/url compress/flate math/big vendor/golang.org/x/crypto/curve25519 vendor/golang.org/x/text/unicode/bidi vendor/golang.org/x/text/unicode/norm vendor/golang.org/x/net/http2/hpack compress/gzip mime mime/quotedprintable vendor/golang.org/x/text/secure/bidirule net/http/internal github.com/mattn/go-isatty encoding/gob go/token text/template/parse github.com/mattn/go-colorable flag github.com/fatih/color runtime/trace vendor/golang.org/x/net/idna crypto/rand crypto/elliptic encoding/asn1 crypto/dsa crypto/ed25519 crypto/rsa github.com/hashicorp/go-hclog text/template testing crypto/x509/pkix vendor/golang.org/x/crypto/cryptobyte crypto/ecdsa html/template vendor/golang.org/x/net/http/httpproxy net/textproto crypto/x509 vendor/golang.org/x/net/http/httpguts mime/multipart crypto/tls net/http/httptrace net/http github.com/armon/go-metrics net/rpc github.com/hashicorp/go-msgpack/codec github.com/hashicorp/raft github.com/hashicorp/raft/bench dh_auto_test -O--buildsystem=golang cd obj-i686-linux-gnu && go test -vet=off -v -p 8 github.com/hashicorp/raft github.com/hashicorp/raft/bench === RUN TestCommitment_setVoters --- PASS: TestCommitment_setVoters (0.00s) === RUN TestCommitment_match_max --- PASS: TestCommitment_match_max (0.00s) === RUN TestCommitment_match_nonVoting --- PASS: TestCommitment_match_nonVoting (0.00s) === RUN TestCommitment_recalculate --- PASS: TestCommitment_recalculate (0.00s) === RUN TestCommitment_recalculate_startIndex --- PASS: TestCommitment_recalculate_startIndex (0.00s) === RUN TestCommitment_noVoterSanity --- PASS: TestCommitment_noVoterSanity (0.00s) === RUN TestCommitment_singleVoter --- PASS: TestCommitment_singleVoter (0.00s) === RUN TestConfiguration_Configuration_Clone --- PASS: TestConfiguration_Configuration_Clone (0.00s) === RUN TestConfiguration_configurations_Clone --- PASS: TestConfiguration_configurations_Clone (0.00s) === RUN TestConfiguration_hasVote --- PASS: TestConfiguration_hasVote (0.00s) === RUN TestConfiguration_checkConfiguration --- PASS: TestConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_nextConfiguration_table --- PASS: TestConfiguration_nextConfiguration_table (0.00s) === RUN TestConfiguration_nextConfiguration_prevIndex --- PASS: TestConfiguration_nextConfiguration_prevIndex (0.00s) === RUN TestConfiguration_nextConfiguration_checkConfiguration --- PASS: TestConfiguration_nextConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_encodeDecodePeers --- PASS: TestConfiguration_encodeDecodePeers (0.00s) === RUN TestConfiguration_encodeDecodeConfiguration --- PASS: TestConfiguration_encodeDecodeConfiguration (0.00s) === RUN TestDiscardSnapshotStoreImpl --- PASS: TestDiscardSnapshotStoreImpl (0.00s) === RUN TestDiscardSnapshotSinkImpl --- PASS: TestDiscardSnapshotSinkImpl (0.00s) === RUN TestFileSnapshotStoreImpl --- PASS: TestFileSnapshotStoreImpl (0.00s) === RUN TestFileSnapshotSinkImpl --- PASS: TestFileSnapshotSinkImpl (0.00s) === RUN TestFileSS_CreateSnapshotMissingParentDir testing.go:153: 2024-01-09T05:09:48.750Z [INFO] creating new snapshot: path=/tmp/raft828672824/raft927827767/snapshots/3-10-1704776988750.tmp --- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.07s) === RUN TestFileSS_CreateSnapshot testing.go:153: 2024-01-09T05:09:48.817Z [INFO] creating new snapshot: path=/tmp/raft266540842/snapshots/3-10-1704776988817.tmp testing.go:153: 2024-01-09T05:09:48.831Z [WARN] found temporary snapshot: name=3-10-1704776988817.tmp --- PASS: TestFileSS_CreateSnapshot (0.04s) === RUN TestFileSS_CancelSnapshot testing.go:153: 2024-01-09T05:09:48.855Z [INFO] creating new snapshot: path=/tmp/raft125259905/snapshots/3-10-1704776988855.tmp --- PASS: TestFileSS_CancelSnapshot (0.01s) === RUN TestFileSS_Retention testing.go:153: 2024-01-09T05:09:48.863Z [INFO] creating new snapshot: path=/tmp/raft738891244/snapshots/3-10-1704776988863.tmp testing.go:153: 2024-01-09T05:09:48.873Z [INFO] creating new snapshot: path=/tmp/raft738891244/snapshots/3-11-1704776988873.tmp testing.go:153: 2024-01-09T05:09:48.891Z [INFO] creating new snapshot: path=/tmp/raft738891244/snapshots/3-12-1704776988891.tmp testing.go:153: 2024-01-09T05:09:48.899Z [INFO] reaping snapshot: path=/tmp/raft738891244/snapshots/3-10-1704776988863 testing.go:153: 2024-01-09T05:09:48.900Z [INFO] creating new snapshot: path=/tmp/raft738891244/snapshots/3-13-1704776988900.tmp testing.go:153: 2024-01-09T05:09:48.919Z [INFO] reaping snapshot: path=/tmp/raft738891244/snapshots/3-11-1704776988873 testing.go:153: 2024-01-09T05:09:48.919Z [INFO] creating new snapshot: path=/tmp/raft738891244/snapshots/3-14-1704776988919.tmp testing.go:153: 2024-01-09T05:09:48.943Z [INFO] reaping snapshot: path=/tmp/raft738891244/snapshots/3-12-1704776988891 --- PASS: TestFileSS_Retention (0.08s) === RUN TestFileSS_BadPerm --- PASS: TestFileSS_BadPerm (0.00s) === RUN TestFileSS_MissingParentDir --- PASS: TestFileSS_MissingParentDir (0.00s) === RUN TestFileSS_Ordering testing.go:153: 2024-01-09T05:09:48.944Z [INFO] creating new snapshot: path=/tmp/raft180599231/snapshots/5-130350-1704776988944.tmp testing.go:153: 2024-01-09T05:09:48.971Z [INFO] creating new snapshot: path=/tmp/raft180599231/snapshots/36-204917-1704776988971.tmp --- PASS: TestFileSS_Ordering (0.05s) === RUN TestDeferFutureSuccess --- PASS: TestDeferFutureSuccess (0.00s) === RUN TestDeferFutureError --- PASS: TestDeferFutureError (0.00s) === RUN TestDeferFutureConcurrent --- PASS: TestDeferFutureConcurrent (0.00s) === RUN TestInmemSnapshotStoreImpl --- PASS: TestInmemSnapshotStoreImpl (0.00s) === RUN TestInmemSnapshotSinkImpl --- PASS: TestInmemSnapshotSinkImpl (0.00s) === RUN TestInmemSS_CreateSnapshot --- PASS: TestInmemSS_CreateSnapshot (0.00s) === RUN TestInmemSS_OpenSnapshotTwice --- PASS: TestInmemSS_OpenSnapshotTwice (0.00s) === RUN TestInmemTransportImpl --- PASS: TestInmemTransportImpl (0.00s) === RUN TestInmemTransportWriteTimeout --- PASS: TestInmemTransportWriteTimeout (0.01s) === RUN TestRaft_Integ --- SKIP: TestRaft_Integ (0.00s) === RUN TestLogCache --- PASS: TestLogCache (0.00s) === RUN TestNetworkTransport_CloseStreams --- PASS: TestNetworkTransport_CloseStreams (0.00s) === RUN TestNetworkTransport_StartStop --- PASS: TestNetworkTransport_StartStop (0.00s) === RUN TestNetworkTransport_Heartbeat_FastPath --- PASS: TestNetworkTransport_Heartbeat_FastPath (0.00s) === RUN TestNetworkTransport_AppendEntries --- PASS: TestNetworkTransport_AppendEntries (0.00s) === RUN TestNetworkTransport_AppendEntriesPipeline --- PASS: TestNetworkTransport_AppendEntriesPipeline (0.01s) === RUN TestNetworkTransport_AppendEntriesPipeline_CloseStreams net_transport_test.go:339: Disabled unreliable test --- SKIP: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.00s) === RUN TestNetworkTransport_RequestVote --- PASS: TestNetworkTransport_RequestVote (0.00s) === RUN TestNetworkTransport_InstallSnapshot --- PASS: TestNetworkTransport_InstallSnapshot (0.00s) === RUN TestNetworkTransport_EncodeDecode --- PASS: TestNetworkTransport_EncodeDecode (0.00s) === RUN TestNetworkTransport_EncodeDecode_AddressProvider --- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s) === RUN TestNetworkTransport_PooledConn --- PASS: TestNetworkTransport_PooledConn (0.05s) === RUN TestNetworkTransport_ListenBackoff net_transport_test.go:718: countingWriter: 2024-01-09T05:09:49.075Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2024-01-09T05:09:49.083Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2024-01-09T05:09:49.095Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2024-01-09T05:09:49.118Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2024-01-09T05:09:49.163Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2024-01-09T05:09:49.246Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2024-01-09T05:09:49.411Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2024-01-09T05:09:49.734Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2024-01-09T05:09:50.378Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2024-01-09T05:09:51.382Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2024-01-09T05:09:52.383Z [ERROR] test: failed to accept connection: error="intentional error in test" --- PASS: TestNetworkTransport_ListenBackoff (4.01s) === RUN TestPeersJSON_BadConfiguration --- PASS: TestPeersJSON_BadConfiguration (0.00s) === RUN TestPeersJSON_ReadPeersJSON --- PASS: TestPeersJSON_ReadPeersJSON (0.00s) === RUN TestPeersJSON_ReadConfigJSON --- PASS: TestPeersJSON_ReadConfigJSON (0.00s) === RUN TestRaft_StartStop testLoggerAdapter verbose: server-55abb527-eaf0-2b2a-e90d-2328d695ea45: 2024-01-09T05:09:53.084Z [INFO] server-55abb527-eaf0-2b2a-e90d-2328d695ea45: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-55abb527-eaf0-2b2a-e90d-2328d695ea45 Address:55abb527-eaf0-2b2a-e90d-2328d695ea45}]" testing.go:149: server-55abb527-eaf0-2b2a-e90d-2328d695ea45: 2024-01-09T05:09:53.084Z [INFO] server-55abb527-eaf0-2b2a-e90d-2328d695ea45: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-55abb527-eaf0-2b2a-e90d-2328d695ea45 Address:55abb527-eaf0-2b2a-e90d-2328d695ea45}]" --- PASS: TestRaft_StartStop (0.00s) === RUN TestRaft_AfterShutdown testLoggerAdapter verbose: server-a997f564-6c22-dbcc-2988-3716bf7b1140: 2024-01-09T05:09:53.085Z [INFO] server-a997f564-6c22-dbcc-2988-3716bf7b1140: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a997f564-6c22-dbcc-2988-3716bf7b1140 Address:a997f564-6c22-dbcc-2988-3716bf7b1140}]" testing.go:149: server-a997f564-6c22-dbcc-2988-3716bf7b1140: 2024-01-09T05:09:53.085Z [INFO] server-a997f564-6c22-dbcc-2988-3716bf7b1140: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a997f564-6c22-dbcc-2988-3716bf7b1140 Address:a997f564-6c22-dbcc-2988-3716bf7b1140}]" --- PASS: TestRaft_AfterShutdown (0.00s) === RUN TestRaft_LiveBootstrap testLoggerAdapter verbose: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.086Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: initial configuration: index=0 servers=[] testing.go:149: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.086Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5: 2024-01-09T05:09:53.086Z [INFO] server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5: initial configuration: index=0 servers=[] testing.go:149: server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5: 2024-01-09T05:09:53.086Z [INFO] server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-656011a4-1086-512b-bfcf-946610766331: 2024-01-09T05:09:53.086Z [INFO] server-656011a4-1086-512b-bfcf-946610766331: initial configuration: index=0 servers=[] testing.go:149: server-656011a4-1086-512b-bfcf-946610766331: 2024-01-09T05:09:53.086Z [INFO] server-656011a4-1086-512b-bfcf-946610766331: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.086Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: entering follower state: follower="Node at a507f1fe-3378-abb6-7909-9a89e5cc2e43 [Follower]" leader= testing.go:149: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.086Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: entering follower state: follower="Node at a507f1fe-3378-abb6-7909-9a89e5cc2e43 [Follower]" leader= testLoggerAdapter verbose: server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5: 2024-01-09T05:09:53.086Z [INFO] server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5: entering follower state: follower="Node at 1552d3c7-5e4f-7085-5532-50bfdf8f2ab5 [Follower]" leader= testing.go:149: server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5: 2024-01-09T05:09:53.086Z [INFO] server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5: entering follower state: follower="Node at 1552d3c7-5e4f-7085-5532-50bfdf8f2ab5 [Follower]" leader= testLoggerAdapter verbose: server-656011a4-1086-512b-bfcf-946610766331: 2024-01-09T05:09:53.086Z [INFO] server-656011a4-1086-512b-bfcf-946610766331: entering follower state: follower="Node at 656011a4-1086-512b-bfcf-946610766331 [Follower]" leader= testing.go:149: server-656011a4-1086-512b-bfcf-946610766331: 2024-01-09T05:09:53.086Z [INFO] server-656011a4-1086-512b-bfcf-946610766331: entering follower state: follower="Node at 656011a4-1086-512b-bfcf-946610766331 [Follower]" leader= testLoggerAdapter verbose: cluster: 2024-01-09T05:09:53.086Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-01-09T05:09:53.086Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5: 2024-01-09T05:09:53.155Z [WARN] server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5: no known peers, aborting election testing.go:149: server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5: 2024-01-09T05:09:53.155Z [WARN] server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5: no known peers, aborting election testLoggerAdapter verbose: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.180Z [WARN] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.180Z [WARN] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.180Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: entering candidate state: node="Node at a507f1fe-3378-abb6-7909-9a89e5cc2e43 [Candidate]" term=2 testing.go:149: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.180Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: entering candidate state: node="Node at a507f1fe-3378-abb6-7909-9a89e5cc2e43 [Candidate]" term=2 testLoggerAdapter verbose: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.180Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: election won: tally=2 testing.go:149: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.180Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: election won: tally=2 testLoggerAdapter verbose: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.180Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: entering leader state: leader="Node at a507f1fe-3378-abb6-7909-9a89e5cc2e43 [Leader]" testing.go:149: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.180Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: entering leader state: leader="Node at a507f1fe-3378-abb6-7909-9a89e5cc2e43 [Leader]" testLoggerAdapter verbose: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.180Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: added peer, starting replication: peer=server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5 testing.go:149: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.180Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: added peer, starting replication: peer=server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5 testLoggerAdapter verbose: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.180Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: added peer, starting replication: peer=server-656011a4-1086-512b-bfcf-946610766331 testing.go:149: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.180Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: added peer, starting replication: peer=server-656011a4-1086-512b-bfcf-946610766331 testLoggerAdapter verbose: server-656011a4-1086-512b-bfcf-946610766331: 2024-01-09T05:09:53.181Z [WARN] server-656011a4-1086-512b-bfcf-946610766331: failed to get previous log: previous-index=1 last-index=0 error="log not found" testing.go:149: server-656011a4-1086-512b-bfcf-946610766331: 2024-01-09T05:09:53.181Z [WARN] server-656011a4-1086-512b-bfcf-946610766331: failed to get previous log: previous-index=1 last-index=0 error="log not found" testLoggerAdapter verbose: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.181Z [WARN] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: appendEntries rejected, sending older logs: peer="{Voter server-656011a4-1086-512b-bfcf-946610766331 656011a4-1086-512b-bfcf-946610766331}" next=1 testing.go:149: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.181Z [WARN] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: appendEntries rejected, sending older logs: peer="{Voter server-656011a4-1086-512b-bfcf-946610766331 656011a4-1086-512b-bfcf-946610766331}" next=1 testLoggerAdapter verbose: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.181Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: pipelining replication: peer="{Voter server-656011a4-1086-512b-bfcf-946610766331 656011a4-1086-512b-bfcf-946610766331}" testing.go:149: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.181Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: pipelining replication: peer="{Voter server-656011a4-1086-512b-bfcf-946610766331 656011a4-1086-512b-bfcf-946610766331}" testLoggerAdapter verbose: server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5: 2024-01-09T05:09:53.181Z [WARN] server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5: failed to get previous log: previous-index=1 last-index=0 error="log not found" testing.go:149: server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5: 2024-01-09T05:09:53.181Z [WARN] server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5: failed to get previous log: previous-index=1 last-index=0 error="log not found" testLoggerAdapter verbose: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.181Z [WARN] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: appendEntries rejected, sending older logs: peer="{Voter server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5 1552d3c7-5e4f-7085-5532-50bfdf8f2ab5}" next=1 testing.go:149: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.181Z [WARN] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: appendEntries rejected, sending older logs: peer="{Voter server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5 1552d3c7-5e4f-7085-5532-50bfdf8f2ab5}" next=1 testLoggerAdapter verbose: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.181Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: pipelining replication: peer="{Voter server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5 1552d3c7-5e4f-7085-5532-50bfdf8f2ab5}" testing.go:149: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.181Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: pipelining replication: peer="{Voter server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5 1552d3c7-5e4f-7085-5532-50bfdf8f2ab5}" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:53.286Z [INFO] stable state for Follower reached at 2024-01-09 05:09:53.181389363 +0000 UTC m=+4.434440333 (2 nodes), 94.771481ms from start of poll, 95.253921ms from cluster start. Timeout at 2024-01-09 05:09:53.286504464 +0000 UTC m=+4.539555593, 105.11526ms after stability testing.go:149: cluster: 2024-01-09T05:09:53.286Z [INFO] stable state for Follower reached at 2024-01-09 05:09:53.181389363 +0000 UTC m=+4.434440333 (2 nodes), 94.771481ms from start of poll, 95.253921ms from cluster start. Timeout at 2024-01-09 05:09:53.286504464 +0000 UTC m=+4.539555593, 105.11526ms after stability testLoggerAdapter verbose: cluster: 2024-01-09T05:09:53.286Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:09:53.286Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-01-09T05:09:53.391Z [INFO] stable state for Leader reached at 2024-01-09 05:09:53.286673581 +0000 UTC m=+4.539724578 (1 nodes), 2.159µs from start of poll, 200.538166ms from cluster start. Timeout at 2024-01-09 05:09:53.391729209 +0000 UTC m=+4.644780267, 105.055689ms after stability testing.go:149: cluster: 2024-01-09T05:09:53.391Z [INFO] stable state for Leader reached at 2024-01-09 05:09:53.286673581 +0000 UTC m=+4.539724578 (1 nodes), 2.159µs from start of poll, 200.538166ms from cluster start. Timeout at 2024-01-09 05:09:53.391729209 +0000 UTC m=+4.644780267, 105.055689ms after stability testLoggerAdapter verbose: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.402Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: aborting pipeline replication: peer="{Voter server-656011a4-1086-512b-bfcf-946610766331 656011a4-1086-512b-bfcf-946610766331}" testing.go:149: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.402Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: aborting pipeline replication: peer="{Voter server-656011a4-1086-512b-bfcf-946610766331 656011a4-1086-512b-bfcf-946610766331}" testLoggerAdapter verbose: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.402Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: aborting pipeline replication: peer="{Voter server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5 1552d3c7-5e4f-7085-5532-50bfdf8f2ab5}" testing.go:149: server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: 2024-01-09T05:09:53.402Z [INFO] server-a507f1fe-3378-abb6-7909-9a89e5cc2e43: aborting pipeline replication: peer="{Voter server-1552d3c7-5e4f-7085-5532-50bfdf8f2ab5 1552d3c7-5e4f-7085-5532-50bfdf8f2ab5}" --- PASS: TestRaft_LiveBootstrap (0.32s) === RUN TestRaft_RecoverCluster_NoState testLoggerAdapter verbose: server-7c429492-d4bd-b3ea-1a84-97cfd04660c0: 2024-01-09T05:09:53.403Z [INFO] server-7c429492-d4bd-b3ea-1a84-97cfd04660c0: initial configuration: index=0 servers=[] testing.go:149: server-7c429492-d4bd-b3ea-1a84-97cfd04660c0: 2024-01-09T05:09:53.403Z [INFO] server-7c429492-d4bd-b3ea-1a84-97cfd04660c0: initial configuration: index=0 servers=[] --- PASS: TestRaft_RecoverCluster_NoState (0.00s) === RUN TestRaft_RecoverCluster raft_test.go:119: Disabled unreliable test --- SKIP: TestRaft_RecoverCluster (0.00s) === RUN TestRaft_HasExistingState testLoggerAdapter verbose: server-725cb9e4-97f5-9799-ba23-5965e058cf3b: 2024-01-09T05:09:53.405Z [INFO] server-725cb9e4-97f5-9799-ba23-5965e058cf3b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-725cb9e4-97f5-9799-ba23-5965e058cf3b Address:725cb9e4-97f5-9799-ba23-5965e058cf3b} {Suffrage:Voter ID:server-3cfec63e-0c1f-b6f9-9a35-287122a41b24 Address:3cfec63e-0c1f-b6f9-9a35-287122a41b24}]" testing.go:149: server-725cb9e4-97f5-9799-ba23-5965e058cf3b: 2024-01-09T05:09:53.405Z [INFO] server-725cb9e4-97f5-9799-ba23-5965e058cf3b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-725cb9e4-97f5-9799-ba23-5965e058cf3b Address:725cb9e4-97f5-9799-ba23-5965e058cf3b} {Suffrage:Voter ID:server-3cfec63e-0c1f-b6f9-9a35-287122a41b24 Address:3cfec63e-0c1f-b6f9-9a35-287122a41b24}]" testLoggerAdapter verbose: server-725cb9e4-97f5-9799-ba23-5965e058cf3b: 2024-01-09T05:09:53.405Z [INFO] server-725cb9e4-97f5-9799-ba23-5965e058cf3b: entering follower state: follower="Node at 725cb9e4-97f5-9799-ba23-5965e058cf3b [Follower]" leader= testing.go:149: server-725cb9e4-97f5-9799-ba23-5965e058cf3b: 2024-01-09T05:09:53.405Z [INFO] server-725cb9e4-97f5-9799-ba23-5965e058cf3b: entering follower state: follower="Node at 725cb9e4-97f5-9799-ba23-5965e058cf3b [Follower]" leader= testLoggerAdapter verbose: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.406Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-725cb9e4-97f5-9799-ba23-5965e058cf3b Address:725cb9e4-97f5-9799-ba23-5965e058cf3b} {Suffrage:Voter ID:server-3cfec63e-0c1f-b6f9-9a35-287122a41b24 Address:3cfec63e-0c1f-b6f9-9a35-287122a41b24}]" testing.go:149: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.406Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-725cb9e4-97f5-9799-ba23-5965e058cf3b Address:725cb9e4-97f5-9799-ba23-5965e058cf3b} {Suffrage:Voter ID:server-3cfec63e-0c1f-b6f9-9a35-287122a41b24 Address:3cfec63e-0c1f-b6f9-9a35-287122a41b24}]" testLoggerAdapter verbose: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.406Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: entering follower state: follower="Node at 3cfec63e-0c1f-b6f9-9a35-287122a41b24 [Follower]" leader= testing.go:149: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.406Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: entering follower state: follower="Node at 3cfec63e-0c1f-b6f9-9a35-287122a41b24 [Follower]" leader= testLoggerAdapter verbose: server-b758e9ef-de23-6e38-0468-f7931f6b8f05: 2024-01-09T05:09:53.423Z [INFO] server-b758e9ef-de23-6e38-0468-f7931f6b8f05: initial configuration: index=0 servers=[] testing.go:149: server-b758e9ef-de23-6e38-0468-f7931f6b8f05: 2024-01-09T05:09:53.423Z [INFO] server-b758e9ef-de23-6e38-0468-f7931f6b8f05: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2024-01-09T05:09:53.423Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:09:53.423Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-b758e9ef-de23-6e38-0468-f7931f6b8f05: 2024-01-09T05:09:53.424Z [INFO] server-b758e9ef-de23-6e38-0468-f7931f6b8f05: entering follower state: follower="Node at b758e9ef-de23-6e38-0468-f7931f6b8f05 [Follower]" leader= testing.go:149: server-b758e9ef-de23-6e38-0468-f7931f6b8f05: 2024-01-09T05:09:53.424Z [INFO] server-b758e9ef-de23-6e38-0468-f7931f6b8f05: entering follower state: follower="Node at b758e9ef-de23-6e38-0468-f7931f6b8f05 [Follower]" leader= testLoggerAdapter verbose: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.482Z [WARN] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.482Z [WARN] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.482Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: entering candidate state: node="Node at 3cfec63e-0c1f-b6f9-9a35-287122a41b24 [Candidate]" term=2 testing.go:149: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.482Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: entering candidate state: node="Node at 3cfec63e-0c1f-b6f9-9a35-287122a41b24 [Candidate]" term=2 testLoggerAdapter verbose: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.482Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: election won: tally=2 testing.go:149: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.482Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: election won: tally=2 testLoggerAdapter verbose: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.482Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: entering leader state: leader="Node at 3cfec63e-0c1f-b6f9-9a35-287122a41b24 [Leader]" testing.go:149: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.482Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: entering leader state: leader="Node at 3cfec63e-0c1f-b6f9-9a35-287122a41b24 [Leader]" testLoggerAdapter verbose: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.482Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: added peer, starting replication: peer=server-725cb9e4-97f5-9799-ba23-5965e058cf3b testing.go:149: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.482Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: added peer, starting replication: peer=server-725cb9e4-97f5-9799-ba23-5965e058cf3b testLoggerAdapter verbose: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.483Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: pipelining replication: peer="{Voter server-725cb9e4-97f5-9799-ba23-5965e058cf3b 725cb9e4-97f5-9799-ba23-5965e058cf3b}" testing.go:149: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.483Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: pipelining replication: peer="{Voter server-725cb9e4-97f5-9799-ba23-5965e058cf3b 725cb9e4-97f5-9799-ba23-5965e058cf3b}" testLoggerAdapter verbose: server-b758e9ef-de23-6e38-0468-f7931f6b8f05: 2024-01-09T05:09:53.489Z [WARN] server-b758e9ef-de23-6e38-0468-f7931f6b8f05: no known peers, aborting election testing.go:149: server-b758e9ef-de23-6e38-0468-f7931f6b8f05: 2024-01-09T05:09:53.489Z [WARN] server-b758e9ef-de23-6e38-0468-f7931f6b8f05: no known peers, aborting election testLoggerAdapter verbose: cluster: 2024-01-09T05:09:53.589Z [INFO] stable state for Leader reached at 2024-01-09 05:09:53.483301515 +0000 UTC m=+4.736352615 (1 nodes), 59.316167ms from start of poll, 78.12126ms from cluster start. Timeout at 2024-01-09 05:09:53.589315059 +0000 UTC m=+4.842366230, 106.013615ms after stability testing.go:149: cluster: 2024-01-09T05:09:53.589Z [INFO] stable state for Leader reached at 2024-01-09 05:09:53.483301515 +0000 UTC m=+4.736352615 (1 nodes), 59.316167ms from start of poll, 78.12126ms from cluster start. Timeout at 2024-01-09 05:09:53.589315059 +0000 UTC m=+4.842366230, 106.013615ms after stability testLoggerAdapter verbose: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.589Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: updating configuration: command=AddStaging server-id=server-b758e9ef-de23-6e38-0468-f7931f6b8f05 server-addr=b758e9ef-de23-6e38-0468-f7931f6b8f05 servers="[{Suffrage:Voter ID:server-725cb9e4-97f5-9799-ba23-5965e058cf3b Address:725cb9e4-97f5-9799-ba23-5965e058cf3b} {Suffrage:Voter ID:server-3cfec63e-0c1f-b6f9-9a35-287122a41b24 Address:3cfec63e-0c1f-b6f9-9a35-287122a41b24} {Suffrage:Voter ID:server-b758e9ef-de23-6e38-0468-f7931f6b8f05 Address:b758e9ef-de23-6e38-0468-f7931f6b8f05}]" testing.go:149: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.589Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: updating configuration: command=AddStaging server-id=server-b758e9ef-de23-6e38-0468-f7931f6b8f05 server-addr=b758e9ef-de23-6e38-0468-f7931f6b8f05 servers="[{Suffrage:Voter ID:server-725cb9e4-97f5-9799-ba23-5965e058cf3b Address:725cb9e4-97f5-9799-ba23-5965e058cf3b} {Suffrage:Voter ID:server-3cfec63e-0c1f-b6f9-9a35-287122a41b24 Address:3cfec63e-0c1f-b6f9-9a35-287122a41b24} {Suffrage:Voter ID:server-b758e9ef-de23-6e38-0468-f7931f6b8f05 Address:b758e9ef-de23-6e38-0468-f7931f6b8f05}]" testLoggerAdapter verbose: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.589Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: added peer, starting replication: peer=server-b758e9ef-de23-6e38-0468-f7931f6b8f05 testing.go:149: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.589Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: added peer, starting replication: peer=server-b758e9ef-de23-6e38-0468-f7931f6b8f05 testLoggerAdapter verbose: server-b758e9ef-de23-6e38-0468-f7931f6b8f05: 2024-01-09T05:09:53.589Z [WARN] server-b758e9ef-de23-6e38-0468-f7931f6b8f05: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: server-b758e9ef-de23-6e38-0468-f7931f6b8f05: 2024-01-09T05:09:53.589Z [WARN] server-b758e9ef-de23-6e38-0468-f7931f6b8f05: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.589Z [WARN] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: appendEntries rejected, sending older logs: peer="{Voter server-b758e9ef-de23-6e38-0468-f7931f6b8f05 b758e9ef-de23-6e38-0468-f7931f6b8f05}" next=1 testing.go:149: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.589Z [WARN] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: appendEntries rejected, sending older logs: peer="{Voter server-b758e9ef-de23-6e38-0468-f7931f6b8f05 b758e9ef-de23-6e38-0468-f7931f6b8f05}" next=1 testLoggerAdapter verbose: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.589Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: pipelining replication: peer="{Voter server-b758e9ef-de23-6e38-0468-f7931f6b8f05 b758e9ef-de23-6e38-0468-f7931f6b8f05}" testing.go:149: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.589Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: pipelining replication: peer="{Voter server-b758e9ef-de23-6e38-0468-f7931f6b8f05 b758e9ef-de23-6e38-0468-f7931f6b8f05}" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:53.590Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:09:53.590Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-01-09T05:09:53.695Z [INFO] stable state for Leader reached at 2024-01-09 05:09:53.590053259 +0000 UTC m=+4.843104338 (1 nodes), 2.388µs from start of poll, 184.872983ms from cluster start. Timeout at 2024-01-09 05:09:53.695062572 +0000 UTC m=+4.948113581, 105.009243ms after stability testing.go:149: cluster: 2024-01-09T05:09:53.695Z [INFO] stable state for Leader reached at 2024-01-09 05:09:53.590053259 +0000 UTC m=+4.843104338 (1 nodes), 2.388µs from start of poll, 184.872983ms from cluster start. Timeout at 2024-01-09 05:09:53.695062572 +0000 UTC m=+4.948113581, 105.009243ms after stability testLoggerAdapter verbose: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.695Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: aborting pipeline replication: peer="{Voter server-b758e9ef-de23-6e38-0468-f7931f6b8f05 b758e9ef-de23-6e38-0468-f7931f6b8f05}" testing.go:149: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.695Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: aborting pipeline replication: peer="{Voter server-b758e9ef-de23-6e38-0468-f7931f6b8f05 b758e9ef-de23-6e38-0468-f7931f6b8f05}" testLoggerAdapter verbose: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.695Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: aborting pipeline replication: peer="{Voter server-725cb9e4-97f5-9799-ba23-5965e058cf3b 725cb9e4-97f5-9799-ba23-5965e058cf3b}" testing.go:149: server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: 2024-01-09T05:09:53.695Z [INFO] server-3cfec63e-0c1f-b6f9-9a35-287122a41b24: aborting pipeline replication: peer="{Voter server-725cb9e4-97f5-9799-ba23-5965e058cf3b 725cb9e4-97f5-9799-ba23-5965e058cf3b}" --- PASS: TestRaft_HasExistingState (0.29s) === RUN TestRaft_SingleNode testLoggerAdapter verbose: server-71bee22d-37bd-2ab1-9807-7add819f0e21: 2024-01-09T05:09:53.696Z [INFO] server-71bee22d-37bd-2ab1-9807-7add819f0e21: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-71bee22d-37bd-2ab1-9807-7add819f0e21 Address:71bee22d-37bd-2ab1-9807-7add819f0e21}]" testing.go:149: server-71bee22d-37bd-2ab1-9807-7add819f0e21: 2024-01-09T05:09:53.696Z [INFO] server-71bee22d-37bd-2ab1-9807-7add819f0e21: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-71bee22d-37bd-2ab1-9807-7add819f0e21 Address:71bee22d-37bd-2ab1-9807-7add819f0e21}]" testLoggerAdapter verbose: server-71bee22d-37bd-2ab1-9807-7add819f0e21: 2024-01-09T05:09:53.696Z [INFO] server-71bee22d-37bd-2ab1-9807-7add819f0e21: entering follower state: follower="Node at 71bee22d-37bd-2ab1-9807-7add819f0e21 [Follower]" leader= testing.go:149: server-71bee22d-37bd-2ab1-9807-7add819f0e21: 2024-01-09T05:09:53.696Z [INFO] server-71bee22d-37bd-2ab1-9807-7add819f0e21: entering follower state: follower="Node at 71bee22d-37bd-2ab1-9807-7add819f0e21 [Follower]" leader= testLoggerAdapter verbose: server-71bee22d-37bd-2ab1-9807-7add819f0e21: 2024-01-09T05:09:53.757Z [WARN] server-71bee22d-37bd-2ab1-9807-7add819f0e21: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-71bee22d-37bd-2ab1-9807-7add819f0e21: 2024-01-09T05:09:53.757Z [WARN] server-71bee22d-37bd-2ab1-9807-7add819f0e21: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-71bee22d-37bd-2ab1-9807-7add819f0e21: 2024-01-09T05:09:53.757Z [INFO] server-71bee22d-37bd-2ab1-9807-7add819f0e21: entering candidate state: node="Node at 71bee22d-37bd-2ab1-9807-7add819f0e21 [Candidate]" term=2 testing.go:149: server-71bee22d-37bd-2ab1-9807-7add819f0e21: 2024-01-09T05:09:53.757Z [INFO] server-71bee22d-37bd-2ab1-9807-7add819f0e21: entering candidate state: node="Node at 71bee22d-37bd-2ab1-9807-7add819f0e21 [Candidate]" term=2 testLoggerAdapter verbose: server-71bee22d-37bd-2ab1-9807-7add819f0e21: 2024-01-09T05:09:53.757Z [INFO] server-71bee22d-37bd-2ab1-9807-7add819f0e21: election won: tally=1 testing.go:149: server-71bee22d-37bd-2ab1-9807-7add819f0e21: 2024-01-09T05:09:53.757Z [INFO] server-71bee22d-37bd-2ab1-9807-7add819f0e21: election won: tally=1 testLoggerAdapter verbose: server-71bee22d-37bd-2ab1-9807-7add819f0e21: 2024-01-09T05:09:53.757Z [INFO] server-71bee22d-37bd-2ab1-9807-7add819f0e21: entering leader state: leader="Node at 71bee22d-37bd-2ab1-9807-7add819f0e21 [Leader]" testing.go:149: server-71bee22d-37bd-2ab1-9807-7add819f0e21: 2024-01-09T05:09:53.757Z [INFO] server-71bee22d-37bd-2ab1-9807-7add819f0e21: entering leader state: leader="Node at 71bee22d-37bd-2ab1-9807-7add819f0e21 [Leader]" --- PASS: TestRaft_SingleNode (0.06s) === RUN TestRaft_TripleNode testLoggerAdapter verbose: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.758Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-02569c5a-a550-e67a-f071-c8cfcd65d351 Address:02569c5a-a550-e67a-f071-c8cfcd65d351} {Suffrage:Voter ID:server-b7d18231-0a9e-a900-556c-207650145640 Address:b7d18231-0a9e-a900-556c-207650145640} {Suffrage:Voter ID:server-52e1b013-6bf0-0a59-529f-985d3543858f Address:52e1b013-6bf0-0a59-529f-985d3543858f}]" testing.go:149: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.758Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-02569c5a-a550-e67a-f071-c8cfcd65d351 Address:02569c5a-a550-e67a-f071-c8cfcd65d351} {Suffrage:Voter ID:server-b7d18231-0a9e-a900-556c-207650145640 Address:b7d18231-0a9e-a900-556c-207650145640} {Suffrage:Voter ID:server-52e1b013-6bf0-0a59-529f-985d3543858f Address:52e1b013-6bf0-0a59-529f-985d3543858f}]" testLoggerAdapter verbose: server-b7d18231-0a9e-a900-556c-207650145640: 2024-01-09T05:09:53.758Z [INFO] server-b7d18231-0a9e-a900-556c-207650145640: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-02569c5a-a550-e67a-f071-c8cfcd65d351 Address:02569c5a-a550-e67a-f071-c8cfcd65d351} {Suffrage:Voter ID:server-b7d18231-0a9e-a900-556c-207650145640 Address:b7d18231-0a9e-a900-556c-207650145640} {Suffrage:Voter ID:server-52e1b013-6bf0-0a59-529f-985d3543858f Address:52e1b013-6bf0-0a59-529f-985d3543858f}]" testing.go:149: server-b7d18231-0a9e-a900-556c-207650145640: 2024-01-09T05:09:53.758Z [INFO] server-b7d18231-0a9e-a900-556c-207650145640: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-02569c5a-a550-e67a-f071-c8cfcd65d351 Address:02569c5a-a550-e67a-f071-c8cfcd65d351} {Suffrage:Voter ID:server-b7d18231-0a9e-a900-556c-207650145640 Address:b7d18231-0a9e-a900-556c-207650145640} {Suffrage:Voter ID:server-52e1b013-6bf0-0a59-529f-985d3543858f Address:52e1b013-6bf0-0a59-529f-985d3543858f}]" testLoggerAdapter verbose: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.758Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: entering follower state: follower="Node at 02569c5a-a550-e67a-f071-c8cfcd65d351 [Follower]" leader= testing.go:149: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.758Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: entering follower state: follower="Node at 02569c5a-a550-e67a-f071-c8cfcd65d351 [Follower]" leader= testLoggerAdapter verbose: server-b7d18231-0a9e-a900-556c-207650145640: 2024-01-09T05:09:53.759Z [INFO] server-b7d18231-0a9e-a900-556c-207650145640: entering follower state: follower="Node at b7d18231-0a9e-a900-556c-207650145640 [Follower]" leader= testing.go:149: server-b7d18231-0a9e-a900-556c-207650145640: 2024-01-09T05:09:53.759Z [INFO] server-b7d18231-0a9e-a900-556c-207650145640: entering follower state: follower="Node at b7d18231-0a9e-a900-556c-207650145640 [Follower]" leader= testLoggerAdapter verbose: server-52e1b013-6bf0-0a59-529f-985d3543858f: 2024-01-09T05:09:53.759Z [INFO] server-52e1b013-6bf0-0a59-529f-985d3543858f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-02569c5a-a550-e67a-f071-c8cfcd65d351 Address:02569c5a-a550-e67a-f071-c8cfcd65d351} {Suffrage:Voter ID:server-b7d18231-0a9e-a900-556c-207650145640 Address:b7d18231-0a9e-a900-556c-207650145640} {Suffrage:Voter ID:server-52e1b013-6bf0-0a59-529f-985d3543858f Address:52e1b013-6bf0-0a59-529f-985d3543858f}]" testing.go:149: server-52e1b013-6bf0-0a59-529f-985d3543858f: 2024-01-09T05:09:53.759Z [INFO] server-52e1b013-6bf0-0a59-529f-985d3543858f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-02569c5a-a550-e67a-f071-c8cfcd65d351 Address:02569c5a-a550-e67a-f071-c8cfcd65d351} {Suffrage:Voter ID:server-b7d18231-0a9e-a900-556c-207650145640 Address:b7d18231-0a9e-a900-556c-207650145640} {Suffrage:Voter ID:server-52e1b013-6bf0-0a59-529f-985d3543858f Address:52e1b013-6bf0-0a59-529f-985d3543858f}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:53.759Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-01-09T05:09:53.759Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-52e1b013-6bf0-0a59-529f-985d3543858f: 2024-01-09T05:09:53.759Z [INFO] server-52e1b013-6bf0-0a59-529f-985d3543858f: entering follower state: follower="Node at 52e1b013-6bf0-0a59-529f-985d3543858f [Follower]" leader= testing.go:149: server-52e1b013-6bf0-0a59-529f-985d3543858f: 2024-01-09T05:09:53.759Z [INFO] server-52e1b013-6bf0-0a59-529f-985d3543858f: entering follower state: follower="Node at 52e1b013-6bf0-0a59-529f-985d3543858f [Follower]" leader= testLoggerAdapter verbose: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.812Z [WARN] server-02569c5a-a550-e67a-f071-c8cfcd65d351: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.812Z [WARN] server-02569c5a-a550-e67a-f071-c8cfcd65d351: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.812Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: entering candidate state: node="Node at 02569c5a-a550-e67a-f071-c8cfcd65d351 [Candidate]" term=2 testing.go:149: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.812Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: entering candidate state: node="Node at 02569c5a-a550-e67a-f071-c8cfcd65d351 [Candidate]" term=2 testLoggerAdapter verbose: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.813Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: election won: tally=2 testing.go:149: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.813Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: election won: tally=2 testLoggerAdapter verbose: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.813Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: entering leader state: leader="Node at 02569c5a-a550-e67a-f071-c8cfcd65d351 [Leader]" testing.go:149: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.813Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: entering leader state: leader="Node at 02569c5a-a550-e67a-f071-c8cfcd65d351 [Leader]" testLoggerAdapter verbose: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.813Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: added peer, starting replication: peer=server-b7d18231-0a9e-a900-556c-207650145640 testing.go:149: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.813Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: added peer, starting replication: peer=server-b7d18231-0a9e-a900-556c-207650145640 testLoggerAdapter verbose: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.813Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: added peer, starting replication: peer=server-52e1b013-6bf0-0a59-529f-985d3543858f testing.go:149: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.813Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: added peer, starting replication: peer=server-52e1b013-6bf0-0a59-529f-985d3543858f testLoggerAdapter verbose: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.813Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: pipelining replication: peer="{Voter server-b7d18231-0a9e-a900-556c-207650145640 b7d18231-0a9e-a900-556c-207650145640}" testing.go:149: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.813Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: pipelining replication: peer="{Voter server-b7d18231-0a9e-a900-556c-207650145640 b7d18231-0a9e-a900-556c-207650145640}" testLoggerAdapter verbose: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.814Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: pipelining replication: peer="{Voter server-52e1b013-6bf0-0a59-529f-985d3543858f 52e1b013-6bf0-0a59-529f-985d3543858f}" testing.go:149: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:53.814Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: pipelining replication: peer="{Voter server-52e1b013-6bf0-0a59-529f-985d3543858f 52e1b013-6bf0-0a59-529f-985d3543858f}" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:53.918Z [INFO] stable state for Follower reached at 2024-01-09 05:09:53.813778294 +0000 UTC m=+5.066829460 (2 nodes), 54.536048ms from start of poll, 55.13129ms from cluster start. Timeout at 2024-01-09 05:09:53.918842028 +0000 UTC m=+5.171893074, 105.063614ms after stability testing.go:149: cluster: 2024-01-09T05:09:53.918Z [INFO] stable state for Follower reached at 2024-01-09 05:09:53.813778294 +0000 UTC m=+5.066829460 (2 nodes), 54.536048ms from start of poll, 55.13129ms from cluster start. Timeout at 2024-01-09 05:09:53.918842028 +0000 UTC m=+5.171893074, 105.063614ms after stability testLoggerAdapter verbose: cluster: 2024-01-09T05:09:53.918Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:09:53.918Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-01-09T05:09:54.024Z [INFO] stable state for Leader reached at 2024-01-09 05:09:53.919018369 +0000 UTC m=+5.172069369 (1 nodes), 2.021µs from start of poll, 160.371199ms from cluster start. Timeout at 2024-01-09 05:09:54.02411078 +0000 UTC m=+5.277161843, 105.092474ms after stability testing.go:149: cluster: 2024-01-09T05:09:54.024Z [INFO] stable state for Leader reached at 2024-01-09 05:09:53.919018369 +0000 UTC m=+5.172069369 (1 nodes), 2.021µs from start of poll, 160.371199ms from cluster start. Timeout at 2024-01-09 05:09:54.02411078 +0000 UTC m=+5.277161843, 105.092474ms after stability testLoggerAdapter verbose: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:54.037Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: aborting pipeline replication: peer="{Voter server-52e1b013-6bf0-0a59-529f-985d3543858f 52e1b013-6bf0-0a59-529f-985d3543858f}" testing.go:149: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:54.037Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: aborting pipeline replication: peer="{Voter server-52e1b013-6bf0-0a59-529f-985d3543858f 52e1b013-6bf0-0a59-529f-985d3543858f}" testLoggerAdapter verbose: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:54.037Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: aborting pipeline replication: peer="{Voter server-b7d18231-0a9e-a900-556c-207650145640 b7d18231-0a9e-a900-556c-207650145640}" testing.go:149: server-02569c5a-a550-e67a-f071-c8cfcd65d351: 2024-01-09T05:09:54.037Z [INFO] server-02569c5a-a550-e67a-f071-c8cfcd65d351: aborting pipeline replication: peer="{Voter server-b7d18231-0a9e-a900-556c-207650145640 b7d18231-0a9e-a900-556c-207650145640}" --- PASS: TestRaft_TripleNode (0.28s) === RUN TestRaft_LeaderFail testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.039Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d6d0910-06fa-854f-5c8a-d43091411912 Address:4d6d0910-06fa-854f-5c8a-d43091411912} {Suffrage:Voter ID:server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea Address:a20fdaeb-68ed-be57-a26d-07cd322fc2ea} {Suffrage:Voter ID:server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 Address:f251bea8-a509-cbe0-decd-8c0e1d364fa2}]" testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.039Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d6d0910-06fa-854f-5c8a-d43091411912 Address:4d6d0910-06fa-854f-5c8a-d43091411912} {Suffrage:Voter ID:server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea Address:a20fdaeb-68ed-be57-a26d-07cd322fc2ea} {Suffrage:Voter ID:server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 Address:f251bea8-a509-cbe0-decd-8c0e1d364fa2}]" testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.039Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: entering follower state: follower="Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Follower]" leader= testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.039Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: entering follower state: follower="Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Follower]" leader= testLoggerAdapter verbose: server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: 2024-01-09T05:09:54.039Z [INFO] server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d6d0910-06fa-854f-5c8a-d43091411912 Address:4d6d0910-06fa-854f-5c8a-d43091411912} {Suffrage:Voter ID:server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea Address:a20fdaeb-68ed-be57-a26d-07cd322fc2ea} {Suffrage:Voter ID:server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 Address:f251bea8-a509-cbe0-decd-8c0e1d364fa2}]" testing.go:149: server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: 2024-01-09T05:09:54.039Z [INFO] server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d6d0910-06fa-854f-5c8a-d43091411912 Address:4d6d0910-06fa-854f-5c8a-d43091411912} {Suffrage:Voter ID:server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea Address:a20fdaeb-68ed-be57-a26d-07cd322fc2ea} {Suffrage:Voter ID:server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 Address:f251bea8-a509-cbe0-decd-8c0e1d364fa2}]" testLoggerAdapter verbose: server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: 2024-01-09T05:09:54.039Z [INFO] server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: entering follower state: follower="Node at a20fdaeb-68ed-be57-a26d-07cd322fc2ea [Follower]" leader= testing.go:149: server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: 2024-01-09T05:09:54.039Z [INFO] server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: entering follower state: follower="Node at a20fdaeb-68ed-be57-a26d-07cd322fc2ea [Follower]" leader= testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.039Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d6d0910-06fa-854f-5c8a-d43091411912 Address:4d6d0910-06fa-854f-5c8a-d43091411912} {Suffrage:Voter ID:server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea Address:a20fdaeb-68ed-be57-a26d-07cd322fc2ea} {Suffrage:Voter ID:server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 Address:f251bea8-a509-cbe0-decd-8c0e1d364fa2}]" testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.039Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d6d0910-06fa-854f-5c8a-d43091411912 Address:4d6d0910-06fa-854f-5c8a-d43091411912} {Suffrage:Voter ID:server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea Address:a20fdaeb-68ed-be57-a26d-07cd322fc2ea} {Suffrage:Voter ID:server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 Address:f251bea8-a509-cbe0-decd-8c0e1d364fa2}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:54.039Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-01-09T05:09:54.039Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.039Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: entering follower state: follower="Node at f251bea8-a509-cbe0-decd-8c0e1d364fa2 [Follower]" leader= testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.039Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: entering follower state: follower="Node at f251bea8-a509-cbe0-decd-8c0e1d364fa2 [Follower]" leader= testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.094Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.094Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.094Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: entering candidate state: node="Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Candidate]" term=2 testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.094Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: entering candidate state: node="Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Candidate]" term=2 testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.094Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: election won: tally=2 testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.094Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: election won: tally=2 testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.094Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: entering leader state: leader="Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Leader]" testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.094Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: entering leader state: leader="Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Leader]" testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.094Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: added peer, starting replication: peer=server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.094Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: added peer, starting replication: peer=server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.095Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: added peer, starting replication: peer=server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.095Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: added peer, starting replication: peer=server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.095Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: pipelining replication: peer="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.095Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: pipelining replication: peer="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.095Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: pipelining replication: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.095Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: pipelining replication: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:54.206Z [INFO] stable state for Follower reached at 2024-01-09 05:09:54.095399776 +0000 UTC m=+5.348451128 (2 nodes), 55.804982ms from start of poll, 56.360248ms from cluster start. Timeout at 2024-01-09 05:09:54.206333162 +0000 UTC m=+5.459384225, 110.933097ms after stability testing.go:149: cluster: 2024-01-09T05:09:54.206Z [INFO] stable state for Follower reached at 2024-01-09 05:09:54.095399776 +0000 UTC m=+5.348451128 (2 nodes), 55.804982ms from start of poll, 56.360248ms from cluster start. Timeout at 2024-01-09 05:09:54.206333162 +0000 UTC m=+5.459384225, 110.933097ms after stability testLoggerAdapter verbose: cluster: 2024-01-09T05:09:54.206Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:09:54.206Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-01-09T05:09:54.311Z [INFO] stable state for Leader reached at 2024-01-09 05:09:54.206507028 +0000 UTC m=+5.459558208 (1 nodes), 2.879µs from start of poll, 167.467328ms from cluster start. Timeout at 2024-01-09 05:09:54.311641412 +0000 UTC m=+5.564692589, 105.134381ms after stability testing.go:149: cluster: 2024-01-09T05:09:54.311Z [INFO] stable state for Leader reached at 2024-01-09 05:09:54.206507028 +0000 UTC m=+5.459558208 (1 nodes), 2.879µs from start of poll, 167.467328ms from cluster start. Timeout at 2024-01-09 05:09:54.311641412 +0000 UTC m=+5.564692589, 105.134381ms after stability raft_test.go:336: [INFO] Disconnecting Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Leader] testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.329Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to heartbeat to: peer=a20fdaeb-68ed-be57-a26d-07cd322fc2ea error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.329Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to heartbeat to: peer=a20fdaeb-68ed-be57-a26d-07cd322fc2ea error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.331Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to pipeline appendEntries: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" error="append pipeline closed" testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.331Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to pipeline appendEntries: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" error="append pipeline closed" testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.331Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: aborting pipeline replication: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.331Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: aborting pipeline replication: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.331Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to heartbeat to: peer=f251bea8-a509-cbe0-decd-8c0e1d364fa2 error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.331Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to heartbeat to: peer=f251bea8-a509-cbe0-decd-8c0e1d364fa2 error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.332Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to pipeline appendEntries: peer="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" error="append pipeline closed" testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.332Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to pipeline appendEntries: peer="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" error="append pipeline closed" testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.332Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: aborting pipeline replication: peer="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.332Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: aborting pipeline replication: peer="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:54.333Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:09:54.333Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.341Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to appendEntries to: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.341Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to appendEntries to: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.341Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to appendEntries to: peer="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.341Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to appendEntries to: peer="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.347Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to heartbeat to: peer=f251bea8-a509-cbe0-decd-8c0e1d364fa2 error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.347Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to heartbeat to: peer=f251bea8-a509-cbe0-decd-8c0e1d364fa2 error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.348Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to heartbeat to: peer=a20fdaeb-68ed-be57-a26d-07cd322fc2ea error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.348Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to heartbeat to: peer=a20fdaeb-68ed-be57-a26d-07cd322fc2ea error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.357Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to appendEntries to: peer="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.357Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to appendEntries to: peer="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.358Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to appendEntries to: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.358Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to appendEntries to: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.363Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to heartbeat to: peer=f251bea8-a509-cbe0-decd-8c0e1d364fa2 error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.363Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to heartbeat to: peer=f251bea8-a509-cbe0-decd-8c0e1d364fa2 error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.368Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to heartbeat to: peer=a20fdaeb-68ed-be57-a26d-07cd322fc2ea error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.368Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to heartbeat to: peer=a20fdaeb-68ed-be57-a26d-07cd322fc2ea error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.373Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to contact: server-id=server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea time=50.064143ms testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.373Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to contact: server-id=server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea time=50.064143ms testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.376Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to appendEntries to: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.376Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to appendEntries to: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.377Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to appendEntries to: peer="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.377Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to appendEntries to: peer="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.383Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to contact: server-id=server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea time=60.37515ms testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.383Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to contact: server-id=server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea time=60.37515ms testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.383Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to contact: server-id=server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 time=59.344272ms testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.383Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to contact: server-id=server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 time=59.344272ms testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.384Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to contact quorum of nodes, stepping down testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.384Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to contact quorum of nodes, stepping down testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.384Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: entering follower state: follower="Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Follower]" leader= testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.384Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: entering follower state: follower="Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Follower]" leader= testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.392Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to heartbeat to: peer=f251bea8-a509-cbe0-decd-8c0e1d364fa2 error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.392Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to heartbeat to: peer=f251bea8-a509-cbe0-decd-8c0e1d364fa2 error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" testLoggerAdapter verbose: server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: 2024-01-09T05:09:54.396Z [WARN] server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: heartbeat timeout reached, starting election: last-leader=4d6d0910-06fa-854f-5c8a-d43091411912 testing.go:149: server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: 2024-01-09T05:09:54.396Z [WARN] server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: heartbeat timeout reached, starting election: last-leader=4d6d0910-06fa-854f-5c8a-d43091411912 testLoggerAdapter verbose: server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: 2024-01-09T05:09:54.396Z [INFO] server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: entering candidate state: node="Node at a20fdaeb-68ed-be57-a26d-07cd322fc2ea [Candidate]" term=3 testing.go:149: server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: 2024-01-09T05:09:54.396Z [INFO] server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: entering candidate state: node="Node at a20fdaeb-68ed-be57-a26d-07cd322fc2ea [Candidate]" term=3 testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.396Z [WARN] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: rejecting vote request since we have a leader: from=a20fdaeb-68ed-be57-a26d-07cd322fc2ea leader=4d6d0910-06fa-854f-5c8a-d43091411912 testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.396Z [WARN] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: rejecting vote request since we have a leader: from=a20fdaeb-68ed-be57-a26d-07cd322fc2ea leader=4d6d0910-06fa-854f-5c8a-d43091411912 testLoggerAdapter verbose: server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: 2024-01-09T05:09:54.397Z [ERROR] server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: failed to make requestVote RPC: target="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" testing.go:149: server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: 2024-01-09T05:09:54.397Z [ERROR] server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: failed to make requestVote RPC: target="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.398Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to heartbeat to: peer=a20fdaeb-68ed-be57-a26d-07cd322fc2ea error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.398Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to heartbeat to: peer=a20fdaeb-68ed-be57-a26d-07cd322fc2ea error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.403Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to appendEntries to: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.403Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to appendEntries to: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.404Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to appendEntries to: peer="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.404Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to appendEntries to: peer="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.409Z [WARN] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: heartbeat timeout reached, starting election: last-leader=4d6d0910-06fa-854f-5c8a-d43091411912 testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.409Z [WARN] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: heartbeat timeout reached, starting election: last-leader=4d6d0910-06fa-854f-5c8a-d43091411912 testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.409Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: entering candidate state: node="Node at f251bea8-a509-cbe0-decd-8c0e1d364fa2 [Candidate]" term=3 testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.409Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: entering candidate state: node="Node at f251bea8-a509-cbe0-decd-8c0e1d364fa2 [Candidate]" term=3 testLoggerAdapter verbose: server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: 2024-01-09T05:09:54.409Z [INFO] server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: duplicate requestVote for same term: term=3 testing.go:149: server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: 2024-01-09T05:09:54.409Z [INFO] server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: duplicate requestVote for same term: term=3 testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.409Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to make requestVote RPC: target="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.409Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to make requestVote RPC: target="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.452Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.452Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.452Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: entering candidate state: node="Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Candidate]" term=3 testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.452Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: entering candidate state: node="Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Candidate]" term=3 testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.452Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to make requestVote RPC: target="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.452Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to make requestVote RPC: target="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.452Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to make requestVote RPC: target="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.452Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to make requestVote RPC: target="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.473Z [WARN] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: Election timeout reached, restarting election testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.473Z [WARN] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: Election timeout reached, restarting election testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.473Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: entering candidate state: node="Node at f251bea8-a509-cbe0-decd-8c0e1d364fa2 [Candidate]" term=4 testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.473Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: entering candidate state: node="Node at f251bea8-a509-cbe0-decd-8c0e1d364fa2 [Candidate]" term=4 testLoggerAdapter verbose: server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: 2024-01-09T05:09:54.473Z [INFO] server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: entering follower state: follower="Node at a20fdaeb-68ed-be57-a26d-07cd322fc2ea [Follower]" leader= testing.go:149: server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: 2024-01-09T05:09:54.473Z [INFO] server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: entering follower state: follower="Node at a20fdaeb-68ed-be57-a26d-07cd322fc2ea [Follower]" leader= testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.473Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: election won: tally=2 testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.473Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: election won: tally=2 testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.473Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: entering leader state: leader="Node at f251bea8-a509-cbe0-decd-8c0e1d364fa2 [Leader]" testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.473Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: entering leader state: leader="Node at f251bea8-a509-cbe0-decd-8c0e1d364fa2 [Leader]" testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.473Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: added peer, starting replication: peer=server-4d6d0910-06fa-854f-5c8a-d43091411912 testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.473Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: added peer, starting replication: peer=server-4d6d0910-06fa-854f-5c8a-d43091411912 testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.473Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: added peer, starting replication: peer=server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.473Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: added peer, starting replication: peer=server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.473Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: pipelining replication: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.473Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: pipelining replication: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.473Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to make requestVote RPC: target="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.473Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to make requestVote RPC: target="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.473Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to appendEntries to: peer="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.473Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to appendEntries to: peer="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.482Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to heartbeat to: peer=4d6d0910-06fa-854f-5c8a-d43091411912 error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.482Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to heartbeat to: peer=4d6d0910-06fa-854f-5c8a-d43091411912 error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.489Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to appendEntries to: peer="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.489Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to appendEntries to: peer="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.502Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to heartbeat to: peer=4d6d0910-06fa-854f-5c8a-d43091411912 error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.502Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to heartbeat to: peer=4d6d0910-06fa-854f-5c8a-d43091411912 error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.507Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to appendEntries to: peer="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.507Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to appendEntries to: peer="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.520Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to heartbeat to: peer=4d6d0910-06fa-854f-5c8a-d43091411912 error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.520Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to heartbeat to: peer=4d6d0910-06fa-854f-5c8a-d43091411912 error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.521Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: Election timeout reached, restarting election testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.521Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: Election timeout reached, restarting election testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.521Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: entering candidate state: node="Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Candidate]" term=4 testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.521Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: entering candidate state: node="Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Candidate]" term=4 testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.521Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to make requestVote RPC: target="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.521Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to make requestVote RPC: target="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.521Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to make requestVote RPC: target="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.521Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to make requestVote RPC: target="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.523Z [WARN] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to contact: server-id=server-4d6d0910-06fa-854f-5c8a-d43091411912 time=50.175494ms testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.523Z [WARN] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to contact: server-id=server-4d6d0910-06fa-854f-5c8a-d43091411912 time=50.175494ms testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.533Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to appendEntries to: peer="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.533Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to appendEntries to: peer="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.550Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to heartbeat to: peer=4d6d0910-06fa-854f-5c8a-d43091411912 error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.550Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to heartbeat to: peer=4d6d0910-06fa-854f-5c8a-d43091411912 error="failed to connect to peer: 4d6d0910-06fa-854f-5c8a-d43091411912" testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.571Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: Election timeout reached, restarting election testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.571Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: Election timeout reached, restarting election testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.571Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: entering candidate state: node="Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Candidate]" term=5 testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.571Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: entering candidate state: node="Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Candidate]" term=5 testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.571Z [WARN] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to contact: server-id=server-4d6d0910-06fa-854f-5c8a-d43091411912 time=98.233819ms testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.571Z [WARN] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: failed to contact: server-id=server-4d6d0910-06fa-854f-5c8a-d43091411912 time=98.233819ms testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.571Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to make requestVote RPC: target="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.571Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to make requestVote RPC: target="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" error="failed to connect to peer: a20fdaeb-68ed-be57-a26d-07cd322fc2ea" testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.572Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to make requestVote RPC: target="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.572Z [ERROR] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to make requestVote RPC: target="{Voter server-f251bea8-a509-cbe0-decd-8c0e1d364fa2 f251bea8-a509-cbe0-decd-8c0e1d364fa2}" error="failed to connect to peer: f251bea8-a509-cbe0-decd-8c0e1d364fa2" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:54.578Z [INFO] stable state for Leader reached at 2024-01-09 05:09:54.473660943 +0000 UTC m=+5.726712308 (1 nodes), 140.223516ms from start of poll, 434.621428ms from cluster start. Timeout at 2024-01-09 05:09:54.578682913 +0000 UTC m=+5.831733967, 105.021659ms after stability testing.go:149: cluster: 2024-01-09T05:09:54.578Z [INFO] stable state for Leader reached at 2024-01-09 05:09:54.473660943 +0000 UTC m=+5.726712308 (1 nodes), 140.223516ms from start of poll, 434.621428ms from cluster start. Timeout at 2024-01-09 05:09:54.578682913 +0000 UTC m=+5.831733967, 105.021659ms after stability raft_test.go:371: [INFO] Reconnecting Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Candidate] testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.580Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: peer has newer term, stopping replication: peer="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.580Z [ERROR] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: peer has newer term, stopping replication: peer="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.580Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: entering follower state: follower="Node at f251bea8-a509-cbe0-decd-8c0e1d364fa2 [Follower]" leader= testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.580Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: entering follower state: follower="Node at f251bea8-a509-cbe0-decd-8c0e1d364fa2 [Follower]" leader= testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.580Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: aborting pipeline replication: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.580Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: aborting pipeline replication: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.621Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: Election timeout reached, restarting election testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.621Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: Election timeout reached, restarting election testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.621Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: entering candidate state: node="Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Candidate]" term=6 testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.621Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: entering candidate state: node="Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Candidate]" term=6 testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.622Z [WARN] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: rejecting vote request since our last term is greater: candidate=4d6d0910-06fa-854f-5c8a-d43091411912 last-term=4 last-candidate-term=2 testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.622Z [WARN] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: rejecting vote request since our last term is greater: candidate=4d6d0910-06fa-854f-5c8a-d43091411912 last-term=4 last-candidate-term=2 testLoggerAdapter verbose: server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: 2024-01-09T05:09:54.622Z [WARN] server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: rejecting vote request since we have a leader: from=4d6d0910-06fa-854f-5c8a-d43091411912 leader=f251bea8-a509-cbe0-decd-8c0e1d364fa2 testing.go:149: server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: 2024-01-09T05:09:54.622Z [WARN] server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea: rejecting vote request since we have a leader: from=4d6d0910-06fa-854f-5c8a-d43091411912 leader=f251bea8-a509-cbe0-decd-8c0e1d364fa2 testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.642Z [WARN] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.642Z [WARN] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.642Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: entering candidate state: node="Node at f251bea8-a509-cbe0-decd-8c0e1d364fa2 [Candidate]" term=7 testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.642Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: entering candidate state: node="Node at f251bea8-a509-cbe0-decd-8c0e1d364fa2 [Candidate]" term=7 testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.642Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: election won: tally=2 testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.642Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: election won: tally=2 testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.642Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: entering leader state: leader="Node at f251bea8-a509-cbe0-decd-8c0e1d364fa2 [Leader]" testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.642Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: entering leader state: leader="Node at f251bea8-a509-cbe0-decd-8c0e1d364fa2 [Leader]" testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.642Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: added peer, starting replication: peer=server-4d6d0910-06fa-854f-5c8a-d43091411912 testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.642Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: added peer, starting replication: peer=server-4d6d0910-06fa-854f-5c8a-d43091411912 testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.642Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: added peer, starting replication: peer=server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.642Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: added peer, starting replication: peer=server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.643Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: pipelining replication: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.643Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: pipelining replication: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.643Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: entering follower state: follower="Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Follower]" leader= testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.643Z [INFO] server-4d6d0910-06fa-854f-5c8a-d43091411912: entering follower state: follower="Node at 4d6d0910-06fa-854f-5c8a-d43091411912 [Follower]" leader= testLoggerAdapter verbose: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.643Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to get previous log: previous-index=5 last-index=3 error="log not found" testing.go:149: server-4d6d0910-06fa-854f-5c8a-d43091411912: 2024-01-09T05:09:54.643Z [WARN] server-4d6d0910-06fa-854f-5c8a-d43091411912: failed to get previous log: previous-index=5 last-index=3 error="log not found" testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.643Z [WARN] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: appendEntries rejected, sending older logs: peer="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" next=4 testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.643Z [WARN] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: appendEntries rejected, sending older logs: peer="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" next=4 testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.643Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: pipelining replication: peer="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.643Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: pipelining replication: peer="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.648Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: aborting pipeline replication: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.648Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: aborting pipeline replication: peer="{Voter server-a20fdaeb-68ed-be57-a26d-07cd322fc2ea a20fdaeb-68ed-be57-a26d-07cd322fc2ea}" testLoggerAdapter verbose: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.648Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: aborting pipeline replication: peer="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" testing.go:149: server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: 2024-01-09T05:09:54.648Z [INFO] server-f251bea8-a509-cbe0-decd-8c0e1d364fa2: aborting pipeline replication: peer="{Voter server-4d6d0910-06fa-854f-5c8a-d43091411912 4d6d0910-06fa-854f-5c8a-d43091411912}" --- PASS: TestRaft_LeaderFail (0.61s) === RUN TestRaft_BehindFollower testLoggerAdapter verbose: server-49c4da0a-d151-c41c-77a6-4eef96aaf71f: 2024-01-09T05:09:54.650Z [INFO] server-49c4da0a-d151-c41c-77a6-4eef96aaf71f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49c4da0a-d151-c41c-77a6-4eef96aaf71f Address:49c4da0a-d151-c41c-77a6-4eef96aaf71f} {Suffrage:Voter ID:server-27bbf3ad-bb6e-b691-d788-e28474238269 Address:27bbf3ad-bb6e-b691-d788-e28474238269} {Suffrage:Voter ID:server-63264def-4c66-c838-4ee0-f04dbd85cbd3 Address:63264def-4c66-c838-4ee0-f04dbd85cbd3}]" testing.go:149: server-49c4da0a-d151-c41c-77a6-4eef96aaf71f: 2024-01-09T05:09:54.650Z [INFO] server-49c4da0a-d151-c41c-77a6-4eef96aaf71f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49c4da0a-d151-c41c-77a6-4eef96aaf71f Address:49c4da0a-d151-c41c-77a6-4eef96aaf71f} {Suffrage:Voter ID:server-27bbf3ad-bb6e-b691-d788-e28474238269 Address:27bbf3ad-bb6e-b691-d788-e28474238269} {Suffrage:Voter ID:server-63264def-4c66-c838-4ee0-f04dbd85cbd3 Address:63264def-4c66-c838-4ee0-f04dbd85cbd3}]" testLoggerAdapter verbose: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.650Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49c4da0a-d151-c41c-77a6-4eef96aaf71f Address:49c4da0a-d151-c41c-77a6-4eef96aaf71f} {Suffrage:Voter ID:server-27bbf3ad-bb6e-b691-d788-e28474238269 Address:27bbf3ad-bb6e-b691-d788-e28474238269} {Suffrage:Voter ID:server-63264def-4c66-c838-4ee0-f04dbd85cbd3 Address:63264def-4c66-c838-4ee0-f04dbd85cbd3}]" testing.go:149: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.650Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49c4da0a-d151-c41c-77a6-4eef96aaf71f Address:49c4da0a-d151-c41c-77a6-4eef96aaf71f} {Suffrage:Voter ID:server-27bbf3ad-bb6e-b691-d788-e28474238269 Address:27bbf3ad-bb6e-b691-d788-e28474238269} {Suffrage:Voter ID:server-63264def-4c66-c838-4ee0-f04dbd85cbd3 Address:63264def-4c66-c838-4ee0-f04dbd85cbd3}]" testLoggerAdapter verbose: server-63264def-4c66-c838-4ee0-f04dbd85cbd3: 2024-01-09T05:09:54.650Z [INFO] server-63264def-4c66-c838-4ee0-f04dbd85cbd3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49c4da0a-d151-c41c-77a6-4eef96aaf71f Address:49c4da0a-d151-c41c-77a6-4eef96aaf71f} {Suffrage:Voter ID:server-27bbf3ad-bb6e-b691-d788-e28474238269 Address:27bbf3ad-bb6e-b691-d788-e28474238269} {Suffrage:Voter ID:server-63264def-4c66-c838-4ee0-f04dbd85cbd3 Address:63264def-4c66-c838-4ee0-f04dbd85cbd3}]" testing.go:149: server-63264def-4c66-c838-4ee0-f04dbd85cbd3: 2024-01-09T05:09:54.650Z [INFO] server-63264def-4c66-c838-4ee0-f04dbd85cbd3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49c4da0a-d151-c41c-77a6-4eef96aaf71f Address:49c4da0a-d151-c41c-77a6-4eef96aaf71f} {Suffrage:Voter ID:server-27bbf3ad-bb6e-b691-d788-e28474238269 Address:27bbf3ad-bb6e-b691-d788-e28474238269} {Suffrage:Voter ID:server-63264def-4c66-c838-4ee0-f04dbd85cbd3 Address:63264def-4c66-c838-4ee0-f04dbd85cbd3}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:54.650Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:09:54.650Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-49c4da0a-d151-c41c-77a6-4eef96aaf71f: 2024-01-09T05:09:54.650Z [INFO] server-49c4da0a-d151-c41c-77a6-4eef96aaf71f: entering follower state: follower="Node at 49c4da0a-d151-c41c-77a6-4eef96aaf71f [Follower]" leader= testing.go:149: server-49c4da0a-d151-c41c-77a6-4eef96aaf71f: 2024-01-09T05:09:54.650Z [INFO] server-49c4da0a-d151-c41c-77a6-4eef96aaf71f: entering follower state: follower="Node at 49c4da0a-d151-c41c-77a6-4eef96aaf71f [Follower]" leader= testLoggerAdapter verbose: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.650Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: entering follower state: follower="Node at 27bbf3ad-bb6e-b691-d788-e28474238269 [Follower]" leader= testing.go:149: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.650Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: entering follower state: follower="Node at 27bbf3ad-bb6e-b691-d788-e28474238269 [Follower]" leader= testLoggerAdapter verbose: server-63264def-4c66-c838-4ee0-f04dbd85cbd3: 2024-01-09T05:09:54.650Z [INFO] server-63264def-4c66-c838-4ee0-f04dbd85cbd3: entering follower state: follower="Node at 63264def-4c66-c838-4ee0-f04dbd85cbd3 [Follower]" leader= testing.go:149: server-63264def-4c66-c838-4ee0-f04dbd85cbd3: 2024-01-09T05:09:54.650Z [INFO] server-63264def-4c66-c838-4ee0-f04dbd85cbd3: entering follower state: follower="Node at 63264def-4c66-c838-4ee0-f04dbd85cbd3 [Follower]" leader= testLoggerAdapter verbose: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.716Z [WARN] server-27bbf3ad-bb6e-b691-d788-e28474238269: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.716Z [WARN] server-27bbf3ad-bb6e-b691-d788-e28474238269: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.716Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: entering candidate state: node="Node at 27bbf3ad-bb6e-b691-d788-e28474238269 [Candidate]" term=2 testing.go:149: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.716Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: entering candidate state: node="Node at 27bbf3ad-bb6e-b691-d788-e28474238269 [Candidate]" term=2 testLoggerAdapter verbose: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.716Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: election won: tally=2 testing.go:149: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.716Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: election won: tally=2 testLoggerAdapter verbose: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.716Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: entering leader state: leader="Node at 27bbf3ad-bb6e-b691-d788-e28474238269 [Leader]" testing.go:149: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.716Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: entering leader state: leader="Node at 27bbf3ad-bb6e-b691-d788-e28474238269 [Leader]" testLoggerAdapter verbose: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.716Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: added peer, starting replication: peer=server-49c4da0a-d151-c41c-77a6-4eef96aaf71f testing.go:149: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.716Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: added peer, starting replication: peer=server-49c4da0a-d151-c41c-77a6-4eef96aaf71f testLoggerAdapter verbose: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.716Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: added peer, starting replication: peer=server-63264def-4c66-c838-4ee0-f04dbd85cbd3 testing.go:149: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.716Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: added peer, starting replication: peer=server-63264def-4c66-c838-4ee0-f04dbd85cbd3 testLoggerAdapter verbose: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.716Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: pipelining replication: peer="{Voter server-63264def-4c66-c838-4ee0-f04dbd85cbd3 63264def-4c66-c838-4ee0-f04dbd85cbd3}" testing.go:149: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.716Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: pipelining replication: peer="{Voter server-63264def-4c66-c838-4ee0-f04dbd85cbd3 63264def-4c66-c838-4ee0-f04dbd85cbd3}" testLoggerAdapter verbose: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.716Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: pipelining replication: peer="{Voter server-49c4da0a-d151-c41c-77a6-4eef96aaf71f 49c4da0a-d151-c41c-77a6-4eef96aaf71f}" testing.go:149: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.716Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: pipelining replication: peer="{Voter server-49c4da0a-d151-c41c-77a6-4eef96aaf71f 49c4da0a-d151-c41c-77a6-4eef96aaf71f}" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:54.822Z [INFO] stable state for Leader reached at 2024-01-09 05:09:54.716838203 +0000 UTC m=+5.969889276 (1 nodes), 66.18785ms from start of poll, 66.668652ms from cluster start. Timeout at 2024-01-09 05:09:54.822185831 +0000 UTC m=+6.075236891, 105.347615ms after stability testing.go:149: cluster: 2024-01-09T05:09:54.822Z [INFO] stable state for Leader reached at 2024-01-09 05:09:54.716838203 +0000 UTC m=+5.969889276 (1 nodes), 66.18785ms from start of poll, 66.668652ms from cluster start. Timeout at 2024-01-09 05:09:54.822185831 +0000 UTC m=+6.075236891, 105.347615ms after stability testLoggerAdapter verbose: cluster: 2024-01-09T05:09:54.822Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-01-09T05:09:54.822Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2024-01-09T05:09:54.927Z [INFO] stable state for Follower reached at 2024-01-09 05:09:54.82236484 +0000 UTC m=+6.075415880 (2 nodes), 3.231µs from start of poll, 172.195256ms from cluster start. Timeout at 2024-01-09 05:09:54.927484484 +0000 UTC m=+6.180535517, 105.119637ms after stability testing.go:149: cluster: 2024-01-09T05:09:54.927Z [INFO] stable state for Follower reached at 2024-01-09 05:09:54.82236484 +0000 UTC m=+6.075415880 (2 nodes), 3.231µs from start of poll, 172.195256ms from cluster start. Timeout at 2024-01-09 05:09:54.927484484 +0000 UTC m=+6.180535517, 105.119637ms after stability testLoggerAdapter verbose: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.927Z [ERROR] server-27bbf3ad-bb6e-b691-d788-e28474238269: failed to pipeline appendEntries: peer="{Voter server-49c4da0a-d151-c41c-77a6-4eef96aaf71f 49c4da0a-d151-c41c-77a6-4eef96aaf71f}" error="append pipeline closed" testing.go:149: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.927Z [ERROR] server-27bbf3ad-bb6e-b691-d788-e28474238269: failed to pipeline appendEntries: peer="{Voter server-49c4da0a-d151-c41c-77a6-4eef96aaf71f 49c4da0a-d151-c41c-77a6-4eef96aaf71f}" error="append pipeline closed" testLoggerAdapter verbose: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.927Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: aborting pipeline replication: peer="{Voter server-49c4da0a-d151-c41c-77a6-4eef96aaf71f 49c4da0a-d151-c41c-77a6-4eef96aaf71f}" testing.go:149: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.927Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: aborting pipeline replication: peer="{Voter server-49c4da0a-d151-c41c-77a6-4eef96aaf71f 49c4da0a-d151-c41c-77a6-4eef96aaf71f}" testLoggerAdapter verbose: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.927Z [ERROR] server-27bbf3ad-bb6e-b691-d788-e28474238269: failed to appendEntries to: peer="{Voter server-49c4da0a-d151-c41c-77a6-4eef96aaf71f 49c4da0a-d151-c41c-77a6-4eef96aaf71f}" error="failed to connect to peer: 49c4da0a-d151-c41c-77a6-4eef96aaf71f" testing.go:149: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.927Z [ERROR] server-27bbf3ad-bb6e-b691-d788-e28474238269: failed to appendEntries to: peer="{Voter server-49c4da0a-d151-c41c-77a6-4eef96aaf71f 49c4da0a-d151-c41c-77a6-4eef96aaf71f}" error="failed to connect to peer: 49c4da0a-d151-c41c-77a6-4eef96aaf71f" raft_test.go:420: [INFO] Finished apply without behind follower testLoggerAdapter verbose: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.938Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: pipelining replication: peer="{Voter server-49c4da0a-d151-c41c-77a6-4eef96aaf71f 49c4da0a-d151-c41c-77a6-4eef96aaf71f}" testing.go:149: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:54.938Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: pipelining replication: peer="{Voter server-49c4da0a-d151-c41c-77a6-4eef96aaf71f 49c4da0a-d151-c41c-77a6-4eef96aaf71f}" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:54.950Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:09:54.950Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-01-09T05:09:55.055Z [INFO] stable state for Leader reached at 2024-01-09 05:09:54.950409839 +0000 UTC m=+6.203460861 (1 nodes), 2.201µs from start of poll, 300.240237ms from cluster start. Timeout at 2024-01-09 05:09:55.05551688 +0000 UTC m=+6.308568127, 105.107266ms after stability testing.go:149: cluster: 2024-01-09T05:09:55.055Z [INFO] stable state for Leader reached at 2024-01-09 05:09:54.950409839 +0000 UTC m=+6.203460861 (1 nodes), 2.201µs from start of poll, 300.240237ms from cluster start. Timeout at 2024-01-09 05:09:55.05551688 +0000 UTC m=+6.308568127, 105.107266ms after stability testLoggerAdapter verbose: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:55.055Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: aborting pipeline replication: peer="{Voter server-63264def-4c66-c838-4ee0-f04dbd85cbd3 63264def-4c66-c838-4ee0-f04dbd85cbd3}" testing.go:149: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:55.055Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: aborting pipeline replication: peer="{Voter server-63264def-4c66-c838-4ee0-f04dbd85cbd3 63264def-4c66-c838-4ee0-f04dbd85cbd3}" testLoggerAdapter verbose: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:55.055Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: aborting pipeline replication: peer="{Voter server-49c4da0a-d151-c41c-77a6-4eef96aaf71f 49c4da0a-d151-c41c-77a6-4eef96aaf71f}" testing.go:149: server-27bbf3ad-bb6e-b691-d788-e28474238269: 2024-01-09T05:09:55.055Z [INFO] server-27bbf3ad-bb6e-b691-d788-e28474238269: aborting pipeline replication: peer="{Voter server-49c4da0a-d151-c41c-77a6-4eef96aaf71f 49c4da0a-d151-c41c-77a6-4eef96aaf71f}" --- PASS: TestRaft_BehindFollower (0.41s) === RUN TestRaft_ApplyNonLeader testLoggerAdapter verbose: server-0b4ed842-f581-ab95-5247-4f33547ec339: 2024-01-09T05:09:55.058Z [INFO] server-0b4ed842-f581-ab95-5247-4f33547ec339: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b4ed842-f581-ab95-5247-4f33547ec339 Address:0b4ed842-f581-ab95-5247-4f33547ec339} {Suffrage:Voter ID:server-59592c2d-859d-7af8-e969-a9b189bef710 Address:59592c2d-859d-7af8-e969-a9b189bef710} {Suffrage:Voter ID:server-2e794893-82a8-228d-d44e-33ad22a55e36 Address:2e794893-82a8-228d-d44e-33ad22a55e36}]" testing.go:149: server-0b4ed842-f581-ab95-5247-4f33547ec339: 2024-01-09T05:09:55.058Z [INFO] server-0b4ed842-f581-ab95-5247-4f33547ec339: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b4ed842-f581-ab95-5247-4f33547ec339 Address:0b4ed842-f581-ab95-5247-4f33547ec339} {Suffrage:Voter ID:server-59592c2d-859d-7af8-e969-a9b189bef710 Address:59592c2d-859d-7af8-e969-a9b189bef710} {Suffrage:Voter ID:server-2e794893-82a8-228d-d44e-33ad22a55e36 Address:2e794893-82a8-228d-d44e-33ad22a55e36}]" testLoggerAdapter verbose: server-59592c2d-859d-7af8-e969-a9b189bef710: 2024-01-09T05:09:55.058Z [INFO] server-59592c2d-859d-7af8-e969-a9b189bef710: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b4ed842-f581-ab95-5247-4f33547ec339 Address:0b4ed842-f581-ab95-5247-4f33547ec339} {Suffrage:Voter ID:server-59592c2d-859d-7af8-e969-a9b189bef710 Address:59592c2d-859d-7af8-e969-a9b189bef710} {Suffrage:Voter ID:server-2e794893-82a8-228d-d44e-33ad22a55e36 Address:2e794893-82a8-228d-d44e-33ad22a55e36}]" testing.go:149: server-59592c2d-859d-7af8-e969-a9b189bef710: 2024-01-09T05:09:55.058Z [INFO] server-59592c2d-859d-7af8-e969-a9b189bef710: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b4ed842-f581-ab95-5247-4f33547ec339 Address:0b4ed842-f581-ab95-5247-4f33547ec339} {Suffrage:Voter ID:server-59592c2d-859d-7af8-e969-a9b189bef710 Address:59592c2d-859d-7af8-e969-a9b189bef710} {Suffrage:Voter ID:server-2e794893-82a8-228d-d44e-33ad22a55e36 Address:2e794893-82a8-228d-d44e-33ad22a55e36}]" testLoggerAdapter verbose: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.058Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b4ed842-f581-ab95-5247-4f33547ec339 Address:0b4ed842-f581-ab95-5247-4f33547ec339} {Suffrage:Voter ID:server-59592c2d-859d-7af8-e969-a9b189bef710 Address:59592c2d-859d-7af8-e969-a9b189bef710} {Suffrage:Voter ID:server-2e794893-82a8-228d-d44e-33ad22a55e36 Address:2e794893-82a8-228d-d44e-33ad22a55e36}]" testing.go:149: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.058Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b4ed842-f581-ab95-5247-4f33547ec339 Address:0b4ed842-f581-ab95-5247-4f33547ec339} {Suffrage:Voter ID:server-59592c2d-859d-7af8-e969-a9b189bef710 Address:59592c2d-859d-7af8-e969-a9b189bef710} {Suffrage:Voter ID:server-2e794893-82a8-228d-d44e-33ad22a55e36 Address:2e794893-82a8-228d-d44e-33ad22a55e36}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:55.059Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-59592c2d-859d-7af8-e969-a9b189bef710: 2024-01-09T05:09:55.059Z [INFO] server-59592c2d-859d-7af8-e969-a9b189bef710: entering follower state: follower="Node at 59592c2d-859d-7af8-e969-a9b189bef710 [Follower]" leader= testLoggerAdapter verbose: server-0b4ed842-f581-ab95-5247-4f33547ec339: 2024-01-09T05:09:55.059Z [INFO] server-0b4ed842-f581-ab95-5247-4f33547ec339: entering follower state: follower="Node at 0b4ed842-f581-ab95-5247-4f33547ec339 [Follower]" leader= testLoggerAdapter verbose: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.059Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: entering follower state: follower="Node at 2e794893-82a8-228d-d44e-33ad22a55e36 [Follower]" leader= testing.go:149: cluster: 2024-01-09T05:09:55.059Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-59592c2d-859d-7af8-e969-a9b189bef710: 2024-01-09T05:09:55.059Z [INFO] server-59592c2d-859d-7af8-e969-a9b189bef710: entering follower state: follower="Node at 59592c2d-859d-7af8-e969-a9b189bef710 [Follower]" leader= testing.go:149: server-0b4ed842-f581-ab95-5247-4f33547ec339: 2024-01-09T05:09:55.059Z [INFO] server-0b4ed842-f581-ab95-5247-4f33547ec339: entering follower state: follower="Node at 0b4ed842-f581-ab95-5247-4f33547ec339 [Follower]" leader= testing.go:149: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.059Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: entering follower state: follower="Node at 2e794893-82a8-228d-d44e-33ad22a55e36 [Follower]" leader= testLoggerAdapter verbose: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.124Z [WARN] server-2e794893-82a8-228d-d44e-33ad22a55e36: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.124Z [WARN] server-2e794893-82a8-228d-d44e-33ad22a55e36: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.124Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: entering candidate state: node="Node at 2e794893-82a8-228d-d44e-33ad22a55e36 [Candidate]" term=2 testing.go:149: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.124Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: entering candidate state: node="Node at 2e794893-82a8-228d-d44e-33ad22a55e36 [Candidate]" term=2 testLoggerAdapter verbose: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.124Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: election won: tally=2 testing.go:149: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.124Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: election won: tally=2 testLoggerAdapter verbose: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.124Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: entering leader state: leader="Node at 2e794893-82a8-228d-d44e-33ad22a55e36 [Leader]" testing.go:149: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.124Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: entering leader state: leader="Node at 2e794893-82a8-228d-d44e-33ad22a55e36 [Leader]" testLoggerAdapter verbose: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.124Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: added peer, starting replication: peer=server-0b4ed842-f581-ab95-5247-4f33547ec339 testing.go:149: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.124Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: added peer, starting replication: peer=server-0b4ed842-f581-ab95-5247-4f33547ec339 testLoggerAdapter verbose: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.124Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: added peer, starting replication: peer=server-59592c2d-859d-7af8-e969-a9b189bef710 testing.go:149: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.124Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: added peer, starting replication: peer=server-59592c2d-859d-7af8-e969-a9b189bef710 testLoggerAdapter verbose: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.124Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: pipelining replication: peer="{Voter server-59592c2d-859d-7af8-e969-a9b189bef710 59592c2d-859d-7af8-e969-a9b189bef710}" testing.go:149: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.124Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: pipelining replication: peer="{Voter server-59592c2d-859d-7af8-e969-a9b189bef710 59592c2d-859d-7af8-e969-a9b189bef710}" testLoggerAdapter verbose: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.124Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: pipelining replication: peer="{Voter server-0b4ed842-f581-ab95-5247-4f33547ec339 0b4ed842-f581-ab95-5247-4f33547ec339}" testing.go:149: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.124Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: pipelining replication: peer="{Voter server-0b4ed842-f581-ab95-5247-4f33547ec339 0b4ed842-f581-ab95-5247-4f33547ec339}" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:55.230Z [INFO] stable state for Leader reached at 2024-01-09 05:09:55.125061163 +0000 UTC m=+6.378112175 (1 nodes), 65.523595ms from start of poll, 66.767561ms from cluster start. Timeout at 2024-01-09 05:09:55.230241081 +0000 UTC m=+6.483292171, 105.179996ms after stability testing.go:149: cluster: 2024-01-09T05:09:55.230Z [INFO] stable state for Leader reached at 2024-01-09 05:09:55.125061163 +0000 UTC m=+6.378112175 (1 nodes), 65.523595ms from start of poll, 66.767561ms from cluster start. Timeout at 2024-01-09 05:09:55.230241081 +0000 UTC m=+6.483292171, 105.179996ms after stability testLoggerAdapter verbose: cluster: 2024-01-09T05:09:55.230Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-01-09T05:09:55.230Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2024-01-09T05:09:55.336Z [INFO] stable state for Follower reached at 2024-01-09 05:09:55.231027263 +0000 UTC m=+6.484078289 (2 nodes), 2.875µs from start of poll, 172.733675ms from cluster start. Timeout at 2024-01-09 05:09:55.336115841 +0000 UTC m=+6.589166854, 105.088565ms after stability testing.go:149: cluster: 2024-01-09T05:09:55.336Z [INFO] stable state for Follower reached at 2024-01-09 05:09:55.231027263 +0000 UTC m=+6.484078289 (2 nodes), 2.875µs from start of poll, 172.733675ms from cluster start. Timeout at 2024-01-09 05:09:55.336115841 +0000 UTC m=+6.589166854, 105.088565ms after stability testLoggerAdapter verbose: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.336Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: aborting pipeline replication: peer="{Voter server-59592c2d-859d-7af8-e969-a9b189bef710 59592c2d-859d-7af8-e969-a9b189bef710}" testing.go:149: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.336Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: aborting pipeline replication: peer="{Voter server-59592c2d-859d-7af8-e969-a9b189bef710 59592c2d-859d-7af8-e969-a9b189bef710}" testLoggerAdapter verbose: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.336Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: aborting pipeline replication: peer="{Voter server-0b4ed842-f581-ab95-5247-4f33547ec339 0b4ed842-f581-ab95-5247-4f33547ec339}" testing.go:149: server-2e794893-82a8-228d-d44e-33ad22a55e36: 2024-01-09T05:09:55.336Z [INFO] server-2e794893-82a8-228d-d44e-33ad22a55e36: aborting pipeline replication: peer="{Voter server-0b4ed842-f581-ab95-5247-4f33547ec339 0b4ed842-f581-ab95-5247-4f33547ec339}" --- PASS: TestRaft_ApplyNonLeader (0.28s) === RUN TestRaft_ApplyConcurrent testLoggerAdapter verbose: server-a39db28d-8f3a-024e-8ef6-61762d94d488: 2024-01-09T05:09:55.337Z [INFO] server-a39db28d-8f3a-024e-8ef6-61762d94d488: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a39db28d-8f3a-024e-8ef6-61762d94d488 Address:a39db28d-8f3a-024e-8ef6-61762d94d488} {Suffrage:Voter ID:server-ac2de96e-8850-c488-3df4-bc550f9a55ec Address:ac2de96e-8850-c488-3df4-bc550f9a55ec} {Suffrage:Voter ID:server-9d2584ff-67d8-dff1-816e-661ae3882d48 Address:9d2584ff-67d8-dff1-816e-661ae3882d48}]" testing.go:149: server-a39db28d-8f3a-024e-8ef6-61762d94d488: 2024-01-09T05:09:55.337Z [INFO] server-a39db28d-8f3a-024e-8ef6-61762d94d488: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a39db28d-8f3a-024e-8ef6-61762d94d488 Address:a39db28d-8f3a-024e-8ef6-61762d94d488} {Suffrage:Voter ID:server-ac2de96e-8850-c488-3df4-bc550f9a55ec Address:ac2de96e-8850-c488-3df4-bc550f9a55ec} {Suffrage:Voter ID:server-9d2584ff-67d8-dff1-816e-661ae3882d48 Address:9d2584ff-67d8-dff1-816e-661ae3882d48}]" testLoggerAdapter verbose: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.337Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a39db28d-8f3a-024e-8ef6-61762d94d488 Address:a39db28d-8f3a-024e-8ef6-61762d94d488} {Suffrage:Voter ID:server-ac2de96e-8850-c488-3df4-bc550f9a55ec Address:ac2de96e-8850-c488-3df4-bc550f9a55ec} {Suffrage:Voter ID:server-9d2584ff-67d8-dff1-816e-661ae3882d48 Address:9d2584ff-67d8-dff1-816e-661ae3882d48}]" testing.go:149: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.337Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a39db28d-8f3a-024e-8ef6-61762d94d488 Address:a39db28d-8f3a-024e-8ef6-61762d94d488} {Suffrage:Voter ID:server-ac2de96e-8850-c488-3df4-bc550f9a55ec Address:ac2de96e-8850-c488-3df4-bc550f9a55ec} {Suffrage:Voter ID:server-9d2584ff-67d8-dff1-816e-661ae3882d48 Address:9d2584ff-67d8-dff1-816e-661ae3882d48}]" testLoggerAdapter verbose: server-9d2584ff-67d8-dff1-816e-661ae3882d48: 2024-01-09T05:09:55.337Z [INFO] server-9d2584ff-67d8-dff1-816e-661ae3882d48: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a39db28d-8f3a-024e-8ef6-61762d94d488 Address:a39db28d-8f3a-024e-8ef6-61762d94d488} {Suffrage:Voter ID:server-ac2de96e-8850-c488-3df4-bc550f9a55ec Address:ac2de96e-8850-c488-3df4-bc550f9a55ec} {Suffrage:Voter ID:server-9d2584ff-67d8-dff1-816e-661ae3882d48 Address:9d2584ff-67d8-dff1-816e-661ae3882d48}]" testing.go:149: server-9d2584ff-67d8-dff1-816e-661ae3882d48: 2024-01-09T05:09:55.337Z [INFO] server-9d2584ff-67d8-dff1-816e-661ae3882d48: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a39db28d-8f3a-024e-8ef6-61762d94d488 Address:a39db28d-8f3a-024e-8ef6-61762d94d488} {Suffrage:Voter ID:server-ac2de96e-8850-c488-3df4-bc550f9a55ec Address:ac2de96e-8850-c488-3df4-bc550f9a55ec} {Suffrage:Voter ID:server-9d2584ff-67d8-dff1-816e-661ae3882d48 Address:9d2584ff-67d8-dff1-816e-661ae3882d48}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:55.337Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:09:55.337Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-a39db28d-8f3a-024e-8ef6-61762d94d488: 2024-01-09T05:09:55.338Z [INFO] server-a39db28d-8f3a-024e-8ef6-61762d94d488: entering follower state: follower="Node at a39db28d-8f3a-024e-8ef6-61762d94d488 [Follower]" leader= testing.go:149: server-a39db28d-8f3a-024e-8ef6-61762d94d488: 2024-01-09T05:09:55.338Z [INFO] server-a39db28d-8f3a-024e-8ef6-61762d94d488: entering follower state: follower="Node at a39db28d-8f3a-024e-8ef6-61762d94d488 [Follower]" leader= testLoggerAdapter verbose: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.338Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: entering follower state: follower="Node at ac2de96e-8850-c488-3df4-bc550f9a55ec [Follower]" leader= testing.go:149: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.338Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: entering follower state: follower="Node at ac2de96e-8850-c488-3df4-bc550f9a55ec [Follower]" leader= testLoggerAdapter verbose: server-9d2584ff-67d8-dff1-816e-661ae3882d48: 2024-01-09T05:09:55.338Z [INFO] server-9d2584ff-67d8-dff1-816e-661ae3882d48: entering follower state: follower="Node at 9d2584ff-67d8-dff1-816e-661ae3882d48 [Follower]" leader= testing.go:149: server-9d2584ff-67d8-dff1-816e-661ae3882d48: 2024-01-09T05:09:55.338Z [INFO] server-9d2584ff-67d8-dff1-816e-661ae3882d48: entering follower state: follower="Node at 9d2584ff-67d8-dff1-816e-661ae3882d48 [Follower]" leader= testLoggerAdapter verbose: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.447Z [WARN] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.447Z [WARN] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.447Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: entering candidate state: node="Node at ac2de96e-8850-c488-3df4-bc550f9a55ec [Candidate]" term=2 testing.go:149: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.447Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: entering candidate state: node="Node at ac2de96e-8850-c488-3df4-bc550f9a55ec [Candidate]" term=2 testLoggerAdapter verbose: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.447Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: election won: tally=2 testing.go:149: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.447Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: election won: tally=2 testLoggerAdapter verbose: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.447Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: entering leader state: leader="Node at ac2de96e-8850-c488-3df4-bc550f9a55ec [Leader]" testing.go:149: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.447Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: entering leader state: leader="Node at ac2de96e-8850-c488-3df4-bc550f9a55ec [Leader]" testLoggerAdapter verbose: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.448Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: added peer, starting replication: peer=server-a39db28d-8f3a-024e-8ef6-61762d94d488 testing.go:149: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.448Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: added peer, starting replication: peer=server-a39db28d-8f3a-024e-8ef6-61762d94d488 testLoggerAdapter verbose: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.448Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: added peer, starting replication: peer=server-9d2584ff-67d8-dff1-816e-661ae3882d48 testing.go:149: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.448Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: added peer, starting replication: peer=server-9d2584ff-67d8-dff1-816e-661ae3882d48 testLoggerAdapter verbose: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.448Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: pipelining replication: peer="{Voter server-9d2584ff-67d8-dff1-816e-661ae3882d48 9d2584ff-67d8-dff1-816e-661ae3882d48}" testing.go:149: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.448Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: pipelining replication: peer="{Voter server-9d2584ff-67d8-dff1-816e-661ae3882d48 9d2584ff-67d8-dff1-816e-661ae3882d48}" testLoggerAdapter verbose: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.448Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: pipelining replication: peer="{Voter server-a39db28d-8f3a-024e-8ef6-61762d94d488 a39db28d-8f3a-024e-8ef6-61762d94d488}" testing.go:149: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.448Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: pipelining replication: peer="{Voter server-a39db28d-8f3a-024e-8ef6-61762d94d488 a39db28d-8f3a-024e-8ef6-61762d94d488}" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:55.653Z [INFO] stable state for Leader reached at 2024-01-09 05:09:55.448192383 +0000 UTC m=+6.701244041 (1 nodes), 110.207265ms from start of poll, 110.624866ms from cluster start. Timeout at 2024-01-09 05:09:55.653584874 +0000 UTC m=+6.906636046, 205.392005ms after stability testing.go:149: cluster: 2024-01-09T05:09:55.653Z [INFO] stable state for Leader reached at 2024-01-09 05:09:55.448192383 +0000 UTC m=+6.701244041 (1 nodes), 110.207265ms from start of poll, 110.624866ms from cluster start. Timeout at 2024-01-09 05:09:55.653584874 +0000 UTC m=+6.906636046, 205.392005ms after stability testLoggerAdapter verbose: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.674Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: aborting pipeline replication: peer="{Voter server-9d2584ff-67d8-dff1-816e-661ae3882d48 9d2584ff-67d8-dff1-816e-661ae3882d48}" testing.go:149: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.674Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: aborting pipeline replication: peer="{Voter server-9d2584ff-67d8-dff1-816e-661ae3882d48 9d2584ff-67d8-dff1-816e-661ae3882d48}" testLoggerAdapter verbose: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.674Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: aborting pipeline replication: peer="{Voter server-a39db28d-8f3a-024e-8ef6-61762d94d488 a39db28d-8f3a-024e-8ef6-61762d94d488}" testing.go:149: server-ac2de96e-8850-c488-3df4-bc550f9a55ec: 2024-01-09T05:09:55.674Z [INFO] server-ac2de96e-8850-c488-3df4-bc550f9a55ec: aborting pipeline replication: peer="{Voter server-a39db28d-8f3a-024e-8ef6-61762d94d488 a39db28d-8f3a-024e-8ef6-61762d94d488}" --- PASS: TestRaft_ApplyConcurrent (0.34s) === RUN TestRaft_ApplyConcurrent_Timeout testLoggerAdapter verbose: server-7ca408a2-f45d-b934-ce21-049cc0a2f197: 2024-01-09T05:09:55.675Z [INFO] server-7ca408a2-f45d-b934-ce21-049cc0a2f197: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7ca408a2-f45d-b934-ce21-049cc0a2f197 Address:7ca408a2-f45d-b934-ce21-049cc0a2f197}]" testing.go:149: server-7ca408a2-f45d-b934-ce21-049cc0a2f197: 2024-01-09T05:09:55.675Z [INFO] server-7ca408a2-f45d-b934-ce21-049cc0a2f197: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7ca408a2-f45d-b934-ce21-049cc0a2f197 Address:7ca408a2-f45d-b934-ce21-049cc0a2f197}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:55.675Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:09:55.675Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-7ca408a2-f45d-b934-ce21-049cc0a2f197: 2024-01-09T05:09:55.675Z [INFO] server-7ca408a2-f45d-b934-ce21-049cc0a2f197: entering follower state: follower="Node at 7ca408a2-f45d-b934-ce21-049cc0a2f197 [Follower]" leader= testing.go:149: server-7ca408a2-f45d-b934-ce21-049cc0a2f197: 2024-01-09T05:09:55.675Z [INFO] server-7ca408a2-f45d-b934-ce21-049cc0a2f197: entering follower state: follower="Node at 7ca408a2-f45d-b934-ce21-049cc0a2f197 [Follower]" leader= testLoggerAdapter verbose: server-7ca408a2-f45d-b934-ce21-049cc0a2f197: 2024-01-09T05:09:55.856Z [WARN] server-7ca408a2-f45d-b934-ce21-049cc0a2f197: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-7ca408a2-f45d-b934-ce21-049cc0a2f197: 2024-01-09T05:09:55.856Z [WARN] server-7ca408a2-f45d-b934-ce21-049cc0a2f197: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-7ca408a2-f45d-b934-ce21-049cc0a2f197: 2024-01-09T05:09:55.856Z [INFO] server-7ca408a2-f45d-b934-ce21-049cc0a2f197: entering candidate state: node="Node at 7ca408a2-f45d-b934-ce21-049cc0a2f197 [Candidate]" term=2 testing.go:149: server-7ca408a2-f45d-b934-ce21-049cc0a2f197: 2024-01-09T05:09:55.856Z [INFO] server-7ca408a2-f45d-b934-ce21-049cc0a2f197: entering candidate state: node="Node at 7ca408a2-f45d-b934-ce21-049cc0a2f197 [Candidate]" term=2 testLoggerAdapter verbose: server-7ca408a2-f45d-b934-ce21-049cc0a2f197: 2024-01-09T05:09:55.856Z [INFO] server-7ca408a2-f45d-b934-ce21-049cc0a2f197: election won: tally=1 testing.go:149: server-7ca408a2-f45d-b934-ce21-049cc0a2f197: 2024-01-09T05:09:55.856Z [INFO] server-7ca408a2-f45d-b934-ce21-049cc0a2f197: election won: tally=1 testLoggerAdapter verbose: server-7ca408a2-f45d-b934-ce21-049cc0a2f197: 2024-01-09T05:09:55.856Z [INFO] server-7ca408a2-f45d-b934-ce21-049cc0a2f197: entering leader state: leader="Node at 7ca408a2-f45d-b934-ce21-049cc0a2f197 [Leader]" testing.go:149: server-7ca408a2-f45d-b934-ce21-049cc0a2f197: 2024-01-09T05:09:55.856Z [INFO] server-7ca408a2-f45d-b934-ce21-049cc0a2f197: entering leader state: leader="Node at 7ca408a2-f45d-b934-ce21-049cc0a2f197 [Leader]" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:56.057Z [INFO] stable state for Leader reached at 2024-01-09 05:09:55.856495202 +0000 UTC m=+7.109546247 (1 nodes), 180.709198ms from start of poll, 180.951814ms from cluster start. Timeout at 2024-01-09 05:09:56.057627124 +0000 UTC m=+7.310678365, 201.132118ms after stability testing.go:149: cluster: 2024-01-09T05:09:56.057Z [INFO] stable state for Leader reached at 2024-01-09 05:09:55.856495202 +0000 UTC m=+7.109546247 (1 nodes), 180.709198ms from start of poll, 180.951814ms from cluster start. Timeout at 2024-01-09 05:09:56.057627124 +0000 UTC m=+7.310678365, 201.132118ms after stability --- PASS: TestRaft_ApplyConcurrent_Timeout (0.38s) === RUN TestRaft_JoinNode testLoggerAdapter verbose: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.060Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3147839-f31a-e42c-3c68-b2f2bf175735 Address:c3147839-f31a-e42c-3c68-b2f2bf175735} {Suffrage:Voter ID:server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89 Address:c3131786-6d4c-2e2e-0788-7fd8fa0b9b89}]" testing.go:149: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.060Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3147839-f31a-e42c-3c68-b2f2bf175735 Address:c3147839-f31a-e42c-3c68-b2f2bf175735} {Suffrage:Voter ID:server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89 Address:c3131786-6d4c-2e2e-0788-7fd8fa0b9b89}]" testLoggerAdapter verbose: server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89: 2024-01-09T05:09:56.060Z [INFO] server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3147839-f31a-e42c-3c68-b2f2bf175735 Address:c3147839-f31a-e42c-3c68-b2f2bf175735} {Suffrage:Voter ID:server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89 Address:c3131786-6d4c-2e2e-0788-7fd8fa0b9b89}]" testing.go:149: server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89: 2024-01-09T05:09:56.060Z [INFO] server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3147839-f31a-e42c-3c68-b2f2bf175735 Address:c3147839-f31a-e42c-3c68-b2f2bf175735} {Suffrage:Voter ID:server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89 Address:c3131786-6d4c-2e2e-0788-7fd8fa0b9b89}]" testLoggerAdapter verbose: server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1: 2024-01-09T05:09:56.061Z [INFO] server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1: initial configuration: index=0 servers=[] testing.go:149: server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1: 2024-01-09T05:09:56.061Z [INFO] server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2024-01-09T05:09:56.061Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:09:56.061Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.061Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: entering follower state: follower="Node at c3147839-f31a-e42c-3c68-b2f2bf175735 [Follower]" leader= testing.go:149: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.061Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: entering follower state: follower="Node at c3147839-f31a-e42c-3c68-b2f2bf175735 [Follower]" leader= testLoggerAdapter verbose: server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1: 2024-01-09T05:09:56.061Z [INFO] server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1: entering follower state: follower="Node at eaf7e51b-eef0-35ee-2498-9c3d8f4366c1 [Follower]" leader= testLoggerAdapter verbose: server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89: 2024-01-09T05:09:56.061Z [INFO] server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89: entering follower state: follower="Node at c3131786-6d4c-2e2e-0788-7fd8fa0b9b89 [Follower]" leader= testing.go:149: server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1: 2024-01-09T05:09:56.061Z [INFO] server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1: entering follower state: follower="Node at eaf7e51b-eef0-35ee-2498-9c3d8f4366c1 [Follower]" leader= testing.go:149: server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89: 2024-01-09T05:09:56.061Z [INFO] server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89: entering follower state: follower="Node at c3131786-6d4c-2e2e-0788-7fd8fa0b9b89 [Follower]" leader= testLoggerAdapter verbose: server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1: 2024-01-09T05:09:56.116Z [WARN] server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1: no known peers, aborting election testing.go:149: server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1: 2024-01-09T05:09:56.116Z [WARN] server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1: no known peers, aborting election testLoggerAdapter verbose: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.124Z [WARN] server-c3147839-f31a-e42c-3c68-b2f2bf175735: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.124Z [WARN] server-c3147839-f31a-e42c-3c68-b2f2bf175735: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.124Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: entering candidate state: node="Node at c3147839-f31a-e42c-3c68-b2f2bf175735 [Candidate]" term=2 testing.go:149: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.124Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: entering candidate state: node="Node at c3147839-f31a-e42c-3c68-b2f2bf175735 [Candidate]" term=2 testLoggerAdapter verbose: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.125Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: election won: tally=2 testing.go:149: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.125Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: election won: tally=2 testLoggerAdapter verbose: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.125Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: entering leader state: leader="Node at c3147839-f31a-e42c-3c68-b2f2bf175735 [Leader]" testing.go:149: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.125Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: entering leader state: leader="Node at c3147839-f31a-e42c-3c68-b2f2bf175735 [Leader]" testLoggerAdapter verbose: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.125Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: added peer, starting replication: peer=server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89 testing.go:149: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.125Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: added peer, starting replication: peer=server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89 testLoggerAdapter verbose: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.125Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: pipelining replication: peer="{Voter server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89 c3131786-6d4c-2e2e-0788-7fd8fa0b9b89}" testing.go:149: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.125Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: pipelining replication: peer="{Voter server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89 c3131786-6d4c-2e2e-0788-7fd8fa0b9b89}" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:56.235Z [INFO] stable state for Leader reached at 2024-01-09 05:09:56.125291561 +0000 UTC m=+7.378343029 (1 nodes), 63.809808ms from start of poll, 65.170298ms from cluster start. Timeout at 2024-01-09 05:09:56.235075666 +0000 UTC m=+7.488126682, 109.783653ms after stability testing.go:149: cluster: 2024-01-09T05:09:56.235Z [INFO] stable state for Leader reached at 2024-01-09 05:09:56.125291561 +0000 UTC m=+7.378343029 (1 nodes), 63.809808ms from start of poll, 65.170298ms from cluster start. Timeout at 2024-01-09 05:09:56.235075666 +0000 UTC m=+7.488126682, 109.783653ms after stability testLoggerAdapter verbose: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.235Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: updating configuration: command=AddStaging server-id=server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1 server-addr=eaf7e51b-eef0-35ee-2498-9c3d8f4366c1 servers="[{Suffrage:Voter ID:server-c3147839-f31a-e42c-3c68-b2f2bf175735 Address:c3147839-f31a-e42c-3c68-b2f2bf175735} {Suffrage:Voter ID:server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89 Address:c3131786-6d4c-2e2e-0788-7fd8fa0b9b89} {Suffrage:Voter ID:server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1 Address:eaf7e51b-eef0-35ee-2498-9c3d8f4366c1}]" testing.go:149: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.235Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: updating configuration: command=AddStaging server-id=server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1 server-addr=eaf7e51b-eef0-35ee-2498-9c3d8f4366c1 servers="[{Suffrage:Voter ID:server-c3147839-f31a-e42c-3c68-b2f2bf175735 Address:c3147839-f31a-e42c-3c68-b2f2bf175735} {Suffrage:Voter ID:server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89 Address:c3131786-6d4c-2e2e-0788-7fd8fa0b9b89} {Suffrage:Voter ID:server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1 Address:eaf7e51b-eef0-35ee-2498-9c3d8f4366c1}]" testLoggerAdapter verbose: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.235Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: added peer, starting replication: peer=server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1 testing.go:149: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.235Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: added peer, starting replication: peer=server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1 testLoggerAdapter verbose: server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1: 2024-01-09T05:09:56.235Z [WARN] server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1: 2024-01-09T05:09:56.235Z [WARN] server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.235Z [WARN] server-c3147839-f31a-e42c-3c68-b2f2bf175735: appendEntries rejected, sending older logs: peer="{Voter server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1 eaf7e51b-eef0-35ee-2498-9c3d8f4366c1}" next=1 testing.go:149: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.235Z [WARN] server-c3147839-f31a-e42c-3c68-b2f2bf175735: appendEntries rejected, sending older logs: peer="{Voter server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1 eaf7e51b-eef0-35ee-2498-9c3d8f4366c1}" next=1 testLoggerAdapter verbose: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.235Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: pipelining replication: peer="{Voter server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1 eaf7e51b-eef0-35ee-2498-9c3d8f4366c1}" testing.go:149: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.235Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: pipelining replication: peer="{Voter server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1 eaf7e51b-eef0-35ee-2498-9c3d8f4366c1}" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:56.235Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:09:56.235Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-01-09T05:09:56.341Z [INFO] stable state for Leader reached at 2024-01-09 05:09:56.235776046 +0000 UTC m=+7.488834643 (1 nodes), 9.779µs from start of poll, 175.661912ms from cluster start. Timeout at 2024-01-09 05:09:56.341143553 +0000 UTC m=+7.594194585, 105.359942ms after stability testing.go:149: cluster: 2024-01-09T05:09:56.341Z [INFO] stable state for Leader reached at 2024-01-09 05:09:56.235776046 +0000 UTC m=+7.488834643 (1 nodes), 9.779µs from start of poll, 175.661912ms from cluster start. Timeout at 2024-01-09 05:09:56.341143553 +0000 UTC m=+7.594194585, 105.359942ms after stability testLoggerAdapter verbose: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.341Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: aborting pipeline replication: peer="{Voter server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1 eaf7e51b-eef0-35ee-2498-9c3d8f4366c1}" testing.go:149: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.341Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: aborting pipeline replication: peer="{Voter server-eaf7e51b-eef0-35ee-2498-9c3d8f4366c1 eaf7e51b-eef0-35ee-2498-9c3d8f4366c1}" testLoggerAdapter verbose: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.341Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: aborting pipeline replication: peer="{Voter server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89 c3131786-6d4c-2e2e-0788-7fd8fa0b9b89}" testing.go:149: server-c3147839-f31a-e42c-3c68-b2f2bf175735: 2024-01-09T05:09:56.341Z [INFO] server-c3147839-f31a-e42c-3c68-b2f2bf175735: aborting pipeline replication: peer="{Voter server-c3131786-6d4c-2e2e-0788-7fd8fa0b9b89 c3131786-6d4c-2e2e-0788-7fd8fa0b9b89}" --- PASS: TestRaft_JoinNode (0.28s) === RUN TestRaft_JoinNode_ConfigStore testLoggerAdapter verbose: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.343Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-97371e1f-22e0-76af-4a81-5afa5cfd8315 Address:97371e1f-22e0-76af-4a81-5afa5cfd8315}]" testing.go:149: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.343Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-97371e1f-22e0-76af-4a81-5afa5cfd8315 Address:97371e1f-22e0-76af-4a81-5afa5cfd8315}]" testLoggerAdapter verbose: server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69: 2024-01-09T05:09:56.343Z [INFO] server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69: initial configuration: index=0 servers=[] testing.go:149: server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69: 2024-01-09T05:09:56.343Z [INFO] server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f: 2024-01-09T05:09:56.343Z [INFO] server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f: initial configuration: index=0 servers=[] testing.go:149: server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f: 2024-01-09T05:09:56.343Z [INFO] server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2024-01-09T05:09:56.343Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:09:56.343Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.343Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: entering follower state: follower="Node at 97371e1f-22e0-76af-4a81-5afa5cfd8315 [Follower]" leader= testing.go:149: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.343Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: entering follower state: follower="Node at 97371e1f-22e0-76af-4a81-5afa5cfd8315 [Follower]" leader= testLoggerAdapter verbose: server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69: 2024-01-09T05:09:56.343Z [INFO] server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69: entering follower state: follower="Node at b1d2a2c9-ea13-4c34-4d91-d26b22449f69 [Follower]" leader= testing.go:149: server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69: 2024-01-09T05:09:56.343Z [INFO] server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69: entering follower state: follower="Node at b1d2a2c9-ea13-4c34-4d91-d26b22449f69 [Follower]" leader= testLoggerAdapter verbose: server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f: 2024-01-09T05:09:56.343Z [INFO] server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f: entering follower state: follower="Node at cdb7c38b-a9bd-110e-c6fd-d856526f9d2f [Follower]" leader= testing.go:149: server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f: 2024-01-09T05:09:56.343Z [INFO] server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f: entering follower state: follower="Node at cdb7c38b-a9bd-110e-c6fd-d856526f9d2f [Follower]" leader= testLoggerAdapter verbose: server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69: 2024-01-09T05:09:56.409Z [WARN] server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69: no known peers, aborting election testing.go:149: server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69: 2024-01-09T05:09:56.409Z [WARN] server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69: no known peers, aborting election testLoggerAdapter verbose: server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f: 2024-01-09T05:09:56.422Z [WARN] server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f: no known peers, aborting election testing.go:149: server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f: 2024-01-09T05:09:56.422Z [WARN] server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f: no known peers, aborting election testLoggerAdapter verbose: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.433Z [WARN] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.433Z [WARN] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.433Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: entering candidate state: node="Node at 97371e1f-22e0-76af-4a81-5afa5cfd8315 [Candidate]" term=2 testing.go:149: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.433Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: entering candidate state: node="Node at 97371e1f-22e0-76af-4a81-5afa5cfd8315 [Candidate]" term=2 testLoggerAdapter verbose: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.433Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: election won: tally=1 testing.go:149: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.433Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: election won: tally=1 testLoggerAdapter verbose: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.433Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: entering leader state: leader="Node at 97371e1f-22e0-76af-4a81-5afa5cfd8315 [Leader]" testing.go:149: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.433Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: entering leader state: leader="Node at 97371e1f-22e0-76af-4a81-5afa5cfd8315 [Leader]" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:56.538Z [INFO] stable state for Leader reached at 2024-01-09 05:09:56.43364299 +0000 UTC m=+7.686694602 (1 nodes), 89.816074ms from start of poll, 90.73522ms from cluster start. Timeout at 2024-01-09 05:09:56.538766278 +0000 UTC m=+7.791817515, 105.122913ms after stability testing.go:149: cluster: 2024-01-09T05:09:56.538Z [INFO] stable state for Leader reached at 2024-01-09 05:09:56.43364299 +0000 UTC m=+7.686694602 (1 nodes), 89.816074ms from start of poll, 90.73522ms from cluster start. Timeout at 2024-01-09 05:09:56.538766278 +0000 UTC m=+7.791817515, 105.122913ms after stability testLoggerAdapter verbose: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.538Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: updating configuration: command=AddStaging server-id=server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69 server-addr=b1d2a2c9-ea13-4c34-4d91-d26b22449f69 servers="[{Suffrage:Voter ID:server-97371e1f-22e0-76af-4a81-5afa5cfd8315 Address:97371e1f-22e0-76af-4a81-5afa5cfd8315} {Suffrage:Voter ID:server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69 Address:b1d2a2c9-ea13-4c34-4d91-d26b22449f69}]" testing.go:149: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.538Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: updating configuration: command=AddStaging server-id=server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69 server-addr=b1d2a2c9-ea13-4c34-4d91-d26b22449f69 servers="[{Suffrage:Voter ID:server-97371e1f-22e0-76af-4a81-5afa5cfd8315 Address:97371e1f-22e0-76af-4a81-5afa5cfd8315} {Suffrage:Voter ID:server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69 Address:b1d2a2c9-ea13-4c34-4d91-d26b22449f69}]" testLoggerAdapter verbose: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.539Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: added peer, starting replication: peer=server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69 testing.go:149: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.539Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: added peer, starting replication: peer=server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69 testLoggerAdapter verbose: cluster: 2024-01-09T05:09:56.539Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:09:56.539Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69: 2024-01-09T05:09:56.539Z [WARN] server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69: 2024-01-09T05:09:56.539Z [WARN] server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.539Z [WARN] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: appendEntries rejected, sending older logs: peer="{Voter server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69 b1d2a2c9-ea13-4c34-4d91-d26b22449f69}" next=1 testing.go:149: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.539Z [WARN] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: appendEntries rejected, sending older logs: peer="{Voter server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69 b1d2a2c9-ea13-4c34-4d91-d26b22449f69}" next=1 testLoggerAdapter verbose: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.539Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: pipelining replication: peer="{Voter server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69 b1d2a2c9-ea13-4c34-4d91-d26b22449f69}" testing.go:149: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.539Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: pipelining replication: peer="{Voter server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69 b1d2a2c9-ea13-4c34-4d91-d26b22449f69}" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:56.644Z [INFO] stable state for Leader reached at 2024-01-09 05:09:56.539227581 +0000 UTC m=+7.792278707 (1 nodes), 3.752µs from start of poll, 196.319325ms from cluster start. Timeout at 2024-01-09 05:09:56.644333904 +0000 UTC m=+7.897385032, 105.106325ms after stability testing.go:149: cluster: 2024-01-09T05:09:56.644Z [INFO] stable state for Leader reached at 2024-01-09 05:09:56.539227581 +0000 UTC m=+7.792278707 (1 nodes), 3.752µs from start of poll, 196.319325ms from cluster start. Timeout at 2024-01-09 05:09:56.644333904 +0000 UTC m=+7.897385032, 105.106325ms after stability testLoggerAdapter verbose: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.644Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: updating configuration: command=AddStaging server-id=server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f server-addr=cdb7c38b-a9bd-110e-c6fd-d856526f9d2f servers="[{Suffrage:Voter ID:server-97371e1f-22e0-76af-4a81-5afa5cfd8315 Address:97371e1f-22e0-76af-4a81-5afa5cfd8315} {Suffrage:Voter ID:server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69 Address:b1d2a2c9-ea13-4c34-4d91-d26b22449f69} {Suffrage:Voter ID:server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f Address:cdb7c38b-a9bd-110e-c6fd-d856526f9d2f}]" testing.go:149: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.644Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: updating configuration: command=AddStaging server-id=server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f server-addr=cdb7c38b-a9bd-110e-c6fd-d856526f9d2f servers="[{Suffrage:Voter ID:server-97371e1f-22e0-76af-4a81-5afa5cfd8315 Address:97371e1f-22e0-76af-4a81-5afa5cfd8315} {Suffrage:Voter ID:server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69 Address:b1d2a2c9-ea13-4c34-4d91-d26b22449f69} {Suffrage:Voter ID:server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f Address:cdb7c38b-a9bd-110e-c6fd-d856526f9d2f}]" testLoggerAdapter verbose: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.644Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: added peer, starting replication: peer=server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f testing.go:149: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.644Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: added peer, starting replication: peer=server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f testLoggerAdapter verbose: server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f: 2024-01-09T05:09:56.644Z [WARN] server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f: failed to get previous log: previous-index=4 last-index=0 error="log not found" testing.go:149: server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f: 2024-01-09T05:09:56.644Z [WARN] server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f: failed to get previous log: previous-index=4 last-index=0 error="log not found" testLoggerAdapter verbose: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.644Z [WARN] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: appendEntries rejected, sending older logs: peer="{Voter server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f cdb7c38b-a9bd-110e-c6fd-d856526f9d2f}" next=1 testing.go:149: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.644Z [WARN] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: appendEntries rejected, sending older logs: peer="{Voter server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f cdb7c38b-a9bd-110e-c6fd-d856526f9d2f}" next=1 testLoggerAdapter verbose: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.644Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: pipelining replication: peer="{Voter server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f cdb7c38b-a9bd-110e-c6fd-d856526f9d2f}" testing.go:149: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.644Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: pipelining replication: peer="{Voter server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f cdb7c38b-a9bd-110e-c6fd-d856526f9d2f}" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:56.645Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:09:56.645Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-01-09T05:09:56.755Z [INFO] stable state for Leader reached at 2024-01-09 05:09:56.645113715 +0000 UTC m=+7.898164838 (1 nodes), 2.552µs from start of poll, 302.205456ms from cluster start. Timeout at 2024-01-09 05:09:56.755051989 +0000 UTC m=+8.008102985, 109.938147ms after stability testing.go:149: cluster: 2024-01-09T05:09:56.755Z [INFO] stable state for Leader reached at 2024-01-09 05:09:56.645113715 +0000 UTC m=+7.898164838 (1 nodes), 2.552µs from start of poll, 302.205456ms from cluster start. Timeout at 2024-01-09 05:09:56.755051989 +0000 UTC m=+8.008102985, 109.938147ms after stability testLoggerAdapter verbose: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.755Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: aborting pipeline replication: peer="{Voter server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f cdb7c38b-a9bd-110e-c6fd-d856526f9d2f}" testing.go:149: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.755Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: aborting pipeline replication: peer="{Voter server-cdb7c38b-a9bd-110e-c6fd-d856526f9d2f cdb7c38b-a9bd-110e-c6fd-d856526f9d2f}" testLoggerAdapter verbose: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.755Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: aborting pipeline replication: peer="{Voter server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69 b1d2a2c9-ea13-4c34-4d91-d26b22449f69}" testing.go:149: server-97371e1f-22e0-76af-4a81-5afa5cfd8315: 2024-01-09T05:09:56.755Z [INFO] server-97371e1f-22e0-76af-4a81-5afa5cfd8315: aborting pipeline replication: peer="{Voter server-b1d2a2c9-ea13-4c34-4d91-d26b22449f69 b1d2a2c9-ea13-4c34-4d91-d26b22449f69}" --- PASS: TestRaft_JoinNode_ConfigStore (0.41s) === RUN TestRaft_RemoveFollower testLoggerAdapter verbose: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.757Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25c93f94-7b17-9861-cbac-b88b06a3a827 Address:25c93f94-7b17-9861-cbac-b88b06a3a827} {Suffrage:Voter ID:server-e591c761-60ba-b3d1-bba1-3608c0fde319 Address:e591c761-60ba-b3d1-bba1-3608c0fde319} {Suffrage:Voter ID:server-a3ddce4b-7390-67a1-253c-807365b0096c Address:a3ddce4b-7390-67a1-253c-807365b0096c}]" testing.go:149: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.757Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25c93f94-7b17-9861-cbac-b88b06a3a827 Address:25c93f94-7b17-9861-cbac-b88b06a3a827} {Suffrage:Voter ID:server-e591c761-60ba-b3d1-bba1-3608c0fde319 Address:e591c761-60ba-b3d1-bba1-3608c0fde319} {Suffrage:Voter ID:server-a3ddce4b-7390-67a1-253c-807365b0096c Address:a3ddce4b-7390-67a1-253c-807365b0096c}]" testLoggerAdapter verbose: server-e591c761-60ba-b3d1-bba1-3608c0fde319: 2024-01-09T05:09:56.757Z [INFO] server-e591c761-60ba-b3d1-bba1-3608c0fde319: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25c93f94-7b17-9861-cbac-b88b06a3a827 Address:25c93f94-7b17-9861-cbac-b88b06a3a827} {Suffrage:Voter ID:server-e591c761-60ba-b3d1-bba1-3608c0fde319 Address:e591c761-60ba-b3d1-bba1-3608c0fde319} {Suffrage:Voter ID:server-a3ddce4b-7390-67a1-253c-807365b0096c Address:a3ddce4b-7390-67a1-253c-807365b0096c}]" testing.go:149: server-e591c761-60ba-b3d1-bba1-3608c0fde319: 2024-01-09T05:09:56.757Z [INFO] server-e591c761-60ba-b3d1-bba1-3608c0fde319: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25c93f94-7b17-9861-cbac-b88b06a3a827 Address:25c93f94-7b17-9861-cbac-b88b06a3a827} {Suffrage:Voter ID:server-e591c761-60ba-b3d1-bba1-3608c0fde319 Address:e591c761-60ba-b3d1-bba1-3608c0fde319} {Suffrage:Voter ID:server-a3ddce4b-7390-67a1-253c-807365b0096c Address:a3ddce4b-7390-67a1-253c-807365b0096c}]" testLoggerAdapter verbose: server-a3ddce4b-7390-67a1-253c-807365b0096c: 2024-01-09T05:09:56.757Z [INFO] server-a3ddce4b-7390-67a1-253c-807365b0096c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25c93f94-7b17-9861-cbac-b88b06a3a827 Address:25c93f94-7b17-9861-cbac-b88b06a3a827} {Suffrage:Voter ID:server-e591c761-60ba-b3d1-bba1-3608c0fde319 Address:e591c761-60ba-b3d1-bba1-3608c0fde319} {Suffrage:Voter ID:server-a3ddce4b-7390-67a1-253c-807365b0096c Address:a3ddce4b-7390-67a1-253c-807365b0096c}]" testing.go:149: server-a3ddce4b-7390-67a1-253c-807365b0096c: 2024-01-09T05:09:56.757Z [INFO] server-a3ddce4b-7390-67a1-253c-807365b0096c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25c93f94-7b17-9861-cbac-b88b06a3a827 Address:25c93f94-7b17-9861-cbac-b88b06a3a827} {Suffrage:Voter ID:server-e591c761-60ba-b3d1-bba1-3608c0fde319 Address:e591c761-60ba-b3d1-bba1-3608c0fde319} {Suffrage:Voter ID:server-a3ddce4b-7390-67a1-253c-807365b0096c Address:a3ddce4b-7390-67a1-253c-807365b0096c}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:56.757Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:09:56.757Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.757Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: entering follower state: follower="Node at 25c93f94-7b17-9861-cbac-b88b06a3a827 [Follower]" leader= testing.go:149: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.757Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: entering follower state: follower="Node at 25c93f94-7b17-9861-cbac-b88b06a3a827 [Follower]" leader= testLoggerAdapter verbose: server-e591c761-60ba-b3d1-bba1-3608c0fde319: 2024-01-09T05:09:56.757Z [INFO] server-e591c761-60ba-b3d1-bba1-3608c0fde319: entering follower state: follower="Node at e591c761-60ba-b3d1-bba1-3608c0fde319 [Follower]" leader= testing.go:149: server-e591c761-60ba-b3d1-bba1-3608c0fde319: 2024-01-09T05:09:56.757Z [INFO] server-e591c761-60ba-b3d1-bba1-3608c0fde319: entering follower state: follower="Node at e591c761-60ba-b3d1-bba1-3608c0fde319 [Follower]" leader= testLoggerAdapter verbose: server-a3ddce4b-7390-67a1-253c-807365b0096c: 2024-01-09T05:09:56.757Z [INFO] server-a3ddce4b-7390-67a1-253c-807365b0096c: entering follower state: follower="Node at a3ddce4b-7390-67a1-253c-807365b0096c [Follower]" leader= testing.go:149: server-a3ddce4b-7390-67a1-253c-807365b0096c: 2024-01-09T05:09:56.757Z [INFO] server-a3ddce4b-7390-67a1-253c-807365b0096c: entering follower state: follower="Node at a3ddce4b-7390-67a1-253c-807365b0096c [Follower]" leader= testLoggerAdapter verbose: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.813Z [WARN] server-25c93f94-7b17-9861-cbac-b88b06a3a827: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.813Z [WARN] server-25c93f94-7b17-9861-cbac-b88b06a3a827: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.813Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: entering candidate state: node="Node at 25c93f94-7b17-9861-cbac-b88b06a3a827 [Candidate]" term=2 testing.go:149: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.813Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: entering candidate state: node="Node at 25c93f94-7b17-9861-cbac-b88b06a3a827 [Candidate]" term=2 testLoggerAdapter verbose: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.813Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: election won: tally=2 testing.go:149: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.813Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: election won: tally=2 testLoggerAdapter verbose: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.813Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: entering leader state: leader="Node at 25c93f94-7b17-9861-cbac-b88b06a3a827 [Leader]" testing.go:149: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.813Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: entering leader state: leader="Node at 25c93f94-7b17-9861-cbac-b88b06a3a827 [Leader]" testLoggerAdapter verbose: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.813Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: added peer, starting replication: peer=server-e591c761-60ba-b3d1-bba1-3608c0fde319 testing.go:149: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.813Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: added peer, starting replication: peer=server-e591c761-60ba-b3d1-bba1-3608c0fde319 testLoggerAdapter verbose: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.813Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: added peer, starting replication: peer=server-a3ddce4b-7390-67a1-253c-807365b0096c testing.go:149: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.813Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: added peer, starting replication: peer=server-a3ddce4b-7390-67a1-253c-807365b0096c testLoggerAdapter verbose: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.813Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: pipelining replication: peer="{Voter server-a3ddce4b-7390-67a1-253c-807365b0096c a3ddce4b-7390-67a1-253c-807365b0096c}" testing.go:149: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.813Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: pipelining replication: peer="{Voter server-a3ddce4b-7390-67a1-253c-807365b0096c a3ddce4b-7390-67a1-253c-807365b0096c}" testLoggerAdapter verbose: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.813Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: pipelining replication: peer="{Voter server-e591c761-60ba-b3d1-bba1-3608c0fde319 e591c761-60ba-b3d1-bba1-3608c0fde319}" testing.go:149: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:56.813Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: pipelining replication: peer="{Voter server-e591c761-60ba-b3d1-bba1-3608c0fde319 e591c761-60ba-b3d1-bba1-3608c0fde319}" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:56.919Z [INFO] stable state for Leader reached at 2024-01-09 05:09:56.814098 +0000 UTC m=+8.067149020 (1 nodes), 56.63438ms from start of poll, 57.226071ms from cluster start. Timeout at 2024-01-09 05:09:56.919122922 +0000 UTC m=+8.172174016, 105.024996ms after stability testing.go:149: cluster: 2024-01-09T05:09:56.919Z [INFO] stable state for Leader reached at 2024-01-09 05:09:56.814098 +0000 UTC m=+8.067149020 (1 nodes), 56.63438ms from start of poll, 57.226071ms from cluster start. Timeout at 2024-01-09 05:09:56.919122922 +0000 UTC m=+8.172174016, 105.024996ms after stability testLoggerAdapter verbose: cluster: 2024-01-09T05:09:56.924Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-01-09T05:09:56.924Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2024-01-09T05:09:57.035Z [INFO] stable state for Follower reached at 2024-01-09 05:09:56.924618469 +0000 UTC m=+8.177669469 (2 nodes), 2.96µs from start of poll, 167.74652ms from cluster start. Timeout at 2024-01-09 05:09:57.035117491 +0000 UTC m=+8.288168508, 110.499039ms after stability testing.go:149: cluster: 2024-01-09T05:09:57.035Z [INFO] stable state for Follower reached at 2024-01-09 05:09:56.924618469 +0000 UTC m=+8.177669469 (2 nodes), 2.96µs from start of poll, 167.74652ms from cluster start. Timeout at 2024-01-09 05:09:57.035117491 +0000 UTC m=+8.288168508, 110.499039ms after stability testLoggerAdapter verbose: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:57.035Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: updating configuration: command=RemoveServer server-id=server-e591c761-60ba-b3d1-bba1-3608c0fde319 server-addr= servers="[{Suffrage:Voter ID:server-25c93f94-7b17-9861-cbac-b88b06a3a827 Address:25c93f94-7b17-9861-cbac-b88b06a3a827} {Suffrage:Voter ID:server-a3ddce4b-7390-67a1-253c-807365b0096c Address:a3ddce4b-7390-67a1-253c-807365b0096c}]" testing.go:149: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:57.035Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: updating configuration: command=RemoveServer server-id=server-e591c761-60ba-b3d1-bba1-3608c0fde319 server-addr= servers="[{Suffrage:Voter ID:server-25c93f94-7b17-9861-cbac-b88b06a3a827 Address:25c93f94-7b17-9861-cbac-b88b06a3a827} {Suffrage:Voter ID:server-a3ddce4b-7390-67a1-253c-807365b0096c Address:a3ddce4b-7390-67a1-253c-807365b0096c}]" testLoggerAdapter verbose: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:57.035Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: removed peer, stopping replication: peer=server-e591c761-60ba-b3d1-bba1-3608c0fde319 last-index=3 testing.go:149: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:57.035Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: removed peer, stopping replication: peer=server-e591c761-60ba-b3d1-bba1-3608c0fde319 last-index=3 testLoggerAdapter verbose: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:57.035Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: aborting pipeline replication: peer="{Voter server-e591c761-60ba-b3d1-bba1-3608c0fde319 e591c761-60ba-b3d1-bba1-3608c0fde319}" testing.go:149: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:57.035Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: aborting pipeline replication: peer="{Voter server-e591c761-60ba-b3d1-bba1-3608c0fde319 e591c761-60ba-b3d1-bba1-3608c0fde319}" testLoggerAdapter verbose: server-e591c761-60ba-b3d1-bba1-3608c0fde319: 2024-01-09T05:09:57.128Z [WARN] server-e591c761-60ba-b3d1-bba1-3608c0fde319: heartbeat timeout reached, starting election: last-leader=25c93f94-7b17-9861-cbac-b88b06a3a827 testing.go:149: server-e591c761-60ba-b3d1-bba1-3608c0fde319: 2024-01-09T05:09:57.128Z [WARN] server-e591c761-60ba-b3d1-bba1-3608c0fde319: heartbeat timeout reached, starting election: last-leader=25c93f94-7b17-9861-cbac-b88b06a3a827 testLoggerAdapter verbose: server-e591c761-60ba-b3d1-bba1-3608c0fde319: 2024-01-09T05:09:57.128Z [INFO] server-e591c761-60ba-b3d1-bba1-3608c0fde319: entering candidate state: node="Node at e591c761-60ba-b3d1-bba1-3608c0fde319 [Candidate]" term=3 testing.go:149: server-e591c761-60ba-b3d1-bba1-3608c0fde319: 2024-01-09T05:09:57.128Z [INFO] server-e591c761-60ba-b3d1-bba1-3608c0fde319: entering candidate state: node="Node at e591c761-60ba-b3d1-bba1-3608c0fde319 [Candidate]" term=3 testLoggerAdapter verbose: server-a3ddce4b-7390-67a1-253c-807365b0096c: 2024-01-09T05:09:57.128Z [WARN] server-a3ddce4b-7390-67a1-253c-807365b0096c: rejecting vote request since we have a leader: from=e591c761-60ba-b3d1-bba1-3608c0fde319 leader=25c93f94-7b17-9861-cbac-b88b06a3a827 testing.go:149: server-a3ddce4b-7390-67a1-253c-807365b0096c: 2024-01-09T05:09:57.128Z [WARN] server-a3ddce4b-7390-67a1-253c-807365b0096c: rejecting vote request since we have a leader: from=e591c761-60ba-b3d1-bba1-3608c0fde319 leader=25c93f94-7b17-9861-cbac-b88b06a3a827 testLoggerAdapter verbose: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:57.128Z [WARN] server-25c93f94-7b17-9861-cbac-b88b06a3a827: rejecting vote request since we have a leader: from=e591c761-60ba-b3d1-bba1-3608c0fde319 leader=25c93f94-7b17-9861-cbac-b88b06a3a827 testing.go:149: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:57.128Z [WARN] server-25c93f94-7b17-9861-cbac-b88b06a3a827: rejecting vote request since we have a leader: from=e591c761-60ba-b3d1-bba1-3608c0fde319 leader=25c93f94-7b17-9861-cbac-b88b06a3a827 testLoggerAdapter verbose: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:57.147Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: aborting pipeline replication: peer="{Voter server-a3ddce4b-7390-67a1-253c-807365b0096c a3ddce4b-7390-67a1-253c-807365b0096c}" testing.go:149: server-25c93f94-7b17-9861-cbac-b88b06a3a827: 2024-01-09T05:09:57.147Z [INFO] server-25c93f94-7b17-9861-cbac-b88b06a3a827: aborting pipeline replication: peer="{Voter server-a3ddce4b-7390-67a1-253c-807365b0096c a3ddce4b-7390-67a1-253c-807365b0096c}" --- PASS: TestRaft_RemoveFollower (0.39s) === RUN TestRaft_RemoveLeader testLoggerAdapter verbose: server-c8a06ca4-2690-98b7-5baa-63759e639581: 2024-01-09T05:09:57.148Z [INFO] server-c8a06ca4-2690-98b7-5baa-63759e639581: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c8a06ca4-2690-98b7-5baa-63759e639581 Address:c8a06ca4-2690-98b7-5baa-63759e639581} {Suffrage:Voter ID:server-41223f4a-7243-9254-7581-ddae91d497b1 Address:41223f4a-7243-9254-7581-ddae91d497b1} {Suffrage:Voter ID:server-2c7c7829-a541-c68f-c891-302be9f19edd Address:2c7c7829-a541-c68f-c891-302be9f19edd}]" testing.go:149: server-c8a06ca4-2690-98b7-5baa-63759e639581: 2024-01-09T05:09:57.148Z [INFO] server-c8a06ca4-2690-98b7-5baa-63759e639581: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c8a06ca4-2690-98b7-5baa-63759e639581 Address:c8a06ca4-2690-98b7-5baa-63759e639581} {Suffrage:Voter ID:server-41223f4a-7243-9254-7581-ddae91d497b1 Address:41223f4a-7243-9254-7581-ddae91d497b1} {Suffrage:Voter ID:server-2c7c7829-a541-c68f-c891-302be9f19edd Address:2c7c7829-a541-c68f-c891-302be9f19edd}]" testLoggerAdapter verbose: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.148Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c8a06ca4-2690-98b7-5baa-63759e639581 Address:c8a06ca4-2690-98b7-5baa-63759e639581} {Suffrage:Voter ID:server-41223f4a-7243-9254-7581-ddae91d497b1 Address:41223f4a-7243-9254-7581-ddae91d497b1} {Suffrage:Voter ID:server-2c7c7829-a541-c68f-c891-302be9f19edd Address:2c7c7829-a541-c68f-c891-302be9f19edd}]" testing.go:149: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.148Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c8a06ca4-2690-98b7-5baa-63759e639581 Address:c8a06ca4-2690-98b7-5baa-63759e639581} {Suffrage:Voter ID:server-41223f4a-7243-9254-7581-ddae91d497b1 Address:41223f4a-7243-9254-7581-ddae91d497b1} {Suffrage:Voter ID:server-2c7c7829-a541-c68f-c891-302be9f19edd Address:2c7c7829-a541-c68f-c891-302be9f19edd}]" testLoggerAdapter verbose: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.149Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c8a06ca4-2690-98b7-5baa-63759e639581 Address:c8a06ca4-2690-98b7-5baa-63759e639581} {Suffrage:Voter ID:server-41223f4a-7243-9254-7581-ddae91d497b1 Address:41223f4a-7243-9254-7581-ddae91d497b1} {Suffrage:Voter ID:server-2c7c7829-a541-c68f-c891-302be9f19edd Address:2c7c7829-a541-c68f-c891-302be9f19edd}]" testing.go:149: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.149Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c8a06ca4-2690-98b7-5baa-63759e639581 Address:c8a06ca4-2690-98b7-5baa-63759e639581} {Suffrage:Voter ID:server-41223f4a-7243-9254-7581-ddae91d497b1 Address:41223f4a-7243-9254-7581-ddae91d497b1} {Suffrage:Voter ID:server-2c7c7829-a541-c68f-c891-302be9f19edd Address:2c7c7829-a541-c68f-c891-302be9f19edd}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:57.149Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:09:57.149Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-c8a06ca4-2690-98b7-5baa-63759e639581: 2024-01-09T05:09:57.149Z [INFO] server-c8a06ca4-2690-98b7-5baa-63759e639581: entering follower state: follower="Node at c8a06ca4-2690-98b7-5baa-63759e639581 [Follower]" leader= testing.go:149: server-c8a06ca4-2690-98b7-5baa-63759e639581: 2024-01-09T05:09:57.149Z [INFO] server-c8a06ca4-2690-98b7-5baa-63759e639581: entering follower state: follower="Node at c8a06ca4-2690-98b7-5baa-63759e639581 [Follower]" leader= testLoggerAdapter verbose: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.149Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: entering follower state: follower="Node at 41223f4a-7243-9254-7581-ddae91d497b1 [Follower]" leader= testing.go:149: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.149Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: entering follower state: follower="Node at 41223f4a-7243-9254-7581-ddae91d497b1 [Follower]" leader= testLoggerAdapter verbose: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.149Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: entering follower state: follower="Node at 2c7c7829-a541-c68f-c891-302be9f19edd [Follower]" leader= testing.go:149: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.149Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: entering follower state: follower="Node at 2c7c7829-a541-c68f-c891-302be9f19edd [Follower]" leader= testLoggerAdapter verbose: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.223Z [WARN] server-2c7c7829-a541-c68f-c891-302be9f19edd: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.223Z [WARN] server-2c7c7829-a541-c68f-c891-302be9f19edd: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.223Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: entering candidate state: node="Node at 2c7c7829-a541-c68f-c891-302be9f19edd [Candidate]" term=2 testing.go:149: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.223Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: entering candidate state: node="Node at 2c7c7829-a541-c68f-c891-302be9f19edd [Candidate]" term=2 testLoggerAdapter verbose: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.223Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: election won: tally=2 testing.go:149: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.223Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: election won: tally=2 testLoggerAdapter verbose: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.223Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: entering leader state: leader="Node at 2c7c7829-a541-c68f-c891-302be9f19edd [Leader]" testing.go:149: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.223Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: entering leader state: leader="Node at 2c7c7829-a541-c68f-c891-302be9f19edd [Leader]" testLoggerAdapter verbose: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.223Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: added peer, starting replication: peer=server-c8a06ca4-2690-98b7-5baa-63759e639581 testing.go:149: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.223Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: added peer, starting replication: peer=server-c8a06ca4-2690-98b7-5baa-63759e639581 testLoggerAdapter verbose: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.223Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: added peer, starting replication: peer=server-41223f4a-7243-9254-7581-ddae91d497b1 testing.go:149: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.223Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: added peer, starting replication: peer=server-41223f4a-7243-9254-7581-ddae91d497b1 testLoggerAdapter verbose: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.223Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: pipelining replication: peer="{Voter server-41223f4a-7243-9254-7581-ddae91d497b1 41223f4a-7243-9254-7581-ddae91d497b1}" testing.go:149: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.223Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: pipelining replication: peer="{Voter server-41223f4a-7243-9254-7581-ddae91d497b1 41223f4a-7243-9254-7581-ddae91d497b1}" testLoggerAdapter verbose: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.223Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: pipelining replication: peer="{Voter server-c8a06ca4-2690-98b7-5baa-63759e639581 c8a06ca4-2690-98b7-5baa-63759e639581}" testing.go:149: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.223Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: pipelining replication: peer="{Voter server-c8a06ca4-2690-98b7-5baa-63759e639581 c8a06ca4-2690-98b7-5baa-63759e639581}" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:57.328Z [INFO] stable state for Leader reached at 2024-01-09 05:09:57.223803261 +0000 UTC m=+8.476854291 (1 nodes), 74.708753ms from start of poll, 75.185186ms from cluster start. Timeout at 2024-01-09 05:09:57.328909408 +0000 UTC m=+8.581960556, 105.106265ms after stability testing.go:149: cluster: 2024-01-09T05:09:57.328Z [INFO] stable state for Leader reached at 2024-01-09 05:09:57.223803261 +0000 UTC m=+8.476854291 (1 nodes), 74.708753ms from start of poll, 75.185186ms from cluster start. Timeout at 2024-01-09 05:09:57.328909408 +0000 UTC m=+8.581960556, 105.106265ms after stability testLoggerAdapter verbose: cluster: 2024-01-09T05:09:57.334Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-01-09T05:09:57.334Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2024-01-09T05:09:57.439Z [INFO] stable state for Follower reached at 2024-01-09 05:09:57.334251502 +0000 UTC m=+8.587302618 (2 nodes), 2.876µs from start of poll, 185.633513ms from cluster start. Timeout at 2024-01-09 05:09:57.439428118 +0000 UTC m=+8.692479116, 105.176498ms after stability testing.go:149: cluster: 2024-01-09T05:09:57.439Z [INFO] stable state for Follower reached at 2024-01-09 05:09:57.334251502 +0000 UTC m=+8.587302618 (2 nodes), 2.876µs from start of poll, 185.633513ms from cluster start. Timeout at 2024-01-09 05:09:57.439428118 +0000 UTC m=+8.692479116, 105.176498ms after stability testLoggerAdapter verbose: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.439Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: updating configuration: command=RemoveServer server-id=server-2c7c7829-a541-c68f-c891-302be9f19edd server-addr= servers="[{Suffrage:Voter ID:server-c8a06ca4-2690-98b7-5baa-63759e639581 Address:c8a06ca4-2690-98b7-5baa-63759e639581} {Suffrage:Voter ID:server-41223f4a-7243-9254-7581-ddae91d497b1 Address:41223f4a-7243-9254-7581-ddae91d497b1}]" testing.go:149: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.439Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: updating configuration: command=RemoveServer server-id=server-2c7c7829-a541-c68f-c891-302be9f19edd server-addr= servers="[{Suffrage:Voter ID:server-c8a06ca4-2690-98b7-5baa-63759e639581 Address:c8a06ca4-2690-98b7-5baa-63759e639581} {Suffrage:Voter ID:server-41223f4a-7243-9254-7581-ddae91d497b1 Address:41223f4a-7243-9254-7581-ddae91d497b1}]" testLoggerAdapter verbose: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.439Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: removed ourself, shutting down testing.go:149: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.439Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: removed ourself, shutting down testLoggerAdapter verbose: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.439Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: aborting pipeline replication: peer="{Voter server-41223f4a-7243-9254-7581-ddae91d497b1 41223f4a-7243-9254-7581-ddae91d497b1}" testing.go:149: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.439Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: aborting pipeline replication: peer="{Voter server-41223f4a-7243-9254-7581-ddae91d497b1 41223f4a-7243-9254-7581-ddae91d497b1}" testLoggerAdapter verbose: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.439Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: aborting pipeline replication: peer="{Voter server-c8a06ca4-2690-98b7-5baa-63759e639581 c8a06ca4-2690-98b7-5baa-63759e639581}" testing.go:149: server-2c7c7829-a541-c68f-c891-302be9f19edd: 2024-01-09T05:09:57.439Z [INFO] server-2c7c7829-a541-c68f-c891-302be9f19edd: aborting pipeline replication: peer="{Voter server-c8a06ca4-2690-98b7-5baa-63759e639581 c8a06ca4-2690-98b7-5baa-63759e639581}" testLoggerAdapter verbose: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.519Z [WARN] server-41223f4a-7243-9254-7581-ddae91d497b1: heartbeat timeout reached, starting election: last-leader=2c7c7829-a541-c68f-c891-302be9f19edd testing.go:149: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.519Z [WARN] server-41223f4a-7243-9254-7581-ddae91d497b1: heartbeat timeout reached, starting election: last-leader=2c7c7829-a541-c68f-c891-302be9f19edd testLoggerAdapter verbose: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.519Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: entering candidate state: node="Node at 41223f4a-7243-9254-7581-ddae91d497b1 [Candidate]" term=3 testing.go:149: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.519Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: entering candidate state: node="Node at 41223f4a-7243-9254-7581-ddae91d497b1 [Candidate]" term=3 testLoggerAdapter verbose: server-c8a06ca4-2690-98b7-5baa-63759e639581: 2024-01-09T05:09:57.519Z [WARN] server-c8a06ca4-2690-98b7-5baa-63759e639581: rejecting vote request since we have a leader: from=41223f4a-7243-9254-7581-ddae91d497b1 leader=2c7c7829-a541-c68f-c891-302be9f19edd testing.go:149: server-c8a06ca4-2690-98b7-5baa-63759e639581: 2024-01-09T05:09:57.519Z [WARN] server-c8a06ca4-2690-98b7-5baa-63759e639581: rejecting vote request since we have a leader: from=41223f4a-7243-9254-7581-ddae91d497b1 leader=2c7c7829-a541-c68f-c891-302be9f19edd testLoggerAdapter verbose: server-c8a06ca4-2690-98b7-5baa-63759e639581: 2024-01-09T05:09:57.543Z [WARN] server-c8a06ca4-2690-98b7-5baa-63759e639581: heartbeat timeout reached, starting election: last-leader=2c7c7829-a541-c68f-c891-302be9f19edd testing.go:149: server-c8a06ca4-2690-98b7-5baa-63759e639581: 2024-01-09T05:09:57.543Z [WARN] server-c8a06ca4-2690-98b7-5baa-63759e639581: heartbeat timeout reached, starting election: last-leader=2c7c7829-a541-c68f-c891-302be9f19edd testLoggerAdapter verbose: server-c8a06ca4-2690-98b7-5baa-63759e639581: 2024-01-09T05:09:57.543Z [INFO] server-c8a06ca4-2690-98b7-5baa-63759e639581: entering candidate state: node="Node at c8a06ca4-2690-98b7-5baa-63759e639581 [Candidate]" term=3 testing.go:149: server-c8a06ca4-2690-98b7-5baa-63759e639581: 2024-01-09T05:09:57.543Z [INFO] server-c8a06ca4-2690-98b7-5baa-63759e639581: entering candidate state: node="Node at c8a06ca4-2690-98b7-5baa-63759e639581 [Candidate]" term=3 testLoggerAdapter verbose: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.543Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: duplicate requestVote for same term: term=3 testing.go:149: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.543Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: duplicate requestVote for same term: term=3 testLoggerAdapter verbose: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.594Z [WARN] server-41223f4a-7243-9254-7581-ddae91d497b1: Election timeout reached, restarting election testing.go:149: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.594Z [WARN] server-41223f4a-7243-9254-7581-ddae91d497b1: Election timeout reached, restarting election testLoggerAdapter verbose: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.595Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: entering candidate state: node="Node at 41223f4a-7243-9254-7581-ddae91d497b1 [Candidate]" term=4 testing.go:149: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.595Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: entering candidate state: node="Node at 41223f4a-7243-9254-7581-ddae91d497b1 [Candidate]" term=4 testLoggerAdapter verbose: server-c8a06ca4-2690-98b7-5baa-63759e639581: 2024-01-09T05:09:57.595Z [INFO] server-c8a06ca4-2690-98b7-5baa-63759e639581: entering follower state: follower="Node at c8a06ca4-2690-98b7-5baa-63759e639581 [Follower]" leader= testing.go:149: server-c8a06ca4-2690-98b7-5baa-63759e639581: 2024-01-09T05:09:57.595Z [INFO] server-c8a06ca4-2690-98b7-5baa-63759e639581: entering follower state: follower="Node at c8a06ca4-2690-98b7-5baa-63759e639581 [Follower]" leader= testLoggerAdapter verbose: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.595Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: election won: tally=2 testing.go:149: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.595Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: election won: tally=2 testLoggerAdapter verbose: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.595Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: entering leader state: leader="Node at 41223f4a-7243-9254-7581-ddae91d497b1 [Leader]" testing.go:149: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.595Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: entering leader state: leader="Node at 41223f4a-7243-9254-7581-ddae91d497b1 [Leader]" testLoggerAdapter verbose: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.595Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: added peer, starting replication: peer=server-c8a06ca4-2690-98b7-5baa-63759e639581 testing.go:149: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.595Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: added peer, starting replication: peer=server-c8a06ca4-2690-98b7-5baa-63759e639581 testLoggerAdapter verbose: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.595Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: pipelining replication: peer="{Voter server-c8a06ca4-2690-98b7-5baa-63759e639581 c8a06ca4-2690-98b7-5baa-63759e639581}" testing.go:149: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.595Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: pipelining replication: peer="{Voter server-c8a06ca4-2690-98b7-5baa-63759e639581 c8a06ca4-2690-98b7-5baa-63759e639581}" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:57.676Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:09:57.676Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-01-09T05:09:57.781Z [INFO] stable state for Leader reached at 2024-01-09 05:09:57.676182452 +0000 UTC m=+8.929233462 (1 nodes), 54.813µs from start of poll, 527.564357ms from cluster start. Timeout at 2024-01-09 05:09:57.781301951 +0000 UTC m=+9.034353068, 105.119606ms after stability testing.go:149: cluster: 2024-01-09T05:09:57.781Z [INFO] stable state for Leader reached at 2024-01-09 05:09:57.676182452 +0000 UTC m=+8.929233462 (1 nodes), 54.813µs from start of poll, 527.564357ms from cluster start. Timeout at 2024-01-09 05:09:57.781301951 +0000 UTC m=+9.034353068, 105.119606ms after stability testLoggerAdapter verbose: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.781Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: aborting pipeline replication: peer="{Voter server-c8a06ca4-2690-98b7-5baa-63759e639581 c8a06ca4-2690-98b7-5baa-63759e639581}" testing.go:149: server-41223f4a-7243-9254-7581-ddae91d497b1: 2024-01-09T05:09:57.781Z [INFO] server-41223f4a-7243-9254-7581-ddae91d497b1: aborting pipeline replication: peer="{Voter server-c8a06ca4-2690-98b7-5baa-63759e639581 c8a06ca4-2690-98b7-5baa-63759e639581}" --- PASS: TestRaft_RemoveLeader (0.63s) === RUN TestRaft_RemoveLeader_NoShutdown testLoggerAdapter verbose: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:57.783Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd Address:9a5bce64-da87-eed0-1c8c-d753d3ab43bd} {Suffrage:Voter ID:server-e8bf385c-490a-2395-38fd-ba0d812e9332 Address:e8bf385c-490a-2395-38fd-ba0d812e9332} {Suffrage:Voter ID:server-637e3d76-fedb-7ba5-2cc2-895de972afce Address:637e3d76-fedb-7ba5-2cc2-895de972afce}]" testing.go:149: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:57.783Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd Address:9a5bce64-da87-eed0-1c8c-d753d3ab43bd} {Suffrage:Voter ID:server-e8bf385c-490a-2395-38fd-ba0d812e9332 Address:e8bf385c-490a-2395-38fd-ba0d812e9332} {Suffrage:Voter ID:server-637e3d76-fedb-7ba5-2cc2-895de972afce Address:637e3d76-fedb-7ba5-2cc2-895de972afce}]" testLoggerAdapter verbose: server-e8bf385c-490a-2395-38fd-ba0d812e9332: 2024-01-09T05:09:57.783Z [INFO] server-e8bf385c-490a-2395-38fd-ba0d812e9332: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd Address:9a5bce64-da87-eed0-1c8c-d753d3ab43bd} {Suffrage:Voter ID:server-e8bf385c-490a-2395-38fd-ba0d812e9332 Address:e8bf385c-490a-2395-38fd-ba0d812e9332} {Suffrage:Voter ID:server-637e3d76-fedb-7ba5-2cc2-895de972afce Address:637e3d76-fedb-7ba5-2cc2-895de972afce}]" testing.go:149: server-e8bf385c-490a-2395-38fd-ba0d812e9332: 2024-01-09T05:09:57.783Z [INFO] server-e8bf385c-490a-2395-38fd-ba0d812e9332: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd Address:9a5bce64-da87-eed0-1c8c-d753d3ab43bd} {Suffrage:Voter ID:server-e8bf385c-490a-2395-38fd-ba0d812e9332 Address:e8bf385c-490a-2395-38fd-ba0d812e9332} {Suffrage:Voter ID:server-637e3d76-fedb-7ba5-2cc2-895de972afce Address:637e3d76-fedb-7ba5-2cc2-895de972afce}]" testLoggerAdapter verbose: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.783Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd Address:9a5bce64-da87-eed0-1c8c-d753d3ab43bd} {Suffrage:Voter ID:server-e8bf385c-490a-2395-38fd-ba0d812e9332 Address:e8bf385c-490a-2395-38fd-ba0d812e9332} {Suffrage:Voter ID:server-637e3d76-fedb-7ba5-2cc2-895de972afce Address:637e3d76-fedb-7ba5-2cc2-895de972afce}]" testing.go:149: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.783Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd Address:9a5bce64-da87-eed0-1c8c-d753d3ab43bd} {Suffrage:Voter ID:server-e8bf385c-490a-2395-38fd-ba0d812e9332 Address:e8bf385c-490a-2395-38fd-ba0d812e9332} {Suffrage:Voter ID:server-637e3d76-fedb-7ba5-2cc2-895de972afce Address:637e3d76-fedb-7ba5-2cc2-895de972afce}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:57.783Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-01-09T05:09:57.783Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:57.783Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: entering follower state: follower="Node at 9a5bce64-da87-eed0-1c8c-d753d3ab43bd [Follower]" leader= testing.go:149: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:57.783Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: entering follower state: follower="Node at 9a5bce64-da87-eed0-1c8c-d753d3ab43bd [Follower]" leader= testLoggerAdapter verbose: server-e8bf385c-490a-2395-38fd-ba0d812e9332: 2024-01-09T05:09:57.783Z [INFO] server-e8bf385c-490a-2395-38fd-ba0d812e9332: entering follower state: follower="Node at e8bf385c-490a-2395-38fd-ba0d812e9332 [Follower]" leader= testing.go:149: server-e8bf385c-490a-2395-38fd-ba0d812e9332: 2024-01-09T05:09:57.783Z [INFO] server-e8bf385c-490a-2395-38fd-ba0d812e9332: entering follower state: follower="Node at e8bf385c-490a-2395-38fd-ba0d812e9332 [Follower]" leader= testLoggerAdapter verbose: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.783Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: entering follower state: follower="Node at 637e3d76-fedb-7ba5-2cc2-895de972afce [Follower]" leader= testing.go:149: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.783Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: entering follower state: follower="Node at 637e3d76-fedb-7ba5-2cc2-895de972afce [Follower]" leader= testLoggerAdapter verbose: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.848Z [WARN] server-637e3d76-fedb-7ba5-2cc2-895de972afce: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.848Z [WARN] server-637e3d76-fedb-7ba5-2cc2-895de972afce: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.848Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: entering candidate state: node="Node at 637e3d76-fedb-7ba5-2cc2-895de972afce [Candidate]" term=2 testing.go:149: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.848Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: entering candidate state: node="Node at 637e3d76-fedb-7ba5-2cc2-895de972afce [Candidate]" term=2 testLoggerAdapter verbose: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.848Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: election won: tally=2 testing.go:149: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.848Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: election won: tally=2 testLoggerAdapter verbose: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.849Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: entering leader state: leader="Node at 637e3d76-fedb-7ba5-2cc2-895de972afce [Leader]" testing.go:149: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.849Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: entering leader state: leader="Node at 637e3d76-fedb-7ba5-2cc2-895de972afce [Leader]" testLoggerAdapter verbose: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.849Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: added peer, starting replication: peer=server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd testing.go:149: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.849Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: added peer, starting replication: peer=server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd testLoggerAdapter verbose: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.849Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: added peer, starting replication: peer=server-e8bf385c-490a-2395-38fd-ba0d812e9332 testing.go:149: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.849Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: added peer, starting replication: peer=server-e8bf385c-490a-2395-38fd-ba0d812e9332 testLoggerAdapter verbose: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.849Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: pipelining replication: peer="{Voter server-e8bf385c-490a-2395-38fd-ba0d812e9332 e8bf385c-490a-2395-38fd-ba0d812e9332}" testing.go:149: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.849Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: pipelining replication: peer="{Voter server-e8bf385c-490a-2395-38fd-ba0d812e9332 e8bf385c-490a-2395-38fd-ba0d812e9332}" testLoggerAdapter verbose: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.849Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: pipelining replication: peer="{Voter server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd 9a5bce64-da87-eed0-1c8c-d753d3ab43bd}" testing.go:149: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:57.849Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: pipelining replication: peer="{Voter server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd 9a5bce64-da87-eed0-1c8c-d753d3ab43bd}" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:57.954Z [INFO] stable state for Follower reached at 2024-01-09 05:09:57.849456457 +0000 UTC m=+9.102507588 (2 nodes), 65.83085ms from start of poll, 66.473384ms from cluster start. Timeout at 2024-01-09 05:09:57.95455978 +0000 UTC m=+9.207610974, 105.103386ms after stability testing.go:149: cluster: 2024-01-09T05:09:57.954Z [INFO] stable state for Follower reached at 2024-01-09 05:09:57.849456457 +0000 UTC m=+9.102507588 (2 nodes), 65.83085ms from start of poll, 66.473384ms from cluster start. Timeout at 2024-01-09 05:09:57.95455978 +0000 UTC m=+9.207610974, 105.103386ms after stability testLoggerAdapter verbose: cluster: 2024-01-09T05:09:57.954Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:09:57.954Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-01-09T05:09:58.059Z [INFO] stable state for Leader reached at 2024-01-09 05:09:57.954774353 +0000 UTC m=+9.207825470 (1 nodes), 2.417µs from start of poll, 171.791266ms from cluster start. Timeout at 2024-01-09 05:09:58.0598681 +0000 UTC m=+9.312919240, 105.09377ms after stability testing.go:149: cluster: 2024-01-09T05:09:58.059Z [INFO] stable state for Leader reached at 2024-01-09 05:09:57.954774353 +0000 UTC m=+9.207825470 (1 nodes), 2.417µs from start of poll, 171.791266ms from cluster start. Timeout at 2024-01-09 05:09:58.0598681 +0000 UTC m=+9.312919240, 105.09377ms after stability testLoggerAdapter verbose: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:58.066Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: updating configuration: command=RemoveServer server-id=server-637e3d76-fedb-7ba5-2cc2-895de972afce server-addr= servers="[{Suffrage:Voter ID:server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd Address:9a5bce64-da87-eed0-1c8c-d753d3ab43bd} {Suffrage:Voter ID:server-e8bf385c-490a-2395-38fd-ba0d812e9332 Address:e8bf385c-490a-2395-38fd-ba0d812e9332}]" testing.go:149: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:58.066Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: updating configuration: command=RemoveServer server-id=server-637e3d76-fedb-7ba5-2cc2-895de972afce server-addr= servers="[{Suffrage:Voter ID:server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd Address:9a5bce64-da87-eed0-1c8c-d753d3ab43bd} {Suffrage:Voter ID:server-e8bf385c-490a-2395-38fd-ba0d812e9332 Address:e8bf385c-490a-2395-38fd-ba0d812e9332}]" testLoggerAdapter verbose: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:58.066Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: removed ourself, transitioning to follower testing.go:149: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:58.066Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: removed ourself, transitioning to follower testLoggerAdapter verbose: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:58.067Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: entering follower state: follower="Node at 637e3d76-fedb-7ba5-2cc2-895de972afce [Follower]" leader= testing.go:149: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:58.067Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: entering follower state: follower="Node at 637e3d76-fedb-7ba5-2cc2-895de972afce [Follower]" leader= testLoggerAdapter verbose: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:58.067Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: aborting pipeline replication: peer="{Voter server-e8bf385c-490a-2395-38fd-ba0d812e9332 e8bf385c-490a-2395-38fd-ba0d812e9332}" testing.go:149: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:58.067Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: aborting pipeline replication: peer="{Voter server-e8bf385c-490a-2395-38fd-ba0d812e9332 e8bf385c-490a-2395-38fd-ba0d812e9332}" testLoggerAdapter verbose: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:58.067Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: aborting pipeline replication: peer="{Voter server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd 9a5bce64-da87-eed0-1c8c-d753d3ab43bd}" testing.go:149: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:58.067Z [INFO] server-637e3d76-fedb-7ba5-2cc2-895de972afce: aborting pipeline replication: peer="{Voter server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd 9a5bce64-da87-eed0-1c8c-d753d3ab43bd}" testLoggerAdapter verbose: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.117Z [WARN] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: heartbeat timeout reached, starting election: last-leader=637e3d76-fedb-7ba5-2cc2-895de972afce testing.go:149: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.117Z [WARN] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: heartbeat timeout reached, starting election: last-leader=637e3d76-fedb-7ba5-2cc2-895de972afce testLoggerAdapter verbose: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.117Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: entering candidate state: node="Node at 9a5bce64-da87-eed0-1c8c-d753d3ab43bd [Candidate]" term=3 testing.go:149: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.117Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: entering candidate state: node="Node at 9a5bce64-da87-eed0-1c8c-d753d3ab43bd [Candidate]" term=3 testLoggerAdapter verbose: server-e8bf385c-490a-2395-38fd-ba0d812e9332: 2024-01-09T05:09:58.118Z [WARN] server-e8bf385c-490a-2395-38fd-ba0d812e9332: rejecting vote request since we have a leader: from=9a5bce64-da87-eed0-1c8c-d753d3ab43bd leader=637e3d76-fedb-7ba5-2cc2-895de972afce testing.go:149: server-e8bf385c-490a-2395-38fd-ba0d812e9332: 2024-01-09T05:09:58.118Z [WARN] server-e8bf385c-490a-2395-38fd-ba0d812e9332: rejecting vote request since we have a leader: from=9a5bce64-da87-eed0-1c8c-d753d3ab43bd leader=637e3d76-fedb-7ba5-2cc2-895de972afce testLoggerAdapter verbose: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:58.160Z [WARN] server-637e3d76-fedb-7ba5-2cc2-895de972afce: not part of stable configuration, aborting election testing.go:149: server-637e3d76-fedb-7ba5-2cc2-895de972afce: 2024-01-09T05:09:58.160Z [WARN] server-637e3d76-fedb-7ba5-2cc2-895de972afce: not part of stable configuration, aborting election testLoggerAdapter verbose: server-e8bf385c-490a-2395-38fd-ba0d812e9332: 2024-01-09T05:09:58.170Z [WARN] server-e8bf385c-490a-2395-38fd-ba0d812e9332: heartbeat timeout reached, starting election: last-leader=637e3d76-fedb-7ba5-2cc2-895de972afce testing.go:149: server-e8bf385c-490a-2395-38fd-ba0d812e9332: 2024-01-09T05:09:58.170Z [WARN] server-e8bf385c-490a-2395-38fd-ba0d812e9332: heartbeat timeout reached, starting election: last-leader=637e3d76-fedb-7ba5-2cc2-895de972afce testLoggerAdapter verbose: server-e8bf385c-490a-2395-38fd-ba0d812e9332: 2024-01-09T05:09:58.170Z [INFO] server-e8bf385c-490a-2395-38fd-ba0d812e9332: entering candidate state: node="Node at e8bf385c-490a-2395-38fd-ba0d812e9332 [Candidate]" term=3 testing.go:149: server-e8bf385c-490a-2395-38fd-ba0d812e9332: 2024-01-09T05:09:58.170Z [INFO] server-e8bf385c-490a-2395-38fd-ba0d812e9332: entering candidate state: node="Node at e8bf385c-490a-2395-38fd-ba0d812e9332 [Candidate]" term=3 testLoggerAdapter verbose: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.170Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: duplicate requestVote for same term: term=3 testing.go:149: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.170Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: duplicate requestVote for same term: term=3 testLoggerAdapter verbose: cluster: 2024-01-09T05:09:58.173Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:09:58.173Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.216Z [WARN] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: Election timeout reached, restarting election testing.go:149: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.216Z [WARN] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: Election timeout reached, restarting election testLoggerAdapter verbose: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.216Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: entering candidate state: node="Node at 9a5bce64-da87-eed0-1c8c-d753d3ab43bd [Candidate]" term=4 testing.go:149: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.216Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: entering candidate state: node="Node at 9a5bce64-da87-eed0-1c8c-d753d3ab43bd [Candidate]" term=4 testLoggerAdapter verbose: server-e8bf385c-490a-2395-38fd-ba0d812e9332: 2024-01-09T05:09:58.216Z [INFO] server-e8bf385c-490a-2395-38fd-ba0d812e9332: entering follower state: follower="Node at e8bf385c-490a-2395-38fd-ba0d812e9332 [Follower]" leader= testing.go:149: server-e8bf385c-490a-2395-38fd-ba0d812e9332: 2024-01-09T05:09:58.216Z [INFO] server-e8bf385c-490a-2395-38fd-ba0d812e9332: entering follower state: follower="Node at e8bf385c-490a-2395-38fd-ba0d812e9332 [Follower]" leader= testLoggerAdapter verbose: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.216Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: election won: tally=2 testing.go:149: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.216Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: election won: tally=2 testLoggerAdapter verbose: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.216Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: entering leader state: leader="Node at 9a5bce64-da87-eed0-1c8c-d753d3ab43bd [Leader]" testing.go:149: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.216Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: entering leader state: leader="Node at 9a5bce64-da87-eed0-1c8c-d753d3ab43bd [Leader]" testLoggerAdapter verbose: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.217Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: added peer, starting replication: peer=server-e8bf385c-490a-2395-38fd-ba0d812e9332 testing.go:149: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.217Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: added peer, starting replication: peer=server-e8bf385c-490a-2395-38fd-ba0d812e9332 testLoggerAdapter verbose: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.217Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: pipelining replication: peer="{Voter server-e8bf385c-490a-2395-38fd-ba0d812e9332 e8bf385c-490a-2395-38fd-ba0d812e9332}" testing.go:149: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.217Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: pipelining replication: peer="{Voter server-e8bf385c-490a-2395-38fd-ba0d812e9332 e8bf385c-490a-2395-38fd-ba0d812e9332}" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:58.323Z [INFO] stable state for Leader reached at 2024-01-09 05:09:58.217332903 +0000 UTC m=+9.470384039 (1 nodes), 43.958323ms from start of poll, 434.349835ms from cluster start. Timeout at 2024-01-09 05:09:58.322960517 +0000 UTC m=+9.576011784, 105.627745ms after stability testing.go:149: cluster: 2024-01-09T05:09:58.323Z [INFO] stable state for Leader reached at 2024-01-09 05:09:58.217332903 +0000 UTC m=+9.470384039 (1 nodes), 43.958323ms from start of poll, 434.349835ms from cluster start. Timeout at 2024-01-09 05:09:58.322960517 +0000 UTC m=+9.576011784, 105.627745ms after stability testLoggerAdapter verbose: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.435Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: aborting pipeline replication: peer="{Voter server-e8bf385c-490a-2395-38fd-ba0d812e9332 e8bf385c-490a-2395-38fd-ba0d812e9332}" testing.go:149: server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: 2024-01-09T05:09:58.435Z [INFO] server-9a5bce64-da87-eed0-1c8c-d753d3ab43bd: aborting pipeline replication: peer="{Voter server-e8bf385c-490a-2395-38fd-ba0d812e9332 e8bf385c-490a-2395-38fd-ba0d812e9332}" --- PASS: TestRaft_RemoveLeader_NoShutdown (0.65s) === RUN TestRaft_RemoveFollower_SplitCluster testLoggerAdapter verbose: server-8e63e08b-8779-5ecf-6579-0337bd0b7259: 2024-01-09T05:09:58.437Z [INFO] server-8e63e08b-8779-5ecf-6579-0337bd0b7259: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e63e08b-8779-5ecf-6579-0337bd0b7259 Address:8e63e08b-8779-5ecf-6579-0337bd0b7259} {Suffrage:Voter ID:server-5db5b671-affb-8f8c-8841-e12234d7bd6b Address:5db5b671-affb-8f8c-8841-e12234d7bd6b} {Suffrage:Voter ID:server-dfab7669-1d70-4fbc-1a53-ab8358b776d5 Address:dfab7669-1d70-4fbc-1a53-ab8358b776d5} {Suffrage:Voter ID:server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0 Address:f82cb951-6b0e-596d-ad0a-c1c6c66295c0}]" testing.go:149: server-8e63e08b-8779-5ecf-6579-0337bd0b7259: 2024-01-09T05:09:58.437Z [INFO] server-8e63e08b-8779-5ecf-6579-0337bd0b7259: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e63e08b-8779-5ecf-6579-0337bd0b7259 Address:8e63e08b-8779-5ecf-6579-0337bd0b7259} {Suffrage:Voter ID:server-5db5b671-affb-8f8c-8841-e12234d7bd6b Address:5db5b671-affb-8f8c-8841-e12234d7bd6b} {Suffrage:Voter ID:server-dfab7669-1d70-4fbc-1a53-ab8358b776d5 Address:dfab7669-1d70-4fbc-1a53-ab8358b776d5} {Suffrage:Voter ID:server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0 Address:f82cb951-6b0e-596d-ad0a-c1c6c66295c0}]" testLoggerAdapter verbose: server-5db5b671-affb-8f8c-8841-e12234d7bd6b: 2024-01-09T05:09:58.437Z [INFO] server-5db5b671-affb-8f8c-8841-e12234d7bd6b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e63e08b-8779-5ecf-6579-0337bd0b7259 Address:8e63e08b-8779-5ecf-6579-0337bd0b7259} {Suffrage:Voter ID:server-5db5b671-affb-8f8c-8841-e12234d7bd6b Address:5db5b671-affb-8f8c-8841-e12234d7bd6b} {Suffrage:Voter ID:server-dfab7669-1d70-4fbc-1a53-ab8358b776d5 Address:dfab7669-1d70-4fbc-1a53-ab8358b776d5} {Suffrage:Voter ID:server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0 Address:f82cb951-6b0e-596d-ad0a-c1c6c66295c0}]" testing.go:149: server-5db5b671-affb-8f8c-8841-e12234d7bd6b: 2024-01-09T05:09:58.437Z [INFO] server-5db5b671-affb-8f8c-8841-e12234d7bd6b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e63e08b-8779-5ecf-6579-0337bd0b7259 Address:8e63e08b-8779-5ecf-6579-0337bd0b7259} {Suffrage:Voter ID:server-5db5b671-affb-8f8c-8841-e12234d7bd6b Address:5db5b671-affb-8f8c-8841-e12234d7bd6b} {Suffrage:Voter ID:server-dfab7669-1d70-4fbc-1a53-ab8358b776d5 Address:dfab7669-1d70-4fbc-1a53-ab8358b776d5} {Suffrage:Voter ID:server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0 Address:f82cb951-6b0e-596d-ad0a-c1c6c66295c0}]" testLoggerAdapter verbose: server-8e63e08b-8779-5ecf-6579-0337bd0b7259: 2024-01-09T05:09:58.437Z [INFO] server-8e63e08b-8779-5ecf-6579-0337bd0b7259: entering follower state: follower="Node at 8e63e08b-8779-5ecf-6579-0337bd0b7259 [Follower]" leader= testing.go:149: server-8e63e08b-8779-5ecf-6579-0337bd0b7259: 2024-01-09T05:09:58.437Z [INFO] server-8e63e08b-8779-5ecf-6579-0337bd0b7259: entering follower state: follower="Node at 8e63e08b-8779-5ecf-6579-0337bd0b7259 [Follower]" leader= testLoggerAdapter verbose: server-5db5b671-affb-8f8c-8841-e12234d7bd6b: 2024-01-09T05:09:58.437Z [INFO] server-5db5b671-affb-8f8c-8841-e12234d7bd6b: entering follower state: follower="Node at 5db5b671-affb-8f8c-8841-e12234d7bd6b [Follower]" leader= testing.go:149: server-5db5b671-affb-8f8c-8841-e12234d7bd6b: 2024-01-09T05:09:58.437Z [INFO] server-5db5b671-affb-8f8c-8841-e12234d7bd6b: entering follower state: follower="Node at 5db5b671-affb-8f8c-8841-e12234d7bd6b [Follower]" leader= testLoggerAdapter verbose: server-dfab7669-1d70-4fbc-1a53-ab8358b776d5: 2024-01-09T05:09:58.437Z [INFO] server-dfab7669-1d70-4fbc-1a53-ab8358b776d5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e63e08b-8779-5ecf-6579-0337bd0b7259 Address:8e63e08b-8779-5ecf-6579-0337bd0b7259} {Suffrage:Voter ID:server-5db5b671-affb-8f8c-8841-e12234d7bd6b Address:5db5b671-affb-8f8c-8841-e12234d7bd6b} {Suffrage:Voter ID:server-dfab7669-1d70-4fbc-1a53-ab8358b776d5 Address:dfab7669-1d70-4fbc-1a53-ab8358b776d5} {Suffrage:Voter ID:server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0 Address:f82cb951-6b0e-596d-ad0a-c1c6c66295c0}]" testing.go:149: server-dfab7669-1d70-4fbc-1a53-ab8358b776d5: 2024-01-09T05:09:58.437Z [INFO] server-dfab7669-1d70-4fbc-1a53-ab8358b776d5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e63e08b-8779-5ecf-6579-0337bd0b7259 Address:8e63e08b-8779-5ecf-6579-0337bd0b7259} {Suffrage:Voter ID:server-5db5b671-affb-8f8c-8841-e12234d7bd6b Address:5db5b671-affb-8f8c-8841-e12234d7bd6b} {Suffrage:Voter ID:server-dfab7669-1d70-4fbc-1a53-ab8358b776d5 Address:dfab7669-1d70-4fbc-1a53-ab8358b776d5} {Suffrage:Voter ID:server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0 Address:f82cb951-6b0e-596d-ad0a-c1c6c66295c0}]" testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.437Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e63e08b-8779-5ecf-6579-0337bd0b7259 Address:8e63e08b-8779-5ecf-6579-0337bd0b7259} {Suffrage:Voter ID:server-5db5b671-affb-8f8c-8841-e12234d7bd6b Address:5db5b671-affb-8f8c-8841-e12234d7bd6b} {Suffrage:Voter ID:server-dfab7669-1d70-4fbc-1a53-ab8358b776d5 Address:dfab7669-1d70-4fbc-1a53-ab8358b776d5} {Suffrage:Voter ID:server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0 Address:f82cb951-6b0e-596d-ad0a-c1c6c66295c0}]" testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.437Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e63e08b-8779-5ecf-6579-0337bd0b7259 Address:8e63e08b-8779-5ecf-6579-0337bd0b7259} {Suffrage:Voter ID:server-5db5b671-affb-8f8c-8841-e12234d7bd6b Address:5db5b671-affb-8f8c-8841-e12234d7bd6b} {Suffrage:Voter ID:server-dfab7669-1d70-4fbc-1a53-ab8358b776d5 Address:dfab7669-1d70-4fbc-1a53-ab8358b776d5} {Suffrage:Voter ID:server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0 Address:f82cb951-6b0e-596d-ad0a-c1c6c66295c0}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:58.437Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:09:58.437Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-dfab7669-1d70-4fbc-1a53-ab8358b776d5: 2024-01-09T05:09:58.437Z [INFO] server-dfab7669-1d70-4fbc-1a53-ab8358b776d5: entering follower state: follower="Node at dfab7669-1d70-4fbc-1a53-ab8358b776d5 [Follower]" leader= testing.go:149: server-dfab7669-1d70-4fbc-1a53-ab8358b776d5: 2024-01-09T05:09:58.437Z [INFO] server-dfab7669-1d70-4fbc-1a53-ab8358b776d5: entering follower state: follower="Node at dfab7669-1d70-4fbc-1a53-ab8358b776d5 [Follower]" leader= testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.437Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: entering follower state: follower="Node at f82cb951-6b0e-596d-ad0a-c1c6c66295c0 [Follower]" leader= testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.437Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: entering follower state: follower="Node at f82cb951-6b0e-596d-ad0a-c1c6c66295c0 [Follower]" leader= testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.519Z [WARN] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.519Z [WARN] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.519Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: entering candidate state: node="Node at f82cb951-6b0e-596d-ad0a-c1c6c66295c0 [Candidate]" term=2 testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.519Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: entering candidate state: node="Node at f82cb951-6b0e-596d-ad0a-c1c6c66295c0 [Candidate]" term=2 testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.519Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: election won: tally=3 testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.519Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: election won: tally=3 testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.519Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: entering leader state: leader="Node at f82cb951-6b0e-596d-ad0a-c1c6c66295c0 [Leader]" testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.519Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: entering leader state: leader="Node at f82cb951-6b0e-596d-ad0a-c1c6c66295c0 [Leader]" testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.519Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: added peer, starting replication: peer=server-8e63e08b-8779-5ecf-6579-0337bd0b7259 testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.519Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: added peer, starting replication: peer=server-8e63e08b-8779-5ecf-6579-0337bd0b7259 testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.519Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: added peer, starting replication: peer=server-5db5b671-affb-8f8c-8841-e12234d7bd6b testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.519Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: added peer, starting replication: peer=server-5db5b671-affb-8f8c-8841-e12234d7bd6b testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.519Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: added peer, starting replication: peer=server-dfab7669-1d70-4fbc-1a53-ab8358b776d5 testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.519Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: added peer, starting replication: peer=server-dfab7669-1d70-4fbc-1a53-ab8358b776d5 testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.519Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: pipelining replication: peer="{Voter server-dfab7669-1d70-4fbc-1a53-ab8358b776d5 dfab7669-1d70-4fbc-1a53-ab8358b776d5}" testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.519Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: pipelining replication: peer="{Voter server-dfab7669-1d70-4fbc-1a53-ab8358b776d5 dfab7669-1d70-4fbc-1a53-ab8358b776d5}" testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.519Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: pipelining replication: peer="{Voter server-8e63e08b-8779-5ecf-6579-0337bd0b7259 8e63e08b-8779-5ecf-6579-0337bd0b7259}" testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.519Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: pipelining replication: peer="{Voter server-8e63e08b-8779-5ecf-6579-0337bd0b7259 8e63e08b-8779-5ecf-6579-0337bd0b7259}" testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.520Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: pipelining replication: peer="{Voter server-5db5b671-affb-8f8c-8841-e12234d7bd6b 5db5b671-affb-8f8c-8841-e12234d7bd6b}" testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.520Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: pipelining replication: peer="{Voter server-5db5b671-affb-8f8c-8841-e12234d7bd6b 5db5b671-affb-8f8c-8841-e12234d7bd6b}" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:58.625Z [INFO] stable state for Leader reached at 2024-01-09 05:09:58.519887674 +0000 UTC m=+9.772938728 (1 nodes), 82.068495ms from start of poll, 82.860903ms from cluster start. Timeout at 2024-01-09 05:09:58.625002025 +0000 UTC m=+9.878053192, 105.114464ms after stability testing.go:149: cluster: 2024-01-09T05:09:58.625Z [INFO] stable state for Leader reached at 2024-01-09 05:09:58.519887674 +0000 UTC m=+9.772938728 (1 nodes), 82.068495ms from start of poll, 82.860903ms from cluster start. Timeout at 2024-01-09 05:09:58.625002025 +0000 UTC m=+9.878053192, 105.114464ms after stability testLoggerAdapter verbose: cluster: 2024-01-09T05:09:58.735Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-01-09T05:09:58.735Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2024-01-09T05:09:58.840Z [INFO] stable state for Follower reached at 2024-01-09 05:09:58.735221379 +0000 UTC m=+9.988272520 (3 nodes), 2.938µs from start of poll, 298.194695ms from cluster start. Timeout at 2024-01-09 05:09:58.840315884 +0000 UTC m=+10.093366986, 105.094466ms after stability testing.go:149: cluster: 2024-01-09T05:09:58.840Z [INFO] stable state for Follower reached at 2024-01-09 05:09:58.735221379 +0000 UTC m=+9.988272520 (3 nodes), 2.938µs from start of poll, 298.194695ms from cluster start. Timeout at 2024-01-09 05:09:58.840315884 +0000 UTC m=+10.093366986, 105.094466ms after stability testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.840Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: updating configuration: command=RemoveServer server-id=server-dfab7669-1d70-4fbc-1a53-ab8358b776d5 server-addr= servers="[{Suffrage:Voter ID:server-8e63e08b-8779-5ecf-6579-0337bd0b7259 Address:8e63e08b-8779-5ecf-6579-0337bd0b7259} {Suffrage:Voter ID:server-5db5b671-affb-8f8c-8841-e12234d7bd6b Address:5db5b671-affb-8f8c-8841-e12234d7bd6b} {Suffrage:Voter ID:server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0 Address:f82cb951-6b0e-596d-ad0a-c1c6c66295c0}]" testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.840Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: updating configuration: command=RemoveServer server-id=server-dfab7669-1d70-4fbc-1a53-ab8358b776d5 server-addr= servers="[{Suffrage:Voter ID:server-8e63e08b-8779-5ecf-6579-0337bd0b7259 Address:8e63e08b-8779-5ecf-6579-0337bd0b7259} {Suffrage:Voter ID:server-5db5b671-affb-8f8c-8841-e12234d7bd6b Address:5db5b671-affb-8f8c-8841-e12234d7bd6b} {Suffrage:Voter ID:server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0 Address:f82cb951-6b0e-596d-ad0a-c1c6c66295c0}]" testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.840Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: removed peer, stopping replication: peer=server-dfab7669-1d70-4fbc-1a53-ab8358b776d5 last-index=3 testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.840Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: removed peer, stopping replication: peer=server-dfab7669-1d70-4fbc-1a53-ab8358b776d5 last-index=3 testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.840Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to pipeline appendEntries: peer="{Voter server-8e63e08b-8779-5ecf-6579-0337bd0b7259 8e63e08b-8779-5ecf-6579-0337bd0b7259}" error="append pipeline closed" testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.840Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to pipeline appendEntries: peer="{Voter server-8e63e08b-8779-5ecf-6579-0337bd0b7259 8e63e08b-8779-5ecf-6579-0337bd0b7259}" error="append pipeline closed" testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.840Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: aborting pipeline replication: peer="{Voter server-8e63e08b-8779-5ecf-6579-0337bd0b7259 8e63e08b-8779-5ecf-6579-0337bd0b7259}" testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.840Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: aborting pipeline replication: peer="{Voter server-8e63e08b-8779-5ecf-6579-0337bd0b7259 8e63e08b-8779-5ecf-6579-0337bd0b7259}" testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.840Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to pipeline appendEntries: peer="{Voter server-5db5b671-affb-8f8c-8841-e12234d7bd6b 5db5b671-affb-8f8c-8841-e12234d7bd6b}" error="append pipeline closed" testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.840Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to pipeline appendEntries: peer="{Voter server-5db5b671-affb-8f8c-8841-e12234d7bd6b 5db5b671-affb-8f8c-8841-e12234d7bd6b}" error="append pipeline closed" testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.840Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: aborting pipeline replication: peer="{Voter server-5db5b671-affb-8f8c-8841-e12234d7bd6b 5db5b671-affb-8f8c-8841-e12234d7bd6b}" testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.840Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: aborting pipeline replication: peer="{Voter server-5db5b671-affb-8f8c-8841-e12234d7bd6b 5db5b671-affb-8f8c-8841-e12234d7bd6b}" testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.840Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: aborting pipeline replication: peer="{Voter server-dfab7669-1d70-4fbc-1a53-ab8358b776d5 dfab7669-1d70-4fbc-1a53-ab8358b776d5}" testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.840Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: aborting pipeline replication: peer="{Voter server-dfab7669-1d70-4fbc-1a53-ab8358b776d5 dfab7669-1d70-4fbc-1a53-ab8358b776d5}" testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.841Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to heartbeat to: peer=5db5b671-affb-8f8c-8841-e12234d7bd6b error="failed to connect to peer: 5db5b671-affb-8f8c-8841-e12234d7bd6b" testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.841Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to heartbeat to: peer=5db5b671-affb-8f8c-8841-e12234d7bd6b error="failed to connect to peer: 5db5b671-affb-8f8c-8841-e12234d7bd6b" testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.843Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to heartbeat to: peer=8e63e08b-8779-5ecf-6579-0337bd0b7259 error="failed to connect to peer: 8e63e08b-8779-5ecf-6579-0337bd0b7259" testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.843Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to heartbeat to: peer=8e63e08b-8779-5ecf-6579-0337bd0b7259 error="failed to connect to peer: 8e63e08b-8779-5ecf-6579-0337bd0b7259" testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.848Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to appendEntries to: peer="{Voter server-8e63e08b-8779-5ecf-6579-0337bd0b7259 8e63e08b-8779-5ecf-6579-0337bd0b7259}" error="failed to connect to peer: 8e63e08b-8779-5ecf-6579-0337bd0b7259" testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.848Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to appendEntries to: peer="{Voter server-8e63e08b-8779-5ecf-6579-0337bd0b7259 8e63e08b-8779-5ecf-6579-0337bd0b7259}" error="failed to connect to peer: 8e63e08b-8779-5ecf-6579-0337bd0b7259" testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.850Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to appendEntries to: peer="{Voter server-5db5b671-affb-8f8c-8841-e12234d7bd6b 5db5b671-affb-8f8c-8841-e12234d7bd6b}" error="failed to connect to peer: 5db5b671-affb-8f8c-8841-e12234d7bd6b" testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.850Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to appendEntries to: peer="{Voter server-5db5b671-affb-8f8c-8841-e12234d7bd6b 5db5b671-affb-8f8c-8841-e12234d7bd6b}" error="failed to connect to peer: 5db5b671-affb-8f8c-8841-e12234d7bd6b" testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.859Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to heartbeat to: peer=8e63e08b-8779-5ecf-6579-0337bd0b7259 error="failed to connect to peer: 8e63e08b-8779-5ecf-6579-0337bd0b7259" testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.859Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to heartbeat to: peer=8e63e08b-8779-5ecf-6579-0337bd0b7259 error="failed to connect to peer: 8e63e08b-8779-5ecf-6579-0337bd0b7259" testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.860Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to heartbeat to: peer=5db5b671-affb-8f8c-8841-e12234d7bd6b error="failed to connect to peer: 5db5b671-affb-8f8c-8841-e12234d7bd6b" testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.860Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to heartbeat to: peer=5db5b671-affb-8f8c-8841-e12234d7bd6b error="failed to connect to peer: 5db5b671-affb-8f8c-8841-e12234d7bd6b" testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.868Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to appendEntries to: peer="{Voter server-8e63e08b-8779-5ecf-6579-0337bd0b7259 8e63e08b-8779-5ecf-6579-0337bd0b7259}" error="failed to connect to peer: 8e63e08b-8779-5ecf-6579-0337bd0b7259" testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.868Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to appendEntries to: peer="{Voter server-8e63e08b-8779-5ecf-6579-0337bd0b7259 8e63e08b-8779-5ecf-6579-0337bd0b7259}" error="failed to connect to peer: 8e63e08b-8779-5ecf-6579-0337bd0b7259" testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.868Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to appendEntries to: peer="{Voter server-5db5b671-affb-8f8c-8841-e12234d7bd6b 5db5b671-affb-8f8c-8841-e12234d7bd6b}" error="failed to connect to peer: 5db5b671-affb-8f8c-8841-e12234d7bd6b" testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.868Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to appendEntries to: peer="{Voter server-5db5b671-affb-8f8c-8841-e12234d7bd6b 5db5b671-affb-8f8c-8841-e12234d7bd6b}" error="failed to connect to peer: 5db5b671-affb-8f8c-8841-e12234d7bd6b" testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.877Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to heartbeat to: peer=8e63e08b-8779-5ecf-6579-0337bd0b7259 error="failed to connect to peer: 8e63e08b-8779-5ecf-6579-0337bd0b7259" testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.877Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to heartbeat to: peer=8e63e08b-8779-5ecf-6579-0337bd0b7259 error="failed to connect to peer: 8e63e08b-8779-5ecf-6579-0337bd0b7259" testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.879Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to heartbeat to: peer=5db5b671-affb-8f8c-8841-e12234d7bd6b error="failed to connect to peer: 5db5b671-affb-8f8c-8841-e12234d7bd6b" testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.879Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to heartbeat to: peer=5db5b671-affb-8f8c-8841-e12234d7bd6b error="failed to connect to peer: 5db5b671-affb-8f8c-8841-e12234d7bd6b" testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.886Z [WARN] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to contact: server-id=server-5db5b671-affb-8f8c-8841-e12234d7bd6b time=50.273422ms testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.886Z [WARN] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to contact: server-id=server-5db5b671-affb-8f8c-8841-e12234d7bd6b time=50.273422ms testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.886Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to appendEntries to: peer="{Voter server-8e63e08b-8779-5ecf-6579-0337bd0b7259 8e63e08b-8779-5ecf-6579-0337bd0b7259}" error="failed to connect to peer: 8e63e08b-8779-5ecf-6579-0337bd0b7259" testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.886Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to appendEntries to: peer="{Voter server-8e63e08b-8779-5ecf-6579-0337bd0b7259 8e63e08b-8779-5ecf-6579-0337bd0b7259}" error="failed to connect to peer: 8e63e08b-8779-5ecf-6579-0337bd0b7259" testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.887Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to appendEntries to: peer="{Voter server-5db5b671-affb-8f8c-8841-e12234d7bd6b 5db5b671-affb-8f8c-8841-e12234d7bd6b}" error="failed to connect to peer: 5db5b671-affb-8f8c-8841-e12234d7bd6b" testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.887Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to appendEntries to: peer="{Voter server-5db5b671-affb-8f8c-8841-e12234d7bd6b 5db5b671-affb-8f8c-8841-e12234d7bd6b}" error="failed to connect to peer: 5db5b671-affb-8f8c-8841-e12234d7bd6b" testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.896Z [WARN] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to contact: server-id=server-8e63e08b-8779-5ecf-6579-0337bd0b7259 time=59.445109ms testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.896Z [WARN] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to contact: server-id=server-8e63e08b-8779-5ecf-6579-0337bd0b7259 time=59.445109ms testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.896Z [WARN] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to contact: server-id=server-5db5b671-affb-8f8c-8841-e12234d7bd6b time=60.646174ms testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.896Z [WARN] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to contact: server-id=server-5db5b671-affb-8f8c-8841-e12234d7bd6b time=60.646174ms testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.896Z [WARN] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to contact quorum of nodes, stepping down testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.896Z [WARN] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to contact quorum of nodes, stepping down testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.896Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: entering follower state: follower="Node at f82cb951-6b0e-596d-ad0a-c1c6c66295c0 [Follower]" leader= testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.896Z [INFO] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: entering follower state: follower="Node at f82cb951-6b0e-596d-ad0a-c1c6c66295c0 [Follower]" leader= testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.896Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to appendEntries to: peer="{Voter server-8e63e08b-8779-5ecf-6579-0337bd0b7259 8e63e08b-8779-5ecf-6579-0337bd0b7259}" error="failed to connect to peer: 8e63e08b-8779-5ecf-6579-0337bd0b7259" testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.896Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to appendEntries to: peer="{Voter server-8e63e08b-8779-5ecf-6579-0337bd0b7259 8e63e08b-8779-5ecf-6579-0337bd0b7259}" error="failed to connect to peer: 8e63e08b-8779-5ecf-6579-0337bd0b7259" testLoggerAdapter verbose: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.896Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to appendEntries to: peer="{Voter server-5db5b671-affb-8f8c-8841-e12234d7bd6b 5db5b671-affb-8f8c-8841-e12234d7bd6b}" error="failed to connect to peer: 5db5b671-affb-8f8c-8841-e12234d7bd6b" testing.go:149: server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: 2024-01-09T05:09:58.896Z [ERROR] server-f82cb951-6b0e-596d-ad0a-c1c6c66295c0: failed to appendEntries to: peer="{Voter server-5db5b671-affb-8f8c-8841-e12234d7bd6b 5db5b671-affb-8f8c-8841-e12234d7bd6b}" error="failed to connect to peer: 5db5b671-affb-8f8c-8841-e12234d7bd6b" --- PASS: TestRaft_RemoveFollower_SplitCluster (0.46s) === RUN TestRaft_AddKnownPeer testLoggerAdapter verbose: server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75: 2024-01-09T05:09:58.898Z [INFO] server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 Address:37d7450a-7d4c-fb10-d0c8-2f2dfef49c75} {Suffrage:Voter ID:server-0d37ee14-011c-72df-7fd6-5553a7037a2b Address:0d37ee14-011c-72df-7fd6-5553a7037a2b} {Suffrage:Voter ID:server-483f3ce8-5fdf-56e2-efdf-e06b182e266f Address:483f3ce8-5fdf-56e2-efdf-e06b182e266f}]" testing.go:149: server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75: 2024-01-09T05:09:58.898Z [INFO] server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 Address:37d7450a-7d4c-fb10-d0c8-2f2dfef49c75} {Suffrage:Voter ID:server-0d37ee14-011c-72df-7fd6-5553a7037a2b Address:0d37ee14-011c-72df-7fd6-5553a7037a2b} {Suffrage:Voter ID:server-483f3ce8-5fdf-56e2-efdf-e06b182e266f Address:483f3ce8-5fdf-56e2-efdf-e06b182e266f}]" testLoggerAdapter verbose: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.898Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 Address:37d7450a-7d4c-fb10-d0c8-2f2dfef49c75} {Suffrage:Voter ID:server-0d37ee14-011c-72df-7fd6-5553a7037a2b Address:0d37ee14-011c-72df-7fd6-5553a7037a2b} {Suffrage:Voter ID:server-483f3ce8-5fdf-56e2-efdf-e06b182e266f Address:483f3ce8-5fdf-56e2-efdf-e06b182e266f}]" testing.go:149: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.898Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 Address:37d7450a-7d4c-fb10-d0c8-2f2dfef49c75} {Suffrage:Voter ID:server-0d37ee14-011c-72df-7fd6-5553a7037a2b Address:0d37ee14-011c-72df-7fd6-5553a7037a2b} {Suffrage:Voter ID:server-483f3ce8-5fdf-56e2-efdf-e06b182e266f Address:483f3ce8-5fdf-56e2-efdf-e06b182e266f}]" testLoggerAdapter verbose: server-483f3ce8-5fdf-56e2-efdf-e06b182e266f: 2024-01-09T05:09:58.898Z [INFO] server-483f3ce8-5fdf-56e2-efdf-e06b182e266f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 Address:37d7450a-7d4c-fb10-d0c8-2f2dfef49c75} {Suffrage:Voter ID:server-0d37ee14-011c-72df-7fd6-5553a7037a2b Address:0d37ee14-011c-72df-7fd6-5553a7037a2b} {Suffrage:Voter ID:server-483f3ce8-5fdf-56e2-efdf-e06b182e266f Address:483f3ce8-5fdf-56e2-efdf-e06b182e266f}]" testing.go:149: server-483f3ce8-5fdf-56e2-efdf-e06b182e266f: 2024-01-09T05:09:58.898Z [INFO] server-483f3ce8-5fdf-56e2-efdf-e06b182e266f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 Address:37d7450a-7d4c-fb10-d0c8-2f2dfef49c75} {Suffrage:Voter ID:server-0d37ee14-011c-72df-7fd6-5553a7037a2b Address:0d37ee14-011c-72df-7fd6-5553a7037a2b} {Suffrage:Voter ID:server-483f3ce8-5fdf-56e2-efdf-e06b182e266f Address:483f3ce8-5fdf-56e2-efdf-e06b182e266f}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:58.898Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:09:58.898Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75: 2024-01-09T05:09:58.899Z [INFO] server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75: entering follower state: follower="Node at 37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 [Follower]" leader= testing.go:149: server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75: 2024-01-09T05:09:58.899Z [INFO] server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75: entering follower state: follower="Node at 37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 [Follower]" leader= testLoggerAdapter verbose: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.899Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: entering follower state: follower="Node at 0d37ee14-011c-72df-7fd6-5553a7037a2b [Follower]" leader= testing.go:149: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.899Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: entering follower state: follower="Node at 0d37ee14-011c-72df-7fd6-5553a7037a2b [Follower]" leader= testLoggerAdapter verbose: server-483f3ce8-5fdf-56e2-efdf-e06b182e266f: 2024-01-09T05:09:58.899Z [INFO] server-483f3ce8-5fdf-56e2-efdf-e06b182e266f: entering follower state: follower="Node at 483f3ce8-5fdf-56e2-efdf-e06b182e266f [Follower]" leader= testing.go:149: server-483f3ce8-5fdf-56e2-efdf-e06b182e266f: 2024-01-09T05:09:58.899Z [INFO] server-483f3ce8-5fdf-56e2-efdf-e06b182e266f: entering follower state: follower="Node at 483f3ce8-5fdf-56e2-efdf-e06b182e266f [Follower]" leader= testLoggerAdapter verbose: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.975Z [WARN] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.975Z [WARN] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.976Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: entering candidate state: node="Node at 0d37ee14-011c-72df-7fd6-5553a7037a2b [Candidate]" term=2 testing.go:149: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.976Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: entering candidate state: node="Node at 0d37ee14-011c-72df-7fd6-5553a7037a2b [Candidate]" term=2 testLoggerAdapter verbose: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.976Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: election won: tally=2 testing.go:149: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.976Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: election won: tally=2 testLoggerAdapter verbose: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.976Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: entering leader state: leader="Node at 0d37ee14-011c-72df-7fd6-5553a7037a2b [Leader]" testing.go:149: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.976Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: entering leader state: leader="Node at 0d37ee14-011c-72df-7fd6-5553a7037a2b [Leader]" testLoggerAdapter verbose: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.976Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: added peer, starting replication: peer=server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 testing.go:149: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.976Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: added peer, starting replication: peer=server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 testLoggerAdapter verbose: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.976Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: added peer, starting replication: peer=server-483f3ce8-5fdf-56e2-efdf-e06b182e266f testing.go:149: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.976Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: added peer, starting replication: peer=server-483f3ce8-5fdf-56e2-efdf-e06b182e266f testLoggerAdapter verbose: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.976Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: pipelining replication: peer="{Voter server-483f3ce8-5fdf-56e2-efdf-e06b182e266f 483f3ce8-5fdf-56e2-efdf-e06b182e266f}" testing.go:149: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.976Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: pipelining replication: peer="{Voter server-483f3ce8-5fdf-56e2-efdf-e06b182e266f 483f3ce8-5fdf-56e2-efdf-e06b182e266f}" testLoggerAdapter verbose: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.976Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: pipelining replication: peer="{Voter server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 37d7450a-7d4c-fb10-d0c8-2f2dfef49c75}" testing.go:149: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:58.976Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: pipelining replication: peer="{Voter server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 37d7450a-7d4c-fb10-d0c8-2f2dfef49c75}" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:59.081Z [INFO] stable state for Leader reached at 2024-01-09 05:09:58.976515905 +0000 UTC m=+10.229567139 (1 nodes), 77.48489ms from start of poll, 78.042221ms from cluster start. Timeout at 2024-01-09 05:09:59.081621458 +0000 UTC m=+10.334672589, 105.10545ms after stability testing.go:149: cluster: 2024-01-09T05:09:59.081Z [INFO] stable state for Leader reached at 2024-01-09 05:09:58.976515905 +0000 UTC m=+10.229567139 (1 nodes), 77.48489ms from start of poll, 78.042221ms from cluster start. Timeout at 2024-01-09 05:09:59.081621458 +0000 UTC m=+10.334672589, 105.10545ms after stability testLoggerAdapter verbose: cluster: 2024-01-09T05:09:59.081Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-01-09T05:09:59.081Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2024-01-09T05:09:59.187Z [INFO] stable state for Follower reached at 2024-01-09 05:09:59.081769751 +0000 UTC m=+10.334820785 (2 nodes), 2.547µs from start of poll, 183.295867ms from cluster start. Timeout at 2024-01-09 05:09:59.187393592 +0000 UTC m=+10.440444720, 105.623935ms after stability testing.go:149: cluster: 2024-01-09T05:09:59.187Z [INFO] stable state for Follower reached at 2024-01-09 05:09:59.081769751 +0000 UTC m=+10.334820785 (2 nodes), 2.547µs from start of poll, 183.295867ms from cluster start. Timeout at 2024-01-09 05:09:59.187393592 +0000 UTC m=+10.440444720, 105.623935ms after stability testLoggerAdapter verbose: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:59.187Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: updating configuration: command=AddStaging server-id=server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 server-addr=37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 servers="[{Suffrage:Voter ID:server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 Address:37d7450a-7d4c-fb10-d0c8-2f2dfef49c75} {Suffrage:Voter ID:server-0d37ee14-011c-72df-7fd6-5553a7037a2b Address:0d37ee14-011c-72df-7fd6-5553a7037a2b} {Suffrage:Voter ID:server-483f3ce8-5fdf-56e2-efdf-e06b182e266f Address:483f3ce8-5fdf-56e2-efdf-e06b182e266f}]" testing.go:149: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:59.187Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: updating configuration: command=AddStaging server-id=server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 server-addr=37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 servers="[{Suffrage:Voter ID:server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 Address:37d7450a-7d4c-fb10-d0c8-2f2dfef49c75} {Suffrage:Voter ID:server-0d37ee14-011c-72df-7fd6-5553a7037a2b Address:0d37ee14-011c-72df-7fd6-5553a7037a2b} {Suffrage:Voter ID:server-483f3ce8-5fdf-56e2-efdf-e06b182e266f Address:483f3ce8-5fdf-56e2-efdf-e06b182e266f}]" testLoggerAdapter verbose: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:59.190Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: aborting pipeline replication: peer="{Voter server-483f3ce8-5fdf-56e2-efdf-e06b182e266f 483f3ce8-5fdf-56e2-efdf-e06b182e266f}" testing.go:149: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:59.190Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: aborting pipeline replication: peer="{Voter server-483f3ce8-5fdf-56e2-efdf-e06b182e266f 483f3ce8-5fdf-56e2-efdf-e06b182e266f}" testLoggerAdapter verbose: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:59.190Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: aborting pipeline replication: peer="{Voter server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 37d7450a-7d4c-fb10-d0c8-2f2dfef49c75}" testing.go:149: server-0d37ee14-011c-72df-7fd6-5553a7037a2b: 2024-01-09T05:09:59.190Z [INFO] server-0d37ee14-011c-72df-7fd6-5553a7037a2b: aborting pipeline replication: peer="{Voter server-37d7450a-7d4c-fb10-d0c8-2f2dfef49c75 37d7450a-7d4c-fb10-d0c8-2f2dfef49c75}" --- PASS: TestRaft_AddKnownPeer (0.30s) === RUN TestRaft_RemoveUnknownPeer testLoggerAdapter verbose: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.199Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b62f0521-7d78-49be-e421-c343bee52c3c Address:b62f0521-7d78-49be-e421-c343bee52c3c} {Suffrage:Voter ID:server-650c6d6f-3e9c-4281-fa05-7671ff6196c6 Address:650c6d6f-3e9c-4281-fa05-7671ff6196c6} {Suffrage:Voter ID:server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f Address:8fd7c004-241a-b68b-3e7b-abd2b1e8006f}]" testing.go:149: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.199Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b62f0521-7d78-49be-e421-c343bee52c3c Address:b62f0521-7d78-49be-e421-c343bee52c3c} {Suffrage:Voter ID:server-650c6d6f-3e9c-4281-fa05-7671ff6196c6 Address:650c6d6f-3e9c-4281-fa05-7671ff6196c6} {Suffrage:Voter ID:server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f Address:8fd7c004-241a-b68b-3e7b-abd2b1e8006f}]" testLoggerAdapter verbose: server-650c6d6f-3e9c-4281-fa05-7671ff6196c6: 2024-01-09T05:09:59.199Z [INFO] server-650c6d6f-3e9c-4281-fa05-7671ff6196c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b62f0521-7d78-49be-e421-c343bee52c3c Address:b62f0521-7d78-49be-e421-c343bee52c3c} {Suffrage:Voter ID:server-650c6d6f-3e9c-4281-fa05-7671ff6196c6 Address:650c6d6f-3e9c-4281-fa05-7671ff6196c6} {Suffrage:Voter ID:server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f Address:8fd7c004-241a-b68b-3e7b-abd2b1e8006f}]" testing.go:149: server-650c6d6f-3e9c-4281-fa05-7671ff6196c6: 2024-01-09T05:09:59.199Z [INFO] server-650c6d6f-3e9c-4281-fa05-7671ff6196c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b62f0521-7d78-49be-e421-c343bee52c3c Address:b62f0521-7d78-49be-e421-c343bee52c3c} {Suffrage:Voter ID:server-650c6d6f-3e9c-4281-fa05-7671ff6196c6 Address:650c6d6f-3e9c-4281-fa05-7671ff6196c6} {Suffrage:Voter ID:server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f Address:8fd7c004-241a-b68b-3e7b-abd2b1e8006f}]" testLoggerAdapter verbose: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.200Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: entering follower state: follower="Node at b62f0521-7d78-49be-e421-c343bee52c3c [Follower]" leader= testing.go:149: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.200Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: entering follower state: follower="Node at b62f0521-7d78-49be-e421-c343bee52c3c [Follower]" leader= testLoggerAdapter verbose: server-650c6d6f-3e9c-4281-fa05-7671ff6196c6: 2024-01-09T05:09:59.200Z [INFO] server-650c6d6f-3e9c-4281-fa05-7671ff6196c6: entering follower state: follower="Node at 650c6d6f-3e9c-4281-fa05-7671ff6196c6 [Follower]" leader= testing.go:149: server-650c6d6f-3e9c-4281-fa05-7671ff6196c6: 2024-01-09T05:09:59.200Z [INFO] server-650c6d6f-3e9c-4281-fa05-7671ff6196c6: entering follower state: follower="Node at 650c6d6f-3e9c-4281-fa05-7671ff6196c6 [Follower]" leader= testLoggerAdapter verbose: server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f: 2024-01-09T05:09:59.200Z [INFO] server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b62f0521-7d78-49be-e421-c343bee52c3c Address:b62f0521-7d78-49be-e421-c343bee52c3c} {Suffrage:Voter ID:server-650c6d6f-3e9c-4281-fa05-7671ff6196c6 Address:650c6d6f-3e9c-4281-fa05-7671ff6196c6} {Suffrage:Voter ID:server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f Address:8fd7c004-241a-b68b-3e7b-abd2b1e8006f}]" testing.go:149: server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f: 2024-01-09T05:09:59.200Z [INFO] server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b62f0521-7d78-49be-e421-c343bee52c3c Address:b62f0521-7d78-49be-e421-c343bee52c3c} {Suffrage:Voter ID:server-650c6d6f-3e9c-4281-fa05-7671ff6196c6 Address:650c6d6f-3e9c-4281-fa05-7671ff6196c6} {Suffrage:Voter ID:server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f Address:8fd7c004-241a-b68b-3e7b-abd2b1e8006f}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:59.200Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:09:59.200Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f: 2024-01-09T05:09:59.200Z [INFO] server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f: entering follower state: follower="Node at 8fd7c004-241a-b68b-3e7b-abd2b1e8006f [Follower]" leader= testing.go:149: server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f: 2024-01-09T05:09:59.200Z [INFO] server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f: entering follower state: follower="Node at 8fd7c004-241a-b68b-3e7b-abd2b1e8006f [Follower]" leader= testLoggerAdapter verbose: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.251Z [WARN] server-b62f0521-7d78-49be-e421-c343bee52c3c: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.251Z [WARN] server-b62f0521-7d78-49be-e421-c343bee52c3c: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.251Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: entering candidate state: node="Node at b62f0521-7d78-49be-e421-c343bee52c3c [Candidate]" term=2 testing.go:149: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.251Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: entering candidate state: node="Node at b62f0521-7d78-49be-e421-c343bee52c3c [Candidate]" term=2 testLoggerAdapter verbose: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.253Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: election won: tally=2 testing.go:149: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.253Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: election won: tally=2 testLoggerAdapter verbose: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.253Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: entering leader state: leader="Node at b62f0521-7d78-49be-e421-c343bee52c3c [Leader]" testing.go:149: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.253Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: entering leader state: leader="Node at b62f0521-7d78-49be-e421-c343bee52c3c [Leader]" testLoggerAdapter verbose: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.253Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: added peer, starting replication: peer=server-650c6d6f-3e9c-4281-fa05-7671ff6196c6 testing.go:149: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.253Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: added peer, starting replication: peer=server-650c6d6f-3e9c-4281-fa05-7671ff6196c6 testLoggerAdapter verbose: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.254Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: added peer, starting replication: peer=server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f testing.go:149: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.254Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: added peer, starting replication: peer=server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f testLoggerAdapter verbose: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.254Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: pipelining replication: peer="{Voter server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f 8fd7c004-241a-b68b-3e7b-abd2b1e8006f}" testing.go:149: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.254Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: pipelining replication: peer="{Voter server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f 8fd7c004-241a-b68b-3e7b-abd2b1e8006f}" testLoggerAdapter verbose: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.255Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: pipelining replication: peer="{Voter server-650c6d6f-3e9c-4281-fa05-7671ff6196c6 650c6d6f-3e9c-4281-fa05-7671ff6196c6}" testing.go:149: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.255Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: pipelining replication: peer="{Voter server-650c6d6f-3e9c-4281-fa05-7671ff6196c6 650c6d6f-3e9c-4281-fa05-7671ff6196c6}" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:59.359Z [INFO] stable state for Leader reached at 2024-01-09 05:09:59.254617444 +0000 UTC m=+10.507668480 (1 nodes), 53.918384ms from start of poll, 55.593048ms from cluster start. Timeout at 2024-01-09 05:09:59.359698059 +0000 UTC m=+10.612749120, 105.08064ms after stability testing.go:149: cluster: 2024-01-09T05:09:59.359Z [INFO] stable state for Leader reached at 2024-01-09 05:09:59.254617444 +0000 UTC m=+10.507668480 (1 nodes), 53.918384ms from start of poll, 55.593048ms from cluster start. Timeout at 2024-01-09 05:09:59.359698059 +0000 UTC m=+10.612749120, 105.08064ms after stability testLoggerAdapter verbose: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.359Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: updating configuration: command=RemoveServer server-id=a98d063c-8213-edb6-fa4a-32463f278c19 server-addr= servers="[{Suffrage:Voter ID:server-b62f0521-7d78-49be-e421-c343bee52c3c Address:b62f0521-7d78-49be-e421-c343bee52c3c} {Suffrage:Voter ID:server-650c6d6f-3e9c-4281-fa05-7671ff6196c6 Address:650c6d6f-3e9c-4281-fa05-7671ff6196c6} {Suffrage:Voter ID:server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f Address:8fd7c004-241a-b68b-3e7b-abd2b1e8006f}]" testing.go:149: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.359Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: updating configuration: command=RemoveServer server-id=a98d063c-8213-edb6-fa4a-32463f278c19 server-addr= servers="[{Suffrage:Voter ID:server-b62f0521-7d78-49be-e421-c343bee52c3c Address:b62f0521-7d78-49be-e421-c343bee52c3c} {Suffrage:Voter ID:server-650c6d6f-3e9c-4281-fa05-7671ff6196c6 Address:650c6d6f-3e9c-4281-fa05-7671ff6196c6} {Suffrage:Voter ID:server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f Address:8fd7c004-241a-b68b-3e7b-abd2b1e8006f}]" testLoggerAdapter verbose: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.360Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: aborting pipeline replication: peer="{Voter server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f 8fd7c004-241a-b68b-3e7b-abd2b1e8006f}" testing.go:149: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.360Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: aborting pipeline replication: peer="{Voter server-8fd7c004-241a-b68b-3e7b-abd2b1e8006f 8fd7c004-241a-b68b-3e7b-abd2b1e8006f}" testLoggerAdapter verbose: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.360Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: aborting pipeline replication: peer="{Voter server-650c6d6f-3e9c-4281-fa05-7671ff6196c6 650c6d6f-3e9c-4281-fa05-7671ff6196c6}" testing.go:149: server-b62f0521-7d78-49be-e421-c343bee52c3c: 2024-01-09T05:09:59.360Z [INFO] server-b62f0521-7d78-49be-e421-c343bee52c3c: aborting pipeline replication: peer="{Voter server-650c6d6f-3e9c-4281-fa05-7671ff6196c6 650c6d6f-3e9c-4281-fa05-7671ff6196c6}" --- PASS: TestRaft_RemoveUnknownPeer (0.16s) === RUN TestRaft_SnapshotRestore testLoggerAdapter verbose: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.361Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31783a95-4ae1-8861-c5c6-d38a313ca10d Address:31783a95-4ae1-8861-c5c6-d38a313ca10d}]" testing.go:149: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.361Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31783a95-4ae1-8861-c5c6-d38a313ca10d Address:31783a95-4ae1-8861-c5c6-d38a313ca10d}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:59.361Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:09:59.361Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.362Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: entering follower state: follower="Node at 31783a95-4ae1-8861-c5c6-d38a313ca10d [Follower]" leader= testing.go:149: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.362Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: entering follower state: follower="Node at 31783a95-4ae1-8861-c5c6-d38a313ca10d [Follower]" leader= testLoggerAdapter verbose: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.429Z [WARN] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.429Z [WARN] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.430Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: entering candidate state: node="Node at 31783a95-4ae1-8861-c5c6-d38a313ca10d [Candidate]" term=2 testing.go:149: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.430Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: entering candidate state: node="Node at 31783a95-4ae1-8861-c5c6-d38a313ca10d [Candidate]" term=2 testLoggerAdapter verbose: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.430Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: election won: tally=1 testing.go:149: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.430Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: election won: tally=1 testLoggerAdapter verbose: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.430Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: entering leader state: leader="Node at 31783a95-4ae1-8861-c5c6-d38a313ca10d [Leader]" testing.go:149: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.430Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: entering leader state: leader="Node at 31783a95-4ae1-8861-c5c6-d38a313ca10d [Leader]" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:59.535Z [INFO] stable state for Leader reached at 2024-01-09 05:09:59.430248822 +0000 UTC m=+10.683299941 (1 nodes), 68.225783ms from start of poll, 68.436468ms from cluster start. Timeout at 2024-01-09 05:09:59.535368354 +0000 UTC m=+10.788419555, 105.119614ms after stability testing.go:149: cluster: 2024-01-09T05:09:59.535Z [INFO] stable state for Leader reached at 2024-01-09 05:09:59.430248822 +0000 UTC m=+10.683299941 (1 nodes), 68.225783ms from start of poll, 68.436468ms from cluster start. Timeout at 2024-01-09 05:09:59.535368354 +0000 UTC m=+10.788419555, 105.119614ms after stability testLoggerAdapter verbose: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.538Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: starting snapshot up to: index=102 testing.go:149: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.538Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: starting snapshot up to: index=102 testing.go:153: 2024-01-09T05:09:59.538Z [INFO] creating new snapshot: path=/tmp/raft833869084/snapshots/2-102-1704776999538.tmp testLoggerAdapter verbose: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.549Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: compacting logs: from=1 to=92 testing.go:149: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.549Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.549Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: snapshot complete up to: index=102 testing.go:149: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.549Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: snapshot complete up to: index=102 testLoggerAdapter verbose: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.549Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: restored from snapshot: id=2-102-1704776999538 testing.go:149: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.549Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: restored from snapshot: id=2-102-1704776999538 testLoggerAdapter verbose: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.549Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31783a95-4ae1-8861-c5c6-d38a313ca10d Address:31783a95-4ae1-8861-c5c6-d38a313ca10d}]" testing.go:149: server-31783a95-4ae1-8861-c5c6-d38a313ca10d: 2024-01-09T05:09:59.549Z [INFO] server-31783a95-4ae1-8861-c5c6-d38a313ca10d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31783a95-4ae1-8861-c5c6-d38a313ca10d Address:31783a95-4ae1-8861-c5c6-d38a313ca10d}]" --- PASS: TestRaft_SnapshotRestore (0.19s) === RUN TestRaft_NoRestoreOnStart testLoggerAdapter verbose: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.550Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5bda9bf0-2bdf-11b5-a986-342347309877 Address:5bda9bf0-2bdf-11b5-a986-342347309877}]" testing.go:149: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.550Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5bda9bf0-2bdf-11b5-a986-342347309877 Address:5bda9bf0-2bdf-11b5-a986-342347309877}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:59.550Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:09:59.550Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.550Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: entering follower state: follower="Node at 5bda9bf0-2bdf-11b5-a986-342347309877 [Follower]" leader= testing.go:149: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.550Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: entering follower state: follower="Node at 5bda9bf0-2bdf-11b5-a986-342347309877 [Follower]" leader= testLoggerAdapter verbose: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.647Z [WARN] server-5bda9bf0-2bdf-11b5-a986-342347309877: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.647Z [WARN] server-5bda9bf0-2bdf-11b5-a986-342347309877: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.647Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: entering candidate state: node="Node at 5bda9bf0-2bdf-11b5-a986-342347309877 [Candidate]" term=2 testing.go:149: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.647Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: entering candidate state: node="Node at 5bda9bf0-2bdf-11b5-a986-342347309877 [Candidate]" term=2 testLoggerAdapter verbose: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.648Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: election won: tally=1 testing.go:149: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.648Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: election won: tally=1 testLoggerAdapter verbose: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.648Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: entering leader state: leader="Node at 5bda9bf0-2bdf-11b5-a986-342347309877 [Leader]" testing.go:149: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.648Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: entering leader state: leader="Node at 5bda9bf0-2bdf-11b5-a986-342347309877 [Leader]" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:59.753Z [INFO] stable state for Leader reached at 2024-01-09 05:09:59.648225691 +0000 UTC m=+10.901276853 (1 nodes), 97.627937ms from start of poll, 97.811798ms from cluster start. Timeout at 2024-01-09 05:09:59.753301073 +0000 UTC m=+11.006352095, 105.075242ms after stability testing.go:149: cluster: 2024-01-09T05:09:59.753Z [INFO] stable state for Leader reached at 2024-01-09 05:09:59.648225691 +0000 UTC m=+10.901276853 (1 nodes), 97.627937ms from start of poll, 97.811798ms from cluster start. Timeout at 2024-01-09 05:09:59.753301073 +0000 UTC m=+11.006352095, 105.075242ms after stability testLoggerAdapter verbose: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.754Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: starting snapshot up to: index=102 testing.go:149: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.754Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: starting snapshot up to: index=102 testing.go:153: 2024-01-09T05:09:59.754Z [INFO] creating new snapshot: path=/tmp/raft460926126/snapshots/2-102-1704776999754.tmp testLoggerAdapter verbose: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.762Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: compacting logs: from=1 to=92 testing.go:149: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.762Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.762Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: snapshot complete up to: index=102 testing.go:149: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.762Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: snapshot complete up to: index=102 testLoggerAdapter verbose: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.762Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5bda9bf0-2bdf-11b5-a986-342347309877 Address:5bda9bf0-2bdf-11b5-a986-342347309877}]" testing.go:149: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.762Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5bda9bf0-2bdf-11b5-a986-342347309877 Address:5bda9bf0-2bdf-11b5-a986-342347309877}]" --- PASS: TestRaft_NoRestoreOnStart (0.21s) === RUN TestRaft_SnapshotRestore_PeerChange raft_test.go:1026: Disabled unreliable test --- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s) === RUN TestRaft_AutoSnapshot raft_test.go:1132: Disabled unreliable test --- SKIP: TestRaft_AutoSnapshot (0.00s) === RUN TestRaft_UserSnapshot testLoggerAdapter verbose: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.763Z [INFO] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06 Address:b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06}]" testing.go:149: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.763Z [INFO] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06 Address:b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:59.763Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:09:59.763Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.763Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: entering follower state: follower="Node at 5bda9bf0-2bdf-11b5-a986-342347309877 [Follower]" leader= testLoggerAdapter verbose: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.763Z [INFO] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: entering follower state: follower="Node at b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06 [Follower]" leader= testing.go:149: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.763Z [INFO] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: entering follower state: follower="Node at b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06 [Follower]" leader= testLoggerAdapter verbose: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.822Z [WARN] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.822Z [WARN] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.822Z [INFO] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: entering candidate state: node="Node at b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06 [Candidate]" term=2 testing.go:149: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.822Z [INFO] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: entering candidate state: node="Node at b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06 [Candidate]" term=2 testLoggerAdapter verbose: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.822Z [INFO] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: election won: tally=1 testing.go:149: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.822Z [INFO] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: election won: tally=1 testLoggerAdapter verbose: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.822Z [INFO] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: entering leader state: leader="Node at b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06 [Leader]" testing.go:149: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.822Z [INFO] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: entering leader state: leader="Node at b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06 [Leader]" testLoggerAdapter verbose: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.835Z [WARN] server-5bda9bf0-2bdf-11b5-a986-342347309877: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.835Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: entering candidate state: node="Node at 5bda9bf0-2bdf-11b5-a986-342347309877 [Candidate]" term=3 testLoggerAdapter verbose: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.835Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: election won: tally=1 testLoggerAdapter verbose: server-5bda9bf0-2bdf-11b5-a986-342347309877: 2024-01-09T05:09:59.835Z [INFO] server-5bda9bf0-2bdf-11b5-a986-342347309877: entering leader state: leader="Node at 5bda9bf0-2bdf-11b5-a986-342347309877 [Leader]" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:59.927Z [INFO] stable state for Leader reached at 2024-01-09 05:09:59.822760144 +0000 UTC m=+11.075811216 (1 nodes), 59.254034ms from start of poll, 59.390684ms from cluster start. Timeout at 2024-01-09 05:09:59.927858419 +0000 UTC m=+11.180909549, 105.098333ms after stability testing.go:149: cluster: 2024-01-09T05:09:59.927Z [INFO] stable state for Leader reached at 2024-01-09 05:09:59.822760144 +0000 UTC m=+11.075811216 (1 nodes), 59.254034ms from start of poll, 59.390684ms from cluster start. Timeout at 2024-01-09 05:09:59.927858419 +0000 UTC m=+11.180909549, 105.098333ms after stability testLoggerAdapter verbose: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.928Z [ERROR] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: failed to take snapshot: error="nothing new to snapshot" testing.go:149: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.928Z [ERROR] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: failed to take snapshot: error="nothing new to snapshot" testLoggerAdapter verbose: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.928Z [INFO] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: starting snapshot up to: index=12 testing.go:149: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.928Z [INFO] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: starting snapshot up to: index=12 testing.go:153: 2024-01-09T05:09:59.928Z [INFO] creating new snapshot: path=/tmp/raft285144592/snapshots/2-12-1704776999928.tmp testLoggerAdapter verbose: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.947Z [INFO] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: compacting logs: from=1 to=2 testing.go:149: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.947Z [INFO] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: compacting logs: from=1 to=2 testLoggerAdapter verbose: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.947Z [INFO] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: snapshot complete up to: index=12 testing.go:149: server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: 2024-01-09T05:09:59.947Z [INFO] server-b8f0b984-0dd8-a531-6d9a-e1da4d2f9b06: snapshot complete up to: index=12 --- PASS: TestRaft_UserSnapshot (0.19s) === RUN TestRaft_UserRestore testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:09:59.951Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a68b9448-5aac-228e-387f-635da2a03624 Address:a68b9448-5aac-228e-387f-635da2a03624} {Suffrage:Voter ID:server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9 Address:39d4e4e1-1ef2-2bb4-7d89-481da9d138f9} {Suffrage:Voter ID:server-7b17c738-d926-4a54-385d-51c86c237c47 Address:7b17c738-d926-4a54-385d-51c86c237c47}]" testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:09:59.951Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a68b9448-5aac-228e-387f-635da2a03624 Address:a68b9448-5aac-228e-387f-635da2a03624} {Suffrage:Voter ID:server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9 Address:39d4e4e1-1ef2-2bb4-7d89-481da9d138f9} {Suffrage:Voter ID:server-7b17c738-d926-4a54-385d-51c86c237c47 Address:7b17c738-d926-4a54-385d-51c86c237c47}]" testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:09:59.952Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: entering follower state: follower="Node at a68b9448-5aac-228e-387f-635da2a03624 [Follower]" leader= testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:09:59.952Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: entering follower state: follower="Node at a68b9448-5aac-228e-387f-635da2a03624 [Follower]" leader= testLoggerAdapter verbose: server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9: 2024-01-09T05:09:59.952Z [INFO] server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a68b9448-5aac-228e-387f-635da2a03624 Address:a68b9448-5aac-228e-387f-635da2a03624} {Suffrage:Voter ID:server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9 Address:39d4e4e1-1ef2-2bb4-7d89-481da9d138f9} {Suffrage:Voter ID:server-7b17c738-d926-4a54-385d-51c86c237c47 Address:7b17c738-d926-4a54-385d-51c86c237c47}]" testing.go:149: server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9: 2024-01-09T05:09:59.952Z [INFO] server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a68b9448-5aac-228e-387f-635da2a03624 Address:a68b9448-5aac-228e-387f-635da2a03624} {Suffrage:Voter ID:server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9 Address:39d4e4e1-1ef2-2bb4-7d89-481da9d138f9} {Suffrage:Voter ID:server-7b17c738-d926-4a54-385d-51c86c237c47 Address:7b17c738-d926-4a54-385d-51c86c237c47}]" testLoggerAdapter verbose: server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9: 2024-01-09T05:09:59.953Z [INFO] server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9: entering follower state: follower="Node at 39d4e4e1-1ef2-2bb4-7d89-481da9d138f9 [Follower]" leader= testing.go:149: server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9: 2024-01-09T05:09:59.953Z [INFO] server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9: entering follower state: follower="Node at 39d4e4e1-1ef2-2bb4-7d89-481da9d138f9 [Follower]" leader= testLoggerAdapter verbose: server-7b17c738-d926-4a54-385d-51c86c237c47: 2024-01-09T05:09:59.954Z [INFO] server-7b17c738-d926-4a54-385d-51c86c237c47: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a68b9448-5aac-228e-387f-635da2a03624 Address:a68b9448-5aac-228e-387f-635da2a03624} {Suffrage:Voter ID:server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9 Address:39d4e4e1-1ef2-2bb4-7d89-481da9d138f9} {Suffrage:Voter ID:server-7b17c738-d926-4a54-385d-51c86c237c47 Address:7b17c738-d926-4a54-385d-51c86c237c47}]" testing.go:149: server-7b17c738-d926-4a54-385d-51c86c237c47: 2024-01-09T05:09:59.954Z [INFO] server-7b17c738-d926-4a54-385d-51c86c237c47: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a68b9448-5aac-228e-387f-635da2a03624 Address:a68b9448-5aac-228e-387f-635da2a03624} {Suffrage:Voter ID:server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9 Address:39d4e4e1-1ef2-2bb4-7d89-481da9d138f9} {Suffrage:Voter ID:server-7b17c738-d926-4a54-385d-51c86c237c47 Address:7b17c738-d926-4a54-385d-51c86c237c47}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:09:59.954Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:09:59.954Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-7b17c738-d926-4a54-385d-51c86c237c47: 2024-01-09T05:09:59.954Z [INFO] server-7b17c738-d926-4a54-385d-51c86c237c47: entering follower state: follower="Node at 7b17c738-d926-4a54-385d-51c86c237c47 [Follower]" leader= testing.go:149: server-7b17c738-d926-4a54-385d-51c86c237c47: 2024-01-09T05:09:59.954Z [INFO] server-7b17c738-d926-4a54-385d-51c86c237c47: entering follower state: follower="Node at 7b17c738-d926-4a54-385d-51c86c237c47 [Follower]" leader= testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.024Z [WARN] server-a68b9448-5aac-228e-387f-635da2a03624: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.024Z [WARN] server-a68b9448-5aac-228e-387f-635da2a03624: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.024Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: entering candidate state: node="Node at a68b9448-5aac-228e-387f-635da2a03624 [Candidate]" term=2 testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.024Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: entering candidate state: node="Node at a68b9448-5aac-228e-387f-635da2a03624 [Candidate]" term=2 testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.024Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: election won: tally=2 testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.024Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: election won: tally=2 testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.024Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: entering leader state: leader="Node at a68b9448-5aac-228e-387f-635da2a03624 [Leader]" testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.024Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: entering leader state: leader="Node at a68b9448-5aac-228e-387f-635da2a03624 [Leader]" testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.025Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: added peer, starting replication: peer=server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9 testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.025Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: added peer, starting replication: peer=server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9 testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.025Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: added peer, starting replication: peer=server-7b17c738-d926-4a54-385d-51c86c237c47 testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.025Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: added peer, starting replication: peer=server-7b17c738-d926-4a54-385d-51c86c237c47 testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.025Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: pipelining replication: peer="{Voter server-7b17c738-d926-4a54-385d-51c86c237c47 7b17c738-d926-4a54-385d-51c86c237c47}" testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.025Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: pipelining replication: peer="{Voter server-7b17c738-d926-4a54-385d-51c86c237c47 7b17c738-d926-4a54-385d-51c86c237c47}" testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.025Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: pipelining replication: peer="{Voter server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9 39d4e4e1-1ef2-2bb4-7d89-481da9d138f9}" testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.025Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: pipelining replication: peer="{Voter server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9 39d4e4e1-1ef2-2bb4-7d89-481da9d138f9}" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:00.130Z [INFO] stable state for Leader reached at 2024-01-09 05:10:00.025329694 +0000 UTC m=+11.278380755 (1 nodes), 71.166644ms from start of poll, 74.778508ms from cluster start. Timeout at 2024-01-09 05:10:00.130391623 +0000 UTC m=+11.383442610, 105.061855ms after stability testing.go:149: cluster: 2024-01-09T05:10:00.130Z [INFO] stable state for Leader reached at 2024-01-09 05:10:00.025329694 +0000 UTC m=+11.278380755 (1 nodes), 71.166644ms from start of poll, 74.778508ms from cluster start. Timeout at 2024-01-09 05:10:00.130391623 +0000 UTC m=+11.383442610, 105.061855ms after stability testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.131Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: starting snapshot up to: index=12 testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.131Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: starting snapshot up to: index=12 testing.go:153: 2024-01-09T05:10:00.131Z [INFO] creating new snapshot: path=/tmp/raft072134594/snapshots/2-12-1704777000131.tmp testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.138Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: snapshot complete up to: index=12 testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.138Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: snapshot complete up to: index=12 testing.go:153: 2024-01-09T05:10:00.138Z [INFO] creating new snapshot: path=/tmp/raft072134594/snapshots/2-23-1704777000138.tmp testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.147Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: copied to local snapshot: bytes=71 testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.147Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.147Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: restored user snapshot: index=1 testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.147Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: restored user snapshot: index=1 testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.147Z [ERROR] server-a68b9448-5aac-228e-387f-635da2a03624: failed to get log: index=23 error="log not found" testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.147Z [ERROR] server-a68b9448-5aac-228e-387f-635da2a03624: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.147Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: aborting pipeline replication: peer="{Voter server-7b17c738-d926-4a54-385d-51c86c237c47 7b17c738-d926-4a54-385d-51c86c237c47}" testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.147Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: aborting pipeline replication: peer="{Voter server-7b17c738-d926-4a54-385d-51c86c237c47 7b17c738-d926-4a54-385d-51c86c237c47}" testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.147Z [ERROR] server-a68b9448-5aac-228e-387f-635da2a03624: failed to get log: index=23 error="log not found" testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.147Z [ERROR] server-a68b9448-5aac-228e-387f-635da2a03624: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.147Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: aborting pipeline replication: peer="{Voter server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9 39d4e4e1-1ef2-2bb4-7d89-481da9d138f9}" testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.147Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: aborting pipeline replication: peer="{Voter server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9 39d4e4e1-1ef2-2bb4-7d89-481da9d138f9}" testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.155Z [ERROR] server-a68b9448-5aac-228e-387f-635da2a03624: failed to get log: index=23 error="log not found" testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.155Z [ERROR] server-a68b9448-5aac-228e-387f-635da2a03624: failed to get log: index=23 error="log not found" testing.go:153: 2024-01-09T05:10:00.156Z [INFO] creating new snapshot: path=/tmp/raft968478230/snapshots/2-23-1704777000156.tmp testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.158Z [ERROR] server-a68b9448-5aac-228e-387f-635da2a03624: failed to get log: index=23 error="log not found" testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.158Z [ERROR] server-a68b9448-5aac-228e-387f-635da2a03624: failed to get log: index=23 error="log not found" testing.go:153: 2024-01-09T05:10:00.158Z [INFO] creating new snapshot: path=/tmp/raft182449732/snapshots/2-23-1704777000158.tmp testLoggerAdapter verbose: server-7b17c738-d926-4a54-385d-51c86c237c47: 2024-01-09T05:10:00.183Z [INFO] server-7b17c738-d926-4a54-385d-51c86c237c47: copied to local snapshot: bytes=71 testing.go:149: server-7b17c738-d926-4a54-385d-51c86c237c47: 2024-01-09T05:10:00.183Z [INFO] server-7b17c738-d926-4a54-385d-51c86c237c47: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-7b17c738-d926-4a54-385d-51c86c237c47: 2024-01-09T05:10:00.183Z [INFO] server-7b17c738-d926-4a54-385d-51c86c237c47: Installed remote snapshot testing.go:149: server-7b17c738-d926-4a54-385d-51c86c237c47: 2024-01-09T05:10:00.183Z [INFO] server-7b17c738-d926-4a54-385d-51c86c237c47: Installed remote snapshot testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.184Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: pipelining replication: peer="{Voter server-7b17c738-d926-4a54-385d-51c86c237c47 7b17c738-d926-4a54-385d-51c86c237c47}" testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.184Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: pipelining replication: peer="{Voter server-7b17c738-d926-4a54-385d-51c86c237c47 7b17c738-d926-4a54-385d-51c86c237c47}" testLoggerAdapter verbose: server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9: 2024-01-09T05:10:00.184Z [INFO] server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9: copied to local snapshot: bytes=71 testing.go:149: server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9: 2024-01-09T05:10:00.184Z [INFO] server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9: 2024-01-09T05:10:00.185Z [INFO] server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9: Installed remote snapshot testing.go:149: server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9: 2024-01-09T05:10:00.185Z [INFO] server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9: Installed remote snapshot testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.185Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: pipelining replication: peer="{Voter server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9 39d4e4e1-1ef2-2bb4-7d89-481da9d138f9}" testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.185Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: pipelining replication: peer="{Voter server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9 39d4e4e1-1ef2-2bb4-7d89-481da9d138f9}" testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.200Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: aborting pipeline replication: peer="{Voter server-7b17c738-d926-4a54-385d-51c86c237c47 7b17c738-d926-4a54-385d-51c86c237c47}" testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.200Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: aborting pipeline replication: peer="{Voter server-7b17c738-d926-4a54-385d-51c86c237c47 7b17c738-d926-4a54-385d-51c86c237c47}" testLoggerAdapter verbose: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.200Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: aborting pipeline replication: peer="{Voter server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9 39d4e4e1-1ef2-2bb4-7d89-481da9d138f9}" testing.go:149: server-a68b9448-5aac-228e-387f-635da2a03624: 2024-01-09T05:10:00.200Z [INFO] server-a68b9448-5aac-228e-387f-635da2a03624: aborting pipeline replication: peer="{Voter server-39d4e4e1-1ef2-2bb4-7d89-481da9d138f9 39d4e4e1-1ef2-2bb4-7d89-481da9d138f9}" testLoggerAdapter verbose: server-ec231c35-cbda-d116-0747-707f226cf159: 2024-01-09T05:10:00.202Z [INFO] server-ec231c35-cbda-d116-0747-707f226cf159: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ec231c35-cbda-d116-0747-707f226cf159 Address:ec231c35-cbda-d116-0747-707f226cf159} {Suffrage:Voter ID:server-bbfdd75e-6d44-738b-78d8-87f6425a0d08 Address:bbfdd75e-6d44-738b-78d8-87f6425a0d08} {Suffrage:Voter ID:server-bcaf3bbb-754c-2939-4a43-462cc5e29f87 Address:bcaf3bbb-754c-2939-4a43-462cc5e29f87}]" testing.go:149: server-ec231c35-cbda-d116-0747-707f226cf159: 2024-01-09T05:10:00.202Z [INFO] server-ec231c35-cbda-d116-0747-707f226cf159: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ec231c35-cbda-d116-0747-707f226cf159 Address:ec231c35-cbda-d116-0747-707f226cf159} {Suffrage:Voter ID:server-bbfdd75e-6d44-738b-78d8-87f6425a0d08 Address:bbfdd75e-6d44-738b-78d8-87f6425a0d08} {Suffrage:Voter ID:server-bcaf3bbb-754c-2939-4a43-462cc5e29f87 Address:bcaf3bbb-754c-2939-4a43-462cc5e29f87}]" testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.202Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ec231c35-cbda-d116-0747-707f226cf159 Address:ec231c35-cbda-d116-0747-707f226cf159} {Suffrage:Voter ID:server-bbfdd75e-6d44-738b-78d8-87f6425a0d08 Address:bbfdd75e-6d44-738b-78d8-87f6425a0d08} {Suffrage:Voter ID:server-bcaf3bbb-754c-2939-4a43-462cc5e29f87 Address:bcaf3bbb-754c-2939-4a43-462cc5e29f87}]" testLoggerAdapter verbose: server-ec231c35-cbda-d116-0747-707f226cf159: 2024-01-09T05:10:00.202Z [INFO] server-ec231c35-cbda-d116-0747-707f226cf159: entering follower state: follower="Node at ec231c35-cbda-d116-0747-707f226cf159 [Follower]" leader= testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.202Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ec231c35-cbda-d116-0747-707f226cf159 Address:ec231c35-cbda-d116-0747-707f226cf159} {Suffrage:Voter ID:server-bbfdd75e-6d44-738b-78d8-87f6425a0d08 Address:bbfdd75e-6d44-738b-78d8-87f6425a0d08} {Suffrage:Voter ID:server-bcaf3bbb-754c-2939-4a43-462cc5e29f87 Address:bcaf3bbb-754c-2939-4a43-462cc5e29f87}]" testLoggerAdapter verbose: server-bcaf3bbb-754c-2939-4a43-462cc5e29f87: 2024-01-09T05:10:00.203Z [INFO] server-bcaf3bbb-754c-2939-4a43-462cc5e29f87: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ec231c35-cbda-d116-0747-707f226cf159 Address:ec231c35-cbda-d116-0747-707f226cf159} {Suffrage:Voter ID:server-bbfdd75e-6d44-738b-78d8-87f6425a0d08 Address:bbfdd75e-6d44-738b-78d8-87f6425a0d08} {Suffrage:Voter ID:server-bcaf3bbb-754c-2939-4a43-462cc5e29f87 Address:bcaf3bbb-754c-2939-4a43-462cc5e29f87}]" testing.go:149: server-bcaf3bbb-754c-2939-4a43-462cc5e29f87: 2024-01-09T05:10:00.203Z [INFO] server-bcaf3bbb-754c-2939-4a43-462cc5e29f87: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ec231c35-cbda-d116-0747-707f226cf159 Address:ec231c35-cbda-d116-0747-707f226cf159} {Suffrage:Voter ID:server-bbfdd75e-6d44-738b-78d8-87f6425a0d08 Address:bbfdd75e-6d44-738b-78d8-87f6425a0d08} {Suffrage:Voter ID:server-bcaf3bbb-754c-2939-4a43-462cc5e29f87 Address:bcaf3bbb-754c-2939-4a43-462cc5e29f87}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:00.203Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:10:00.203Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-ec231c35-cbda-d116-0747-707f226cf159: 2024-01-09T05:10:00.202Z [INFO] server-ec231c35-cbda-d116-0747-707f226cf159: entering follower state: follower="Node at ec231c35-cbda-d116-0747-707f226cf159 [Follower]" leader= testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.203Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: entering follower state: follower="Node at bbfdd75e-6d44-738b-78d8-87f6425a0d08 [Follower]" leader= testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.203Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: entering follower state: follower="Node at bbfdd75e-6d44-738b-78d8-87f6425a0d08 [Follower]" leader= testLoggerAdapter verbose: server-bcaf3bbb-754c-2939-4a43-462cc5e29f87: 2024-01-09T05:10:00.203Z [INFO] server-bcaf3bbb-754c-2939-4a43-462cc5e29f87: entering follower state: follower="Node at bcaf3bbb-754c-2939-4a43-462cc5e29f87 [Follower]" leader= testing.go:149: server-bcaf3bbb-754c-2939-4a43-462cc5e29f87: 2024-01-09T05:10:00.203Z [INFO] server-bcaf3bbb-754c-2939-4a43-462cc5e29f87: entering follower state: follower="Node at bcaf3bbb-754c-2939-4a43-462cc5e29f87 [Follower]" leader= testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.257Z [WARN] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.257Z [WARN] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.258Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: entering candidate state: node="Node at bbfdd75e-6d44-738b-78d8-87f6425a0d08 [Candidate]" term=2 testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.258Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: entering candidate state: node="Node at bbfdd75e-6d44-738b-78d8-87f6425a0d08 [Candidate]" term=2 testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.258Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: election won: tally=2 testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.258Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: election won: tally=2 testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.258Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: entering leader state: leader="Node at bbfdd75e-6d44-738b-78d8-87f6425a0d08 [Leader]" testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.258Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: entering leader state: leader="Node at bbfdd75e-6d44-738b-78d8-87f6425a0d08 [Leader]" testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.258Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: added peer, starting replication: peer=server-ec231c35-cbda-d116-0747-707f226cf159 testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.258Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: added peer, starting replication: peer=server-ec231c35-cbda-d116-0747-707f226cf159 testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.258Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: added peer, starting replication: peer=server-bcaf3bbb-754c-2939-4a43-462cc5e29f87 testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.258Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: added peer, starting replication: peer=server-bcaf3bbb-754c-2939-4a43-462cc5e29f87 testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.258Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: pipelining replication: peer="{Voter server-bcaf3bbb-754c-2939-4a43-462cc5e29f87 bcaf3bbb-754c-2939-4a43-462cc5e29f87}" testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.258Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: pipelining replication: peer="{Voter server-bcaf3bbb-754c-2939-4a43-462cc5e29f87 bcaf3bbb-754c-2939-4a43-462cc5e29f87}" testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.258Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: pipelining replication: peer="{Voter server-ec231c35-cbda-d116-0747-707f226cf159 ec231c35-cbda-d116-0747-707f226cf159}" testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.258Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: pipelining replication: peer="{Voter server-ec231c35-cbda-d116-0747-707f226cf159 ec231c35-cbda-d116-0747-707f226cf159}" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:00.363Z [INFO] stable state for Leader reached at 2024-01-09 05:10:00.258370623 +0000 UTC m=+11.511421812 (1 nodes), 54.669167ms from start of poll, 55.771458ms from cluster start. Timeout at 2024-01-09 05:10:00.363533126 +0000 UTC m=+11.616584229, 105.162417ms after stability testing.go:149: cluster: 2024-01-09T05:10:00.363Z [INFO] stable state for Leader reached at 2024-01-09 05:10:00.258370623 +0000 UTC m=+11.511421812 (1 nodes), 54.669167ms from start of poll, 55.771458ms from cluster start. Timeout at 2024-01-09 05:10:00.363533126 +0000 UTC m=+11.616584229, 105.162417ms after stability testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.364Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: starting snapshot up to: index=12 testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.364Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: starting snapshot up to: index=12 testing.go:153: 2024-01-09T05:10:00.364Z [INFO] creating new snapshot: path=/tmp/raft016711082/snapshots/2-12-1704777000364.tmp testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.371Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: snapshot complete up to: index=12 testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.371Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: snapshot complete up to: index=12 testing.go:153: 2024-01-09T05:10:00.372Z [INFO] creating new snapshot: path=/tmp/raft016711082/snapshots/2-23-1704777000372.tmp testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.381Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: copied to local snapshot: bytes=71 testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.381Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.381Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: restored user snapshot: index=1 testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.381Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: restored user snapshot: index=1 testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.381Z [ERROR] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: failed to get log: index=23 error="log not found" testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.381Z [ERROR] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.381Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: aborting pipeline replication: peer="{Voter server-bcaf3bbb-754c-2939-4a43-462cc5e29f87 bcaf3bbb-754c-2939-4a43-462cc5e29f87}" testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.381Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: aborting pipeline replication: peer="{Voter server-bcaf3bbb-754c-2939-4a43-462cc5e29f87 bcaf3bbb-754c-2939-4a43-462cc5e29f87}" testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.381Z [ERROR] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: failed to get log: index=23 error="log not found" testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.381Z [ERROR] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.381Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: aborting pipeline replication: peer="{Voter server-ec231c35-cbda-d116-0747-707f226cf159 ec231c35-cbda-d116-0747-707f226cf159}" testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.381Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: aborting pipeline replication: peer="{Voter server-ec231c35-cbda-d116-0747-707f226cf159 ec231c35-cbda-d116-0747-707f226cf159}" testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.388Z [ERROR] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: failed to get log: index=23 error="log not found" testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.388Z [ERROR] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: failed to get log: index=23 error="log not found" testing.go:153: 2024-01-09T05:10:00.388Z [INFO] creating new snapshot: path=/tmp/raft039767480/snapshots/2-23-1704777000388.tmp testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.392Z [ERROR] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: failed to get log: index=23 error="log not found" testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.392Z [ERROR] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: failed to get log: index=23 error="log not found" testing.go:153: 2024-01-09T05:10:00.392Z [INFO] creating new snapshot: path=/tmp/raft246478444/snapshots/2-23-1704777000392.tmp testLoggerAdapter verbose: server-ec231c35-cbda-d116-0747-707f226cf159: 2024-01-09T05:10:00.400Z [INFO] server-ec231c35-cbda-d116-0747-707f226cf159: copied to local snapshot: bytes=71 testing.go:149: server-ec231c35-cbda-d116-0747-707f226cf159: 2024-01-09T05:10:00.400Z [INFO] server-ec231c35-cbda-d116-0747-707f226cf159: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-ec231c35-cbda-d116-0747-707f226cf159: 2024-01-09T05:10:00.401Z [INFO] server-ec231c35-cbda-d116-0747-707f226cf159: Installed remote snapshot testing.go:149: server-ec231c35-cbda-d116-0747-707f226cf159: 2024-01-09T05:10:00.401Z [INFO] server-ec231c35-cbda-d116-0747-707f226cf159: Installed remote snapshot testLoggerAdapter verbose: server-ec231c35-cbda-d116-0747-707f226cf159: 2024-01-09T05:10:00.401Z [WARN] server-ec231c35-cbda-d116-0747-707f226cf159: skipping application of old log: index=22 testing.go:149: server-ec231c35-cbda-d116-0747-707f226cf159: 2024-01-09T05:10:00.401Z [WARN] server-ec231c35-cbda-d116-0747-707f226cf159: skipping application of old log: index=22 testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.401Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: pipelining replication: peer="{Voter server-ec231c35-cbda-d116-0747-707f226cf159 ec231c35-cbda-d116-0747-707f226cf159}" testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.401Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: pipelining replication: peer="{Voter server-ec231c35-cbda-d116-0747-707f226cf159 ec231c35-cbda-d116-0747-707f226cf159}" testLoggerAdapter verbose: server-bcaf3bbb-754c-2939-4a43-462cc5e29f87: 2024-01-09T05:10:00.402Z [INFO] server-bcaf3bbb-754c-2939-4a43-462cc5e29f87: copied to local snapshot: bytes=71 testing.go:149: server-bcaf3bbb-754c-2939-4a43-462cc5e29f87: 2024-01-09T05:10:00.402Z [INFO] server-bcaf3bbb-754c-2939-4a43-462cc5e29f87: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-bcaf3bbb-754c-2939-4a43-462cc5e29f87: 2024-01-09T05:10:00.403Z [INFO] server-bcaf3bbb-754c-2939-4a43-462cc5e29f87: Installed remote snapshot testing.go:149: server-bcaf3bbb-754c-2939-4a43-462cc5e29f87: 2024-01-09T05:10:00.403Z [INFO] server-bcaf3bbb-754c-2939-4a43-462cc5e29f87: Installed remote snapshot testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.403Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: pipelining replication: peer="{Voter server-bcaf3bbb-754c-2939-4a43-462cc5e29f87 bcaf3bbb-754c-2939-4a43-462cc5e29f87}" testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.403Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: pipelining replication: peer="{Voter server-bcaf3bbb-754c-2939-4a43-462cc5e29f87 bcaf3bbb-754c-2939-4a43-462cc5e29f87}" testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.420Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: aborting pipeline replication: peer="{Voter server-bcaf3bbb-754c-2939-4a43-462cc5e29f87 bcaf3bbb-754c-2939-4a43-462cc5e29f87}" testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.420Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: aborting pipeline replication: peer="{Voter server-bcaf3bbb-754c-2939-4a43-462cc5e29f87 bcaf3bbb-754c-2939-4a43-462cc5e29f87}" testLoggerAdapter verbose: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.420Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: aborting pipeline replication: peer="{Voter server-ec231c35-cbda-d116-0747-707f226cf159 ec231c35-cbda-d116-0747-707f226cf159}" testing.go:149: server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: 2024-01-09T05:10:00.420Z [INFO] server-bbfdd75e-6d44-738b-78d8-87f6425a0d08: aborting pipeline replication: peer="{Voter server-ec231c35-cbda-d116-0747-707f226cf159 ec231c35-cbda-d116-0747-707f226cf159}" testLoggerAdapter verbose: server-e64a76d6-af6a-9995-419b-a7cad3a65539: 2024-01-09T05:10:00.427Z [INFO] server-e64a76d6-af6a-9995-419b-a7cad3a65539: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e64a76d6-af6a-9995-419b-a7cad3a65539 Address:e64a76d6-af6a-9995-419b-a7cad3a65539} {Suffrage:Voter ID:server-b5e31636-90a1-b17c-5e9f-69b7b408058f Address:b5e31636-90a1-b17c-5e9f-69b7b408058f} {Suffrage:Voter ID:server-82fff768-5fd2-c8d5-68b3-669b8047ea37 Address:82fff768-5fd2-c8d5-68b3-669b8047ea37}]" testing.go:149: server-e64a76d6-af6a-9995-419b-a7cad3a65539: 2024-01-09T05:10:00.427Z [INFO] server-e64a76d6-af6a-9995-419b-a7cad3a65539: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e64a76d6-af6a-9995-419b-a7cad3a65539 Address:e64a76d6-af6a-9995-419b-a7cad3a65539} {Suffrage:Voter ID:server-b5e31636-90a1-b17c-5e9f-69b7b408058f Address:b5e31636-90a1-b17c-5e9f-69b7b408058f} {Suffrage:Voter ID:server-82fff768-5fd2-c8d5-68b3-669b8047ea37 Address:82fff768-5fd2-c8d5-68b3-669b8047ea37}]" testLoggerAdapter verbose: server-b5e31636-90a1-b17c-5e9f-69b7b408058f: 2024-01-09T05:10:00.427Z [INFO] server-b5e31636-90a1-b17c-5e9f-69b7b408058f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e64a76d6-af6a-9995-419b-a7cad3a65539 Address:e64a76d6-af6a-9995-419b-a7cad3a65539} {Suffrage:Voter ID:server-b5e31636-90a1-b17c-5e9f-69b7b408058f Address:b5e31636-90a1-b17c-5e9f-69b7b408058f} {Suffrage:Voter ID:server-82fff768-5fd2-c8d5-68b3-669b8047ea37 Address:82fff768-5fd2-c8d5-68b3-669b8047ea37}]" testing.go:149: server-b5e31636-90a1-b17c-5e9f-69b7b408058f: 2024-01-09T05:10:00.427Z [INFO] server-b5e31636-90a1-b17c-5e9f-69b7b408058f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e64a76d6-af6a-9995-419b-a7cad3a65539 Address:e64a76d6-af6a-9995-419b-a7cad3a65539} {Suffrage:Voter ID:server-b5e31636-90a1-b17c-5e9f-69b7b408058f Address:b5e31636-90a1-b17c-5e9f-69b7b408058f} {Suffrage:Voter ID:server-82fff768-5fd2-c8d5-68b3-669b8047ea37 Address:82fff768-5fd2-c8d5-68b3-669b8047ea37}]" testLoggerAdapter verbose: server-e64a76d6-af6a-9995-419b-a7cad3a65539: 2024-01-09T05:10:00.427Z [INFO] server-e64a76d6-af6a-9995-419b-a7cad3a65539: entering follower state: follower="Node at e64a76d6-af6a-9995-419b-a7cad3a65539 [Follower]" leader= testing.go:149: server-e64a76d6-af6a-9995-419b-a7cad3a65539: 2024-01-09T05:10:00.427Z [INFO] server-e64a76d6-af6a-9995-419b-a7cad3a65539: entering follower state: follower="Node at e64a76d6-af6a-9995-419b-a7cad3a65539 [Follower]" leader= testLoggerAdapter verbose: server-b5e31636-90a1-b17c-5e9f-69b7b408058f: 2024-01-09T05:10:00.427Z [INFO] server-b5e31636-90a1-b17c-5e9f-69b7b408058f: entering follower state: follower="Node at b5e31636-90a1-b17c-5e9f-69b7b408058f [Follower]" leader= testing.go:149: server-b5e31636-90a1-b17c-5e9f-69b7b408058f: 2024-01-09T05:10:00.427Z [INFO] server-b5e31636-90a1-b17c-5e9f-69b7b408058f: entering follower state: follower="Node at b5e31636-90a1-b17c-5e9f-69b7b408058f [Follower]" leader= testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.428Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e64a76d6-af6a-9995-419b-a7cad3a65539 Address:e64a76d6-af6a-9995-419b-a7cad3a65539} {Suffrage:Voter ID:server-b5e31636-90a1-b17c-5e9f-69b7b408058f Address:b5e31636-90a1-b17c-5e9f-69b7b408058f} {Suffrage:Voter ID:server-82fff768-5fd2-c8d5-68b3-669b8047ea37 Address:82fff768-5fd2-c8d5-68b3-669b8047ea37}]" testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.428Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e64a76d6-af6a-9995-419b-a7cad3a65539 Address:e64a76d6-af6a-9995-419b-a7cad3a65539} {Suffrage:Voter ID:server-b5e31636-90a1-b17c-5e9f-69b7b408058f Address:b5e31636-90a1-b17c-5e9f-69b7b408058f} {Suffrage:Voter ID:server-82fff768-5fd2-c8d5-68b3-669b8047ea37 Address:82fff768-5fd2-c8d5-68b3-669b8047ea37}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:00.428Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:10:00.428Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.429Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: entering follower state: follower="Node at 82fff768-5fd2-c8d5-68b3-669b8047ea37 [Follower]" leader= testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.429Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: entering follower state: follower="Node at 82fff768-5fd2-c8d5-68b3-669b8047ea37 [Follower]" leader= testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.487Z [WARN] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.487Z [WARN] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.487Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: entering candidate state: node="Node at 82fff768-5fd2-c8d5-68b3-669b8047ea37 [Candidate]" term=2 testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.487Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: entering candidate state: node="Node at 82fff768-5fd2-c8d5-68b3-669b8047ea37 [Candidate]" term=2 testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.487Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: election won: tally=2 testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.487Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: election won: tally=2 testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.487Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: entering leader state: leader="Node at 82fff768-5fd2-c8d5-68b3-669b8047ea37 [Leader]" testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.487Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: entering leader state: leader="Node at 82fff768-5fd2-c8d5-68b3-669b8047ea37 [Leader]" testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.487Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: added peer, starting replication: peer=server-e64a76d6-af6a-9995-419b-a7cad3a65539 testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.487Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: added peer, starting replication: peer=server-e64a76d6-af6a-9995-419b-a7cad3a65539 testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.487Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: added peer, starting replication: peer=server-b5e31636-90a1-b17c-5e9f-69b7b408058f testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.487Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: added peer, starting replication: peer=server-b5e31636-90a1-b17c-5e9f-69b7b408058f testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.487Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: pipelining replication: peer="{Voter server-b5e31636-90a1-b17c-5e9f-69b7b408058f b5e31636-90a1-b17c-5e9f-69b7b408058f}" testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.487Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: pipelining replication: peer="{Voter server-b5e31636-90a1-b17c-5e9f-69b7b408058f b5e31636-90a1-b17c-5e9f-69b7b408058f}" testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.487Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: pipelining replication: peer="{Voter server-e64a76d6-af6a-9995-419b-a7cad3a65539 e64a76d6-af6a-9995-419b-a7cad3a65539}" testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.487Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: pipelining replication: peer="{Voter server-e64a76d6-af6a-9995-419b-a7cad3a65539 e64a76d6-af6a-9995-419b-a7cad3a65539}" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:00.593Z [INFO] stable state for Leader reached at 2024-01-09 05:10:00.488034825 +0000 UTC m=+11.741085915 (1 nodes), 59.449482ms from start of poll, 61.518318ms from cluster start. Timeout at 2024-01-09 05:10:00.593602385 +0000 UTC m=+11.846653638, 105.567723ms after stability testing.go:149: cluster: 2024-01-09T05:10:00.593Z [INFO] stable state for Leader reached at 2024-01-09 05:10:00.488034825 +0000 UTC m=+11.741085915 (1 nodes), 59.449482ms from start of poll, 61.518318ms from cluster start. Timeout at 2024-01-09 05:10:00.593602385 +0000 UTC m=+11.846653638, 105.567723ms after stability testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.594Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: starting snapshot up to: index=12 testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.594Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: starting snapshot up to: index=12 testing.go:153: 2024-01-09T05:10:00.594Z [INFO] creating new snapshot: path=/tmp/raft209861266/snapshots/2-12-1704777000594.tmp testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.603Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: snapshot complete up to: index=12 testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.603Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: snapshot complete up to: index=12 testing.go:153: 2024-01-09T05:10:00.603Z [INFO] creating new snapshot: path=/tmp/raft209861266/snapshots/2-23-1704777000603.tmp testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.611Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: copied to local snapshot: bytes=71 testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.611Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.612Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: restored user snapshot: index=1 testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.612Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: restored user snapshot: index=1 testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.612Z [ERROR] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: failed to get log: index=23 error="log not found" testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.612Z [ERROR] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.612Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: aborting pipeline replication: peer="{Voter server-e64a76d6-af6a-9995-419b-a7cad3a65539 e64a76d6-af6a-9995-419b-a7cad3a65539}" testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.612Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: aborting pipeline replication: peer="{Voter server-e64a76d6-af6a-9995-419b-a7cad3a65539 e64a76d6-af6a-9995-419b-a7cad3a65539}" testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.612Z [ERROR] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: failed to get log: index=23 error="log not found" testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.612Z [ERROR] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.612Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: aborting pipeline replication: peer="{Voter server-b5e31636-90a1-b17c-5e9f-69b7b408058f b5e31636-90a1-b17c-5e9f-69b7b408058f}" testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.612Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: aborting pipeline replication: peer="{Voter server-b5e31636-90a1-b17c-5e9f-69b7b408058f b5e31636-90a1-b17c-5e9f-69b7b408058f}" testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.619Z [ERROR] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: failed to get log: index=23 error="log not found" testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.619Z [ERROR] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: failed to get log: index=23 error="log not found" testing.go:153: 2024-01-09T05:10:00.620Z [INFO] creating new snapshot: path=/tmp/raft489713248/snapshots/2-23-1704777000620.tmp testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.622Z [ERROR] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: failed to get log: index=23 error="log not found" testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.622Z [ERROR] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: failed to get log: index=23 error="log not found" testing.go:153: 2024-01-09T05:10:00.623Z [INFO] creating new snapshot: path=/tmp/raft580825214/snapshots/2-23-1704777000623.tmp testLoggerAdapter verbose: server-b5e31636-90a1-b17c-5e9f-69b7b408058f: 2024-01-09T05:10:00.632Z [INFO] server-b5e31636-90a1-b17c-5e9f-69b7b408058f: copied to local snapshot: bytes=71 testing.go:149: server-b5e31636-90a1-b17c-5e9f-69b7b408058f: 2024-01-09T05:10:00.632Z [INFO] server-b5e31636-90a1-b17c-5e9f-69b7b408058f: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-e64a76d6-af6a-9995-419b-a7cad3a65539: 2024-01-09T05:10:00.633Z [INFO] server-e64a76d6-af6a-9995-419b-a7cad3a65539: copied to local snapshot: bytes=71 testing.go:149: server-e64a76d6-af6a-9995-419b-a7cad3a65539: 2024-01-09T05:10:00.633Z [INFO] server-e64a76d6-af6a-9995-419b-a7cad3a65539: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-e64a76d6-af6a-9995-419b-a7cad3a65539: 2024-01-09T05:10:00.633Z [INFO] server-e64a76d6-af6a-9995-419b-a7cad3a65539: Installed remote snapshot testing.go:149: server-e64a76d6-af6a-9995-419b-a7cad3a65539: 2024-01-09T05:10:00.633Z [INFO] server-e64a76d6-af6a-9995-419b-a7cad3a65539: Installed remote snapshot testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.633Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: pipelining replication: peer="{Voter server-e64a76d6-af6a-9995-419b-a7cad3a65539 e64a76d6-af6a-9995-419b-a7cad3a65539}" testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.633Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: pipelining replication: peer="{Voter server-e64a76d6-af6a-9995-419b-a7cad3a65539 e64a76d6-af6a-9995-419b-a7cad3a65539}" testLoggerAdapter verbose: server-b5e31636-90a1-b17c-5e9f-69b7b408058f: 2024-01-09T05:10:00.633Z [INFO] server-b5e31636-90a1-b17c-5e9f-69b7b408058f: Installed remote snapshot testing.go:149: server-b5e31636-90a1-b17c-5e9f-69b7b408058f: 2024-01-09T05:10:00.633Z [INFO] server-b5e31636-90a1-b17c-5e9f-69b7b408058f: Installed remote snapshot testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.633Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: pipelining replication: peer="{Voter server-b5e31636-90a1-b17c-5e9f-69b7b408058f b5e31636-90a1-b17c-5e9f-69b7b408058f}" testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.633Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: pipelining replication: peer="{Voter server-b5e31636-90a1-b17c-5e9f-69b7b408058f b5e31636-90a1-b17c-5e9f-69b7b408058f}" testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.651Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: aborting pipeline replication: peer="{Voter server-b5e31636-90a1-b17c-5e9f-69b7b408058f b5e31636-90a1-b17c-5e9f-69b7b408058f}" testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.651Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: aborting pipeline replication: peer="{Voter server-b5e31636-90a1-b17c-5e9f-69b7b408058f b5e31636-90a1-b17c-5e9f-69b7b408058f}" testLoggerAdapter verbose: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.651Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: aborting pipeline replication: peer="{Voter server-e64a76d6-af6a-9995-419b-a7cad3a65539 e64a76d6-af6a-9995-419b-a7cad3a65539}" testing.go:149: server-82fff768-5fd2-c8d5-68b3-669b8047ea37: 2024-01-09T05:10:00.651Z [INFO] server-82fff768-5fd2-c8d5-68b3-669b8047ea37: aborting pipeline replication: peer="{Voter server-e64a76d6-af6a-9995-419b-a7cad3a65539 e64a76d6-af6a-9995-419b-a7cad3a65539}" testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.653Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62b05e7b-869f-f84a-ab0d-250af0adb19a Address:62b05e7b-869f-f84a-ab0d-250af0adb19a} {Suffrage:Voter ID:server-f28f3437-45ec-cfb0-a8bb-291c891734ad Address:f28f3437-45ec-cfb0-a8bb-291c891734ad} {Suffrage:Voter ID:server-d64eed0d-cc3d-74e5-402b-904bfe0618d8 Address:d64eed0d-cc3d-74e5-402b-904bfe0618d8}]" testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.653Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62b05e7b-869f-f84a-ab0d-250af0adb19a Address:62b05e7b-869f-f84a-ab0d-250af0adb19a} {Suffrage:Voter ID:server-f28f3437-45ec-cfb0-a8bb-291c891734ad Address:f28f3437-45ec-cfb0-a8bb-291c891734ad} {Suffrage:Voter ID:server-d64eed0d-cc3d-74e5-402b-904bfe0618d8 Address:d64eed0d-cc3d-74e5-402b-904bfe0618d8}]" testLoggerAdapter verbose: server-f28f3437-45ec-cfb0-a8bb-291c891734ad: 2024-01-09T05:10:00.653Z [INFO] server-f28f3437-45ec-cfb0-a8bb-291c891734ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62b05e7b-869f-f84a-ab0d-250af0adb19a Address:62b05e7b-869f-f84a-ab0d-250af0adb19a} {Suffrage:Voter ID:server-f28f3437-45ec-cfb0-a8bb-291c891734ad Address:f28f3437-45ec-cfb0-a8bb-291c891734ad} {Suffrage:Voter ID:server-d64eed0d-cc3d-74e5-402b-904bfe0618d8 Address:d64eed0d-cc3d-74e5-402b-904bfe0618d8}]" testing.go:149: server-f28f3437-45ec-cfb0-a8bb-291c891734ad: 2024-01-09T05:10:00.653Z [INFO] server-f28f3437-45ec-cfb0-a8bb-291c891734ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62b05e7b-869f-f84a-ab0d-250af0adb19a Address:62b05e7b-869f-f84a-ab0d-250af0adb19a} {Suffrage:Voter ID:server-f28f3437-45ec-cfb0-a8bb-291c891734ad Address:f28f3437-45ec-cfb0-a8bb-291c891734ad} {Suffrage:Voter ID:server-d64eed0d-cc3d-74e5-402b-904bfe0618d8 Address:d64eed0d-cc3d-74e5-402b-904bfe0618d8}]" testLoggerAdapter verbose: server-d64eed0d-cc3d-74e5-402b-904bfe0618d8: 2024-01-09T05:10:00.654Z [INFO] server-d64eed0d-cc3d-74e5-402b-904bfe0618d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62b05e7b-869f-f84a-ab0d-250af0adb19a Address:62b05e7b-869f-f84a-ab0d-250af0adb19a} {Suffrage:Voter ID:server-f28f3437-45ec-cfb0-a8bb-291c891734ad Address:f28f3437-45ec-cfb0-a8bb-291c891734ad} {Suffrage:Voter ID:server-d64eed0d-cc3d-74e5-402b-904bfe0618d8 Address:d64eed0d-cc3d-74e5-402b-904bfe0618d8}]" testing.go:149: server-d64eed0d-cc3d-74e5-402b-904bfe0618d8: 2024-01-09T05:10:00.654Z [INFO] server-d64eed0d-cc3d-74e5-402b-904bfe0618d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62b05e7b-869f-f84a-ab0d-250af0adb19a Address:62b05e7b-869f-f84a-ab0d-250af0adb19a} {Suffrage:Voter ID:server-f28f3437-45ec-cfb0-a8bb-291c891734ad Address:f28f3437-45ec-cfb0-a8bb-291c891734ad} {Suffrage:Voter ID:server-d64eed0d-cc3d-74e5-402b-904bfe0618d8 Address:d64eed0d-cc3d-74e5-402b-904bfe0618d8}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:00.654Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:10:00.654Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.654Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: entering follower state: follower="Node at 62b05e7b-869f-f84a-ab0d-250af0adb19a [Follower]" leader= testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.654Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: entering follower state: follower="Node at 62b05e7b-869f-f84a-ab0d-250af0adb19a [Follower]" leader= testLoggerAdapter verbose: server-f28f3437-45ec-cfb0-a8bb-291c891734ad: 2024-01-09T05:10:00.654Z [INFO] server-f28f3437-45ec-cfb0-a8bb-291c891734ad: entering follower state: follower="Node at f28f3437-45ec-cfb0-a8bb-291c891734ad [Follower]" leader= testing.go:149: server-f28f3437-45ec-cfb0-a8bb-291c891734ad: 2024-01-09T05:10:00.654Z [INFO] server-f28f3437-45ec-cfb0-a8bb-291c891734ad: entering follower state: follower="Node at f28f3437-45ec-cfb0-a8bb-291c891734ad [Follower]" leader= testLoggerAdapter verbose: server-d64eed0d-cc3d-74e5-402b-904bfe0618d8: 2024-01-09T05:10:00.654Z [INFO] server-d64eed0d-cc3d-74e5-402b-904bfe0618d8: entering follower state: follower="Node at d64eed0d-cc3d-74e5-402b-904bfe0618d8 [Follower]" leader= testing.go:149: server-d64eed0d-cc3d-74e5-402b-904bfe0618d8: 2024-01-09T05:10:00.654Z [INFO] server-d64eed0d-cc3d-74e5-402b-904bfe0618d8: entering follower state: follower="Node at d64eed0d-cc3d-74e5-402b-904bfe0618d8 [Follower]" leader= testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.727Z [WARN] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.727Z [WARN] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.727Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: entering candidate state: node="Node at 62b05e7b-869f-f84a-ab0d-250af0adb19a [Candidate]" term=2 testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.727Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: entering candidate state: node="Node at 62b05e7b-869f-f84a-ab0d-250af0adb19a [Candidate]" term=2 testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.727Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: election won: tally=2 testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.727Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: election won: tally=2 testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.727Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: entering leader state: leader="Node at 62b05e7b-869f-f84a-ab0d-250af0adb19a [Leader]" testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.727Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: entering leader state: leader="Node at 62b05e7b-869f-f84a-ab0d-250af0adb19a [Leader]" testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.727Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: added peer, starting replication: peer=server-f28f3437-45ec-cfb0-a8bb-291c891734ad testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.727Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: added peer, starting replication: peer=server-f28f3437-45ec-cfb0-a8bb-291c891734ad testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.727Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: added peer, starting replication: peer=server-d64eed0d-cc3d-74e5-402b-904bfe0618d8 testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.727Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: added peer, starting replication: peer=server-d64eed0d-cc3d-74e5-402b-904bfe0618d8 testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.727Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: pipelining replication: peer="{Voter server-d64eed0d-cc3d-74e5-402b-904bfe0618d8 d64eed0d-cc3d-74e5-402b-904bfe0618d8}" testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.727Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: pipelining replication: peer="{Voter server-d64eed0d-cc3d-74e5-402b-904bfe0618d8 d64eed0d-cc3d-74e5-402b-904bfe0618d8}" testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.727Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: pipelining replication: peer="{Voter server-f28f3437-45ec-cfb0-a8bb-291c891734ad f28f3437-45ec-cfb0-a8bb-291c891734ad}" testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.727Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: pipelining replication: peer="{Voter server-f28f3437-45ec-cfb0-a8bb-291c891734ad f28f3437-45ec-cfb0-a8bb-291c891734ad}" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:00.833Z [INFO] stable state for Leader reached at 2024-01-09 05:10:00.727874005 +0000 UTC m=+11.980925143 (1 nodes), 73.487736ms from start of poll, 74.292803ms from cluster start. Timeout at 2024-01-09 05:10:00.832977705 +0000 UTC m=+12.086028715, 105.103572ms after stability testing.go:149: cluster: 2024-01-09T05:10:00.833Z [INFO] stable state for Leader reached at 2024-01-09 05:10:00.727874005 +0000 UTC m=+11.980925143 (1 nodes), 73.487736ms from start of poll, 74.292803ms from cluster start. Timeout at 2024-01-09 05:10:00.832977705 +0000 UTC m=+12.086028715, 105.103572ms after stability testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.833Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: starting snapshot up to: index=12 testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.833Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: starting snapshot up to: index=12 testing.go:153: 2024-01-09T05:10:00.833Z [INFO] creating new snapshot: path=/tmp/raft145392020/snapshots/2-12-1704777000833.tmp testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.844Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: snapshot complete up to: index=12 testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.844Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: snapshot complete up to: index=12 testing.go:153: 2024-01-09T05:10:00.845Z [INFO] creating new snapshot: path=/tmp/raft145392020/snapshots/2-113-1704777000845.tmp testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.856Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: copied to local snapshot: bytes=71 testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.856Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.856Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: restored user snapshot: index=1 testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.856Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: restored user snapshot: index=1 testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.856Z [ERROR] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: failed to get log: index=23 error="log not found" testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.856Z [ERROR] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.856Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: aborting pipeline replication: peer="{Voter server-d64eed0d-cc3d-74e5-402b-904bfe0618d8 d64eed0d-cc3d-74e5-402b-904bfe0618d8}" testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.856Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: aborting pipeline replication: peer="{Voter server-d64eed0d-cc3d-74e5-402b-904bfe0618d8 d64eed0d-cc3d-74e5-402b-904bfe0618d8}" testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.856Z [ERROR] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: failed to get log: index=23 error="log not found" testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.856Z [ERROR] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.856Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: aborting pipeline replication: peer="{Voter server-f28f3437-45ec-cfb0-a8bb-291c891734ad f28f3437-45ec-cfb0-a8bb-291c891734ad}" testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.856Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: aborting pipeline replication: peer="{Voter server-f28f3437-45ec-cfb0-a8bb-291c891734ad f28f3437-45ec-cfb0-a8bb-291c891734ad}" testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.864Z [ERROR] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: failed to get log: index=23 error="log not found" testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.864Z [ERROR] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: failed to get log: index=23 error="log not found" testing.go:153: 2024-01-09T05:10:00.865Z [INFO] creating new snapshot: path=/tmp/raft395268104/snapshots/2-113-1704777000865.tmp testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.866Z [ERROR] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: failed to get log: index=23 error="log not found" testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.866Z [ERROR] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: failed to get log: index=23 error="log not found" testing.go:153: 2024-01-09T05:10:00.867Z [INFO] creating new snapshot: path=/tmp/raft345521638/snapshots/2-113-1704777000867.tmp testLoggerAdapter verbose: server-f28f3437-45ec-cfb0-a8bb-291c891734ad: 2024-01-09T05:10:00.884Z [INFO] server-f28f3437-45ec-cfb0-a8bb-291c891734ad: copied to local snapshot: bytes=71 testing.go:149: server-f28f3437-45ec-cfb0-a8bb-291c891734ad: 2024-01-09T05:10:00.884Z [INFO] server-f28f3437-45ec-cfb0-a8bb-291c891734ad: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-f28f3437-45ec-cfb0-a8bb-291c891734ad: 2024-01-09T05:10:00.884Z [INFO] server-f28f3437-45ec-cfb0-a8bb-291c891734ad: Installed remote snapshot testing.go:149: server-f28f3437-45ec-cfb0-a8bb-291c891734ad: 2024-01-09T05:10:00.884Z [INFO] server-f28f3437-45ec-cfb0-a8bb-291c891734ad: Installed remote snapshot testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.884Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: pipelining replication: peer="{Voter server-f28f3437-45ec-cfb0-a8bb-291c891734ad f28f3437-45ec-cfb0-a8bb-291c891734ad}" testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.884Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: pipelining replication: peer="{Voter server-f28f3437-45ec-cfb0-a8bb-291c891734ad f28f3437-45ec-cfb0-a8bb-291c891734ad}" testLoggerAdapter verbose: server-d64eed0d-cc3d-74e5-402b-904bfe0618d8: 2024-01-09T05:10:00.889Z [INFO] server-d64eed0d-cc3d-74e5-402b-904bfe0618d8: copied to local snapshot: bytes=71 testing.go:149: server-d64eed0d-cc3d-74e5-402b-904bfe0618d8: 2024-01-09T05:10:00.889Z [INFO] server-d64eed0d-cc3d-74e5-402b-904bfe0618d8: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-d64eed0d-cc3d-74e5-402b-904bfe0618d8: 2024-01-09T05:10:00.890Z [INFO] server-d64eed0d-cc3d-74e5-402b-904bfe0618d8: Installed remote snapshot testing.go:149: server-d64eed0d-cc3d-74e5-402b-904bfe0618d8: 2024-01-09T05:10:00.890Z [INFO] server-d64eed0d-cc3d-74e5-402b-904bfe0618d8: Installed remote snapshot testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.890Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: pipelining replication: peer="{Voter server-d64eed0d-cc3d-74e5-402b-904bfe0618d8 d64eed0d-cc3d-74e5-402b-904bfe0618d8}" testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.890Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: pipelining replication: peer="{Voter server-d64eed0d-cc3d-74e5-402b-904bfe0618d8 d64eed0d-cc3d-74e5-402b-904bfe0618d8}" testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.901Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: aborting pipeline replication: peer="{Voter server-d64eed0d-cc3d-74e5-402b-904bfe0618d8 d64eed0d-cc3d-74e5-402b-904bfe0618d8}" testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.901Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: aborting pipeline replication: peer="{Voter server-d64eed0d-cc3d-74e5-402b-904bfe0618d8 d64eed0d-cc3d-74e5-402b-904bfe0618d8}" testLoggerAdapter verbose: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.901Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: aborting pipeline replication: peer="{Voter server-f28f3437-45ec-cfb0-a8bb-291c891734ad f28f3437-45ec-cfb0-a8bb-291c891734ad}" testing.go:149: server-62b05e7b-869f-f84a-ab0d-250af0adb19a: 2024-01-09T05:10:00.901Z [INFO] server-62b05e7b-869f-f84a-ab0d-250af0adb19a: aborting pipeline replication: peer="{Voter server-f28f3437-45ec-cfb0-a8bb-291c891734ad f28f3437-45ec-cfb0-a8bb-291c891734ad}" testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.903Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b Address:9c29f5e2-1fc9-edd7-e97a-f9abc960667b} {Suffrage:Voter ID:server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7 Address:f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7} {Suffrage:Voter ID:server-97d3eaf6-e477-f733-c692-4e3e39392f22 Address:97d3eaf6-e477-f733-c692-4e3e39392f22}]" testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.903Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b Address:9c29f5e2-1fc9-edd7-e97a-f9abc960667b} {Suffrage:Voter ID:server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7 Address:f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7} {Suffrage:Voter ID:server-97d3eaf6-e477-f733-c692-4e3e39392f22 Address:97d3eaf6-e477-f733-c692-4e3e39392f22}]" testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.903Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: entering follower state: follower="Node at 9c29f5e2-1fc9-edd7-e97a-f9abc960667b [Follower]" leader= testLoggerAdapter verbose: server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7: 2024-01-09T05:10:00.903Z [INFO] server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b Address:9c29f5e2-1fc9-edd7-e97a-f9abc960667b} {Suffrage:Voter ID:server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7 Address:f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7} {Suffrage:Voter ID:server-97d3eaf6-e477-f733-c692-4e3e39392f22 Address:97d3eaf6-e477-f733-c692-4e3e39392f22}]" testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.903Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: entering follower state: follower="Node at 9c29f5e2-1fc9-edd7-e97a-f9abc960667b [Follower]" leader= testing.go:149: server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7: 2024-01-09T05:10:00.903Z [INFO] server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b Address:9c29f5e2-1fc9-edd7-e97a-f9abc960667b} {Suffrage:Voter ID:server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7 Address:f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7} {Suffrage:Voter ID:server-97d3eaf6-e477-f733-c692-4e3e39392f22 Address:97d3eaf6-e477-f733-c692-4e3e39392f22}]" testLoggerAdapter verbose: server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7: 2024-01-09T05:10:00.903Z [INFO] server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7: entering follower state: follower="Node at f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7 [Follower]" leader= testing.go:149: server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7: 2024-01-09T05:10:00.903Z [INFO] server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7: entering follower state: follower="Node at f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7 [Follower]" leader= testLoggerAdapter verbose: server-97d3eaf6-e477-f733-c692-4e3e39392f22: 2024-01-09T05:10:00.903Z [INFO] server-97d3eaf6-e477-f733-c692-4e3e39392f22: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b Address:9c29f5e2-1fc9-edd7-e97a-f9abc960667b} {Suffrage:Voter ID:server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7 Address:f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7} {Suffrage:Voter ID:server-97d3eaf6-e477-f733-c692-4e3e39392f22 Address:97d3eaf6-e477-f733-c692-4e3e39392f22}]" testing.go:149: server-97d3eaf6-e477-f733-c692-4e3e39392f22: 2024-01-09T05:10:00.903Z [INFO] server-97d3eaf6-e477-f733-c692-4e3e39392f22: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b Address:9c29f5e2-1fc9-edd7-e97a-f9abc960667b} {Suffrage:Voter ID:server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7 Address:f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7} {Suffrage:Voter ID:server-97d3eaf6-e477-f733-c692-4e3e39392f22 Address:97d3eaf6-e477-f733-c692-4e3e39392f22}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:00.903Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:10:00.903Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-97d3eaf6-e477-f733-c692-4e3e39392f22: 2024-01-09T05:10:00.903Z [INFO] server-97d3eaf6-e477-f733-c692-4e3e39392f22: entering follower state: follower="Node at 97d3eaf6-e477-f733-c692-4e3e39392f22 [Follower]" leader= testing.go:149: server-97d3eaf6-e477-f733-c692-4e3e39392f22: 2024-01-09T05:10:00.903Z [INFO] server-97d3eaf6-e477-f733-c692-4e3e39392f22: entering follower state: follower="Node at 97d3eaf6-e477-f733-c692-4e3e39392f22 [Follower]" leader= testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.970Z [WARN] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.970Z [WARN] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.971Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: entering candidate state: node="Node at 9c29f5e2-1fc9-edd7-e97a-f9abc960667b [Candidate]" term=2 testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.971Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: entering candidate state: node="Node at 9c29f5e2-1fc9-edd7-e97a-f9abc960667b [Candidate]" term=2 testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.971Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: election won: tally=2 testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.971Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: election won: tally=2 testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.971Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: entering leader state: leader="Node at 9c29f5e2-1fc9-edd7-e97a-f9abc960667b [Leader]" testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.971Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: entering leader state: leader="Node at 9c29f5e2-1fc9-edd7-e97a-f9abc960667b [Leader]" testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.971Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: added peer, starting replication: peer=server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7 testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.971Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: added peer, starting replication: peer=server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7 testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.971Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: added peer, starting replication: peer=server-97d3eaf6-e477-f733-c692-4e3e39392f22 testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.971Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: added peer, starting replication: peer=server-97d3eaf6-e477-f733-c692-4e3e39392f22 testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.971Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: pipelining replication: peer="{Voter server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7 f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7}" testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.971Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: pipelining replication: peer="{Voter server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7 f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7}" testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.971Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: pipelining replication: peer="{Voter server-97d3eaf6-e477-f733-c692-4e3e39392f22 97d3eaf6-e477-f733-c692-4e3e39392f22}" testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:00.971Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: pipelining replication: peer="{Voter server-97d3eaf6-e477-f733-c692-4e3e39392f22 97d3eaf6-e477-f733-c692-4e3e39392f22}" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:01.076Z [INFO] stable state for Leader reached at 2024-01-09 05:10:00.971380126 +0000 UTC m=+12.224431187 (1 nodes), 67.577002ms from start of poll, 68.185887ms from cluster start. Timeout at 2024-01-09 05:10:01.076474619 +0000 UTC m=+12.329525687, 105.0945ms after stability testing.go:149: cluster: 2024-01-09T05:10:01.076Z [INFO] stable state for Leader reached at 2024-01-09 05:10:00.971380126 +0000 UTC m=+12.224431187 (1 nodes), 67.577002ms from start of poll, 68.185887ms from cluster start. Timeout at 2024-01-09 05:10:01.076474619 +0000 UTC m=+12.329525687, 105.0945ms after stability testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.077Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: starting snapshot up to: index=12 testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.077Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: starting snapshot up to: index=12 testing.go:153: 2024-01-09T05:10:01.077Z [INFO] creating new snapshot: path=/tmp/raft214232186/snapshots/2-12-1704777001077.tmp testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.084Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: snapshot complete up to: index=12 testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.084Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: snapshot complete up to: index=12 testing.go:153: 2024-01-09T05:10:01.084Z [INFO] creating new snapshot: path=/tmp/raft214232186/snapshots/2-1013-1704777001084.tmp testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.100Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: copied to local snapshot: bytes=71 testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.100Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.100Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: restored user snapshot: index=1 testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.100Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: restored user snapshot: index=1 testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.103Z [ERROR] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: failed to get log: index=23 error="log not found" testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.103Z [ERROR] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.105Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: aborting pipeline replication: peer="{Voter server-97d3eaf6-e477-f733-c692-4e3e39392f22 97d3eaf6-e477-f733-c692-4e3e39392f22}" testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.105Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: aborting pipeline replication: peer="{Voter server-97d3eaf6-e477-f733-c692-4e3e39392f22 97d3eaf6-e477-f733-c692-4e3e39392f22}" testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.103Z [ERROR] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: failed to get log: index=23 error="log not found" testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.103Z [ERROR] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.105Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: aborting pipeline replication: peer="{Voter server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7 f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7}" testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.105Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: aborting pipeline replication: peer="{Voter server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7 f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7}" testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.105Z [ERROR] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: failed to get log: index=23 error="log not found" testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.105Z [ERROR] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: failed to get log: index=23 error="log not found" testing.go:153: 2024-01-09T05:10:01.105Z [INFO] creating new snapshot: path=/tmp/raft721165756/snapshots/2-1013-1704777001105.tmp testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.113Z [ERROR] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: failed to get log: index=23 error="log not found" testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.113Z [ERROR] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: failed to get log: index=23 error="log not found" testing.go:153: 2024-01-09T05:10:01.113Z [INFO] creating new snapshot: path=/tmp/raft811189838/snapshots/2-1013-1704777001113.tmp testLoggerAdapter verbose: server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7: 2024-01-09T05:10:01.113Z [INFO] server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7: copied to local snapshot: bytes=71 testing.go:149: server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7: 2024-01-09T05:10:01.113Z [INFO] server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7: 2024-01-09T05:10:01.113Z [INFO] server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7: Installed remote snapshot testing.go:149: server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7: 2024-01-09T05:10:01.113Z [INFO] server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7: Installed remote snapshot testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.114Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: pipelining replication: peer="{Voter server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7 f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7}" testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.114Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: pipelining replication: peer="{Voter server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7 f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7}" testLoggerAdapter verbose: server-97d3eaf6-e477-f733-c692-4e3e39392f22: 2024-01-09T05:10:01.123Z [INFO] server-97d3eaf6-e477-f733-c692-4e3e39392f22: copied to local snapshot: bytes=71 testing.go:149: server-97d3eaf6-e477-f733-c692-4e3e39392f22: 2024-01-09T05:10:01.123Z [INFO] server-97d3eaf6-e477-f733-c692-4e3e39392f22: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-97d3eaf6-e477-f733-c692-4e3e39392f22: 2024-01-09T05:10:01.124Z [INFO] server-97d3eaf6-e477-f733-c692-4e3e39392f22: Installed remote snapshot testing.go:149: server-97d3eaf6-e477-f733-c692-4e3e39392f22: 2024-01-09T05:10:01.124Z [INFO] server-97d3eaf6-e477-f733-c692-4e3e39392f22: Installed remote snapshot testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.124Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: pipelining replication: peer="{Voter server-97d3eaf6-e477-f733-c692-4e3e39392f22 97d3eaf6-e477-f733-c692-4e3e39392f22}" testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.124Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: pipelining replication: peer="{Voter server-97d3eaf6-e477-f733-c692-4e3e39392f22 97d3eaf6-e477-f733-c692-4e3e39392f22}" testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.135Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: aborting pipeline replication: peer="{Voter server-97d3eaf6-e477-f733-c692-4e3e39392f22 97d3eaf6-e477-f733-c692-4e3e39392f22}" testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.135Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: aborting pipeline replication: peer="{Voter server-97d3eaf6-e477-f733-c692-4e3e39392f22 97d3eaf6-e477-f733-c692-4e3e39392f22}" testLoggerAdapter verbose: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.136Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: aborting pipeline replication: peer="{Voter server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7 f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7}" testing.go:149: server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: 2024-01-09T05:10:01.136Z [INFO] server-9c29f5e2-1fc9-edd7-e97a-f9abc960667b: aborting pipeline replication: peer="{Voter server-f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7 f9ba8ccc-16e1-2a1a-f9e6-7848fe2ed9b7}" testLoggerAdapter verbose: server-98380281-4484-82ac-6934-e75f6883688f: 2024-01-09T05:10:01.140Z [INFO] server-98380281-4484-82ac-6934-e75f6883688f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98380281-4484-82ac-6934-e75f6883688f Address:98380281-4484-82ac-6934-e75f6883688f} {Suffrage:Voter ID:server-9b6d9996-37d1-7f97-639b-f943fafa4261 Address:9b6d9996-37d1-7f97-639b-f943fafa4261} {Suffrage:Voter ID:server-591947dd-710c-7eb7-8b46-22a7944f4442 Address:591947dd-710c-7eb7-8b46-22a7944f4442}]" testing.go:149: server-98380281-4484-82ac-6934-e75f6883688f: 2024-01-09T05:10:01.140Z [INFO] server-98380281-4484-82ac-6934-e75f6883688f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98380281-4484-82ac-6934-e75f6883688f Address:98380281-4484-82ac-6934-e75f6883688f} {Suffrage:Voter ID:server-9b6d9996-37d1-7f97-639b-f943fafa4261 Address:9b6d9996-37d1-7f97-639b-f943fafa4261} {Suffrage:Voter ID:server-591947dd-710c-7eb7-8b46-22a7944f4442 Address:591947dd-710c-7eb7-8b46-22a7944f4442}]" testLoggerAdapter verbose: server-98380281-4484-82ac-6934-e75f6883688f: 2024-01-09T05:10:01.140Z [INFO] server-98380281-4484-82ac-6934-e75f6883688f: entering follower state: follower="Node at 98380281-4484-82ac-6934-e75f6883688f [Follower]" leader= testing.go:149: server-98380281-4484-82ac-6934-e75f6883688f: 2024-01-09T05:10:01.140Z [INFO] server-98380281-4484-82ac-6934-e75f6883688f: entering follower state: follower="Node at 98380281-4484-82ac-6934-e75f6883688f [Follower]" leader= testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.141Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98380281-4484-82ac-6934-e75f6883688f Address:98380281-4484-82ac-6934-e75f6883688f} {Suffrage:Voter ID:server-9b6d9996-37d1-7f97-639b-f943fafa4261 Address:9b6d9996-37d1-7f97-639b-f943fafa4261} {Suffrage:Voter ID:server-591947dd-710c-7eb7-8b46-22a7944f4442 Address:591947dd-710c-7eb7-8b46-22a7944f4442}]" testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.141Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98380281-4484-82ac-6934-e75f6883688f Address:98380281-4484-82ac-6934-e75f6883688f} {Suffrage:Voter ID:server-9b6d9996-37d1-7f97-639b-f943fafa4261 Address:9b6d9996-37d1-7f97-639b-f943fafa4261} {Suffrage:Voter ID:server-591947dd-710c-7eb7-8b46-22a7944f4442 Address:591947dd-710c-7eb7-8b46-22a7944f4442}]" testLoggerAdapter verbose: server-591947dd-710c-7eb7-8b46-22a7944f4442: 2024-01-09T05:10:01.141Z [INFO] server-591947dd-710c-7eb7-8b46-22a7944f4442: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98380281-4484-82ac-6934-e75f6883688f Address:98380281-4484-82ac-6934-e75f6883688f} {Suffrage:Voter ID:server-9b6d9996-37d1-7f97-639b-f943fafa4261 Address:9b6d9996-37d1-7f97-639b-f943fafa4261} {Suffrage:Voter ID:server-591947dd-710c-7eb7-8b46-22a7944f4442 Address:591947dd-710c-7eb7-8b46-22a7944f4442}]" testing.go:149: server-591947dd-710c-7eb7-8b46-22a7944f4442: 2024-01-09T05:10:01.141Z [INFO] server-591947dd-710c-7eb7-8b46-22a7944f4442: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98380281-4484-82ac-6934-e75f6883688f Address:98380281-4484-82ac-6934-e75f6883688f} {Suffrage:Voter ID:server-9b6d9996-37d1-7f97-639b-f943fafa4261 Address:9b6d9996-37d1-7f97-639b-f943fafa4261} {Suffrage:Voter ID:server-591947dd-710c-7eb7-8b46-22a7944f4442 Address:591947dd-710c-7eb7-8b46-22a7944f4442}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:01.141Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:10:01.141Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.141Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: entering follower state: follower="Node at 9b6d9996-37d1-7f97-639b-f943fafa4261 [Follower]" leader= testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.141Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: entering follower state: follower="Node at 9b6d9996-37d1-7f97-639b-f943fafa4261 [Follower]" leader= testLoggerAdapter verbose: server-591947dd-710c-7eb7-8b46-22a7944f4442: 2024-01-09T05:10:01.141Z [INFO] server-591947dd-710c-7eb7-8b46-22a7944f4442: entering follower state: follower="Node at 591947dd-710c-7eb7-8b46-22a7944f4442 [Follower]" leader= testing.go:149: server-591947dd-710c-7eb7-8b46-22a7944f4442: 2024-01-09T05:10:01.141Z [INFO] server-591947dd-710c-7eb7-8b46-22a7944f4442: entering follower state: follower="Node at 591947dd-710c-7eb7-8b46-22a7944f4442 [Follower]" leader= testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.194Z [WARN] server-9b6d9996-37d1-7f97-639b-f943fafa4261: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.194Z [WARN] server-9b6d9996-37d1-7f97-639b-f943fafa4261: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.195Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: entering candidate state: node="Node at 9b6d9996-37d1-7f97-639b-f943fafa4261 [Candidate]" term=2 testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.195Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: entering candidate state: node="Node at 9b6d9996-37d1-7f97-639b-f943fafa4261 [Candidate]" term=2 testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.195Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: election won: tally=2 testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.195Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: election won: tally=2 testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.195Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: entering leader state: leader="Node at 9b6d9996-37d1-7f97-639b-f943fafa4261 [Leader]" testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.195Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: entering leader state: leader="Node at 9b6d9996-37d1-7f97-639b-f943fafa4261 [Leader]" testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.195Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: added peer, starting replication: peer=server-98380281-4484-82ac-6934-e75f6883688f testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.195Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: added peer, starting replication: peer=server-98380281-4484-82ac-6934-e75f6883688f testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.195Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: added peer, starting replication: peer=server-591947dd-710c-7eb7-8b46-22a7944f4442 testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.195Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: added peer, starting replication: peer=server-591947dd-710c-7eb7-8b46-22a7944f4442 testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.195Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: pipelining replication: peer="{Voter server-98380281-4484-82ac-6934-e75f6883688f 98380281-4484-82ac-6934-e75f6883688f}" testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.195Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: pipelining replication: peer="{Voter server-98380281-4484-82ac-6934-e75f6883688f 98380281-4484-82ac-6934-e75f6883688f}" testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.195Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: pipelining replication: peer="{Voter server-591947dd-710c-7eb7-8b46-22a7944f4442 591947dd-710c-7eb7-8b46-22a7944f4442}" testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.195Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: pipelining replication: peer="{Voter server-591947dd-710c-7eb7-8b46-22a7944f4442 591947dd-710c-7eb7-8b46-22a7944f4442}" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:01.301Z [INFO] stable state for Leader reached at 2024-01-09 05:10:01.195422288 +0000 UTC m=+12.448473426 (1 nodes), 53.819278ms from start of poll, 55.104715ms from cluster start. Timeout at 2024-01-09 05:10:01.30112509 +0000 UTC m=+12.554176091, 105.702665ms after stability testing.go:149: cluster: 2024-01-09T05:10:01.301Z [INFO] stable state for Leader reached at 2024-01-09 05:10:01.195422288 +0000 UTC m=+12.448473426 (1 nodes), 53.819278ms from start of poll, 55.104715ms from cluster start. Timeout at 2024-01-09 05:10:01.30112509 +0000 UTC m=+12.554176091, 105.702665ms after stability testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.301Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: starting snapshot up to: index=12 testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.301Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: starting snapshot up to: index=12 testing.go:153: 2024-01-09T05:10:01.301Z [INFO] creating new snapshot: path=/tmp/raft518721890/snapshots/2-12-1704777001301.tmp testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.307Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: snapshot complete up to: index=12 testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.307Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: snapshot complete up to: index=12 testing.go:153: 2024-01-09T05:10:01.309Z [INFO] creating new snapshot: path=/tmp/raft518721890/snapshots/2-10013-1704777001309.tmp testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.323Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: copied to local snapshot: bytes=71 testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.323Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.324Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: restored user snapshot: index=1 testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.324Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: restored user snapshot: index=1 testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.324Z [ERROR] server-9b6d9996-37d1-7f97-639b-f943fafa4261: failed to get log: index=23 error="log not found" testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.324Z [ERROR] server-9b6d9996-37d1-7f97-639b-f943fafa4261: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.324Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: aborting pipeline replication: peer="{Voter server-591947dd-710c-7eb7-8b46-22a7944f4442 591947dd-710c-7eb7-8b46-22a7944f4442}" testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.324Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: aborting pipeline replication: peer="{Voter server-591947dd-710c-7eb7-8b46-22a7944f4442 591947dd-710c-7eb7-8b46-22a7944f4442}" testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.324Z [ERROR] server-9b6d9996-37d1-7f97-639b-f943fafa4261: failed to get log: index=23 error="log not found" testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.324Z [ERROR] server-9b6d9996-37d1-7f97-639b-f943fafa4261: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.324Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: aborting pipeline replication: peer="{Voter server-98380281-4484-82ac-6934-e75f6883688f 98380281-4484-82ac-6934-e75f6883688f}" testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.324Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: aborting pipeline replication: peer="{Voter server-98380281-4484-82ac-6934-e75f6883688f 98380281-4484-82ac-6934-e75f6883688f}" testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.332Z [ERROR] server-9b6d9996-37d1-7f97-639b-f943fafa4261: failed to get log: index=23 error="log not found" testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.332Z [ERROR] server-9b6d9996-37d1-7f97-639b-f943fafa4261: failed to get log: index=23 error="log not found" testing.go:153: 2024-01-09T05:10:01.332Z [INFO] creating new snapshot: path=/tmp/raft025824484/snapshots/2-10013-1704777001332.tmp testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.333Z [ERROR] server-9b6d9996-37d1-7f97-639b-f943fafa4261: failed to get log: index=23 error="log not found" testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.333Z [ERROR] server-9b6d9996-37d1-7f97-639b-f943fafa4261: failed to get log: index=23 error="log not found" testing.go:153: 2024-01-09T05:10:01.334Z [INFO] creating new snapshot: path=/tmp/raft745983664/snapshots/2-10013-1704777001334.tmp testLoggerAdapter verbose: server-98380281-4484-82ac-6934-e75f6883688f: 2024-01-09T05:10:01.353Z [INFO] server-98380281-4484-82ac-6934-e75f6883688f: copied to local snapshot: bytes=71 testing.go:149: server-98380281-4484-82ac-6934-e75f6883688f: 2024-01-09T05:10:01.353Z [INFO] server-98380281-4484-82ac-6934-e75f6883688f: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-591947dd-710c-7eb7-8b46-22a7944f4442: 2024-01-09T05:10:01.354Z [INFO] server-591947dd-710c-7eb7-8b46-22a7944f4442: copied to local snapshot: bytes=71 testing.go:149: server-591947dd-710c-7eb7-8b46-22a7944f4442: 2024-01-09T05:10:01.354Z [INFO] server-591947dd-710c-7eb7-8b46-22a7944f4442: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-591947dd-710c-7eb7-8b46-22a7944f4442: 2024-01-09T05:10:01.354Z [INFO] server-591947dd-710c-7eb7-8b46-22a7944f4442: Installed remote snapshot testing.go:149: server-591947dd-710c-7eb7-8b46-22a7944f4442: 2024-01-09T05:10:01.354Z [INFO] server-591947dd-710c-7eb7-8b46-22a7944f4442: Installed remote snapshot testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.354Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: pipelining replication: peer="{Voter server-591947dd-710c-7eb7-8b46-22a7944f4442 591947dd-710c-7eb7-8b46-22a7944f4442}" testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.354Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: pipelining replication: peer="{Voter server-591947dd-710c-7eb7-8b46-22a7944f4442 591947dd-710c-7eb7-8b46-22a7944f4442}" testLoggerAdapter verbose: server-98380281-4484-82ac-6934-e75f6883688f: 2024-01-09T05:10:01.355Z [INFO] server-98380281-4484-82ac-6934-e75f6883688f: Installed remote snapshot testing.go:149: server-98380281-4484-82ac-6934-e75f6883688f: 2024-01-09T05:10:01.355Z [INFO] server-98380281-4484-82ac-6934-e75f6883688f: Installed remote snapshot testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.355Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: pipelining replication: peer="{Voter server-98380281-4484-82ac-6934-e75f6883688f 98380281-4484-82ac-6934-e75f6883688f}" testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.355Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: pipelining replication: peer="{Voter server-98380281-4484-82ac-6934-e75f6883688f 98380281-4484-82ac-6934-e75f6883688f}" testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.371Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: aborting pipeline replication: peer="{Voter server-591947dd-710c-7eb7-8b46-22a7944f4442 591947dd-710c-7eb7-8b46-22a7944f4442}" testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.371Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: aborting pipeline replication: peer="{Voter server-591947dd-710c-7eb7-8b46-22a7944f4442 591947dd-710c-7eb7-8b46-22a7944f4442}" testLoggerAdapter verbose: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.371Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: aborting pipeline replication: peer="{Voter server-98380281-4484-82ac-6934-e75f6883688f 98380281-4484-82ac-6934-e75f6883688f}" testing.go:149: server-9b6d9996-37d1-7f97-639b-f943fafa4261: 2024-01-09T05:10:01.371Z [INFO] server-9b6d9996-37d1-7f97-639b-f943fafa4261: aborting pipeline replication: peer="{Voter server-98380281-4484-82ac-6934-e75f6883688f 98380281-4484-82ac-6934-e75f6883688f}" --- PASS: TestRaft_UserRestore (1.42s) === RUN TestRaft_SendSnapshotFollower testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.374Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-138c9853-4660-c7de-1ab8-eed6328689ca Address:138c9853-4660-c7de-1ab8-eed6328689ca} {Suffrage:Voter ID:server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc Address:cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc} {Suffrage:Voter ID:server-cc3c790b-3981-510f-d759-353e8ab43bff Address:cc3c790b-3981-510f-d759-353e8ab43bff}]" testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.374Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-138c9853-4660-c7de-1ab8-eed6328689ca Address:138c9853-4660-c7de-1ab8-eed6328689ca} {Suffrage:Voter ID:server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc Address:cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc} {Suffrage:Voter ID:server-cc3c790b-3981-510f-d759-353e8ab43bff Address:cc3c790b-3981-510f-d759-353e8ab43bff}]" testLoggerAdapter verbose: server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: 2024-01-09T05:10:01.374Z [INFO] server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-138c9853-4660-c7de-1ab8-eed6328689ca Address:138c9853-4660-c7de-1ab8-eed6328689ca} {Suffrage:Voter ID:server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc Address:cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc} {Suffrage:Voter ID:server-cc3c790b-3981-510f-d759-353e8ab43bff Address:cc3c790b-3981-510f-d759-353e8ab43bff}]" testing.go:149: server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: 2024-01-09T05:10:01.374Z [INFO] server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-138c9853-4660-c7de-1ab8-eed6328689ca Address:138c9853-4660-c7de-1ab8-eed6328689ca} {Suffrage:Voter ID:server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc Address:cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc} {Suffrage:Voter ID:server-cc3c790b-3981-510f-d759-353e8ab43bff Address:cc3c790b-3981-510f-d759-353e8ab43bff}]" testLoggerAdapter verbose: server-cc3c790b-3981-510f-d759-353e8ab43bff: 2024-01-09T05:10:01.374Z [INFO] server-cc3c790b-3981-510f-d759-353e8ab43bff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-138c9853-4660-c7de-1ab8-eed6328689ca Address:138c9853-4660-c7de-1ab8-eed6328689ca} {Suffrage:Voter ID:server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc Address:cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc} {Suffrage:Voter ID:server-cc3c790b-3981-510f-d759-353e8ab43bff Address:cc3c790b-3981-510f-d759-353e8ab43bff}]" testing.go:149: server-cc3c790b-3981-510f-d759-353e8ab43bff: 2024-01-09T05:10:01.374Z [INFO] server-cc3c790b-3981-510f-d759-353e8ab43bff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-138c9853-4660-c7de-1ab8-eed6328689ca Address:138c9853-4660-c7de-1ab8-eed6328689ca} {Suffrage:Voter ID:server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc Address:cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc} {Suffrage:Voter ID:server-cc3c790b-3981-510f-d759-353e8ab43bff Address:cc3c790b-3981-510f-d759-353e8ab43bff}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:01.374Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-01-09T05:10:01.374Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: 2024-01-09T05:10:01.374Z [INFO] server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: entering follower state: follower="Node at cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc [Follower]" leader= testing.go:149: server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: 2024-01-09T05:10:01.374Z [INFO] server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: entering follower state: follower="Node at cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc [Follower]" leader= testLoggerAdapter verbose: server-cc3c790b-3981-510f-d759-353e8ab43bff: 2024-01-09T05:10:01.374Z [INFO] server-cc3c790b-3981-510f-d759-353e8ab43bff: entering follower state: follower="Node at cc3c790b-3981-510f-d759-353e8ab43bff [Follower]" leader= testing.go:149: server-cc3c790b-3981-510f-d759-353e8ab43bff: 2024-01-09T05:10:01.374Z [INFO] server-cc3c790b-3981-510f-d759-353e8ab43bff: entering follower state: follower="Node at cc3c790b-3981-510f-d759-353e8ab43bff [Follower]" leader= testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.374Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: entering follower state: follower="Node at 138c9853-4660-c7de-1ab8-eed6328689ca [Follower]" leader= testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.374Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: entering follower state: follower="Node at 138c9853-4660-c7de-1ab8-eed6328689ca [Follower]" leader= testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.451Z [WARN] server-138c9853-4660-c7de-1ab8-eed6328689ca: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.451Z [WARN] server-138c9853-4660-c7de-1ab8-eed6328689ca: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.451Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: entering candidate state: node="Node at 138c9853-4660-c7de-1ab8-eed6328689ca [Candidate]" term=2 testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.451Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: entering candidate state: node="Node at 138c9853-4660-c7de-1ab8-eed6328689ca [Candidate]" term=2 testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.451Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: election won: tally=2 testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.451Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: election won: tally=2 testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.451Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: entering leader state: leader="Node at 138c9853-4660-c7de-1ab8-eed6328689ca [Leader]" testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.451Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: entering leader state: leader="Node at 138c9853-4660-c7de-1ab8-eed6328689ca [Leader]" testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.451Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: added peer, starting replication: peer=server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.451Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: added peer, starting replication: peer=server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.452Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: added peer, starting replication: peer=server-cc3c790b-3981-510f-d759-353e8ab43bff testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.452Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: added peer, starting replication: peer=server-cc3c790b-3981-510f-d759-353e8ab43bff testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.452Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: pipelining replication: peer="{Voter server-cc3c790b-3981-510f-d759-353e8ab43bff cc3c790b-3981-510f-d759-353e8ab43bff}" testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.452Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: pipelining replication: peer="{Voter server-cc3c790b-3981-510f-d759-353e8ab43bff cc3c790b-3981-510f-d759-353e8ab43bff}" testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.452Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: pipelining replication: peer="{Voter server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc}" testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.452Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: pipelining replication: peer="{Voter server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc}" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:01.557Z [INFO] stable state for Follower reached at 2024-01-09 05:10:01.452263018 +0000 UTC m=+12.705313991 (2 nodes), 77.905776ms from start of poll, 78.631545ms from cluster start. Timeout at 2024-01-09 05:10:01.557352629 +0000 UTC m=+12.810403601, 105.08961ms after stability testing.go:149: cluster: 2024-01-09T05:10:01.557Z [INFO] stable state for Follower reached at 2024-01-09 05:10:01.452263018 +0000 UTC m=+12.705313991 (2 nodes), 77.905776ms from start of poll, 78.631545ms from cluster start. Timeout at 2024-01-09 05:10:01.557352629 +0000 UTC m=+12.810403601, 105.08961ms after stability testLoggerAdapter verbose: cluster: 2024-01-09T05:10:01.557Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:10:01.557Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-01-09T05:10:01.662Z [INFO] stable state for Leader reached at 2024-01-09 05:10:01.557521543 +0000 UTC m=+12.810572501 (1 nodes), 1.867µs from start of poll, 183.890055ms from cluster start. Timeout at 2024-01-09 05:10:01.662672805 +0000 UTC m=+12.915723785, 105.151284ms after stability testing.go:149: cluster: 2024-01-09T05:10:01.662Z [INFO] stable state for Leader reached at 2024-01-09 05:10:01.557521543 +0000 UTC m=+12.810572501 (1 nodes), 1.867µs from start of poll, 183.890055ms from cluster start. Timeout at 2024-01-09 05:10:01.662672805 +0000 UTC m=+12.915723785, 105.151284ms after stability testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.662Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to pipeline appendEntries: peer="{Voter server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc}" error="append pipeline closed" testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.662Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to pipeline appendEntries: peer="{Voter server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc}" error="append pipeline closed" testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.662Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: aborting pipeline replication: peer="{Voter server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc}" testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.662Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: aborting pipeline replication: peer="{Voter server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc}" testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.662Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to appendEntries to: peer="{Voter server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc}" error="failed to connect to peer: cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc" testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.662Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to appendEntries to: peer="{Voter server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc}" error="failed to connect to peer: cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc" testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.668Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to heartbeat to: peer=cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc error="failed to connect to peer: cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc" testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.668Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to heartbeat to: peer=cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc error="failed to connect to peer: cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc" testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.673Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to appendEntries to: peer="{Voter server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc}" error="failed to connect to peer: cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc" testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.673Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to appendEntries to: peer="{Voter server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc}" error="failed to connect to peer: cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc" raft_test.go:1320: [INFO] Finished apply without behind follower testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.676Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: starting snapshot up to: index=102 testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.676Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: starting snapshot up to: index=102 testing.go:153: 2024-01-09T05:10:01.676Z [INFO] creating new snapshot: path=/tmp/raft708945334/snapshots/2-102-1704777001676.tmp testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.683Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to appendEntries to: peer="{Voter server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc}" error="failed to connect to peer: cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc" testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.683Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to appendEntries to: peer="{Voter server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc}" error="failed to connect to peer: cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc" testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.689Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to heartbeat to: peer=cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc error="failed to connect to peer: cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc" testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.689Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to heartbeat to: peer=cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc error="failed to connect to peer: cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc" testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.706Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to heartbeat to: peer=cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc error="failed to connect to peer: cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc" testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.706Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to heartbeat to: peer=cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc error="failed to connect to peer: cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc" testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.708Z [WARN] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to contact: server-id=server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc time=50.064496ms testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.708Z [WARN] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to contact: server-id=server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc time=50.064496ms testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.712Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to appendEntries to: peer="{Voter server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc}" error="failed to connect to peer: cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc" testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.712Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to appendEntries to: peer="{Voter server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc}" error="failed to connect to peer: cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc" testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.718Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: compacting logs: from=1 to=92 testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.718Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.718Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: snapshot complete up to: index=102 testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.718Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: snapshot complete up to: index=102 testLoggerAdapter verbose: server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: 2024-01-09T05:10:01.718Z [ERROR] server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: failed to take snapshot: error="nothing new to snapshot" testing.go:149: server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: 2024-01-09T05:10:01.718Z [ERROR] server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: failed to take snapshot: error="nothing new to snapshot" testLoggerAdapter verbose: server-cc3c790b-3981-510f-d759-353e8ab43bff: 2024-01-09T05:10:01.718Z [INFO] server-cc3c790b-3981-510f-d759-353e8ab43bff: starting snapshot up to: index=102 testing.go:149: server-cc3c790b-3981-510f-d759-353e8ab43bff: 2024-01-09T05:10:01.718Z [INFO] server-cc3c790b-3981-510f-d759-353e8ab43bff: starting snapshot up to: index=102 testing.go:153: 2024-01-09T05:10:01.718Z [INFO] creating new snapshot: path=/tmp/raft250731850/snapshots/2-102-1704777001718.tmp testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.736Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to heartbeat to: peer=cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc error="failed to connect to peer: cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc" testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.736Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to heartbeat to: peer=cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc error="failed to connect to peer: cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc" testLoggerAdapter verbose: server-cc3c790b-3981-510f-d759-353e8ab43bff: 2024-01-09T05:10:01.756Z [INFO] server-cc3c790b-3981-510f-d759-353e8ab43bff: compacting logs: from=1 to=92 testing.go:149: server-cc3c790b-3981-510f-d759-353e8ab43bff: 2024-01-09T05:10:01.756Z [INFO] server-cc3c790b-3981-510f-d759-353e8ab43bff: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-cc3c790b-3981-510f-d759-353e8ab43bff: 2024-01-09T05:10:01.756Z [INFO] server-cc3c790b-3981-510f-d759-353e8ab43bff: snapshot complete up to: index=102 testing.go:149: server-cc3c790b-3981-510f-d759-353e8ab43bff: 2024-01-09T05:10:01.756Z [INFO] server-cc3c790b-3981-510f-d759-353e8ab43bff: snapshot complete up to: index=102 testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.756Z [WARN] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to contact: server-id=server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc time=97.870203ms testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.756Z [WARN] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to contact: server-id=server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc time=97.870203ms testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.761Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to get log: index=2 error="log not found" testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.761Z [ERROR] server-138c9853-4660-c7de-1ab8-eed6328689ca: failed to get log: index=2 error="log not found" testing.go:153: 2024-01-09T05:10:01.761Z [INFO] creating new snapshot: path=/tmp/raft680109656/snapshots/2-102-1704777001761.tmp testLoggerAdapter verbose: server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: 2024-01-09T05:10:01.769Z [INFO] server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: copied to local snapshot: bytes=693 testing.go:149: server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: 2024-01-09T05:10:01.769Z [INFO] server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: copied to local snapshot: bytes=693 testLoggerAdapter verbose: server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: 2024-01-09T05:10:01.769Z [INFO] server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: Installed remote snapshot testing.go:149: server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: 2024-01-09T05:10:01.769Z [INFO] server-cd9c6260-d7e3-3303-b7b5-5afbb37b6ecc: Installed remote snapshot testLoggerAdapter verbose: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.771Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: aborting pipeline replication: peer="{Voter server-cc3c790b-3981-510f-d759-353e8ab43bff cc3c790b-3981-510f-d759-353e8ab43bff}" testing.go:149: server-138c9853-4660-c7de-1ab8-eed6328689ca: 2024-01-09T05:10:01.771Z [INFO] server-138c9853-4660-c7de-1ab8-eed6328689ca: aborting pipeline replication: peer="{Voter server-cc3c790b-3981-510f-d759-353e8ab43bff cc3c790b-3981-510f-d759-353e8ab43bff}" --- PASS: TestRaft_SendSnapshotFollower (0.40s) === RUN TestRaft_SendSnapshotAndLogsFollower testLoggerAdapter verbose: server-019f2655-7706-c399-525b-39ceb5b19abb: 2024-01-09T05:10:01.775Z [INFO] server-019f2655-7706-c399-525b-39ceb5b19abb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-019f2655-7706-c399-525b-39ceb5b19abb Address:019f2655-7706-c399-525b-39ceb5b19abb} {Suffrage:Voter ID:server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6 Address:2a03ca26-444b-58cf-c7ba-203a0d05a1b6} {Suffrage:Voter ID:server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8 Address:cf5bea6e-9ea4-4c40-0729-d455d24e67c8}]" testing.go:149: server-019f2655-7706-c399-525b-39ceb5b19abb: 2024-01-09T05:10:01.775Z [INFO] server-019f2655-7706-c399-525b-39ceb5b19abb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-019f2655-7706-c399-525b-39ceb5b19abb Address:019f2655-7706-c399-525b-39ceb5b19abb} {Suffrage:Voter ID:server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6 Address:2a03ca26-444b-58cf-c7ba-203a0d05a1b6} {Suffrage:Voter ID:server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8 Address:cf5bea6e-9ea4-4c40-0729-d455d24e67c8}]" testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.776Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-019f2655-7706-c399-525b-39ceb5b19abb Address:019f2655-7706-c399-525b-39ceb5b19abb} {Suffrage:Voter ID:server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6 Address:2a03ca26-444b-58cf-c7ba-203a0d05a1b6} {Suffrage:Voter ID:server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8 Address:cf5bea6e-9ea4-4c40-0729-d455d24e67c8}]" testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.776Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-019f2655-7706-c399-525b-39ceb5b19abb Address:019f2655-7706-c399-525b-39ceb5b19abb} {Suffrage:Voter ID:server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6 Address:2a03ca26-444b-58cf-c7ba-203a0d05a1b6} {Suffrage:Voter ID:server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8 Address:cf5bea6e-9ea4-4c40-0729-d455d24e67c8}]" testLoggerAdapter verbose: server-019f2655-7706-c399-525b-39ceb5b19abb: 2024-01-09T05:10:01.776Z [INFO] server-019f2655-7706-c399-525b-39ceb5b19abb: entering follower state: follower="Node at 019f2655-7706-c399-525b-39ceb5b19abb [Follower]" leader= testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.776Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: entering follower state: follower="Node at 2a03ca26-444b-58cf-c7ba-203a0d05a1b6 [Follower]" leader= testing.go:149: server-019f2655-7706-c399-525b-39ceb5b19abb: 2024-01-09T05:10:01.776Z [INFO] server-019f2655-7706-c399-525b-39ceb5b19abb: entering follower state: follower="Node at 019f2655-7706-c399-525b-39ceb5b19abb [Follower]" leader= testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.776Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: entering follower state: follower="Node at 2a03ca26-444b-58cf-c7ba-203a0d05a1b6 [Follower]" leader= testLoggerAdapter verbose: server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: 2024-01-09T05:10:01.776Z [INFO] server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-019f2655-7706-c399-525b-39ceb5b19abb Address:019f2655-7706-c399-525b-39ceb5b19abb} {Suffrage:Voter ID:server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6 Address:2a03ca26-444b-58cf-c7ba-203a0d05a1b6} {Suffrage:Voter ID:server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8 Address:cf5bea6e-9ea4-4c40-0729-d455d24e67c8}]" testing.go:149: server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: 2024-01-09T05:10:01.776Z [INFO] server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-019f2655-7706-c399-525b-39ceb5b19abb Address:019f2655-7706-c399-525b-39ceb5b19abb} {Suffrage:Voter ID:server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6 Address:2a03ca26-444b-58cf-c7ba-203a0d05a1b6} {Suffrage:Voter ID:server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8 Address:cf5bea6e-9ea4-4c40-0729-d455d24e67c8}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:01.777Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: 2024-01-09T05:10:01.777Z [INFO] server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: entering follower state: follower="Node at cf5bea6e-9ea4-4c40-0729-d455d24e67c8 [Follower]" leader= testing.go:149: server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: 2024-01-09T05:10:01.777Z [INFO] server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: entering follower state: follower="Node at cf5bea6e-9ea4-4c40-0729-d455d24e67c8 [Follower]" leader= testing.go:149: cluster: 2024-01-09T05:10:01.777Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.854Z [WARN] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.854Z [WARN] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.854Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: entering candidate state: node="Node at 2a03ca26-444b-58cf-c7ba-203a0d05a1b6 [Candidate]" term=2 testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.854Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: entering candidate state: node="Node at 2a03ca26-444b-58cf-c7ba-203a0d05a1b6 [Candidate]" term=2 testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.854Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: election won: tally=2 testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.854Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: election won: tally=2 testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.854Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: entering leader state: leader="Node at 2a03ca26-444b-58cf-c7ba-203a0d05a1b6 [Leader]" testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.854Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: entering leader state: leader="Node at 2a03ca26-444b-58cf-c7ba-203a0d05a1b6 [Leader]" testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.854Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: added peer, starting replication: peer=server-019f2655-7706-c399-525b-39ceb5b19abb testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.854Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: added peer, starting replication: peer=server-019f2655-7706-c399-525b-39ceb5b19abb testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.854Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: added peer, starting replication: peer=server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8 testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.854Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: added peer, starting replication: peer=server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8 testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.854Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: pipelining replication: peer="{Voter server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8 cf5bea6e-9ea4-4c40-0729-d455d24e67c8}" testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.854Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: pipelining replication: peer="{Voter server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8 cf5bea6e-9ea4-4c40-0729-d455d24e67c8}" testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.855Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: pipelining replication: peer="{Voter server-019f2655-7706-c399-525b-39ceb5b19abb 019f2655-7706-c399-525b-39ceb5b19abb}" testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:01.855Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: pipelining replication: peer="{Voter server-019f2655-7706-c399-525b-39ceb5b19abb 019f2655-7706-c399-525b-39ceb5b19abb}" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:01.960Z [INFO] stable state for Follower reached at 2024-01-09 05:10:01.855223132 +0000 UTC m=+13.108274199 (2 nodes), 77.892793ms from start of poll, 79.728102ms from cluster start. Timeout at 2024-01-09 05:10:01.960328179 +0000 UTC m=+13.213379289, 105.10509ms after stability testing.go:149: cluster: 2024-01-09T05:10:01.960Z [INFO] stable state for Follower reached at 2024-01-09 05:10:01.855223132 +0000 UTC m=+13.108274199 (2 nodes), 77.892793ms from start of poll, 79.728102ms from cluster start. Timeout at 2024-01-09 05:10:01.960328179 +0000 UTC m=+13.213379289, 105.10509ms after stability testLoggerAdapter verbose: cluster: 2024-01-09T05:10:01.960Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:10:01.960Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-01-09T05:10:02.065Z [INFO] stable state for Leader reached at 2024-01-09 05:10:01.960492513 +0000 UTC m=+13.213543557 (1 nodes), 2.405µs from start of poll, 184.99746ms from cluster start. Timeout at 2024-01-09 05:10:02.065598472 +0000 UTC m=+13.318649667, 105.10611ms after stability testing.go:149: cluster: 2024-01-09T05:10:02.065Z [INFO] stable state for Leader reached at 2024-01-09 05:10:01.960492513 +0000 UTC m=+13.213543557 (1 nodes), 2.405µs from start of poll, 184.99746ms from cluster start. Timeout at 2024-01-09 05:10:02.065598472 +0000 UTC m=+13.318649667, 105.10611ms after stability testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.065Z [ERROR] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: failed to pipeline appendEntries: peer="{Voter server-019f2655-7706-c399-525b-39ceb5b19abb 019f2655-7706-c399-525b-39ceb5b19abb}" error="append pipeline closed" testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.065Z [ERROR] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: failed to pipeline appendEntries: peer="{Voter server-019f2655-7706-c399-525b-39ceb5b19abb 019f2655-7706-c399-525b-39ceb5b19abb}" error="append pipeline closed" testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.065Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: aborting pipeline replication: peer="{Voter server-019f2655-7706-c399-525b-39ceb5b19abb 019f2655-7706-c399-525b-39ceb5b19abb}" testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.065Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: aborting pipeline replication: peer="{Voter server-019f2655-7706-c399-525b-39ceb5b19abb 019f2655-7706-c399-525b-39ceb5b19abb}" testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.065Z [ERROR] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: failed to appendEntries to: peer="{Voter server-019f2655-7706-c399-525b-39ceb5b19abb 019f2655-7706-c399-525b-39ceb5b19abb}" error="failed to connect to peer: 019f2655-7706-c399-525b-39ceb5b19abb" testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.065Z [ERROR] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: failed to appendEntries to: peer="{Voter server-019f2655-7706-c399-525b-39ceb5b19abb 019f2655-7706-c399-525b-39ceb5b19abb}" error="failed to connect to peer: 019f2655-7706-c399-525b-39ceb5b19abb" testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.070Z [ERROR] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: failed to heartbeat to: peer=019f2655-7706-c399-525b-39ceb5b19abb error="failed to connect to peer: 019f2655-7706-c399-525b-39ceb5b19abb" testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.070Z [ERROR] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: failed to heartbeat to: peer=019f2655-7706-c399-525b-39ceb5b19abb error="failed to connect to peer: 019f2655-7706-c399-525b-39ceb5b19abb" testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.076Z [ERROR] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: failed to appendEntries to: peer="{Voter server-019f2655-7706-c399-525b-39ceb5b19abb 019f2655-7706-c399-525b-39ceb5b19abb}" error="failed to connect to peer: 019f2655-7706-c399-525b-39ceb5b19abb" testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.076Z [ERROR] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: failed to appendEntries to: peer="{Voter server-019f2655-7706-c399-525b-39ceb5b19abb 019f2655-7706-c399-525b-39ceb5b19abb}" error="failed to connect to peer: 019f2655-7706-c399-525b-39ceb5b19abb" raft_test.go:1362: [INFO] Finished apply without behind follower testLoggerAdapter verbose: server-019f2655-7706-c399-525b-39ceb5b19abb: 2024-01-09T05:10:02.076Z [ERROR] server-019f2655-7706-c399-525b-39ceb5b19abb: failed to take snapshot: error="nothing new to snapshot" testing.go:149: server-019f2655-7706-c399-525b-39ceb5b19abb: 2024-01-09T05:10:02.076Z [ERROR] server-019f2655-7706-c399-525b-39ceb5b19abb: failed to take snapshot: error="nothing new to snapshot" testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.076Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: starting snapshot up to: index=102 testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.076Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: starting snapshot up to: index=102 testing.go:153: 2024-01-09T05:10:02.076Z [INFO] creating new snapshot: path=/tmp/raft073156126/snapshots/2-102-1704777002076.tmp testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.083Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: compacting logs: from=1 to=92 testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.083Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.083Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: snapshot complete up to: index=102 testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.083Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: snapshot complete up to: index=102 testLoggerAdapter verbose: server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: 2024-01-09T05:10:02.083Z [INFO] server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: starting snapshot up to: index=87 testing.go:149: server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: 2024-01-09T05:10:02.083Z [INFO] server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: starting snapshot up to: index=87 testing.go:153: 2024-01-09T05:10:02.083Z [INFO] creating new snapshot: path=/tmp/raft911629056/snapshots/2-87-1704777002083.tmp testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.086Z [ERROR] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: failed to get log: index=2 error="log not found" testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.086Z [ERROR] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: failed to get log: index=2 error="log not found" testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.087Z [ERROR] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: failed to install snapshot: id=2-102-1704777002076 error="failed to connect to peer: 019f2655-7706-c399-525b-39ceb5b19abb" testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.087Z [ERROR] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: failed to install snapshot: id=2-102-1704777002076 error="failed to connect to peer: 019f2655-7706-c399-525b-39ceb5b19abb" testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.088Z [ERROR] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: failed to send snapshot to: peer="{Voter server-019f2655-7706-c399-525b-39ceb5b19abb 019f2655-7706-c399-525b-39ceb5b19abb}" error="failed to connect to peer: 019f2655-7706-c399-525b-39ceb5b19abb" testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.088Z [ERROR] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: failed to send snapshot to: peer="{Voter server-019f2655-7706-c399-525b-39ceb5b19abb 019f2655-7706-c399-525b-39ceb5b19abb}" error="failed to connect to peer: 019f2655-7706-c399-525b-39ceb5b19abb" testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.089Z [ERROR] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: failed to heartbeat to: peer=019f2655-7706-c399-525b-39ceb5b19abb error="failed to connect to peer: 019f2655-7706-c399-525b-39ceb5b19abb" testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.089Z [ERROR] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: failed to heartbeat to: peer=019f2655-7706-c399-525b-39ceb5b19abb error="failed to connect to peer: 019f2655-7706-c399-525b-39ceb5b19abb" testLoggerAdapter verbose: server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: 2024-01-09T05:10:02.091Z [INFO] server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: compacting logs: from=1 to=87 testing.go:149: server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: 2024-01-09T05:10:02.091Z [INFO] server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: compacting logs: from=1 to=87 testLoggerAdapter verbose: server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: 2024-01-09T05:10:02.091Z [INFO] server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: snapshot complete up to: index=87 testing.go:149: server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: 2024-01-09T05:10:02.091Z [INFO] server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8: snapshot complete up to: index=87 raft_test.go:1383: [INFO] Finished apply without behind follower testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.111Z [ERROR] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: failed to get log: index=2 error="log not found" testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.111Z [ERROR] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: failed to get log: index=2 error="log not found" testing.go:153: 2024-01-09T05:10:02.112Z [INFO] creating new snapshot: path=/tmp/raft215901964/snapshots/2-102-1704777002112.tmp testLoggerAdapter verbose: server-019f2655-7706-c399-525b-39ceb5b19abb: 2024-01-09T05:10:02.118Z [INFO] server-019f2655-7706-c399-525b-39ceb5b19abb: copied to local snapshot: bytes=693 testing.go:149: server-019f2655-7706-c399-525b-39ceb5b19abb: 2024-01-09T05:10:02.118Z [INFO] server-019f2655-7706-c399-525b-39ceb5b19abb: copied to local snapshot: bytes=693 testLoggerAdapter verbose: server-019f2655-7706-c399-525b-39ceb5b19abb: 2024-01-09T05:10:02.118Z [INFO] server-019f2655-7706-c399-525b-39ceb5b19abb: Installed remote snapshot testing.go:149: server-019f2655-7706-c399-525b-39ceb5b19abb: 2024-01-09T05:10:02.118Z [INFO] server-019f2655-7706-c399-525b-39ceb5b19abb: Installed remote snapshot testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.118Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: pipelining replication: peer="{Voter server-019f2655-7706-c399-525b-39ceb5b19abb 019f2655-7706-c399-525b-39ceb5b19abb}" testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.118Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: pipelining replication: peer="{Voter server-019f2655-7706-c399-525b-39ceb5b19abb 019f2655-7706-c399-525b-39ceb5b19abb}" testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.131Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: aborting pipeline replication: peer="{Voter server-019f2655-7706-c399-525b-39ceb5b19abb 019f2655-7706-c399-525b-39ceb5b19abb}" testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.131Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: aborting pipeline replication: peer="{Voter server-019f2655-7706-c399-525b-39ceb5b19abb 019f2655-7706-c399-525b-39ceb5b19abb}" testLoggerAdapter verbose: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.131Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: aborting pipeline replication: peer="{Voter server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8 cf5bea6e-9ea4-4c40-0729-d455d24e67c8}" testing.go:149: server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: 2024-01-09T05:10:02.131Z [INFO] server-2a03ca26-444b-58cf-c7ba-203a0d05a1b6: aborting pipeline replication: peer="{Voter server-cf5bea6e-9ea4-4c40-0729-d455d24e67c8 cf5bea6e-9ea4-4c40-0729-d455d24e67c8}" --- PASS: TestRaft_SendSnapshotAndLogsFollower (0.36s) === RUN TestRaft_ReJoinFollower testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.133Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b Address:0dc470d6-a011-e4fd-046f-e7ec51b30d4b} {Suffrage:Voter ID:server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 Address:22f7ee5a-2e3f-78b5-a317-00ccd58a6c30} {Suffrage:Voter ID:server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 Address:005b5ae9-219d-6c40-ef9c-25f3ac9723e1}]" testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.133Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b Address:0dc470d6-a011-e4fd-046f-e7ec51b30d4b} {Suffrage:Voter ID:server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 Address:22f7ee5a-2e3f-78b5-a317-00ccd58a6c30} {Suffrage:Voter ID:server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 Address:005b5ae9-219d-6c40-ef9c-25f3ac9723e1}]" testLoggerAdapter verbose: server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: 2024-01-09T05:10:02.133Z [INFO] server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b Address:0dc470d6-a011-e4fd-046f-e7ec51b30d4b} {Suffrage:Voter ID:server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 Address:22f7ee5a-2e3f-78b5-a317-00ccd58a6c30} {Suffrage:Voter ID:server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 Address:005b5ae9-219d-6c40-ef9c-25f3ac9723e1}]" testing.go:149: server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: 2024-01-09T05:10:02.133Z [INFO] server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b Address:0dc470d6-a011-e4fd-046f-e7ec51b30d4b} {Suffrage:Voter ID:server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 Address:22f7ee5a-2e3f-78b5-a317-00ccd58a6c30} {Suffrage:Voter ID:server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 Address:005b5ae9-219d-6c40-ef9c-25f3ac9723e1}]" testLoggerAdapter verbose: server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1: 2024-01-09T05:10:02.133Z [INFO] server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b Address:0dc470d6-a011-e4fd-046f-e7ec51b30d4b} {Suffrage:Voter ID:server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 Address:22f7ee5a-2e3f-78b5-a317-00ccd58a6c30} {Suffrage:Voter ID:server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 Address:005b5ae9-219d-6c40-ef9c-25f3ac9723e1}]" testing.go:149: server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1: 2024-01-09T05:10:02.133Z [INFO] server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b Address:0dc470d6-a011-e4fd-046f-e7ec51b30d4b} {Suffrage:Voter ID:server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 Address:22f7ee5a-2e3f-78b5-a317-00ccd58a6c30} {Suffrage:Voter ID:server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 Address:005b5ae9-219d-6c40-ef9c-25f3ac9723e1}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:02.133Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:10:02.133Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.133Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: entering follower state: follower="Node at 0dc470d6-a011-e4fd-046f-e7ec51b30d4b [Follower]" leader= testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.133Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: entering follower state: follower="Node at 0dc470d6-a011-e4fd-046f-e7ec51b30d4b [Follower]" leader= testLoggerAdapter verbose: server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: 2024-01-09T05:10:02.133Z [INFO] server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: entering follower state: follower="Node at 22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 [Follower]" leader= testing.go:149: server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: 2024-01-09T05:10:02.133Z [INFO] server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: entering follower state: follower="Node at 22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 [Follower]" leader= testLoggerAdapter verbose: server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1: 2024-01-09T05:10:02.133Z [INFO] server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1: entering follower state: follower="Node at 005b5ae9-219d-6c40-ef9c-25f3ac9723e1 [Follower]" leader= testing.go:149: server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1: 2024-01-09T05:10:02.133Z [INFO] server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1: entering follower state: follower="Node at 005b5ae9-219d-6c40-ef9c-25f3ac9723e1 [Follower]" leader= testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.202Z [WARN] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.202Z [WARN] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.202Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: entering candidate state: node="Node at 0dc470d6-a011-e4fd-046f-e7ec51b30d4b [Candidate]" term=2 testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.202Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: entering candidate state: node="Node at 0dc470d6-a011-e4fd-046f-e7ec51b30d4b [Candidate]" term=2 testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.202Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: election won: tally=2 testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.202Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: election won: tally=2 testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.202Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: entering leader state: leader="Node at 0dc470d6-a011-e4fd-046f-e7ec51b30d4b [Leader]" testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.202Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: entering leader state: leader="Node at 0dc470d6-a011-e4fd-046f-e7ec51b30d4b [Leader]" testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.202Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: added peer, starting replication: peer=server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.202Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: added peer, starting replication: peer=server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.202Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: added peer, starting replication: peer=server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.202Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: added peer, starting replication: peer=server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.203Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: pipelining replication: peer="{Voter server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 005b5ae9-219d-6c40-ef9c-25f3ac9723e1}" testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.203Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: pipelining replication: peer="{Voter server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 005b5ae9-219d-6c40-ef9c-25f3ac9723e1}" testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.203Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: pipelining replication: peer="{Voter server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 22f7ee5a-2e3f-78b5-a317-00ccd58a6c30}" testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.203Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: pipelining replication: peer="{Voter server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 22f7ee5a-2e3f-78b5-a317-00ccd58a6c30}" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:02.308Z [INFO] stable state for Leader reached at 2024-01-09 05:10:02.203852697 +0000 UTC m=+13.456903662 (1 nodes), 70.195837ms from start of poll, 70.609195ms from cluster start. Timeout at 2024-01-09 05:10:02.308937202 +0000 UTC m=+13.561988189, 105.084527ms after stability testing.go:149: cluster: 2024-01-09T05:10:02.308Z [INFO] stable state for Leader reached at 2024-01-09 05:10:02.203852697 +0000 UTC m=+13.456903662 (1 nodes), 70.195837ms from start of poll, 70.609195ms from cluster start. Timeout at 2024-01-09 05:10:02.308937202 +0000 UTC m=+13.561988189, 105.084527ms after stability testLoggerAdapter verbose: cluster: 2024-01-09T05:10:02.314Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-01-09T05:10:02.314Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2024-01-09T05:10:02.419Z [INFO] stable state for Follower reached at 2024-01-09 05:10:02.314317619 +0000 UTC m=+13.567368576 (2 nodes), 2.362µs from start of poll, 181.074109ms from cluster start. Timeout at 2024-01-09 05:10:02.419325728 +0000 UTC m=+13.672376772, 105.008196ms after stability testing.go:149: cluster: 2024-01-09T05:10:02.419Z [INFO] stable state for Follower reached at 2024-01-09 05:10:02.314317619 +0000 UTC m=+13.567368576 (2 nodes), 2.362µs from start of poll, 181.074109ms from cluster start. Timeout at 2024-01-09 05:10:02.419325728 +0000 UTC m=+13.672376772, 105.008196ms after stability testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.419Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: updating configuration: command=RemoveServer server-id=server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 server-addr= servers="[{Suffrage:Voter ID:server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b Address:0dc470d6-a011-e4fd-046f-e7ec51b30d4b} {Suffrage:Voter ID:server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 Address:005b5ae9-219d-6c40-ef9c-25f3ac9723e1}]" testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.419Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: updating configuration: command=RemoveServer server-id=server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 server-addr= servers="[{Suffrage:Voter ID:server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b Address:0dc470d6-a011-e4fd-046f-e7ec51b30d4b} {Suffrage:Voter ID:server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 Address:005b5ae9-219d-6c40-ef9c-25f3ac9723e1}]" testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.419Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: removed peer, stopping replication: peer=server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 last-index=3 testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.419Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: removed peer, stopping replication: peer=server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 last-index=3 testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.419Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: aborting pipeline replication: peer="{Voter server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 22f7ee5a-2e3f-78b5-a317-00ccd58a6c30}" testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.419Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: aborting pipeline replication: peer="{Voter server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 22f7ee5a-2e3f-78b5-a317-00ccd58a6c30}" testLoggerAdapter verbose: server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: 2024-01-09T05:10:02.508Z [WARN] server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: heartbeat timeout reached, starting election: last-leader=0dc470d6-a011-e4fd-046f-e7ec51b30d4b testing.go:149: server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: 2024-01-09T05:10:02.508Z [WARN] server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: heartbeat timeout reached, starting election: last-leader=0dc470d6-a011-e4fd-046f-e7ec51b30d4b testLoggerAdapter verbose: server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: 2024-01-09T05:10:02.508Z [INFO] server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: entering candidate state: node="Node at 22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 [Candidate]" term=3 testing.go:149: server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: 2024-01-09T05:10:02.508Z [INFO] server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: entering candidate state: node="Node at 22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 [Candidate]" term=3 testLoggerAdapter verbose: server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1: 2024-01-09T05:10:02.508Z [WARN] server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1: rejecting vote request since we have a leader: from=22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 leader=0dc470d6-a011-e4fd-046f-e7ec51b30d4b testing.go:149: server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1: 2024-01-09T05:10:02.508Z [WARN] server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1: rejecting vote request since we have a leader: from=22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 leader=0dc470d6-a011-e4fd-046f-e7ec51b30d4b testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.508Z [WARN] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: rejecting vote request since we have a leader: from=22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 leader=0dc470d6-a011-e4fd-046f-e7ec51b30d4b testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.508Z [WARN] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: rejecting vote request since we have a leader: from=22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 leader=0dc470d6-a011-e4fd-046f-e7ec51b30d4b testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.525Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: updating configuration: command=AddStaging server-id=server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 server-addr=22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 servers="[{Suffrage:Voter ID:server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b Address:0dc470d6-a011-e4fd-046f-e7ec51b30d4b} {Suffrage:Voter ID:server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 Address:005b5ae9-219d-6c40-ef9c-25f3ac9723e1} {Suffrage:Voter ID:server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 Address:22f7ee5a-2e3f-78b5-a317-00ccd58a6c30}]" testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.525Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: updating configuration: command=AddStaging server-id=server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 server-addr=22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 servers="[{Suffrage:Voter ID:server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b Address:0dc470d6-a011-e4fd-046f-e7ec51b30d4b} {Suffrage:Voter ID:server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 Address:005b5ae9-219d-6c40-ef9c-25f3ac9723e1} {Suffrage:Voter ID:server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 Address:22f7ee5a-2e3f-78b5-a317-00ccd58a6c30}]" testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.525Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: added peer, starting replication: peer=server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.525Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: added peer, starting replication: peer=server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.526Z [ERROR] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: peer has newer term, stopping replication: peer="{Voter server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 22f7ee5a-2e3f-78b5-a317-00ccd58a6c30}" testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.526Z [ERROR] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: peer has newer term, stopping replication: peer="{Voter server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 22f7ee5a-2e3f-78b5-a317-00ccd58a6c30}" testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.526Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: entering follower state: follower="Node at 0dc470d6-a011-e4fd-046f-e7ec51b30d4b [Follower]" leader= testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.526Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: entering follower state: follower="Node at 0dc470d6-a011-e4fd-046f-e7ec51b30d4b [Follower]" leader= testLoggerAdapter verbose: cluster: 2024-01-09T05:10:02.526Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:10:02.526Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.526Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: aborting pipeline replication: peer="{Voter server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 005b5ae9-219d-6c40-ef9c-25f3ac9723e1}" testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.526Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: aborting pipeline replication: peer="{Voter server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 005b5ae9-219d-6c40-ef9c-25f3ac9723e1}" testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.591Z [WARN] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.591Z [WARN] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.591Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: entering candidate state: node="Node at 0dc470d6-a011-e4fd-046f-e7ec51b30d4b [Candidate]" term=3 testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.591Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: entering candidate state: node="Node at 0dc470d6-a011-e4fd-046f-e7ec51b30d4b [Candidate]" term=3 testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.591Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: election won: tally=2 testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.591Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: election won: tally=2 testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.591Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: entering leader state: leader="Node at 0dc470d6-a011-e4fd-046f-e7ec51b30d4b [Leader]" testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.591Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: entering leader state: leader="Node at 0dc470d6-a011-e4fd-046f-e7ec51b30d4b [Leader]" testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.591Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: added peer, starting replication: peer=server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.591Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: added peer, starting replication: peer=server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.591Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: added peer, starting replication: peer=server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.591Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: added peer, starting replication: peer=server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 testLoggerAdapter verbose: server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: 2024-01-09T05:10:02.591Z [WARN] server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: failed to get previous log: previous-index=4 last-index=3 error="log not found" testing.go:149: server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: 2024-01-09T05:10:02.591Z [WARN] server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: failed to get previous log: previous-index=4 last-index=3 error="log not found" testLoggerAdapter verbose: server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: 2024-01-09T05:10:02.591Z [INFO] server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: entering follower state: follower="Node at 22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 [Follower]" leader=0dc470d6-a011-e4fd-046f-e7ec51b30d4b testing.go:149: server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: 2024-01-09T05:10:02.591Z [INFO] server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30: entering follower state: follower="Node at 22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 [Follower]" leader=0dc470d6-a011-e4fd-046f-e7ec51b30d4b testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.591Z [WARN] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: appendEntries rejected, sending older logs: peer="{Voter server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 22f7ee5a-2e3f-78b5-a317-00ccd58a6c30}" next=4 testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.591Z [WARN] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: appendEntries rejected, sending older logs: peer="{Voter server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 22f7ee5a-2e3f-78b5-a317-00ccd58a6c30}" next=4 testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.591Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: pipelining replication: peer="{Voter server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 22f7ee5a-2e3f-78b5-a317-00ccd58a6c30}" testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.591Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: pipelining replication: peer="{Voter server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 22f7ee5a-2e3f-78b5-a317-00ccd58a6c30}" testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.591Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: pipelining replication: peer="{Voter server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 005b5ae9-219d-6c40-ef9c-25f3ac9723e1}" testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.591Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: pipelining replication: peer="{Voter server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 005b5ae9-219d-6c40-ef9c-25f3ac9723e1}" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:02.696Z [INFO] stable state for Leader reached at 2024-01-09 05:10:02.59167866 +0000 UTC m=+13.844729648 (1 nodes), 65.55793ms from start of poll, 458.435181ms from cluster start. Timeout at 2024-01-09 05:10:02.696696391 +0000 UTC m=+13.949747444, 105.017796ms after stability testing.go:149: cluster: 2024-01-09T05:10:02.696Z [INFO] stable state for Leader reached at 2024-01-09 05:10:02.59167866 +0000 UTC m=+13.844729648 (1 nodes), 65.55793ms from start of poll, 458.435181ms from cluster start. Timeout at 2024-01-09 05:10:02.696696391 +0000 UTC m=+13.949747444, 105.017796ms after stability testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.696Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: aborting pipeline replication: peer="{Voter server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 005b5ae9-219d-6c40-ef9c-25f3ac9723e1}" testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.696Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: aborting pipeline replication: peer="{Voter server-005b5ae9-219d-6c40-ef9c-25f3ac9723e1 005b5ae9-219d-6c40-ef9c-25f3ac9723e1}" testLoggerAdapter verbose: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.697Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: aborting pipeline replication: peer="{Voter server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 22f7ee5a-2e3f-78b5-a317-00ccd58a6c30}" testing.go:149: server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: 2024-01-09T05:10:02.697Z [INFO] server-0dc470d6-a011-e4fd-046f-e7ec51b30d4b: aborting pipeline replication: peer="{Voter server-22f7ee5a-2e3f-78b5-a317-00ccd58a6c30 22f7ee5a-2e3f-78b5-a317-00ccd58a6c30}" --- PASS: TestRaft_ReJoinFollower (0.57s) === RUN TestRaft_LeaderLeaseExpire testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.699Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8617de21-0aaf-40bb-57ea-6dd62fb443e3 Address:8617de21-0aaf-40bb-57ea-6dd62fb443e3} {Suffrage:Voter ID:server-709f3e06-aa57-e0d5-215e-a9989d22e81c Address:709f3e06-aa57-e0d5-215e-a9989d22e81c}]" testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.699Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8617de21-0aaf-40bb-57ea-6dd62fb443e3 Address:8617de21-0aaf-40bb-57ea-6dd62fb443e3} {Suffrage:Voter ID:server-709f3e06-aa57-e0d5-215e-a9989d22e81c Address:709f3e06-aa57-e0d5-215e-a9989d22e81c}]" testLoggerAdapter verbose: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:02.699Z [INFO] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8617de21-0aaf-40bb-57ea-6dd62fb443e3 Address:8617de21-0aaf-40bb-57ea-6dd62fb443e3} {Suffrage:Voter ID:server-709f3e06-aa57-e0d5-215e-a9989d22e81c Address:709f3e06-aa57-e0d5-215e-a9989d22e81c}]" testing.go:149: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:02.699Z [INFO] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8617de21-0aaf-40bb-57ea-6dd62fb443e3 Address:8617de21-0aaf-40bb-57ea-6dd62fb443e3} {Suffrage:Voter ID:server-709f3e06-aa57-e0d5-215e-a9989d22e81c Address:709f3e06-aa57-e0d5-215e-a9989d22e81c}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:02.699Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:10:02.699Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:02.699Z [INFO] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: entering follower state: follower="Node at 709f3e06-aa57-e0d5-215e-a9989d22e81c [Follower]" leader= testing.go:149: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:02.699Z [INFO] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: entering follower state: follower="Node at 709f3e06-aa57-e0d5-215e-a9989d22e81c [Follower]" leader= testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.699Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: entering follower state: follower="Node at 8617de21-0aaf-40bb-57ea-6dd62fb443e3 [Follower]" leader= testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.699Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: entering follower state: follower="Node at 8617de21-0aaf-40bb-57ea-6dd62fb443e3 [Follower]" leader= testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.750Z [WARN] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.750Z [WARN] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.751Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: entering candidate state: node="Node at 8617de21-0aaf-40bb-57ea-6dd62fb443e3 [Candidate]" term=2 testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.751Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: entering candidate state: node="Node at 8617de21-0aaf-40bb-57ea-6dd62fb443e3 [Candidate]" term=2 testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.751Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: election won: tally=2 testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.751Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: election won: tally=2 testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.751Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: entering leader state: leader="Node at 8617de21-0aaf-40bb-57ea-6dd62fb443e3 [Leader]" testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.751Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: entering leader state: leader="Node at 8617de21-0aaf-40bb-57ea-6dd62fb443e3 [Leader]" testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.751Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: added peer, starting replication: peer=server-709f3e06-aa57-e0d5-215e-a9989d22e81c testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.751Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: added peer, starting replication: peer=server-709f3e06-aa57-e0d5-215e-a9989d22e81c testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.751Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: pipelining replication: peer="{Voter server-709f3e06-aa57-e0d5-215e-a9989d22e81c 709f3e06-aa57-e0d5-215e-a9989d22e81c}" testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.751Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: pipelining replication: peer="{Voter server-709f3e06-aa57-e0d5-215e-a9989d22e81c 709f3e06-aa57-e0d5-215e-a9989d22e81c}" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:02.856Z [INFO] stable state for Leader reached at 2024-01-09 05:10:02.751340387 +0000 UTC m=+14.004391361 (1 nodes), 52.063478ms from start of poll, 52.390421ms from cluster start. Timeout at 2024-01-09 05:10:02.856582914 +0000 UTC m=+14.109633895, 105.242534ms after stability testing.go:149: cluster: 2024-01-09T05:10:02.856Z [INFO] stable state for Leader reached at 2024-01-09 05:10:02.751340387 +0000 UTC m=+14.004391361 (1 nodes), 52.063478ms from start of poll, 52.390421ms from cluster start. Timeout at 2024-01-09 05:10:02.856582914 +0000 UTC m=+14.109633895, 105.242534ms after stability testLoggerAdapter verbose: cluster: 2024-01-09T05:10:02.861Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-01-09T05:10:02.861Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2024-01-09T05:10:02.967Z [INFO] stable state for Follower reached at 2024-01-09 05:10:02.86185176 +0000 UTC m=+14.114902728 (1 nodes), 2.075µs from start of poll, 162.901788ms from cluster start. Timeout at 2024-01-09 05:10:02.967508918 +0000 UTC m=+14.220559947, 105.657219ms after stability testing.go:149: cluster: 2024-01-09T05:10:02.967Z [INFO] stable state for Follower reached at 2024-01-09 05:10:02.86185176 +0000 UTC m=+14.114902728 (1 nodes), 2.075µs from start of poll, 162.901788ms from cluster start. Timeout at 2024-01-09 05:10:02.967508918 +0000 UTC m=+14.220559947, 105.657219ms after stability raft_test.go:1492: [INFO] Disconnecting Node at 709f3e06-aa57-e0d5-215e-a9989d22e81c [Follower] testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.969Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to heartbeat to: peer=709f3e06-aa57-e0d5-215e-a9989d22e81c error="failed to connect to peer: 709f3e06-aa57-e0d5-215e-a9989d22e81c" testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.969Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to heartbeat to: peer=709f3e06-aa57-e0d5-215e-a9989d22e81c error="failed to connect to peer: 709f3e06-aa57-e0d5-215e-a9989d22e81c" testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.970Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to pipeline appendEntries: peer="{Voter server-709f3e06-aa57-e0d5-215e-a9989d22e81c 709f3e06-aa57-e0d5-215e-a9989d22e81c}" error="append pipeline closed" testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.970Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to pipeline appendEntries: peer="{Voter server-709f3e06-aa57-e0d5-215e-a9989d22e81c 709f3e06-aa57-e0d5-215e-a9989d22e81c}" error="append pipeline closed" testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.970Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: aborting pipeline replication: peer="{Voter server-709f3e06-aa57-e0d5-215e-a9989d22e81c 709f3e06-aa57-e0d5-215e-a9989d22e81c}" testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.970Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: aborting pipeline replication: peer="{Voter server-709f3e06-aa57-e0d5-215e-a9989d22e81c 709f3e06-aa57-e0d5-215e-a9989d22e81c}" testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.978Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to appendEntries to: peer="{Voter server-709f3e06-aa57-e0d5-215e-a9989d22e81c 709f3e06-aa57-e0d5-215e-a9989d22e81c}" error="failed to connect to peer: 709f3e06-aa57-e0d5-215e-a9989d22e81c" testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.978Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to appendEntries to: peer="{Voter server-709f3e06-aa57-e0d5-215e-a9989d22e81c 709f3e06-aa57-e0d5-215e-a9989d22e81c}" error="failed to connect to peer: 709f3e06-aa57-e0d5-215e-a9989d22e81c" testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.987Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to heartbeat to: peer=709f3e06-aa57-e0d5-215e-a9989d22e81c error="failed to connect to peer: 709f3e06-aa57-e0d5-215e-a9989d22e81c" testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.987Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to heartbeat to: peer=709f3e06-aa57-e0d5-215e-a9989d22e81c error="failed to connect to peer: 709f3e06-aa57-e0d5-215e-a9989d22e81c" testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.995Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to appendEntries to: peer="{Voter server-709f3e06-aa57-e0d5-215e-a9989d22e81c 709f3e06-aa57-e0d5-215e-a9989d22e81c}" error="failed to connect to peer: 709f3e06-aa57-e0d5-215e-a9989d22e81c" testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:02.995Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to appendEntries to: peer="{Voter server-709f3e06-aa57-e0d5-215e-a9989d22e81c 709f3e06-aa57-e0d5-215e-a9989d22e81c}" error="failed to connect to peer: 709f3e06-aa57-e0d5-215e-a9989d22e81c" testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.008Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to heartbeat to: peer=709f3e06-aa57-e0d5-215e-a9989d22e81c error="failed to connect to peer: 709f3e06-aa57-e0d5-215e-a9989d22e81c" testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.008Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to heartbeat to: peer=709f3e06-aa57-e0d5-215e-a9989d22e81c error="failed to connect to peer: 709f3e06-aa57-e0d5-215e-a9989d22e81c" testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.013Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to appendEntries to: peer="{Voter server-709f3e06-aa57-e0d5-215e-a9989d22e81c 709f3e06-aa57-e0d5-215e-a9989d22e81c}" error="failed to connect to peer: 709f3e06-aa57-e0d5-215e-a9989d22e81c" testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.013Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to appendEntries to: peer="{Voter server-709f3e06-aa57-e0d5-215e-a9989d22e81c 709f3e06-aa57-e0d5-215e-a9989d22e81c}" error="failed to connect to peer: 709f3e06-aa57-e0d5-215e-a9989d22e81c" testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.015Z [WARN] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to contact: server-id=server-709f3e06-aa57-e0d5-215e-a9989d22e81c time=50.092686ms testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.015Z [WARN] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to contact: server-id=server-709f3e06-aa57-e0d5-215e-a9989d22e81c time=50.092686ms testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.015Z [WARN] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to contact quorum of nodes, stepping down testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.015Z [WARN] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to contact quorum of nodes, stepping down testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.015Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: entering follower state: follower="Node at 8617de21-0aaf-40bb-57ea-6dd62fb443e3 [Follower]" leader= testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.015Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: entering follower state: follower="Node at 8617de21-0aaf-40bb-57ea-6dd62fb443e3 [Follower]" leader= testLoggerAdapter verbose: cluster: 2024-01-09T05:10:03.015Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:10:03.015Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:03.095Z [WARN] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: heartbeat timeout reached, starting election: last-leader=8617de21-0aaf-40bb-57ea-6dd62fb443e3 testing.go:149: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:03.095Z [WARN] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: heartbeat timeout reached, starting election: last-leader=8617de21-0aaf-40bb-57ea-6dd62fb443e3 testLoggerAdapter verbose: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:03.095Z [INFO] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: entering candidate state: node="Node at 709f3e06-aa57-e0d5-215e-a9989d22e81c [Candidate]" term=3 testing.go:149: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:03.095Z [INFO] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: entering candidate state: node="Node at 709f3e06-aa57-e0d5-215e-a9989d22e81c [Candidate]" term=3 testLoggerAdapter verbose: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:03.095Z [ERROR] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: failed to make requestVote RPC: target="{Voter server-8617de21-0aaf-40bb-57ea-6dd62fb443e3 8617de21-0aaf-40bb-57ea-6dd62fb443e3}" error="failed to connect to peer: 8617de21-0aaf-40bb-57ea-6dd62fb443e3" testing.go:149: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:03.095Z [ERROR] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: failed to make requestVote RPC: target="{Voter server-8617de21-0aaf-40bb-57ea-6dd62fb443e3 8617de21-0aaf-40bb-57ea-6dd62fb443e3}" error="failed to connect to peer: 8617de21-0aaf-40bb-57ea-6dd62fb443e3" testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.097Z [WARN] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.097Z [WARN] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.097Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: entering candidate state: node="Node at 8617de21-0aaf-40bb-57ea-6dd62fb443e3 [Candidate]" term=3 testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.097Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: entering candidate state: node="Node at 8617de21-0aaf-40bb-57ea-6dd62fb443e3 [Candidate]" term=3 testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.098Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to make requestVote RPC: target="{Voter server-709f3e06-aa57-e0d5-215e-a9989d22e81c 709f3e06-aa57-e0d5-215e-a9989d22e81c}" error="failed to connect to peer: 709f3e06-aa57-e0d5-215e-a9989d22e81c" testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.098Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to make requestVote RPC: target="{Voter server-709f3e06-aa57-e0d5-215e-a9989d22e81c 709f3e06-aa57-e0d5-215e-a9989d22e81c}" error="failed to connect to peer: 709f3e06-aa57-e0d5-215e-a9989d22e81c" testLoggerAdapter verbose: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:03.164Z [WARN] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: Election timeout reached, restarting election testing.go:149: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:03.164Z [WARN] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: Election timeout reached, restarting election testLoggerAdapter verbose: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:03.164Z [INFO] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: entering candidate state: node="Node at 709f3e06-aa57-e0d5-215e-a9989d22e81c [Candidate]" term=4 testing.go:149: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:03.164Z [INFO] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: entering candidate state: node="Node at 709f3e06-aa57-e0d5-215e-a9989d22e81c [Candidate]" term=4 testLoggerAdapter verbose: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:03.164Z [ERROR] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: failed to make requestVote RPC: target="{Voter server-8617de21-0aaf-40bb-57ea-6dd62fb443e3 8617de21-0aaf-40bb-57ea-6dd62fb443e3}" error="failed to connect to peer: 8617de21-0aaf-40bb-57ea-6dd62fb443e3" testing.go:149: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:03.164Z [ERROR] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: failed to make requestVote RPC: target="{Voter server-8617de21-0aaf-40bb-57ea-6dd62fb443e3 8617de21-0aaf-40bb-57ea-6dd62fb443e3}" error="failed to connect to peer: 8617de21-0aaf-40bb-57ea-6dd62fb443e3" testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.190Z [WARN] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: Election timeout reached, restarting election testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.190Z [WARN] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: Election timeout reached, restarting election testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.191Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: entering candidate state: node="Node at 8617de21-0aaf-40bb-57ea-6dd62fb443e3 [Candidate]" term=4 testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.191Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: entering candidate state: node="Node at 8617de21-0aaf-40bb-57ea-6dd62fb443e3 [Candidate]" term=4 testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.191Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to make requestVote RPC: target="{Voter server-709f3e06-aa57-e0d5-215e-a9989d22e81c 709f3e06-aa57-e0d5-215e-a9989d22e81c}" error="failed to connect to peer: 709f3e06-aa57-e0d5-215e-a9989d22e81c" testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.191Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to make requestVote RPC: target="{Voter server-709f3e06-aa57-e0d5-215e-a9989d22e81c 709f3e06-aa57-e0d5-215e-a9989d22e81c}" error="failed to connect to peer: 709f3e06-aa57-e0d5-215e-a9989d22e81c" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:03.200Z [INFO] stable state for Leader reached at 2024-01-09 05:10:03.095257568 +0000 UTC m=+14.348308620 (0 nodes), 79.274905ms from start of poll, 396.30768ms from cluster start. Timeout at 2024-01-09 05:10:03.20036977 +0000 UTC m=+14.453420837, 105.112217ms after stability testing.go:149: cluster: 2024-01-09T05:10:03.200Z [INFO] stable state for Leader reached at 2024-01-09 05:10:03.095257568 +0000 UTC m=+14.348308620 (0 nodes), 79.274905ms from start of poll, 396.30768ms from cluster start. Timeout at 2024-01-09 05:10:03.20036977 +0000 UTC m=+14.453420837, 105.112217ms after stability testLoggerAdapter verbose: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:03.243Z [WARN] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: Election timeout reached, restarting election testing.go:149: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:03.243Z [WARN] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: Election timeout reached, restarting election testLoggerAdapter verbose: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:03.243Z [INFO] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: entering candidate state: node="Node at 709f3e06-aa57-e0d5-215e-a9989d22e81c [Candidate]" term=5 testing.go:149: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:03.243Z [INFO] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: entering candidate state: node="Node at 709f3e06-aa57-e0d5-215e-a9989d22e81c [Candidate]" term=5 testLoggerAdapter verbose: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:03.243Z [ERROR] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: failed to make requestVote RPC: target="{Voter server-8617de21-0aaf-40bb-57ea-6dd62fb443e3 8617de21-0aaf-40bb-57ea-6dd62fb443e3}" error="failed to connect to peer: 8617de21-0aaf-40bb-57ea-6dd62fb443e3" testing.go:149: server-709f3e06-aa57-e0d5-215e-a9989d22e81c: 2024-01-09T05:10:03.243Z [ERROR] server-709f3e06-aa57-e0d5-215e-a9989d22e81c: failed to make requestVote RPC: target="{Voter server-8617de21-0aaf-40bb-57ea-6dd62fb443e3 8617de21-0aaf-40bb-57ea-6dd62fb443e3}" error="failed to connect to peer: 8617de21-0aaf-40bb-57ea-6dd62fb443e3" testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.273Z [WARN] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: Election timeout reached, restarting election testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.273Z [WARN] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: Election timeout reached, restarting election testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.273Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: entering candidate state: node="Node at 8617de21-0aaf-40bb-57ea-6dd62fb443e3 [Candidate]" term=5 testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.273Z [INFO] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: entering candidate state: node="Node at 8617de21-0aaf-40bb-57ea-6dd62fb443e3 [Candidate]" term=5 testLoggerAdapter verbose: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.273Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to make requestVote RPC: target="{Voter server-709f3e06-aa57-e0d5-215e-a9989d22e81c 709f3e06-aa57-e0d5-215e-a9989d22e81c}" error="failed to connect to peer: 709f3e06-aa57-e0d5-215e-a9989d22e81c" testing.go:149: server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: 2024-01-09T05:10:03.273Z [ERROR] server-8617de21-0aaf-40bb-57ea-6dd62fb443e3: failed to make requestVote RPC: target="{Voter server-709f3e06-aa57-e0d5-215e-a9989d22e81c 709f3e06-aa57-e0d5-215e-a9989d22e81c}" error="failed to connect to peer: 709f3e06-aa57-e0d5-215e-a9989d22e81c" --- PASS: TestRaft_LeaderLeaseExpire (0.61s) === RUN TestRaft_Barrier testLoggerAdapter verbose: server-9a014d33-ecd4-ff65-9072-f7ec18431f63: 2024-01-09T05:10:03.308Z [INFO] server-9a014d33-ecd4-ff65-9072-f7ec18431f63: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a014d33-ecd4-ff65-9072-f7ec18431f63 Address:9a014d33-ecd4-ff65-9072-f7ec18431f63} {Suffrage:Voter ID:server-2488494b-ec57-e41d-646d-b34026aff9e6 Address:2488494b-ec57-e41d-646d-b34026aff9e6} {Suffrage:Voter ID:server-bebf1eda-9b5a-62fc-4805-da04bddb943a Address:bebf1eda-9b5a-62fc-4805-da04bddb943a}]" testing.go:149: server-9a014d33-ecd4-ff65-9072-f7ec18431f63: 2024-01-09T05:10:03.308Z [INFO] server-9a014d33-ecd4-ff65-9072-f7ec18431f63: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a014d33-ecd4-ff65-9072-f7ec18431f63 Address:9a014d33-ecd4-ff65-9072-f7ec18431f63} {Suffrage:Voter ID:server-2488494b-ec57-e41d-646d-b34026aff9e6 Address:2488494b-ec57-e41d-646d-b34026aff9e6} {Suffrage:Voter ID:server-bebf1eda-9b5a-62fc-4805-da04bddb943a Address:bebf1eda-9b5a-62fc-4805-da04bddb943a}]" testLoggerAdapter verbose: server-2488494b-ec57-e41d-646d-b34026aff9e6: 2024-01-09T05:10:03.309Z [INFO] server-2488494b-ec57-e41d-646d-b34026aff9e6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a014d33-ecd4-ff65-9072-f7ec18431f63 Address:9a014d33-ecd4-ff65-9072-f7ec18431f63} {Suffrage:Voter ID:server-2488494b-ec57-e41d-646d-b34026aff9e6 Address:2488494b-ec57-e41d-646d-b34026aff9e6} {Suffrage:Voter ID:server-bebf1eda-9b5a-62fc-4805-da04bddb943a Address:bebf1eda-9b5a-62fc-4805-da04bddb943a}]" testing.go:149: server-2488494b-ec57-e41d-646d-b34026aff9e6: 2024-01-09T05:10:03.309Z [INFO] server-2488494b-ec57-e41d-646d-b34026aff9e6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a014d33-ecd4-ff65-9072-f7ec18431f63 Address:9a014d33-ecd4-ff65-9072-f7ec18431f63} {Suffrage:Voter ID:server-2488494b-ec57-e41d-646d-b34026aff9e6 Address:2488494b-ec57-e41d-646d-b34026aff9e6} {Suffrage:Voter ID:server-bebf1eda-9b5a-62fc-4805-da04bddb943a Address:bebf1eda-9b5a-62fc-4805-da04bddb943a}]" testLoggerAdapter verbose: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.309Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a014d33-ecd4-ff65-9072-f7ec18431f63 Address:9a014d33-ecd4-ff65-9072-f7ec18431f63} {Suffrage:Voter ID:server-2488494b-ec57-e41d-646d-b34026aff9e6 Address:2488494b-ec57-e41d-646d-b34026aff9e6} {Suffrage:Voter ID:server-bebf1eda-9b5a-62fc-4805-da04bddb943a Address:bebf1eda-9b5a-62fc-4805-da04bddb943a}]" testing.go:149: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.309Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a014d33-ecd4-ff65-9072-f7ec18431f63 Address:9a014d33-ecd4-ff65-9072-f7ec18431f63} {Suffrage:Voter ID:server-2488494b-ec57-e41d-646d-b34026aff9e6 Address:2488494b-ec57-e41d-646d-b34026aff9e6} {Suffrage:Voter ID:server-bebf1eda-9b5a-62fc-4805-da04bddb943a Address:bebf1eda-9b5a-62fc-4805-da04bddb943a}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:03.309Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:10:03.309Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-9a014d33-ecd4-ff65-9072-f7ec18431f63: 2024-01-09T05:10:03.309Z [INFO] server-9a014d33-ecd4-ff65-9072-f7ec18431f63: entering follower state: follower="Node at 9a014d33-ecd4-ff65-9072-f7ec18431f63 [Follower]" leader= testing.go:149: server-9a014d33-ecd4-ff65-9072-f7ec18431f63: 2024-01-09T05:10:03.309Z [INFO] server-9a014d33-ecd4-ff65-9072-f7ec18431f63: entering follower state: follower="Node at 9a014d33-ecd4-ff65-9072-f7ec18431f63 [Follower]" leader= testLoggerAdapter verbose: server-2488494b-ec57-e41d-646d-b34026aff9e6: 2024-01-09T05:10:03.310Z [INFO] server-2488494b-ec57-e41d-646d-b34026aff9e6: entering follower state: follower="Node at 2488494b-ec57-e41d-646d-b34026aff9e6 [Follower]" leader= testing.go:149: server-2488494b-ec57-e41d-646d-b34026aff9e6: 2024-01-09T05:10:03.310Z [INFO] server-2488494b-ec57-e41d-646d-b34026aff9e6: entering follower state: follower="Node at 2488494b-ec57-e41d-646d-b34026aff9e6 [Follower]" leader= testLoggerAdapter verbose: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.310Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: entering follower state: follower="Node at bebf1eda-9b5a-62fc-4805-da04bddb943a [Follower]" leader= testing.go:149: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.310Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: entering follower state: follower="Node at bebf1eda-9b5a-62fc-4805-da04bddb943a [Follower]" leader= testLoggerAdapter verbose: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.393Z [WARN] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.393Z [WARN] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.393Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: entering candidate state: node="Node at bebf1eda-9b5a-62fc-4805-da04bddb943a [Candidate]" term=2 testing.go:149: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.393Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: entering candidate state: node="Node at bebf1eda-9b5a-62fc-4805-da04bddb943a [Candidate]" term=2 testLoggerAdapter verbose: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.393Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: election won: tally=2 testing.go:149: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.393Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: election won: tally=2 testLoggerAdapter verbose: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.393Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: entering leader state: leader="Node at bebf1eda-9b5a-62fc-4805-da04bddb943a [Leader]" testing.go:149: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.393Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: entering leader state: leader="Node at bebf1eda-9b5a-62fc-4805-da04bddb943a [Leader]" testLoggerAdapter verbose: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.393Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: added peer, starting replication: peer=server-9a014d33-ecd4-ff65-9072-f7ec18431f63 testing.go:149: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.393Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: added peer, starting replication: peer=server-9a014d33-ecd4-ff65-9072-f7ec18431f63 testLoggerAdapter verbose: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.393Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: added peer, starting replication: peer=server-2488494b-ec57-e41d-646d-b34026aff9e6 testing.go:149: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.393Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: added peer, starting replication: peer=server-2488494b-ec57-e41d-646d-b34026aff9e6 testLoggerAdapter verbose: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.393Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: pipelining replication: peer="{Voter server-9a014d33-ecd4-ff65-9072-f7ec18431f63 9a014d33-ecd4-ff65-9072-f7ec18431f63}" testing.go:149: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.393Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: pipelining replication: peer="{Voter server-9a014d33-ecd4-ff65-9072-f7ec18431f63 9a014d33-ecd4-ff65-9072-f7ec18431f63}" testLoggerAdapter verbose: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.393Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: pipelining replication: peer="{Voter server-2488494b-ec57-e41d-646d-b34026aff9e6 2488494b-ec57-e41d-646d-b34026aff9e6}" testing.go:149: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.393Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: pipelining replication: peer="{Voter server-2488494b-ec57-e41d-646d-b34026aff9e6 2488494b-ec57-e41d-646d-b34026aff9e6}" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:03.499Z [INFO] stable state for Leader reached at 2024-01-09 05:10:03.393609191 +0000 UTC m=+14.646660196 (1 nodes), 84.200457ms from start of poll, 84.800943ms from cluster start. Timeout at 2024-01-09 05:10:03.498963557 +0000 UTC m=+14.752014791, 105.354595ms after stability testing.go:149: cluster: 2024-01-09T05:10:03.499Z [INFO] stable state for Leader reached at 2024-01-09 05:10:03.393609191 +0000 UTC m=+14.646660196 (1 nodes), 84.200457ms from start of poll, 84.800943ms from cluster start. Timeout at 2024-01-09 05:10:03.498963557 +0000 UTC m=+14.752014791, 105.354595ms after stability testLoggerAdapter verbose: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.517Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: aborting pipeline replication: peer="{Voter server-9a014d33-ecd4-ff65-9072-f7ec18431f63 9a014d33-ecd4-ff65-9072-f7ec18431f63}" testing.go:149: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.517Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: aborting pipeline replication: peer="{Voter server-9a014d33-ecd4-ff65-9072-f7ec18431f63 9a014d33-ecd4-ff65-9072-f7ec18431f63}" testLoggerAdapter verbose: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.517Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: aborting pipeline replication: peer="{Voter server-2488494b-ec57-e41d-646d-b34026aff9e6 2488494b-ec57-e41d-646d-b34026aff9e6}" testing.go:149: server-bebf1eda-9b5a-62fc-4805-da04bddb943a: 2024-01-09T05:10:03.517Z [INFO] server-bebf1eda-9b5a-62fc-4805-da04bddb943a: aborting pipeline replication: peer="{Voter server-2488494b-ec57-e41d-646d-b34026aff9e6 2488494b-ec57-e41d-646d-b34026aff9e6}" --- PASS: TestRaft_Barrier (0.21s) === RUN TestRaft_VerifyLeader testLoggerAdapter verbose: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.519Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1c7c37b4-200b-3db9-5c3a-b14689d5edba Address:1c7c37b4-200b-3db9-5c3a-b14689d5edba} {Suffrage:Voter ID:server-3920bd6f-c487-705e-b37a-27e1c98948b6 Address:3920bd6f-c487-705e-b37a-27e1c98948b6} {Suffrage:Voter ID:server-c982249b-8fe4-c5b8-b954-c74b18753511 Address:c982249b-8fe4-c5b8-b954-c74b18753511}]" testing.go:149: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.519Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1c7c37b4-200b-3db9-5c3a-b14689d5edba Address:1c7c37b4-200b-3db9-5c3a-b14689d5edba} {Suffrage:Voter ID:server-3920bd6f-c487-705e-b37a-27e1c98948b6 Address:3920bd6f-c487-705e-b37a-27e1c98948b6} {Suffrage:Voter ID:server-c982249b-8fe4-c5b8-b954-c74b18753511 Address:c982249b-8fe4-c5b8-b954-c74b18753511}]" testLoggerAdapter verbose: server-3920bd6f-c487-705e-b37a-27e1c98948b6: 2024-01-09T05:10:03.519Z [INFO] server-3920bd6f-c487-705e-b37a-27e1c98948b6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1c7c37b4-200b-3db9-5c3a-b14689d5edba Address:1c7c37b4-200b-3db9-5c3a-b14689d5edba} {Suffrage:Voter ID:server-3920bd6f-c487-705e-b37a-27e1c98948b6 Address:3920bd6f-c487-705e-b37a-27e1c98948b6} {Suffrage:Voter ID:server-c982249b-8fe4-c5b8-b954-c74b18753511 Address:c982249b-8fe4-c5b8-b954-c74b18753511}]" testing.go:149: server-3920bd6f-c487-705e-b37a-27e1c98948b6: 2024-01-09T05:10:03.519Z [INFO] server-3920bd6f-c487-705e-b37a-27e1c98948b6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1c7c37b4-200b-3db9-5c3a-b14689d5edba Address:1c7c37b4-200b-3db9-5c3a-b14689d5edba} {Suffrage:Voter ID:server-3920bd6f-c487-705e-b37a-27e1c98948b6 Address:3920bd6f-c487-705e-b37a-27e1c98948b6} {Suffrage:Voter ID:server-c982249b-8fe4-c5b8-b954-c74b18753511 Address:c982249b-8fe4-c5b8-b954-c74b18753511}]" testLoggerAdapter verbose: server-c982249b-8fe4-c5b8-b954-c74b18753511: 2024-01-09T05:10:03.519Z [INFO] server-c982249b-8fe4-c5b8-b954-c74b18753511: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1c7c37b4-200b-3db9-5c3a-b14689d5edba Address:1c7c37b4-200b-3db9-5c3a-b14689d5edba} {Suffrage:Voter ID:server-3920bd6f-c487-705e-b37a-27e1c98948b6 Address:3920bd6f-c487-705e-b37a-27e1c98948b6} {Suffrage:Voter ID:server-c982249b-8fe4-c5b8-b954-c74b18753511 Address:c982249b-8fe4-c5b8-b954-c74b18753511}]" testing.go:149: server-c982249b-8fe4-c5b8-b954-c74b18753511: 2024-01-09T05:10:03.519Z [INFO] server-c982249b-8fe4-c5b8-b954-c74b18753511: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1c7c37b4-200b-3db9-5c3a-b14689d5edba Address:1c7c37b4-200b-3db9-5c3a-b14689d5edba} {Suffrage:Voter ID:server-3920bd6f-c487-705e-b37a-27e1c98948b6 Address:3920bd6f-c487-705e-b37a-27e1c98948b6} {Suffrage:Voter ID:server-c982249b-8fe4-c5b8-b954-c74b18753511 Address:c982249b-8fe4-c5b8-b954-c74b18753511}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:03.519Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:10:03.519Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.519Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: entering follower state: follower="Node at 1c7c37b4-200b-3db9-5c3a-b14689d5edba [Follower]" leader= testing.go:149: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.519Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: entering follower state: follower="Node at 1c7c37b4-200b-3db9-5c3a-b14689d5edba [Follower]" leader= testLoggerAdapter verbose: server-3920bd6f-c487-705e-b37a-27e1c98948b6: 2024-01-09T05:10:03.519Z [INFO] server-3920bd6f-c487-705e-b37a-27e1c98948b6: entering follower state: follower="Node at 3920bd6f-c487-705e-b37a-27e1c98948b6 [Follower]" leader= testing.go:149: server-3920bd6f-c487-705e-b37a-27e1c98948b6: 2024-01-09T05:10:03.519Z [INFO] server-3920bd6f-c487-705e-b37a-27e1c98948b6: entering follower state: follower="Node at 3920bd6f-c487-705e-b37a-27e1c98948b6 [Follower]" leader= testLoggerAdapter verbose: server-c982249b-8fe4-c5b8-b954-c74b18753511: 2024-01-09T05:10:03.520Z [INFO] server-c982249b-8fe4-c5b8-b954-c74b18753511: entering follower state: follower="Node at c982249b-8fe4-c5b8-b954-c74b18753511 [Follower]" leader= testing.go:149: server-c982249b-8fe4-c5b8-b954-c74b18753511: 2024-01-09T05:10:03.520Z [INFO] server-c982249b-8fe4-c5b8-b954-c74b18753511: entering follower state: follower="Node at c982249b-8fe4-c5b8-b954-c74b18753511 [Follower]" leader= testLoggerAdapter verbose: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.571Z [WARN] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.571Z [WARN] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.571Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: entering candidate state: node="Node at 1c7c37b4-200b-3db9-5c3a-b14689d5edba [Candidate]" term=2 testing.go:149: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.571Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: entering candidate state: node="Node at 1c7c37b4-200b-3db9-5c3a-b14689d5edba [Candidate]" term=2 testLoggerAdapter verbose: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.571Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: election won: tally=2 testing.go:149: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.571Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: election won: tally=2 testLoggerAdapter verbose: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.571Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: entering leader state: leader="Node at 1c7c37b4-200b-3db9-5c3a-b14689d5edba [Leader]" testing.go:149: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.571Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: entering leader state: leader="Node at 1c7c37b4-200b-3db9-5c3a-b14689d5edba [Leader]" testLoggerAdapter verbose: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.571Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: added peer, starting replication: peer=server-3920bd6f-c487-705e-b37a-27e1c98948b6 testing.go:149: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.571Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: added peer, starting replication: peer=server-3920bd6f-c487-705e-b37a-27e1c98948b6 testLoggerAdapter verbose: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.571Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: added peer, starting replication: peer=server-c982249b-8fe4-c5b8-b954-c74b18753511 testing.go:149: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.571Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: added peer, starting replication: peer=server-c982249b-8fe4-c5b8-b954-c74b18753511 testLoggerAdapter verbose: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.571Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: pipelining replication: peer="{Voter server-3920bd6f-c487-705e-b37a-27e1c98948b6 3920bd6f-c487-705e-b37a-27e1c98948b6}" testing.go:149: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.571Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: pipelining replication: peer="{Voter server-3920bd6f-c487-705e-b37a-27e1c98948b6 3920bd6f-c487-705e-b37a-27e1c98948b6}" testLoggerAdapter verbose: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.571Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: pipelining replication: peer="{Voter server-c982249b-8fe4-c5b8-b954-c74b18753511 c982249b-8fe4-c5b8-b954-c74b18753511}" testing.go:149: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.571Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: pipelining replication: peer="{Voter server-c982249b-8fe4-c5b8-b954-c74b18753511 c982249b-8fe4-c5b8-b954-c74b18753511}" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:03.677Z [INFO] stable state for Leader reached at 2024-01-09 05:10:03.571655419 +0000 UTC m=+14.824706500 (1 nodes), 51.773319ms from start of poll, 52.327196ms from cluster start. Timeout at 2024-01-09 05:10:03.67708264 +0000 UTC m=+14.930133622, 105.427122ms after stability testing.go:149: cluster: 2024-01-09T05:10:03.677Z [INFO] stable state for Leader reached at 2024-01-09 05:10:03.571655419 +0000 UTC m=+14.824706500 (1 nodes), 51.773319ms from start of poll, 52.327196ms from cluster start. Timeout at 2024-01-09 05:10:03.67708264 +0000 UTC m=+14.930133622, 105.427122ms after stability testLoggerAdapter verbose: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.677Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: aborting pipeline replication: peer="{Voter server-c982249b-8fe4-c5b8-b954-c74b18753511 c982249b-8fe4-c5b8-b954-c74b18753511}" testing.go:149: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.677Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: aborting pipeline replication: peer="{Voter server-c982249b-8fe4-c5b8-b954-c74b18753511 c982249b-8fe4-c5b8-b954-c74b18753511}" testLoggerAdapter verbose: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.677Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: aborting pipeline replication: peer="{Voter server-3920bd6f-c487-705e-b37a-27e1c98948b6 3920bd6f-c487-705e-b37a-27e1c98948b6}" testing.go:149: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.677Z [INFO] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: aborting pipeline replication: peer="{Voter server-3920bd6f-c487-705e-b37a-27e1c98948b6 3920bd6f-c487-705e-b37a-27e1c98948b6}" testLoggerAdapter verbose: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.727Z [ERROR] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: failed to heartbeat to: peer=3920bd6f-c487-705e-b37a-27e1c98948b6 error="command timed out" testing.go:149: server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: 2024-01-09T05:10:03.727Z [ERROR] server-1c7c37b4-200b-3db9-5c3a-b14689d5edba: failed to heartbeat to: peer=3920bd6f-c487-705e-b37a-27e1c98948b6 error="command timed out" --- PASS: TestRaft_VerifyLeader (0.21s) === RUN TestRaft_VerifyLeader_Single testLoggerAdapter verbose: server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: 2024-01-09T05:10:03.728Z [INFO] server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3e68225b-b745-0c7c-2387-ad9ed2c891b1 Address:3e68225b-b745-0c7c-2387-ad9ed2c891b1}]" testing.go:149: server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: 2024-01-09T05:10:03.728Z [INFO] server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3e68225b-b745-0c7c-2387-ad9ed2c891b1 Address:3e68225b-b745-0c7c-2387-ad9ed2c891b1}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:03.728Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:10:03.728Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: 2024-01-09T05:10:03.728Z [INFO] server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: entering follower state: follower="Node at 3e68225b-b745-0c7c-2387-ad9ed2c891b1 [Follower]" leader= testing.go:149: server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: 2024-01-09T05:10:03.728Z [INFO] server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: entering follower state: follower="Node at 3e68225b-b745-0c7c-2387-ad9ed2c891b1 [Follower]" leader= testLoggerAdapter verbose: server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: 2024-01-09T05:10:03.830Z [WARN] server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: 2024-01-09T05:10:03.830Z [WARN] server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: 2024-01-09T05:10:03.831Z [INFO] server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: entering candidate state: node="Node at 3e68225b-b745-0c7c-2387-ad9ed2c891b1 [Candidate]" term=2 testing.go:149: server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: 2024-01-09T05:10:03.831Z [INFO] server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: entering candidate state: node="Node at 3e68225b-b745-0c7c-2387-ad9ed2c891b1 [Candidate]" term=2 testLoggerAdapter verbose: server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: 2024-01-09T05:10:03.831Z [INFO] server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: election won: tally=1 testing.go:149: server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: 2024-01-09T05:10:03.831Z [INFO] server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: election won: tally=1 testLoggerAdapter verbose: server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: 2024-01-09T05:10:03.831Z [INFO] server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: entering leader state: leader="Node at 3e68225b-b745-0c7c-2387-ad9ed2c891b1 [Leader]" testing.go:149: server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: 2024-01-09T05:10:03.831Z [INFO] server-3e68225b-b745-0c7c-2387-ad9ed2c891b1: entering leader state: leader="Node at 3e68225b-b745-0c7c-2387-ad9ed2c891b1 [Leader]" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:03.936Z [INFO] stable state for Leader reached at 2024-01-09 05:10:03.831268687 +0000 UTC m=+15.084319711 (1 nodes), 102.548012ms from start of poll, 102.755078ms from cluster start. Timeout at 2024-01-09 05:10:03.936411313 +0000 UTC m=+15.189462630, 105.142919ms after stability testing.go:149: cluster: 2024-01-09T05:10:03.936Z [INFO] stable state for Leader reached at 2024-01-09 05:10:03.831268687 +0000 UTC m=+15.084319711 (1 nodes), 102.548012ms from start of poll, 102.755078ms from cluster start. Timeout at 2024-01-09 05:10:03.936411313 +0000 UTC m=+15.189462630, 105.142919ms after stability --- PASS: TestRaft_VerifyLeader_Single (0.21s) === RUN TestRaft_VerifyLeader_Fail testLoggerAdapter verbose: server-1e63e209-64c4-2b05-52c3-18126e9220e0: 2024-01-09T05:10:03.937Z [INFO] server-1e63e209-64c4-2b05-52c3-18126e9220e0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e63e209-64c4-2b05-52c3-18126e9220e0 Address:1e63e209-64c4-2b05-52c3-18126e9220e0} {Suffrage:Voter ID:server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd Address:ed1a0628-4a30-7e8e-ce29-f1b582d055fd}]" testing.go:149: server-1e63e209-64c4-2b05-52c3-18126e9220e0: 2024-01-09T05:10:03.937Z [INFO] server-1e63e209-64c4-2b05-52c3-18126e9220e0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e63e209-64c4-2b05-52c3-18126e9220e0 Address:1e63e209-64c4-2b05-52c3-18126e9220e0} {Suffrage:Voter ID:server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd Address:ed1a0628-4a30-7e8e-ce29-f1b582d055fd}]" testLoggerAdapter verbose: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:03.937Z [INFO] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e63e209-64c4-2b05-52c3-18126e9220e0 Address:1e63e209-64c4-2b05-52c3-18126e9220e0} {Suffrage:Voter ID:server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd Address:ed1a0628-4a30-7e8e-ce29-f1b582d055fd}]" testing.go:149: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:03.937Z [INFO] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e63e209-64c4-2b05-52c3-18126e9220e0 Address:1e63e209-64c4-2b05-52c3-18126e9220e0} {Suffrage:Voter ID:server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd Address:ed1a0628-4a30-7e8e-ce29-f1b582d055fd}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:03.937Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:10:03.937Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-1e63e209-64c4-2b05-52c3-18126e9220e0: 2024-01-09T05:10:03.937Z [INFO] server-1e63e209-64c4-2b05-52c3-18126e9220e0: entering follower state: follower="Node at 1e63e209-64c4-2b05-52c3-18126e9220e0 [Follower]" leader= testing.go:149: server-1e63e209-64c4-2b05-52c3-18126e9220e0: 2024-01-09T05:10:03.937Z [INFO] server-1e63e209-64c4-2b05-52c3-18126e9220e0: entering follower state: follower="Node at 1e63e209-64c4-2b05-52c3-18126e9220e0 [Follower]" leader= testLoggerAdapter verbose: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:03.937Z [INFO] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: entering follower state: follower="Node at ed1a0628-4a30-7e8e-ce29-f1b582d055fd [Follower]" leader= testing.go:149: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:03.937Z [INFO] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: entering follower state: follower="Node at ed1a0628-4a30-7e8e-ce29-f1b582d055fd [Follower]" leader= testLoggerAdapter verbose: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:04.006Z [WARN] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:04.006Z [WARN] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:04.006Z [INFO] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: entering candidate state: node="Node at ed1a0628-4a30-7e8e-ce29-f1b582d055fd [Candidate]" term=2 testing.go:149: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:04.006Z [INFO] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: entering candidate state: node="Node at ed1a0628-4a30-7e8e-ce29-f1b582d055fd [Candidate]" term=2 testLoggerAdapter verbose: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:04.006Z [INFO] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: election won: tally=2 testing.go:149: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:04.006Z [INFO] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: election won: tally=2 testLoggerAdapter verbose: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:04.006Z [INFO] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: entering leader state: leader="Node at ed1a0628-4a30-7e8e-ce29-f1b582d055fd [Leader]" testing.go:149: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:04.006Z [INFO] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: entering leader state: leader="Node at ed1a0628-4a30-7e8e-ce29-f1b582d055fd [Leader]" testLoggerAdapter verbose: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:04.006Z [INFO] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: added peer, starting replication: peer=server-1e63e209-64c4-2b05-52c3-18126e9220e0 testing.go:149: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:04.006Z [INFO] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: added peer, starting replication: peer=server-1e63e209-64c4-2b05-52c3-18126e9220e0 testLoggerAdapter verbose: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:04.006Z [INFO] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: pipelining replication: peer="{Voter server-1e63e209-64c4-2b05-52c3-18126e9220e0 1e63e209-64c4-2b05-52c3-18126e9220e0}" testing.go:149: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:04.006Z [INFO] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: pipelining replication: peer="{Voter server-1e63e209-64c4-2b05-52c3-18126e9220e0 1e63e209-64c4-2b05-52c3-18126e9220e0}" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:04.111Z [INFO] stable state for Leader reached at 2024-01-09 05:10:04.006601085 +0000 UTC m=+15.259652265 (1 nodes), 68.881808ms from start of poll, 69.230638ms from cluster start. Timeout at 2024-01-09 05:10:04.111632579 +0000 UTC m=+15.364683629, 105.031364ms after stability testing.go:149: cluster: 2024-01-09T05:10:04.111Z [INFO] stable state for Leader reached at 2024-01-09 05:10:04.006601085 +0000 UTC m=+15.259652265 (1 nodes), 68.881808ms from start of poll, 69.230638ms from cluster start. Timeout at 2024-01-09 05:10:04.111632579 +0000 UTC m=+15.364683629, 105.031364ms after stability testLoggerAdapter verbose: cluster: 2024-01-09T05:10:04.111Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-01-09T05:10:04.111Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2024-01-09T05:10:04.217Z [INFO] stable state for Follower reached at 2024-01-09 05:10:04.111827989 +0000 UTC m=+15.364878993 (1 nodes), 2.149µs from start of poll, 174.457366ms from cluster start. Timeout at 2024-01-09 05:10:04.217572196 +0000 UTC m=+15.470623402, 105.744409ms after stability testing.go:149: cluster: 2024-01-09T05:10:04.217Z [INFO] stable state for Follower reached at 2024-01-09 05:10:04.111827989 +0000 UTC m=+15.364878993 (1 nodes), 2.149µs from start of poll, 174.457366ms from cluster start. Timeout at 2024-01-09 05:10:04.217572196 +0000 UTC m=+15.470623402, 105.744409ms after stability testLoggerAdapter verbose: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:04.217Z [WARN] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: new leader elected, stepping down testing.go:149: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:04.217Z [WARN] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: new leader elected, stepping down testLoggerAdapter verbose: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:04.217Z [INFO] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: entering follower state: follower="Node at ed1a0628-4a30-7e8e-ce29-f1b582d055fd [Follower]" leader= testing.go:149: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:04.217Z [INFO] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: entering follower state: follower="Node at ed1a0628-4a30-7e8e-ce29-f1b582d055fd [Follower]" leader= testLoggerAdapter verbose: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:04.217Z [INFO] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: aborting pipeline replication: peer="{Voter server-1e63e209-64c4-2b05-52c3-18126e9220e0 1e63e209-64c4-2b05-52c3-18126e9220e0}" testing.go:149: server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: 2024-01-09T05:10:04.217Z [INFO] server-ed1a0628-4a30-7e8e-ce29-f1b582d055fd: aborting pipeline replication: peer="{Voter server-1e63e209-64c4-2b05-52c3-18126e9220e0 1e63e209-64c4-2b05-52c3-18126e9220e0}" --- PASS: TestRaft_VerifyLeader_Fail (0.28s) === RUN TestRaft_VerifyLeader_PartialConnect testLoggerAdapter verbose: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.219Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-261bdfd8-57b8-4b39-9ec0-85027b02a719 Address:261bdfd8-57b8-4b39-9ec0-85027b02a719} {Suffrage:Voter ID:server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba Address:dd4c5831-2bac-9bc1-3abf-300dec9f0eba} {Suffrage:Voter ID:server-6d20a6da-abe8-43f1-271c-64588c3397d2 Address:6d20a6da-abe8-43f1-271c-64588c3397d2}]" testing.go:149: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.219Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-261bdfd8-57b8-4b39-9ec0-85027b02a719 Address:261bdfd8-57b8-4b39-9ec0-85027b02a719} {Suffrage:Voter ID:server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba Address:dd4c5831-2bac-9bc1-3abf-300dec9f0eba} {Suffrage:Voter ID:server-6d20a6da-abe8-43f1-271c-64588c3397d2 Address:6d20a6da-abe8-43f1-271c-64588c3397d2}]" testLoggerAdapter verbose: server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba: 2024-01-09T05:10:04.219Z [INFO] server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-261bdfd8-57b8-4b39-9ec0-85027b02a719 Address:261bdfd8-57b8-4b39-9ec0-85027b02a719} {Suffrage:Voter ID:server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba Address:dd4c5831-2bac-9bc1-3abf-300dec9f0eba} {Suffrage:Voter ID:server-6d20a6da-abe8-43f1-271c-64588c3397d2 Address:6d20a6da-abe8-43f1-271c-64588c3397d2}]" testing.go:149: server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba: 2024-01-09T05:10:04.219Z [INFO] server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-261bdfd8-57b8-4b39-9ec0-85027b02a719 Address:261bdfd8-57b8-4b39-9ec0-85027b02a719} {Suffrage:Voter ID:server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba Address:dd4c5831-2bac-9bc1-3abf-300dec9f0eba} {Suffrage:Voter ID:server-6d20a6da-abe8-43f1-271c-64588c3397d2 Address:6d20a6da-abe8-43f1-271c-64588c3397d2}]" testLoggerAdapter verbose: server-6d20a6da-abe8-43f1-271c-64588c3397d2: 2024-01-09T05:10:04.219Z [INFO] server-6d20a6da-abe8-43f1-271c-64588c3397d2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-261bdfd8-57b8-4b39-9ec0-85027b02a719 Address:261bdfd8-57b8-4b39-9ec0-85027b02a719} {Suffrage:Voter ID:server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba Address:dd4c5831-2bac-9bc1-3abf-300dec9f0eba} {Suffrage:Voter ID:server-6d20a6da-abe8-43f1-271c-64588c3397d2 Address:6d20a6da-abe8-43f1-271c-64588c3397d2}]" testing.go:149: server-6d20a6da-abe8-43f1-271c-64588c3397d2: 2024-01-09T05:10:04.219Z [INFO] server-6d20a6da-abe8-43f1-271c-64588c3397d2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-261bdfd8-57b8-4b39-9ec0-85027b02a719 Address:261bdfd8-57b8-4b39-9ec0-85027b02a719} {Suffrage:Voter ID:server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba Address:dd4c5831-2bac-9bc1-3abf-300dec9f0eba} {Suffrage:Voter ID:server-6d20a6da-abe8-43f1-271c-64588c3397d2 Address:6d20a6da-abe8-43f1-271c-64588c3397d2}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:04.219Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:10:04.219Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.219Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: entering follower state: follower="Node at 261bdfd8-57b8-4b39-9ec0-85027b02a719 [Follower]" leader= testing.go:149: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.219Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: entering follower state: follower="Node at 261bdfd8-57b8-4b39-9ec0-85027b02a719 [Follower]" leader= testLoggerAdapter verbose: server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba: 2024-01-09T05:10:04.220Z [INFO] server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba: entering follower state: follower="Node at dd4c5831-2bac-9bc1-3abf-300dec9f0eba [Follower]" leader= testing.go:149: server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba: 2024-01-09T05:10:04.220Z [INFO] server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba: entering follower state: follower="Node at dd4c5831-2bac-9bc1-3abf-300dec9f0eba [Follower]" leader= testLoggerAdapter verbose: server-6d20a6da-abe8-43f1-271c-64588c3397d2: 2024-01-09T05:10:04.220Z [INFO] server-6d20a6da-abe8-43f1-271c-64588c3397d2: entering follower state: follower="Node at 6d20a6da-abe8-43f1-271c-64588c3397d2 [Follower]" leader= testing.go:149: server-6d20a6da-abe8-43f1-271c-64588c3397d2: 2024-01-09T05:10:04.220Z [INFO] server-6d20a6da-abe8-43f1-271c-64588c3397d2: entering follower state: follower="Node at 6d20a6da-abe8-43f1-271c-64588c3397d2 [Follower]" leader= testLoggerAdapter verbose: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.292Z [WARN] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.292Z [WARN] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.292Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: entering candidate state: node="Node at 261bdfd8-57b8-4b39-9ec0-85027b02a719 [Candidate]" term=2 testing.go:149: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.292Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: entering candidate state: node="Node at 261bdfd8-57b8-4b39-9ec0-85027b02a719 [Candidate]" term=2 testLoggerAdapter verbose: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.292Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: election won: tally=2 testing.go:149: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.292Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: election won: tally=2 testLoggerAdapter verbose: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.292Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: entering leader state: leader="Node at 261bdfd8-57b8-4b39-9ec0-85027b02a719 [Leader]" testing.go:149: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.292Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: entering leader state: leader="Node at 261bdfd8-57b8-4b39-9ec0-85027b02a719 [Leader]" testLoggerAdapter verbose: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.292Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: added peer, starting replication: peer=server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba testing.go:149: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.292Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: added peer, starting replication: peer=server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba testLoggerAdapter verbose: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.292Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: added peer, starting replication: peer=server-6d20a6da-abe8-43f1-271c-64588c3397d2 testing.go:149: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.292Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: added peer, starting replication: peer=server-6d20a6da-abe8-43f1-271c-64588c3397d2 testLoggerAdapter verbose: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.292Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: pipelining replication: peer="{Voter server-6d20a6da-abe8-43f1-271c-64588c3397d2 6d20a6da-abe8-43f1-271c-64588c3397d2}" testing.go:149: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.292Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: pipelining replication: peer="{Voter server-6d20a6da-abe8-43f1-271c-64588c3397d2 6d20a6da-abe8-43f1-271c-64588c3397d2}" testLoggerAdapter verbose: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.292Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: pipelining replication: peer="{Voter server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba dd4c5831-2bac-9bc1-3abf-300dec9f0eba}" testing.go:149: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.292Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: pipelining replication: peer="{Voter server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba dd4c5831-2bac-9bc1-3abf-300dec9f0eba}" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:04.398Z [INFO] stable state for Leader reached at 2024-01-09 05:10:04.292872676 +0000 UTC m=+15.545923742 (1 nodes), 72.910326ms from start of poll, 73.459003ms from cluster start. Timeout at 2024-01-09 05:10:04.398089643 +0000 UTC m=+15.651148361, 105.224619ms after stability testing.go:149: cluster: 2024-01-09T05:10:04.398Z [INFO] stable state for Leader reached at 2024-01-09 05:10:04.292872676 +0000 UTC m=+15.545923742 (1 nodes), 72.910326ms from start of poll, 73.459003ms from cluster start. Timeout at 2024-01-09 05:10:04.398089643 +0000 UTC m=+15.651148361, 105.224619ms after stability testLoggerAdapter verbose: cluster: 2024-01-09T05:10:04.403Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-01-09T05:10:04.403Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2024-01-09T05:10:04.509Z [INFO] stable state for Follower reached at 2024-01-09 05:10:04.40409861 +0000 UTC m=+15.657149745 (2 nodes), 3.019µs from start of poll, 184.685006ms from cluster start. Timeout at 2024-01-09 05:10:04.509273419 +0000 UTC m=+15.762324500, 105.174755ms after stability testing.go:149: cluster: 2024-01-09T05:10:04.509Z [INFO] stable state for Follower reached at 2024-01-09 05:10:04.40409861 +0000 UTC m=+15.657149745 (2 nodes), 3.019µs from start of poll, 184.685006ms from cluster start. Timeout at 2024-01-09 05:10:04.509273419 +0000 UTC m=+15.762324500, 105.174755ms after stability raft_test.go:1647: [INFO] Disconnecting Node at dd4c5831-2bac-9bc1-3abf-300dec9f0eba [Follower] testLoggerAdapter verbose: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.509Z [ERROR] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: failed to heartbeat to: peer=dd4c5831-2bac-9bc1-3abf-300dec9f0eba error="failed to connect to peer: dd4c5831-2bac-9bc1-3abf-300dec9f0eba" testing.go:149: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.509Z [ERROR] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: failed to heartbeat to: peer=dd4c5831-2bac-9bc1-3abf-300dec9f0eba error="failed to connect to peer: dd4c5831-2bac-9bc1-3abf-300dec9f0eba" testLoggerAdapter verbose: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.509Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: aborting pipeline replication: peer="{Voter server-6d20a6da-abe8-43f1-271c-64588c3397d2 6d20a6da-abe8-43f1-271c-64588c3397d2}" testing.go:149: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.509Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: aborting pipeline replication: peer="{Voter server-6d20a6da-abe8-43f1-271c-64588c3397d2 6d20a6da-abe8-43f1-271c-64588c3397d2}" testLoggerAdapter verbose: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.510Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: aborting pipeline replication: peer="{Voter server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba dd4c5831-2bac-9bc1-3abf-300dec9f0eba}" testing.go:149: server-261bdfd8-57b8-4b39-9ec0-85027b02a719: 2024-01-09T05:10:04.510Z [INFO] server-261bdfd8-57b8-4b39-9ec0-85027b02a719: aborting pipeline replication: peer="{Voter server-dd4c5831-2bac-9bc1-3abf-300dec9f0eba dd4c5831-2bac-9bc1-3abf-300dec9f0eba}" --- PASS: TestRaft_VerifyLeader_PartialConnect (0.29s) === RUN TestRaft_StartAsLeader testLoggerAdapter verbose: server-2b62827e-6ae8-577c-5095-5baed64efa70: 2024-01-09T05:10:04.512Z [INFO] server-2b62827e-6ae8-577c-5095-5baed64efa70: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2b62827e-6ae8-577c-5095-5baed64efa70 Address:2b62827e-6ae8-577c-5095-5baed64efa70}]" testing.go:149: server-2b62827e-6ae8-577c-5095-5baed64efa70: 2024-01-09T05:10:04.512Z [INFO] server-2b62827e-6ae8-577c-5095-5baed64efa70: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2b62827e-6ae8-577c-5095-5baed64efa70 Address:2b62827e-6ae8-577c-5095-5baed64efa70}]" testLoggerAdapter verbose: server-2b62827e-6ae8-577c-5095-5baed64efa70: 2024-01-09T05:10:04.512Z [INFO] server-2b62827e-6ae8-577c-5095-5baed64efa70: entering leader state: leader="Node at 2b62827e-6ae8-577c-5095-5baed64efa70 [Leader]" testing.go:149: server-2b62827e-6ae8-577c-5095-5baed64efa70: 2024-01-09T05:10:04.512Z [INFO] server-2b62827e-6ae8-577c-5095-5baed64efa70: entering leader state: leader="Node at 2b62827e-6ae8-577c-5095-5baed64efa70 [Leader]" --- PASS: TestRaft_StartAsLeader (0.00s) === RUN TestRaft_NotifyCh testLoggerAdapter verbose: server-da418d31-c1b2-2c51-7e6e-31147f72440d: 2024-01-09T05:10:04.514Z [INFO] server-da418d31-c1b2-2c51-7e6e-31147f72440d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-da418d31-c1b2-2c51-7e6e-31147f72440d Address:da418d31-c1b2-2c51-7e6e-31147f72440d}]" testing.go:149: server-da418d31-c1b2-2c51-7e6e-31147f72440d: 2024-01-09T05:10:04.514Z [INFO] server-da418d31-c1b2-2c51-7e6e-31147f72440d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-da418d31-c1b2-2c51-7e6e-31147f72440d Address:da418d31-c1b2-2c51-7e6e-31147f72440d}]" testLoggerAdapter verbose: server-da418d31-c1b2-2c51-7e6e-31147f72440d: 2024-01-09T05:10:04.514Z [INFO] server-da418d31-c1b2-2c51-7e6e-31147f72440d: entering follower state: follower="Node at da418d31-c1b2-2c51-7e6e-31147f72440d [Follower]" leader= testing.go:149: server-da418d31-c1b2-2c51-7e6e-31147f72440d: 2024-01-09T05:10:04.514Z [INFO] server-da418d31-c1b2-2c51-7e6e-31147f72440d: entering follower state: follower="Node at da418d31-c1b2-2c51-7e6e-31147f72440d [Follower]" leader= testLoggerAdapter verbose: server-da418d31-c1b2-2c51-7e6e-31147f72440d: 2024-01-09T05:10:04.578Z [WARN] server-da418d31-c1b2-2c51-7e6e-31147f72440d: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-da418d31-c1b2-2c51-7e6e-31147f72440d: 2024-01-09T05:10:04.578Z [WARN] server-da418d31-c1b2-2c51-7e6e-31147f72440d: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-da418d31-c1b2-2c51-7e6e-31147f72440d: 2024-01-09T05:10:04.578Z [INFO] server-da418d31-c1b2-2c51-7e6e-31147f72440d: entering candidate state: node="Node at da418d31-c1b2-2c51-7e6e-31147f72440d [Candidate]" term=2 testing.go:149: server-da418d31-c1b2-2c51-7e6e-31147f72440d: 2024-01-09T05:10:04.578Z [INFO] server-da418d31-c1b2-2c51-7e6e-31147f72440d: entering candidate state: node="Node at da418d31-c1b2-2c51-7e6e-31147f72440d [Candidate]" term=2 testLoggerAdapter verbose: server-da418d31-c1b2-2c51-7e6e-31147f72440d: 2024-01-09T05:10:04.578Z [INFO] server-da418d31-c1b2-2c51-7e6e-31147f72440d: election won: tally=1 testing.go:149: server-da418d31-c1b2-2c51-7e6e-31147f72440d: 2024-01-09T05:10:04.578Z [INFO] server-da418d31-c1b2-2c51-7e6e-31147f72440d: election won: tally=1 testLoggerAdapter verbose: server-da418d31-c1b2-2c51-7e6e-31147f72440d: 2024-01-09T05:10:04.578Z [INFO] server-da418d31-c1b2-2c51-7e6e-31147f72440d: entering leader state: leader="Node at da418d31-c1b2-2c51-7e6e-31147f72440d [Leader]" testing.go:149: server-da418d31-c1b2-2c51-7e6e-31147f72440d: 2024-01-09T05:10:04.578Z [INFO] server-da418d31-c1b2-2c51-7e6e-31147f72440d: entering leader state: leader="Node at da418d31-c1b2-2c51-7e6e-31147f72440d [Leader]" --- PASS: TestRaft_NotifyCh (0.07s) === RUN TestRaft_Voting testLoggerAdapter verbose: server-3620caaf-fe14-274d-b295-300856ae8562: 2024-01-09T05:10:04.580Z [INFO] server-3620caaf-fe14-274d-b295-300856ae8562: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3620caaf-fe14-274d-b295-300856ae8562 Address:3620caaf-fe14-274d-b295-300856ae8562} {Suffrage:Voter ID:server-40772aff-3515-f043-13c1-d269661cbc18 Address:40772aff-3515-f043-13c1-d269661cbc18} {Suffrage:Voter ID:server-537d52a1-1d79-f546-03cd-2e13d8290052 Address:537d52a1-1d79-f546-03cd-2e13d8290052}]" testing.go:149: server-3620caaf-fe14-274d-b295-300856ae8562: 2024-01-09T05:10:04.580Z [INFO] server-3620caaf-fe14-274d-b295-300856ae8562: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3620caaf-fe14-274d-b295-300856ae8562 Address:3620caaf-fe14-274d-b295-300856ae8562} {Suffrage:Voter ID:server-40772aff-3515-f043-13c1-d269661cbc18 Address:40772aff-3515-f043-13c1-d269661cbc18} {Suffrage:Voter ID:server-537d52a1-1d79-f546-03cd-2e13d8290052 Address:537d52a1-1d79-f546-03cd-2e13d8290052}]" testLoggerAdapter verbose: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.580Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3620caaf-fe14-274d-b295-300856ae8562 Address:3620caaf-fe14-274d-b295-300856ae8562} {Suffrage:Voter ID:server-40772aff-3515-f043-13c1-d269661cbc18 Address:40772aff-3515-f043-13c1-d269661cbc18} {Suffrage:Voter ID:server-537d52a1-1d79-f546-03cd-2e13d8290052 Address:537d52a1-1d79-f546-03cd-2e13d8290052}]" testing.go:149: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.580Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3620caaf-fe14-274d-b295-300856ae8562 Address:3620caaf-fe14-274d-b295-300856ae8562} {Suffrage:Voter ID:server-40772aff-3515-f043-13c1-d269661cbc18 Address:40772aff-3515-f043-13c1-d269661cbc18} {Suffrage:Voter ID:server-537d52a1-1d79-f546-03cd-2e13d8290052 Address:537d52a1-1d79-f546-03cd-2e13d8290052}]" testLoggerAdapter verbose: server-537d52a1-1d79-f546-03cd-2e13d8290052: 2024-01-09T05:10:04.580Z [INFO] server-537d52a1-1d79-f546-03cd-2e13d8290052: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3620caaf-fe14-274d-b295-300856ae8562 Address:3620caaf-fe14-274d-b295-300856ae8562} {Suffrage:Voter ID:server-40772aff-3515-f043-13c1-d269661cbc18 Address:40772aff-3515-f043-13c1-d269661cbc18} {Suffrage:Voter ID:server-537d52a1-1d79-f546-03cd-2e13d8290052 Address:537d52a1-1d79-f546-03cd-2e13d8290052}]" testing.go:149: server-537d52a1-1d79-f546-03cd-2e13d8290052: 2024-01-09T05:10:04.580Z [INFO] server-537d52a1-1d79-f546-03cd-2e13d8290052: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3620caaf-fe14-274d-b295-300856ae8562 Address:3620caaf-fe14-274d-b295-300856ae8562} {Suffrage:Voter ID:server-40772aff-3515-f043-13c1-d269661cbc18 Address:40772aff-3515-f043-13c1-d269661cbc18} {Suffrage:Voter ID:server-537d52a1-1d79-f546-03cd-2e13d8290052 Address:537d52a1-1d79-f546-03cd-2e13d8290052}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:04.580Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-01-09T05:10:04.580Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-3620caaf-fe14-274d-b295-300856ae8562: 2024-01-09T05:10:04.580Z [INFO] server-3620caaf-fe14-274d-b295-300856ae8562: entering follower state: follower="Node at 3620caaf-fe14-274d-b295-300856ae8562 [Follower]" leader= testing.go:149: server-3620caaf-fe14-274d-b295-300856ae8562: 2024-01-09T05:10:04.580Z [INFO] server-3620caaf-fe14-274d-b295-300856ae8562: entering follower state: follower="Node at 3620caaf-fe14-274d-b295-300856ae8562 [Follower]" leader= testLoggerAdapter verbose: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.580Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: entering follower state: follower="Node at 40772aff-3515-f043-13c1-d269661cbc18 [Follower]" leader= testing.go:149: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.580Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: entering follower state: follower="Node at 40772aff-3515-f043-13c1-d269661cbc18 [Follower]" leader= testLoggerAdapter verbose: server-537d52a1-1d79-f546-03cd-2e13d8290052: 2024-01-09T05:10:04.580Z [INFO] server-537d52a1-1d79-f546-03cd-2e13d8290052: entering follower state: follower="Node at 537d52a1-1d79-f546-03cd-2e13d8290052 [Follower]" leader= testing.go:149: server-537d52a1-1d79-f546-03cd-2e13d8290052: 2024-01-09T05:10:04.580Z [INFO] server-537d52a1-1d79-f546-03cd-2e13d8290052: entering follower state: follower="Node at 537d52a1-1d79-f546-03cd-2e13d8290052 [Follower]" leader= testLoggerAdapter verbose: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.660Z [WARN] server-40772aff-3515-f043-13c1-d269661cbc18: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.660Z [WARN] server-40772aff-3515-f043-13c1-d269661cbc18: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.660Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: entering candidate state: node="Node at 40772aff-3515-f043-13c1-d269661cbc18 [Candidate]" term=2 testing.go:149: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.660Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: entering candidate state: node="Node at 40772aff-3515-f043-13c1-d269661cbc18 [Candidate]" term=2 testLoggerAdapter verbose: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.660Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: election won: tally=2 testing.go:149: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.660Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: election won: tally=2 testLoggerAdapter verbose: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.660Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: entering leader state: leader="Node at 40772aff-3515-f043-13c1-d269661cbc18 [Leader]" testing.go:149: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.660Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: entering leader state: leader="Node at 40772aff-3515-f043-13c1-d269661cbc18 [Leader]" testLoggerAdapter verbose: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.660Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: added peer, starting replication: peer=server-3620caaf-fe14-274d-b295-300856ae8562 testing.go:149: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.660Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: added peer, starting replication: peer=server-3620caaf-fe14-274d-b295-300856ae8562 testLoggerAdapter verbose: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.660Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: added peer, starting replication: peer=server-537d52a1-1d79-f546-03cd-2e13d8290052 testing.go:149: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.660Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: added peer, starting replication: peer=server-537d52a1-1d79-f546-03cd-2e13d8290052 testLoggerAdapter verbose: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.660Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: pipelining replication: peer="{Voter server-537d52a1-1d79-f546-03cd-2e13d8290052 537d52a1-1d79-f546-03cd-2e13d8290052}" testing.go:149: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.660Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: pipelining replication: peer="{Voter server-537d52a1-1d79-f546-03cd-2e13d8290052 537d52a1-1d79-f546-03cd-2e13d8290052}" testLoggerAdapter verbose: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.660Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: pipelining replication: peer="{Voter server-3620caaf-fe14-274d-b295-300856ae8562 3620caaf-fe14-274d-b295-300856ae8562}" testing.go:149: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.660Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: pipelining replication: peer="{Voter server-3620caaf-fe14-274d-b295-300856ae8562 3620caaf-fe14-274d-b295-300856ae8562}" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:04.766Z [INFO] stable state for Follower reached at 2024-01-09 05:10:04.660952146 +0000 UTC m=+15.914003288 (2 nodes), 80.371784ms from start of poll, 80.882219ms from cluster start. Timeout at 2024-01-09 05:10:04.766178865 +0000 UTC m=+16.019229914, 105.226626ms after stability testing.go:149: cluster: 2024-01-09T05:10:04.766Z [INFO] stable state for Follower reached at 2024-01-09 05:10:04.660952146 +0000 UTC m=+15.914003288 (2 nodes), 80.371784ms from start of poll, 80.882219ms from cluster start. Timeout at 2024-01-09 05:10:04.766178865 +0000 UTC m=+16.019229914, 105.226626ms after stability testLoggerAdapter verbose: cluster: 2024-01-09T05:10:04.766Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:10:04.766Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-01-09T05:10:04.872Z [INFO] stable state for Leader reached at 2024-01-09 05:10:04.76745132 +0000 UTC m=+16.020502450 (1 nodes), 3.18µs from start of poll, 187.381381ms from cluster start. Timeout at 2024-01-09 05:10:04.872564895 +0000 UTC m=+16.125616108, 105.113658ms after stability testing.go:149: cluster: 2024-01-09T05:10:04.872Z [INFO] stable state for Leader reached at 2024-01-09 05:10:04.76745132 +0000 UTC m=+16.020502450 (1 nodes), 3.18µs from start of poll, 187.381381ms from cluster start. Timeout at 2024-01-09 05:10:04.872564895 +0000 UTC m=+16.125616108, 105.113658ms after stability testLoggerAdapter verbose: server-537d52a1-1d79-f546-03cd-2e13d8290052: 2024-01-09T05:10:04.873Z [WARN] server-537d52a1-1d79-f546-03cd-2e13d8290052: rejecting vote request since we have a leader: from=3620caaf-fe14-274d-b295-300856ae8562 leader=40772aff-3515-f043-13c1-d269661cbc18 testing.go:149: server-537d52a1-1d79-f546-03cd-2e13d8290052: 2024-01-09T05:10:04.873Z [WARN] server-537d52a1-1d79-f546-03cd-2e13d8290052: rejecting vote request since we have a leader: from=3620caaf-fe14-274d-b295-300856ae8562 leader=40772aff-3515-f043-13c1-d269661cbc18 testLoggerAdapter verbose: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.873Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: aborting pipeline replication: peer="{Voter server-537d52a1-1d79-f546-03cd-2e13d8290052 537d52a1-1d79-f546-03cd-2e13d8290052}" testing.go:149: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.873Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: aborting pipeline replication: peer="{Voter server-537d52a1-1d79-f546-03cd-2e13d8290052 537d52a1-1d79-f546-03cd-2e13d8290052}" testLoggerAdapter verbose: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.873Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: aborting pipeline replication: peer="{Voter server-3620caaf-fe14-274d-b295-300856ae8562 3620caaf-fe14-274d-b295-300856ae8562}" testing.go:149: server-40772aff-3515-f043-13c1-d269661cbc18: 2024-01-09T05:10:04.873Z [INFO] server-40772aff-3515-f043-13c1-d269661cbc18: aborting pipeline replication: peer="{Voter server-3620caaf-fe14-274d-b295-300856ae8562 3620caaf-fe14-274d-b295-300856ae8562}" --- PASS: TestRaft_Voting (0.30s) === RUN TestRaft_ProtocolVersion_RejectRPC testLoggerAdapter verbose: server-8b6a6558-ed08-a050-331e-a0a39f8a19c6: 2024-01-09T05:10:04.877Z [INFO] server-8b6a6558-ed08-a050-331e-a0a39f8a19c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b6a6558-ed08-a050-331e-a0a39f8a19c6 Address:8b6a6558-ed08-a050-331e-a0a39f8a19c6} {Suffrage:Voter ID:server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac Address:5463e2ba-ed6b-6c8c-f662-21da3fb822ac} {Suffrage:Voter ID:server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b Address:e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b}]" testing.go:149: server-8b6a6558-ed08-a050-331e-a0a39f8a19c6: 2024-01-09T05:10:04.877Z [INFO] server-8b6a6558-ed08-a050-331e-a0a39f8a19c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b6a6558-ed08-a050-331e-a0a39f8a19c6 Address:8b6a6558-ed08-a050-331e-a0a39f8a19c6} {Suffrage:Voter ID:server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac Address:5463e2ba-ed6b-6c8c-f662-21da3fb822ac} {Suffrage:Voter ID:server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b Address:e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b}]" testLoggerAdapter verbose: server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac: 2024-01-09T05:10:04.878Z [INFO] server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b6a6558-ed08-a050-331e-a0a39f8a19c6 Address:8b6a6558-ed08-a050-331e-a0a39f8a19c6} {Suffrage:Voter ID:server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac Address:5463e2ba-ed6b-6c8c-f662-21da3fb822ac} {Suffrage:Voter ID:server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b Address:e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b}]" testing.go:149: server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac: 2024-01-09T05:10:04.878Z [INFO] server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b6a6558-ed08-a050-331e-a0a39f8a19c6 Address:8b6a6558-ed08-a050-331e-a0a39f8a19c6} {Suffrage:Voter ID:server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac Address:5463e2ba-ed6b-6c8c-f662-21da3fb822ac} {Suffrage:Voter ID:server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b Address:e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b}]" testLoggerAdapter verbose: server-8b6a6558-ed08-a050-331e-a0a39f8a19c6: 2024-01-09T05:10:04.878Z [INFO] server-8b6a6558-ed08-a050-331e-a0a39f8a19c6: entering follower state: follower="Node at 8b6a6558-ed08-a050-331e-a0a39f8a19c6 [Follower]" leader= testing.go:149: server-8b6a6558-ed08-a050-331e-a0a39f8a19c6: 2024-01-09T05:10:04.878Z [INFO] server-8b6a6558-ed08-a050-331e-a0a39f8a19c6: entering follower state: follower="Node at 8b6a6558-ed08-a050-331e-a0a39f8a19c6 [Follower]" leader= testLoggerAdapter verbose: server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac: 2024-01-09T05:10:04.878Z [INFO] server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac: entering follower state: follower="Node at 5463e2ba-ed6b-6c8c-f662-21da3fb822ac [Follower]" leader= testing.go:149: server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac: 2024-01-09T05:10:04.878Z [INFO] server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac: entering follower state: follower="Node at 5463e2ba-ed6b-6c8c-f662-21da3fb822ac [Follower]" leader= testLoggerAdapter verbose: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.879Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b6a6558-ed08-a050-331e-a0a39f8a19c6 Address:8b6a6558-ed08-a050-331e-a0a39f8a19c6} {Suffrage:Voter ID:server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac Address:5463e2ba-ed6b-6c8c-f662-21da3fb822ac} {Suffrage:Voter ID:server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b Address:e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b}]" testing.go:149: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.879Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b6a6558-ed08-a050-331e-a0a39f8a19c6 Address:8b6a6558-ed08-a050-331e-a0a39f8a19c6} {Suffrage:Voter ID:server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac Address:5463e2ba-ed6b-6c8c-f662-21da3fb822ac} {Suffrage:Voter ID:server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b Address:e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:04.879Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-01-09T05:10:04.879Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.879Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: entering follower state: follower="Node at e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b [Follower]" leader= testing.go:149: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.879Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: entering follower state: follower="Node at e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b [Follower]" leader= testLoggerAdapter verbose: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.931Z [WARN] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.931Z [WARN] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.932Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: entering candidate state: node="Node at e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b [Candidate]" term=2 testing.go:149: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.932Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: entering candidate state: node="Node at e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b [Candidate]" term=2 testLoggerAdapter verbose: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.932Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: election won: tally=2 testing.go:149: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.932Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: election won: tally=2 testLoggerAdapter verbose: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.932Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: entering leader state: leader="Node at e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b [Leader]" testing.go:149: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.932Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: entering leader state: leader="Node at e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b [Leader]" testLoggerAdapter verbose: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.932Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: added peer, starting replication: peer=server-8b6a6558-ed08-a050-331e-a0a39f8a19c6 testing.go:149: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.932Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: added peer, starting replication: peer=server-8b6a6558-ed08-a050-331e-a0a39f8a19c6 testLoggerAdapter verbose: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.932Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: added peer, starting replication: peer=server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac testing.go:149: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.932Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: added peer, starting replication: peer=server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac testLoggerAdapter verbose: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.932Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: pipelining replication: peer="{Voter server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac 5463e2ba-ed6b-6c8c-f662-21da3fb822ac}" testing.go:149: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.932Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: pipelining replication: peer="{Voter server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac 5463e2ba-ed6b-6c8c-f662-21da3fb822ac}" testLoggerAdapter verbose: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.932Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: pipelining replication: peer="{Voter server-8b6a6558-ed08-a050-331e-a0a39f8a19c6 8b6a6558-ed08-a050-331e-a0a39f8a19c6}" testing.go:149: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:04.932Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: pipelining replication: peer="{Voter server-8b6a6558-ed08-a050-331e-a0a39f8a19c6 8b6a6558-ed08-a050-331e-a0a39f8a19c6}" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:05.038Z [INFO] stable state for Follower reached at 2024-01-09 05:10:04.932925112 +0000 UTC m=+16.185976253 (2 nodes), 53.543674ms from start of poll, 55.847804ms from cluster start. Timeout at 2024-01-09 05:10:05.038041127 +0000 UTC m=+16.291092273, 105.11602ms after stability testing.go:149: cluster: 2024-01-09T05:10:05.038Z [INFO] stable state for Follower reached at 2024-01-09 05:10:04.932925112 +0000 UTC m=+16.185976253 (2 nodes), 53.543674ms from start of poll, 55.847804ms from cluster start. Timeout at 2024-01-09 05:10:05.038041127 +0000 UTC m=+16.291092273, 105.11602ms after stability testLoggerAdapter verbose: cluster: 2024-01-09T05:10:05.038Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:10:05.038Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-01-09T05:10:05.143Z [INFO] stable state for Leader reached at 2024-01-09 05:10:05.038190268 +0000 UTC m=+16.291241410 (1 nodes), 2.573µs from start of poll, 161.112961ms from cluster start. Timeout at 2024-01-09 05:10:05.143296571 +0000 UTC m=+16.396347663, 105.106253ms after stability testing.go:149: cluster: 2024-01-09T05:10:05.143Z [INFO] stable state for Leader reached at 2024-01-09 05:10:05.038190268 +0000 UTC m=+16.291241410 (1 nodes), 2.573µs from start of poll, 161.112961ms from cluster start. Timeout at 2024-01-09 05:10:05.143296571 +0000 UTC m=+16.396347663, 105.106253ms after stability testLoggerAdapter verbose: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:05.143Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: aborting pipeline replication: peer="{Voter server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac 5463e2ba-ed6b-6c8c-f662-21da3fb822ac}" testing.go:149: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:05.143Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: aborting pipeline replication: peer="{Voter server-5463e2ba-ed6b-6c8c-f662-21da3fb822ac 5463e2ba-ed6b-6c8c-f662-21da3fb822ac}" testLoggerAdapter verbose: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:05.143Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: aborting pipeline replication: peer="{Voter server-8b6a6558-ed08-a050-331e-a0a39f8a19c6 8b6a6558-ed08-a050-331e-a0a39f8a19c6}" testing.go:149: server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: 2024-01-09T05:10:05.143Z [INFO] server-e0a37819-e1fc-f82d-07a2-7b76ea7dbb8b: aborting pipeline replication: peer="{Voter server-8b6a6558-ed08-a050-331e-a0a39f8a19c6 8b6a6558-ed08-a050-331e-a0a39f8a19c6}" --- PASS: TestRaft_ProtocolVersion_RejectRPC (0.27s) === RUN TestRaft_ProtocolVersion_Upgrade_1_2 testLoggerAdapter verbose: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.151Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: initial configuration: index=1 servers="[{Suffrage:Voter ID:a0ea6c91-e115-e43e-e35e-3e180635831d Address:a0ea6c91-e115-e43e-e35e-3e180635831d} {Suffrage:Voter ID:a02787f6-97d3-11ae-ae8a-eff88e75b250 Address:a02787f6-97d3-11ae-ae8a-eff88e75b250}]" testing.go:149: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.151Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: initial configuration: index=1 servers="[{Suffrage:Voter ID:a0ea6c91-e115-e43e-e35e-3e180635831d Address:a0ea6c91-e115-e43e-e35e-3e180635831d} {Suffrage:Voter ID:a02787f6-97d3-11ae-ae8a-eff88e75b250 Address:a02787f6-97d3-11ae-ae8a-eff88e75b250}]" testLoggerAdapter verbose: a02787f6-97d3-11ae-ae8a-eff88e75b250: 2024-01-09T05:10:05.151Z [INFO] a02787f6-97d3-11ae-ae8a-eff88e75b250: initial configuration: index=1 servers="[{Suffrage:Voter ID:a0ea6c91-e115-e43e-e35e-3e180635831d Address:a0ea6c91-e115-e43e-e35e-3e180635831d} {Suffrage:Voter ID:a02787f6-97d3-11ae-ae8a-eff88e75b250 Address:a02787f6-97d3-11ae-ae8a-eff88e75b250}]" testing.go:149: a02787f6-97d3-11ae-ae8a-eff88e75b250: 2024-01-09T05:10:05.151Z [INFO] a02787f6-97d3-11ae-ae8a-eff88e75b250: initial configuration: index=1 servers="[{Suffrage:Voter ID:a0ea6c91-e115-e43e-e35e-3e180635831d Address:a0ea6c91-e115-e43e-e35e-3e180635831d} {Suffrage:Voter ID:a02787f6-97d3-11ae-ae8a-eff88e75b250 Address:a02787f6-97d3-11ae-ae8a-eff88e75b250}]" testLoggerAdapter verbose: aef09d44-1ee3-903b-a640-190442a2e065: 2024-01-09T05:10:05.152Z [INFO] aef09d44-1ee3-903b-a640-190442a2e065: initial configuration: index=0 servers=[] testing.go:149: aef09d44-1ee3-903b-a640-190442a2e065: 2024-01-09T05:10:05.152Z [INFO] aef09d44-1ee3-903b-a640-190442a2e065: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2024-01-09T05:10:05.152Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:10:05.152Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.152Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: entering follower state: follower="Node at a0ea6c91-e115-e43e-e35e-3e180635831d [Follower]" leader= testing.go:149: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.152Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: entering follower state: follower="Node at a0ea6c91-e115-e43e-e35e-3e180635831d [Follower]" leader= testLoggerAdapter verbose: a02787f6-97d3-11ae-ae8a-eff88e75b250: 2024-01-09T05:10:05.153Z [INFO] a02787f6-97d3-11ae-ae8a-eff88e75b250: entering follower state: follower="Node at a02787f6-97d3-11ae-ae8a-eff88e75b250 [Follower]" leader= testing.go:149: a02787f6-97d3-11ae-ae8a-eff88e75b250: 2024-01-09T05:10:05.153Z [INFO] a02787f6-97d3-11ae-ae8a-eff88e75b250: entering follower state: follower="Node at a02787f6-97d3-11ae-ae8a-eff88e75b250 [Follower]" leader= testLoggerAdapter verbose: aef09d44-1ee3-903b-a640-190442a2e065: 2024-01-09T05:10:05.153Z [INFO] aef09d44-1ee3-903b-a640-190442a2e065: entering follower state: follower="Node at aef09d44-1ee3-903b-a640-190442a2e065 [Follower]" leader= testing.go:149: aef09d44-1ee3-903b-a640-190442a2e065: 2024-01-09T05:10:05.153Z [INFO] aef09d44-1ee3-903b-a640-190442a2e065: entering follower state: follower="Node at aef09d44-1ee3-903b-a640-190442a2e065 [Follower]" leader= testLoggerAdapter verbose: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.224Z [WARN] a0ea6c91-e115-e43e-e35e-3e180635831d: heartbeat timeout reached, starting election: last-leader= testing.go:149: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.224Z [WARN] a0ea6c91-e115-e43e-e35e-3e180635831d: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.224Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: entering candidate state: node="Node at a0ea6c91-e115-e43e-e35e-3e180635831d [Candidate]" term=2 testing.go:149: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.224Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: entering candidate state: node="Node at a0ea6c91-e115-e43e-e35e-3e180635831d [Candidate]" term=2 testLoggerAdapter verbose: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.224Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: election won: tally=2 testing.go:149: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.224Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: election won: tally=2 testLoggerAdapter verbose: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.224Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: entering leader state: leader="Node at a0ea6c91-e115-e43e-e35e-3e180635831d [Leader]" testing.go:149: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.224Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: entering leader state: leader="Node at a0ea6c91-e115-e43e-e35e-3e180635831d [Leader]" testLoggerAdapter verbose: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.224Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: added peer, starting replication: peer=a02787f6-97d3-11ae-ae8a-eff88e75b250 testing.go:149: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.224Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: added peer, starting replication: peer=a02787f6-97d3-11ae-ae8a-eff88e75b250 testLoggerAdapter verbose: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.224Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: pipelining replication: peer="{Voter a02787f6-97d3-11ae-ae8a-eff88e75b250 a02787f6-97d3-11ae-ae8a-eff88e75b250}" testing.go:149: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.224Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: pipelining replication: peer="{Voter a02787f6-97d3-11ae-ae8a-eff88e75b250 a02787f6-97d3-11ae-ae8a-eff88e75b250}" testLoggerAdapter verbose: aef09d44-1ee3-903b-a640-190442a2e065: 2024-01-09T05:10:05.253Z [WARN] aef09d44-1ee3-903b-a640-190442a2e065: no known peers, aborting election testing.go:149: aef09d44-1ee3-903b-a640-190442a2e065: 2024-01-09T05:10:05.253Z [WARN] aef09d44-1ee3-903b-a640-190442a2e065: no known peers, aborting election testLoggerAdapter verbose: cluster: 2024-01-09T05:10:05.330Z [INFO] stable state for Leader reached at 2024-01-09 05:10:05.224727087 +0000 UTC m=+16.477778132 (1 nodes), 72.288579ms from start of poll, 73.157815ms from cluster start. Timeout at 2024-01-09 05:10:05.330416078 +0000 UTC m=+16.583467148, 105.689016ms after stability testing.go:149: cluster: 2024-01-09T05:10:05.330Z [INFO] stable state for Leader reached at 2024-01-09 05:10:05.224727087 +0000 UTC m=+16.477778132 (1 nodes), 72.288579ms from start of poll, 73.157815ms from cluster start. Timeout at 2024-01-09 05:10:05.330416078 +0000 UTC m=+16.583467148, 105.689016ms after stability testLoggerAdapter verbose: cluster: 2024-01-09T05:10:05.330Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:10:05.330Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-01-09T05:10:05.435Z [INFO] stable state for Leader reached at 2024-01-09 05:10:05.330613883 +0000 UTC m=+16.583664974 (1 nodes), 3.116µs from start of poll, 179.044657ms from cluster start. Timeout at 2024-01-09 05:10:05.435731646 +0000 UTC m=+16.688782753, 105.117779ms after stability testing.go:149: cluster: 2024-01-09T05:10:05.435Z [INFO] stable state for Leader reached at 2024-01-09 05:10:05.330613883 +0000 UTC m=+16.583664974 (1 nodes), 3.116µs from start of poll, 179.044657ms from cluster start. Timeout at 2024-01-09 05:10:05.435731646 +0000 UTC m=+16.688782753, 105.117779ms after stability testLoggerAdapter verbose: cluster: 2024-01-09T05:10:05.435Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:10:05.435Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-01-09T05:10:05.541Z [INFO] stable state for Leader reached at 2024-01-09 05:10:05.435978659 +0000 UTC m=+16.689029766 (1 nodes), 2.359µs from start of poll, 284.409449ms from cluster start. Timeout at 2024-01-09 05:10:05.541090086 +0000 UTC m=+16.794141161, 105.111395ms after stability testing.go:149: cluster: 2024-01-09T05:10:05.541Z [INFO] stable state for Leader reached at 2024-01-09 05:10:05.435978659 +0000 UTC m=+16.689029766 (1 nodes), 2.359µs from start of poll, 284.409449ms from cluster start. Timeout at 2024-01-09 05:10:05.541090086 +0000 UTC m=+16.794141161, 105.111395ms after stability testLoggerAdapter verbose: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.541Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: updating configuration: command=AddStaging server-id=aef09d44-1ee3-903b-a640-190442a2e065 server-addr=aef09d44-1ee3-903b-a640-190442a2e065 servers="[{Suffrage:Voter ID:a0ea6c91-e115-e43e-e35e-3e180635831d Address:a0ea6c91-e115-e43e-e35e-3e180635831d} {Suffrage:Voter ID:a02787f6-97d3-11ae-ae8a-eff88e75b250 Address:a02787f6-97d3-11ae-ae8a-eff88e75b250} {Suffrage:Voter ID:aef09d44-1ee3-903b-a640-190442a2e065 Address:aef09d44-1ee3-903b-a640-190442a2e065}]" testing.go:149: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.541Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: updating configuration: command=AddStaging server-id=aef09d44-1ee3-903b-a640-190442a2e065 server-addr=aef09d44-1ee3-903b-a640-190442a2e065 servers="[{Suffrage:Voter ID:a0ea6c91-e115-e43e-e35e-3e180635831d Address:a0ea6c91-e115-e43e-e35e-3e180635831d} {Suffrage:Voter ID:a02787f6-97d3-11ae-ae8a-eff88e75b250 Address:a02787f6-97d3-11ae-ae8a-eff88e75b250} {Suffrage:Voter ID:aef09d44-1ee3-903b-a640-190442a2e065 Address:aef09d44-1ee3-903b-a640-190442a2e065}]" testLoggerAdapter verbose: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.541Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: added peer, starting replication: peer=aef09d44-1ee3-903b-a640-190442a2e065 testing.go:149: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.541Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: added peer, starting replication: peer=aef09d44-1ee3-903b-a640-190442a2e065 testLoggerAdapter verbose: aef09d44-1ee3-903b-a640-190442a2e065: 2024-01-09T05:10:05.541Z [WARN] aef09d44-1ee3-903b-a640-190442a2e065: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: aef09d44-1ee3-903b-a640-190442a2e065: 2024-01-09T05:10:05.541Z [WARN] aef09d44-1ee3-903b-a640-190442a2e065: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.541Z [WARN] a0ea6c91-e115-e43e-e35e-3e180635831d: appendEntries rejected, sending older logs: peer="{Voter aef09d44-1ee3-903b-a640-190442a2e065 aef09d44-1ee3-903b-a640-190442a2e065}" next=1 testing.go:149: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.541Z [WARN] a0ea6c91-e115-e43e-e35e-3e180635831d: appendEntries rejected, sending older logs: peer="{Voter aef09d44-1ee3-903b-a640-190442a2e065 aef09d44-1ee3-903b-a640-190442a2e065}" next=1 testLoggerAdapter verbose: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.541Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: pipelining replication: peer="{Voter aef09d44-1ee3-903b-a640-190442a2e065 aef09d44-1ee3-903b-a640-190442a2e065}" testing.go:149: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.541Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: pipelining replication: peer="{Voter aef09d44-1ee3-903b-a640-190442a2e065 aef09d44-1ee3-903b-a640-190442a2e065}" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:05.546Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:10:05.546Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-01-09T05:10:05.652Z [INFO] stable state for Leader reached at 2024-01-09 05:10:05.54707786 +0000 UTC m=+16.800128902 (1 nodes), 2.54µs from start of poll, 395.508585ms from cluster start. Timeout at 2024-01-09 05:10:05.652168671 +0000 UTC m=+16.905219675, 105.090773ms after stability testing.go:149: cluster: 2024-01-09T05:10:05.652Z [INFO] stable state for Leader reached at 2024-01-09 05:10:05.54707786 +0000 UTC m=+16.800128902 (1 nodes), 2.54µs from start of poll, 395.508585ms from cluster start. Timeout at 2024-01-09 05:10:05.652168671 +0000 UTC m=+16.905219675, 105.090773ms after stability testLoggerAdapter verbose: cluster: 2024-01-09T05:10:05.652Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:10:05.652Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-01-09T05:10:05.757Z [INFO] stable state for Leader reached at 2024-01-09 05:10:05.652355379 +0000 UTC m=+16.905406372 (1 nodes), 2.051µs from start of poll, 500.786055ms from cluster start. Timeout at 2024-01-09 05:10:05.757380951 +0000 UTC m=+17.010432061, 105.025689ms after stability testing.go:149: cluster: 2024-01-09T05:10:05.757Z [INFO] stable state for Leader reached at 2024-01-09 05:10:05.652355379 +0000 UTC m=+16.905406372 (1 nodes), 2.051µs from start of poll, 500.786055ms from cluster start. Timeout at 2024-01-09 05:10:05.757380951 +0000 UTC m=+17.010432061, 105.025689ms after stability testLoggerAdapter verbose: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.757Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: updating configuration: command=RemoveServer server-id=aef09d44-1ee3-903b-a640-190442a2e065 server-addr= servers="[{Suffrage:Voter ID:a0ea6c91-e115-e43e-e35e-3e180635831d Address:a0ea6c91-e115-e43e-e35e-3e180635831d} {Suffrage:Voter ID:a02787f6-97d3-11ae-ae8a-eff88e75b250 Address:a02787f6-97d3-11ae-ae8a-eff88e75b250}]" testing.go:149: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.757Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: updating configuration: command=RemoveServer server-id=aef09d44-1ee3-903b-a640-190442a2e065 server-addr= servers="[{Suffrage:Voter ID:a0ea6c91-e115-e43e-e35e-3e180635831d Address:a0ea6c91-e115-e43e-e35e-3e180635831d} {Suffrage:Voter ID:a02787f6-97d3-11ae-ae8a-eff88e75b250 Address:a02787f6-97d3-11ae-ae8a-eff88e75b250}]" testLoggerAdapter verbose: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.757Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: removed peer, stopping replication: peer=aef09d44-1ee3-903b-a640-190442a2e065 last-index=4 testing.go:149: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.757Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: removed peer, stopping replication: peer=aef09d44-1ee3-903b-a640-190442a2e065 last-index=4 testLoggerAdapter verbose: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.757Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: aborting pipeline replication: peer="{Voter aef09d44-1ee3-903b-a640-190442a2e065 aef09d44-1ee3-903b-a640-190442a2e065}" testing.go:149: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.757Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: aborting pipeline replication: peer="{Voter aef09d44-1ee3-903b-a640-190442a2e065 aef09d44-1ee3-903b-a640-190442a2e065}" testLoggerAdapter verbose: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.757Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: aborting pipeline replication: peer="{Voter a02787f6-97d3-11ae-ae8a-eff88e75b250 a02787f6-97d3-11ae-ae8a-eff88e75b250}" testing.go:149: a0ea6c91-e115-e43e-e35e-3e180635831d: 2024-01-09T05:10:05.757Z [INFO] a0ea6c91-e115-e43e-e35e-3e180635831d: aborting pipeline replication: peer="{Voter a02787f6-97d3-11ae-ae8a-eff88e75b250 a02787f6-97d3-11ae-ae8a-eff88e75b250}" --- PASS: TestRaft_ProtocolVersion_Upgrade_1_2 (0.61s) === RUN TestRaft_ProtocolVersion_Upgrade_2_3 testLoggerAdapter verbose: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:05.759Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: initial configuration: index=1 servers="[{Suffrage:Voter ID:4ef9a0f8-fdeb-c797-0840-35b5ebd602cc Address:4ef9a0f8-fdeb-c797-0840-35b5ebd602cc} {Suffrage:Voter ID:a0acff85-887c-9857-f396-95021c279817 Address:a0acff85-887c-9857-f396-95021c279817}]" testing.go:149: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:05.759Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: initial configuration: index=1 servers="[{Suffrage:Voter ID:4ef9a0f8-fdeb-c797-0840-35b5ebd602cc Address:4ef9a0f8-fdeb-c797-0840-35b5ebd602cc} {Suffrage:Voter ID:a0acff85-887c-9857-f396-95021c279817 Address:a0acff85-887c-9857-f396-95021c279817}]" testLoggerAdapter verbose: a0acff85-887c-9857-f396-95021c279817: 2024-01-09T05:10:05.759Z [INFO] a0acff85-887c-9857-f396-95021c279817: initial configuration: index=1 servers="[{Suffrage:Voter ID:4ef9a0f8-fdeb-c797-0840-35b5ebd602cc Address:4ef9a0f8-fdeb-c797-0840-35b5ebd602cc} {Suffrage:Voter ID:a0acff85-887c-9857-f396-95021c279817 Address:a0acff85-887c-9857-f396-95021c279817}]" testing.go:149: a0acff85-887c-9857-f396-95021c279817: 2024-01-09T05:10:05.759Z [INFO] a0acff85-887c-9857-f396-95021c279817: initial configuration: index=1 servers="[{Suffrage:Voter ID:4ef9a0f8-fdeb-c797-0840-35b5ebd602cc Address:4ef9a0f8-fdeb-c797-0840-35b5ebd602cc} {Suffrage:Voter ID:a0acff85-887c-9857-f396-95021c279817 Address:a0acff85-887c-9857-f396-95021c279817}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:05.759Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-01-09T05:10:05.759Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:05.759Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: entering follower state: follower="Node at 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc [Follower]" leader= testing.go:149: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:05.759Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: entering follower state: follower="Node at 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc [Follower]" leader= testLoggerAdapter verbose: a0acff85-887c-9857-f396-95021c279817: 2024-01-09T05:10:05.759Z [INFO] a0acff85-887c-9857-f396-95021c279817: entering follower state: follower="Node at a0acff85-887c-9857-f396-95021c279817 [Follower]" leader= testing.go:149: a0acff85-887c-9857-f396-95021c279817: 2024-01-09T05:10:05.759Z [INFO] a0acff85-887c-9857-f396-95021c279817: entering follower state: follower="Node at a0acff85-887c-9857-f396-95021c279817 [Follower]" leader= testLoggerAdapter verbose: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:05.814Z [WARN] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: heartbeat timeout reached, starting election: last-leader= testing.go:149: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:05.814Z [WARN] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:05.814Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: entering candidate state: node="Node at 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc [Candidate]" term=2 testing.go:149: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:05.814Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: entering candidate state: node="Node at 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc [Candidate]" term=2 testLoggerAdapter verbose: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:05.814Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: election won: tally=2 testing.go:149: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:05.814Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: election won: tally=2 testLoggerAdapter verbose: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:05.814Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: entering leader state: leader="Node at 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc [Leader]" testing.go:149: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:05.814Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: entering leader state: leader="Node at 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc [Leader]" testLoggerAdapter verbose: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:05.814Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: added peer, starting replication: peer=a0acff85-887c-9857-f396-95021c279817 testing.go:149: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:05.814Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: added peer, starting replication: peer=a0acff85-887c-9857-f396-95021c279817 testLoggerAdapter verbose: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:05.814Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: pipelining replication: peer="{Voter a0acff85-887c-9857-f396-95021c279817 a0acff85-887c-9857-f396-95021c279817}" testing.go:149: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:05.814Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: pipelining replication: peer="{Voter a0acff85-887c-9857-f396-95021c279817 a0acff85-887c-9857-f396-95021c279817}" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:05.919Z [INFO] stable state for Follower reached at 2024-01-09 05:10:05.814747193 +0000 UTC m=+17.067798297 (1 nodes), 55.273817ms from start of poll, 55.619466ms from cluster start. Timeout at 2024-01-09 05:10:05.919921604 +0000 UTC m=+17.172972731, 105.174434ms after stability testing.go:149: cluster: 2024-01-09T05:10:05.919Z [INFO] stable state for Follower reached at 2024-01-09 05:10:05.814747193 +0000 UTC m=+17.067798297 (1 nodes), 55.273817ms from start of poll, 55.619466ms from cluster start. Timeout at 2024-01-09 05:10:05.919921604 +0000 UTC m=+17.172972731, 105.174434ms after stability testLoggerAdapter verbose: server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2: 2024-01-09T05:10:05.920Z [INFO] server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2: initial configuration: index=0 servers=[] testing.go:149: server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2: 2024-01-09T05:10:05.920Z [INFO] server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2024-01-09T05:10:05.920Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:10:05.920Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2: 2024-01-09T05:10:05.920Z [INFO] server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2: entering follower state: follower="Node at 19e3d1f0-6a40-2083-2d89-cd1e0db04dc2 [Follower]" leader= testing.go:149: server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2: 2024-01-09T05:10:05.920Z [INFO] server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2: entering follower state: follower="Node at 19e3d1f0-6a40-2083-2d89-cd1e0db04dc2 [Follower]" leader= testLoggerAdapter verbose: server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2: 2024-01-09T05:10:05.997Z [WARN] server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2: no known peers, aborting election testing.go:149: server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2: 2024-01-09T05:10:05.997Z [WARN] server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2: no known peers, aborting election testLoggerAdapter verbose: cluster: 2024-01-09T05:10:06.025Z [INFO] stable state for Leader reached at 2024-01-09 05:10:05.920501325 +0000 UTC m=+17.173552337 (1 nodes), 2.272µs from start of poll, 161.373506ms from cluster start. Timeout at 2024-01-09 05:10:06.025637916 +0000 UTC m=+17.278689066, 105.136729ms after stability testing.go:149: cluster: 2024-01-09T05:10:06.025Z [INFO] stable state for Leader reached at 2024-01-09 05:10:05.920501325 +0000 UTC m=+17.173552337 (1 nodes), 2.272µs from start of poll, 161.373506ms from cluster start. Timeout at 2024-01-09 05:10:06.025637916 +0000 UTC m=+17.278689066, 105.136729ms after stability testLoggerAdapter verbose: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:06.025Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: updating configuration: command=AddStaging server-id=server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2 server-addr=19e3d1f0-6a40-2083-2d89-cd1e0db04dc2 servers="[{Suffrage:Voter ID:4ef9a0f8-fdeb-c797-0840-35b5ebd602cc Address:4ef9a0f8-fdeb-c797-0840-35b5ebd602cc} {Suffrage:Voter ID:a0acff85-887c-9857-f396-95021c279817 Address:a0acff85-887c-9857-f396-95021c279817} {Suffrage:Voter ID:server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2 Address:19e3d1f0-6a40-2083-2d89-cd1e0db04dc2}]" testing.go:149: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:06.025Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: updating configuration: command=AddStaging server-id=server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2 server-addr=19e3d1f0-6a40-2083-2d89-cd1e0db04dc2 servers="[{Suffrage:Voter ID:4ef9a0f8-fdeb-c797-0840-35b5ebd602cc Address:4ef9a0f8-fdeb-c797-0840-35b5ebd602cc} {Suffrage:Voter ID:a0acff85-887c-9857-f396-95021c279817 Address:a0acff85-887c-9857-f396-95021c279817} {Suffrage:Voter ID:server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2 Address:19e3d1f0-6a40-2083-2d89-cd1e0db04dc2}]" testLoggerAdapter verbose: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:06.025Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: added peer, starting replication: peer=server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2 testing.go:149: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:06.025Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: added peer, starting replication: peer=server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2 testLoggerAdapter verbose: server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2: 2024-01-09T05:10:06.026Z [WARN] server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2: 2024-01-09T05:10:06.026Z [WARN] server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:06.026Z [WARN] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: appendEntries rejected, sending older logs: peer="{Voter server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2 19e3d1f0-6a40-2083-2d89-cd1e0db04dc2}" next=1 testing.go:149: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:06.026Z [WARN] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: appendEntries rejected, sending older logs: peer="{Voter server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2 19e3d1f0-6a40-2083-2d89-cd1e0db04dc2}" next=1 testLoggerAdapter verbose: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:06.026Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: pipelining replication: peer="{Voter server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2 19e3d1f0-6a40-2083-2d89-cd1e0db04dc2}" testing.go:149: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:06.026Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: pipelining replication: peer="{Voter server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2 19e3d1f0-6a40-2083-2d89-cd1e0db04dc2}" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:06.033Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:10:06.033Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-01-09T05:10:06.139Z [INFO] stable state for Leader reached at 2024-01-09 05:10:06.033852415 +0000 UTC m=+17.286903434 (1 nodes), 2.076µs from start of poll, 274.724603ms from cluster start. Timeout at 2024-01-09 05:10:06.138928335 +0000 UTC m=+17.391979595, 105.076161ms after stability testing.go:149: cluster: 2024-01-09T05:10:06.139Z [INFO] stable state for Leader reached at 2024-01-09 05:10:06.033852415 +0000 UTC m=+17.286903434 (1 nodes), 2.076µs from start of poll, 274.724603ms from cluster start. Timeout at 2024-01-09 05:10:06.138928335 +0000 UTC m=+17.391979595, 105.076161ms after stability testLoggerAdapter verbose: cluster: 2024-01-09T05:10:06.139Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:10:06.139Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-01-09T05:10:06.248Z [INFO] stable state for Leader reached at 2024-01-09 05:10:06.139128975 +0000 UTC m=+17.392180133 (1 nodes), 2.436µs from start of poll, 380.001302ms from cluster start. Timeout at 2024-01-09 05:10:06.248379404 +0000 UTC m=+17.501430492, 109.250359ms after stability testing.go:149: cluster: 2024-01-09T05:10:06.248Z [INFO] stable state for Leader reached at 2024-01-09 05:10:06.139128975 +0000 UTC m=+17.392180133 (1 nodes), 2.436µs from start of poll, 380.001302ms from cluster start. Timeout at 2024-01-09 05:10:06.248379404 +0000 UTC m=+17.501430492, 109.250359ms after stability testLoggerAdapter verbose: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:06.248Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: updating configuration: command=RemoveServer server-id=a0acff85-887c-9857-f396-95021c279817 server-addr= servers="[{Suffrage:Voter ID:4ef9a0f8-fdeb-c797-0840-35b5ebd602cc Address:4ef9a0f8-fdeb-c797-0840-35b5ebd602cc} {Suffrage:Voter ID:server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2 Address:19e3d1f0-6a40-2083-2d89-cd1e0db04dc2}]" testing.go:149: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:06.248Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: updating configuration: command=RemoveServer server-id=a0acff85-887c-9857-f396-95021c279817 server-addr= servers="[{Suffrage:Voter ID:4ef9a0f8-fdeb-c797-0840-35b5ebd602cc Address:4ef9a0f8-fdeb-c797-0840-35b5ebd602cc} {Suffrage:Voter ID:server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2 Address:19e3d1f0-6a40-2083-2d89-cd1e0db04dc2}]" testLoggerAdapter verbose: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:06.248Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: removed peer, stopping replication: peer=a0acff85-887c-9857-f396-95021c279817 last-index=4 testing.go:149: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:06.248Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: removed peer, stopping replication: peer=a0acff85-887c-9857-f396-95021c279817 last-index=4 testLoggerAdapter verbose: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:06.248Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: aborting pipeline replication: peer="{Voter a0acff85-887c-9857-f396-95021c279817 a0acff85-887c-9857-f396-95021c279817}" testing.go:149: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:06.248Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: aborting pipeline replication: peer="{Voter a0acff85-887c-9857-f396-95021c279817 a0acff85-887c-9857-f396-95021c279817}" testLoggerAdapter verbose: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:06.248Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: aborting pipeline replication: peer="{Voter server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2 19e3d1f0-6a40-2083-2d89-cd1e0db04dc2}" testing.go:149: 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: 2024-01-09T05:10:06.248Z [INFO] 4ef9a0f8-fdeb-c797-0840-35b5ebd602cc: aborting pipeline replication: peer="{Voter server-19e3d1f0-6a40-2083-2d89-cd1e0db04dc2 19e3d1f0-6a40-2083-2d89-cd1e0db04dc2}" --- PASS: TestRaft_ProtocolVersion_Upgrade_2_3 (0.49s) === RUN TestRaft_LeadershipTransferInProgress --- PASS: TestRaft_LeadershipTransferInProgress (0.00s) === RUN TestRaft_LeadershipTransferPickServer --- PASS: TestRaft_LeadershipTransferPickServer (0.00s) === RUN TestRaft_LeadershipTransfer testLoggerAdapter verbose: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.250Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a19ff899-04fa-609d-505d-f02c746b4e5a Address:a19ff899-04fa-609d-505d-f02c746b4e5a} {Suffrage:Voter ID:server-5edbc0c0-7378-358b-2799-d963b1edf0fe Address:5edbc0c0-7378-358b-2799-d963b1edf0fe} {Suffrage:Voter ID:server-c41e38b3-6a17-7080-5877-70799996b8ea Address:c41e38b3-6a17-7080-5877-70799996b8ea}]" testing.go:149: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.250Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a19ff899-04fa-609d-505d-f02c746b4e5a Address:a19ff899-04fa-609d-505d-f02c746b4e5a} {Suffrage:Voter ID:server-5edbc0c0-7378-358b-2799-d963b1edf0fe Address:5edbc0c0-7378-358b-2799-d963b1edf0fe} {Suffrage:Voter ID:server-c41e38b3-6a17-7080-5877-70799996b8ea Address:c41e38b3-6a17-7080-5877-70799996b8ea}]" testLoggerAdapter verbose: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.250Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: entering follower state: follower="Node at a19ff899-04fa-609d-505d-f02c746b4e5a [Follower]" leader= testing.go:149: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.250Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: entering follower state: follower="Node at a19ff899-04fa-609d-505d-f02c746b4e5a [Follower]" leader= testLoggerAdapter verbose: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.250Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a19ff899-04fa-609d-505d-f02c746b4e5a Address:a19ff899-04fa-609d-505d-f02c746b4e5a} {Suffrage:Voter ID:server-5edbc0c0-7378-358b-2799-d963b1edf0fe Address:5edbc0c0-7378-358b-2799-d963b1edf0fe} {Suffrage:Voter ID:server-c41e38b3-6a17-7080-5877-70799996b8ea Address:c41e38b3-6a17-7080-5877-70799996b8ea}]" testing.go:149: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.250Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a19ff899-04fa-609d-505d-f02c746b4e5a Address:a19ff899-04fa-609d-505d-f02c746b4e5a} {Suffrage:Voter ID:server-5edbc0c0-7378-358b-2799-d963b1edf0fe Address:5edbc0c0-7378-358b-2799-d963b1edf0fe} {Suffrage:Voter ID:server-c41e38b3-6a17-7080-5877-70799996b8ea Address:c41e38b3-6a17-7080-5877-70799996b8ea}]" testLoggerAdapter verbose: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.251Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: entering follower state: follower="Node at 5edbc0c0-7378-358b-2799-d963b1edf0fe [Follower]" leader= testing.go:149: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.251Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: entering follower state: follower="Node at 5edbc0c0-7378-358b-2799-d963b1edf0fe [Follower]" leader= testLoggerAdapter verbose: server-c41e38b3-6a17-7080-5877-70799996b8ea: 2024-01-09T05:10:06.251Z [INFO] server-c41e38b3-6a17-7080-5877-70799996b8ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a19ff899-04fa-609d-505d-f02c746b4e5a Address:a19ff899-04fa-609d-505d-f02c746b4e5a} {Suffrage:Voter ID:server-5edbc0c0-7378-358b-2799-d963b1edf0fe Address:5edbc0c0-7378-358b-2799-d963b1edf0fe} {Suffrage:Voter ID:server-c41e38b3-6a17-7080-5877-70799996b8ea Address:c41e38b3-6a17-7080-5877-70799996b8ea}]" testing.go:149: server-c41e38b3-6a17-7080-5877-70799996b8ea: 2024-01-09T05:10:06.251Z [INFO] server-c41e38b3-6a17-7080-5877-70799996b8ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a19ff899-04fa-609d-505d-f02c746b4e5a Address:a19ff899-04fa-609d-505d-f02c746b4e5a} {Suffrage:Voter ID:server-5edbc0c0-7378-358b-2799-d963b1edf0fe Address:5edbc0c0-7378-358b-2799-d963b1edf0fe} {Suffrage:Voter ID:server-c41e38b3-6a17-7080-5877-70799996b8ea Address:c41e38b3-6a17-7080-5877-70799996b8ea}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:06.251Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:10:06.251Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-c41e38b3-6a17-7080-5877-70799996b8ea: 2024-01-09T05:10:06.251Z [INFO] server-c41e38b3-6a17-7080-5877-70799996b8ea: entering follower state: follower="Node at c41e38b3-6a17-7080-5877-70799996b8ea [Follower]" leader= testing.go:149: server-c41e38b3-6a17-7080-5877-70799996b8ea: 2024-01-09T05:10:06.251Z [INFO] server-c41e38b3-6a17-7080-5877-70799996b8ea: entering follower state: follower="Node at c41e38b3-6a17-7080-5877-70799996b8ea [Follower]" leader= testLoggerAdapter verbose: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.308Z [WARN] server-a19ff899-04fa-609d-505d-f02c746b4e5a: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.308Z [WARN] server-a19ff899-04fa-609d-505d-f02c746b4e5a: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.308Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: entering candidate state: node="Node at a19ff899-04fa-609d-505d-f02c746b4e5a [Candidate]" term=2 testing.go:149: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.308Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: entering candidate state: node="Node at a19ff899-04fa-609d-505d-f02c746b4e5a [Candidate]" term=2 testLoggerAdapter verbose: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.308Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: election won: tally=2 testing.go:149: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.308Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: election won: tally=2 testLoggerAdapter verbose: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.308Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: entering leader state: leader="Node at a19ff899-04fa-609d-505d-f02c746b4e5a [Leader]" testing.go:149: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.308Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: entering leader state: leader="Node at a19ff899-04fa-609d-505d-f02c746b4e5a [Leader]" testLoggerAdapter verbose: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.308Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: added peer, starting replication: peer=server-5edbc0c0-7378-358b-2799-d963b1edf0fe testing.go:149: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.308Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: added peer, starting replication: peer=server-5edbc0c0-7378-358b-2799-d963b1edf0fe testLoggerAdapter verbose: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.308Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: added peer, starting replication: peer=server-c41e38b3-6a17-7080-5877-70799996b8ea testing.go:149: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.308Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: added peer, starting replication: peer=server-c41e38b3-6a17-7080-5877-70799996b8ea testLoggerAdapter verbose: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.308Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: pipelining replication: peer="{Voter server-c41e38b3-6a17-7080-5877-70799996b8ea c41e38b3-6a17-7080-5877-70799996b8ea}" testing.go:149: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.308Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: pipelining replication: peer="{Voter server-c41e38b3-6a17-7080-5877-70799996b8ea c41e38b3-6a17-7080-5877-70799996b8ea}" testLoggerAdapter verbose: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.309Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: pipelining replication: peer="{Voter server-5edbc0c0-7378-358b-2799-d963b1edf0fe 5edbc0c0-7378-358b-2799-d963b1edf0fe}" testing.go:149: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.309Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: pipelining replication: peer="{Voter server-5edbc0c0-7378-358b-2799-d963b1edf0fe 5edbc0c0-7378-358b-2799-d963b1edf0fe}" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:06.414Z [INFO] stable state for Leader reached at 2024-01-09 05:10:06.309102467 +0000 UTC m=+17.562153655 (1 nodes), 57.754861ms from start of poll, 58.562369ms from cluster start. Timeout at 2024-01-09 05:10:06.414600017 +0000 UTC m=+17.667651000, 105.497345ms after stability testing.go:149: cluster: 2024-01-09T05:10:06.414Z [INFO] stable state for Leader reached at 2024-01-09 05:10:06.309102467 +0000 UTC m=+17.562153655 (1 nodes), 57.754861ms from start of poll, 58.562369ms from cluster start. Timeout at 2024-01-09 05:10:06.414600017 +0000 UTC m=+17.667651000, 105.497345ms after stability testLoggerAdapter verbose: cluster: 2024-01-09T05:10:06.414Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:10:06.414Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-01-09T05:10:06.520Z [INFO] stable state for Leader reached at 2024-01-09 05:10:06.41478448 +0000 UTC m=+17.667835507 (1 nodes), 2.527µs from start of poll, 164.244221ms from cluster start. Timeout at 2024-01-09 05:10:06.520923969 +0000 UTC m=+17.773975067, 106.13956ms after stability testing.go:149: cluster: 2024-01-09T05:10:06.520Z [INFO] stable state for Leader reached at 2024-01-09 05:10:06.41478448 +0000 UTC m=+17.667835507 (1 nodes), 2.527µs from start of poll, 164.244221ms from cluster start. Timeout at 2024-01-09 05:10:06.520923969 +0000 UTC m=+17.773975067, 106.13956ms after stability testLoggerAdapter verbose: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.521Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: entering candidate state: node="Node at 5edbc0c0-7378-358b-2799-d963b1edf0fe [Candidate]" term=3 testing.go:149: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.521Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: entering candidate state: node="Node at 5edbc0c0-7378-358b-2799-d963b1edf0fe [Candidate]" term=3 testLoggerAdapter verbose: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.521Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: election won: tally=2 testing.go:149: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.521Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: election won: tally=2 testLoggerAdapter verbose: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.521Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: entering leader state: leader="Node at 5edbc0c0-7378-358b-2799-d963b1edf0fe [Leader]" testing.go:149: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.521Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: entering leader state: leader="Node at 5edbc0c0-7378-358b-2799-d963b1edf0fe [Leader]" testLoggerAdapter verbose: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.521Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: added peer, starting replication: peer=server-a19ff899-04fa-609d-505d-f02c746b4e5a testing.go:149: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.521Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: added peer, starting replication: peer=server-a19ff899-04fa-609d-505d-f02c746b4e5a testLoggerAdapter verbose: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.521Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: added peer, starting replication: peer=server-c41e38b3-6a17-7080-5877-70799996b8ea testing.go:149: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.521Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: added peer, starting replication: peer=server-c41e38b3-6a17-7080-5877-70799996b8ea testLoggerAdapter verbose: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.521Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: pipelining replication: peer="{Voter server-c41e38b3-6a17-7080-5877-70799996b8ea c41e38b3-6a17-7080-5877-70799996b8ea}" testing.go:149: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.521Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: pipelining replication: peer="{Voter server-c41e38b3-6a17-7080-5877-70799996b8ea c41e38b3-6a17-7080-5877-70799996b8ea}" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:06.521Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:10:06.521Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.521Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: entering follower state: follower="Node at a19ff899-04fa-609d-505d-f02c746b4e5a [Follower]" leader= testing.go:149: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.521Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: entering follower state: follower="Node at a19ff899-04fa-609d-505d-f02c746b4e5a [Follower]" leader= testLoggerAdapter verbose: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.521Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: aborting pipeline replication: peer="{Voter server-c41e38b3-6a17-7080-5877-70799996b8ea c41e38b3-6a17-7080-5877-70799996b8ea}" testing.go:149: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.521Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: aborting pipeline replication: peer="{Voter server-c41e38b3-6a17-7080-5877-70799996b8ea c41e38b3-6a17-7080-5877-70799996b8ea}" testLoggerAdapter verbose: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.521Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: pipelining replication: peer="{Voter server-a19ff899-04fa-609d-505d-f02c746b4e5a a19ff899-04fa-609d-505d-f02c746b4e5a}" testing.go:149: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.521Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: pipelining replication: peer="{Voter server-a19ff899-04fa-609d-505d-f02c746b4e5a a19ff899-04fa-609d-505d-f02c746b4e5a}" testLoggerAdapter verbose: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.521Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: aborting pipeline replication: peer="{Voter server-5edbc0c0-7378-358b-2799-d963b1edf0fe 5edbc0c0-7378-358b-2799-d963b1edf0fe}" testing.go:149: server-a19ff899-04fa-609d-505d-f02c746b4e5a: 2024-01-09T05:10:06.521Z [INFO] server-a19ff899-04fa-609d-505d-f02c746b4e5a: aborting pipeline replication: peer="{Voter server-5edbc0c0-7378-358b-2799-d963b1edf0fe 5edbc0c0-7378-358b-2799-d963b1edf0fe}" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:06.627Z [INFO] stable state for Leader reached at 2024-01-09 05:10:06.521861338 +0000 UTC m=+17.774912374 (1 nodes), 370.172µs from start of poll, 271.321088ms from cluster start. Timeout at 2024-01-09 05:10:06.627069519 +0000 UTC m=+17.880120631, 105.208257ms after stability testing.go:149: cluster: 2024-01-09T05:10:06.627Z [INFO] stable state for Leader reached at 2024-01-09 05:10:06.521861338 +0000 UTC m=+17.774912374 (1 nodes), 370.172µs from start of poll, 271.321088ms from cluster start. Timeout at 2024-01-09 05:10:06.627069519 +0000 UTC m=+17.880120631, 105.208257ms after stability testLoggerAdapter verbose: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.627Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: aborting pipeline replication: peer="{Voter server-c41e38b3-6a17-7080-5877-70799996b8ea c41e38b3-6a17-7080-5877-70799996b8ea}" testing.go:149: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.627Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: aborting pipeline replication: peer="{Voter server-c41e38b3-6a17-7080-5877-70799996b8ea c41e38b3-6a17-7080-5877-70799996b8ea}" testLoggerAdapter verbose: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.627Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: aborting pipeline replication: peer="{Voter server-a19ff899-04fa-609d-505d-f02c746b4e5a a19ff899-04fa-609d-505d-f02c746b4e5a}" testing.go:149: server-5edbc0c0-7378-358b-2799-d963b1edf0fe: 2024-01-09T05:10:06.627Z [INFO] server-5edbc0c0-7378-358b-2799-d963b1edf0fe: aborting pipeline replication: peer="{Voter server-a19ff899-04fa-609d-505d-f02c746b4e5a a19ff899-04fa-609d-505d-f02c746b4e5a}" --- PASS: TestRaft_LeadershipTransfer (0.38s) === RUN TestRaft_LeadershipTransferWithOneNode testLoggerAdapter verbose: server-365a107c-90e7-6fc4-be0e-524397881df2: 2024-01-09T05:10:06.628Z [INFO] server-365a107c-90e7-6fc4-be0e-524397881df2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-365a107c-90e7-6fc4-be0e-524397881df2 Address:365a107c-90e7-6fc4-be0e-524397881df2}]" testing.go:149: server-365a107c-90e7-6fc4-be0e-524397881df2: 2024-01-09T05:10:06.628Z [INFO] server-365a107c-90e7-6fc4-be0e-524397881df2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-365a107c-90e7-6fc4-be0e-524397881df2 Address:365a107c-90e7-6fc4-be0e-524397881df2}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:06.628Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:10:06.628Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-365a107c-90e7-6fc4-be0e-524397881df2: 2024-01-09T05:10:06.628Z [INFO] server-365a107c-90e7-6fc4-be0e-524397881df2: entering follower state: follower="Node at 365a107c-90e7-6fc4-be0e-524397881df2 [Follower]" leader= testing.go:149: server-365a107c-90e7-6fc4-be0e-524397881df2: 2024-01-09T05:10:06.628Z [INFO] server-365a107c-90e7-6fc4-be0e-524397881df2: entering follower state: follower="Node at 365a107c-90e7-6fc4-be0e-524397881df2 [Follower]" leader= testLoggerAdapter verbose: server-365a107c-90e7-6fc4-be0e-524397881df2: 2024-01-09T05:10:06.694Z [WARN] server-365a107c-90e7-6fc4-be0e-524397881df2: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-365a107c-90e7-6fc4-be0e-524397881df2: 2024-01-09T05:10:06.694Z [WARN] server-365a107c-90e7-6fc4-be0e-524397881df2: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-365a107c-90e7-6fc4-be0e-524397881df2: 2024-01-09T05:10:06.694Z [INFO] server-365a107c-90e7-6fc4-be0e-524397881df2: entering candidate state: node="Node at 365a107c-90e7-6fc4-be0e-524397881df2 [Candidate]" term=2 testing.go:149: server-365a107c-90e7-6fc4-be0e-524397881df2: 2024-01-09T05:10:06.694Z [INFO] server-365a107c-90e7-6fc4-be0e-524397881df2: entering candidate state: node="Node at 365a107c-90e7-6fc4-be0e-524397881df2 [Candidate]" term=2 testLoggerAdapter verbose: server-365a107c-90e7-6fc4-be0e-524397881df2: 2024-01-09T05:10:06.695Z [INFO] server-365a107c-90e7-6fc4-be0e-524397881df2: election won: tally=1 testing.go:149: server-365a107c-90e7-6fc4-be0e-524397881df2: 2024-01-09T05:10:06.695Z [INFO] server-365a107c-90e7-6fc4-be0e-524397881df2: election won: tally=1 testLoggerAdapter verbose: server-365a107c-90e7-6fc4-be0e-524397881df2: 2024-01-09T05:10:06.695Z [INFO] server-365a107c-90e7-6fc4-be0e-524397881df2: entering leader state: leader="Node at 365a107c-90e7-6fc4-be0e-524397881df2 [Leader]" testing.go:149: server-365a107c-90e7-6fc4-be0e-524397881df2: 2024-01-09T05:10:06.695Z [INFO] server-365a107c-90e7-6fc4-be0e-524397881df2: entering leader state: leader="Node at 365a107c-90e7-6fc4-be0e-524397881df2 [Leader]" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:06.806Z [INFO] stable state for Leader reached at 2024-01-09 05:10:06.695143229 +0000 UTC m=+17.948194276 (1 nodes), 66.599257ms from start of poll, 66.804662ms from cluster start. Timeout at 2024-01-09 05:10:06.8069404 +0000 UTC m=+18.059991508, 111.797232ms after stability testing.go:149: cluster: 2024-01-09T05:10:06.806Z [INFO] stable state for Leader reached at 2024-01-09 05:10:06.695143229 +0000 UTC m=+17.948194276 (1 nodes), 66.599257ms from start of poll, 66.804662ms from cluster start. Timeout at 2024-01-09 05:10:06.8069404 +0000 UTC m=+18.059991508, 111.797232ms after stability --- PASS: TestRaft_LeadershipTransferWithOneNode (0.18s) === RUN TestRaft_LeadershipTransferWithSevenNodes testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:06.808Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 Address:c9b91bba-dc30-e04b-fb6e-30317aaae7f0} {Suffrage:Voter ID:server-78458f2b-7852-b1e6-0e47-696b8d2b327a Address:78458f2b-7852-b1e6-0e47-696b8d2b327a} {Suffrage:Voter ID:server-7690a8da-ca4b-74d6-905a-84887782f70a Address:7690a8da-ca4b-74d6-905a-84887782f70a} {Suffrage:Voter ID:server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb Address:b1cf2ae1-f267-abda-3212-ac286ecd9fdb} {Suffrage:Voter ID:server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 Address:e5774e4f-7d02-8923-9fa1-7d61c66aeee0} {Suffrage:Voter ID:server-a96c7b98-21ec-611e-8239-dea35c98ee1e Address:a96c7b98-21ec-611e-8239-dea35c98ee1e} {Suffrage:Voter ID:server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff Address:7e15329d-e3f8-73ab-e49c-21e6c404a4ff}]" testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:06.808Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 Address:c9b91bba-dc30-e04b-fb6e-30317aaae7f0} {Suffrage:Voter ID:server-78458f2b-7852-b1e6-0e47-696b8d2b327a Address:78458f2b-7852-b1e6-0e47-696b8d2b327a} {Suffrage:Voter ID:server-7690a8da-ca4b-74d6-905a-84887782f70a Address:7690a8da-ca4b-74d6-905a-84887782f70a} {Suffrage:Voter ID:server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb Address:b1cf2ae1-f267-abda-3212-ac286ecd9fdb} {Suffrage:Voter ID:server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 Address:e5774e4f-7d02-8923-9fa1-7d61c66aeee0} {Suffrage:Voter ID:server-a96c7b98-21ec-611e-8239-dea35c98ee1e Address:a96c7b98-21ec-611e-8239-dea35c98ee1e} {Suffrage:Voter ID:server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff Address:7e15329d-e3f8-73ab-e49c-21e6c404a4ff}]" testLoggerAdapter verbose: server-78458f2b-7852-b1e6-0e47-696b8d2b327a: 2024-01-09T05:10:06.808Z [INFO] server-78458f2b-7852-b1e6-0e47-696b8d2b327a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 Address:c9b91bba-dc30-e04b-fb6e-30317aaae7f0} {Suffrage:Voter ID:server-78458f2b-7852-b1e6-0e47-696b8d2b327a Address:78458f2b-7852-b1e6-0e47-696b8d2b327a} {Suffrage:Voter ID:server-7690a8da-ca4b-74d6-905a-84887782f70a Address:7690a8da-ca4b-74d6-905a-84887782f70a} {Suffrage:Voter ID:server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb Address:b1cf2ae1-f267-abda-3212-ac286ecd9fdb} {Suffrage:Voter ID:server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 Address:e5774e4f-7d02-8923-9fa1-7d61c66aeee0} {Suffrage:Voter ID:server-a96c7b98-21ec-611e-8239-dea35c98ee1e Address:a96c7b98-21ec-611e-8239-dea35c98ee1e} {Suffrage:Voter ID:server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff Address:7e15329d-e3f8-73ab-e49c-21e6c404a4ff}]" testing.go:149: server-78458f2b-7852-b1e6-0e47-696b8d2b327a: 2024-01-09T05:10:06.808Z [INFO] server-78458f2b-7852-b1e6-0e47-696b8d2b327a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 Address:c9b91bba-dc30-e04b-fb6e-30317aaae7f0} {Suffrage:Voter ID:server-78458f2b-7852-b1e6-0e47-696b8d2b327a Address:78458f2b-7852-b1e6-0e47-696b8d2b327a} {Suffrage:Voter ID:server-7690a8da-ca4b-74d6-905a-84887782f70a Address:7690a8da-ca4b-74d6-905a-84887782f70a} {Suffrage:Voter ID:server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb Address:b1cf2ae1-f267-abda-3212-ac286ecd9fdb} {Suffrage:Voter ID:server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 Address:e5774e4f-7d02-8923-9fa1-7d61c66aeee0} {Suffrage:Voter ID:server-a96c7b98-21ec-611e-8239-dea35c98ee1e Address:a96c7b98-21ec-611e-8239-dea35c98ee1e} {Suffrage:Voter ID:server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff Address:7e15329d-e3f8-73ab-e49c-21e6c404a4ff}]" testLoggerAdapter verbose: server-7690a8da-ca4b-74d6-905a-84887782f70a: 2024-01-09T05:10:06.809Z [INFO] server-7690a8da-ca4b-74d6-905a-84887782f70a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 Address:c9b91bba-dc30-e04b-fb6e-30317aaae7f0} {Suffrage:Voter ID:server-78458f2b-7852-b1e6-0e47-696b8d2b327a Address:78458f2b-7852-b1e6-0e47-696b8d2b327a} {Suffrage:Voter ID:server-7690a8da-ca4b-74d6-905a-84887782f70a Address:7690a8da-ca4b-74d6-905a-84887782f70a} {Suffrage:Voter ID:server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb Address:b1cf2ae1-f267-abda-3212-ac286ecd9fdb} {Suffrage:Voter ID:server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 Address:e5774e4f-7d02-8923-9fa1-7d61c66aeee0} {Suffrage:Voter ID:server-a96c7b98-21ec-611e-8239-dea35c98ee1e Address:a96c7b98-21ec-611e-8239-dea35c98ee1e} {Suffrage:Voter ID:server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff Address:7e15329d-e3f8-73ab-e49c-21e6c404a4ff}]" testing.go:149: server-7690a8da-ca4b-74d6-905a-84887782f70a: 2024-01-09T05:10:06.809Z [INFO] server-7690a8da-ca4b-74d6-905a-84887782f70a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 Address:c9b91bba-dc30-e04b-fb6e-30317aaae7f0} {Suffrage:Voter ID:server-78458f2b-7852-b1e6-0e47-696b8d2b327a Address:78458f2b-7852-b1e6-0e47-696b8d2b327a} {Suffrage:Voter ID:server-7690a8da-ca4b-74d6-905a-84887782f70a Address:7690a8da-ca4b-74d6-905a-84887782f70a} {Suffrage:Voter ID:server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb Address:b1cf2ae1-f267-abda-3212-ac286ecd9fdb} {Suffrage:Voter ID:server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 Address:e5774e4f-7d02-8923-9fa1-7d61c66aeee0} {Suffrage:Voter ID:server-a96c7b98-21ec-611e-8239-dea35c98ee1e Address:a96c7b98-21ec-611e-8239-dea35c98ee1e} {Suffrage:Voter ID:server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff Address:7e15329d-e3f8-73ab-e49c-21e6c404a4ff}]" testLoggerAdapter verbose: server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb: 2024-01-09T05:10:06.809Z [INFO] server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 Address:c9b91bba-dc30-e04b-fb6e-30317aaae7f0} {Suffrage:Voter ID:server-78458f2b-7852-b1e6-0e47-696b8d2b327a Address:78458f2b-7852-b1e6-0e47-696b8d2b327a} {Suffrage:Voter ID:server-7690a8da-ca4b-74d6-905a-84887782f70a Address:7690a8da-ca4b-74d6-905a-84887782f70a} {Suffrage:Voter ID:server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb Address:b1cf2ae1-f267-abda-3212-ac286ecd9fdb} {Suffrage:Voter ID:server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 Address:e5774e4f-7d02-8923-9fa1-7d61c66aeee0} {Suffrage:Voter ID:server-a96c7b98-21ec-611e-8239-dea35c98ee1e Address:a96c7b98-21ec-611e-8239-dea35c98ee1e} {Suffrage:Voter ID:server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff Address:7e15329d-e3f8-73ab-e49c-21e6c404a4ff}]" testing.go:149: server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb: 2024-01-09T05:10:06.809Z [INFO] server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 Address:c9b91bba-dc30-e04b-fb6e-30317aaae7f0} {Suffrage:Voter ID:server-78458f2b-7852-b1e6-0e47-696b8d2b327a Address:78458f2b-7852-b1e6-0e47-696b8d2b327a} {Suffrage:Voter ID:server-7690a8da-ca4b-74d6-905a-84887782f70a Address:7690a8da-ca4b-74d6-905a-84887782f70a} {Suffrage:Voter ID:server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb Address:b1cf2ae1-f267-abda-3212-ac286ecd9fdb} {Suffrage:Voter ID:server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 Address:e5774e4f-7d02-8923-9fa1-7d61c66aeee0} {Suffrage:Voter ID:server-a96c7b98-21ec-611e-8239-dea35c98ee1e Address:a96c7b98-21ec-611e-8239-dea35c98ee1e} {Suffrage:Voter ID:server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff Address:7e15329d-e3f8-73ab-e49c-21e6c404a4ff}]" testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.809Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 Address:c9b91bba-dc30-e04b-fb6e-30317aaae7f0} {Suffrage:Voter ID:server-78458f2b-7852-b1e6-0e47-696b8d2b327a Address:78458f2b-7852-b1e6-0e47-696b8d2b327a} {Suffrage:Voter ID:server-7690a8da-ca4b-74d6-905a-84887782f70a Address:7690a8da-ca4b-74d6-905a-84887782f70a} {Suffrage:Voter ID:server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb Address:b1cf2ae1-f267-abda-3212-ac286ecd9fdb} {Suffrage:Voter ID:server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 Address:e5774e4f-7d02-8923-9fa1-7d61c66aeee0} {Suffrage:Voter ID:server-a96c7b98-21ec-611e-8239-dea35c98ee1e Address:a96c7b98-21ec-611e-8239-dea35c98ee1e} {Suffrage:Voter ID:server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff Address:7e15329d-e3f8-73ab-e49c-21e6c404a4ff}]" testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.809Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 Address:c9b91bba-dc30-e04b-fb6e-30317aaae7f0} {Suffrage:Voter ID:server-78458f2b-7852-b1e6-0e47-696b8d2b327a Address:78458f2b-7852-b1e6-0e47-696b8d2b327a} {Suffrage:Voter ID:server-7690a8da-ca4b-74d6-905a-84887782f70a Address:7690a8da-ca4b-74d6-905a-84887782f70a} {Suffrage:Voter ID:server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb Address:b1cf2ae1-f267-abda-3212-ac286ecd9fdb} {Suffrage:Voter ID:server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 Address:e5774e4f-7d02-8923-9fa1-7d61c66aeee0} {Suffrage:Voter ID:server-a96c7b98-21ec-611e-8239-dea35c98ee1e Address:a96c7b98-21ec-611e-8239-dea35c98ee1e} {Suffrage:Voter ID:server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff Address:7e15329d-e3f8-73ab-e49c-21e6c404a4ff}]" testLoggerAdapter verbose: server-a96c7b98-21ec-611e-8239-dea35c98ee1e: 2024-01-09T05:10:06.809Z [INFO] server-a96c7b98-21ec-611e-8239-dea35c98ee1e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 Address:c9b91bba-dc30-e04b-fb6e-30317aaae7f0} {Suffrage:Voter ID:server-78458f2b-7852-b1e6-0e47-696b8d2b327a Address:78458f2b-7852-b1e6-0e47-696b8d2b327a} {Suffrage:Voter ID:server-7690a8da-ca4b-74d6-905a-84887782f70a Address:7690a8da-ca4b-74d6-905a-84887782f70a} {Suffrage:Voter ID:server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb Address:b1cf2ae1-f267-abda-3212-ac286ecd9fdb} {Suffrage:Voter ID:server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 Address:e5774e4f-7d02-8923-9fa1-7d61c66aeee0} {Suffrage:Voter ID:server-a96c7b98-21ec-611e-8239-dea35c98ee1e Address:a96c7b98-21ec-611e-8239-dea35c98ee1e} {Suffrage:Voter ID:server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff Address:7e15329d-e3f8-73ab-e49c-21e6c404a4ff}]" testing.go:149: server-a96c7b98-21ec-611e-8239-dea35c98ee1e: 2024-01-09T05:10:06.809Z [INFO] server-a96c7b98-21ec-611e-8239-dea35c98ee1e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 Address:c9b91bba-dc30-e04b-fb6e-30317aaae7f0} {Suffrage:Voter ID:server-78458f2b-7852-b1e6-0e47-696b8d2b327a Address:78458f2b-7852-b1e6-0e47-696b8d2b327a} {Suffrage:Voter ID:server-7690a8da-ca4b-74d6-905a-84887782f70a Address:7690a8da-ca4b-74d6-905a-84887782f70a} {Suffrage:Voter ID:server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb Address:b1cf2ae1-f267-abda-3212-ac286ecd9fdb} {Suffrage:Voter ID:server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 Address:e5774e4f-7d02-8923-9fa1-7d61c66aeee0} {Suffrage:Voter ID:server-a96c7b98-21ec-611e-8239-dea35c98ee1e Address:a96c7b98-21ec-611e-8239-dea35c98ee1e} {Suffrage:Voter ID:server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff Address:7e15329d-e3f8-73ab-e49c-21e6c404a4ff}]" testLoggerAdapter verbose: server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff: 2024-01-09T05:10:06.809Z [INFO] server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 Address:c9b91bba-dc30-e04b-fb6e-30317aaae7f0} {Suffrage:Voter ID:server-78458f2b-7852-b1e6-0e47-696b8d2b327a Address:78458f2b-7852-b1e6-0e47-696b8d2b327a} {Suffrage:Voter ID:server-7690a8da-ca4b-74d6-905a-84887782f70a Address:7690a8da-ca4b-74d6-905a-84887782f70a} {Suffrage:Voter ID:server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb Address:b1cf2ae1-f267-abda-3212-ac286ecd9fdb} {Suffrage:Voter ID:server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 Address:e5774e4f-7d02-8923-9fa1-7d61c66aeee0} {Suffrage:Voter ID:server-a96c7b98-21ec-611e-8239-dea35c98ee1e Address:a96c7b98-21ec-611e-8239-dea35c98ee1e} {Suffrage:Voter ID:server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff Address:7e15329d-e3f8-73ab-e49c-21e6c404a4ff}]" testing.go:149: server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff: 2024-01-09T05:10:06.809Z [INFO] server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 Address:c9b91bba-dc30-e04b-fb6e-30317aaae7f0} {Suffrage:Voter ID:server-78458f2b-7852-b1e6-0e47-696b8d2b327a Address:78458f2b-7852-b1e6-0e47-696b8d2b327a} {Suffrage:Voter ID:server-7690a8da-ca4b-74d6-905a-84887782f70a Address:7690a8da-ca4b-74d6-905a-84887782f70a} {Suffrage:Voter ID:server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb Address:b1cf2ae1-f267-abda-3212-ac286ecd9fdb} {Suffrage:Voter ID:server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 Address:e5774e4f-7d02-8923-9fa1-7d61c66aeee0} {Suffrage:Voter ID:server-a96c7b98-21ec-611e-8239-dea35c98ee1e Address:a96c7b98-21ec-611e-8239-dea35c98ee1e} {Suffrage:Voter ID:server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff Address:7e15329d-e3f8-73ab-e49c-21e6c404a4ff}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:06.809Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:10:06.809Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:06.809Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: entering follower state: follower="Node at c9b91bba-dc30-e04b-fb6e-30317aaae7f0 [Follower]" leader= testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:06.809Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: entering follower state: follower="Node at c9b91bba-dc30-e04b-fb6e-30317aaae7f0 [Follower]" leader= testLoggerAdapter verbose: server-78458f2b-7852-b1e6-0e47-696b8d2b327a: 2024-01-09T05:10:06.809Z [INFO] server-78458f2b-7852-b1e6-0e47-696b8d2b327a: entering follower state: follower="Node at 78458f2b-7852-b1e6-0e47-696b8d2b327a [Follower]" leader= testing.go:149: server-78458f2b-7852-b1e6-0e47-696b8d2b327a: 2024-01-09T05:10:06.809Z [INFO] server-78458f2b-7852-b1e6-0e47-696b8d2b327a: entering follower state: follower="Node at 78458f2b-7852-b1e6-0e47-696b8d2b327a [Follower]" leader= testLoggerAdapter verbose: server-7690a8da-ca4b-74d6-905a-84887782f70a: 2024-01-09T05:10:06.809Z [INFO] server-7690a8da-ca4b-74d6-905a-84887782f70a: entering follower state: follower="Node at 7690a8da-ca4b-74d6-905a-84887782f70a [Follower]" leader= testing.go:149: server-7690a8da-ca4b-74d6-905a-84887782f70a: 2024-01-09T05:10:06.809Z [INFO] server-7690a8da-ca4b-74d6-905a-84887782f70a: entering follower state: follower="Node at 7690a8da-ca4b-74d6-905a-84887782f70a [Follower]" leader= testLoggerAdapter verbose: server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb: 2024-01-09T05:10:06.810Z [INFO] server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb: entering follower state: follower="Node at b1cf2ae1-f267-abda-3212-ac286ecd9fdb [Follower]" leader= testing.go:149: server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb: 2024-01-09T05:10:06.810Z [INFO] server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb: entering follower state: follower="Node at b1cf2ae1-f267-abda-3212-ac286ecd9fdb [Follower]" leader= testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.810Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: entering follower state: follower="Node at e5774e4f-7d02-8923-9fa1-7d61c66aeee0 [Follower]" leader= testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.810Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: entering follower state: follower="Node at e5774e4f-7d02-8923-9fa1-7d61c66aeee0 [Follower]" leader= testLoggerAdapter verbose: server-a96c7b98-21ec-611e-8239-dea35c98ee1e: 2024-01-09T05:10:06.810Z [INFO] server-a96c7b98-21ec-611e-8239-dea35c98ee1e: entering follower state: follower="Node at a96c7b98-21ec-611e-8239-dea35c98ee1e [Follower]" leader= testing.go:149: server-a96c7b98-21ec-611e-8239-dea35c98ee1e: 2024-01-09T05:10:06.810Z [INFO] server-a96c7b98-21ec-611e-8239-dea35c98ee1e: entering follower state: follower="Node at a96c7b98-21ec-611e-8239-dea35c98ee1e [Follower]" leader= testLoggerAdapter verbose: server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff: 2024-01-09T05:10:06.810Z [INFO] server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff: entering follower state: follower="Node at 7e15329d-e3f8-73ab-e49c-21e6c404a4ff [Follower]" leader= testing.go:149: server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff: 2024-01-09T05:10:06.810Z [INFO] server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff: entering follower state: follower="Node at 7e15329d-e3f8-73ab-e49c-21e6c404a4ff [Follower]" leader= testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.862Z [WARN] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.862Z [WARN] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: entering candidate state: node="Node at e5774e4f-7d02-8923-9fa1-7d61c66aeee0 [Candidate]" term=2 testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: entering candidate state: node="Node at e5774e4f-7d02-8923-9fa1-7d61c66aeee0 [Candidate]" term=2 testLoggerAdapter verbose: server-78458f2b-7852-b1e6-0e47-696b8d2b327a: 2024-01-09T05:10:06.863Z [WARN] server-78458f2b-7852-b1e6-0e47-696b8d2b327a: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-78458f2b-7852-b1e6-0e47-696b8d2b327a: 2024-01-09T05:10:06.863Z [WARN] server-78458f2b-7852-b1e6-0e47-696b8d2b327a: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-78458f2b-7852-b1e6-0e47-696b8d2b327a: 2024-01-09T05:10:06.863Z [INFO] server-78458f2b-7852-b1e6-0e47-696b8d2b327a: entering candidate state: node="Node at 78458f2b-7852-b1e6-0e47-696b8d2b327a [Candidate]" term=2 testing.go:149: server-78458f2b-7852-b1e6-0e47-696b8d2b327a: 2024-01-09T05:10:06.863Z [INFO] server-78458f2b-7852-b1e6-0e47-696b8d2b327a: entering candidate state: node="Node at 78458f2b-7852-b1e6-0e47-696b8d2b327a [Candidate]" term=2 testLoggerAdapter verbose: server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff: 2024-01-09T05:10:06.863Z [INFO] server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff: duplicate requestVote for same term: term=2 testing.go:149: server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff: 2024-01-09T05:10:06.863Z [INFO] server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-78458f2b-7852-b1e6-0e47-696b8d2b327a: 2024-01-09T05:10:06.863Z [INFO] server-78458f2b-7852-b1e6-0e47-696b8d2b327a: duplicate requestVote for same term: term=2 testing.go:149: server-78458f2b-7852-b1e6-0e47-696b8d2b327a: 2024-01-09T05:10:06.863Z [INFO] server-78458f2b-7852-b1e6-0e47-696b8d2b327a: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: election won: tally=4 testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: election won: tally=4 testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: entering leader state: leader="Node at e5774e4f-7d02-8923-9fa1-7d61c66aeee0 [Leader]" testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: entering leader state: leader="Node at e5774e4f-7d02-8923-9fa1-7d61c66aeee0 [Leader]" testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: added peer, starting replication: peer=server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: added peer, starting replication: peer=server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: added peer, starting replication: peer=server-78458f2b-7852-b1e6-0e47-696b8d2b327a testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: added peer, starting replication: peer=server-78458f2b-7852-b1e6-0e47-696b8d2b327a testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: added peer, starting replication: peer=server-7690a8da-ca4b-74d6-905a-84887782f70a testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: added peer, starting replication: peer=server-7690a8da-ca4b-74d6-905a-84887782f70a testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: added peer, starting replication: peer=server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: added peer, starting replication: peer=server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: added peer, starting replication: peer=server-a96c7b98-21ec-611e-8239-dea35c98ee1e testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: added peer, starting replication: peer=server-a96c7b98-21ec-611e-8239-dea35c98ee1e testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: added peer, starting replication: peer=server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: added peer, starting replication: peer=server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: pipelining replication: peer="{Voter server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff 7e15329d-e3f8-73ab-e49c-21e6c404a4ff}" testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.863Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: pipelining replication: peer="{Voter server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff 7e15329d-e3f8-73ab-e49c-21e6c404a4ff}" testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:06.863Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: duplicate requestVote for same term: term=2 testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:06.863Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-7690a8da-ca4b-74d6-905a-84887782f70a: 2024-01-09T05:10:06.863Z [INFO] server-7690a8da-ca4b-74d6-905a-84887782f70a: duplicate requestVote for same term: term=2 testing.go:149: server-7690a8da-ca4b-74d6-905a-84887782f70a: 2024-01-09T05:10:06.863Z [INFO] server-7690a8da-ca4b-74d6-905a-84887782f70a: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb: 2024-01-09T05:10:06.863Z [INFO] server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb: duplicate requestVote for same term: term=2 testing.go:149: server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb: 2024-01-09T05:10:06.863Z [INFO] server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.864Z [WARN] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: rejecting vote request since we have a leader: from=78458f2b-7852-b1e6-0e47-696b8d2b327a leader=e5774e4f-7d02-8923-9fa1-7d61c66aeee0 testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.864Z [WARN] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: rejecting vote request since we have a leader: from=78458f2b-7852-b1e6-0e47-696b8d2b327a leader=e5774e4f-7d02-8923-9fa1-7d61c66aeee0 testLoggerAdapter verbose: server-a96c7b98-21ec-611e-8239-dea35c98ee1e: 2024-01-09T05:10:06.864Z [INFO] server-a96c7b98-21ec-611e-8239-dea35c98ee1e: duplicate requestVote for same term: term=2 testing.go:149: server-a96c7b98-21ec-611e-8239-dea35c98ee1e: 2024-01-09T05:10:06.864Z [INFO] server-a96c7b98-21ec-611e-8239-dea35c98ee1e: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.864Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: pipelining replication: peer="{Voter server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 c9b91bba-dc30-e04b-fb6e-30317aaae7f0}" testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.864Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: pipelining replication: peer="{Voter server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 c9b91bba-dc30-e04b-fb6e-30317aaae7f0}" testLoggerAdapter verbose: server-78458f2b-7852-b1e6-0e47-696b8d2b327a: 2024-01-09T05:10:06.864Z [INFO] server-78458f2b-7852-b1e6-0e47-696b8d2b327a: entering follower state: follower="Node at 78458f2b-7852-b1e6-0e47-696b8d2b327a [Follower]" leader=e5774e4f-7d02-8923-9fa1-7d61c66aeee0 testing.go:149: server-78458f2b-7852-b1e6-0e47-696b8d2b327a: 2024-01-09T05:10:06.864Z [INFO] server-78458f2b-7852-b1e6-0e47-696b8d2b327a: entering follower state: follower="Node at 78458f2b-7852-b1e6-0e47-696b8d2b327a [Follower]" leader=e5774e4f-7d02-8923-9fa1-7d61c66aeee0 testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.864Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: pipelining replication: peer="{Voter server-78458f2b-7852-b1e6-0e47-696b8d2b327a 78458f2b-7852-b1e6-0e47-696b8d2b327a}" testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.864Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: pipelining replication: peer="{Voter server-78458f2b-7852-b1e6-0e47-696b8d2b327a 78458f2b-7852-b1e6-0e47-696b8d2b327a}" testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.864Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: pipelining replication: peer="{Voter server-7690a8da-ca4b-74d6-905a-84887782f70a 7690a8da-ca4b-74d6-905a-84887782f70a}" testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.864Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: pipelining replication: peer="{Voter server-7690a8da-ca4b-74d6-905a-84887782f70a 7690a8da-ca4b-74d6-905a-84887782f70a}" testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.864Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: pipelining replication: peer="{Voter server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb b1cf2ae1-f267-abda-3212-ac286ecd9fdb}" testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.864Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: pipelining replication: peer="{Voter server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb b1cf2ae1-f267-abda-3212-ac286ecd9fdb}" testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.864Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: pipelining replication: peer="{Voter server-a96c7b98-21ec-611e-8239-dea35c98ee1e a96c7b98-21ec-611e-8239-dea35c98ee1e}" testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:06.864Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: pipelining replication: peer="{Voter server-a96c7b98-21ec-611e-8239-dea35c98ee1e a96c7b98-21ec-611e-8239-dea35c98ee1e}" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:06.971Z [INFO] stable state for Leader reached at 2024-01-09 05:10:06.864652799 +0000 UTC m=+18.117703871 (1 nodes), 54.803395ms from start of poll, 56.060659ms from cluster start. Timeout at 2024-01-09 05:10:06.971145069 +0000 UTC m=+18.224196037, 106.492166ms after stability testing.go:149: cluster: 2024-01-09T05:10:06.971Z [INFO] stable state for Leader reached at 2024-01-09 05:10:06.864652799 +0000 UTC m=+18.117703871 (1 nodes), 54.803395ms from start of poll, 56.060659ms from cluster start. Timeout at 2024-01-09 05:10:06.971145069 +0000 UTC m=+18.224196037, 106.492166ms after stability testLoggerAdapter verbose: cluster: 2024-01-09T05:10:06.971Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-01-09T05:10:06.971Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2024-01-09T05:10:07.078Z [INFO] stable state for Follower reached at 2024-01-09 05:10:06.971274314 +0000 UTC m=+18.224325289 (6 nodes), 3.036µs from start of poll, 162.682077ms from cluster start. Timeout at 2024-01-09 05:10:07.078938838 +0000 UTC m=+18.331989960, 107.664671ms after stability testing.go:149: cluster: 2024-01-09T05:10:07.078Z [INFO] stable state for Follower reached at 2024-01-09 05:10:06.971274314 +0000 UTC m=+18.224325289 (6 nodes), 3.036µs from start of poll, 162.682077ms from cluster start. Timeout at 2024-01-09 05:10:07.078938838 +0000 UTC m=+18.331989960, 107.664671ms after stability testLoggerAdapter verbose: cluster: 2024-01-09T05:10:07.079Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:10:07.079Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-01-09T05:10:07.184Z [INFO] stable state for Leader reached at 2024-01-09 05:10:07.079085435 +0000 UTC m=+18.332136436 (1 nodes), 2.79µs from start of poll, 270.493224ms from cluster start. Timeout at 2024-01-09 05:10:07.184875429 +0000 UTC m=+18.437926459, 105.790023ms after stability testing.go:149: cluster: 2024-01-09T05:10:07.184Z [INFO] stable state for Leader reached at 2024-01-09 05:10:07.079085435 +0000 UTC m=+18.332136436 (1 nodes), 2.79µs from start of poll, 270.493224ms from cluster start. Timeout at 2024-01-09 05:10:07.184875429 +0000 UTC m=+18.437926459, 105.790023ms after stability testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.185Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: entering candidate state: node="Node at c9b91bba-dc30-e04b-fb6e-30317aaae7f0 [Candidate]" term=3 testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.185Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: entering candidate state: node="Node at c9b91bba-dc30-e04b-fb6e-30317aaae7f0 [Candidate]" term=3 testLoggerAdapter verbose: cluster: 2024-01-09T05:10:07.185Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:10:07.185Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.185Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: election won: tally=4 testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.185Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: election won: tally=4 testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.185Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: entering leader state: leader="Node at c9b91bba-dc30-e04b-fb6e-30317aaae7f0 [Leader]" testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.185Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: entering leader state: leader="Node at c9b91bba-dc30-e04b-fb6e-30317aaae7f0 [Leader]" testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.185Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: added peer, starting replication: peer=server-78458f2b-7852-b1e6-0e47-696b8d2b327a testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.185Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: added peer, starting replication: peer=server-78458f2b-7852-b1e6-0e47-696b8d2b327a testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.185Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: added peer, starting replication: peer=server-7690a8da-ca4b-74d6-905a-84887782f70a testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.185Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: added peer, starting replication: peer=server-7690a8da-ca4b-74d6-905a-84887782f70a testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.185Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: added peer, starting replication: peer=server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.185Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: added peer, starting replication: peer=server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.185Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: added peer, starting replication: peer=server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.185Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: added peer, starting replication: peer=server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.185Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: added peer, starting replication: peer=server-a96c7b98-21ec-611e-8239-dea35c98ee1e testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.185Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: added peer, starting replication: peer=server-a96c7b98-21ec-611e-8239-dea35c98ee1e testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.185Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: added peer, starting replication: peer=server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:07.186Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: entering follower state: follower="Node at e5774e4f-7d02-8923-9fa1-7d61c66aeee0 [Follower]" leader= testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.185Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: added peer, starting replication: peer=server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:07.186Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: entering follower state: follower="Node at e5774e4f-7d02-8923-9fa1-7d61c66aeee0 [Follower]" leader= testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.191Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: pipelining replication: peer="{Voter server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 e5774e4f-7d02-8923-9fa1-7d61c66aeee0}" testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.191Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: pipelining replication: peer="{Voter server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 e5774e4f-7d02-8923-9fa1-7d61c66aeee0}" testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.186Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: pipelining replication: peer="{Voter server-7690a8da-ca4b-74d6-905a-84887782f70a 7690a8da-ca4b-74d6-905a-84887782f70a}" testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.186Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: pipelining replication: peer="{Voter server-7690a8da-ca4b-74d6-905a-84887782f70a 7690a8da-ca4b-74d6-905a-84887782f70a}" testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.191Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: pipelining replication: peer="{Voter server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff 7e15329d-e3f8-73ab-e49c-21e6c404a4ff}" testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.191Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: pipelining replication: peer="{Voter server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff 7e15329d-e3f8-73ab-e49c-21e6c404a4ff}" testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:07.186Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: aborting pipeline replication: peer="{Voter server-78458f2b-7852-b1e6-0e47-696b8d2b327a 78458f2b-7852-b1e6-0e47-696b8d2b327a}" testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:07.186Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: aborting pipeline replication: peer="{Voter server-78458f2b-7852-b1e6-0e47-696b8d2b327a 78458f2b-7852-b1e6-0e47-696b8d2b327a}" testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.186Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: pipelining replication: peer="{Voter server-78458f2b-7852-b1e6-0e47-696b8d2b327a 78458f2b-7852-b1e6-0e47-696b8d2b327a}" testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.186Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: pipelining replication: peer="{Voter server-78458f2b-7852-b1e6-0e47-696b8d2b327a 78458f2b-7852-b1e6-0e47-696b8d2b327a}" testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:07.186Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: aborting pipeline replication: peer="{Voter server-7690a8da-ca4b-74d6-905a-84887782f70a 7690a8da-ca4b-74d6-905a-84887782f70a}" testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:07.186Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: aborting pipeline replication: peer="{Voter server-7690a8da-ca4b-74d6-905a-84887782f70a 7690a8da-ca4b-74d6-905a-84887782f70a}" testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.186Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: pipelining replication: peer="{Voter server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb b1cf2ae1-f267-abda-3212-ac286ecd9fdb}" testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.186Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: pipelining replication: peer="{Voter server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb b1cf2ae1-f267-abda-3212-ac286ecd9fdb}" testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:07.186Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: aborting pipeline replication: peer="{Voter server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb b1cf2ae1-f267-abda-3212-ac286ecd9fdb}" testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:07.186Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: aborting pipeline replication: peer="{Voter server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb b1cf2ae1-f267-abda-3212-ac286ecd9fdb}" testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.191Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: pipelining replication: peer="{Voter server-a96c7b98-21ec-611e-8239-dea35c98ee1e a96c7b98-21ec-611e-8239-dea35c98ee1e}" testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.191Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: pipelining replication: peer="{Voter server-a96c7b98-21ec-611e-8239-dea35c98ee1e a96c7b98-21ec-611e-8239-dea35c98ee1e}" testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:07.186Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: aborting pipeline replication: peer="{Voter server-a96c7b98-21ec-611e-8239-dea35c98ee1e a96c7b98-21ec-611e-8239-dea35c98ee1e}" testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:07.186Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: aborting pipeline replication: peer="{Voter server-a96c7b98-21ec-611e-8239-dea35c98ee1e a96c7b98-21ec-611e-8239-dea35c98ee1e}" testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:07.186Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: aborting pipeline replication: peer="{Voter server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff 7e15329d-e3f8-73ab-e49c-21e6c404a4ff}" testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:07.186Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: aborting pipeline replication: peer="{Voter server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff 7e15329d-e3f8-73ab-e49c-21e6c404a4ff}" testLoggerAdapter verbose: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:07.186Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: aborting pipeline replication: peer="{Voter server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 c9b91bba-dc30-e04b-fb6e-30317aaae7f0}" testing.go:149: server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: 2024-01-09T05:10:07.186Z [INFO] server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0: aborting pipeline replication: peer="{Voter server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0 c9b91bba-dc30-e04b-fb6e-30317aaae7f0}" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:07.295Z [INFO] stable state for Leader reached at 2024-01-09 05:10:07.186429062 +0000 UTC m=+18.439480108 (1 nodes), 1.040734ms from start of poll, 377.836896ms from cluster start. Timeout at 2024-01-09 05:10:07.295900801 +0000 UTC m=+18.548951868, 109.47176ms after stability testing.go:149: cluster: 2024-01-09T05:10:07.295Z [INFO] stable state for Leader reached at 2024-01-09 05:10:07.186429062 +0000 UTC m=+18.439480108 (1 nodes), 1.040734ms from start of poll, 377.836896ms from cluster start. Timeout at 2024-01-09 05:10:07.295900801 +0000 UTC m=+18.548951868, 109.47176ms after stability testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.296Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: aborting pipeline replication: peer="{Voter server-7690a8da-ca4b-74d6-905a-84887782f70a 7690a8da-ca4b-74d6-905a-84887782f70a}" testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.296Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: aborting pipeline replication: peer="{Voter server-7690a8da-ca4b-74d6-905a-84887782f70a 7690a8da-ca4b-74d6-905a-84887782f70a}" testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.296Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: aborting pipeline replication: peer="{Voter server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb b1cf2ae1-f267-abda-3212-ac286ecd9fdb}" testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.296Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: aborting pipeline replication: peer="{Voter server-b1cf2ae1-f267-abda-3212-ac286ecd9fdb b1cf2ae1-f267-abda-3212-ac286ecd9fdb}" testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.296Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: aborting pipeline replication: peer="{Voter server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 e5774e4f-7d02-8923-9fa1-7d61c66aeee0}" testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.296Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: aborting pipeline replication: peer="{Voter server-e5774e4f-7d02-8923-9fa1-7d61c66aeee0 e5774e4f-7d02-8923-9fa1-7d61c66aeee0}" testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.296Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: aborting pipeline replication: peer="{Voter server-a96c7b98-21ec-611e-8239-dea35c98ee1e a96c7b98-21ec-611e-8239-dea35c98ee1e}" testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.296Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: aborting pipeline replication: peer="{Voter server-a96c7b98-21ec-611e-8239-dea35c98ee1e a96c7b98-21ec-611e-8239-dea35c98ee1e}" testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.296Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: aborting pipeline replication: peer="{Voter server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff 7e15329d-e3f8-73ab-e49c-21e6c404a4ff}" testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.296Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: aborting pipeline replication: peer="{Voter server-7e15329d-e3f8-73ab-e49c-21e6c404a4ff 7e15329d-e3f8-73ab-e49c-21e6c404a4ff}" testLoggerAdapter verbose: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.296Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: aborting pipeline replication: peer="{Voter server-78458f2b-7852-b1e6-0e47-696b8d2b327a 78458f2b-7852-b1e6-0e47-696b8d2b327a}" testing.go:149: server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: 2024-01-09T05:10:07.296Z [INFO] server-c9b91bba-dc30-e04b-fb6e-30317aaae7f0: aborting pipeline replication: peer="{Voter server-78458f2b-7852-b1e6-0e47-696b8d2b327a 78458f2b-7852-b1e6-0e47-696b8d2b327a}" --- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.49s) === RUN TestRaft_LeadershipTransferToInvalidID testLoggerAdapter verbose: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.298Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5534b285-4693-2480-b4c7-7d20e1baeaea Address:5534b285-4693-2480-b4c7-7d20e1baeaea} {Suffrage:Voter ID:server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab Address:c7e6cb42-46e1-712e-ad2f-90b187bc6aab} {Suffrage:Voter ID:server-e54fb9e6-99fd-c048-711f-571f96b294bc Address:e54fb9e6-99fd-c048-711f-571f96b294bc}]" testing.go:149: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.298Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5534b285-4693-2480-b4c7-7d20e1baeaea Address:5534b285-4693-2480-b4c7-7d20e1baeaea} {Suffrage:Voter ID:server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab Address:c7e6cb42-46e1-712e-ad2f-90b187bc6aab} {Suffrage:Voter ID:server-e54fb9e6-99fd-c048-711f-571f96b294bc Address:e54fb9e6-99fd-c048-711f-571f96b294bc}]" testLoggerAdapter verbose: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.299Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: entering follower state: follower="Node at 5534b285-4693-2480-b4c7-7d20e1baeaea [Follower]" leader= testing.go:149: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.299Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: entering follower state: follower="Node at 5534b285-4693-2480-b4c7-7d20e1baeaea [Follower]" leader= testLoggerAdapter verbose: server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab: 2024-01-09T05:10:07.298Z [INFO] server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5534b285-4693-2480-b4c7-7d20e1baeaea Address:5534b285-4693-2480-b4c7-7d20e1baeaea} {Suffrage:Voter ID:server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab Address:c7e6cb42-46e1-712e-ad2f-90b187bc6aab} {Suffrage:Voter ID:server-e54fb9e6-99fd-c048-711f-571f96b294bc Address:e54fb9e6-99fd-c048-711f-571f96b294bc}]" testing.go:149: server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab: 2024-01-09T05:10:07.298Z [INFO] server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5534b285-4693-2480-b4c7-7d20e1baeaea Address:5534b285-4693-2480-b4c7-7d20e1baeaea} {Suffrage:Voter ID:server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab Address:c7e6cb42-46e1-712e-ad2f-90b187bc6aab} {Suffrage:Voter ID:server-e54fb9e6-99fd-c048-711f-571f96b294bc Address:e54fb9e6-99fd-c048-711f-571f96b294bc}]" testLoggerAdapter verbose: server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab: 2024-01-09T05:10:07.299Z [INFO] server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab: entering follower state: follower="Node at c7e6cb42-46e1-712e-ad2f-90b187bc6aab [Follower]" leader= testing.go:149: server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab: 2024-01-09T05:10:07.299Z [INFO] server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab: entering follower state: follower="Node at c7e6cb42-46e1-712e-ad2f-90b187bc6aab [Follower]" leader= testLoggerAdapter verbose: server-e54fb9e6-99fd-c048-711f-571f96b294bc: 2024-01-09T05:10:07.299Z [INFO] server-e54fb9e6-99fd-c048-711f-571f96b294bc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5534b285-4693-2480-b4c7-7d20e1baeaea Address:5534b285-4693-2480-b4c7-7d20e1baeaea} {Suffrage:Voter ID:server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab Address:c7e6cb42-46e1-712e-ad2f-90b187bc6aab} {Suffrage:Voter ID:server-e54fb9e6-99fd-c048-711f-571f96b294bc Address:e54fb9e6-99fd-c048-711f-571f96b294bc}]" testing.go:149: server-e54fb9e6-99fd-c048-711f-571f96b294bc: 2024-01-09T05:10:07.299Z [INFO] server-e54fb9e6-99fd-c048-711f-571f96b294bc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5534b285-4693-2480-b4c7-7d20e1baeaea Address:5534b285-4693-2480-b4c7-7d20e1baeaea} {Suffrage:Voter ID:server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab Address:c7e6cb42-46e1-712e-ad2f-90b187bc6aab} {Suffrage:Voter ID:server-e54fb9e6-99fd-c048-711f-571f96b294bc Address:e54fb9e6-99fd-c048-711f-571f96b294bc}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:07.299Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:10:07.299Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-e54fb9e6-99fd-c048-711f-571f96b294bc: 2024-01-09T05:10:07.299Z [INFO] server-e54fb9e6-99fd-c048-711f-571f96b294bc: entering follower state: follower="Node at e54fb9e6-99fd-c048-711f-571f96b294bc [Follower]" leader= testing.go:149: server-e54fb9e6-99fd-c048-711f-571f96b294bc: 2024-01-09T05:10:07.299Z [INFO] server-e54fb9e6-99fd-c048-711f-571f96b294bc: entering follower state: follower="Node at e54fb9e6-99fd-c048-711f-571f96b294bc [Follower]" leader= testLoggerAdapter verbose: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.351Z [WARN] server-5534b285-4693-2480-b4c7-7d20e1baeaea: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.351Z [WARN] server-5534b285-4693-2480-b4c7-7d20e1baeaea: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.351Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: entering candidate state: node="Node at 5534b285-4693-2480-b4c7-7d20e1baeaea [Candidate]" term=2 testing.go:149: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.351Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: entering candidate state: node="Node at 5534b285-4693-2480-b4c7-7d20e1baeaea [Candidate]" term=2 testLoggerAdapter verbose: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.352Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: election won: tally=2 testing.go:149: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.352Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: election won: tally=2 testLoggerAdapter verbose: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.352Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: entering leader state: leader="Node at 5534b285-4693-2480-b4c7-7d20e1baeaea [Leader]" testing.go:149: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.352Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: entering leader state: leader="Node at 5534b285-4693-2480-b4c7-7d20e1baeaea [Leader]" testLoggerAdapter verbose: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.352Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: added peer, starting replication: peer=server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab testing.go:149: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.352Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: added peer, starting replication: peer=server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab testLoggerAdapter verbose: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.352Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: added peer, starting replication: peer=server-e54fb9e6-99fd-c048-711f-571f96b294bc testing.go:149: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.352Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: added peer, starting replication: peer=server-e54fb9e6-99fd-c048-711f-571f96b294bc testLoggerAdapter verbose: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.352Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: pipelining replication: peer="{Voter server-e54fb9e6-99fd-c048-711f-571f96b294bc e54fb9e6-99fd-c048-711f-571f96b294bc}" testing.go:149: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.352Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: pipelining replication: peer="{Voter server-e54fb9e6-99fd-c048-711f-571f96b294bc e54fb9e6-99fd-c048-711f-571f96b294bc}" testLoggerAdapter verbose: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.352Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: pipelining replication: peer="{Voter server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab c7e6cb42-46e1-712e-ad2f-90b187bc6aab}" testing.go:149: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.352Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: pipelining replication: peer="{Voter server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab c7e6cb42-46e1-712e-ad2f-90b187bc6aab}" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:07.457Z [INFO] stable state for Leader reached at 2024-01-09 05:10:07.352539423 +0000 UTC m=+18.605590523 (1 nodes), 52.947522ms from start of poll, 54.005282ms from cluster start. Timeout at 2024-01-09 05:10:07.457655023 +0000 UTC m=+18.710706104, 105.115581ms after stability testing.go:149: cluster: 2024-01-09T05:10:07.457Z [INFO] stable state for Leader reached at 2024-01-09 05:10:07.352539423 +0000 UTC m=+18.605590523 (1 nodes), 52.947522ms from start of poll, 54.005282ms from cluster start. Timeout at 2024-01-09 05:10:07.457655023 +0000 UTC m=+18.710706104, 105.115581ms after stability testLoggerAdapter verbose: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.457Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: aborting pipeline replication: peer="{Voter server-e54fb9e6-99fd-c048-711f-571f96b294bc e54fb9e6-99fd-c048-711f-571f96b294bc}" testing.go:149: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.457Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: aborting pipeline replication: peer="{Voter server-e54fb9e6-99fd-c048-711f-571f96b294bc e54fb9e6-99fd-c048-711f-571f96b294bc}" testLoggerAdapter verbose: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.458Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: aborting pipeline replication: peer="{Voter server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab c7e6cb42-46e1-712e-ad2f-90b187bc6aab}" testing.go:149: server-5534b285-4693-2480-b4c7-7d20e1baeaea: 2024-01-09T05:10:07.458Z [INFO] server-5534b285-4693-2480-b4c7-7d20e1baeaea: aborting pipeline replication: peer="{Voter server-c7e6cb42-46e1-712e-ad2f-90b187bc6aab c7e6cb42-46e1-712e-ad2f-90b187bc6aab}" --- PASS: TestRaft_LeadershipTransferToInvalidID (0.16s) === RUN TestRaft_LeadershipTransferToInvalidAddress testLoggerAdapter verbose: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.459Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9 Address:1f32a78b-3df2-ac0b-fcdd-5eebf03488a9} {Suffrage:Voter ID:server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e Address:05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e} {Suffrage:Voter ID:server-8e44a857-c59b-1833-4366-2673c370165a Address:8e44a857-c59b-1833-4366-2673c370165a}]" testing.go:149: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.459Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9 Address:1f32a78b-3df2-ac0b-fcdd-5eebf03488a9} {Suffrage:Voter ID:server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e Address:05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e} {Suffrage:Voter ID:server-8e44a857-c59b-1833-4366-2673c370165a Address:8e44a857-c59b-1833-4366-2673c370165a}]" testLoggerAdapter verbose: server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e: 2024-01-09T05:10:07.459Z [INFO] server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9 Address:1f32a78b-3df2-ac0b-fcdd-5eebf03488a9} {Suffrage:Voter ID:server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e Address:05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e} {Suffrage:Voter ID:server-8e44a857-c59b-1833-4366-2673c370165a Address:8e44a857-c59b-1833-4366-2673c370165a}]" testing.go:149: server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e: 2024-01-09T05:10:07.459Z [INFO] server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9 Address:1f32a78b-3df2-ac0b-fcdd-5eebf03488a9} {Suffrage:Voter ID:server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e Address:05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e} {Suffrage:Voter ID:server-8e44a857-c59b-1833-4366-2673c370165a Address:8e44a857-c59b-1833-4366-2673c370165a}]" testLoggerAdapter verbose: server-8e44a857-c59b-1833-4366-2673c370165a: 2024-01-09T05:10:07.460Z [INFO] server-8e44a857-c59b-1833-4366-2673c370165a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9 Address:1f32a78b-3df2-ac0b-fcdd-5eebf03488a9} {Suffrage:Voter ID:server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e Address:05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e} {Suffrage:Voter ID:server-8e44a857-c59b-1833-4366-2673c370165a Address:8e44a857-c59b-1833-4366-2673c370165a}]" testing.go:149: server-8e44a857-c59b-1833-4366-2673c370165a: 2024-01-09T05:10:07.460Z [INFO] server-8e44a857-c59b-1833-4366-2673c370165a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9 Address:1f32a78b-3df2-ac0b-fcdd-5eebf03488a9} {Suffrage:Voter ID:server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e Address:05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e} {Suffrage:Voter ID:server-8e44a857-c59b-1833-4366-2673c370165a Address:8e44a857-c59b-1833-4366-2673c370165a}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:07.460Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-01-09T05:10:07.460Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.460Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: entering follower state: follower="Node at 1f32a78b-3df2-ac0b-fcdd-5eebf03488a9 [Follower]" leader= testing.go:149: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.460Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: entering follower state: follower="Node at 1f32a78b-3df2-ac0b-fcdd-5eebf03488a9 [Follower]" leader= testLoggerAdapter verbose: server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e: 2024-01-09T05:10:07.460Z [INFO] server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e: entering follower state: follower="Node at 05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e [Follower]" leader= testing.go:149: server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e: 2024-01-09T05:10:07.460Z [INFO] server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e: entering follower state: follower="Node at 05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e [Follower]" leader= testLoggerAdapter verbose: server-8e44a857-c59b-1833-4366-2673c370165a: 2024-01-09T05:10:07.460Z [INFO] server-8e44a857-c59b-1833-4366-2673c370165a: entering follower state: follower="Node at 8e44a857-c59b-1833-4366-2673c370165a [Follower]" leader= testing.go:149: server-8e44a857-c59b-1833-4366-2673c370165a: 2024-01-09T05:10:07.460Z [INFO] server-8e44a857-c59b-1833-4366-2673c370165a: entering follower state: follower="Node at 8e44a857-c59b-1833-4366-2673c370165a [Follower]" leader= testLoggerAdapter verbose: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.524Z [WARN] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.524Z [WARN] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.524Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: entering candidate state: node="Node at 1f32a78b-3df2-ac0b-fcdd-5eebf03488a9 [Candidate]" term=2 testing.go:149: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.524Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: entering candidate state: node="Node at 1f32a78b-3df2-ac0b-fcdd-5eebf03488a9 [Candidate]" term=2 testLoggerAdapter verbose: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.524Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: election won: tally=2 testing.go:149: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.524Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: election won: tally=2 testLoggerAdapter verbose: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.524Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: entering leader state: leader="Node at 1f32a78b-3df2-ac0b-fcdd-5eebf03488a9 [Leader]" testing.go:149: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.524Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: entering leader state: leader="Node at 1f32a78b-3df2-ac0b-fcdd-5eebf03488a9 [Leader]" testLoggerAdapter verbose: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.524Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: added peer, starting replication: peer=server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e testing.go:149: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.524Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: added peer, starting replication: peer=server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e testLoggerAdapter verbose: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.524Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: added peer, starting replication: peer=server-8e44a857-c59b-1833-4366-2673c370165a testing.go:149: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.524Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: added peer, starting replication: peer=server-8e44a857-c59b-1833-4366-2673c370165a testLoggerAdapter verbose: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.524Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: pipelining replication: peer="{Voter server-8e44a857-c59b-1833-4366-2673c370165a 8e44a857-c59b-1833-4366-2673c370165a}" testing.go:149: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.524Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: pipelining replication: peer="{Voter server-8e44a857-c59b-1833-4366-2673c370165a 8e44a857-c59b-1833-4366-2673c370165a}" testLoggerAdapter verbose: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.524Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: pipelining replication: peer="{Voter server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e 05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e}" testing.go:149: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.524Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: pipelining replication: peer="{Voter server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e 05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e}" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:07.629Z [INFO] stable state for Follower reached at 2024-01-09 05:10:07.52482094 +0000 UTC m=+18.777871995 (2 nodes), 64.711139ms from start of poll, 65.27158ms from cluster start. Timeout at 2024-01-09 05:10:07.629920362 +0000 UTC m=+18.882971392, 105.099397ms after stability testing.go:149: cluster: 2024-01-09T05:10:07.629Z [INFO] stable state for Follower reached at 2024-01-09 05:10:07.52482094 +0000 UTC m=+18.777871995 (2 nodes), 64.711139ms from start of poll, 65.27158ms from cluster start. Timeout at 2024-01-09 05:10:07.629920362 +0000 UTC m=+18.882971392, 105.099397ms after stability testLoggerAdapter verbose: cluster: 2024-01-09T05:10:07.630Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:10:07.630Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-01-09T05:10:07.735Z [INFO] stable state for Leader reached at 2024-01-09 05:10:07.630078372 +0000 UTC m=+18.883129399 (1 nodes), 2.443µs from start of poll, 170.528984ms from cluster start. Timeout at 2024-01-09 05:10:07.735262215 +0000 UTC m=+18.988313307, 105.183908ms after stability testing.go:149: cluster: 2024-01-09T05:10:07.735Z [INFO] stable state for Leader reached at 2024-01-09 05:10:07.630078372 +0000 UTC m=+18.883129399 (1 nodes), 2.443µs from start of poll, 170.528984ms from cluster start. Timeout at 2024-01-09 05:10:07.735262215 +0000 UTC m=+18.988313307, 105.183908ms after stability testLoggerAdapter verbose: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.735Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: aborting pipeline replication: peer="{Voter server-8e44a857-c59b-1833-4366-2673c370165a 8e44a857-c59b-1833-4366-2673c370165a}" testing.go:149: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.735Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: aborting pipeline replication: peer="{Voter server-8e44a857-c59b-1833-4366-2673c370165a 8e44a857-c59b-1833-4366-2673c370165a}" testLoggerAdapter verbose: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.735Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: aborting pipeline replication: peer="{Voter server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e 05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e}" testing.go:149: server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: 2024-01-09T05:10:07.735Z [INFO] server-1f32a78b-3df2-ac0b-fcdd-5eebf03488a9: aborting pipeline replication: peer="{Voter server-05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e 05fff4e6-4dab-03ae-5d0e-e1c56ffdf06e}" --- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.28s) === RUN TestRaft_LeadershipTransferToBehindServer testLoggerAdapter verbose: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:07.737Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0189b842-19da-c595-de45-ddabfae2712d Address:0189b842-19da-c595-de45-ddabfae2712d} {Suffrage:Voter ID:server-3006c307-f604-7a72-ba1c-eeb8283298b2 Address:3006c307-f604-7a72-ba1c-eeb8283298b2} {Suffrage:Voter ID:server-688e118c-253e-992c-61f7-a58ca9e008bd Address:688e118c-253e-992c-61f7-a58ca9e008bd}]" testing.go:149: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:07.737Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0189b842-19da-c595-de45-ddabfae2712d Address:0189b842-19da-c595-de45-ddabfae2712d} {Suffrage:Voter ID:server-3006c307-f604-7a72-ba1c-eeb8283298b2 Address:3006c307-f604-7a72-ba1c-eeb8283298b2} {Suffrage:Voter ID:server-688e118c-253e-992c-61f7-a58ca9e008bd Address:688e118c-253e-992c-61f7-a58ca9e008bd}]" testLoggerAdapter verbose: server-3006c307-f604-7a72-ba1c-eeb8283298b2: 2024-01-09T05:10:07.737Z [INFO] server-3006c307-f604-7a72-ba1c-eeb8283298b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0189b842-19da-c595-de45-ddabfae2712d Address:0189b842-19da-c595-de45-ddabfae2712d} {Suffrage:Voter ID:server-3006c307-f604-7a72-ba1c-eeb8283298b2 Address:3006c307-f604-7a72-ba1c-eeb8283298b2} {Suffrage:Voter ID:server-688e118c-253e-992c-61f7-a58ca9e008bd Address:688e118c-253e-992c-61f7-a58ca9e008bd}]" testing.go:149: server-3006c307-f604-7a72-ba1c-eeb8283298b2: 2024-01-09T05:10:07.737Z [INFO] server-3006c307-f604-7a72-ba1c-eeb8283298b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0189b842-19da-c595-de45-ddabfae2712d Address:0189b842-19da-c595-de45-ddabfae2712d} {Suffrage:Voter ID:server-3006c307-f604-7a72-ba1c-eeb8283298b2 Address:3006c307-f604-7a72-ba1c-eeb8283298b2} {Suffrage:Voter ID:server-688e118c-253e-992c-61f7-a58ca9e008bd Address:688e118c-253e-992c-61f7-a58ca9e008bd}]" testLoggerAdapter verbose: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.737Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0189b842-19da-c595-de45-ddabfae2712d Address:0189b842-19da-c595-de45-ddabfae2712d} {Suffrage:Voter ID:server-3006c307-f604-7a72-ba1c-eeb8283298b2 Address:3006c307-f604-7a72-ba1c-eeb8283298b2} {Suffrage:Voter ID:server-688e118c-253e-992c-61f7-a58ca9e008bd Address:688e118c-253e-992c-61f7-a58ca9e008bd}]" testing.go:149: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.737Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0189b842-19da-c595-de45-ddabfae2712d Address:0189b842-19da-c595-de45-ddabfae2712d} {Suffrage:Voter ID:server-3006c307-f604-7a72-ba1c-eeb8283298b2 Address:3006c307-f604-7a72-ba1c-eeb8283298b2} {Suffrage:Voter ID:server-688e118c-253e-992c-61f7-a58ca9e008bd Address:688e118c-253e-992c-61f7-a58ca9e008bd}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:07.737Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:10:07.737Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:07.737Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: entering follower state: follower="Node at 0189b842-19da-c595-de45-ddabfae2712d [Follower]" leader= testing.go:149: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:07.737Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: entering follower state: follower="Node at 0189b842-19da-c595-de45-ddabfae2712d [Follower]" leader= testLoggerAdapter verbose: server-3006c307-f604-7a72-ba1c-eeb8283298b2: 2024-01-09T05:10:07.737Z [INFO] server-3006c307-f604-7a72-ba1c-eeb8283298b2: entering follower state: follower="Node at 3006c307-f604-7a72-ba1c-eeb8283298b2 [Follower]" leader= testing.go:149: server-3006c307-f604-7a72-ba1c-eeb8283298b2: 2024-01-09T05:10:07.737Z [INFO] server-3006c307-f604-7a72-ba1c-eeb8283298b2: entering follower state: follower="Node at 3006c307-f604-7a72-ba1c-eeb8283298b2 [Follower]" leader= testLoggerAdapter verbose: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.737Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: entering follower state: follower="Node at 688e118c-253e-992c-61f7-a58ca9e008bd [Follower]" leader= testing.go:149: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.737Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: entering follower state: follower="Node at 688e118c-253e-992c-61f7-a58ca9e008bd [Follower]" leader= testLoggerAdapter verbose: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.788Z [WARN] server-688e118c-253e-992c-61f7-a58ca9e008bd: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.788Z [WARN] server-688e118c-253e-992c-61f7-a58ca9e008bd: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.788Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: entering candidate state: node="Node at 688e118c-253e-992c-61f7-a58ca9e008bd [Candidate]" term=2 testing.go:149: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.788Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: entering candidate state: node="Node at 688e118c-253e-992c-61f7-a58ca9e008bd [Candidate]" term=2 testLoggerAdapter verbose: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.788Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: election won: tally=2 testing.go:149: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.788Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: election won: tally=2 testLoggerAdapter verbose: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.788Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: entering leader state: leader="Node at 688e118c-253e-992c-61f7-a58ca9e008bd [Leader]" testing.go:149: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.788Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: entering leader state: leader="Node at 688e118c-253e-992c-61f7-a58ca9e008bd [Leader]" testLoggerAdapter verbose: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.788Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: added peer, starting replication: peer=server-0189b842-19da-c595-de45-ddabfae2712d testing.go:149: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.788Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: added peer, starting replication: peer=server-0189b842-19da-c595-de45-ddabfae2712d testLoggerAdapter verbose: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.788Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: added peer, starting replication: peer=server-3006c307-f604-7a72-ba1c-eeb8283298b2 testing.go:149: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.788Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: added peer, starting replication: peer=server-3006c307-f604-7a72-ba1c-eeb8283298b2 testLoggerAdapter verbose: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.788Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: pipelining replication: peer="{Voter server-3006c307-f604-7a72-ba1c-eeb8283298b2 3006c307-f604-7a72-ba1c-eeb8283298b2}" testing.go:149: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.788Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: pipelining replication: peer="{Voter server-3006c307-f604-7a72-ba1c-eeb8283298b2 3006c307-f604-7a72-ba1c-eeb8283298b2}" testLoggerAdapter verbose: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.789Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: pipelining replication: peer="{Voter server-0189b842-19da-c595-de45-ddabfae2712d 0189b842-19da-c595-de45-ddabfae2712d}" testing.go:149: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:07.789Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: pipelining replication: peer="{Voter server-0189b842-19da-c595-de45-ddabfae2712d 0189b842-19da-c595-de45-ddabfae2712d}" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:07.894Z [INFO] stable state for Leader reached at 2024-01-09 05:10:07.789103217 +0000 UTC m=+19.042154346 (1 nodes), 51.670716ms from start of poll, 52.175154ms from cluster start. Timeout at 2024-01-09 05:10:07.894230459 +0000 UTC m=+19.147281589, 105.127243ms after stability testing.go:149: cluster: 2024-01-09T05:10:07.894Z [INFO] stable state for Leader reached at 2024-01-09 05:10:07.789103217 +0000 UTC m=+19.042154346 (1 nodes), 51.670716ms from start of poll, 52.175154ms from cluster start. Timeout at 2024-01-09 05:10:07.894230459 +0000 UTC m=+19.147281589, 105.127243ms after stability testLoggerAdapter verbose: cluster: 2024-01-09T05:10:07.894Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-01-09T05:10:07.894Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2024-01-09T05:10:08.000Z [INFO] stable state for Follower reached at 2024-01-09 05:10:07.895793067 +0000 UTC m=+19.148844060 (2 nodes), 2.976µs from start of poll, 158.864868ms from cluster start. Timeout at 2024-01-09 05:10:08.000816822 +0000 UTC m=+19.253867842, 105.023782ms after stability testing.go:149: cluster: 2024-01-09T05:10:08.000Z [INFO] stable state for Follower reached at 2024-01-09 05:10:07.895793067 +0000 UTC m=+19.148844060 (2 nodes), 2.976µs from start of poll, 158.864868ms from cluster start. Timeout at 2024-01-09 05:10:08.000816822 +0000 UTC m=+19.253867842, 105.023782ms after stability testLoggerAdapter verbose: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:08.051Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: entering candidate state: node="Node at 0189b842-19da-c595-de45-ddabfae2712d [Candidate]" term=3 testing.go:149: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:08.051Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: entering candidate state: node="Node at 0189b842-19da-c595-de45-ddabfae2712d [Candidate]" term=3 testLoggerAdapter verbose: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:08.052Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: entering follower state: follower="Node at 688e118c-253e-992c-61f7-a58ca9e008bd [Follower]" leader= testing.go:149: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:08.052Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: entering follower state: follower="Node at 688e118c-253e-992c-61f7-a58ca9e008bd [Follower]" leader= testLoggerAdapter verbose: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:08.052Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: aborting pipeline replication: peer="{Voter server-3006c307-f604-7a72-ba1c-eeb8283298b2 3006c307-f604-7a72-ba1c-eeb8283298b2}" testLoggerAdapter verbose: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:08.052Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: election won: tally=2 testing.go:149: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:08.052Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: aborting pipeline replication: peer="{Voter server-3006c307-f604-7a72-ba1c-eeb8283298b2 3006c307-f604-7a72-ba1c-eeb8283298b2}" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:08.052Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:08.052Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: aborting pipeline replication: peer="{Voter server-0189b842-19da-c595-de45-ddabfae2712d 0189b842-19da-c595-de45-ddabfae2712d}" testing.go:149: cluster: 2024-01-09T05:10:08.052Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-688e118c-253e-992c-61f7-a58ca9e008bd: 2024-01-09T05:10:08.052Z [INFO] server-688e118c-253e-992c-61f7-a58ca9e008bd: aborting pipeline replication: peer="{Voter server-0189b842-19da-c595-de45-ddabfae2712d 0189b842-19da-c595-de45-ddabfae2712d}" testing.go:149: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:08.052Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: election won: tally=2 testLoggerAdapter verbose: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:08.052Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: entering leader state: leader="Node at 0189b842-19da-c595-de45-ddabfae2712d [Leader]" testing.go:149: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:08.052Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: entering leader state: leader="Node at 0189b842-19da-c595-de45-ddabfae2712d [Leader]" testLoggerAdapter verbose: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:08.052Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: added peer, starting replication: peer=server-3006c307-f604-7a72-ba1c-eeb8283298b2 testing.go:149: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:08.052Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: added peer, starting replication: peer=server-3006c307-f604-7a72-ba1c-eeb8283298b2 testLoggerAdapter verbose: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:08.052Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: added peer, starting replication: peer=server-688e118c-253e-992c-61f7-a58ca9e008bd testing.go:149: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:08.052Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: added peer, starting replication: peer=server-688e118c-253e-992c-61f7-a58ca9e008bd testLoggerAdapter verbose: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:08.052Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: pipelining replication: peer="{Voter server-688e118c-253e-992c-61f7-a58ca9e008bd 688e118c-253e-992c-61f7-a58ca9e008bd}" testing.go:149: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:08.052Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: pipelining replication: peer="{Voter server-688e118c-253e-992c-61f7-a58ca9e008bd 688e118c-253e-992c-61f7-a58ca9e008bd}" testLoggerAdapter verbose: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:08.052Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: pipelining replication: peer="{Voter server-3006c307-f604-7a72-ba1c-eeb8283298b2 3006c307-f604-7a72-ba1c-eeb8283298b2}" testing.go:149: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:08.052Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: pipelining replication: peer="{Voter server-3006c307-f604-7a72-ba1c-eeb8283298b2 3006c307-f604-7a72-ba1c-eeb8283298b2}" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:08.157Z [INFO] stable state for Leader reached at 2024-01-09 05:10:08.052448031 +0000 UTC m=+19.305499248 (1 nodes), 325.177µs from start of poll, 315.520056ms from cluster start. Timeout at 2024-01-09 05:10:08.157579275 +0000 UTC m=+19.410630342, 105.131094ms after stability testing.go:149: cluster: 2024-01-09T05:10:08.157Z [INFO] stable state for Leader reached at 2024-01-09 05:10:08.052448031 +0000 UTC m=+19.305499248 (1 nodes), 325.177µs from start of poll, 315.520056ms from cluster start. Timeout at 2024-01-09 05:10:08.157579275 +0000 UTC m=+19.410630342, 105.131094ms after stability testLoggerAdapter verbose: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:08.157Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: aborting pipeline replication: peer="{Voter server-688e118c-253e-992c-61f7-a58ca9e008bd 688e118c-253e-992c-61f7-a58ca9e008bd}" testing.go:149: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:08.157Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: aborting pipeline replication: peer="{Voter server-688e118c-253e-992c-61f7-a58ca9e008bd 688e118c-253e-992c-61f7-a58ca9e008bd}" testLoggerAdapter verbose: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:08.157Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: aborting pipeline replication: peer="{Voter server-3006c307-f604-7a72-ba1c-eeb8283298b2 3006c307-f604-7a72-ba1c-eeb8283298b2}" testing.go:149: server-0189b842-19da-c595-de45-ddabfae2712d: 2024-01-09T05:10:08.157Z [INFO] server-0189b842-19da-c595-de45-ddabfae2712d: aborting pipeline replication: peer="{Voter server-3006c307-f604-7a72-ba1c-eeb8283298b2 3006c307-f604-7a72-ba1c-eeb8283298b2}" --- PASS: TestRaft_LeadershipTransferToBehindServer (0.42s) === RUN TestRaft_LeadershipTransferToItself testLoggerAdapter verbose: server-30b2f93d-5285-84e7-551b-a48d711d6846: 2024-01-09T05:10:08.159Z [INFO] server-30b2f93d-5285-84e7-551b-a48d711d6846: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30b2f93d-5285-84e7-551b-a48d711d6846 Address:30b2f93d-5285-84e7-551b-a48d711d6846} {Suffrage:Voter ID:server-039a2132-e84d-30df-b4b5-6980889216e6 Address:039a2132-e84d-30df-b4b5-6980889216e6} {Suffrage:Voter ID:server-ce43a170-4e3e-1419-64a7-6b5df072e9b0 Address:ce43a170-4e3e-1419-64a7-6b5df072e9b0}]" testing.go:149: server-30b2f93d-5285-84e7-551b-a48d711d6846: 2024-01-09T05:10:08.159Z [INFO] server-30b2f93d-5285-84e7-551b-a48d711d6846: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30b2f93d-5285-84e7-551b-a48d711d6846 Address:30b2f93d-5285-84e7-551b-a48d711d6846} {Suffrage:Voter ID:server-039a2132-e84d-30df-b4b5-6980889216e6 Address:039a2132-e84d-30df-b4b5-6980889216e6} {Suffrage:Voter ID:server-ce43a170-4e3e-1419-64a7-6b5df072e9b0 Address:ce43a170-4e3e-1419-64a7-6b5df072e9b0}]" testLoggerAdapter verbose: server-039a2132-e84d-30df-b4b5-6980889216e6: 2024-01-09T05:10:08.159Z [INFO] server-039a2132-e84d-30df-b4b5-6980889216e6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30b2f93d-5285-84e7-551b-a48d711d6846 Address:30b2f93d-5285-84e7-551b-a48d711d6846} {Suffrage:Voter ID:server-039a2132-e84d-30df-b4b5-6980889216e6 Address:039a2132-e84d-30df-b4b5-6980889216e6} {Suffrage:Voter ID:server-ce43a170-4e3e-1419-64a7-6b5df072e9b0 Address:ce43a170-4e3e-1419-64a7-6b5df072e9b0}]" testing.go:149: server-039a2132-e84d-30df-b4b5-6980889216e6: 2024-01-09T05:10:08.159Z [INFO] server-039a2132-e84d-30df-b4b5-6980889216e6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30b2f93d-5285-84e7-551b-a48d711d6846 Address:30b2f93d-5285-84e7-551b-a48d711d6846} {Suffrage:Voter ID:server-039a2132-e84d-30df-b4b5-6980889216e6 Address:039a2132-e84d-30df-b4b5-6980889216e6} {Suffrage:Voter ID:server-ce43a170-4e3e-1419-64a7-6b5df072e9b0 Address:ce43a170-4e3e-1419-64a7-6b5df072e9b0}]" testLoggerAdapter verbose: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.160Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30b2f93d-5285-84e7-551b-a48d711d6846 Address:30b2f93d-5285-84e7-551b-a48d711d6846} {Suffrage:Voter ID:server-039a2132-e84d-30df-b4b5-6980889216e6 Address:039a2132-e84d-30df-b4b5-6980889216e6} {Suffrage:Voter ID:server-ce43a170-4e3e-1419-64a7-6b5df072e9b0 Address:ce43a170-4e3e-1419-64a7-6b5df072e9b0}]" testing.go:149: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.160Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30b2f93d-5285-84e7-551b-a48d711d6846 Address:30b2f93d-5285-84e7-551b-a48d711d6846} {Suffrage:Voter ID:server-039a2132-e84d-30df-b4b5-6980889216e6 Address:039a2132-e84d-30df-b4b5-6980889216e6} {Suffrage:Voter ID:server-ce43a170-4e3e-1419-64a7-6b5df072e9b0 Address:ce43a170-4e3e-1419-64a7-6b5df072e9b0}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:08.160Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:10:08.160Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-30b2f93d-5285-84e7-551b-a48d711d6846: 2024-01-09T05:10:08.160Z [INFO] server-30b2f93d-5285-84e7-551b-a48d711d6846: entering follower state: follower="Node at 30b2f93d-5285-84e7-551b-a48d711d6846 [Follower]" leader= testing.go:149: server-30b2f93d-5285-84e7-551b-a48d711d6846: 2024-01-09T05:10:08.160Z [INFO] server-30b2f93d-5285-84e7-551b-a48d711d6846: entering follower state: follower="Node at 30b2f93d-5285-84e7-551b-a48d711d6846 [Follower]" leader= testLoggerAdapter verbose: server-039a2132-e84d-30df-b4b5-6980889216e6: 2024-01-09T05:10:08.160Z [INFO] server-039a2132-e84d-30df-b4b5-6980889216e6: entering follower state: follower="Node at 039a2132-e84d-30df-b4b5-6980889216e6 [Follower]" leader= testing.go:149: server-039a2132-e84d-30df-b4b5-6980889216e6: 2024-01-09T05:10:08.160Z [INFO] server-039a2132-e84d-30df-b4b5-6980889216e6: entering follower state: follower="Node at 039a2132-e84d-30df-b4b5-6980889216e6 [Follower]" leader= testLoggerAdapter verbose: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.160Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: entering follower state: follower="Node at ce43a170-4e3e-1419-64a7-6b5df072e9b0 [Follower]" leader= testing.go:149: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.160Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: entering follower state: follower="Node at ce43a170-4e3e-1419-64a7-6b5df072e9b0 [Follower]" leader= testLoggerAdapter verbose: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.231Z [WARN] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.231Z [WARN] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.231Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: entering candidate state: node="Node at ce43a170-4e3e-1419-64a7-6b5df072e9b0 [Candidate]" term=2 testing.go:149: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.231Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: entering candidate state: node="Node at ce43a170-4e3e-1419-64a7-6b5df072e9b0 [Candidate]" term=2 testLoggerAdapter verbose: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.231Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: election won: tally=2 testing.go:149: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.231Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: election won: tally=2 testLoggerAdapter verbose: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.231Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: entering leader state: leader="Node at ce43a170-4e3e-1419-64a7-6b5df072e9b0 [Leader]" testing.go:149: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.231Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: entering leader state: leader="Node at ce43a170-4e3e-1419-64a7-6b5df072e9b0 [Leader]" testLoggerAdapter verbose: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.231Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: added peer, starting replication: peer=server-30b2f93d-5285-84e7-551b-a48d711d6846 testing.go:149: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.231Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: added peer, starting replication: peer=server-30b2f93d-5285-84e7-551b-a48d711d6846 testLoggerAdapter verbose: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.232Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: added peer, starting replication: peer=server-039a2132-e84d-30df-b4b5-6980889216e6 testing.go:149: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.232Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: added peer, starting replication: peer=server-039a2132-e84d-30df-b4b5-6980889216e6 testLoggerAdapter verbose: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.232Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: pipelining replication: peer="{Voter server-039a2132-e84d-30df-b4b5-6980889216e6 039a2132-e84d-30df-b4b5-6980889216e6}" testing.go:149: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.232Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: pipelining replication: peer="{Voter server-039a2132-e84d-30df-b4b5-6980889216e6 039a2132-e84d-30df-b4b5-6980889216e6}" testLoggerAdapter verbose: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.232Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: pipelining replication: peer="{Voter server-30b2f93d-5285-84e7-551b-a48d711d6846 30b2f93d-5285-84e7-551b-a48d711d6846}" testing.go:149: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.232Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: pipelining replication: peer="{Voter server-30b2f93d-5285-84e7-551b-a48d711d6846 30b2f93d-5285-84e7-551b-a48d711d6846}" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:08.337Z [INFO] stable state for Leader reached at 2024-01-09 05:10:08.232348768 +0000 UTC m=+19.485399956 (1 nodes), 72.229917ms from start of poll, 72.7115ms from cluster start. Timeout at 2024-01-09 05:10:08.337443471 +0000 UTC m=+19.590494499, 105.094543ms after stability testing.go:149: cluster: 2024-01-09T05:10:08.337Z [INFO] stable state for Leader reached at 2024-01-09 05:10:08.232348768 +0000 UTC m=+19.485399956 (1 nodes), 72.229917ms from start of poll, 72.7115ms from cluster start. Timeout at 2024-01-09 05:10:08.337443471 +0000 UTC m=+19.590494499, 105.094543ms after stability testLoggerAdapter verbose: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.337Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: cannot transfer leadership to itself testing.go:149: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.337Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: cannot transfer leadership to itself testLoggerAdapter verbose: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.337Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: aborting pipeline replication: peer="{Voter server-039a2132-e84d-30df-b4b5-6980889216e6 039a2132-e84d-30df-b4b5-6980889216e6}" testing.go:149: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.337Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: aborting pipeline replication: peer="{Voter server-039a2132-e84d-30df-b4b5-6980889216e6 039a2132-e84d-30df-b4b5-6980889216e6}" testLoggerAdapter verbose: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.337Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: aborting pipeline replication: peer="{Voter server-30b2f93d-5285-84e7-551b-a48d711d6846 30b2f93d-5285-84e7-551b-a48d711d6846}" testing.go:149: server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: 2024-01-09T05:10:08.337Z [INFO] server-ce43a170-4e3e-1419-64a7-6b5df072e9b0: aborting pipeline replication: peer="{Voter server-30b2f93d-5285-84e7-551b-a48d711d6846 30b2f93d-5285-84e7-551b-a48d711d6846}" --- PASS: TestRaft_LeadershipTransferToItself (0.18s) === RUN TestRaft_LeadershipTransferLeaderRejectsClientRequests raft_test.go:2064: Disabled unreliable test --- SKIP: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.00s) === RUN TestRaft_LeadershipTransferLeaderReplicationTimeout testLoggerAdapter verbose: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.339Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080 Address:b30cf42e-8ebf-2b2c-c15a-b8848dd9b080} {Suffrage:Voter ID:server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810 Address:3c3001ba-c8f3-6e64-1597-3e71fc8e3810} {Suffrage:Voter ID:server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf Address:fa31d492-ba2b-6ed1-5267-4d4af9584eaf}]" testing.go:149: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.339Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080 Address:b30cf42e-8ebf-2b2c-c15a-b8848dd9b080} {Suffrage:Voter ID:server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810 Address:3c3001ba-c8f3-6e64-1597-3e71fc8e3810} {Suffrage:Voter ID:server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf Address:fa31d492-ba2b-6ed1-5267-4d4af9584eaf}]" testLoggerAdapter verbose: server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810: 2024-01-09T05:10:08.339Z [INFO] server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080 Address:b30cf42e-8ebf-2b2c-c15a-b8848dd9b080} {Suffrage:Voter ID:server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810 Address:3c3001ba-c8f3-6e64-1597-3e71fc8e3810} {Suffrage:Voter ID:server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf Address:fa31d492-ba2b-6ed1-5267-4d4af9584eaf}]" testing.go:149: server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810: 2024-01-09T05:10:08.339Z [INFO] server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080 Address:b30cf42e-8ebf-2b2c-c15a-b8848dd9b080} {Suffrage:Voter ID:server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810 Address:3c3001ba-c8f3-6e64-1597-3e71fc8e3810} {Suffrage:Voter ID:server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf Address:fa31d492-ba2b-6ed1-5267-4d4af9584eaf}]" testLoggerAdapter verbose: server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf: 2024-01-09T05:10:08.339Z [INFO] server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080 Address:b30cf42e-8ebf-2b2c-c15a-b8848dd9b080} {Suffrage:Voter ID:server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810 Address:3c3001ba-c8f3-6e64-1597-3e71fc8e3810} {Suffrage:Voter ID:server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf Address:fa31d492-ba2b-6ed1-5267-4d4af9584eaf}]" testing.go:149: server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf: 2024-01-09T05:10:08.339Z [INFO] server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080 Address:b30cf42e-8ebf-2b2c-c15a-b8848dd9b080} {Suffrage:Voter ID:server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810 Address:3c3001ba-c8f3-6e64-1597-3e71fc8e3810} {Suffrage:Voter ID:server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf Address:fa31d492-ba2b-6ed1-5267-4d4af9584eaf}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:08.339Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:10:08.339Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.339Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: entering follower state: follower="Node at b30cf42e-8ebf-2b2c-c15a-b8848dd9b080 [Follower]" leader= testing.go:149: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.339Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: entering follower state: follower="Node at b30cf42e-8ebf-2b2c-c15a-b8848dd9b080 [Follower]" leader= testLoggerAdapter verbose: server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810: 2024-01-09T05:10:08.339Z [INFO] server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810: entering follower state: follower="Node at 3c3001ba-c8f3-6e64-1597-3e71fc8e3810 [Follower]" leader= testing.go:149: server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810: 2024-01-09T05:10:08.339Z [INFO] server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810: entering follower state: follower="Node at 3c3001ba-c8f3-6e64-1597-3e71fc8e3810 [Follower]" leader= testLoggerAdapter verbose: server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf: 2024-01-09T05:10:08.339Z [INFO] server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf: entering follower state: follower="Node at fa31d492-ba2b-6ed1-5267-4d4af9584eaf [Follower]" leader= testing.go:149: server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf: 2024-01-09T05:10:08.339Z [INFO] server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf: entering follower state: follower="Node at fa31d492-ba2b-6ed1-5267-4d4af9584eaf [Follower]" leader= testLoggerAdapter verbose: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.404Z [WARN] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.404Z [WARN] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.404Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: entering candidate state: node="Node at b30cf42e-8ebf-2b2c-c15a-b8848dd9b080 [Candidate]" term=2 testing.go:149: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.404Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: entering candidate state: node="Node at b30cf42e-8ebf-2b2c-c15a-b8848dd9b080 [Candidate]" term=2 testLoggerAdapter verbose: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.404Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: election won: tally=2 testing.go:149: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.404Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: election won: tally=2 testLoggerAdapter verbose: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.404Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: entering leader state: leader="Node at b30cf42e-8ebf-2b2c-c15a-b8848dd9b080 [Leader]" testing.go:149: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.404Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: entering leader state: leader="Node at b30cf42e-8ebf-2b2c-c15a-b8848dd9b080 [Leader]" testLoggerAdapter verbose: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.404Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: added peer, starting replication: peer=server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810 testing.go:149: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.404Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: added peer, starting replication: peer=server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810 testLoggerAdapter verbose: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.405Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: added peer, starting replication: peer=server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf testing.go:149: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.405Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: added peer, starting replication: peer=server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf testLoggerAdapter verbose: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.405Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: pipelining replication: peer="{Voter server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf fa31d492-ba2b-6ed1-5267-4d4af9584eaf}" testing.go:149: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.405Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: pipelining replication: peer="{Voter server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf fa31d492-ba2b-6ed1-5267-4d4af9584eaf}" testLoggerAdapter verbose: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.405Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: pipelining replication: peer="{Voter server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810 3c3001ba-c8f3-6e64-1597-3e71fc8e3810}" testing.go:149: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:08.405Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: pipelining replication: peer="{Voter server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810 3c3001ba-c8f3-6e64-1597-3e71fc8e3810}" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:08.514Z [INFO] stable state for Leader reached at 2024-01-09 05:10:08.405333527 +0000 UTC m=+19.658384553 (1 nodes), 65.765646ms from start of poll, 66.238579ms from cluster start. Timeout at 2024-01-09 05:10:08.513943353 +0000 UTC m=+19.766994375, 108.609822ms after stability testing.go:149: cluster: 2024-01-09T05:10:08.514Z [INFO] stable state for Leader reached at 2024-01-09 05:10:08.405333527 +0000 UTC m=+19.658384553 (1 nodes), 65.765646ms from start of poll, 66.238579ms from cluster start. Timeout at 2024-01-09 05:10:08.513943353 +0000 UTC m=+19.766994375, 108.609822ms after stability testLoggerAdapter verbose: cluster: 2024-01-09T05:10:08.514Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-01-09T05:10:08.514Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2024-01-09T05:10:08.619Z [INFO] stable state for Follower reached at 2024-01-09 05:10:08.514110956 +0000 UTC m=+19.767161959 (2 nodes), 2.505µs from start of poll, 175.015985ms from cluster start. Timeout at 2024-01-09 05:10:08.619219897 +0000 UTC m=+19.872270951, 105.108992ms after stability testing.go:149: cluster: 2024-01-09T05:10:08.619Z [INFO] stable state for Follower reached at 2024-01-09 05:10:08.514110956 +0000 UTC m=+19.767161959 (2 nodes), 2.505µs from start of poll, 175.015985ms from cluster start. Timeout at 2024-01-09 05:10:08.619219897 +0000 UTC m=+19.872270951, 105.108992ms after stability testLoggerAdapter verbose: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:09.125Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: aborting pipeline replication: peer="{Voter server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf fa31d492-ba2b-6ed1-5267-4d4af9584eaf}" testing.go:149: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:09.125Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: aborting pipeline replication: peer="{Voter server-fa31d492-ba2b-6ed1-5267-4d4af9584eaf fa31d492-ba2b-6ed1-5267-4d4af9584eaf}" testLoggerAdapter verbose: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:09.125Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: aborting pipeline replication: peer="{Voter server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810 3c3001ba-c8f3-6e64-1597-3e71fc8e3810}" testing.go:149: server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: 2024-01-09T05:10:09.125Z [INFO] server-b30cf42e-8ebf-2b2c-c15a-b8848dd9b080: aborting pipeline replication: peer="{Voter server-3c3001ba-c8f3-6e64-1597-3e71fc8e3810 3c3001ba-c8f3-6e64-1597-3e71fc8e3810}" --- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (0.79s) === RUN TestRaft_LeadershipTransferStopRightAway --- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s) === RUN TestRaft_GetConfigurationNoBootstrap testLoggerAdapter verbose: server-3b39c6a3-adad-a25e-1c81-808a4efb2831: 2024-01-09T05:10:09.127Z [INFO] server-3b39c6a3-adad-a25e-1c81-808a4efb2831: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3b39c6a3-adad-a25e-1c81-808a4efb2831 Address:3b39c6a3-adad-a25e-1c81-808a4efb2831} {Suffrage:Voter ID:server-6be33f16-9a5a-4a59-346e-9f2d05853da4 Address:6be33f16-9a5a-4a59-346e-9f2d05853da4}]" testing.go:149: server-3b39c6a3-adad-a25e-1c81-808a4efb2831: 2024-01-09T05:10:09.127Z [INFO] server-3b39c6a3-adad-a25e-1c81-808a4efb2831: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3b39c6a3-adad-a25e-1c81-808a4efb2831 Address:3b39c6a3-adad-a25e-1c81-808a4efb2831} {Suffrage:Voter ID:server-6be33f16-9a5a-4a59-346e-9f2d05853da4 Address:6be33f16-9a5a-4a59-346e-9f2d05853da4}]" testLoggerAdapter verbose: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.127Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3b39c6a3-adad-a25e-1c81-808a4efb2831 Address:3b39c6a3-adad-a25e-1c81-808a4efb2831} {Suffrage:Voter ID:server-6be33f16-9a5a-4a59-346e-9f2d05853da4 Address:6be33f16-9a5a-4a59-346e-9f2d05853da4}]" testing.go:149: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.127Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3b39c6a3-adad-a25e-1c81-808a4efb2831 Address:3b39c6a3-adad-a25e-1c81-808a4efb2831} {Suffrage:Voter ID:server-6be33f16-9a5a-4a59-346e-9f2d05853da4 Address:6be33f16-9a5a-4a59-346e-9f2d05853da4}]" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:09.127Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-01-09T05:10:09.127Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-3b39c6a3-adad-a25e-1c81-808a4efb2831: 2024-01-09T05:10:09.127Z [INFO] server-3b39c6a3-adad-a25e-1c81-808a4efb2831: entering follower state: follower="Node at 3b39c6a3-adad-a25e-1c81-808a4efb2831 [Follower]" leader= testing.go:149: server-3b39c6a3-adad-a25e-1c81-808a4efb2831: 2024-01-09T05:10:09.127Z [INFO] server-3b39c6a3-adad-a25e-1c81-808a4efb2831: entering follower state: follower="Node at 3b39c6a3-adad-a25e-1c81-808a4efb2831 [Follower]" leader= testLoggerAdapter verbose: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.127Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: entering follower state: follower="Node at 6be33f16-9a5a-4a59-346e-9f2d05853da4 [Follower]" leader= testing.go:149: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.127Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: entering follower state: follower="Node at 6be33f16-9a5a-4a59-346e-9f2d05853da4 [Follower]" leader= testLoggerAdapter verbose: server-3b39c6a3-adad-a25e-1c81-808a4efb2831: 2024-01-09T05:10:09.210Z [WARN] server-3b39c6a3-adad-a25e-1c81-808a4efb2831: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-3b39c6a3-adad-a25e-1c81-808a4efb2831: 2024-01-09T05:10:09.210Z [WARN] server-3b39c6a3-adad-a25e-1c81-808a4efb2831: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-3b39c6a3-adad-a25e-1c81-808a4efb2831: 2024-01-09T05:10:09.211Z [INFO] server-3b39c6a3-adad-a25e-1c81-808a4efb2831: entering candidate state: node="Node at 3b39c6a3-adad-a25e-1c81-808a4efb2831 [Candidate]" term=2 testing.go:149: server-3b39c6a3-adad-a25e-1c81-808a4efb2831: 2024-01-09T05:10:09.211Z [INFO] server-3b39c6a3-adad-a25e-1c81-808a4efb2831: entering candidate state: node="Node at 3b39c6a3-adad-a25e-1c81-808a4efb2831 [Candidate]" term=2 testLoggerAdapter verbose: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.211Z [WARN] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.211Z [WARN] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.211Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: entering candidate state: node="Node at 6be33f16-9a5a-4a59-346e-9f2d05853da4 [Candidate]" term=2 testing.go:149: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.211Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: entering candidate state: node="Node at 6be33f16-9a5a-4a59-346e-9f2d05853da4 [Candidate]" term=2 testLoggerAdapter verbose: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.211Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: duplicate requestVote for same term: term=2 testing.go:149: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.211Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-3b39c6a3-adad-a25e-1c81-808a4efb2831: 2024-01-09T05:10:09.211Z [INFO] server-3b39c6a3-adad-a25e-1c81-808a4efb2831: duplicate requestVote for same term: term=2 testing.go:149: server-3b39c6a3-adad-a25e-1c81-808a4efb2831: 2024-01-09T05:10:09.211Z [INFO] server-3b39c6a3-adad-a25e-1c81-808a4efb2831: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.295Z [WARN] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: Election timeout reached, restarting election testing.go:149: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.295Z [WARN] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: Election timeout reached, restarting election testLoggerAdapter verbose: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.295Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: entering candidate state: node="Node at 6be33f16-9a5a-4a59-346e-9f2d05853da4 [Candidate]" term=3 testing.go:149: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.295Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: entering candidate state: node="Node at 6be33f16-9a5a-4a59-346e-9f2d05853da4 [Candidate]" term=3 testLoggerAdapter verbose: server-3b39c6a3-adad-a25e-1c81-808a4efb2831: 2024-01-09T05:10:09.295Z [INFO] server-3b39c6a3-adad-a25e-1c81-808a4efb2831: entering follower state: follower="Node at 3b39c6a3-adad-a25e-1c81-808a4efb2831 [Follower]" leader= testing.go:149: server-3b39c6a3-adad-a25e-1c81-808a4efb2831: 2024-01-09T05:10:09.295Z [INFO] server-3b39c6a3-adad-a25e-1c81-808a4efb2831: entering follower state: follower="Node at 3b39c6a3-adad-a25e-1c81-808a4efb2831 [Follower]" leader= testLoggerAdapter verbose: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.295Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: election won: tally=2 testing.go:149: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.295Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: election won: tally=2 testLoggerAdapter verbose: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.295Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: entering leader state: leader="Node at 6be33f16-9a5a-4a59-346e-9f2d05853da4 [Leader]" testing.go:149: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.295Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: entering leader state: leader="Node at 6be33f16-9a5a-4a59-346e-9f2d05853da4 [Leader]" testLoggerAdapter verbose: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.295Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: added peer, starting replication: peer=server-3b39c6a3-adad-a25e-1c81-808a4efb2831 testing.go:149: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.295Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: added peer, starting replication: peer=server-3b39c6a3-adad-a25e-1c81-808a4efb2831 testLoggerAdapter verbose: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.295Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: pipelining replication: peer="{Voter server-3b39c6a3-adad-a25e-1c81-808a4efb2831 3b39c6a3-adad-a25e-1c81-808a4efb2831}" testing.go:149: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.295Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: pipelining replication: peer="{Voter server-3b39c6a3-adad-a25e-1c81-808a4efb2831 3b39c6a3-adad-a25e-1c81-808a4efb2831}" testLoggerAdapter verbose: cluster: 2024-01-09T05:10:09.409Z [INFO] stable state for Follower reached at 2024-01-09 05:10:09.295662499 +0000 UTC m=+20.548713546 (1 nodes), 167.777211ms from start of poll, 168.244616ms from cluster start. Timeout at 2024-01-09 05:10:09.409853883 +0000 UTC m=+20.662904995, 114.191449ms after stability testing.go:149: cluster: 2024-01-09T05:10:09.409Z [INFO] stable state for Follower reached at 2024-01-09 05:10:09.295662499 +0000 UTC m=+20.548713546 (1 nodes), 167.777211ms from start of poll, 168.244616ms from cluster start. Timeout at 2024-01-09 05:10:09.409853883 +0000 UTC m=+20.662904995, 114.191449ms after stability testLoggerAdapter verbose: cluster: 2024-01-09T05:10:09.409Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-01-09T05:10:09.409Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-01-09T05:10:09.527Z [INFO] stable state for Leader reached at 2024-01-09 05:10:09.410041706 +0000 UTC m=+20.663092804 (1 nodes), 2.58µs from start of poll, 282.623874ms from cluster start. Timeout at 2024-01-09 05:10:09.526937397 +0000 UTC m=+20.779988429, 116.895625ms after stability testing.go:149: cluster: 2024-01-09T05:10:09.527Z [INFO] stable state for Leader reached at 2024-01-09 05:10:09.410041706 +0000 UTC m=+20.663092804 (1 nodes), 2.58µs from start of poll, 282.623874ms from cluster start. Timeout at 2024-01-09 05:10:09.526937397 +0000 UTC m=+20.779988429, 116.895625ms after stability testLoggerAdapter verbose: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.542Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3b39c6a3-adad-a25e-1c81-808a4efb2831 Address:3b39c6a3-adad-a25e-1c81-808a4efb2831} {Suffrage:Voter ID:server-6be33f16-9a5a-4a59-346e-9f2d05853da4 Address:6be33f16-9a5a-4a59-346e-9f2d05853da4}]" testing.go:149: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.542Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3b39c6a3-adad-a25e-1c81-808a4efb2831 Address:3b39c6a3-adad-a25e-1c81-808a4efb2831} {Suffrage:Voter ID:server-6be33f16-9a5a-4a59-346e-9f2d05853da4 Address:6be33f16-9a5a-4a59-346e-9f2d05853da4}]" testLoggerAdapter verbose: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.542Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: aborting pipeline replication: peer="{Voter server-3b39c6a3-adad-a25e-1c81-808a4efb2831 3b39c6a3-adad-a25e-1c81-808a4efb2831}" testing.go:149: server-6be33f16-9a5a-4a59-346e-9f2d05853da4: 2024-01-09T05:10:09.542Z [INFO] server-6be33f16-9a5a-4a59-346e-9f2d05853da4: aborting pipeline replication: peer="{Voter server-3b39c6a3-adad-a25e-1c81-808a4efb2831 3b39c6a3-adad-a25e-1c81-808a4efb2831}" --- PASS: TestRaft_GetConfigurationNoBootstrap (0.42s) === RUN TestTCPTransport_BadAddr --- PASS: TestTCPTransport_BadAddr (0.00s) === RUN TestTCPTransport_WithAdvertise --- PASS: TestTCPTransport_WithAdvertise (0.00s) === RUN TestTransport_StartStop --- PASS: TestTransport_StartStop (0.00s) === RUN TestTransport_AppendEntries --- PASS: TestTransport_AppendEntries (0.00s) === RUN TestTransport_AppendEntriesPipeline --- PASS: TestTransport_AppendEntriesPipeline (0.00s) === RUN TestTransport_RequestVote --- PASS: TestTransport_RequestVote (0.00s) === RUN TestTransport_InstallSnapshot --- PASS: TestTransport_InstallSnapshot (0.00s) === RUN TestTransport_EncodeDecode --- PASS: TestTransport_EncodeDecode (0.00s) === RUN TestRandomTimeout --- PASS: TestRandomTimeout (0.00s) === RUN TestNewSeed --- PASS: TestNewSeed (0.00s) === RUN TestRandomTimeout_NoTime --- PASS: TestRandomTimeout_NoTime (0.00s) === RUN TestMin --- PASS: TestMin (0.00s) === RUN TestMax --- PASS: TestMax (0.00s) === RUN TestGenerateUUID --- PASS: TestGenerateUUID (0.00s) === RUN TestBackoff --- PASS: TestBackoff (0.00s) PASS ok github.com/hashicorp/raft 20.832s ? github.com/hashicorp/raft/bench [no test files] create-stamp debian/debhelper-build-stamp fakeroot debian/rules binary dh binary --buildsystem=golang --with=golang dh_testroot -O--buildsystem=golang dh_prep -O--buildsystem=golang dh_auto_install -O--buildsystem=golang dh_installdocs -O--buildsystem=golang dh_installchangelogs -O--buildsystem=golang dh_installinit -O--buildsystem=golang dh_installsystemduser -O--buildsystem=golang dh_perl -O--buildsystem=golang dh_link -O--buildsystem=golang dh_strip_nondeterminism -O--buildsystem=golang dh_compress -O--buildsystem=golang dh_fixperms -O--buildsystem=golang dh_missing -O--buildsystem=golang dh_installdeb -O--buildsystem=golang dh_golang -O--buildsystem=golang dh_gencontrol -O--buildsystem=golang dpkg-gencontrol: warning: Depends field of package golang-github-hashicorp-raft-dev: substitution variable ${shlibs:Depends} used, but is not defined dh_md5sums -O--buildsystem=golang dh_builddeb -O--buildsystem=golang dpkg-deb: building package 'golang-github-hashicorp-raft-dev' in '../golang-github-hashicorp-raft-dev_1.1.2-1_all.deb'. dpkg-genbuildinfo --build=binary dpkg-genchanges --build=binary >../golang-github-hashicorp-raft_1.1.2-1_i386.changes dpkg-genchanges: info: binary-only upload (no source code included) dpkg-source --after-build . dpkg-buildpackage: info: binary-only upload (no source included) dpkg-genchanges: info: including full source code in upload I: copying local configuration I: unmounting dev/ptmx filesystem I: unmounting dev/pts filesystem I: unmounting dev/shm filesystem I: unmounting proc filesystem I: unmounting sys filesystem I: cleaning the build env I: removing directory /srv/workspace/pbuilder/904 and its subdirectories I: Current time: Mon Jan 8 17:10:20 -12 2024 I: pbuilder-time-stamp: 1704777020