Sat Jul 10 01:03:30 UTC 2021 I: starting to build golang-github-hashicorp-raft/bullseye/arm64 on jenkins on '2021-07-10 01:03' Sat Jul 10 01:03:30 UTC 2021 I: The jenkins build log is/was available at https://jenkins.debian.net/userContent/reproducible/debian/build_service/arm64_30/11277/console.log Sat Jul 10 01:03:30 UTC 2021 I: Downloading source for bullseye/golang-github-hashicorp-raft=1.1.2-1 --2021-07-10 01:03:30-- http://cdn-fastly.deb.debian.org/debian/pool/main/g/golang-github-hashicorp-raft/golang-github-hashicorp-raft_1.1.2-1.dsc Connecting to 78.137.99.97:3128... connected. Proxy request sent, awaiting response... 200 OK Length: 2408 (2.4K) Saving to: ‘golang-github-hashicorp-raft_1.1.2-1.dsc’ 0K .. 100% 142M=0s 2021-07-10 01:03:30 (142 MB/s) - ‘golang-github-hashicorp-raft_1.1.2-1.dsc’ saved [2408/2408] Sat Jul 10 01:03:30 UTC 2021 I: golang-github-hashicorp-raft_1.1.2-1.dsc -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Format: 3.0 (quilt) Source: golang-github-hashicorp-raft Binary: golang-github-hashicorp-raft-dev Architecture: all Version: 1.1.2-1 Maintainer: Debian Go Packaging Team Uploaders: Tim Potter , Dmitry Smirnov , Homepage: https://github.com/hashicorp/raft Standards-Version: 4.4.1 Vcs-Browser: https://salsa.debian.org/go-team/packages/golang-github-hashicorp-raft Vcs-Git: https://salsa.debian.org/go-team/packages/golang-github-hashicorp-raft.git Testsuite: autopkgtest-pkg-go Build-Depends: debhelper (>= 12~), dh-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~) Package-List: golang-github-hashicorp-raft-dev deb devel optional arch=all Checksums-Sha1: 6eb88916d5b8568cc5e49dfd778e53f425f02612 103160 golang-github-hashicorp-raft_1.1.2.orig.tar.xz 3efa08a30605d234327a7c1293b36ecde63ffbbe 8980 golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz Checksums-Sha256: 6c3383ee430aa6b421cc4cb4cbc485b0e612bda2e94e88d416d13d65d375dc89 103160 golang-github-hashicorp-raft_1.1.2.orig.tar.xz 3877bf052d5719266221189613502262d1be1af152c736d3dc8ed49734ad6dd8 8980 golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz Files: 65844d5b08afa2d62ea4dedaf43e6047 103160 golang-github-hashicorp-raft_1.1.2.orig.tar.xz 036528358012213daa0588e86be2a3d4 8980 golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz Go-Import-Path: github.com/hashicorp/raft -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEULx8+TnSDCcqawZWUra72VOWjRsFAl4kBCwACgkQUra72VOW jRsPQBAAxzZChwaTUFMUIQz9GgdwyR5o2SjXw48JJ//lfygslStaiz8ju52lp33x a1d56obRwTy2QmY2s6+oV3Mq4sQWs7U/1PGnywKeA71Us10A4AAv5d9wMRq1eqzp PcyVdhfM0zFcmJf8Wr93Mi8wbCr9PT+oTxc59+zWDruS7xCoe3aDtaekVfb8QZyj OZPsaTtneSxKjqk+V5Jyx8X6Y6P2cwv89GDd4nxTxapw8H38e2cBk7BsYmXUNXye 6N7dOChDN+x6gqgBU0O97oex+QOpd4LNoFBl2gEi7kukw+jrjrdyKQCANPh0CKx0 K4MPq2BeI3qKJV21l2z7S30BSjduvyktC7efckw5CX4zgT9/kJ5e7ZAf/og+M2vc CoCV+aoAH1OOWbkPu0dsuJiA5z1ZkKGa8ZB5MgvUentk7LVd/HaXW9QDdRTNZT8z Jdh6/wnpSn8ZZEhAixTtqNt64s0sNWmh4krF0WgGNZ1djzYd6gVI1LqlJTaSRKFo nxGsEe+jnC6kv2u8XpLG+xevTKcu4UIFHq2ebpanFvxzqxAv1dWTBDDI13NApMZb Po5huhcdLapUW4NGW/3pkdJkpcY5u26zH6YvTxDICU/tHXDo5/BqI9E1jEj+5m5J XrNqpKLFjW2I9gRcmT5vy3O4edLCPBFs6MGQBK7u1V+nEkh8pJc= =IObb -----END PGP SIGNATURE----- Sat Jul 10 01:03:31 UTC 2021 I: Checking whether the package is not for us Sat Jul 10 01:03:31 UTC 2021 I: Starting 1st build on remote node codethink16-arm64.debian.net. Sat Jul 10 01:03:31 UTC 2021 I: Preparing to do remote build '1' on codethink16-arm64.debian.net. Sat Jul 10 01:05:24 UTC 2021 I: Deleting $TMPDIR on codethink16-arm64.debian.net. I: pbuilder: network access will be disabled during build I: Current time: Fri Jul 9 13:03:34 -12 2021 I: pbuilder-time-stamp: 1625879014 I: Building the build Environment I: extracting base tarball [/var/cache/pbuilder/bullseye-reproducible-base.tgz] I: copying local configuration I: mounting /proc filesystem I: mounting /sys filesystem I: creating /{dev,run}/shm I: mounting /dev/pts filesystem I: redirecting /dev/ptmx to /dev/pts/ptmx I: policy-rc.d already exists I: Copying source file I: copying [golang-github-hashicorp-raft_1.1.2-1.dsc] I: copying [./golang-github-hashicorp-raft_1.1.2.orig.tar.xz] I: copying [./golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz] I: Extracting source gpgv: unknown type of key resource 'trustedkeys.kbx' gpgv: keyblock resource '/tmp/dpkg-verify-sig.vJtVmMUa/trustedkeys.kbx': General error gpgv: Signature made Sat Jan 18 19:24:28 2020 -12 gpgv: using RSA key 50BC7CF939D20C272A6B065652B6BBD953968D1B gpgv: Can't check signature: No public key dpkg-source: warning: failed to verify signature on ./golang-github-hashicorp-raft_1.1.2-1.dsc dpkg-source: info: extracting golang-github-hashicorp-raft in golang-github-hashicorp-raft-1.1.2 dpkg-source: info: unpacking golang-github-hashicorp-raft_1.1.2.orig.tar.xz dpkg-source: info: unpacking golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz dpkg-source: info: using patch list from debian/patches/series dpkg-source: info: applying t-skip--TestNetworkTransport_AppendEntriesPipeline_CloseStreams.patch dpkg-source: info: applying t-skip--TestRaft_AutoSnapshot.patch dpkg-source: info: applying t-skip--TestRaft_LeadershipTransferLeaderRejectsClientRequests.patch dpkg-source: info: applying t-skip--TestRaft_RecoverCluster.patch dpkg-source: info: applying t-skip--TestRaft_SnapshotRestore_PeerChange.patch I: using fakeroot in build. I: Installing the build-deps I: user script /srv/workspace/pbuilder/15724/tmp/hooks/D02_print_environment starting I: set BUILDDIR='/build' BUILDUSERGECOS='first user,first room,first work-phone,first home-phone,first other' BUILDUSERNAME='pbuilder1' BUILD_ARCH='arm64' DEBIAN_FRONTEND='noninteractive' DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all,-fixfilepath parallel=8' DISTRIBUTION='' HOME='/var/lib/jenkins' HOST_ARCH='arm64' IFS=' ' LANG='C' LANGUAGE='en_US:en' LC_ALL='C' MAIL='/var/mail/root' OPTIND='1' PATH='/usr/sbin:/usr/bin:/sbin:/bin:/usr/games' PBCURRENTCOMMANDLINEOPERATION='build' PBUILDER_OPERATION='build' PBUILDER_PKGDATADIR='/usr/share/pbuilder' PBUILDER_PKGLIBDIR='/usr/lib/pbuilder' PBUILDER_SYSCONFDIR='/etc' PPID='15724' PS1='# ' PS2='> ' PS4='+ ' PWD='/' SHELL='/bin/bash' SHLVL='2' SUDO_COMMAND='/usr/bin/timeout -k 18.1h 18h /usr/bin/ionice -c 3 /usr/bin/nice /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/tmp.tE7qwSSjNo/pbuilderrc_G4FL --hookdir /etc/pbuilder/first-build-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/bullseye-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/tmp.tE7qwSSjNo/b1 --logfile b1/build.log golang-github-hashicorp-raft_1.1.2-1.dsc' SUDO_GID='117' SUDO_UID='110' SUDO_USER='jenkins' TERM='unknown' TZ='/usr/share/zoneinfo/Etc/GMT+12' USER='root' USERNAME='root' _='/usr/bin/systemd-run' http_proxy='http://192.168.101.16:3128' I: uname -a Linux codethink16-arm64 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:18:37 UTC 2021 aarch64 GNU/Linux I: ls -l /bin total 5252 -rwxr-xr-x 1 root root 1282512 Jun 21 14:26 bash -rwxr-xr-x 3 root root 34808 Jul 20 2020 bunzip2 -rwxr-xr-x 3 root root 34808 Jul 20 2020 bzcat lrwxrwxrwx 1 root root 6 Jul 20 2020 bzcmp -> bzdiff -rwxr-xr-x 1 root root 2225 Jul 20 2020 bzdiff lrwxrwxrwx 1 root root 6 Jul 20 2020 bzegrep -> bzgrep -rwxr-xr-x 1 root root 4877 Sep 4 2019 bzexe lrwxrwxrwx 1 root root 6 Jul 20 2020 bzfgrep -> bzgrep -rwxr-xr-x 1 root root 3775 Jul 20 2020 bzgrep -rwxr-xr-x 3 root root 34808 Jul 20 2020 bzip2 -rwxr-xr-x 1 root root 14264 Jul 20 2020 bzip2recover lrwxrwxrwx 1 root root 6 Jul 20 2020 bzless -> bzmore -rwxr-xr-x 1 root root 1297 Jul 20 2020 bzmore -rwxr-xr-x 1 root root 39832 Sep 22 2020 cat -rwxr-xr-x 1 root root 64512 Sep 22 2020 chgrp -rwxr-xr-x 1 root root 60368 Sep 22 2020 chmod -rwxr-xr-x 1 root root 64528 Sep 22 2020 chown -rwxr-xr-x 1 root root 138896 Sep 22 2020 cp -rwxr-xr-x 1 root root 129544 Dec 10 2020 dash -rwxr-xr-x 1 root root 101384 Sep 22 2020 date -rwxr-xr-x 1 root root 80984 Sep 22 2020 dd -rwxr-xr-x 1 root root 89824 Sep 22 2020 df -rwxr-xr-x 1 root root 143088 Sep 22 2020 dir -rwxr-xr-x 1 root root 76152 Feb 7 02:38 dmesg lrwxrwxrwx 1 root root 8 Nov 6 2019 dnsdomainname -> hostname lrwxrwxrwx 1 root root 8 Nov 6 2019 domainname -> hostname -rwxr-xr-x 1 root root 35632 Sep 22 2020 echo -rwxr-xr-x 1 root root 28 Nov 9 2020 egrep -rwxr-xr-x 1 root root 31512 Sep 22 2020 false -rwxr-xr-x 1 root root 28 Nov 9 2020 fgrep -rwxr-xr-x 1 root root 64856 Feb 7 02:38 findmnt -rwsr-xr-x 1 root root 34824 Feb 26 04:12 fusermount -rwxr-xr-x 1 root root 178400 Nov 9 2020 grep -rwxr-xr-x 2 root root 2346 Mar 2 11:30 gunzip -rwxr-xr-x 1 root root 6376 Mar 2 11:30 gzexe -rwxr-xr-x 1 root root 93744 Mar 2 11:30 gzip -rwxr-xr-x 1 root root 18440 Nov 6 2019 hostname -rwxr-xr-x 1 root root 68720 Sep 22 2020 ln -rwxr-xr-x 1 root root 52720 Feb 7 2020 login -rwxr-xr-x 1 root root 143088 Sep 22 2020 ls -rwxr-xr-x 1 root root 161960 Feb 7 02:38 lsblk -rwxr-xr-x 1 root root 85200 Sep 22 2020 mkdir -rwxr-xr-x 1 root root 68744 Sep 22 2020 mknod -rwxr-xr-x 1 root root 43976 Sep 22 2020 mktemp -rwxr-xr-x 1 root root 51368 Feb 7 02:38 more -rwsr-xr-x 1 root root 51360 Feb 7 02:38 mount -rwxr-xr-x 1 root root 14496 Feb 7 02:38 mountpoint -rwxr-xr-x 1 root root 134808 Sep 22 2020 mv lrwxrwxrwx 1 root root 8 Nov 6 2019 nisdomainname -> hostname lrwxrwxrwx 1 root root 14 Apr 18 03:38 pidof -> /sbin/killall5 -rwxr-xr-x 1 root root 35720 Sep 22 2020 pwd lrwxrwxrwx 1 root root 4 Jun 21 14:26 rbash -> bash -rwxr-xr-x 1 root root 43872 Sep 22 2020 readlink -rwxr-xr-x 1 root root 68592 Sep 22 2020 rm -rwxr-xr-x 1 root root 43880 Sep 22 2020 rmdir -rwxr-xr-x 1 root root 19208 Sep 27 2020 run-parts -rwxr-xr-x 1 root root 114016 Dec 22 2018 sed lrwxrwxrwx 1 root root 4 Jul 7 21:24 sh -> dash -rwxr-xr-x 1 root root 35656 Sep 22 2020 sleep -rwxr-xr-x 1 root root 72640 Sep 22 2020 stty -rwsr-xr-x 1 root root 67776 Feb 7 02:38 su -rwxr-xr-x 1 root root 35672 Sep 22 2020 sync -rwxr-xr-x 1 root root 535768 Feb 16 21:55 tar -rwxr-xr-x 1 root root 10568 Sep 27 2020 tempfile -rwxr-xr-x 1 root root 89120 Sep 22 2020 touch -rwxr-xr-x 1 root root 31512 Sep 22 2020 true -rwxr-xr-x 1 root root 14264 Feb 26 04:12 ulockmgr_server -rwsr-xr-x 1 root root 30880 Feb 7 02:38 umount -rwxr-xr-x 1 root root 35640 Sep 22 2020 uname -rwxr-xr-x 2 root root 2346 Mar 2 11:30 uncompress -rwxr-xr-x 1 root root 143088 Sep 22 2020 vdir -rwxr-xr-x 1 root root 59584 Feb 7 02:38 wdctl lrwxrwxrwx 1 root root 8 Nov 6 2019 ypdomainname -> hostname -rwxr-xr-x 1 root root 1984 Mar 2 11:30 zcat -rwxr-xr-x 1 root root 1678 Mar 2 11:30 zcmp -rwxr-xr-x 1 root root 5880 Mar 2 11:30 zdiff -rwxr-xr-x 1 root root 29 Mar 2 11:30 zegrep -rwxr-xr-x 1 root root 29 Mar 2 11:30 zfgrep -rwxr-xr-x 1 root root 2081 Mar 2 11:30 zforce -rwxr-xr-x 1 root root 7585 Mar 2 11:30 zgrep -rwxr-xr-x 1 root root 2206 Mar 2 11:30 zless -rwxr-xr-x 1 root root 1842 Mar 2 11:30 zmore -rwxr-xr-x 1 root root 4553 Mar 2 11:30 znew I: user script /srv/workspace/pbuilder/15724/tmp/hooks/D02_print_environment finished -> Attempting to satisfy build-dependencies -> Creating pbuilder-satisfydepends-dummy package Package: pbuilder-satisfydepends-dummy Version: 0.invalid.0 Architecture: arm64 Maintainer: Debian Pbuilder Team Description: Dummy package to satisfy dependencies with aptitude - created by pbuilder This package was created automatically by pbuilder to satisfy the build-dependencies of the package being currently built. Depends: debhelper (>= 12~), dh-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~) dpkg-deb: building package 'pbuilder-satisfydepends-dummy' in '/tmp/satisfydepends-aptitude/pbuilder-satisfydepends-dummy.deb'. Selecting previously unselected package pbuilder-satisfydepends-dummy. (Reading database ... 19646 files and directories currently installed.) Preparing to unpack .../pbuilder-satisfydepends-dummy.deb ... Unpacking pbuilder-satisfydepends-dummy (0.invalid.0) ... dpkg: pbuilder-satisfydepends-dummy: dependency problems, but configuring anyway as you requested: pbuilder-satisfydepends-dummy depends on debhelper (>= 12~); however: Package debhelper is not installed. pbuilder-satisfydepends-dummy depends on dh-golang; however: Package dh-golang is not installed. pbuilder-satisfydepends-dummy depends on golang-any; however: Package golang-any is not installed. pbuilder-satisfydepends-dummy depends on golang-github-armon-go-metrics-dev; however: Package golang-github-armon-go-metrics-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-hclog-dev (>= 0.9.2~); however: Package golang-github-hashicorp-go-hclog-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~); however: Package golang-github-hashicorp-go-msgpack-dev is not installed. Setting up pbuilder-satisfydepends-dummy (0.invalid.0) ... Reading package lists... Building dependency tree... Reading state information... Initializing package states... Writing extended state information... Building tag database... pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0) pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0) The following NEW packages will be installed: autoconf{a} automake{a} autopoint{a} autotools-dev{a} bsdextrautils{a} debhelper{a} dh-autoreconf{a} dh-golang{a} dh-strip-nondeterminism{a} dwz{a} file{a} gettext{a} gettext-base{a} golang-1.15-go{a} golang-1.15-src{a} golang-any{a} golang-github-alecthomas-units-dev{a} golang-github-armon-go-metrics-dev{a} golang-github-beorn7-perks-dev{a} golang-github-cespare-xxhash-dev{a} golang-github-circonus-labs-circonus-gometrics-dev{a} golang-github-circonus-labs-circonusllhist-dev{a} golang-github-creack-pty-dev{a} golang-github-datadog-datadog-go-dev{a} golang-github-davecgh-go-spew-dev{a} golang-github-dgrijalva-jwt-go-dev{a} golang-github-fatih-color-dev{a} golang-github-go-kit-kit-dev{a} golang-github-go-logfmt-logfmt-dev{a} golang-github-go-stack-stack-dev{a} golang-github-golang-mock-dev{a} golang-github-google-go-cmp-dev{a} golang-github-hashicorp-go-cleanhttp-dev{a} golang-github-hashicorp-go-hclog-dev{a} golang-github-hashicorp-go-immutable-radix-dev{a} golang-github-hashicorp-go-msgpack-dev{a} golang-github-hashicorp-go-retryablehttp-dev{a} golang-github-hashicorp-go-uuid-dev{a} golang-github-hashicorp-golang-lru-dev{a} golang-github-jpillora-backoff-dev{a} golang-github-json-iterator-go-dev{a} golang-github-julienschmidt-httprouter-dev{a} golang-github-kr-pretty-dev{a} golang-github-kr-text-dev{a} golang-github-mattn-go-colorable-dev{a} golang-github-mattn-go-isatty-dev{a} golang-github-modern-go-concurrent-dev{a} golang-github-modern-go-reflect2-dev{a} golang-github-mwitkow-go-conntrack-dev{a} golang-github-opentracing-opentracing-go-dev{a} golang-github-pascaldekloe-goe-dev{a} golang-github-pkg-errors-dev{a} golang-github-pmezard-go-difflib-dev{a} golang-github-prometheus-client-golang-dev{a} golang-github-prometheus-client-model-dev{a} golang-github-prometheus-common-dev{a} golang-github-prometheus-procfs-dev{a} golang-github-sirupsen-logrus-dev{a} golang-github-stretchr-objx-dev{a} golang-github-stretchr-testify-dev{a} golang-github-tv42-httpunix-dev{a} golang-github-vmihailenco-tagparser-dev{a} golang-glog-dev{a} golang-go{a} golang-golang-x-net-dev{a} golang-golang-x-oauth2-dev{a} golang-golang-x-oauth2-google-dev{a} golang-golang-x-sync-dev{a} golang-golang-x-sys-dev{a} golang-golang-x-text-dev{a} golang-golang-x-time-dev{a} golang-golang-x-xerrors-dev{a} golang-google-cloud-compute-metadata-dev{a} golang-google-genproto-dev{a} golang-google-grpc-dev{a} golang-gopkg-alecthomas-kingpin.v2-dev{a} golang-gopkg-check.v1-dev{a} golang-gopkg-mgo.v2-dev{a} golang-gopkg-tomb.v2-dev{a} golang-gopkg-vmihailenco-msgpack.v2-dev{a} golang-gopkg-yaml.v2-dev{a} golang-gopkg-yaml.v3-dev{a} golang-goprotobuf-dev{a} golang-protobuf-extensions-dev{a} golang-src{a} groff-base{a} intltool-debian{a} libarchive-zip-perl{a} libdebhelper-perl{a} libelf1{a} libfile-stripnondeterminism-perl{a} libicu67{a} libmagic-mgc{a} libmagic1{a} libpipeline1{a} libprotobuf-dev{a} libprotobuf-lite23{a} libprotobuf23{a} libprotoc23{a} libsasl2-2{a} libsasl2-dev{a} libsasl2-modules-db{a} libsigsegv2{a} libsub-override-perl{a} libtool{a} libuchardet0{a} libxml2{a} m4{a} man-db{a} po-debconf{a} protobuf-compiler{a} sensible-utils{a} zlib1g-dev{a} The following packages are RECOMMENDED but will NOT be installed: curl libarchive-cpio-perl libltdl-dev libmail-sendmail-perl libsasl2-modules lynx pkg-config pkgconf wget 0 packages upgraded, 113 newly installed, 0 to remove and 0 not upgraded. Need to get 91.1 MB of archives. After unpacking 560 MB will be used. Writing extended state information... Get: 1 http://deb.debian.org/debian bullseye/main arm64 bsdextrautils arm64 2.36.1-7 [141 kB] Get: 2 http://deb.debian.org/debian bullseye/main arm64 libuchardet0 arm64 0.0.7-1 [67.9 kB] Get: 3 http://deb.debian.org/debian bullseye/main arm64 groff-base arm64 1.22.4-6 [883 kB] Get: 4 http://deb.debian.org/debian bullseye/main arm64 libpipeline1 arm64 1.5.3-1 [33.0 kB] Get: 5 http://deb.debian.org/debian bullseye/main arm64 man-db arm64 2.9.4-2 [1336 kB] Get: 6 http://deb.debian.org/debian bullseye/main arm64 golang-golang-x-sys-dev all 0.0~git20210124.22da62e-1 [308 kB] Get: 7 http://deb.debian.org/debian bullseye/main arm64 golang-github-sirupsen-logrus-dev all 1.7.0-2 [45.3 kB] Get: 8 http://deb.debian.org/debian bullseye/main arm64 sensible-utils all 0.0.14 [14.8 kB] Get: 9 http://deb.debian.org/debian bullseye/main arm64 libmagic-mgc arm64 1:5.39-3 [273 kB] Get: 10 http://deb.debian.org/debian bullseye/main arm64 libmagic1 arm64 1:5.39-3 [121 kB] Get: 11 http://deb.debian.org/debian bullseye/main arm64 file arm64 1:5.39-3 [69.1 kB] Get: 12 http://deb.debian.org/debian bullseye/main arm64 gettext-base arm64 0.21-4 [173 kB] Get: 13 http://deb.debian.org/debian bullseye/main arm64 libsigsegv2 arm64 2.13-1 [34.7 kB] Get: 14 http://deb.debian.org/debian bullseye/main arm64 m4 arm64 1.4.18-5 [199 kB] Get: 15 http://deb.debian.org/debian bullseye/main arm64 autoconf all 2.69-14 [313 kB] Get: 16 http://deb.debian.org/debian bullseye/main arm64 autotools-dev all 20180224.1+nmu1 [77.1 kB] Get: 17 http://deb.debian.org/debian bullseye/main arm64 automake all 1:1.16.3-2 [814 kB] Get: 18 http://deb.debian.org/debian bullseye/main arm64 autopoint all 0.21-4 [510 kB] Get: 19 http://deb.debian.org/debian bullseye/main arm64 libdebhelper-perl all 13.3.4 [189 kB] Get: 20 http://deb.debian.org/debian bullseye/main arm64 libtool all 2.4.6-15 [513 kB] Get: 21 http://deb.debian.org/debian bullseye/main arm64 dh-autoreconf all 20 [17.1 kB] Get: 22 http://deb.debian.org/debian bullseye/main arm64 libarchive-zip-perl all 1.68-1 [104 kB] Get: 23 http://deb.debian.org/debian bullseye/main arm64 libsub-override-perl all 0.09-2 [10.2 kB] Get: 24 http://deb.debian.org/debian bullseye/main arm64 libfile-stripnondeterminism-perl all 1.11.0-1 [25.6 kB] Get: 25 http://deb.debian.org/debian bullseye/main arm64 dh-strip-nondeterminism all 1.11.0-1 [15.3 kB] Get: 26 http://deb.debian.org/debian bullseye/main arm64 libelf1 arm64 0.183-1 [164 kB] Get: 27 http://deb.debian.org/debian bullseye/main arm64 dwz arm64 0.13+20210201-1 [155 kB] Get: 28 http://deb.debian.org/debian bullseye/main arm64 libicu67 arm64 67.1-7 [8467 kB] Get: 29 http://deb.debian.org/debian bullseye/main arm64 libxml2 arm64 2.9.10+dfsg-6.7 [629 kB] Get: 30 http://deb.debian.org/debian bullseye/main arm64 gettext arm64 0.21-4 [1261 kB] Get: 31 http://deb.debian.org/debian bullseye/main arm64 intltool-debian all 0.35.0+20060710.5 [26.8 kB] Get: 32 http://deb.debian.org/debian bullseye/main arm64 po-debconf all 1.0.21+nmu1 [248 kB] Get: 33 http://deb.debian.org/debian bullseye/main arm64 debhelper all 13.3.4 [1049 kB] Get: 34 http://deb.debian.org/debian bullseye/main arm64 dh-golang all 1.51 [24.6 kB] Get: 35 http://deb.debian.org/debian bullseye/main arm64 golang-1.15-src arm64 1.15.9-5 [13.8 MB] Get: 36 http://deb.debian.org/debian bullseye/main arm64 golang-1.15-go arm64 1.15.9-5 [42.9 MB] Get: 37 http://deb.debian.org/debian bullseye/main arm64 golang-src arm64 2:1.15~1 [4856 B] Get: 38 http://deb.debian.org/debian bullseye/main arm64 golang-go arm64 2:1.15~1 [23.9 kB] Get: 39 http://deb.debian.org/debian bullseye/main arm64 golang-any arm64 2:1.15~1 [4976 B] Get: 40 http://deb.debian.org/debian bullseye/main arm64 golang-github-alecthomas-units-dev all 0.0~git20201120.1786d5e-1 [7504 B] Get: 41 http://deb.debian.org/debian bullseye/main arm64 golang-github-circonus-labs-circonusllhist-dev all 0.0~git20191022.ec08cde-1 [11.1 kB] Get: 42 http://deb.debian.org/debian bullseye/main arm64 golang-github-hashicorp-go-cleanhttp-dev all 0.5.2-1 [10.8 kB] Get: 43 http://deb.debian.org/debian bullseye/main arm64 golang-github-mattn-go-isatty-dev all 0.0.12-1 [6472 B] Get: 44 http://deb.debian.org/debian bullseye/main arm64 golang-github-mattn-go-colorable-dev all 0.1.7-1 [9936 B] Get: 45 http://deb.debian.org/debian bullseye/main arm64 golang-github-fatih-color-dev all 1.7.0-1 [11.4 kB] Get: 46 http://deb.debian.org/debian bullseye/main arm64 golang-github-davecgh-go-spew-dev all 1.1.1-2 [29.7 kB] Get: 47 http://deb.debian.org/debian bullseye/main arm64 golang-github-pmezard-go-difflib-dev all 1.0.0-3 [12.3 kB] Get: 48 http://deb.debian.org/debian bullseye/main arm64 golang-github-stretchr-objx-dev all 0.3.0-1 [25.4 kB] Get: 49 http://deb.debian.org/debian bullseye/main arm64 golang-gopkg-yaml.v3-dev all 3.0.0~git20200121.a6ecf24-3 [70.5 kB] Get: 50 http://deb.debian.org/debian bullseye/main arm64 golang-github-stretchr-testify-dev all 1.6.1-2 [60.4 kB] Get: 51 http://deb.debian.org/debian bullseye/main arm64 golang-github-hashicorp-go-hclog-dev all 0.11.0-1 [17.9 kB] Get: 52 http://deb.debian.org/debian bullseye/main arm64 golang-github-hashicorp-go-retryablehttp-dev all 0.6.4-2 [17.5 kB] Get: 53 http://deb.debian.org/debian bullseye/main arm64 golang-github-pkg-errors-dev all 0.9.1-1 [13.0 kB] Get: 54 http://deb.debian.org/debian bullseye/main arm64 golang-github-tv42-httpunix-dev all 0.0~git20150427.b75d861-2 [3744 B] Get: 55 http://deb.debian.org/debian bullseye/main arm64 golang-github-circonus-labs-circonus-gometrics-dev all 2.3.1-3 [65.9 kB] Get: 56 http://deb.debian.org/debian bullseye/main arm64 golang-github-datadog-datadog-go-dev all 2.1.0-3 [14.9 kB] Get: 57 http://deb.debian.org/debian bullseye/main arm64 golang-github-hashicorp-go-uuid-dev all 1.0.2-1 [8764 B] Get: 58 http://deb.debian.org/debian bullseye/main arm64 golang-github-hashicorp-golang-lru-dev all 0.5.4-2 [14.7 kB] Get: 59 http://deb.debian.org/debian bullseye/main arm64 golang-github-hashicorp-go-immutable-radix-dev all 1.3.0-1 [25.0 kB] Get: 60 http://deb.debian.org/debian bullseye/main arm64 golang-github-pascaldekloe-goe-dev all 0.1.0-2 [21.7 kB] Get: 61 http://deb.debian.org/debian bullseye/main arm64 golang-github-beorn7-perks-dev all 1.0.1-1 [12.1 kB] Get: 62 http://deb.debian.org/debian bullseye/main arm64 golang-github-cespare-xxhash-dev all 2.1.1-1 [8748 B] Get: 63 http://deb.debian.org/debian bullseye/main arm64 golang-github-modern-go-concurrent-dev all 1.0.3-1.1 [4668 B] Get: 64 http://deb.debian.org/debian bullseye/main arm64 golang-github-modern-go-reflect2-dev all 1.0.1-1 [10.6 kB] Get: 65 http://deb.debian.org/debian bullseye/main arm64 golang-github-json-iterator-go-dev all 1.1.10-2 [69.4 kB] Get: 66 http://deb.debian.org/debian bullseye/main arm64 zlib1g-dev arm64 1:1.2.11.dfsg-2 [189 kB] Get: 67 http://deb.debian.org/debian bullseye/main arm64 libprotobuf23 arm64 3.12.4-1 [780 kB] Get: 68 http://deb.debian.org/debian bullseye/main arm64 libprotobuf-lite23 arm64 3.12.4-1 [218 kB] Get: 69 http://deb.debian.org/debian bullseye/main arm64 libprotobuf-dev arm64 3.12.4-1 [1163 kB] Get: 70 http://deb.debian.org/debian bullseye/main arm64 libprotoc23 arm64 3.12.4-1 [703 kB] Get: 71 http://deb.debian.org/debian bullseye/main arm64 protobuf-compiler arm64 3.12.4-1 [74.9 kB] Get: 72 http://deb.debian.org/debian bullseye/main arm64 golang-goprotobuf-dev arm64 1.3.4-2+b5 [1319 kB] Get: 73 http://deb.debian.org/debian bullseye/main arm64 golang-github-prometheus-client-model-dev all 0.2.0-2 [8476 B] Get: 74 http://deb.debian.org/debian bullseye/main arm64 golang-github-dgrijalva-jwt-go-dev all 3.2.0-3 [33.6 kB] Get: 75 http://deb.debian.org/debian bullseye/main arm64 golang-github-go-logfmt-logfmt-dev all 0.5.0-2 [13.4 kB] Get: 76 http://deb.debian.org/debian bullseye/main arm64 golang-github-go-stack-stack-dev all 1.8.0-1 [8696 B] Get: 77 http://deb.debian.org/debian bullseye/main arm64 golang-github-opentracing-opentracing-go-dev all 1.2.0-1 [29.4 kB] Get: 78 http://deb.debian.org/debian bullseye/main arm64 golang-golang-x-text-dev all 0.3.6-1 [3857 kB] Get: 79 http://deb.debian.org/debian bullseye/main arm64 golang-golang-x-net-dev all 1:0.0+git20210119.5f4716e+dfsg-4 [659 kB] Get: 80 http://deb.debian.org/debian bullseye/main arm64 golang-golang-x-time-dev all 0.0+git20200630.3af7569-1 [10.1 kB] Get: 81 http://deb.debian.org/debian bullseye/main arm64 golang-github-golang-mock-dev all 1.3.1-2 [35.1 kB] Get: 82 http://deb.debian.org/debian bullseye/main arm64 golang-golang-x-xerrors-dev all 0.0~git20191204.9bdfabe-1 [14.2 kB] Get: 83 http://deb.debian.org/debian bullseye/main arm64 golang-github-google-go-cmp-dev all 0.5.4-1 [83.7 kB] Get: 84 http://deb.debian.org/debian bullseye/main arm64 golang-glog-dev all 0.0~git20160126.23def4e-3 [17.3 kB] Get: 85 http://deb.debian.org/debian bullseye/main arm64 golang-golang-x-oauth2-dev all 0.0~git20190604.0f29369-2 [31.9 kB] Get: 86 http://deb.debian.org/debian bullseye/main arm64 golang-google-cloud-compute-metadata-dev all 0.56.0-1 [34.0 kB] Get: 87 http://deb.debian.org/debian bullseye/main arm64 golang-golang-x-oauth2-google-dev all 0.0~git20190604.0f29369-2 [13.2 kB] Get: 88 http://deb.debian.org/debian bullseye/main arm64 golang-google-genproto-dev all 0.0~git20200413.b5235f6-1 [4037 kB] Get: 89 http://deb.debian.org/debian bullseye/main arm64 golang-google-grpc-dev all 1.27.1-1 [478 kB] Get: 90 http://deb.debian.org/debian bullseye/main arm64 golang-github-go-kit-kit-dev all 0.6.0-3 [103 kB] Get: 91 http://deb.debian.org/debian bullseye/main arm64 golang-github-julienschmidt-httprouter-dev all 1.3.0-1 [19.3 kB] Get: 92 http://deb.debian.org/debian bullseye/main arm64 golang-github-jpillora-backoff-dev all 1.0.0-1.1 [3712 B] Get: 93 http://deb.debian.org/debian bullseye/main arm64 golang-github-mwitkow-go-conntrack-dev all 0.0~git20190716.2f06839-2 [14.6 kB] Get: 94 http://deb.debian.org/debian bullseye/main arm64 golang-gopkg-alecthomas-kingpin.v2-dev all 2.2.6-3 [42.8 kB] Get: 95 http://deb.debian.org/debian bullseye/main arm64 golang-gopkg-yaml.v2-dev all 2.4.0-1 [61.3 kB] Get: 96 http://deb.debian.org/debian bullseye/main arm64 golang-protobuf-extensions-dev all 1.0.1-1 [29.6 kB] Get: 97 http://deb.debian.org/debian bullseye/main arm64 golang-github-prometheus-common-dev all 0.15.0-2 [102 kB] Get: 98 http://deb.debian.org/debian bullseye/main arm64 golang-golang-x-sync-dev all 0.0~git20210220.036812b-1 [19.1 kB] Get: 99 http://deb.debian.org/debian bullseye/main arm64 golang-github-prometheus-procfs-dev all 0.3.0-2 [110 kB] Get: 100 http://deb.debian.org/debian bullseye/main arm64 golang-github-prometheus-client-golang-dev all 1.9.0-2 [134 kB] Get: 101 http://deb.debian.org/debian bullseye/main arm64 golang-github-armon-go-metrics-dev all 0.3.4-1 [30.5 kB] Get: 102 http://deb.debian.org/debian bullseye/main arm64 golang-github-creack-pty-dev all 1.1.11-1 [8380 B] Get: 103 http://deb.debian.org/debian bullseye/main arm64 golang-github-vmihailenco-tagparser-dev all 0.1.1-2 [4440 B] Get: 104 http://deb.debian.org/debian bullseye/main arm64 golang-gopkg-vmihailenco-msgpack.v2-dev all 4.3.1-2 [27.5 kB] Get: 105 http://deb.debian.org/debian bullseye/main arm64 golang-gopkg-tomb.v2-dev all 0.0~git20161208.d5d1b58-3 [6840 B] Get: 106 http://deb.debian.org/debian bullseye/main arm64 libsasl2-modules-db arm64 2.1.27+dfsg-2.1 [69.3 kB] Get: 107 http://deb.debian.org/debian bullseye/main arm64 libsasl2-2 arm64 2.1.27+dfsg-2.1 [105 kB] Get: 108 http://deb.debian.org/debian bullseye/main arm64 libsasl2-dev arm64 2.1.27+dfsg-2.1 [277 kB] Get: 109 http://deb.debian.org/debian bullseye/main arm64 golang-github-kr-text-dev all 0.2.0-1 [11.1 kB] Get: 110 http://deb.debian.org/debian bullseye/main arm64 golang-github-kr-pretty-dev all 0.2.1+git20200831.59b4212-1 [14.6 kB] Get: 111 http://deb.debian.org/debian bullseye/main arm64 golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-1 [32.1 kB] Get: 112 http://deb.debian.org/debian bullseye/main arm64 golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB] Get: 113 http://deb.debian.org/debian bullseye/main arm64 golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB] Fetched 91.1 MB in 3s (36.4 MB/s) debconf: delaying package configuration, since apt-utils is not installed Selecting previously unselected package bsdextrautils. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 19646 files and directories currently installed.) Preparing to unpack .../000-bsdextrautils_2.36.1-7_arm64.deb ... Unpacking bsdextrautils (2.36.1-7) ... Selecting previously unselected package libuchardet0:arm64. Preparing to unpack .../001-libuchardet0_0.0.7-1_arm64.deb ... Unpacking libuchardet0:arm64 (0.0.7-1) ... Selecting previously unselected package groff-base. Preparing to unpack .../002-groff-base_1.22.4-6_arm64.deb ... Unpacking groff-base (1.22.4-6) ... Selecting previously unselected package libpipeline1:arm64. Preparing to unpack .../003-libpipeline1_1.5.3-1_arm64.deb ... Unpacking libpipeline1:arm64 (1.5.3-1) ... Selecting previously unselected package man-db. Preparing to unpack .../004-man-db_2.9.4-2_arm64.deb ... Unpacking man-db (2.9.4-2) ... Selecting previously unselected package golang-golang-x-sys-dev. Preparing to unpack .../005-golang-golang-x-sys-dev_0.0~git20210124.22da62e-1_all.deb ... Unpacking golang-golang-x-sys-dev (0.0~git20210124.22da62e-1) ... Selecting previously unselected package golang-github-sirupsen-logrus-dev. Preparing to unpack .../006-golang-github-sirupsen-logrus-dev_1.7.0-2_all.deb ... Unpacking golang-github-sirupsen-logrus-dev (1.7.0-2) ... Selecting previously unselected package sensible-utils. Preparing to unpack .../007-sensible-utils_0.0.14_all.deb ... Unpacking sensible-utils (0.0.14) ... Selecting previously unselected package libmagic-mgc. Preparing to unpack .../008-libmagic-mgc_1%3a5.39-3_arm64.deb ... Unpacking libmagic-mgc (1:5.39-3) ... Selecting previously unselected package libmagic1:arm64. Preparing to unpack .../009-libmagic1_1%3a5.39-3_arm64.deb ... Unpacking libmagic1:arm64 (1:5.39-3) ... Selecting previously unselected package file. Preparing to unpack .../010-file_1%3a5.39-3_arm64.deb ... Unpacking file (1:5.39-3) ... Selecting previously unselected package gettext-base. Preparing to unpack .../011-gettext-base_0.21-4_arm64.deb ... Unpacking gettext-base (0.21-4) ... Selecting previously unselected package libsigsegv2:arm64. Preparing to unpack .../012-libsigsegv2_2.13-1_arm64.deb ... Unpacking libsigsegv2:arm64 (2.13-1) ... Selecting previously unselected package m4. Preparing to unpack .../013-m4_1.4.18-5_arm64.deb ... Unpacking m4 (1.4.18-5) ... Selecting previously unselected package autoconf. Preparing to unpack .../014-autoconf_2.69-14_all.deb ... Unpacking autoconf (2.69-14) ... Selecting previously unselected package autotools-dev. Preparing to unpack .../015-autotools-dev_20180224.1+nmu1_all.deb ... Unpacking autotools-dev (20180224.1+nmu1) ... Selecting previously unselected package automake. Preparing to unpack .../016-automake_1%3a1.16.3-2_all.deb ... Unpacking automake (1:1.16.3-2) ... Selecting previously unselected package autopoint. Preparing to unpack .../017-autopoint_0.21-4_all.deb ... Unpacking autopoint (0.21-4) ... Selecting previously unselected package libdebhelper-perl. Preparing to unpack .../018-libdebhelper-perl_13.3.4_all.deb ... Unpacking libdebhelper-perl (13.3.4) ... Selecting previously unselected package libtool. Preparing to unpack .../019-libtool_2.4.6-15_all.deb ... Unpacking libtool (2.4.6-15) ... Selecting previously unselected package dh-autoreconf. Preparing to unpack .../020-dh-autoreconf_20_all.deb ... Unpacking dh-autoreconf (20) ... Selecting previously unselected package libarchive-zip-perl. Preparing to unpack .../021-libarchive-zip-perl_1.68-1_all.deb ... Unpacking libarchive-zip-perl (1.68-1) ... Selecting previously unselected package libsub-override-perl. Preparing to unpack .../022-libsub-override-perl_0.09-2_all.deb ... Unpacking libsub-override-perl (0.09-2) ... Selecting previously unselected package libfile-stripnondeterminism-perl. Preparing to unpack .../023-libfile-stripnondeterminism-perl_1.11.0-1_all.deb ... Unpacking libfile-stripnondeterminism-perl (1.11.0-1) ... Selecting previously unselected package dh-strip-nondeterminism. Preparing to unpack .../024-dh-strip-nondeterminism_1.11.0-1_all.deb ... Unpacking dh-strip-nondeterminism (1.11.0-1) ... Selecting previously unselected package libelf1:arm64. Preparing to unpack .../025-libelf1_0.183-1_arm64.deb ... Unpacking libelf1:arm64 (0.183-1) ... Selecting previously unselected package dwz. Preparing to unpack .../026-dwz_0.13+20210201-1_arm64.deb ... Unpacking dwz (0.13+20210201-1) ... Selecting previously unselected package libicu67:arm64. Preparing to unpack .../027-libicu67_67.1-7_arm64.deb ... Unpacking libicu67:arm64 (67.1-7) ... Selecting previously unselected package libxml2:arm64. Preparing to unpack .../028-libxml2_2.9.10+dfsg-6.7_arm64.deb ... Unpacking libxml2:arm64 (2.9.10+dfsg-6.7) ... Selecting previously unselected package gettext. Preparing to unpack .../029-gettext_0.21-4_arm64.deb ... Unpacking gettext (0.21-4) ... Selecting previously unselected package intltool-debian. Preparing to unpack .../030-intltool-debian_0.35.0+20060710.5_all.deb ... Unpacking intltool-debian (0.35.0+20060710.5) ... Selecting previously unselected package po-debconf. Preparing to unpack .../031-po-debconf_1.0.21+nmu1_all.deb ... Unpacking po-debconf (1.0.21+nmu1) ... Selecting previously unselected package debhelper. Preparing to unpack .../032-debhelper_13.3.4_all.deb ... Unpacking debhelper (13.3.4) ... Selecting previously unselected package dh-golang. Preparing to unpack .../033-dh-golang_1.51_all.deb ... Unpacking dh-golang (1.51) ... Selecting previously unselected package golang-1.15-src. Preparing to unpack .../034-golang-1.15-src_1.15.9-5_arm64.deb ... Unpacking golang-1.15-src (1.15.9-5) ... Selecting previously unselected package golang-1.15-go. Preparing to unpack .../035-golang-1.15-go_1.15.9-5_arm64.deb ... Unpacking golang-1.15-go (1.15.9-5) ... Selecting previously unselected package golang-src:arm64. Preparing to unpack .../036-golang-src_2%3a1.15~1_arm64.deb ... Unpacking golang-src:arm64 (2:1.15~1) ... Selecting previously unselected package golang-go. Preparing to unpack .../037-golang-go_2%3a1.15~1_arm64.deb ... Unpacking golang-go (2:1.15~1) ... Selecting previously unselected package golang-any:arm64. Preparing to unpack .../038-golang-any_2%3a1.15~1_arm64.deb ... Unpacking golang-any:arm64 (2:1.15~1) ... Selecting previously unselected package golang-github-alecthomas-units-dev. Preparing to unpack .../039-golang-github-alecthomas-units-dev_0.0~git20201120.1786d5e-1_all.deb ... Unpacking golang-github-alecthomas-units-dev (0.0~git20201120.1786d5e-1) ... Selecting previously unselected package golang-github-circonus-labs-circonusllhist-dev. Preparing to unpack .../040-golang-github-circonus-labs-circonusllhist-dev_0.0~git20191022.ec08cde-1_all.deb ... Unpacking golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ... Selecting previously unselected package golang-github-hashicorp-go-cleanhttp-dev. Preparing to unpack .../041-golang-github-hashicorp-go-cleanhttp-dev_0.5.2-1_all.deb ... Unpacking golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ... Selecting previously unselected package golang-github-mattn-go-isatty-dev. Preparing to unpack .../042-golang-github-mattn-go-isatty-dev_0.0.12-1_all.deb ... Unpacking golang-github-mattn-go-isatty-dev (0.0.12-1) ... Selecting previously unselected package golang-github-mattn-go-colorable-dev. Preparing to unpack .../043-golang-github-mattn-go-colorable-dev_0.1.7-1_all.deb ... Unpacking golang-github-mattn-go-colorable-dev (0.1.7-1) ... Selecting previously unselected package golang-github-fatih-color-dev. Preparing to unpack .../044-golang-github-fatih-color-dev_1.7.0-1_all.deb ... Unpacking golang-github-fatih-color-dev (1.7.0-1) ... Selecting previously unselected package golang-github-davecgh-go-spew-dev. Preparing to unpack .../045-golang-github-davecgh-go-spew-dev_1.1.1-2_all.deb ... Unpacking golang-github-davecgh-go-spew-dev (1.1.1-2) ... Selecting previously unselected package golang-github-pmezard-go-difflib-dev. Preparing to unpack .../046-golang-github-pmezard-go-difflib-dev_1.0.0-3_all.deb ... Unpacking golang-github-pmezard-go-difflib-dev (1.0.0-3) ... Selecting previously unselected package golang-github-stretchr-objx-dev. Preparing to unpack .../047-golang-github-stretchr-objx-dev_0.3.0-1_all.deb ... Unpacking golang-github-stretchr-objx-dev (0.3.0-1) ... Selecting previously unselected package golang-gopkg-yaml.v3-dev. Preparing to unpack .../048-golang-gopkg-yaml.v3-dev_3.0.0~git20200121.a6ecf24-3_all.deb ... Unpacking golang-gopkg-yaml.v3-dev (3.0.0~git20200121.a6ecf24-3) ... Selecting previously unselected package golang-github-stretchr-testify-dev. Preparing to unpack .../049-golang-github-stretchr-testify-dev_1.6.1-2_all.deb ... Unpacking golang-github-stretchr-testify-dev (1.6.1-2) ... Selecting previously unselected package golang-github-hashicorp-go-hclog-dev. Preparing to unpack .../050-golang-github-hashicorp-go-hclog-dev_0.11.0-1_all.deb ... Unpacking golang-github-hashicorp-go-hclog-dev (0.11.0-1) ... Selecting previously unselected package golang-github-hashicorp-go-retryablehttp-dev. Preparing to unpack .../051-golang-github-hashicorp-go-retryablehttp-dev_0.6.4-2_all.deb ... Unpacking golang-github-hashicorp-go-retryablehttp-dev (0.6.4-2) ... Selecting previously unselected package golang-github-pkg-errors-dev. Preparing to unpack .../052-golang-github-pkg-errors-dev_0.9.1-1_all.deb ... Unpacking golang-github-pkg-errors-dev (0.9.1-1) ... Selecting previously unselected package golang-github-tv42-httpunix-dev. Preparing to unpack .../053-golang-github-tv42-httpunix-dev_0.0~git20150427.b75d861-2_all.deb ... Unpacking golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-2) ... Selecting previously unselected package golang-github-circonus-labs-circonus-gometrics-dev. Preparing to unpack .../054-golang-github-circonus-labs-circonus-gometrics-dev_2.3.1-3_all.deb ... Unpacking golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-3) ... Selecting previously unselected package golang-github-datadog-datadog-go-dev. Preparing to unpack .../055-golang-github-datadog-datadog-go-dev_2.1.0-3_all.deb ... Unpacking golang-github-datadog-datadog-go-dev (2.1.0-3) ... Selecting previously unselected package golang-github-hashicorp-go-uuid-dev. Preparing to unpack .../056-golang-github-hashicorp-go-uuid-dev_1.0.2-1_all.deb ... Unpacking golang-github-hashicorp-go-uuid-dev (1.0.2-1) ... Selecting previously unselected package golang-github-hashicorp-golang-lru-dev. Preparing to unpack .../057-golang-github-hashicorp-golang-lru-dev_0.5.4-2_all.deb ... Unpacking golang-github-hashicorp-golang-lru-dev (0.5.4-2) ... Selecting previously unselected package golang-github-hashicorp-go-immutable-radix-dev. Preparing to unpack .../058-golang-github-hashicorp-go-immutable-radix-dev_1.3.0-1_all.deb ... Unpacking golang-github-hashicorp-go-immutable-radix-dev (1.3.0-1) ... Selecting previously unselected package golang-github-pascaldekloe-goe-dev. Preparing to unpack .../059-golang-github-pascaldekloe-goe-dev_0.1.0-2_all.deb ... Unpacking golang-github-pascaldekloe-goe-dev (0.1.0-2) ... Selecting previously unselected package golang-github-beorn7-perks-dev. Preparing to unpack .../060-golang-github-beorn7-perks-dev_1.0.1-1_all.deb ... Unpacking golang-github-beorn7-perks-dev (1.0.1-1) ... Selecting previously unselected package golang-github-cespare-xxhash-dev. Preparing to unpack .../061-golang-github-cespare-xxhash-dev_2.1.1-1_all.deb ... Unpacking golang-github-cespare-xxhash-dev (2.1.1-1) ... Selecting previously unselected package golang-github-modern-go-concurrent-dev. Preparing to unpack .../062-golang-github-modern-go-concurrent-dev_1.0.3-1.1_all.deb ... Unpacking golang-github-modern-go-concurrent-dev (1.0.3-1.1) ... Selecting previously unselected package golang-github-modern-go-reflect2-dev. Preparing to unpack .../063-golang-github-modern-go-reflect2-dev_1.0.1-1_all.deb ... Unpacking golang-github-modern-go-reflect2-dev (1.0.1-1) ... Selecting previously unselected package golang-github-json-iterator-go-dev. Preparing to unpack .../064-golang-github-json-iterator-go-dev_1.1.10-2_all.deb ... Unpacking golang-github-json-iterator-go-dev (1.1.10-2) ... Selecting previously unselected package zlib1g-dev:arm64. Preparing to unpack .../065-zlib1g-dev_1%3a1.2.11.dfsg-2_arm64.deb ... Unpacking zlib1g-dev:arm64 (1:1.2.11.dfsg-2) ... Selecting previously unselected package libprotobuf23:arm64. Preparing to unpack .../066-libprotobuf23_3.12.4-1_arm64.deb ... Unpacking libprotobuf23:arm64 (3.12.4-1) ... Selecting previously unselected package libprotobuf-lite23:arm64. Preparing to unpack .../067-libprotobuf-lite23_3.12.4-1_arm64.deb ... Unpacking libprotobuf-lite23:arm64 (3.12.4-1) ... Selecting previously unselected package libprotobuf-dev:arm64. Preparing to unpack .../068-libprotobuf-dev_3.12.4-1_arm64.deb ... Unpacking libprotobuf-dev:arm64 (3.12.4-1) ... Selecting previously unselected package libprotoc23:arm64. Preparing to unpack .../069-libprotoc23_3.12.4-1_arm64.deb ... Unpacking libprotoc23:arm64 (3.12.4-1) ... Selecting previously unselected package protobuf-compiler. Preparing to unpack .../070-protobuf-compiler_3.12.4-1_arm64.deb ... Unpacking protobuf-compiler (3.12.4-1) ... Selecting previously unselected package golang-goprotobuf-dev. Preparing to unpack .../071-golang-goprotobuf-dev_1.3.4-2+b5_arm64.deb ... Unpacking golang-goprotobuf-dev (1.3.4-2+b5) ... Selecting previously unselected package golang-github-prometheus-client-model-dev. Preparing to unpack .../072-golang-github-prometheus-client-model-dev_0.2.0-2_all.deb ... Unpacking golang-github-prometheus-client-model-dev (0.2.0-2) ... Selecting previously unselected package golang-github-dgrijalva-jwt-go-dev. Preparing to unpack .../073-golang-github-dgrijalva-jwt-go-dev_3.2.0-3_all.deb ... Unpacking golang-github-dgrijalva-jwt-go-dev (3.2.0-3) ... Selecting previously unselected package golang-github-go-logfmt-logfmt-dev. Preparing to unpack .../074-golang-github-go-logfmt-logfmt-dev_0.5.0-2_all.deb ... Unpacking golang-github-go-logfmt-logfmt-dev (0.5.0-2) ... Selecting previously unselected package golang-github-go-stack-stack-dev. Preparing to unpack .../075-golang-github-go-stack-stack-dev_1.8.0-1_all.deb ... Unpacking golang-github-go-stack-stack-dev (1.8.0-1) ... Selecting previously unselected package golang-github-opentracing-opentracing-go-dev. Preparing to unpack .../076-golang-github-opentracing-opentracing-go-dev_1.2.0-1_all.deb ... Unpacking golang-github-opentracing-opentracing-go-dev (1.2.0-1) ... Selecting previously unselected package golang-golang-x-text-dev. Preparing to unpack .../077-golang-golang-x-text-dev_0.3.6-1_all.deb ... Unpacking golang-golang-x-text-dev (0.3.6-1) ... Selecting previously unselected package golang-golang-x-net-dev. Preparing to unpack .../078-golang-golang-x-net-dev_1%3a0.0+git20210119.5f4716e+dfsg-4_all.deb ... Unpacking golang-golang-x-net-dev (1:0.0+git20210119.5f4716e+dfsg-4) ... Selecting previously unselected package golang-golang-x-time-dev. Preparing to unpack .../079-golang-golang-x-time-dev_0.0+git20200630.3af7569-1_all.deb ... Unpacking golang-golang-x-time-dev (0.0+git20200630.3af7569-1) ... Selecting previously unselected package golang-github-golang-mock-dev. Preparing to unpack .../080-golang-github-golang-mock-dev_1.3.1-2_all.deb ... Unpacking golang-github-golang-mock-dev (1.3.1-2) ... Selecting previously unselected package golang-golang-x-xerrors-dev. Preparing to unpack .../081-golang-golang-x-xerrors-dev_0.0~git20191204.9bdfabe-1_all.deb ... Unpacking golang-golang-x-xerrors-dev (0.0~git20191204.9bdfabe-1) ... Selecting previously unselected package golang-github-google-go-cmp-dev. Preparing to unpack .../082-golang-github-google-go-cmp-dev_0.5.4-1_all.deb ... Unpacking golang-github-google-go-cmp-dev (0.5.4-1) ... Selecting previously unselected package golang-glog-dev. Preparing to unpack .../083-golang-glog-dev_0.0~git20160126.23def4e-3_all.deb ... Unpacking golang-glog-dev (0.0~git20160126.23def4e-3) ... Selecting previously unselected package golang-golang-x-oauth2-dev. Preparing to unpack .../084-golang-golang-x-oauth2-dev_0.0~git20190604.0f29369-2_all.deb ... Unpacking golang-golang-x-oauth2-dev (0.0~git20190604.0f29369-2) ... Selecting previously unselected package golang-google-cloud-compute-metadata-dev. Preparing to unpack .../085-golang-google-cloud-compute-metadata-dev_0.56.0-1_all.deb ... Unpacking golang-google-cloud-compute-metadata-dev (0.56.0-1) ... Selecting previously unselected package golang-golang-x-oauth2-google-dev. Preparing to unpack .../086-golang-golang-x-oauth2-google-dev_0.0~git20190604.0f29369-2_all.deb ... Unpacking golang-golang-x-oauth2-google-dev (0.0~git20190604.0f29369-2) ... Selecting previously unselected package golang-google-genproto-dev. Preparing to unpack .../087-golang-google-genproto-dev_0.0~git20200413.b5235f6-1_all.deb ... Unpacking golang-google-genproto-dev (0.0~git20200413.b5235f6-1) ... Selecting previously unselected package golang-google-grpc-dev. Preparing to unpack .../088-golang-google-grpc-dev_1.27.1-1_all.deb ... Unpacking golang-google-grpc-dev (1.27.1-1) ... Selecting previously unselected package golang-github-go-kit-kit-dev. Preparing to unpack .../089-golang-github-go-kit-kit-dev_0.6.0-3_all.deb ... Unpacking golang-github-go-kit-kit-dev (0.6.0-3) ... Selecting previously unselected package golang-github-julienschmidt-httprouter-dev. Preparing to unpack .../090-golang-github-julienschmidt-httprouter-dev_1.3.0-1_all.deb ... Unpacking golang-github-julienschmidt-httprouter-dev (1.3.0-1) ... Selecting previously unselected package golang-github-jpillora-backoff-dev. Preparing to unpack .../091-golang-github-jpillora-backoff-dev_1.0.0-1.1_all.deb ... Unpacking golang-github-jpillora-backoff-dev (1.0.0-1.1) ... Selecting previously unselected package golang-github-mwitkow-go-conntrack-dev. Preparing to unpack .../092-golang-github-mwitkow-go-conntrack-dev_0.0~git20190716.2f06839-2_all.deb ... Unpacking golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-2) ... Selecting previously unselected package golang-gopkg-alecthomas-kingpin.v2-dev. Preparing to unpack .../093-golang-gopkg-alecthomas-kingpin.v2-dev_2.2.6-3_all.deb ... Unpacking golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-3) ... Selecting previously unselected package golang-gopkg-yaml.v2-dev. Preparing to unpack .../094-golang-gopkg-yaml.v2-dev_2.4.0-1_all.deb ... Unpacking golang-gopkg-yaml.v2-dev (2.4.0-1) ... Selecting previously unselected package golang-protobuf-extensions-dev. Preparing to unpack .../095-golang-protobuf-extensions-dev_1.0.1-1_all.deb ... Unpacking golang-protobuf-extensions-dev (1.0.1-1) ... Selecting previously unselected package golang-github-prometheus-common-dev. Preparing to unpack .../096-golang-github-prometheus-common-dev_0.15.0-2_all.deb ... Unpacking golang-github-prometheus-common-dev (0.15.0-2) ... Selecting previously unselected package golang-golang-x-sync-dev. Preparing to unpack .../097-golang-golang-x-sync-dev_0.0~git20210220.036812b-1_all.deb ... Unpacking golang-golang-x-sync-dev (0.0~git20210220.036812b-1) ... Selecting previously unselected package golang-github-prometheus-procfs-dev. Preparing to unpack .../098-golang-github-prometheus-procfs-dev_0.3.0-2_all.deb ... Unpacking golang-github-prometheus-procfs-dev (0.3.0-2) ... Selecting previously unselected package golang-github-prometheus-client-golang-dev. Preparing to unpack .../099-golang-github-prometheus-client-golang-dev_1.9.0-2_all.deb ... Unpacking golang-github-prometheus-client-golang-dev (1.9.0-2) ... Selecting previously unselected package golang-github-armon-go-metrics-dev. Preparing to unpack .../100-golang-github-armon-go-metrics-dev_0.3.4-1_all.deb ... Unpacking golang-github-armon-go-metrics-dev (0.3.4-1) ... Selecting previously unselected package golang-github-creack-pty-dev. Preparing to unpack .../101-golang-github-creack-pty-dev_1.1.11-1_all.deb ... Unpacking golang-github-creack-pty-dev (1.1.11-1) ... Selecting previously unselected package golang-github-vmihailenco-tagparser-dev. Preparing to unpack .../102-golang-github-vmihailenco-tagparser-dev_0.1.1-2_all.deb ... Unpacking golang-github-vmihailenco-tagparser-dev (0.1.1-2) ... Selecting previously unselected package golang-gopkg-vmihailenco-msgpack.v2-dev. Preparing to unpack .../103-golang-gopkg-vmihailenco-msgpack.v2-dev_4.3.1-2_all.deb ... Unpacking golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ... Selecting previously unselected package golang-gopkg-tomb.v2-dev. Preparing to unpack .../104-golang-gopkg-tomb.v2-dev_0.0~git20161208.d5d1b58-3_all.deb ... Unpacking golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ... Selecting previously unselected package libsasl2-modules-db:arm64. Preparing to unpack .../105-libsasl2-modules-db_2.1.27+dfsg-2.1_arm64.deb ... Unpacking libsasl2-modules-db:arm64 (2.1.27+dfsg-2.1) ... Selecting previously unselected package libsasl2-2:arm64. Preparing to unpack .../106-libsasl2-2_2.1.27+dfsg-2.1_arm64.deb ... Unpacking libsasl2-2:arm64 (2.1.27+dfsg-2.1) ... Selecting previously unselected package libsasl2-dev. Preparing to unpack .../107-libsasl2-dev_2.1.27+dfsg-2.1_arm64.deb ... Unpacking libsasl2-dev (2.1.27+dfsg-2.1) ... Selecting previously unselected package golang-github-kr-text-dev. Preparing to unpack .../108-golang-github-kr-text-dev_0.2.0-1_all.deb ... Unpacking golang-github-kr-text-dev (0.2.0-1) ... Selecting previously unselected package golang-github-kr-pretty-dev. Preparing to unpack .../109-golang-github-kr-pretty-dev_0.2.1+git20200831.59b4212-1_all.deb ... Unpacking golang-github-kr-pretty-dev (0.2.1+git20200831.59b4212-1) ... Selecting previously unselected package golang-gopkg-check.v1-dev. Preparing to unpack .../110-golang-gopkg-check.v1-dev_0.0+git20200902.038fdea-1_all.deb ... Unpacking golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ... Selecting previously unselected package golang-gopkg-mgo.v2-dev. Preparing to unpack .../111-golang-gopkg-mgo.v2-dev_2016.08.01-7_all.deb ... Unpacking golang-gopkg-mgo.v2-dev (2016.08.01-7) ... Selecting previously unselected package golang-github-hashicorp-go-msgpack-dev. Preparing to unpack .../112-golang-github-hashicorp-go-msgpack-dev_0.5.5-1_all.deb ... Unpacking golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ... Setting up libpipeline1:arm64 (1.5.3-1) ... Setting up golang-gopkg-yaml.v2-dev (2.4.0-1) ... Setting up golang-github-hashicorp-go-uuid-dev (1.0.2-1) ... Setting up bsdextrautils (2.36.1-7) ... update-alternatives: using /usr/bin/write.ul to provide /usr/bin/write (write) in auto mode Setting up golang-github-creack-pty-dev (1.1.11-1) ... Setting up libicu67:arm64 (67.1-7) ... Setting up golang-github-golang-mock-dev (1.3.1-2) ... Setting up golang-github-stretchr-objx-dev (0.3.0-1) ... Setting up libmagic-mgc (1:5.39-3) ... Setting up golang-github-pkg-errors-dev (0.9.1-1) ... Setting up golang-github-hashicorp-golang-lru-dev (0.5.4-2) ... Setting up golang-1.15-src (1.15.9-5) ... Setting up libarchive-zip-perl (1.68-1) ... Setting up golang-gopkg-yaml.v3-dev (3.0.0~git20200121.a6ecf24-3) ... Setting up libdebhelper-perl (13.3.4) ... Setting up golang-golang-x-sys-dev (0.0~git20210124.22da62e-1) ... Setting up libmagic1:arm64 (1:5.39-3) ... Setting up golang-github-pmezard-go-difflib-dev (1.0.0-3) ... Setting up golang-github-modern-go-concurrent-dev (1.0.3-1.1) ... Setting up gettext-base (0.21-4) ... Setting up golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ... Setting up libprotobuf23:arm64 (3.12.4-1) ... Setting up file (1:5.39-3) ... Setting up golang-github-go-stack-stack-dev (1.8.0-1) ... Setting up golang-github-beorn7-perks-dev (1.0.1-1) ... Setting up golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ... Setting up libsasl2-modules-db:arm64 (2.1.27+dfsg-2.1) ... Setting up golang-github-cespare-xxhash-dev (2.1.1-1) ... Setting up golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ... Setting up golang-github-jpillora-backoff-dev (1.0.0-1.1) ... Setting up golang-github-davecgh-go-spew-dev (1.1.1-2) ... Setting up libprotobuf-lite23:arm64 (3.12.4-1) ... Setting up autotools-dev (20180224.1+nmu1) ... Setting up golang-github-pascaldekloe-goe-dev (0.1.0-2) ... Setting up golang-github-go-logfmt-logfmt-dev (0.5.0-2) ... Setting up golang-github-kr-text-dev (0.2.0-1) ... Setting up golang-github-stretchr-testify-dev (1.6.1-2) ... Setting up golang-github-hashicorp-go-immutable-radix-dev (1.3.0-1) ... Setting up libprotoc23:arm64 (3.12.4-1) ... Setting up libsigsegv2:arm64 (2.13-1) ... Setting up golang-golang-x-sync-dev (0.0~git20210220.036812b-1) ... Setting up autopoint (0.21-4) ... Setting up golang-glog-dev (0.0~git20160126.23def4e-3) ... Setting up libsasl2-2:arm64 (2.1.27+dfsg-2.1) ... Setting up golang-github-julienschmidt-httprouter-dev (1.3.0-1) ... Setting up zlib1g-dev:arm64 (1:1.2.11.dfsg-2) ... Setting up golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-2) ... Setting up sensible-utils (0.0.14) ... Setting up golang-golang-x-text-dev (0.3.6-1) ... Setting up libuchardet0:arm64 (0.0.7-1) ... Setting up golang-github-prometheus-procfs-dev (0.3.0-2) ... Setting up golang-github-alecthomas-units-dev (0.0~git20201120.1786d5e-1) ... Setting up golang-github-modern-go-reflect2-dev (1.0.1-1) ... Setting up libsub-override-perl (0.09-2) ... Setting up golang-github-dgrijalva-jwt-go-dev (3.2.0-3) ... Setting up golang-github-datadog-datadog-go-dev (2.1.0-3) ... Setting up golang-golang-x-xerrors-dev (0.0~git20191204.9bdfabe-1) ... Setting up golang-src:arm64 (2:1.15~1) ... Setting up golang-1.15-go (1.15.9-5) ... Setting up libelf1:arm64 (0.183-1) ... Setting up golang-github-vmihailenco-tagparser-dev (0.1.1-2) ... Setting up libxml2:arm64 (2.9.10+dfsg-6.7) ... Setting up golang-github-kr-pretty-dev (0.2.1+git20200831.59b4212-1) ... Setting up libfile-stripnondeterminism-perl (1.11.0-1) ... Setting up golang-github-google-go-cmp-dev (0.5.4-1) ... Setting up golang-github-mattn-go-isatty-dev (0.0.12-1) ... Setting up golang-github-sirupsen-logrus-dev (1.7.0-2) ... Setting up golang-github-opentracing-opentracing-go-dev (1.2.0-1) ... Setting up gettext (0.21-4) ... Setting up protobuf-compiler (3.12.4-1) ... Setting up golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ... Setting up golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-3) ... Setting up libtool (2.4.6-15) ... Setting up golang-go (2:1.15~1) ... Setting up golang-github-mattn-go-colorable-dev (0.1.7-1) ... Setting up m4 (1.4.18-5) ... Setting up golang-github-json-iterator-go-dev (1.1.10-2) ... Setting up golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ... Setting up golang-any:arm64 (2:1.15~1) ... Setting up intltool-debian (0.35.0+20060710.5) ... Setting up libprotobuf-dev:arm64 (3.12.4-1) ... Setting up libsasl2-dev (2.1.27+dfsg-2.1) ... Setting up golang-gopkg-mgo.v2-dev (2016.08.01-7) ... Setting up golang-goprotobuf-dev (1.3.4-2+b5) ... Setting up golang-golang-x-net-dev (1:0.0+git20210119.5f4716e+dfsg-4) ... Setting up autoconf (2.69-14) ... Setting up dh-strip-nondeterminism (1.11.0-1) ... Setting up dwz (0.13+20210201-1) ... Setting up groff-base (1.22.4-6) ... Setting up golang-github-prometheus-client-model-dev (0.2.0-2) ... Setting up golang-golang-x-oauth2-dev (0.0~git20190604.0f29369-2) ... Setting up automake (1:1.16.3-2) ... update-alternatives: using /usr/bin/automake-1.16 to provide /usr/bin/automake (automake) in auto mode Setting up golang-golang-x-time-dev (0.0+git20200630.3af7569-1) ... Setting up golang-github-fatih-color-dev (1.7.0-1) ... Setting up golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ... Setting up po-debconf (1.0.21+nmu1) ... Setting up golang-protobuf-extensions-dev (1.0.1-1) ... Setting up golang-google-cloud-compute-metadata-dev (0.56.0-1) ... Setting up man-db (2.9.4-2) ... Not building database; man-db/auto-update is not 'true'. Setting up dh-autoreconf (20) ... Setting up golang-golang-x-oauth2-google-dev (0.0~git20190604.0f29369-2) ... Setting up golang-github-hashicorp-go-hclog-dev (0.11.0-1) ... Setting up debhelper (13.3.4) ... Setting up golang-github-hashicorp-go-retryablehttp-dev (0.6.4-2) ... Setting up dh-golang (1.51) ... Setting up golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-3) ... Setting up golang-google-genproto-dev (0.0~git20200413.b5235f6-1) ... Setting up golang-github-prometheus-client-golang-dev (1.9.0-2) ... Setting up golang-github-armon-go-metrics-dev (0.3.4-1) ... Setting up golang-google-grpc-dev (1.27.1-1) ... Setting up golang-github-go-kit-kit-dev (0.6.0-3) ... Setting up golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-2) ... Setting up golang-github-prometheus-common-dev (0.15.0-2) ... Processing triggers for libc-bin (2.31-12) ... Reading package lists... Building dependency tree... Reading state information... Reading extended state information... Initializing package states... Writing extended state information... Building tag database... -> Finished parsing the build-deps Reading package lists... Building dependency tree... Reading state information... fakeroot is already the newest version (1.25.3-1.1). 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. I: Building the package I: Running cd /build/golang-github-hashicorp-raft-1.1.2/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-genchanges -S > ../golang-github-hashicorp-raft_1.1.2-1_source.changes dpkg-buildpackage: info: source package golang-github-hashicorp-raft dpkg-buildpackage: info: source version 1.1.2-1 dpkg-buildpackage: info: source distribution unstable dpkg-buildpackage: info: source changed by Dmitry Smirnov dpkg-source --before-build . dpkg-buildpackage: info: host architecture arm64 fakeroot debian/rules clean dh clean --buildsystem=golang --with=golang dh_auto_clean -O--buildsystem=golang dh_autoreconf_clean -O--buildsystem=golang dh_clean -O--buildsystem=golang debian/rules build dh build --buildsystem=golang --with=golang dh_update_autotools_config -O--buildsystem=golang dh_autoreconf -O--buildsystem=golang dh_auto_configure -O--buildsystem=golang dh_auto_build -O--buildsystem=golang cd obj-aarch64-linux-gnu && go install -trimpath -v -p 8 github.com/hashicorp/raft github.com/hashicorp/raft/bench internal/cpu internal/race runtime/internal/sys internal/unsafeheader runtime/internal/atomic unicode/utf8 sync/atomic unicode container/list crypto/internal/subtle runtime/internal/math internal/bytealg crypto/subtle math/bits internal/testlog encoding unicode/utf16 internal/nettrace vendor/golang.org/x/crypto/cryptobyte/asn1 vendor/golang.org/x/crypto/internal/subtle runtime/cgo golang.org/x/sys/internal/unsafeheader math runtime internal/reflectlite sync internal/singleflight math/rand errors sort internal/oserror io github.com/hashicorp/golang-lru/simplelru vendor/golang.org/x/net/dns/dnsmessage strconv syscall bytes hash strings crypto/internal/randutil hash/crc32 crypto/hmac hash/crc64 vendor/golang.org/x/crypto/hkdf crypto crypto/rc4 reflect vendor/golang.org/x/text/transform github.com/hashicorp/go-immutable-radix bufio regexp/syntax path html internal/syscall/execenv internal/syscall/unix time regexp context internal/poll encoding/binary internal/fmtsort os crypto/sha512 crypto/md5 crypto/cipher crypto/sha1 encoding/base64 crypto/ed25519/internal/edwards25519 crypto/sha256 encoding/pem vendor/golang.org/x/crypto/poly1305 golang.org/x/sys/unix crypto/des crypto/aes vendor/golang.org/x/crypto/chacha20 path/filepath fmt net os/signal runtime/debug vendor/golang.org/x/crypto/chacha20poly1305 io/ioutil log encoding/hex encoding/json compress/flate net/url math/big vendor/golang.org/x/crypto/curve25519 vendor/golang.org/x/text/unicode/bidi vendor/golang.org/x/net/http2/hpack vendor/golang.org/x/text/unicode/norm compress/gzip mime vendor/golang.org/x/text/secure/bidirule mime/quotedprintable net/http/internal github.com/mattn/go-isatty encoding/gob github.com/mattn/go-colorable go/token text/template/parse github.com/fatih/color flag runtime/trace github.com/hashicorp/go-hclog crypto/rand crypto/elliptic encoding/asn1 crypto/dsa vendor/golang.org/x/net/idna crypto/ed25519 crypto/rsa testing text/template crypto/x509/pkix vendor/golang.org/x/crypto/cryptobyte crypto/ecdsa html/template vendor/golang.org/x/net/http/httpproxy net/textproto crypto/x509 vendor/golang.org/x/net/http/httpguts mime/multipart crypto/tls net/http/httptrace net/http net/rpc github.com/armon/go-metrics github.com/hashicorp/go-msgpack/codec github.com/hashicorp/raft github.com/hashicorp/raft/bench dh_auto_test -O--buildsystem=golang cd obj-aarch64-linux-gnu && go test -vet=off -v -p 8 github.com/hashicorp/raft github.com/hashicorp/raft/bench === RUN TestCommitment_setVoters --- PASS: TestCommitment_setVoters (0.00s) === RUN TestCommitment_match_max --- PASS: TestCommitment_match_max (0.00s) === RUN TestCommitment_match_nonVoting --- PASS: TestCommitment_match_nonVoting (0.00s) === RUN TestCommitment_recalculate --- PASS: TestCommitment_recalculate (0.00s) === RUN TestCommitment_recalculate_startIndex --- PASS: TestCommitment_recalculate_startIndex (0.00s) === RUN TestCommitment_noVoterSanity --- PASS: TestCommitment_noVoterSanity (0.00s) === RUN TestCommitment_singleVoter --- PASS: TestCommitment_singleVoter (0.00s) === RUN TestConfiguration_Configuration_Clone --- PASS: TestConfiguration_Configuration_Clone (0.00s) === RUN TestConfiguration_configurations_Clone --- PASS: TestConfiguration_configurations_Clone (0.00s) === RUN TestConfiguration_hasVote --- PASS: TestConfiguration_hasVote (0.00s) === RUN TestConfiguration_checkConfiguration --- PASS: TestConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_nextConfiguration_table --- PASS: TestConfiguration_nextConfiguration_table (0.00s) === RUN TestConfiguration_nextConfiguration_prevIndex --- PASS: TestConfiguration_nextConfiguration_prevIndex (0.00s) === RUN TestConfiguration_nextConfiguration_checkConfiguration --- PASS: TestConfiguration_nextConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_encodeDecodePeers --- PASS: TestConfiguration_encodeDecodePeers (0.00s) === RUN TestConfiguration_encodeDecodeConfiguration --- PASS: TestConfiguration_encodeDecodeConfiguration (0.00s) === RUN TestDiscardSnapshotStoreImpl --- PASS: TestDiscardSnapshotStoreImpl (0.00s) === RUN TestDiscardSnapshotSinkImpl --- PASS: TestDiscardSnapshotSinkImpl (0.00s) === RUN TestFileSnapshotStoreImpl --- PASS: TestFileSnapshotStoreImpl (0.00s) === RUN TestFileSnapshotSinkImpl --- PASS: TestFileSnapshotSinkImpl (0.00s) === RUN TestFileSS_CreateSnapshotMissingParentDir testing.go:153: 2021-07-10T01:04:54.617Z [INFO] creating new snapshot: path=/tmp/raft504193843/raft905351414/snapshots/3-10-1625879094617.tmp --- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.00s) === RUN TestFileSS_CreateSnapshot testing.go:153: 2021-07-10T01:04:54.619Z [INFO] creating new snapshot: path=/tmp/raft606177757/snapshots/3-10-1625879094619.tmp testing.go:153: 2021-07-10T01:04:54.619Z [WARN] found temporary snapshot: name=3-10-1625879094619.tmp --- PASS: TestFileSS_CreateSnapshot (0.00s) === RUN TestFileSS_CancelSnapshot testing.go:153: 2021-07-10T01:04:54.620Z [INFO] creating new snapshot: path=/tmp/raft392395160/snapshots/3-10-1625879094620.tmp --- PASS: TestFileSS_CancelSnapshot (0.00s) === RUN TestFileSS_Retention testing.go:153: 2021-07-10T01:04:54.621Z [INFO] creating new snapshot: path=/tmp/raft545531927/snapshots/3-10-1625879094621.tmp testing.go:153: 2021-07-10T01:04:54.621Z [INFO] creating new snapshot: path=/tmp/raft545531927/snapshots/3-11-1625879094621.tmp testing.go:153: 2021-07-10T01:04:54.622Z [INFO] creating new snapshot: path=/tmp/raft545531927/snapshots/3-12-1625879094622.tmp testing.go:153: 2021-07-10T01:04:54.622Z [INFO] reaping snapshot: path=/tmp/raft545531927/snapshots/3-10-1625879094621 testing.go:153: 2021-07-10T01:04:54.622Z [INFO] creating new snapshot: path=/tmp/raft545531927/snapshots/3-13-1625879094622.tmp testing.go:153: 2021-07-10T01:04:54.623Z [INFO] reaping snapshot: path=/tmp/raft545531927/snapshots/3-11-1625879094621 testing.go:153: 2021-07-10T01:04:54.623Z [INFO] creating new snapshot: path=/tmp/raft545531927/snapshots/3-14-1625879094623.tmp testing.go:153: 2021-07-10T01:04:54.624Z [INFO] reaping snapshot: path=/tmp/raft545531927/snapshots/3-12-1625879094622 --- PASS: TestFileSS_Retention (0.00s) === RUN TestFileSS_BadPerm --- PASS: TestFileSS_BadPerm (0.00s) === RUN TestFileSS_MissingParentDir --- PASS: TestFileSS_MissingParentDir (0.00s) === RUN TestFileSS_Ordering testing.go:153: 2021-07-10T01:04:54.625Z [INFO] creating new snapshot: path=/tmp/raft138940254/snapshots/5-130350-1625879094625.tmp testing.go:153: 2021-07-10T01:04:54.626Z [INFO] creating new snapshot: path=/tmp/raft138940254/snapshots/36-204917-1625879094626.tmp --- PASS: TestFileSS_Ordering (0.00s) === RUN TestDeferFutureSuccess --- PASS: TestDeferFutureSuccess (0.00s) === RUN TestDeferFutureError --- PASS: TestDeferFutureError (0.00s) === RUN TestDeferFutureConcurrent --- PASS: TestDeferFutureConcurrent (0.00s) === RUN TestInmemSnapshotStoreImpl --- PASS: TestInmemSnapshotStoreImpl (0.00s) === RUN TestInmemSnapshotSinkImpl --- PASS: TestInmemSnapshotSinkImpl (0.00s) === RUN TestInmemSS_CreateSnapshot --- PASS: TestInmemSS_CreateSnapshot (0.00s) === RUN TestInmemSS_OpenSnapshotTwice --- PASS: TestInmemSS_OpenSnapshotTwice (0.00s) === RUN TestInmemTransportImpl --- PASS: TestInmemTransportImpl (0.00s) === RUN TestInmemTransportWriteTimeout --- PASS: TestInmemTransportWriteTimeout (0.01s) === RUN TestRaft_Integ --- SKIP: TestRaft_Integ (0.00s) === RUN TestLogCache --- PASS: TestLogCache (0.00s) === RUN TestNetworkTransport_CloseStreams --- PASS: TestNetworkTransport_CloseStreams (0.00s) === RUN TestNetworkTransport_StartStop --- PASS: TestNetworkTransport_StartStop (0.00s) === RUN TestNetworkTransport_Heartbeat_FastPath --- PASS: TestNetworkTransport_Heartbeat_FastPath (0.00s) === RUN TestNetworkTransport_AppendEntries --- PASS: TestNetworkTransport_AppendEntries (0.00s) === RUN TestNetworkTransport_AppendEntriesPipeline --- PASS: TestNetworkTransport_AppendEntriesPipeline (0.00s) === RUN TestNetworkTransport_AppendEntriesPipeline_CloseStreams net_transport_test.go:339: Disabled unreliable test --- SKIP: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.00s) === RUN TestNetworkTransport_RequestVote --- PASS: TestNetworkTransport_RequestVote (0.00s) === RUN TestNetworkTransport_InstallSnapshot --- PASS: TestNetworkTransport_InstallSnapshot (0.00s) === RUN TestNetworkTransport_EncodeDecode --- PASS: TestNetworkTransport_EncodeDecode (0.00s) === RUN TestNetworkTransport_EncodeDecode_AddressProvider --- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s) === RUN TestNetworkTransport_PooledConn --- PASS: TestNetworkTransport_PooledConn (0.00s) === RUN TestNetworkTransport_ListenBackoff net_transport_test.go:718: countingWriter: 2021-07-10T01:04:54.651Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-07-10T01:04:54.656Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-07-10T01:04:54.666Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-07-10T01:04:54.687Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-07-10T01:04:54.727Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-07-10T01:04:54.807Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-07-10T01:04:54.968Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-07-10T01:04:55.288Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-07-10T01:04:55.928Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-07-10T01:04:56.928Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-07-10T01:04:57.929Z [ERROR] test: failed to accept connection: error="intentional error in test" --- PASS: TestNetworkTransport_ListenBackoff (4.00s) === RUN TestPeersJSON_BadConfiguration --- PASS: TestPeersJSON_BadConfiguration (0.00s) === RUN TestPeersJSON_ReadPeersJSON --- PASS: TestPeersJSON_ReadPeersJSON (0.00s) === RUN TestPeersJSON_ReadConfigJSON --- PASS: TestPeersJSON_ReadConfigJSON (0.00s) === RUN TestRaft_StartStop testLoggerAdapter verbose: server-bb99c6b6-04d1-6795-0247-6e3c6bfa105b: 2021-07-10T01:04:58.653Z [INFO] server-bb99c6b6-04d1-6795-0247-6e3c6bfa105b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bb99c6b6-04d1-6795-0247-6e3c6bfa105b Address:bb99c6b6-04d1-6795-0247-6e3c6bfa105b}]" testing.go:149: server-bb99c6b6-04d1-6795-0247-6e3c6bfa105b: 2021-07-10T01:04:58.653Z [INFO] server-bb99c6b6-04d1-6795-0247-6e3c6bfa105b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bb99c6b6-04d1-6795-0247-6e3c6bfa105b Address:bb99c6b6-04d1-6795-0247-6e3c6bfa105b}]" --- PASS: TestRaft_StartStop (0.00s) === RUN TestRaft_AfterShutdown testLoggerAdapter verbose: server-25a1494d-f4ca-4ea3-c55d-acac012f35b4: 2021-07-10T01:04:58.655Z [INFO] server-25a1494d-f4ca-4ea3-c55d-acac012f35b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25a1494d-f4ca-4ea3-c55d-acac012f35b4 Address:25a1494d-f4ca-4ea3-c55d-acac012f35b4}]" testing.go:149: server-25a1494d-f4ca-4ea3-c55d-acac012f35b4: 2021-07-10T01:04:58.655Z [INFO] server-25a1494d-f4ca-4ea3-c55d-acac012f35b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25a1494d-f4ca-4ea3-c55d-acac012f35b4 Address:25a1494d-f4ca-4ea3-c55d-acac012f35b4}]" --- PASS: TestRaft_AfterShutdown (0.00s) === RUN TestRaft_LiveBootstrap testLoggerAdapter verbose: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.656Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: initial configuration: index=0 servers=[] testing.go:149: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.656Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-ca294112-77c7-33bf-2167-2c51be1c818f: 2021-07-10T01:04:58.656Z [INFO] server-ca294112-77c7-33bf-2167-2c51be1c818f: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.656Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: entering follower state: follower="Node at 3bc7bf7d-e3f3-b648-cf09-5af67d9cf607 [Follower]" leader= testing.go:149: server-ca294112-77c7-33bf-2167-2c51be1c818f: 2021-07-10T01:04:58.656Z [INFO] server-ca294112-77c7-33bf-2167-2c51be1c818f: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5: 2021-07-10T01:04:58.657Z [INFO] server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5: initial configuration: index=0 servers=[] testing.go:149: server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5: 2021-07-10T01:04:58.657Z [INFO] server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5: initial configuration: index=0 servers=[] testing.go:149: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.656Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: entering follower state: follower="Node at 3bc7bf7d-e3f3-b648-cf09-5af67d9cf607 [Follower]" leader= testLoggerAdapter verbose: cluster: 2021-07-10T01:04:58.657Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-07-10T01:04:58.657Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-ca294112-77c7-33bf-2167-2c51be1c818f: 2021-07-10T01:04:58.660Z [INFO] server-ca294112-77c7-33bf-2167-2c51be1c818f: entering follower state: follower="Node at ca294112-77c7-33bf-2167-2c51be1c818f [Follower]" leader= testing.go:149: server-ca294112-77c7-33bf-2167-2c51be1c818f: 2021-07-10T01:04:58.660Z [INFO] server-ca294112-77c7-33bf-2167-2c51be1c818f: entering follower state: follower="Node at ca294112-77c7-33bf-2167-2c51be1c818f [Follower]" leader= testLoggerAdapter verbose: server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5: 2021-07-10T01:04:58.660Z [INFO] server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5: entering follower state: follower="Node at 8b7f9582-6d45-2c61-320b-9123cdf5dfa5 [Follower]" leader= testing.go:149: server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5: 2021-07-10T01:04:58.660Z [INFO] server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5: entering follower state: follower="Node at 8b7f9582-6d45-2c61-320b-9123cdf5dfa5 [Follower]" leader= testLoggerAdapter verbose: server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5: 2021-07-10T01:04:58.733Z [WARN] server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5: no known peers, aborting election testing.go:149: server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5: 2021-07-10T01:04:58.733Z [WARN] server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5: no known peers, aborting election testLoggerAdapter verbose: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.737Z [WARN] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.737Z [WARN] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.737Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: entering candidate state: node="Node at 3bc7bf7d-e3f3-b648-cf09-5af67d9cf607 [Candidate]" term=2 testing.go:149: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.737Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: entering candidate state: node="Node at 3bc7bf7d-e3f3-b648-cf09-5af67d9cf607 [Candidate]" term=2 testLoggerAdapter verbose: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.737Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: election won: tally=2 testing.go:149: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.737Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: election won: tally=2 testLoggerAdapter verbose: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.737Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: entering leader state: leader="Node at 3bc7bf7d-e3f3-b648-cf09-5af67d9cf607 [Leader]" testing.go:149: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.737Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: entering leader state: leader="Node at 3bc7bf7d-e3f3-b648-cf09-5af67d9cf607 [Leader]" testLoggerAdapter verbose: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.737Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: added peer, starting replication: peer=server-ca294112-77c7-33bf-2167-2c51be1c818f testing.go:149: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.737Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: added peer, starting replication: peer=server-ca294112-77c7-33bf-2167-2c51be1c818f testLoggerAdapter verbose: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.738Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: added peer, starting replication: peer=server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5 testing.go:149: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.738Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: added peer, starting replication: peer=server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5 testLoggerAdapter verbose: server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5: 2021-07-10T01:04:58.738Z [WARN] server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5: failed to get previous log: previous-index=1 last-index=0 error="log not found" testing.go:149: server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5: 2021-07-10T01:04:58.738Z [WARN] server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5: failed to get previous log: previous-index=1 last-index=0 error="log not found" testLoggerAdapter verbose: server-ca294112-77c7-33bf-2167-2c51be1c818f: 2021-07-10T01:04:58.738Z [WARN] server-ca294112-77c7-33bf-2167-2c51be1c818f: failed to get previous log: previous-index=1 last-index=0 error="log not found" testing.go:149: server-ca294112-77c7-33bf-2167-2c51be1c818f: 2021-07-10T01:04:58.738Z [WARN] server-ca294112-77c7-33bf-2167-2c51be1c818f: failed to get previous log: previous-index=1 last-index=0 error="log not found" testLoggerAdapter verbose: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.738Z [WARN] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: appendEntries rejected, sending older logs: peer="{Voter server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5 8b7f9582-6d45-2c61-320b-9123cdf5dfa5}" next=1 testing.go:149: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.738Z [WARN] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: appendEntries rejected, sending older logs: peer="{Voter server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5 8b7f9582-6d45-2c61-320b-9123cdf5dfa5}" next=1 testLoggerAdapter verbose: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.738Z [WARN] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: appendEntries rejected, sending older logs: peer="{Voter server-ca294112-77c7-33bf-2167-2c51be1c818f ca294112-77c7-33bf-2167-2c51be1c818f}" next=1 testing.go:149: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.738Z [WARN] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: appendEntries rejected, sending older logs: peer="{Voter server-ca294112-77c7-33bf-2167-2c51be1c818f ca294112-77c7-33bf-2167-2c51be1c818f}" next=1 testLoggerAdapter verbose: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.738Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: pipelining replication: peer="{Voter server-ca294112-77c7-33bf-2167-2c51be1c818f ca294112-77c7-33bf-2167-2c51be1c818f}" testing.go:149: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.738Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: pipelining replication: peer="{Voter server-ca294112-77c7-33bf-2167-2c51be1c818f ca294112-77c7-33bf-2167-2c51be1c818f}" testLoggerAdapter verbose: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.738Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: pipelining replication: peer="{Voter server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5 8b7f9582-6d45-2c61-320b-9123cdf5dfa5}" testing.go:149: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.738Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: pipelining replication: peer="{Voter server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5 8b7f9582-6d45-2c61-320b-9123cdf5dfa5}" testLoggerAdapter verbose: cluster: 2021-07-10T01:04:58.843Z [INFO] stable state for Follower reached at 2021-07-10 01:04:58.738227512 +0000 UTC m=+4.126385522 (2 nodes), 80.466228ms from start of poll, 81.749342ms from cluster start. Timeout at 2021-07-10 01:04:58.84329306 +0000 UTC m=+4.231451030, 105.065508ms after stability testing.go:149: cluster: 2021-07-10T01:04:58.843Z [INFO] stable state for Follower reached at 2021-07-10 01:04:58.738227512 +0000 UTC m=+4.126385522 (2 nodes), 80.466228ms from start of poll, 81.749342ms from cluster start. Timeout at 2021-07-10 01:04:58.84329306 +0000 UTC m=+4.231451030, 105.065508ms after stability testLoggerAdapter verbose: cluster: 2021-07-10T01:04:58.843Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:04:58.843Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-07-10T01:04:58.948Z [INFO] stable state for Leader reached at 2021-07-10 01:04:58.843585883 +0000 UTC m=+4.231743853 (1 nodes), 2.98µs from start of poll, 187.107673ms from cluster start. Timeout at 2021-07-10 01:04:58.948650731 +0000 UTC m=+4.336808701, 105.064848ms after stability testing.go:149: cluster: 2021-07-10T01:04:58.948Z [INFO] stable state for Leader reached at 2021-07-10 01:04:58.843585883 +0000 UTC m=+4.231743853 (1 nodes), 2.98µs from start of poll, 187.107673ms from cluster start. Timeout at 2021-07-10 01:04:58.948650731 +0000 UTC m=+4.336808701, 105.064848ms after stability testLoggerAdapter verbose: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.959Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: aborting pipeline replication: peer="{Voter server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5 8b7f9582-6d45-2c61-320b-9123cdf5dfa5}" testing.go:149: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.959Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: aborting pipeline replication: peer="{Voter server-8b7f9582-6d45-2c61-320b-9123cdf5dfa5 8b7f9582-6d45-2c61-320b-9123cdf5dfa5}" testLoggerAdapter verbose: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.959Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: aborting pipeline replication: peer="{Voter server-ca294112-77c7-33bf-2167-2c51be1c818f ca294112-77c7-33bf-2167-2c51be1c818f}" testing.go:149: server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: 2021-07-10T01:04:58.959Z [INFO] server-3bc7bf7d-e3f3-b648-cf09-5af67d9cf607: aborting pipeline replication: peer="{Voter server-ca294112-77c7-33bf-2167-2c51be1c818f ca294112-77c7-33bf-2167-2c51be1c818f}" --- PASS: TestRaft_LiveBootstrap (0.30s) === RUN TestRaft_RecoverCluster_NoState testLoggerAdapter verbose: server-ebd8bc83-dd72-d521-8a83-974dfd53fff6: 2021-07-10T01:04:58.960Z [INFO] server-ebd8bc83-dd72-d521-8a83-974dfd53fff6: initial configuration: index=0 servers=[] testing.go:149: server-ebd8bc83-dd72-d521-8a83-974dfd53fff6: 2021-07-10T01:04:58.960Z [INFO] server-ebd8bc83-dd72-d521-8a83-974dfd53fff6: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-ebd8bc83-dd72-d521-8a83-974dfd53fff6: 2021-07-10T01:04:58.960Z [INFO] server-ebd8bc83-dd72-d521-8a83-974dfd53fff6: entering follower state: follower="Node at ebd8bc83-dd72-d521-8a83-974dfd53fff6 [Follower]" leader= testing.go:149: server-ebd8bc83-dd72-d521-8a83-974dfd53fff6: 2021-07-10T01:04:58.960Z [INFO] server-ebd8bc83-dd72-d521-8a83-974dfd53fff6: entering follower state: follower="Node at ebd8bc83-dd72-d521-8a83-974dfd53fff6 [Follower]" leader= --- PASS: TestRaft_RecoverCluster_NoState (0.00s) === RUN TestRaft_RecoverCluster raft_test.go:119: Disabled unreliable test --- SKIP: TestRaft_RecoverCluster (0.00s) === RUN TestRaft_HasExistingState testLoggerAdapter verbose: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:58.961Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4510cfdf-e38a-e76d-5524-8f0140a945fc Address:4510cfdf-e38a-e76d-5524-8f0140a945fc} {Suffrage:Voter ID:server-b07c6965-7364-ac4c-74cc-5d6997bcc545 Address:b07c6965-7364-ac4c-74cc-5d6997bcc545}]" testing.go:149: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:58.961Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4510cfdf-e38a-e76d-5524-8f0140a945fc Address:4510cfdf-e38a-e76d-5524-8f0140a945fc} {Suffrage:Voter ID:server-b07c6965-7364-ac4c-74cc-5d6997bcc545 Address:b07c6965-7364-ac4c-74cc-5d6997bcc545}]" testLoggerAdapter verbose: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:58.962Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: entering follower state: follower="Node at 4510cfdf-e38a-e76d-5524-8f0140a945fc [Follower]" leader= testing.go:149: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:58.962Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: entering follower state: follower="Node at 4510cfdf-e38a-e76d-5524-8f0140a945fc [Follower]" leader= testLoggerAdapter verbose: server-b07c6965-7364-ac4c-74cc-5d6997bcc545: 2021-07-10T01:04:58.962Z [INFO] server-b07c6965-7364-ac4c-74cc-5d6997bcc545: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4510cfdf-e38a-e76d-5524-8f0140a945fc Address:4510cfdf-e38a-e76d-5524-8f0140a945fc} {Suffrage:Voter ID:server-b07c6965-7364-ac4c-74cc-5d6997bcc545 Address:b07c6965-7364-ac4c-74cc-5d6997bcc545}]" testing.go:149: server-b07c6965-7364-ac4c-74cc-5d6997bcc545: 2021-07-10T01:04:58.962Z [INFO] server-b07c6965-7364-ac4c-74cc-5d6997bcc545: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4510cfdf-e38a-e76d-5524-8f0140a945fc Address:4510cfdf-e38a-e76d-5524-8f0140a945fc} {Suffrage:Voter ID:server-b07c6965-7364-ac4c-74cc-5d6997bcc545 Address:b07c6965-7364-ac4c-74cc-5d6997bcc545}]" testLoggerAdapter verbose: server-b07c6965-7364-ac4c-74cc-5d6997bcc545: 2021-07-10T01:04:58.962Z [INFO] server-b07c6965-7364-ac4c-74cc-5d6997bcc545: entering follower state: follower="Node at b07c6965-7364-ac4c-74cc-5d6997bcc545 [Follower]" leader= testing.go:149: server-b07c6965-7364-ac4c-74cc-5d6997bcc545: 2021-07-10T01:04:58.962Z [INFO] server-b07c6965-7364-ac4c-74cc-5d6997bcc545: entering follower state: follower="Node at b07c6965-7364-ac4c-74cc-5d6997bcc545 [Follower]" leader= testLoggerAdapter verbose: server-b1b024f2-7c48-1049-2a38-bdbbc303259e: 2021-07-10T01:04:58.962Z [INFO] server-b1b024f2-7c48-1049-2a38-bdbbc303259e: initial configuration: index=0 servers=[] testing.go:149: server-b1b024f2-7c48-1049-2a38-bdbbc303259e: 2021-07-10T01:04:58.962Z [INFO] server-b1b024f2-7c48-1049-2a38-bdbbc303259e: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-b1b024f2-7c48-1049-2a38-bdbbc303259e: 2021-07-10T01:04:58.962Z [INFO] server-b1b024f2-7c48-1049-2a38-bdbbc303259e: entering follower state: follower="Node at b1b024f2-7c48-1049-2a38-bdbbc303259e [Follower]" leader= testing.go:149: server-b1b024f2-7c48-1049-2a38-bdbbc303259e: 2021-07-10T01:04:58.962Z [INFO] server-b1b024f2-7c48-1049-2a38-bdbbc303259e: entering follower state: follower="Node at b1b024f2-7c48-1049-2a38-bdbbc303259e [Follower]" leader= testLoggerAdapter verbose: cluster: 2021-07-10T01:04:58.962Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:04:58.962Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-b1b024f2-7c48-1049-2a38-bdbbc303259e: 2021-07-10T01:04:59.022Z [WARN] server-b1b024f2-7c48-1049-2a38-bdbbc303259e: no known peers, aborting election testing.go:149: server-b1b024f2-7c48-1049-2a38-bdbbc303259e: 2021-07-10T01:04:59.022Z [WARN] server-b1b024f2-7c48-1049-2a38-bdbbc303259e: no known peers, aborting election testLoggerAdapter verbose: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.023Z [WARN] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.023Z [WARN] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.023Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: entering candidate state: node="Node at 4510cfdf-e38a-e76d-5524-8f0140a945fc [Candidate]" term=2 testing.go:149: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.023Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: entering candidate state: node="Node at 4510cfdf-e38a-e76d-5524-8f0140a945fc [Candidate]" term=2 testLoggerAdapter verbose: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.023Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: election won: tally=2 testing.go:149: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.023Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: election won: tally=2 testLoggerAdapter verbose: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.023Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: entering leader state: leader="Node at 4510cfdf-e38a-e76d-5524-8f0140a945fc [Leader]" testing.go:149: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.023Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: entering leader state: leader="Node at 4510cfdf-e38a-e76d-5524-8f0140a945fc [Leader]" testLoggerAdapter verbose: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.023Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: added peer, starting replication: peer=server-b07c6965-7364-ac4c-74cc-5d6997bcc545 testing.go:149: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.023Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: added peer, starting replication: peer=server-b07c6965-7364-ac4c-74cc-5d6997bcc545 testLoggerAdapter verbose: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.023Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: pipelining replication: peer="{Voter server-b07c6965-7364-ac4c-74cc-5d6997bcc545 b07c6965-7364-ac4c-74cc-5d6997bcc545}" testing.go:149: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.023Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: pipelining replication: peer="{Voter server-b07c6965-7364-ac4c-74cc-5d6997bcc545 b07c6965-7364-ac4c-74cc-5d6997bcc545}" testLoggerAdapter verbose: cluster: 2021-07-10T01:04:59.128Z [INFO] stable state for Leader reached at 2021-07-10 01:04:59.02356428 +0000 UTC m=+4.411722250 (1 nodes), 60.581478ms from start of poll, 61.968253ms from cluster start. Timeout at 2021-07-10 01:04:59.128642848 +0000 UTC m=+4.516800818, 105.078568ms after stability testing.go:149: cluster: 2021-07-10T01:04:59.128Z [INFO] stable state for Leader reached at 2021-07-10 01:04:59.02356428 +0000 UTC m=+4.411722250 (1 nodes), 60.581478ms from start of poll, 61.968253ms from cluster start. Timeout at 2021-07-10 01:04:59.128642848 +0000 UTC m=+4.516800818, 105.078568ms after stability testLoggerAdapter verbose: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.128Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: updating configuration: command=AddStaging server-id=server-b1b024f2-7c48-1049-2a38-bdbbc303259e server-addr=b1b024f2-7c48-1049-2a38-bdbbc303259e servers="[{Suffrage:Voter ID:server-4510cfdf-e38a-e76d-5524-8f0140a945fc Address:4510cfdf-e38a-e76d-5524-8f0140a945fc} {Suffrage:Voter ID:server-b07c6965-7364-ac4c-74cc-5d6997bcc545 Address:b07c6965-7364-ac4c-74cc-5d6997bcc545} {Suffrage:Voter ID:server-b1b024f2-7c48-1049-2a38-bdbbc303259e Address:b1b024f2-7c48-1049-2a38-bdbbc303259e}]" testing.go:149: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.128Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: updating configuration: command=AddStaging server-id=server-b1b024f2-7c48-1049-2a38-bdbbc303259e server-addr=b1b024f2-7c48-1049-2a38-bdbbc303259e servers="[{Suffrage:Voter ID:server-4510cfdf-e38a-e76d-5524-8f0140a945fc Address:4510cfdf-e38a-e76d-5524-8f0140a945fc} {Suffrage:Voter ID:server-b07c6965-7364-ac4c-74cc-5d6997bcc545 Address:b07c6965-7364-ac4c-74cc-5d6997bcc545} {Suffrage:Voter ID:server-b1b024f2-7c48-1049-2a38-bdbbc303259e Address:b1b024f2-7c48-1049-2a38-bdbbc303259e}]" testLoggerAdapter verbose: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.129Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: added peer, starting replication: peer=server-b1b024f2-7c48-1049-2a38-bdbbc303259e testing.go:149: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.129Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: added peer, starting replication: peer=server-b1b024f2-7c48-1049-2a38-bdbbc303259e testLoggerAdapter verbose: server-b1b024f2-7c48-1049-2a38-bdbbc303259e: 2021-07-10T01:04:59.129Z [WARN] server-b1b024f2-7c48-1049-2a38-bdbbc303259e: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: server-b1b024f2-7c48-1049-2a38-bdbbc303259e: 2021-07-10T01:04:59.129Z [WARN] server-b1b024f2-7c48-1049-2a38-bdbbc303259e: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.129Z [WARN] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: appendEntries rejected, sending older logs: peer="{Voter server-b1b024f2-7c48-1049-2a38-bdbbc303259e b1b024f2-7c48-1049-2a38-bdbbc303259e}" next=1 testing.go:149: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.129Z [WARN] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: appendEntries rejected, sending older logs: peer="{Voter server-b1b024f2-7c48-1049-2a38-bdbbc303259e b1b024f2-7c48-1049-2a38-bdbbc303259e}" next=1 testLoggerAdapter verbose: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.129Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: pipelining replication: peer="{Voter server-b1b024f2-7c48-1049-2a38-bdbbc303259e b1b024f2-7c48-1049-2a38-bdbbc303259e}" testing.go:149: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.129Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: pipelining replication: peer="{Voter server-b1b024f2-7c48-1049-2a38-bdbbc303259e b1b024f2-7c48-1049-2a38-bdbbc303259e}" testLoggerAdapter verbose: cluster: 2021-07-10T01:04:59.134Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:04:59.134Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-07-10T01:04:59.239Z [INFO] stable state for Leader reached at 2021-07-10 01:04:59.13453515 +0000 UTC m=+4.522693120 (1 nodes), 2.04µs from start of poll, 172.939123ms from cluster start. Timeout at 2021-07-10 01:04:59.239607078 +0000 UTC m=+4.627765048, 105.071928ms after stability testing.go:149: cluster: 2021-07-10T01:04:59.239Z [INFO] stable state for Leader reached at 2021-07-10 01:04:59.13453515 +0000 UTC m=+4.522693120 (1 nodes), 2.04µs from start of poll, 172.939123ms from cluster start. Timeout at 2021-07-10 01:04:59.239607078 +0000 UTC m=+4.627765048, 105.071928ms after stability testLoggerAdapter verbose: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.239Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: aborting pipeline replication: peer="{Voter server-b1b024f2-7c48-1049-2a38-bdbbc303259e b1b024f2-7c48-1049-2a38-bdbbc303259e}" testing.go:149: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.239Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: aborting pipeline replication: peer="{Voter server-b1b024f2-7c48-1049-2a38-bdbbc303259e b1b024f2-7c48-1049-2a38-bdbbc303259e}" testLoggerAdapter verbose: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.239Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: aborting pipeline replication: peer="{Voter server-b07c6965-7364-ac4c-74cc-5d6997bcc545 b07c6965-7364-ac4c-74cc-5d6997bcc545}" testing.go:149: server-4510cfdf-e38a-e76d-5524-8f0140a945fc: 2021-07-10T01:04:59.239Z [INFO] server-4510cfdf-e38a-e76d-5524-8f0140a945fc: aborting pipeline replication: peer="{Voter server-b07c6965-7364-ac4c-74cc-5d6997bcc545 b07c6965-7364-ac4c-74cc-5d6997bcc545}" --- PASS: TestRaft_HasExistingState (0.28s) === RUN TestRaft_SingleNode testLoggerAdapter verbose: server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: 2021-07-10T01:04:59.241Z [INFO] server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f Address:47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f}]" testing.go:149: server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: 2021-07-10T01:04:59.241Z [INFO] server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f Address:47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f}]" testLoggerAdapter verbose: server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: 2021-07-10T01:04:59.241Z [INFO] server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: entering follower state: follower="Node at 47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f [Follower]" leader= testing.go:149: server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: 2021-07-10T01:04:59.241Z [INFO] server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: entering follower state: follower="Node at 47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f [Follower]" leader= testLoggerAdapter verbose: server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: 2021-07-10T01:04:59.302Z [WARN] server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: 2021-07-10T01:04:59.302Z [WARN] server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: 2021-07-10T01:04:59.302Z [INFO] server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: entering candidate state: node="Node at 47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f [Candidate]" term=2 testing.go:149: server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: 2021-07-10T01:04:59.302Z [INFO] server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: entering candidate state: node="Node at 47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f [Candidate]" term=2 testLoggerAdapter verbose: server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: 2021-07-10T01:04:59.302Z [INFO] server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: election won: tally=1 testing.go:149: server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: 2021-07-10T01:04:59.302Z [INFO] server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: election won: tally=1 testLoggerAdapter verbose: server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: 2021-07-10T01:04:59.303Z [INFO] server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: entering leader state: leader="Node at 47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f [Leader]" testing.go:149: server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: 2021-07-10T01:04:59.303Z [INFO] server-47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f: entering leader state: leader="Node at 47fa6dee-840a-7a96-2f4c-ea49ce3c2a7f [Leader]" --- PASS: TestRaft_SingleNode (0.06s) === RUN TestRaft_TripleNode testLoggerAdapter verbose: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.304Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cdd76ee3-70b9-f7fa-aeb0-3524550db012 Address:cdd76ee3-70b9-f7fa-aeb0-3524550db012} {Suffrage:Voter ID:server-ece067ae-beeb-1de5-b057-f7feaf981b24 Address:ece067ae-beeb-1de5-b057-f7feaf981b24} {Suffrage:Voter ID:server-d59a0575-eef2-c732-f077-0e80a179a16a Address:d59a0575-eef2-c732-f077-0e80a179a16a}]" testing.go:149: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.304Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cdd76ee3-70b9-f7fa-aeb0-3524550db012 Address:cdd76ee3-70b9-f7fa-aeb0-3524550db012} {Suffrage:Voter ID:server-ece067ae-beeb-1de5-b057-f7feaf981b24 Address:ece067ae-beeb-1de5-b057-f7feaf981b24} {Suffrage:Voter ID:server-d59a0575-eef2-c732-f077-0e80a179a16a Address:d59a0575-eef2-c732-f077-0e80a179a16a}]" testLoggerAdapter verbose: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.304Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: entering follower state: follower="Node at cdd76ee3-70b9-f7fa-aeb0-3524550db012 [Follower]" leader= testing.go:149: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.304Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: entering follower state: follower="Node at cdd76ee3-70b9-f7fa-aeb0-3524550db012 [Follower]" leader= testLoggerAdapter verbose: server-ece067ae-beeb-1de5-b057-f7feaf981b24: 2021-07-10T01:04:59.304Z [INFO] server-ece067ae-beeb-1de5-b057-f7feaf981b24: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cdd76ee3-70b9-f7fa-aeb0-3524550db012 Address:cdd76ee3-70b9-f7fa-aeb0-3524550db012} {Suffrage:Voter ID:server-ece067ae-beeb-1de5-b057-f7feaf981b24 Address:ece067ae-beeb-1de5-b057-f7feaf981b24} {Suffrage:Voter ID:server-d59a0575-eef2-c732-f077-0e80a179a16a Address:d59a0575-eef2-c732-f077-0e80a179a16a}]" testing.go:149: server-ece067ae-beeb-1de5-b057-f7feaf981b24: 2021-07-10T01:04:59.304Z [INFO] server-ece067ae-beeb-1de5-b057-f7feaf981b24: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cdd76ee3-70b9-f7fa-aeb0-3524550db012 Address:cdd76ee3-70b9-f7fa-aeb0-3524550db012} {Suffrage:Voter ID:server-ece067ae-beeb-1de5-b057-f7feaf981b24 Address:ece067ae-beeb-1de5-b057-f7feaf981b24} {Suffrage:Voter ID:server-d59a0575-eef2-c732-f077-0e80a179a16a Address:d59a0575-eef2-c732-f077-0e80a179a16a}]" testLoggerAdapter verbose: server-ece067ae-beeb-1de5-b057-f7feaf981b24: 2021-07-10T01:04:59.304Z [INFO] server-ece067ae-beeb-1de5-b057-f7feaf981b24: entering follower state: follower="Node at ece067ae-beeb-1de5-b057-f7feaf981b24 [Follower]" leader= testing.go:149: server-ece067ae-beeb-1de5-b057-f7feaf981b24: 2021-07-10T01:04:59.304Z [INFO] server-ece067ae-beeb-1de5-b057-f7feaf981b24: entering follower state: follower="Node at ece067ae-beeb-1de5-b057-f7feaf981b24 [Follower]" leader= testLoggerAdapter verbose: server-d59a0575-eef2-c732-f077-0e80a179a16a: 2021-07-10T01:04:59.304Z [INFO] server-d59a0575-eef2-c732-f077-0e80a179a16a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cdd76ee3-70b9-f7fa-aeb0-3524550db012 Address:cdd76ee3-70b9-f7fa-aeb0-3524550db012} {Suffrage:Voter ID:server-ece067ae-beeb-1de5-b057-f7feaf981b24 Address:ece067ae-beeb-1de5-b057-f7feaf981b24} {Suffrage:Voter ID:server-d59a0575-eef2-c732-f077-0e80a179a16a Address:d59a0575-eef2-c732-f077-0e80a179a16a}]" testing.go:149: server-d59a0575-eef2-c732-f077-0e80a179a16a: 2021-07-10T01:04:59.304Z [INFO] server-d59a0575-eef2-c732-f077-0e80a179a16a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cdd76ee3-70b9-f7fa-aeb0-3524550db012 Address:cdd76ee3-70b9-f7fa-aeb0-3524550db012} {Suffrage:Voter ID:server-ece067ae-beeb-1de5-b057-f7feaf981b24 Address:ece067ae-beeb-1de5-b057-f7feaf981b24} {Suffrage:Voter ID:server-d59a0575-eef2-c732-f077-0e80a179a16a Address:d59a0575-eef2-c732-f077-0e80a179a16a}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:04:59.305Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-07-10T01:04:59.305Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-d59a0575-eef2-c732-f077-0e80a179a16a: 2021-07-10T01:04:59.305Z [INFO] server-d59a0575-eef2-c732-f077-0e80a179a16a: entering follower state: follower="Node at d59a0575-eef2-c732-f077-0e80a179a16a [Follower]" leader= testing.go:149: server-d59a0575-eef2-c732-f077-0e80a179a16a: 2021-07-10T01:04:59.305Z [INFO] server-d59a0575-eef2-c732-f077-0e80a179a16a: entering follower state: follower="Node at d59a0575-eef2-c732-f077-0e80a179a16a [Follower]" leader= testLoggerAdapter verbose: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.360Z [WARN] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.360Z [WARN] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.360Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: entering candidate state: node="Node at cdd76ee3-70b9-f7fa-aeb0-3524550db012 [Candidate]" term=2 testing.go:149: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.360Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: entering candidate state: node="Node at cdd76ee3-70b9-f7fa-aeb0-3524550db012 [Candidate]" term=2 testLoggerAdapter verbose: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.360Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: election won: tally=2 testing.go:149: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.360Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: election won: tally=2 testLoggerAdapter verbose: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.360Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: entering leader state: leader="Node at cdd76ee3-70b9-f7fa-aeb0-3524550db012 [Leader]" testing.go:149: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.360Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: entering leader state: leader="Node at cdd76ee3-70b9-f7fa-aeb0-3524550db012 [Leader]" testLoggerAdapter verbose: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.360Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: added peer, starting replication: peer=server-ece067ae-beeb-1de5-b057-f7feaf981b24 testing.go:149: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.360Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: added peer, starting replication: peer=server-ece067ae-beeb-1de5-b057-f7feaf981b24 testLoggerAdapter verbose: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.360Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: added peer, starting replication: peer=server-d59a0575-eef2-c732-f077-0e80a179a16a testing.go:149: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.360Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: added peer, starting replication: peer=server-d59a0575-eef2-c732-f077-0e80a179a16a testLoggerAdapter verbose: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.360Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: pipelining replication: peer="{Voter server-d59a0575-eef2-c732-f077-0e80a179a16a d59a0575-eef2-c732-f077-0e80a179a16a}" testing.go:149: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.360Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: pipelining replication: peer="{Voter server-d59a0575-eef2-c732-f077-0e80a179a16a d59a0575-eef2-c732-f077-0e80a179a16a}" testLoggerAdapter verbose: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.361Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: pipelining replication: peer="{Voter server-ece067ae-beeb-1de5-b057-f7feaf981b24 ece067ae-beeb-1de5-b057-f7feaf981b24}" testing.go:149: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.361Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: pipelining replication: peer="{Voter server-ece067ae-beeb-1de5-b057-f7feaf981b24 ece067ae-beeb-1de5-b057-f7feaf981b24}" testLoggerAdapter verbose: cluster: 2021-07-10T01:04:59.465Z [INFO] stable state for Follower reached at 2021-07-10 01:04:59.360798115 +0000 UTC m=+4.748956085 (2 nodes), 55.687307ms from start of poll, 56.708777ms from cluster start. Timeout at 2021-07-10 01:04:59.465877843 +0000 UTC m=+4.854035813, 105.079728ms after stability testing.go:149: cluster: 2021-07-10T01:04:59.465Z [INFO] stable state for Follower reached at 2021-07-10 01:04:59.360798115 +0000 UTC m=+4.748956085 (2 nodes), 55.687307ms from start of poll, 56.708777ms from cluster start. Timeout at 2021-07-10 01:04:59.465877843 +0000 UTC m=+4.854035813, 105.079728ms after stability testLoggerAdapter verbose: cluster: 2021-07-10T01:04:59.466Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:04:59.466Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-07-10T01:04:59.571Z [INFO] stable state for Leader reached at 2021-07-10 01:04:59.466124766 +0000 UTC m=+4.854282736 (1 nodes), 2.16µs from start of poll, 162.035428ms from cluster start. Timeout at 2021-07-10 01:04:59.571186353 +0000 UTC m=+4.959344323, 105.061587ms after stability testing.go:149: cluster: 2021-07-10T01:04:59.571Z [INFO] stable state for Leader reached at 2021-07-10 01:04:59.466124766 +0000 UTC m=+4.854282736 (1 nodes), 2.16µs from start of poll, 162.035428ms from cluster start. Timeout at 2021-07-10 01:04:59.571186353 +0000 UTC m=+4.959344323, 105.061587ms after stability testLoggerAdapter verbose: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.581Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: aborting pipeline replication: peer="{Voter server-d59a0575-eef2-c732-f077-0e80a179a16a d59a0575-eef2-c732-f077-0e80a179a16a}" testing.go:149: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.581Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: aborting pipeline replication: peer="{Voter server-d59a0575-eef2-c732-f077-0e80a179a16a d59a0575-eef2-c732-f077-0e80a179a16a}" testLoggerAdapter verbose: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.581Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: aborting pipeline replication: peer="{Voter server-ece067ae-beeb-1de5-b057-f7feaf981b24 ece067ae-beeb-1de5-b057-f7feaf981b24}" testing.go:149: server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: 2021-07-10T01:04:59.581Z [INFO] server-cdd76ee3-70b9-f7fa-aeb0-3524550db012: aborting pipeline replication: peer="{Voter server-ece067ae-beeb-1de5-b057-f7feaf981b24 ece067ae-beeb-1de5-b057-f7feaf981b24}" --- PASS: TestRaft_TripleNode (0.28s) === RUN TestRaft_LeaderFail testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.583Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-772b73f7-89b2-6b24-3174-ae6a9841a41c Address:772b73f7-89b2-6b24-3174-ae6a9841a41c} {Suffrage:Voter ID:server-dc8eae36-1d7d-8f36-117a-63bca2af9124 Address:dc8eae36-1d7d-8f36-117a-63bca2af9124} {Suffrage:Voter ID:server-7e5a1339-ba92-b6b9-afd4-c4de792836ac Address:7e5a1339-ba92-b6b9-afd4-c4de792836ac}]" testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.583Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-772b73f7-89b2-6b24-3174-ae6a9841a41c Address:772b73f7-89b2-6b24-3174-ae6a9841a41c} {Suffrage:Voter ID:server-dc8eae36-1d7d-8f36-117a-63bca2af9124 Address:dc8eae36-1d7d-8f36-117a-63bca2af9124} {Suffrage:Voter ID:server-7e5a1339-ba92-b6b9-afd4-c4de792836ac Address:7e5a1339-ba92-b6b9-afd4-c4de792836ac}]" testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.583Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: entering follower state: follower="Node at 772b73f7-89b2-6b24-3174-ae6a9841a41c [Follower]" leader= testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.583Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: entering follower state: follower="Node at 772b73f7-89b2-6b24-3174-ae6a9841a41c [Follower]" leader= testLoggerAdapter verbose: server-dc8eae36-1d7d-8f36-117a-63bca2af9124: 2021-07-10T01:04:59.584Z [INFO] server-dc8eae36-1d7d-8f36-117a-63bca2af9124: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-772b73f7-89b2-6b24-3174-ae6a9841a41c Address:772b73f7-89b2-6b24-3174-ae6a9841a41c} {Suffrage:Voter ID:server-dc8eae36-1d7d-8f36-117a-63bca2af9124 Address:dc8eae36-1d7d-8f36-117a-63bca2af9124} {Suffrage:Voter ID:server-7e5a1339-ba92-b6b9-afd4-c4de792836ac Address:7e5a1339-ba92-b6b9-afd4-c4de792836ac}]" testing.go:149: server-dc8eae36-1d7d-8f36-117a-63bca2af9124: 2021-07-10T01:04:59.584Z [INFO] server-dc8eae36-1d7d-8f36-117a-63bca2af9124: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-772b73f7-89b2-6b24-3174-ae6a9841a41c Address:772b73f7-89b2-6b24-3174-ae6a9841a41c} {Suffrage:Voter ID:server-dc8eae36-1d7d-8f36-117a-63bca2af9124 Address:dc8eae36-1d7d-8f36-117a-63bca2af9124} {Suffrage:Voter ID:server-7e5a1339-ba92-b6b9-afd4-c4de792836ac Address:7e5a1339-ba92-b6b9-afd4-c4de792836ac}]" testLoggerAdapter verbose: server-dc8eae36-1d7d-8f36-117a-63bca2af9124: 2021-07-10T01:04:59.584Z [INFO] server-dc8eae36-1d7d-8f36-117a-63bca2af9124: entering follower state: follower="Node at dc8eae36-1d7d-8f36-117a-63bca2af9124 [Follower]" leader= testing.go:149: server-dc8eae36-1d7d-8f36-117a-63bca2af9124: 2021-07-10T01:04:59.584Z [INFO] server-dc8eae36-1d7d-8f36-117a-63bca2af9124: entering follower state: follower="Node at dc8eae36-1d7d-8f36-117a-63bca2af9124 [Follower]" leader= testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:04:59.584Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-772b73f7-89b2-6b24-3174-ae6a9841a41c Address:772b73f7-89b2-6b24-3174-ae6a9841a41c} {Suffrage:Voter ID:server-dc8eae36-1d7d-8f36-117a-63bca2af9124 Address:dc8eae36-1d7d-8f36-117a-63bca2af9124} {Suffrage:Voter ID:server-7e5a1339-ba92-b6b9-afd4-c4de792836ac Address:7e5a1339-ba92-b6b9-afd4-c4de792836ac}]" testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:04:59.584Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-772b73f7-89b2-6b24-3174-ae6a9841a41c Address:772b73f7-89b2-6b24-3174-ae6a9841a41c} {Suffrage:Voter ID:server-dc8eae36-1d7d-8f36-117a-63bca2af9124 Address:dc8eae36-1d7d-8f36-117a-63bca2af9124} {Suffrage:Voter ID:server-7e5a1339-ba92-b6b9-afd4-c4de792836ac Address:7e5a1339-ba92-b6b9-afd4-c4de792836ac}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:04:59.584Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-07-10T01:04:59.584Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:04:59.584Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: entering follower state: follower="Node at 7e5a1339-ba92-b6b9-afd4-c4de792836ac [Follower]" leader= testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:04:59.584Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: entering follower state: follower="Node at 7e5a1339-ba92-b6b9-afd4-c4de792836ac [Follower]" leader= testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.639Z [WARN] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.639Z [WARN] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.639Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: entering candidate state: node="Node at 772b73f7-89b2-6b24-3174-ae6a9841a41c [Candidate]" term=2 testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.639Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: entering candidate state: node="Node at 772b73f7-89b2-6b24-3174-ae6a9841a41c [Candidate]" term=2 testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.640Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: election won: tally=2 testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.640Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: election won: tally=2 testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.640Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: entering leader state: leader="Node at 772b73f7-89b2-6b24-3174-ae6a9841a41c [Leader]" testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.640Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: entering leader state: leader="Node at 772b73f7-89b2-6b24-3174-ae6a9841a41c [Leader]" testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.640Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: added peer, starting replication: peer=server-dc8eae36-1d7d-8f36-117a-63bca2af9124 testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.640Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: added peer, starting replication: peer=server-dc8eae36-1d7d-8f36-117a-63bca2af9124 testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.640Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: added peer, starting replication: peer=server-7e5a1339-ba92-b6b9-afd4-c4de792836ac testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.640Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: added peer, starting replication: peer=server-7e5a1339-ba92-b6b9-afd4-c4de792836ac testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.640Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: pipelining replication: peer="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.640Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: pipelining replication: peer="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.640Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: pipelining replication: peer="{Voter server-7e5a1339-ba92-b6b9-afd4-c4de792836ac 7e5a1339-ba92-b6b9-afd4-c4de792836ac}" testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.640Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: pipelining replication: peer="{Voter server-7e5a1339-ba92-b6b9-afd4-c4de792836ac 7e5a1339-ba92-b6b9-afd4-c4de792836ac}" testLoggerAdapter verbose: cluster: 2021-07-10T01:04:59.745Z [INFO] stable state for Follower reached at 2021-07-10 01:04:59.640240241 +0000 UTC m=+5.028398191 (2 nodes), 55.730567ms from start of poll, 56.806438ms from cluster start. Timeout at 2021-07-10 01:04:59.745307109 +0000 UTC m=+5.133465079, 105.066888ms after stability testing.go:149: cluster: 2021-07-10T01:04:59.745Z [INFO] stable state for Follower reached at 2021-07-10 01:04:59.640240241 +0000 UTC m=+5.028398191 (2 nodes), 55.730567ms from start of poll, 56.806438ms from cluster start. Timeout at 2021-07-10 01:04:59.745307109 +0000 UTC m=+5.133465079, 105.066888ms after stability testLoggerAdapter verbose: cluster: 2021-07-10T01:04:59.745Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:04:59.745Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-07-10T01:04:59.850Z [INFO] stable state for Leader reached at 2021-07-10 01:04:59.745520691 +0000 UTC m=+5.133678641 (1 nodes), 2.4µs from start of poll, 162.086888ms from cluster start. Timeout at 2021-07-10 01:04:59.850582219 +0000 UTC m=+5.238740189, 105.061548ms after stability testing.go:149: cluster: 2021-07-10T01:04:59.850Z [INFO] stable state for Leader reached at 2021-07-10 01:04:59.745520691 +0000 UTC m=+5.133678641 (1 nodes), 2.4µs from start of poll, 162.086888ms from cluster start. Timeout at 2021-07-10 01:04:59.850582219 +0000 UTC m=+5.238740189, 105.061548ms after stability raft_test.go:336: [INFO] Disconnecting Node at 772b73f7-89b2-6b24-3174-ae6a9841a41c [Leader] testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.865Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to pipeline appendEntries: peer="{Voter server-7e5a1339-ba92-b6b9-afd4-c4de792836ac 7e5a1339-ba92-b6b9-afd4-c4de792836ac}" error="append pipeline closed" testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.865Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to pipeline appendEntries: peer="{Voter server-7e5a1339-ba92-b6b9-afd4-c4de792836ac 7e5a1339-ba92-b6b9-afd4-c4de792836ac}" error="append pipeline closed" testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.865Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to heartbeat to: peer=dc8eae36-1d7d-8f36-117a-63bca2af9124 error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.865Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to heartbeat to: peer=dc8eae36-1d7d-8f36-117a-63bca2af9124 error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.865Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: aborting pipeline replication: peer="{Voter server-7e5a1339-ba92-b6b9-afd4-c4de792836ac 7e5a1339-ba92-b6b9-afd4-c4de792836ac}" testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.865Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: aborting pipeline replication: peer="{Voter server-7e5a1339-ba92-b6b9-afd4-c4de792836ac 7e5a1339-ba92-b6b9-afd4-c4de792836ac}" testLoggerAdapter verbose: cluster: 2021-07-10T01:04:59.866Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:04:59.866Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.867Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to pipeline appendEntries: peer="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" error="append pipeline closed" testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.867Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to pipeline appendEntries: peer="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" error="append pipeline closed" testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.867Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to heartbeat to: peer=7e5a1339-ba92-b6b9-afd4-c4de792836ac error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.867Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to heartbeat to: peer=7e5a1339-ba92-b6b9-afd4-c4de792836ac error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.867Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: aborting pipeline replication: peer="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.867Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: aborting pipeline replication: peer="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.873Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to appendEntries to: peer="{Voter server-7e5a1339-ba92-b6b9-afd4-c4de792836ac 7e5a1339-ba92-b6b9-afd4-c4de792836ac}" error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.873Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to appendEntries to: peer="{Voter server-7e5a1339-ba92-b6b9-afd4-c4de792836ac 7e5a1339-ba92-b6b9-afd4-c4de792836ac}" error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.875Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to appendEntries to: peer="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.875Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to appendEntries to: peer="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.884Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to heartbeat to: peer=dc8eae36-1d7d-8f36-117a-63bca2af9124 error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.884Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to heartbeat to: peer=dc8eae36-1d7d-8f36-117a-63bca2af9124 error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.886Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to heartbeat to: peer=7e5a1339-ba92-b6b9-afd4-c4de792836ac error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.886Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to heartbeat to: peer=7e5a1339-ba92-b6b9-afd4-c4de792836ac error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.891Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to appendEntries to: peer="{Voter server-7e5a1339-ba92-b6b9-afd4-c4de792836ac 7e5a1339-ba92-b6b9-afd4-c4de792836ac}" error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.891Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to appendEntries to: peer="{Voter server-7e5a1339-ba92-b6b9-afd4-c4de792836ac 7e5a1339-ba92-b6b9-afd4-c4de792836ac}" error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.891Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to appendEntries to: peer="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.891Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to appendEntries to: peer="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.902Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to heartbeat to: peer=dc8eae36-1d7d-8f36-117a-63bca2af9124 error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.902Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to heartbeat to: peer=dc8eae36-1d7d-8f36-117a-63bca2af9124 error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.902Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to heartbeat to: peer=7e5a1339-ba92-b6b9-afd4-c4de792836ac error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.902Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to heartbeat to: peer=7e5a1339-ba92-b6b9-afd4-c4de792836ac error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.907Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to appendEntries to: peer="{Voter server-7e5a1339-ba92-b6b9-afd4-c4de792836ac 7e5a1339-ba92-b6b9-afd4-c4de792836ac}" error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.907Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to appendEntries to: peer="{Voter server-7e5a1339-ba92-b6b9-afd4-c4de792836ac 7e5a1339-ba92-b6b9-afd4-c4de792836ac}" error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.907Z [WARN] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to contact: server-id=server-dc8eae36-1d7d-8f36-117a-63bca2af9124 time=50.049808ms testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.907Z [WARN] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to contact: server-id=server-dc8eae36-1d7d-8f36-117a-63bca2af9124 time=50.049808ms testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.908Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to appendEntries to: peer="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.908Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to appendEntries to: peer="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.917Z [WARN] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to contact: server-id=server-dc8eae36-1d7d-8f36-117a-63bca2af9124 time=60.271416ms testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.917Z [WARN] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to contact: server-id=server-dc8eae36-1d7d-8f36-117a-63bca2af9124 time=60.271416ms testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.917Z [WARN] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to contact: server-id=server-7e5a1339-ba92-b6b9-afd4-c4de792836ac time=59.134684ms testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.917Z [WARN] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to contact: server-id=server-7e5a1339-ba92-b6b9-afd4-c4de792836ac time=59.134684ms testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.918Z [WARN] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to contact quorum of nodes, stepping down testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.918Z [WARN] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to contact quorum of nodes, stepping down testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.918Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: entering follower state: follower="Node at 772b73f7-89b2-6b24-3174-ae6a9841a41c [Follower]" leader= testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.918Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: entering follower state: follower="Node at 772b73f7-89b2-6b24-3174-ae6a9841a41c [Follower]" leader= testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.928Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to heartbeat to: peer=7e5a1339-ba92-b6b9-afd4-c4de792836ac error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.928Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to heartbeat to: peer=7e5a1339-ba92-b6b9-afd4-c4de792836ac error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.931Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to heartbeat to: peer=dc8eae36-1d7d-8f36-117a-63bca2af9124 error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.931Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to heartbeat to: peer=dc8eae36-1d7d-8f36-117a-63bca2af9124 error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.933Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to appendEntries to: peer="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.933Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to appendEntries to: peer="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:04:59.935Z [WARN] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: heartbeat timeout reached, starting election: last-leader=772b73f7-89b2-6b24-3174-ae6a9841a41c testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:04:59.935Z [WARN] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: heartbeat timeout reached, starting election: last-leader=772b73f7-89b2-6b24-3174-ae6a9841a41c testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:04:59.935Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: entering candidate state: node="Node at 7e5a1339-ba92-b6b9-afd4-c4de792836ac [Candidate]" term=3 testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:04:59.935Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: entering candidate state: node="Node at 7e5a1339-ba92-b6b9-afd4-c4de792836ac [Candidate]" term=3 testLoggerAdapter verbose: server-dc8eae36-1d7d-8f36-117a-63bca2af9124: 2021-07-10T01:04:59.936Z [WARN] server-dc8eae36-1d7d-8f36-117a-63bca2af9124: rejecting vote request since we have a leader: from=7e5a1339-ba92-b6b9-afd4-c4de792836ac leader=772b73f7-89b2-6b24-3174-ae6a9841a41c testing.go:149: server-dc8eae36-1d7d-8f36-117a-63bca2af9124: 2021-07-10T01:04:59.936Z [WARN] server-dc8eae36-1d7d-8f36-117a-63bca2af9124: rejecting vote request since we have a leader: from=7e5a1339-ba92-b6b9-afd4-c4de792836ac leader=772b73f7-89b2-6b24-3174-ae6a9841a41c testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:04:59.935Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to make requestVote RPC: target="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:04:59.935Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to make requestVote RPC: target="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.937Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to appendEntries to: peer="{Voter server-7e5a1339-ba92-b6b9-afd4-c4de792836ac 7e5a1339-ba92-b6b9-afd4-c4de792836ac}" error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:04:59.937Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to appendEntries to: peer="{Voter server-7e5a1339-ba92-b6b9-afd4-c4de792836ac 7e5a1339-ba92-b6b9-afd4-c4de792836ac}" error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" testLoggerAdapter verbose: server-dc8eae36-1d7d-8f36-117a-63bca2af9124: 2021-07-10T01:04:59.979Z [WARN] server-dc8eae36-1d7d-8f36-117a-63bca2af9124: heartbeat timeout reached, starting election: last-leader=772b73f7-89b2-6b24-3174-ae6a9841a41c testing.go:149: server-dc8eae36-1d7d-8f36-117a-63bca2af9124: 2021-07-10T01:04:59.979Z [WARN] server-dc8eae36-1d7d-8f36-117a-63bca2af9124: heartbeat timeout reached, starting election: last-leader=772b73f7-89b2-6b24-3174-ae6a9841a41c testLoggerAdapter verbose: server-dc8eae36-1d7d-8f36-117a-63bca2af9124: 2021-07-10T01:04:59.979Z [INFO] server-dc8eae36-1d7d-8f36-117a-63bca2af9124: entering candidate state: node="Node at dc8eae36-1d7d-8f36-117a-63bca2af9124 [Candidate]" term=3 testing.go:149: server-dc8eae36-1d7d-8f36-117a-63bca2af9124: 2021-07-10T01:04:59.979Z [INFO] server-dc8eae36-1d7d-8f36-117a-63bca2af9124: entering candidate state: node="Node at dc8eae36-1d7d-8f36-117a-63bca2af9124 [Candidate]" term=3 testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:04:59.980Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: duplicate requestVote for same term: term=3 testLoggerAdapter verbose: server-dc8eae36-1d7d-8f36-117a-63bca2af9124: 2021-07-10T01:04:59.980Z [ERROR] server-dc8eae36-1d7d-8f36-117a-63bca2af9124: failed to make requestVote RPC: target="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:04:59.980Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: duplicate requestVote for same term: term=3 testing.go:149: server-dc8eae36-1d7d-8f36-117a-63bca2af9124: 2021-07-10T01:04:59.980Z [ERROR] server-dc8eae36-1d7d-8f36-117a-63bca2af9124: failed to make requestVote RPC: target="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.002Z [WARN] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: Election timeout reached, restarting election testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.002Z [WARN] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: Election timeout reached, restarting election testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.003Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: entering candidate state: node="Node at 7e5a1339-ba92-b6b9-afd4-c4de792836ac [Candidate]" term=4 testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.003Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: entering candidate state: node="Node at 7e5a1339-ba92-b6b9-afd4-c4de792836ac [Candidate]" term=4 testLoggerAdapter verbose: server-dc8eae36-1d7d-8f36-117a-63bca2af9124: 2021-07-10T01:05:00.003Z [INFO] server-dc8eae36-1d7d-8f36-117a-63bca2af9124: entering follower state: follower="Node at dc8eae36-1d7d-8f36-117a-63bca2af9124 [Follower]" leader= testing.go:149: server-dc8eae36-1d7d-8f36-117a-63bca2af9124: 2021-07-10T01:05:00.003Z [INFO] server-dc8eae36-1d7d-8f36-117a-63bca2af9124: entering follower state: follower="Node at dc8eae36-1d7d-8f36-117a-63bca2af9124 [Follower]" leader= testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.003Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: election won: tally=2 testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.003Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: election won: tally=2 testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.003Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: entering leader state: leader="Node at 7e5a1339-ba92-b6b9-afd4-c4de792836ac [Leader]" testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.003Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: entering leader state: leader="Node at 7e5a1339-ba92-b6b9-afd4-c4de792836ac [Leader]" testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.003Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: added peer, starting replication: peer=server-772b73f7-89b2-6b24-3174-ae6a9841a41c testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.003Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: added peer, starting replication: peer=server-772b73f7-89b2-6b24-3174-ae6a9841a41c testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.003Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to make requestVote RPC: target="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.003Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to make requestVote RPC: target="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.003Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: added peer, starting replication: peer=server-dc8eae36-1d7d-8f36-117a-63bca2af9124 testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.003Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: added peer, starting replication: peer=server-dc8eae36-1d7d-8f36-117a-63bca2af9124 testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.003Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to appendEntries to: peer="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.003Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to appendEntries to: peer="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.004Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: pipelining replication: peer="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.004Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: pipelining replication: peer="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.010Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to heartbeat to: peer=772b73f7-89b2-6b24-3174-ae6a9841a41c error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.010Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to heartbeat to: peer=772b73f7-89b2-6b24-3174-ae6a9841a41c error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:05:00.013Z [WARN] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:05:00.013Z [WARN] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:05:00.013Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: entering candidate state: node="Node at 772b73f7-89b2-6b24-3174-ae6a9841a41c [Candidate]" term=3 testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:05:00.013Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: entering candidate state: node="Node at 772b73f7-89b2-6b24-3174-ae6a9841a41c [Candidate]" term=3 testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:05:00.013Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to make requestVote RPC: target="{Voter server-7e5a1339-ba92-b6b9-afd4-c4de792836ac 7e5a1339-ba92-b6b9-afd4-c4de792836ac}" error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.013Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to appendEntries to: peer="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:05:00.013Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to make requestVote RPC: target="{Voter server-7e5a1339-ba92-b6b9-afd4-c4de792836ac 7e5a1339-ba92-b6b9-afd4-c4de792836ac}" error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.013Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to appendEntries to: peer="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:05:00.013Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to make requestVote RPC: target="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:05:00.013Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to make requestVote RPC: target="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.030Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to heartbeat to: peer=772b73f7-89b2-6b24-3174-ae6a9841a41c error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.030Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to heartbeat to: peer=772b73f7-89b2-6b24-3174-ae6a9841a41c error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.030Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to appendEntries to: peer="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.030Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to appendEntries to: peer="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.047Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to heartbeat to: peer=772b73f7-89b2-6b24-3174-ae6a9841a41c error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.047Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to heartbeat to: peer=772b73f7-89b2-6b24-3174-ae6a9841a41c error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.053Z [WARN] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to contact: server-id=server-772b73f7-89b2-6b24-3174-ae6a9841a41c time=50.33111ms testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.053Z [WARN] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to contact: server-id=server-772b73f7-89b2-6b24-3174-ae6a9841a41c time=50.33111ms testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.058Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to appendEntries to: peer="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.058Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to appendEntries to: peer="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.077Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to heartbeat to: peer=772b73f7-89b2-6b24-3174-ae6a9841a41c error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.077Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to heartbeat to: peer=772b73f7-89b2-6b24-3174-ae6a9841a41c error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:05:00.078Z [WARN] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: Election timeout reached, restarting election testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:05:00.078Z [WARN] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: Election timeout reached, restarting election testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:05:00.078Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: entering candidate state: node="Node at 772b73f7-89b2-6b24-3174-ae6a9841a41c [Candidate]" term=4 testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:05:00.078Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: entering candidate state: node="Node at 772b73f7-89b2-6b24-3174-ae6a9841a41c [Candidate]" term=4 testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:05:00.078Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to make requestVote RPC: target="{Voter server-7e5a1339-ba92-b6b9-afd4-c4de792836ac 7e5a1339-ba92-b6b9-afd4-c4de792836ac}" error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:05:00.078Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to make requestVote RPC: target="{Voter server-7e5a1339-ba92-b6b9-afd4-c4de792836ac 7e5a1339-ba92-b6b9-afd4-c4de792836ac}" error="failed to connect to peer: 7e5a1339-ba92-b6b9-afd4-c4de792836ac" testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:05:00.078Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to make requestVote RPC: target="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:05:00.078Z [ERROR] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: failed to make requestVote RPC: target="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" error="failed to connect to peer: dc8eae36-1d7d-8f36-117a-63bca2af9124" testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.103Z [WARN] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to contact: server-id=server-772b73f7-89b2-6b24-3174-ae6a9841a41c time=99.736131ms testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.103Z [WARN] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to contact: server-id=server-772b73f7-89b2-6b24-3174-ae6a9841a41c time=99.736131ms testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.107Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to appendEntries to: peer="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.107Z [ERROR] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: failed to appendEntries to: peer="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" error="failed to connect to peer: 772b73f7-89b2-6b24-3174-ae6a9841a41c" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:00.119Z [INFO] stable state for Leader reached at 2021-07-10 01:05:00.01386486 +0000 UTC m=+5.402022830 (1 nodes), 147.361573ms from start of poll, 430.431077ms from cluster start. Timeout at 2021-07-10 01:05:00.118937068 +0000 UTC m=+5.507095038, 105.072208ms after stability testing.go:149: cluster: 2021-07-10T01:05:00.119Z [INFO] stable state for Leader reached at 2021-07-10 01:05:00.01386486 +0000 UTC m=+5.402022830 (1 nodes), 147.361573ms from start of poll, 430.431077ms from cluster start. Timeout at 2021-07-10 01:05:00.118937068 +0000 UTC m=+5.507095038, 105.072208ms after stability raft_test.go:371: [INFO] Reconnecting Node at 772b73f7-89b2-6b24-3174-ae6a9841a41c [Candidate] testLoggerAdapter verbose: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:05:00.123Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: entering follower state: follower="Node at 772b73f7-89b2-6b24-3174-ae6a9841a41c [Follower]" leader=7e5a1339-ba92-b6b9-afd4-c4de792836ac testing.go:149: server-772b73f7-89b2-6b24-3174-ae6a9841a41c: 2021-07-10T01:05:00.123Z [INFO] server-772b73f7-89b2-6b24-3174-ae6a9841a41c: entering follower state: follower="Node at 772b73f7-89b2-6b24-3174-ae6a9841a41c [Follower]" leader=7e5a1339-ba92-b6b9-afd4-c4de792836ac testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.193Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: pipelining replication: peer="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.193Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: pipelining replication: peer="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.194Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: aborting pipeline replication: peer="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.194Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: aborting pipeline replication: peer="{Voter server-dc8eae36-1d7d-8f36-117a-63bca2af9124 dc8eae36-1d7d-8f36-117a-63bca2af9124}" testLoggerAdapter verbose: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.194Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: aborting pipeline replication: peer="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" testing.go:149: server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: 2021-07-10T01:05:00.194Z [INFO] server-7e5a1339-ba92-b6b9-afd4-c4de792836ac: aborting pipeline replication: peer="{Voter server-772b73f7-89b2-6b24-3174-ae6a9841a41c 772b73f7-89b2-6b24-3174-ae6a9841a41c}" --- PASS: TestRaft_LeaderFail (0.61s) === RUN TestRaft_BehindFollower testLoggerAdapter verbose: server-cc4def27-8c62-c75b-00e3-6804eaf9325d: 2021-07-10T01:05:00.196Z [INFO] server-cc4def27-8c62-c75b-00e3-6804eaf9325d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cc4def27-8c62-c75b-00e3-6804eaf9325d Address:cc4def27-8c62-c75b-00e3-6804eaf9325d} {Suffrage:Voter ID:server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a Address:8fa31dd9-db10-3057-8c4e-7025abaa6e2a} {Suffrage:Voter ID:server-7a0d7856-eca4-bf16-e907-5ebab1a3e751 Address:7a0d7856-eca4-bf16-e907-5ebab1a3e751}]" testing.go:149: server-cc4def27-8c62-c75b-00e3-6804eaf9325d: 2021-07-10T01:05:00.196Z [INFO] server-cc4def27-8c62-c75b-00e3-6804eaf9325d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cc4def27-8c62-c75b-00e3-6804eaf9325d Address:cc4def27-8c62-c75b-00e3-6804eaf9325d} {Suffrage:Voter ID:server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a Address:8fa31dd9-db10-3057-8c4e-7025abaa6e2a} {Suffrage:Voter ID:server-7a0d7856-eca4-bf16-e907-5ebab1a3e751 Address:7a0d7856-eca4-bf16-e907-5ebab1a3e751}]" testLoggerAdapter verbose: server-cc4def27-8c62-c75b-00e3-6804eaf9325d: 2021-07-10T01:05:00.196Z [INFO] server-cc4def27-8c62-c75b-00e3-6804eaf9325d: entering follower state: follower="Node at cc4def27-8c62-c75b-00e3-6804eaf9325d [Follower]" leader= testing.go:149: server-cc4def27-8c62-c75b-00e3-6804eaf9325d: 2021-07-10T01:05:00.196Z [INFO] server-cc4def27-8c62-c75b-00e3-6804eaf9325d: entering follower state: follower="Node at cc4def27-8c62-c75b-00e3-6804eaf9325d [Follower]" leader= testLoggerAdapter verbose: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.196Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cc4def27-8c62-c75b-00e3-6804eaf9325d Address:cc4def27-8c62-c75b-00e3-6804eaf9325d} {Suffrage:Voter ID:server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a Address:8fa31dd9-db10-3057-8c4e-7025abaa6e2a} {Suffrage:Voter ID:server-7a0d7856-eca4-bf16-e907-5ebab1a3e751 Address:7a0d7856-eca4-bf16-e907-5ebab1a3e751}]" testing.go:149: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.196Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cc4def27-8c62-c75b-00e3-6804eaf9325d Address:cc4def27-8c62-c75b-00e3-6804eaf9325d} {Suffrage:Voter ID:server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a Address:8fa31dd9-db10-3057-8c4e-7025abaa6e2a} {Suffrage:Voter ID:server-7a0d7856-eca4-bf16-e907-5ebab1a3e751 Address:7a0d7856-eca4-bf16-e907-5ebab1a3e751}]" testLoggerAdapter verbose: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.196Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: entering follower state: follower="Node at 8fa31dd9-db10-3057-8c4e-7025abaa6e2a [Follower]" leader= testing.go:149: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.196Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: entering follower state: follower="Node at 8fa31dd9-db10-3057-8c4e-7025abaa6e2a [Follower]" leader= testLoggerAdapter verbose: server-7a0d7856-eca4-bf16-e907-5ebab1a3e751: 2021-07-10T01:05:00.197Z [INFO] server-7a0d7856-eca4-bf16-e907-5ebab1a3e751: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cc4def27-8c62-c75b-00e3-6804eaf9325d Address:cc4def27-8c62-c75b-00e3-6804eaf9325d} {Suffrage:Voter ID:server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a Address:8fa31dd9-db10-3057-8c4e-7025abaa6e2a} {Suffrage:Voter ID:server-7a0d7856-eca4-bf16-e907-5ebab1a3e751 Address:7a0d7856-eca4-bf16-e907-5ebab1a3e751}]" testing.go:149: server-7a0d7856-eca4-bf16-e907-5ebab1a3e751: 2021-07-10T01:05:00.197Z [INFO] server-7a0d7856-eca4-bf16-e907-5ebab1a3e751: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cc4def27-8c62-c75b-00e3-6804eaf9325d Address:cc4def27-8c62-c75b-00e3-6804eaf9325d} {Suffrage:Voter ID:server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a Address:8fa31dd9-db10-3057-8c4e-7025abaa6e2a} {Suffrage:Voter ID:server-7a0d7856-eca4-bf16-e907-5ebab1a3e751 Address:7a0d7856-eca4-bf16-e907-5ebab1a3e751}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:00.197Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:00.197Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-7a0d7856-eca4-bf16-e907-5ebab1a3e751: 2021-07-10T01:05:00.197Z [INFO] server-7a0d7856-eca4-bf16-e907-5ebab1a3e751: entering follower state: follower="Node at 7a0d7856-eca4-bf16-e907-5ebab1a3e751 [Follower]" leader= testing.go:149: server-7a0d7856-eca4-bf16-e907-5ebab1a3e751: 2021-07-10T01:05:00.197Z [INFO] server-7a0d7856-eca4-bf16-e907-5ebab1a3e751: entering follower state: follower="Node at 7a0d7856-eca4-bf16-e907-5ebab1a3e751 [Follower]" leader= testLoggerAdapter verbose: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.251Z [WARN] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.251Z [WARN] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.251Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: entering candidate state: node="Node at 8fa31dd9-db10-3057-8c4e-7025abaa6e2a [Candidate]" term=2 testing.go:149: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.251Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: entering candidate state: node="Node at 8fa31dd9-db10-3057-8c4e-7025abaa6e2a [Candidate]" term=2 testLoggerAdapter verbose: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.251Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: election won: tally=2 testing.go:149: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.251Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: election won: tally=2 testLoggerAdapter verbose: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.251Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: entering leader state: leader="Node at 8fa31dd9-db10-3057-8c4e-7025abaa6e2a [Leader]" testing.go:149: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.251Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: entering leader state: leader="Node at 8fa31dd9-db10-3057-8c4e-7025abaa6e2a [Leader]" testLoggerAdapter verbose: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.251Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: added peer, starting replication: peer=server-cc4def27-8c62-c75b-00e3-6804eaf9325d testing.go:149: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.251Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: added peer, starting replication: peer=server-cc4def27-8c62-c75b-00e3-6804eaf9325d testLoggerAdapter verbose: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.252Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: added peer, starting replication: peer=server-7a0d7856-eca4-bf16-e907-5ebab1a3e751 testing.go:149: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.252Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: added peer, starting replication: peer=server-7a0d7856-eca4-bf16-e907-5ebab1a3e751 testLoggerAdapter verbose: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.252Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: pipelining replication: peer="{Voter server-cc4def27-8c62-c75b-00e3-6804eaf9325d cc4def27-8c62-c75b-00e3-6804eaf9325d}" testing.go:149: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.252Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: pipelining replication: peer="{Voter server-cc4def27-8c62-c75b-00e3-6804eaf9325d cc4def27-8c62-c75b-00e3-6804eaf9325d}" testLoggerAdapter verbose: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.252Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: pipelining replication: peer="{Voter server-7a0d7856-eca4-bf16-e907-5ebab1a3e751 7a0d7856-eca4-bf16-e907-5ebab1a3e751}" testing.go:149: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.252Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: pipelining replication: peer="{Voter server-7a0d7856-eca4-bf16-e907-5ebab1a3e751 7a0d7856-eca4-bf16-e907-5ebab1a3e751}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:00.357Z [INFO] stable state for Leader reached at 2021-07-10 01:05:00.251979811 +0000 UTC m=+5.640137801 (1 nodes), 54.825338ms from start of poll, 55.814969ms from cluster start. Timeout at 2021-07-10 01:05:00.357050238 +0000 UTC m=+5.745208208, 105.070407ms after stability testing.go:149: cluster: 2021-07-10T01:05:00.357Z [INFO] stable state for Leader reached at 2021-07-10 01:05:00.251979811 +0000 UTC m=+5.640137801 (1 nodes), 54.825338ms from start of poll, 55.814969ms from cluster start. Timeout at 2021-07-10 01:05:00.357050238 +0000 UTC m=+5.745208208, 105.070407ms after stability testLoggerAdapter verbose: cluster: 2021-07-10T01:05:00.357Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-07-10T01:05:00.357Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-07-10T01:05:00.462Z [INFO] stable state for Follower reached at 2021-07-10 01:05:00.357348541 +0000 UTC m=+5.745506511 (2 nodes), 2.2µs from start of poll, 161.183679ms from cluster start. Timeout at 2021-07-10 01:05:00.462450769 +0000 UTC m=+5.850608759, 105.102248ms after stability testing.go:149: cluster: 2021-07-10T01:05:00.462Z [INFO] stable state for Follower reached at 2021-07-10 01:05:00.357348541 +0000 UTC m=+5.745506511 (2 nodes), 2.2µs from start of poll, 161.183679ms from cluster start. Timeout at 2021-07-10 01:05:00.462450769 +0000 UTC m=+5.850608759, 105.102248ms after stability testLoggerAdapter verbose: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.462Z [ERROR] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: failed to pipeline appendEntries: peer="{Voter server-cc4def27-8c62-c75b-00e3-6804eaf9325d cc4def27-8c62-c75b-00e3-6804eaf9325d}" error="append pipeline closed" testing.go:149: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.462Z [ERROR] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: failed to pipeline appendEntries: peer="{Voter server-cc4def27-8c62-c75b-00e3-6804eaf9325d cc4def27-8c62-c75b-00e3-6804eaf9325d}" error="append pipeline closed" testLoggerAdapter verbose: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.462Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: aborting pipeline replication: peer="{Voter server-cc4def27-8c62-c75b-00e3-6804eaf9325d cc4def27-8c62-c75b-00e3-6804eaf9325d}" testing.go:149: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.462Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: aborting pipeline replication: peer="{Voter server-cc4def27-8c62-c75b-00e3-6804eaf9325d cc4def27-8c62-c75b-00e3-6804eaf9325d}" testLoggerAdapter verbose: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.462Z [ERROR] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: failed to appendEntries to: peer="{Voter server-cc4def27-8c62-c75b-00e3-6804eaf9325d cc4def27-8c62-c75b-00e3-6804eaf9325d}" error="failed to connect to peer: cc4def27-8c62-c75b-00e3-6804eaf9325d" testing.go:149: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.462Z [ERROR] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: failed to appendEntries to: peer="{Voter server-cc4def27-8c62-c75b-00e3-6804eaf9325d cc4def27-8c62-c75b-00e3-6804eaf9325d}" error="failed to connect to peer: cc4def27-8c62-c75b-00e3-6804eaf9325d" raft_test.go:420: [INFO] Finished apply without behind follower testLoggerAdapter verbose: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.473Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: pipelining replication: peer="{Voter server-cc4def27-8c62-c75b-00e3-6804eaf9325d cc4def27-8c62-c75b-00e3-6804eaf9325d}" testing.go:149: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.473Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: pipelining replication: peer="{Voter server-cc4def27-8c62-c75b-00e3-6804eaf9325d cc4def27-8c62-c75b-00e3-6804eaf9325d}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:00.485Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:00.485Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-07-10T01:05:00.591Z [INFO] stable state for Leader reached at 2021-07-10 01:05:00.485859776 +0000 UTC m=+5.874017726 (1 nodes), 2.2µs from start of poll, 289.694894ms from cluster start. Timeout at 2021-07-10 01:05:00.590938324 +0000 UTC m=+5.979096334, 105.078608ms after stability testing.go:149: cluster: 2021-07-10T01:05:00.591Z [INFO] stable state for Leader reached at 2021-07-10 01:05:00.485859776 +0000 UTC m=+5.874017726 (1 nodes), 2.2µs from start of poll, 289.694894ms from cluster start. Timeout at 2021-07-10 01:05:00.590938324 +0000 UTC m=+5.979096334, 105.078608ms after stability testLoggerAdapter verbose: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.591Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: aborting pipeline replication: peer="{Voter server-cc4def27-8c62-c75b-00e3-6804eaf9325d cc4def27-8c62-c75b-00e3-6804eaf9325d}" testing.go:149: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.591Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: aborting pipeline replication: peer="{Voter server-cc4def27-8c62-c75b-00e3-6804eaf9325d cc4def27-8c62-c75b-00e3-6804eaf9325d}" testLoggerAdapter verbose: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.591Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: aborting pipeline replication: peer="{Voter server-7a0d7856-eca4-bf16-e907-5ebab1a3e751 7a0d7856-eca4-bf16-e907-5ebab1a3e751}" testing.go:149: server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: 2021-07-10T01:05:00.591Z [INFO] server-8fa31dd9-db10-3057-8c4e-7025abaa6e2a: aborting pipeline replication: peer="{Voter server-7a0d7856-eca4-bf16-e907-5ebab1a3e751 7a0d7856-eca4-bf16-e907-5ebab1a3e751}" --- PASS: TestRaft_BehindFollower (0.40s) === RUN TestRaft_ApplyNonLeader testLoggerAdapter verbose: server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a: 2021-07-10T01:05:00.593Z [INFO] server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a Address:66db94f2-09bd-b5b9-1c59-dfab786a9e6a} {Suffrage:Voter ID:server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd Address:09dd28b7-7554-e5d9-369d-d0beeb9ccdcd} {Suffrage:Voter ID:server-724f68de-f93b-7654-af86-35f73f0d7331 Address:724f68de-f93b-7654-af86-35f73f0d7331}]" testing.go:149: server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a: 2021-07-10T01:05:00.593Z [INFO] server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a Address:66db94f2-09bd-b5b9-1c59-dfab786a9e6a} {Suffrage:Voter ID:server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd Address:09dd28b7-7554-e5d9-369d-d0beeb9ccdcd} {Suffrage:Voter ID:server-724f68de-f93b-7654-af86-35f73f0d7331 Address:724f68de-f93b-7654-af86-35f73f0d7331}]" testLoggerAdapter verbose: server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd: 2021-07-10T01:05:00.593Z [INFO] server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a Address:66db94f2-09bd-b5b9-1c59-dfab786a9e6a} {Suffrage:Voter ID:server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd Address:09dd28b7-7554-e5d9-369d-d0beeb9ccdcd} {Suffrage:Voter ID:server-724f68de-f93b-7654-af86-35f73f0d7331 Address:724f68de-f93b-7654-af86-35f73f0d7331}]" testing.go:149: server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd: 2021-07-10T01:05:00.593Z [INFO] server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a Address:66db94f2-09bd-b5b9-1c59-dfab786a9e6a} {Suffrage:Voter ID:server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd Address:09dd28b7-7554-e5d9-369d-d0beeb9ccdcd} {Suffrage:Voter ID:server-724f68de-f93b-7654-af86-35f73f0d7331 Address:724f68de-f93b-7654-af86-35f73f0d7331}]" testLoggerAdapter verbose: server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a: 2021-07-10T01:05:00.593Z [INFO] server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a: entering follower state: follower="Node at 66db94f2-09bd-b5b9-1c59-dfab786a9e6a [Follower]" leader= testing.go:149: server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a: 2021-07-10T01:05:00.593Z [INFO] server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a: entering follower state: follower="Node at 66db94f2-09bd-b5b9-1c59-dfab786a9e6a [Follower]" leader= testLoggerAdapter verbose: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.593Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a Address:66db94f2-09bd-b5b9-1c59-dfab786a9e6a} {Suffrage:Voter ID:server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd Address:09dd28b7-7554-e5d9-369d-d0beeb9ccdcd} {Suffrage:Voter ID:server-724f68de-f93b-7654-af86-35f73f0d7331 Address:724f68de-f93b-7654-af86-35f73f0d7331}]" testing.go:149: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.593Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a Address:66db94f2-09bd-b5b9-1c59-dfab786a9e6a} {Suffrage:Voter ID:server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd Address:09dd28b7-7554-e5d9-369d-d0beeb9ccdcd} {Suffrage:Voter ID:server-724f68de-f93b-7654-af86-35f73f0d7331 Address:724f68de-f93b-7654-af86-35f73f0d7331}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:00.593Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:00.593Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.593Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: entering follower state: follower="Node at 724f68de-f93b-7654-af86-35f73f0d7331 [Follower]" leader= testLoggerAdapter verbose: server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd: 2021-07-10T01:05:00.593Z [INFO] server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd: entering follower state: follower="Node at 09dd28b7-7554-e5d9-369d-d0beeb9ccdcd [Follower]" leader= testing.go:149: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.593Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: entering follower state: follower="Node at 724f68de-f93b-7654-af86-35f73f0d7331 [Follower]" leader= testing.go:149: server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd: 2021-07-10T01:05:00.593Z [INFO] server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd: entering follower state: follower="Node at 09dd28b7-7554-e5d9-369d-d0beeb9ccdcd [Follower]" leader= testLoggerAdapter verbose: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.676Z [WARN] server-724f68de-f93b-7654-af86-35f73f0d7331: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.676Z [WARN] server-724f68de-f93b-7654-af86-35f73f0d7331: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.676Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: entering candidate state: node="Node at 724f68de-f93b-7654-af86-35f73f0d7331 [Candidate]" term=2 testing.go:149: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.676Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: entering candidate state: node="Node at 724f68de-f93b-7654-af86-35f73f0d7331 [Candidate]" term=2 testLoggerAdapter verbose: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.676Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: election won: tally=2 testing.go:149: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.676Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: election won: tally=2 testLoggerAdapter verbose: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.676Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: entering leader state: leader="Node at 724f68de-f93b-7654-af86-35f73f0d7331 [Leader]" testing.go:149: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.676Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: entering leader state: leader="Node at 724f68de-f93b-7654-af86-35f73f0d7331 [Leader]" testLoggerAdapter verbose: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.676Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: added peer, starting replication: peer=server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a testing.go:149: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.676Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: added peer, starting replication: peer=server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a testLoggerAdapter verbose: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.676Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: added peer, starting replication: peer=server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd testing.go:149: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.676Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: added peer, starting replication: peer=server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd testLoggerAdapter verbose: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.676Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: pipelining replication: peer="{Voter server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a 66db94f2-09bd-b5b9-1c59-dfab786a9e6a}" testing.go:149: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.676Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: pipelining replication: peer="{Voter server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a 66db94f2-09bd-b5b9-1c59-dfab786a9e6a}" testLoggerAdapter verbose: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.676Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: pipelining replication: peer="{Voter server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd 09dd28b7-7554-e5d9-369d-d0beeb9ccdcd}" testing.go:149: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.676Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: pipelining replication: peer="{Voter server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd 09dd28b7-7554-e5d9-369d-d0beeb9ccdcd}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:00.781Z [INFO] stable state for Leader reached at 2021-07-10 01:05:00.676678028 +0000 UTC m=+6.064836018 (1 nodes), 82.795313ms from start of poll, 83.835964ms from cluster start. Timeout at 2021-07-10 01:05:00.781704375 +0000 UTC m=+6.169862345, 105.026327ms after stability testing.go:149: cluster: 2021-07-10T01:05:00.781Z [INFO] stable state for Leader reached at 2021-07-10 01:05:00.676678028 +0000 UTC m=+6.064836018 (1 nodes), 82.795313ms from start of poll, 83.835964ms from cluster start. Timeout at 2021-07-10 01:05:00.781704375 +0000 UTC m=+6.169862345, 105.026327ms after stability testLoggerAdapter verbose: cluster: 2021-07-10T01:05:00.781Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-07-10T01:05:00.781Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-07-10T01:05:00.887Z [INFO] stable state for Follower reached at 2021-07-10 01:05:00.781942018 +0000 UTC m=+6.170099988 (2 nodes), 2.361µs from start of poll, 189.099934ms from cluster start. Timeout at 2021-07-10 01:05:00.886985905 +0000 UTC m=+6.275143895, 105.043907ms after stability testing.go:149: cluster: 2021-07-10T01:05:00.887Z [INFO] stable state for Follower reached at 2021-07-10 01:05:00.781942018 +0000 UTC m=+6.170099988 (2 nodes), 2.361µs from start of poll, 189.099934ms from cluster start. Timeout at 2021-07-10 01:05:00.886985905 +0000 UTC m=+6.275143895, 105.043907ms after stability testLoggerAdapter verbose: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.887Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: aborting pipeline replication: peer="{Voter server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd 09dd28b7-7554-e5d9-369d-d0beeb9ccdcd}" testing.go:149: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.887Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: aborting pipeline replication: peer="{Voter server-09dd28b7-7554-e5d9-369d-d0beeb9ccdcd 09dd28b7-7554-e5d9-369d-d0beeb9ccdcd}" testLoggerAdapter verbose: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.887Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: aborting pipeline replication: peer="{Voter server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a 66db94f2-09bd-b5b9-1c59-dfab786a9e6a}" testing.go:149: server-724f68de-f93b-7654-af86-35f73f0d7331: 2021-07-10T01:05:00.887Z [INFO] server-724f68de-f93b-7654-af86-35f73f0d7331: aborting pipeline replication: peer="{Voter server-66db94f2-09bd-b5b9-1c59-dfab786a9e6a 66db94f2-09bd-b5b9-1c59-dfab786a9e6a}" --- PASS: TestRaft_ApplyNonLeader (0.30s) === RUN TestRaft_ApplyConcurrent testLoggerAdapter verbose: server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3: 2021-07-10T01:05:00.888Z [INFO] server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3 Address:1e1d953e-3dae-f134-89b6-6cbcda5c8de3} {Suffrage:Voter ID:server-a827d17c-f21d-780f-dd5d-c9471da2e059 Address:a827d17c-f21d-780f-dd5d-c9471da2e059} {Suffrage:Voter ID:server-dc316ee9-f2eb-238f-1020-e5f5db597b81 Address:dc316ee9-f2eb-238f-1020-e5f5db597b81}]" testing.go:149: server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3: 2021-07-10T01:05:00.888Z [INFO] server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3 Address:1e1d953e-3dae-f134-89b6-6cbcda5c8de3} {Suffrage:Voter ID:server-a827d17c-f21d-780f-dd5d-c9471da2e059 Address:a827d17c-f21d-780f-dd5d-c9471da2e059} {Suffrage:Voter ID:server-dc316ee9-f2eb-238f-1020-e5f5db597b81 Address:dc316ee9-f2eb-238f-1020-e5f5db597b81}]" testLoggerAdapter verbose: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:00.889Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3 Address:1e1d953e-3dae-f134-89b6-6cbcda5c8de3} {Suffrage:Voter ID:server-a827d17c-f21d-780f-dd5d-c9471da2e059 Address:a827d17c-f21d-780f-dd5d-c9471da2e059} {Suffrage:Voter ID:server-dc316ee9-f2eb-238f-1020-e5f5db597b81 Address:dc316ee9-f2eb-238f-1020-e5f5db597b81}]" testing.go:149: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:00.889Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3 Address:1e1d953e-3dae-f134-89b6-6cbcda5c8de3} {Suffrage:Voter ID:server-a827d17c-f21d-780f-dd5d-c9471da2e059 Address:a827d17c-f21d-780f-dd5d-c9471da2e059} {Suffrage:Voter ID:server-dc316ee9-f2eb-238f-1020-e5f5db597b81 Address:dc316ee9-f2eb-238f-1020-e5f5db597b81}]" testLoggerAdapter verbose: server-dc316ee9-f2eb-238f-1020-e5f5db597b81: 2021-07-10T01:05:00.889Z [INFO] server-dc316ee9-f2eb-238f-1020-e5f5db597b81: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3 Address:1e1d953e-3dae-f134-89b6-6cbcda5c8de3} {Suffrage:Voter ID:server-a827d17c-f21d-780f-dd5d-c9471da2e059 Address:a827d17c-f21d-780f-dd5d-c9471da2e059} {Suffrage:Voter ID:server-dc316ee9-f2eb-238f-1020-e5f5db597b81 Address:dc316ee9-f2eb-238f-1020-e5f5db597b81}]" testing.go:149: server-dc316ee9-f2eb-238f-1020-e5f5db597b81: 2021-07-10T01:05:00.889Z [INFO] server-dc316ee9-f2eb-238f-1020-e5f5db597b81: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3 Address:1e1d953e-3dae-f134-89b6-6cbcda5c8de3} {Suffrage:Voter ID:server-a827d17c-f21d-780f-dd5d-c9471da2e059 Address:a827d17c-f21d-780f-dd5d-c9471da2e059} {Suffrage:Voter ID:server-dc316ee9-f2eb-238f-1020-e5f5db597b81 Address:dc316ee9-f2eb-238f-1020-e5f5db597b81}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:00.889Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:00.889Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3: 2021-07-10T01:05:00.889Z [INFO] server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3: entering follower state: follower="Node at 1e1d953e-3dae-f134-89b6-6cbcda5c8de3 [Follower]" leader= testing.go:149: server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3: 2021-07-10T01:05:00.889Z [INFO] server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3: entering follower state: follower="Node at 1e1d953e-3dae-f134-89b6-6cbcda5c8de3 [Follower]" leader= testLoggerAdapter verbose: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:00.889Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: entering follower state: follower="Node at a827d17c-f21d-780f-dd5d-c9471da2e059 [Follower]" leader= testing.go:149: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:00.889Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: entering follower state: follower="Node at a827d17c-f21d-780f-dd5d-c9471da2e059 [Follower]" leader= testLoggerAdapter verbose: server-dc316ee9-f2eb-238f-1020-e5f5db597b81: 2021-07-10T01:05:00.889Z [INFO] server-dc316ee9-f2eb-238f-1020-e5f5db597b81: entering follower state: follower="Node at dc316ee9-f2eb-238f-1020-e5f5db597b81 [Follower]" leader= testing.go:149: server-dc316ee9-f2eb-238f-1020-e5f5db597b81: 2021-07-10T01:05:00.889Z [INFO] server-dc316ee9-f2eb-238f-1020-e5f5db597b81: entering follower state: follower="Node at dc316ee9-f2eb-238f-1020-e5f5db597b81 [Follower]" leader= testLoggerAdapter verbose: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.026Z [WARN] server-a827d17c-f21d-780f-dd5d-c9471da2e059: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.026Z [WARN] server-a827d17c-f21d-780f-dd5d-c9471da2e059: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.026Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: entering candidate state: node="Node at a827d17c-f21d-780f-dd5d-c9471da2e059 [Candidate]" term=2 testing.go:149: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.026Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: entering candidate state: node="Node at a827d17c-f21d-780f-dd5d-c9471da2e059 [Candidate]" term=2 testLoggerAdapter verbose: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.027Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: election won: tally=2 testing.go:149: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.027Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: election won: tally=2 testLoggerAdapter verbose: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.027Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: entering leader state: leader="Node at a827d17c-f21d-780f-dd5d-c9471da2e059 [Leader]" testing.go:149: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.027Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: entering leader state: leader="Node at a827d17c-f21d-780f-dd5d-c9471da2e059 [Leader]" testLoggerAdapter verbose: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.027Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: added peer, starting replication: peer=server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3 testing.go:149: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.027Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: added peer, starting replication: peer=server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3 testLoggerAdapter verbose: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.027Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: added peer, starting replication: peer=server-dc316ee9-f2eb-238f-1020-e5f5db597b81 testing.go:149: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.027Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: added peer, starting replication: peer=server-dc316ee9-f2eb-238f-1020-e5f5db597b81 testLoggerAdapter verbose: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.027Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: pipelining replication: peer="{Voter server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3 1e1d953e-3dae-f134-89b6-6cbcda5c8de3}" testing.go:149: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.027Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: pipelining replication: peer="{Voter server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3 1e1d953e-3dae-f134-89b6-6cbcda5c8de3}" testLoggerAdapter verbose: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.027Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: pipelining replication: peer="{Voter server-dc316ee9-f2eb-238f-1020-e5f5db597b81 dc316ee9-f2eb-238f-1020-e5f5db597b81}" testing.go:149: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.027Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: pipelining replication: peer="{Voter server-dc316ee9-f2eb-238f-1020-e5f5db597b81 dc316ee9-f2eb-238f-1020-e5f5db597b81}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:01.232Z [INFO] stable state for Leader reached at 2021-07-10 01:05:01.027322584 +0000 UTC m=+6.415480574 (1 nodes), 137.796732ms from start of poll, 138.738142ms from cluster start. Timeout at 2021-07-10 01:05:01.232389546 +0000 UTC m=+6.620547496, 205.066922ms after stability testing.go:149: cluster: 2021-07-10T01:05:01.232Z [INFO] stable state for Leader reached at 2021-07-10 01:05:01.027322584 +0000 UTC m=+6.415480574 (1 nodes), 137.796732ms from start of poll, 138.738142ms from cluster start. Timeout at 2021-07-10 01:05:01.232389546 +0000 UTC m=+6.620547496, 205.066922ms after stability testLoggerAdapter verbose: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.245Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: aborting pipeline replication: peer="{Voter server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3 1e1d953e-3dae-f134-89b6-6cbcda5c8de3}" testing.go:149: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.245Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: aborting pipeline replication: peer="{Voter server-1e1d953e-3dae-f134-89b6-6cbcda5c8de3 1e1d953e-3dae-f134-89b6-6cbcda5c8de3}" testLoggerAdapter verbose: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.245Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: aborting pipeline replication: peer="{Voter server-dc316ee9-f2eb-238f-1020-e5f5db597b81 dc316ee9-f2eb-238f-1020-e5f5db597b81}" testing.go:149: server-a827d17c-f21d-780f-dd5d-c9471da2e059: 2021-07-10T01:05:01.245Z [INFO] server-a827d17c-f21d-780f-dd5d-c9471da2e059: aborting pipeline replication: peer="{Voter server-dc316ee9-f2eb-238f-1020-e5f5db597b81 dc316ee9-f2eb-238f-1020-e5f5db597b81}" --- PASS: TestRaft_ApplyConcurrent (0.36s) === RUN TestRaft_ApplyConcurrent_Timeout testLoggerAdapter verbose: server-81607767-2f01-59f9-be42-330697676670: 2021-07-10T01:05:01.246Z [INFO] server-81607767-2f01-59f9-be42-330697676670: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-81607767-2f01-59f9-be42-330697676670 Address:81607767-2f01-59f9-be42-330697676670}]" testing.go:149: server-81607767-2f01-59f9-be42-330697676670: 2021-07-10T01:05:01.246Z [INFO] server-81607767-2f01-59f9-be42-330697676670: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-81607767-2f01-59f9-be42-330697676670 Address:81607767-2f01-59f9-be42-330697676670}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:01.246Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:01.246Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-81607767-2f01-59f9-be42-330697676670: 2021-07-10T01:05:01.246Z [INFO] server-81607767-2f01-59f9-be42-330697676670: entering follower state: follower="Node at 81607767-2f01-59f9-be42-330697676670 [Follower]" leader= testing.go:149: server-81607767-2f01-59f9-be42-330697676670: 2021-07-10T01:05:01.246Z [INFO] server-81607767-2f01-59f9-be42-330697676670: entering follower state: follower="Node at 81607767-2f01-59f9-be42-330697676670 [Follower]" leader= testLoggerAdapter verbose: server-81607767-2f01-59f9-be42-330697676670: 2021-07-10T01:05:01.445Z [WARN] server-81607767-2f01-59f9-be42-330697676670: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-81607767-2f01-59f9-be42-330697676670: 2021-07-10T01:05:01.445Z [WARN] server-81607767-2f01-59f9-be42-330697676670: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-81607767-2f01-59f9-be42-330697676670: 2021-07-10T01:05:01.446Z [INFO] server-81607767-2f01-59f9-be42-330697676670: entering candidate state: node="Node at 81607767-2f01-59f9-be42-330697676670 [Candidate]" term=2 testing.go:149: server-81607767-2f01-59f9-be42-330697676670: 2021-07-10T01:05:01.446Z [INFO] server-81607767-2f01-59f9-be42-330697676670: entering candidate state: node="Node at 81607767-2f01-59f9-be42-330697676670 [Candidate]" term=2 testLoggerAdapter verbose: server-81607767-2f01-59f9-be42-330697676670: 2021-07-10T01:05:01.446Z [INFO] server-81607767-2f01-59f9-be42-330697676670: election won: tally=1 testing.go:149: server-81607767-2f01-59f9-be42-330697676670: 2021-07-10T01:05:01.446Z [INFO] server-81607767-2f01-59f9-be42-330697676670: election won: tally=1 testLoggerAdapter verbose: server-81607767-2f01-59f9-be42-330697676670: 2021-07-10T01:05:01.446Z [INFO] server-81607767-2f01-59f9-be42-330697676670: entering leader state: leader="Node at 81607767-2f01-59f9-be42-330697676670 [Leader]" testing.go:149: server-81607767-2f01-59f9-be42-330697676670: 2021-07-10T01:05:01.446Z [INFO] server-81607767-2f01-59f9-be42-330697676670: entering leader state: leader="Node at 81607767-2f01-59f9-be42-330697676670 [Leader]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:01.647Z [INFO] stable state for Leader reached at 2021-07-10 01:05:01.446412503 +0000 UTC m=+6.834570513 (1 nodes), 199.699966ms from start of poll, 200.07635ms from cluster start. Timeout at 2021-07-10 01:05:01.647485242 +0000 UTC m=+7.035643212, 201.072699ms after stability testing.go:149: cluster: 2021-07-10T01:05:01.647Z [INFO] stable state for Leader reached at 2021-07-10 01:05:01.446412503 +0000 UTC m=+6.834570513 (1 nodes), 199.699966ms from start of poll, 200.07635ms from cluster start. Timeout at 2021-07-10 01:05:01.647485242 +0000 UTC m=+7.035643212, 201.072699ms after stability --- PASS: TestRaft_ApplyConcurrent_Timeout (0.40s) === RUN TestRaft_JoinNode testLoggerAdapter verbose: server-77cd786a-e3b1-0556-30fa-9fa725f51fa2: 2021-07-10T01:05:01.648Z [INFO] server-77cd786a-e3b1-0556-30fa-9fa725f51fa2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77cd786a-e3b1-0556-30fa-9fa725f51fa2 Address:77cd786a-e3b1-0556-30fa-9fa725f51fa2} {Suffrage:Voter ID:server-547c628c-b604-48c9-f20b-4f20ab6dd0a0 Address:547c628c-b604-48c9-f20b-4f20ab6dd0a0}]" testing.go:149: server-77cd786a-e3b1-0556-30fa-9fa725f51fa2: 2021-07-10T01:05:01.648Z [INFO] server-77cd786a-e3b1-0556-30fa-9fa725f51fa2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77cd786a-e3b1-0556-30fa-9fa725f51fa2 Address:77cd786a-e3b1-0556-30fa-9fa725f51fa2} {Suffrage:Voter ID:server-547c628c-b604-48c9-f20b-4f20ab6dd0a0 Address:547c628c-b604-48c9-f20b-4f20ab6dd0a0}]" testLoggerAdapter verbose: server-77cd786a-e3b1-0556-30fa-9fa725f51fa2: 2021-07-10T01:05:01.649Z [INFO] server-77cd786a-e3b1-0556-30fa-9fa725f51fa2: entering follower state: follower="Node at 77cd786a-e3b1-0556-30fa-9fa725f51fa2 [Follower]" leader= testing.go:149: server-77cd786a-e3b1-0556-30fa-9fa725f51fa2: 2021-07-10T01:05:01.649Z [INFO] server-77cd786a-e3b1-0556-30fa-9fa725f51fa2: entering follower state: follower="Node at 77cd786a-e3b1-0556-30fa-9fa725f51fa2 [Follower]" leader= testLoggerAdapter verbose: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.649Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77cd786a-e3b1-0556-30fa-9fa725f51fa2 Address:77cd786a-e3b1-0556-30fa-9fa725f51fa2} {Suffrage:Voter ID:server-547c628c-b604-48c9-f20b-4f20ab6dd0a0 Address:547c628c-b604-48c9-f20b-4f20ab6dd0a0}]" testing.go:149: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.649Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77cd786a-e3b1-0556-30fa-9fa725f51fa2 Address:77cd786a-e3b1-0556-30fa-9fa725f51fa2} {Suffrage:Voter ID:server-547c628c-b604-48c9-f20b-4f20ab6dd0a0 Address:547c628c-b604-48c9-f20b-4f20ab6dd0a0}]" testLoggerAdapter verbose: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.649Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: entering follower state: follower="Node at 547c628c-b604-48c9-f20b-4f20ab6dd0a0 [Follower]" leader= testing.go:149: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.649Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: entering follower state: follower="Node at 547c628c-b604-48c9-f20b-4f20ab6dd0a0 [Follower]" leader= testLoggerAdapter verbose: server-d8ae26cb-1997-b273-9593-ba87939f154e: 2021-07-10T01:05:01.649Z [INFO] server-d8ae26cb-1997-b273-9593-ba87939f154e: initial configuration: index=0 servers=[] testing.go:149: server-d8ae26cb-1997-b273-9593-ba87939f154e: 2021-07-10T01:05:01.649Z [INFO] server-d8ae26cb-1997-b273-9593-ba87939f154e: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2021-07-10T01:05:01.649Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-d8ae26cb-1997-b273-9593-ba87939f154e: 2021-07-10T01:05:01.649Z [INFO] server-d8ae26cb-1997-b273-9593-ba87939f154e: entering follower state: follower="Node at d8ae26cb-1997-b273-9593-ba87939f154e [Follower]" leader= testing.go:149: cluster: 2021-07-10T01:05:01.649Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-d8ae26cb-1997-b273-9593-ba87939f154e: 2021-07-10T01:05:01.649Z [INFO] server-d8ae26cb-1997-b273-9593-ba87939f154e: entering follower state: follower="Node at d8ae26cb-1997-b273-9593-ba87939f154e [Follower]" leader= testLoggerAdapter verbose: server-d8ae26cb-1997-b273-9593-ba87939f154e: 2021-07-10T01:05:01.702Z [WARN] server-d8ae26cb-1997-b273-9593-ba87939f154e: no known peers, aborting election testing.go:149: server-d8ae26cb-1997-b273-9593-ba87939f154e: 2021-07-10T01:05:01.702Z [WARN] server-d8ae26cb-1997-b273-9593-ba87939f154e: no known peers, aborting election testLoggerAdapter verbose: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.716Z [WARN] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.716Z [WARN] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.716Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: entering candidate state: node="Node at 547c628c-b604-48c9-f20b-4f20ab6dd0a0 [Candidate]" term=2 testing.go:149: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.716Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: entering candidate state: node="Node at 547c628c-b604-48c9-f20b-4f20ab6dd0a0 [Candidate]" term=2 testLoggerAdapter verbose: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.716Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: election won: tally=2 testing.go:149: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.716Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: election won: tally=2 testLoggerAdapter verbose: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.716Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: entering leader state: leader="Node at 547c628c-b604-48c9-f20b-4f20ab6dd0a0 [Leader]" testing.go:149: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.716Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: entering leader state: leader="Node at 547c628c-b604-48c9-f20b-4f20ab6dd0a0 [Leader]" testLoggerAdapter verbose: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.716Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: added peer, starting replication: peer=server-77cd786a-e3b1-0556-30fa-9fa725f51fa2 testing.go:149: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.716Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: added peer, starting replication: peer=server-77cd786a-e3b1-0556-30fa-9fa725f51fa2 testLoggerAdapter verbose: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.716Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: pipelining replication: peer="{Voter server-77cd786a-e3b1-0556-30fa-9fa725f51fa2 77cd786a-e3b1-0556-30fa-9fa725f51fa2}" testing.go:149: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.716Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: pipelining replication: peer="{Voter server-77cd786a-e3b1-0556-30fa-9fa725f51fa2 77cd786a-e3b1-0556-30fa-9fa725f51fa2}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:01.821Z [INFO] stable state for Leader reached at 2021-07-10 01:05:01.716621291 +0000 UTC m=+7.104779261 (1 nodes), 66.883245ms from start of poll, 67.892336ms from cluster start. Timeout at 2021-07-10 01:05:01.821710419 +0000 UTC m=+7.209868389, 105.089128ms after stability testing.go:149: cluster: 2021-07-10T01:05:01.821Z [INFO] stable state for Leader reached at 2021-07-10 01:05:01.716621291 +0000 UTC m=+7.104779261 (1 nodes), 66.883245ms from start of poll, 67.892336ms from cluster start. Timeout at 2021-07-10 01:05:01.821710419 +0000 UTC m=+7.209868389, 105.089128ms after stability testLoggerAdapter verbose: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.821Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: updating configuration: command=AddStaging server-id=server-d8ae26cb-1997-b273-9593-ba87939f154e server-addr=d8ae26cb-1997-b273-9593-ba87939f154e servers="[{Suffrage:Voter ID:server-77cd786a-e3b1-0556-30fa-9fa725f51fa2 Address:77cd786a-e3b1-0556-30fa-9fa725f51fa2} {Suffrage:Voter ID:server-547c628c-b604-48c9-f20b-4f20ab6dd0a0 Address:547c628c-b604-48c9-f20b-4f20ab6dd0a0} {Suffrage:Voter ID:server-d8ae26cb-1997-b273-9593-ba87939f154e Address:d8ae26cb-1997-b273-9593-ba87939f154e}]" testing.go:149: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.821Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: updating configuration: command=AddStaging server-id=server-d8ae26cb-1997-b273-9593-ba87939f154e server-addr=d8ae26cb-1997-b273-9593-ba87939f154e servers="[{Suffrage:Voter ID:server-77cd786a-e3b1-0556-30fa-9fa725f51fa2 Address:77cd786a-e3b1-0556-30fa-9fa725f51fa2} {Suffrage:Voter ID:server-547c628c-b604-48c9-f20b-4f20ab6dd0a0 Address:547c628c-b604-48c9-f20b-4f20ab6dd0a0} {Suffrage:Voter ID:server-d8ae26cb-1997-b273-9593-ba87939f154e Address:d8ae26cb-1997-b273-9593-ba87939f154e}]" testLoggerAdapter verbose: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.822Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: added peer, starting replication: peer=server-d8ae26cb-1997-b273-9593-ba87939f154e testing.go:149: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.822Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: added peer, starting replication: peer=server-d8ae26cb-1997-b273-9593-ba87939f154e testLoggerAdapter verbose: server-d8ae26cb-1997-b273-9593-ba87939f154e: 2021-07-10T01:05:01.822Z [WARN] server-d8ae26cb-1997-b273-9593-ba87939f154e: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: server-d8ae26cb-1997-b273-9593-ba87939f154e: 2021-07-10T01:05:01.822Z [WARN] server-d8ae26cb-1997-b273-9593-ba87939f154e: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:01.822Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:01.822Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.822Z [WARN] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: appendEntries rejected, sending older logs: peer="{Voter server-d8ae26cb-1997-b273-9593-ba87939f154e d8ae26cb-1997-b273-9593-ba87939f154e}" next=1 testing.go:149: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.822Z [WARN] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: appendEntries rejected, sending older logs: peer="{Voter server-d8ae26cb-1997-b273-9593-ba87939f154e d8ae26cb-1997-b273-9593-ba87939f154e}" next=1 testLoggerAdapter verbose: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.822Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: pipelining replication: peer="{Voter server-d8ae26cb-1997-b273-9593-ba87939f154e d8ae26cb-1997-b273-9593-ba87939f154e}" testing.go:149: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.822Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: pipelining replication: peer="{Voter server-d8ae26cb-1997-b273-9593-ba87939f154e d8ae26cb-1997-b273-9593-ba87939f154e}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:01.927Z [INFO] stable state for Leader reached at 2021-07-10 01:05:01.822609649 +0000 UTC m=+7.210767599 (1 nodes), 1.72µs from start of poll, 173.880674ms from cluster start. Timeout at 2021-07-10 01:05:01.927674576 +0000 UTC m=+7.315832546, 105.064947ms after stability testing.go:149: cluster: 2021-07-10T01:05:01.927Z [INFO] stable state for Leader reached at 2021-07-10 01:05:01.822609649 +0000 UTC m=+7.210767599 (1 nodes), 1.72µs from start of poll, 173.880674ms from cluster start. Timeout at 2021-07-10 01:05:01.927674576 +0000 UTC m=+7.315832546, 105.064947ms after stability testLoggerAdapter verbose: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.928Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: aborting pipeline replication: peer="{Voter server-d8ae26cb-1997-b273-9593-ba87939f154e d8ae26cb-1997-b273-9593-ba87939f154e}" testing.go:149: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.928Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: aborting pipeline replication: peer="{Voter server-d8ae26cb-1997-b273-9593-ba87939f154e d8ae26cb-1997-b273-9593-ba87939f154e}" testLoggerAdapter verbose: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.928Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: aborting pipeline replication: peer="{Voter server-77cd786a-e3b1-0556-30fa-9fa725f51fa2 77cd786a-e3b1-0556-30fa-9fa725f51fa2}" testing.go:149: server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: 2021-07-10T01:05:01.928Z [INFO] server-547c628c-b604-48c9-f20b-4f20ab6dd0a0: aborting pipeline replication: peer="{Voter server-77cd786a-e3b1-0556-30fa-9fa725f51fa2 77cd786a-e3b1-0556-30fa-9fa725f51fa2}" --- PASS: TestRaft_JoinNode (0.28s) === RUN TestRaft_JoinNode_ConfigStore testLoggerAdapter verbose: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:01.929Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b845038-ba98-0d58-7127-c9c89fd5fe60 Address:7b845038-ba98-0d58-7127-c9c89fd5fe60}]" testing.go:149: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:01.929Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b845038-ba98-0d58-7127-c9c89fd5fe60 Address:7b845038-ba98-0d58-7127-c9c89fd5fe60}]" testLoggerAdapter verbose: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:01.929Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: entering follower state: follower="Node at 7b845038-ba98-0d58-7127-c9c89fd5fe60 [Follower]" leader= testing.go:149: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:01.929Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: entering follower state: follower="Node at 7b845038-ba98-0d58-7127-c9c89fd5fe60 [Follower]" leader= testLoggerAdapter verbose: server-3b5470ae-10fd-3291-beba-a039712e0a65: 2021-07-10T01:05:01.929Z [INFO] server-3b5470ae-10fd-3291-beba-a039712e0a65: initial configuration: index=0 servers=[] testing.go:149: server-3b5470ae-10fd-3291-beba-a039712e0a65: 2021-07-10T01:05:01.929Z [INFO] server-3b5470ae-10fd-3291-beba-a039712e0a65: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-3b5470ae-10fd-3291-beba-a039712e0a65: 2021-07-10T01:05:01.929Z [INFO] server-3b5470ae-10fd-3291-beba-a039712e0a65: entering follower state: follower="Node at 3b5470ae-10fd-3291-beba-a039712e0a65 [Follower]" leader= testing.go:149: server-3b5470ae-10fd-3291-beba-a039712e0a65: 2021-07-10T01:05:01.929Z [INFO] server-3b5470ae-10fd-3291-beba-a039712e0a65: entering follower state: follower="Node at 3b5470ae-10fd-3291-beba-a039712e0a65 [Follower]" leader= testLoggerAdapter verbose: server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690: 2021-07-10T01:05:01.930Z [INFO] server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690: initial configuration: index=0 servers=[] testing.go:149: server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690: 2021-07-10T01:05:01.930Z [INFO] server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2021-07-10T01:05:01.930Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690: 2021-07-10T01:05:01.930Z [INFO] server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690: entering follower state: follower="Node at 1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690 [Follower]" leader= testing.go:149: cluster: 2021-07-10T01:05:01.930Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690: 2021-07-10T01:05:01.930Z [INFO] server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690: entering follower state: follower="Node at 1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690 [Follower]" leader= testLoggerAdapter verbose: server-3b5470ae-10fd-3291-beba-a039712e0a65: 2021-07-10T01:05:01.987Z [WARN] server-3b5470ae-10fd-3291-beba-a039712e0a65: no known peers, aborting election testing.go:149: server-3b5470ae-10fd-3291-beba-a039712e0a65: 2021-07-10T01:05:01.987Z [WARN] server-3b5470ae-10fd-3291-beba-a039712e0a65: no known peers, aborting election testLoggerAdapter verbose: server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690: 2021-07-10T01:05:02.005Z [WARN] server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690: no known peers, aborting election testing.go:149: server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690: 2021-07-10T01:05:02.005Z [WARN] server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690: no known peers, aborting election testLoggerAdapter verbose: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.014Z [WARN] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.014Z [WARN] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.014Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: entering candidate state: node="Node at 7b845038-ba98-0d58-7127-c9c89fd5fe60 [Candidate]" term=2 testing.go:149: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.014Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: entering candidate state: node="Node at 7b845038-ba98-0d58-7127-c9c89fd5fe60 [Candidate]" term=2 testLoggerAdapter verbose: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.014Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: election won: tally=1 testing.go:149: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.014Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: election won: tally=1 testLoggerAdapter verbose: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.014Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: entering leader state: leader="Node at 7b845038-ba98-0d58-7127-c9c89fd5fe60 [Leader]" testing.go:149: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.014Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: entering leader state: leader="Node at 7b845038-ba98-0d58-7127-c9c89fd5fe60 [Leader]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:02.120Z [INFO] stable state for Leader reached at 2021-07-10 01:05:02.014785554 +0000 UTC m=+7.402943544 (1 nodes), 84.614392ms from start of poll, 85.748383ms from cluster start. Timeout at 2021-07-10 01:05:02.119865762 +0000 UTC m=+7.508023732, 105.080188ms after stability testing.go:149: cluster: 2021-07-10T01:05:02.120Z [INFO] stable state for Leader reached at 2021-07-10 01:05:02.014785554 +0000 UTC m=+7.402943544 (1 nodes), 84.614392ms from start of poll, 85.748383ms from cluster start. Timeout at 2021-07-10 01:05:02.119865762 +0000 UTC m=+7.508023732, 105.080188ms after stability testLoggerAdapter verbose: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.120Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: updating configuration: command=AddStaging server-id=server-3b5470ae-10fd-3291-beba-a039712e0a65 server-addr=3b5470ae-10fd-3291-beba-a039712e0a65 servers="[{Suffrage:Voter ID:server-7b845038-ba98-0d58-7127-c9c89fd5fe60 Address:7b845038-ba98-0d58-7127-c9c89fd5fe60} {Suffrage:Voter ID:server-3b5470ae-10fd-3291-beba-a039712e0a65 Address:3b5470ae-10fd-3291-beba-a039712e0a65}]" testing.go:149: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.120Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: updating configuration: command=AddStaging server-id=server-3b5470ae-10fd-3291-beba-a039712e0a65 server-addr=3b5470ae-10fd-3291-beba-a039712e0a65 servers="[{Suffrage:Voter ID:server-7b845038-ba98-0d58-7127-c9c89fd5fe60 Address:7b845038-ba98-0d58-7127-c9c89fd5fe60} {Suffrage:Voter ID:server-3b5470ae-10fd-3291-beba-a039712e0a65 Address:3b5470ae-10fd-3291-beba-a039712e0a65}]" testLoggerAdapter verbose: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.120Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: added peer, starting replication: peer=server-3b5470ae-10fd-3291-beba-a039712e0a65 testing.go:149: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.120Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: added peer, starting replication: peer=server-3b5470ae-10fd-3291-beba-a039712e0a65 testLoggerAdapter verbose: cluster: 2021-07-10T01:05:02.120Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:02.120Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-3b5470ae-10fd-3291-beba-a039712e0a65: 2021-07-10T01:05:02.120Z [WARN] server-3b5470ae-10fd-3291-beba-a039712e0a65: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: server-3b5470ae-10fd-3291-beba-a039712e0a65: 2021-07-10T01:05:02.120Z [WARN] server-3b5470ae-10fd-3291-beba-a039712e0a65: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.120Z [WARN] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: appendEntries rejected, sending older logs: peer="{Voter server-3b5470ae-10fd-3291-beba-a039712e0a65 3b5470ae-10fd-3291-beba-a039712e0a65}" next=1 testing.go:149: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.120Z [WARN] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: appendEntries rejected, sending older logs: peer="{Voter server-3b5470ae-10fd-3291-beba-a039712e0a65 3b5470ae-10fd-3291-beba-a039712e0a65}" next=1 testLoggerAdapter verbose: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.120Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: pipelining replication: peer="{Voter server-3b5470ae-10fd-3291-beba-a039712e0a65 3b5470ae-10fd-3291-beba-a039712e0a65}" testing.go:149: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.120Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: pipelining replication: peer="{Voter server-3b5470ae-10fd-3291-beba-a039712e0a65 3b5470ae-10fd-3291-beba-a039712e0a65}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:02.225Z [INFO] stable state for Leader reached at 2021-07-10 01:05:02.12063029 +0000 UTC m=+7.508788280 (1 nodes), 2.14µs from start of poll, 191.593119ms from cluster start. Timeout at 2021-07-10 01:05:02.225687238 +0000 UTC m=+7.613845228, 105.056948ms after stability testing.go:149: cluster: 2021-07-10T01:05:02.225Z [INFO] stable state for Leader reached at 2021-07-10 01:05:02.12063029 +0000 UTC m=+7.508788280 (1 nodes), 2.14µs from start of poll, 191.593119ms from cluster start. Timeout at 2021-07-10 01:05:02.225687238 +0000 UTC m=+7.613845228, 105.056948ms after stability testLoggerAdapter verbose: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.225Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: updating configuration: command=AddStaging server-id=server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690 server-addr=1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690 servers="[{Suffrage:Voter ID:server-7b845038-ba98-0d58-7127-c9c89fd5fe60 Address:7b845038-ba98-0d58-7127-c9c89fd5fe60} {Suffrage:Voter ID:server-3b5470ae-10fd-3291-beba-a039712e0a65 Address:3b5470ae-10fd-3291-beba-a039712e0a65} {Suffrage:Voter ID:server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690 Address:1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690}]" testing.go:149: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.225Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: updating configuration: command=AddStaging server-id=server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690 server-addr=1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690 servers="[{Suffrage:Voter ID:server-7b845038-ba98-0d58-7127-c9c89fd5fe60 Address:7b845038-ba98-0d58-7127-c9c89fd5fe60} {Suffrage:Voter ID:server-3b5470ae-10fd-3291-beba-a039712e0a65 Address:3b5470ae-10fd-3291-beba-a039712e0a65} {Suffrage:Voter ID:server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690 Address:1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690}]" testLoggerAdapter verbose: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.226Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: added peer, starting replication: peer=server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690 testing.go:149: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.226Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: added peer, starting replication: peer=server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690 testLoggerAdapter verbose: server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690: 2021-07-10T01:05:02.226Z [WARN] server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690: failed to get previous log: previous-index=4 last-index=0 error="log not found" testing.go:149: server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690: 2021-07-10T01:05:02.226Z [WARN] server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690: failed to get previous log: previous-index=4 last-index=0 error="log not found" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:02.226Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.226Z [WARN] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: appendEntries rejected, sending older logs: peer="{Voter server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690 1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690}" next=1 testing.go:149: cluster: 2021-07-10T01:05:02.226Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.226Z [WARN] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: appendEntries rejected, sending older logs: peer="{Voter server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690 1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690}" next=1 testLoggerAdapter verbose: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.226Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: pipelining replication: peer="{Voter server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690 1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690}" testing.go:149: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.226Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: pipelining replication: peer="{Voter server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690 1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:02.331Z [INFO] stable state for Leader reached at 2021-07-10 01:05:02.226492646 +0000 UTC m=+7.614650596 (1 nodes), 1.32µs from start of poll, 297.455435ms from cluster start. Timeout at 2021-07-10 01:05:02.331558674 +0000 UTC m=+7.719716664, 105.066068ms after stability testing.go:149: cluster: 2021-07-10T01:05:02.331Z [INFO] stable state for Leader reached at 2021-07-10 01:05:02.226492646 +0000 UTC m=+7.614650596 (1 nodes), 1.32µs from start of poll, 297.455435ms from cluster start. Timeout at 2021-07-10 01:05:02.331558674 +0000 UTC m=+7.719716664, 105.066068ms after stability testLoggerAdapter verbose: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.331Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: aborting pipeline replication: peer="{Voter server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690 1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690}" testing.go:149: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.331Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: aborting pipeline replication: peer="{Voter server-1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690 1b75b6a8-4e1d-84ab-0a0c-9c21cbb17690}" testLoggerAdapter verbose: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.331Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: aborting pipeline replication: peer="{Voter server-3b5470ae-10fd-3291-beba-a039712e0a65 3b5470ae-10fd-3291-beba-a039712e0a65}" testing.go:149: server-7b845038-ba98-0d58-7127-c9c89fd5fe60: 2021-07-10T01:05:02.331Z [INFO] server-7b845038-ba98-0d58-7127-c9c89fd5fe60: aborting pipeline replication: peer="{Voter server-3b5470ae-10fd-3291-beba-a039712e0a65 3b5470ae-10fd-3291-beba-a039712e0a65}" --- PASS: TestRaft_JoinNode_ConfigStore (0.40s) === RUN TestRaft_RemoveFollower testLoggerAdapter verbose: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.333Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-19802584-fb94-dbc0-2edc-c35ec8be4ae1 Address:19802584-fb94-dbc0-2edc-c35ec8be4ae1} {Suffrage:Voter ID:server-37f9809a-6e60-f4c9-5aff-4b0b53e90820 Address:37f9809a-6e60-f4c9-5aff-4b0b53e90820} {Suffrage:Voter ID:server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1 Address:a18410a6-fd05-89ff-3d5b-fb3a48be0ec1}]" testing.go:149: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.333Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-19802584-fb94-dbc0-2edc-c35ec8be4ae1 Address:19802584-fb94-dbc0-2edc-c35ec8be4ae1} {Suffrage:Voter ID:server-37f9809a-6e60-f4c9-5aff-4b0b53e90820 Address:37f9809a-6e60-f4c9-5aff-4b0b53e90820} {Suffrage:Voter ID:server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1 Address:a18410a6-fd05-89ff-3d5b-fb3a48be0ec1}]" testLoggerAdapter verbose: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.333Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: entering follower state: follower="Node at 19802584-fb94-dbc0-2edc-c35ec8be4ae1 [Follower]" leader= testing.go:149: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.333Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: entering follower state: follower="Node at 19802584-fb94-dbc0-2edc-c35ec8be4ae1 [Follower]" leader= testLoggerAdapter verbose: server-37f9809a-6e60-f4c9-5aff-4b0b53e90820: 2021-07-10T01:05:02.333Z [INFO] server-37f9809a-6e60-f4c9-5aff-4b0b53e90820: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-19802584-fb94-dbc0-2edc-c35ec8be4ae1 Address:19802584-fb94-dbc0-2edc-c35ec8be4ae1} {Suffrage:Voter ID:server-37f9809a-6e60-f4c9-5aff-4b0b53e90820 Address:37f9809a-6e60-f4c9-5aff-4b0b53e90820} {Suffrage:Voter ID:server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1 Address:a18410a6-fd05-89ff-3d5b-fb3a48be0ec1}]" testing.go:149: server-37f9809a-6e60-f4c9-5aff-4b0b53e90820: 2021-07-10T01:05:02.333Z [INFO] server-37f9809a-6e60-f4c9-5aff-4b0b53e90820: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-19802584-fb94-dbc0-2edc-c35ec8be4ae1 Address:19802584-fb94-dbc0-2edc-c35ec8be4ae1} {Suffrage:Voter ID:server-37f9809a-6e60-f4c9-5aff-4b0b53e90820 Address:37f9809a-6e60-f4c9-5aff-4b0b53e90820} {Suffrage:Voter ID:server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1 Address:a18410a6-fd05-89ff-3d5b-fb3a48be0ec1}]" testLoggerAdapter verbose: server-37f9809a-6e60-f4c9-5aff-4b0b53e90820: 2021-07-10T01:05:02.333Z [INFO] server-37f9809a-6e60-f4c9-5aff-4b0b53e90820: entering follower state: follower="Node at 37f9809a-6e60-f4c9-5aff-4b0b53e90820 [Follower]" leader= testing.go:149: server-37f9809a-6e60-f4c9-5aff-4b0b53e90820: 2021-07-10T01:05:02.333Z [INFO] server-37f9809a-6e60-f4c9-5aff-4b0b53e90820: entering follower state: follower="Node at 37f9809a-6e60-f4c9-5aff-4b0b53e90820 [Follower]" leader= testLoggerAdapter verbose: server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1: 2021-07-10T01:05:02.333Z [INFO] server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-19802584-fb94-dbc0-2edc-c35ec8be4ae1 Address:19802584-fb94-dbc0-2edc-c35ec8be4ae1} {Suffrage:Voter ID:server-37f9809a-6e60-f4c9-5aff-4b0b53e90820 Address:37f9809a-6e60-f4c9-5aff-4b0b53e90820} {Suffrage:Voter ID:server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1 Address:a18410a6-fd05-89ff-3d5b-fb3a48be0ec1}]" testing.go:149: server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1: 2021-07-10T01:05:02.333Z [INFO] server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-19802584-fb94-dbc0-2edc-c35ec8be4ae1 Address:19802584-fb94-dbc0-2edc-c35ec8be4ae1} {Suffrage:Voter ID:server-37f9809a-6e60-f4c9-5aff-4b0b53e90820 Address:37f9809a-6e60-f4c9-5aff-4b0b53e90820} {Suffrage:Voter ID:server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1 Address:a18410a6-fd05-89ff-3d5b-fb3a48be0ec1}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:02.333Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:02.333Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1: 2021-07-10T01:05:02.333Z [INFO] server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1: entering follower state: follower="Node at a18410a6-fd05-89ff-3d5b-fb3a48be0ec1 [Follower]" leader= testing.go:149: server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1: 2021-07-10T01:05:02.333Z [INFO] server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1: entering follower state: follower="Node at a18410a6-fd05-89ff-3d5b-fb3a48be0ec1 [Follower]" leader= testLoggerAdapter verbose: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.393Z [WARN] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.393Z [WARN] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.393Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: entering candidate state: node="Node at 19802584-fb94-dbc0-2edc-c35ec8be4ae1 [Candidate]" term=2 testing.go:149: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.393Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: entering candidate state: node="Node at 19802584-fb94-dbc0-2edc-c35ec8be4ae1 [Candidate]" term=2 testLoggerAdapter verbose: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.394Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: election won: tally=2 testing.go:149: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.394Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: election won: tally=2 testLoggerAdapter verbose: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.394Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: entering leader state: leader="Node at 19802584-fb94-dbc0-2edc-c35ec8be4ae1 [Leader]" testing.go:149: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.394Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: entering leader state: leader="Node at 19802584-fb94-dbc0-2edc-c35ec8be4ae1 [Leader]" testLoggerAdapter verbose: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.394Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: added peer, starting replication: peer=server-37f9809a-6e60-f4c9-5aff-4b0b53e90820 testing.go:149: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.394Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: added peer, starting replication: peer=server-37f9809a-6e60-f4c9-5aff-4b0b53e90820 testLoggerAdapter verbose: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.394Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: added peer, starting replication: peer=server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1 testing.go:149: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.394Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: added peer, starting replication: peer=server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1 testLoggerAdapter verbose: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.394Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: pipelining replication: peer="{Voter server-37f9809a-6e60-f4c9-5aff-4b0b53e90820 37f9809a-6e60-f4c9-5aff-4b0b53e90820}" testing.go:149: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.394Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: pipelining replication: peer="{Voter server-37f9809a-6e60-f4c9-5aff-4b0b53e90820 37f9809a-6e60-f4c9-5aff-4b0b53e90820}" testLoggerAdapter verbose: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.394Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: pipelining replication: peer="{Voter server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1 a18410a6-fd05-89ff-3d5b-fb3a48be0ec1}" testing.go:149: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.394Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: pipelining replication: peer="{Voter server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1 a18410a6-fd05-89ff-3d5b-fb3a48be0ec1}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:02.499Z [INFO] stable state for Leader reached at 2021-07-10 01:05:02.394300515 +0000 UTC m=+7.782458505 (1 nodes), 60.278555ms from start of poll, 61.239945ms from cluster start. Timeout at 2021-07-10 01:05:02.499395283 +0000 UTC m=+7.887553273, 105.094768ms after stability testing.go:149: cluster: 2021-07-10T01:05:02.499Z [INFO] stable state for Leader reached at 2021-07-10 01:05:02.394300515 +0000 UTC m=+7.782458505 (1 nodes), 60.278555ms from start of poll, 61.239945ms from cluster start. Timeout at 2021-07-10 01:05:02.499395283 +0000 UTC m=+7.887553273, 105.094768ms after stability testLoggerAdapter verbose: cluster: 2021-07-10T01:05:02.504Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-07-10T01:05:02.504Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-07-10T01:05:02.610Z [INFO] stable state for Follower reached at 2021-07-10 01:05:02.504901041 +0000 UTC m=+7.893058991 (2 nodes), 3.22µs from start of poll, 171.840431ms from cluster start. Timeout at 2021-07-10 01:05:02.609974229 +0000 UTC m=+7.998132199, 105.073208ms after stability testing.go:149: cluster: 2021-07-10T01:05:02.610Z [INFO] stable state for Follower reached at 2021-07-10 01:05:02.504901041 +0000 UTC m=+7.893058991 (2 nodes), 3.22µs from start of poll, 171.840431ms from cluster start. Timeout at 2021-07-10 01:05:02.609974229 +0000 UTC m=+7.998132199, 105.073208ms after stability testLoggerAdapter verbose: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.610Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: updating configuration: command=RemoveServer server-id=server-37f9809a-6e60-f4c9-5aff-4b0b53e90820 server-addr= servers="[{Suffrage:Voter ID:server-19802584-fb94-dbc0-2edc-c35ec8be4ae1 Address:19802584-fb94-dbc0-2edc-c35ec8be4ae1} {Suffrage:Voter ID:server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1 Address:a18410a6-fd05-89ff-3d5b-fb3a48be0ec1}]" testing.go:149: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.610Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: updating configuration: command=RemoveServer server-id=server-37f9809a-6e60-f4c9-5aff-4b0b53e90820 server-addr= servers="[{Suffrage:Voter ID:server-19802584-fb94-dbc0-2edc-c35ec8be4ae1 Address:19802584-fb94-dbc0-2edc-c35ec8be4ae1} {Suffrage:Voter ID:server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1 Address:a18410a6-fd05-89ff-3d5b-fb3a48be0ec1}]" testLoggerAdapter verbose: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.610Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: removed peer, stopping replication: peer=server-37f9809a-6e60-f4c9-5aff-4b0b53e90820 last-index=3 testing.go:149: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.610Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: removed peer, stopping replication: peer=server-37f9809a-6e60-f4c9-5aff-4b0b53e90820 last-index=3 testLoggerAdapter verbose: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.610Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: aborting pipeline replication: peer="{Voter server-37f9809a-6e60-f4c9-5aff-4b0b53e90820 37f9809a-6e60-f4c9-5aff-4b0b53e90820}" testing.go:149: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.610Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: aborting pipeline replication: peer="{Voter server-37f9809a-6e60-f4c9-5aff-4b0b53e90820 37f9809a-6e60-f4c9-5aff-4b0b53e90820}" testLoggerAdapter verbose: server-37f9809a-6e60-f4c9-5aff-4b0b53e90820: 2021-07-10T01:05:02.698Z [WARN] server-37f9809a-6e60-f4c9-5aff-4b0b53e90820: heartbeat timeout reached, starting election: last-leader=19802584-fb94-dbc0-2edc-c35ec8be4ae1 testing.go:149: server-37f9809a-6e60-f4c9-5aff-4b0b53e90820: 2021-07-10T01:05:02.698Z [WARN] server-37f9809a-6e60-f4c9-5aff-4b0b53e90820: heartbeat timeout reached, starting election: last-leader=19802584-fb94-dbc0-2edc-c35ec8be4ae1 testLoggerAdapter verbose: server-37f9809a-6e60-f4c9-5aff-4b0b53e90820: 2021-07-10T01:05:02.698Z [INFO] server-37f9809a-6e60-f4c9-5aff-4b0b53e90820: entering candidate state: node="Node at 37f9809a-6e60-f4c9-5aff-4b0b53e90820 [Candidate]" term=3 testing.go:149: server-37f9809a-6e60-f4c9-5aff-4b0b53e90820: 2021-07-10T01:05:02.698Z [INFO] server-37f9809a-6e60-f4c9-5aff-4b0b53e90820: entering candidate state: node="Node at 37f9809a-6e60-f4c9-5aff-4b0b53e90820 [Candidate]" term=3 testLoggerAdapter verbose: server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1: 2021-07-10T01:05:02.698Z [WARN] server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1: rejecting vote request since we have a leader: from=37f9809a-6e60-f4c9-5aff-4b0b53e90820 leader=19802584-fb94-dbc0-2edc-c35ec8be4ae1 testing.go:149: server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1: 2021-07-10T01:05:02.698Z [WARN] server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1: rejecting vote request since we have a leader: from=37f9809a-6e60-f4c9-5aff-4b0b53e90820 leader=19802584-fb94-dbc0-2edc-c35ec8be4ae1 testLoggerAdapter verbose: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.698Z [WARN] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: rejecting vote request since we have a leader: from=37f9809a-6e60-f4c9-5aff-4b0b53e90820 leader=19802584-fb94-dbc0-2edc-c35ec8be4ae1 testing.go:149: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.698Z [WARN] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: rejecting vote request since we have a leader: from=37f9809a-6e60-f4c9-5aff-4b0b53e90820 leader=19802584-fb94-dbc0-2edc-c35ec8be4ae1 testLoggerAdapter verbose: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.715Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: aborting pipeline replication: peer="{Voter server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1 a18410a6-fd05-89ff-3d5b-fb3a48be0ec1}" testing.go:149: server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: 2021-07-10T01:05:02.715Z [INFO] server-19802584-fb94-dbc0-2edc-c35ec8be4ae1: aborting pipeline replication: peer="{Voter server-a18410a6-fd05-89ff-3d5b-fb3a48be0ec1 a18410a6-fd05-89ff-3d5b-fb3a48be0ec1}" --- PASS: TestRaft_RemoveFollower (0.38s) === RUN TestRaft_RemoveLeader testLoggerAdapter verbose: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.717Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab Address:2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab} {Suffrage:Voter ID:server-caae0ef8-5ef3-3038-2a73-a05d64d77146 Address:caae0ef8-5ef3-3038-2a73-a05d64d77146} {Suffrage:Voter ID:server-2bbd6b80-00c9-8a45-10e8-db43464eb80a Address:2bbd6b80-00c9-8a45-10e8-db43464eb80a}]" testing.go:149: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.717Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab Address:2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab} {Suffrage:Voter ID:server-caae0ef8-5ef3-3038-2a73-a05d64d77146 Address:caae0ef8-5ef3-3038-2a73-a05d64d77146} {Suffrage:Voter ID:server-2bbd6b80-00c9-8a45-10e8-db43464eb80a Address:2bbd6b80-00c9-8a45-10e8-db43464eb80a}]" testLoggerAdapter verbose: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.717Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: entering follower state: follower="Node at 2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab [Follower]" leader= testing.go:149: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.717Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: entering follower state: follower="Node at 2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab [Follower]" leader= testLoggerAdapter verbose: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:02.717Z [INFO] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab Address:2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab} {Suffrage:Voter ID:server-caae0ef8-5ef3-3038-2a73-a05d64d77146 Address:caae0ef8-5ef3-3038-2a73-a05d64d77146} {Suffrage:Voter ID:server-2bbd6b80-00c9-8a45-10e8-db43464eb80a Address:2bbd6b80-00c9-8a45-10e8-db43464eb80a}]" testing.go:149: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:02.717Z [INFO] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab Address:2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab} {Suffrage:Voter ID:server-caae0ef8-5ef3-3038-2a73-a05d64d77146 Address:caae0ef8-5ef3-3038-2a73-a05d64d77146} {Suffrage:Voter ID:server-2bbd6b80-00c9-8a45-10e8-db43464eb80a Address:2bbd6b80-00c9-8a45-10e8-db43464eb80a}]" testLoggerAdapter verbose: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:02.717Z [INFO] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: entering follower state: follower="Node at caae0ef8-5ef3-3038-2a73-a05d64d77146 [Follower]" leader= testing.go:149: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:02.717Z [INFO] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: entering follower state: follower="Node at caae0ef8-5ef3-3038-2a73-a05d64d77146 [Follower]" leader= testLoggerAdapter verbose: server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: 2021-07-10T01:05:02.718Z [INFO] server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab Address:2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab} {Suffrage:Voter ID:server-caae0ef8-5ef3-3038-2a73-a05d64d77146 Address:caae0ef8-5ef3-3038-2a73-a05d64d77146} {Suffrage:Voter ID:server-2bbd6b80-00c9-8a45-10e8-db43464eb80a Address:2bbd6b80-00c9-8a45-10e8-db43464eb80a}]" testing.go:149: server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: 2021-07-10T01:05:02.718Z [INFO] server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab Address:2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab} {Suffrage:Voter ID:server-caae0ef8-5ef3-3038-2a73-a05d64d77146 Address:caae0ef8-5ef3-3038-2a73-a05d64d77146} {Suffrage:Voter ID:server-2bbd6b80-00c9-8a45-10e8-db43464eb80a Address:2bbd6b80-00c9-8a45-10e8-db43464eb80a}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:02.718Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:02.718Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: 2021-07-10T01:05:02.718Z [INFO] server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: entering follower state: follower="Node at 2bbd6b80-00c9-8a45-10e8-db43464eb80a [Follower]" leader= testing.go:149: server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: 2021-07-10T01:05:02.718Z [INFO] server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: entering follower state: follower="Node at 2bbd6b80-00c9-8a45-10e8-db43464eb80a [Follower]" leader= testLoggerAdapter verbose: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.774Z [WARN] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.774Z [WARN] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.775Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: entering candidate state: node="Node at 2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab [Candidate]" term=2 testing.go:149: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.775Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: entering candidate state: node="Node at 2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab [Candidate]" term=2 testLoggerAdapter verbose: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.775Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: election won: tally=2 testing.go:149: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.775Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: election won: tally=2 testLoggerAdapter verbose: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.775Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: entering leader state: leader="Node at 2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab [Leader]" testing.go:149: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.775Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: entering leader state: leader="Node at 2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab [Leader]" testLoggerAdapter verbose: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.775Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: added peer, starting replication: peer=server-caae0ef8-5ef3-3038-2a73-a05d64d77146 testing.go:149: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.775Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: added peer, starting replication: peer=server-caae0ef8-5ef3-3038-2a73-a05d64d77146 testLoggerAdapter verbose: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.775Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: added peer, starting replication: peer=server-2bbd6b80-00c9-8a45-10e8-db43464eb80a testing.go:149: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.775Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: added peer, starting replication: peer=server-2bbd6b80-00c9-8a45-10e8-db43464eb80a testLoggerAdapter verbose: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.775Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: pipelining replication: peer="{Voter server-caae0ef8-5ef3-3038-2a73-a05d64d77146 caae0ef8-5ef3-3038-2a73-a05d64d77146}" testing.go:149: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.775Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: pipelining replication: peer="{Voter server-caae0ef8-5ef3-3038-2a73-a05d64d77146 caae0ef8-5ef3-3038-2a73-a05d64d77146}" testLoggerAdapter verbose: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.775Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: pipelining replication: peer="{Voter server-2bbd6b80-00c9-8a45-10e8-db43464eb80a 2bbd6b80-00c9-8a45-10e8-db43464eb80a}" testing.go:149: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.775Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: pipelining replication: peer="{Voter server-2bbd6b80-00c9-8a45-10e8-db43464eb80a 2bbd6b80-00c9-8a45-10e8-db43464eb80a}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:02.880Z [INFO] stable state for Leader reached at 2021-07-10 01:05:02.775450374 +0000 UTC m=+8.163608344 (1 nodes), 57.235604ms from start of poll, 58.260595ms from cluster start. Timeout at 2021-07-10 01:05:02.880514741 +0000 UTC m=+8.268672731, 105.064387ms after stability testing.go:149: cluster: 2021-07-10T01:05:02.880Z [INFO] stable state for Leader reached at 2021-07-10 01:05:02.775450374 +0000 UTC m=+8.163608344 (1 nodes), 57.235604ms from start of poll, 58.260595ms from cluster start. Timeout at 2021-07-10 01:05:02.880514741 +0000 UTC m=+8.268672731, 105.064387ms after stability testLoggerAdapter verbose: cluster: 2021-07-10T01:05:02.885Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-07-10T01:05:02.885Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-07-10T01:05:02.991Z [INFO] stable state for Follower reached at 2021-07-10 01:05:02.885943219 +0000 UTC m=+8.274101189 (2 nodes), 3.441µs from start of poll, 168.75344ms from cluster start. Timeout at 2021-07-10 01:05:02.990997006 +0000 UTC m=+8.379154996, 105.053807ms after stability testing.go:149: cluster: 2021-07-10T01:05:02.991Z [INFO] stable state for Follower reached at 2021-07-10 01:05:02.885943219 +0000 UTC m=+8.274101189 (2 nodes), 3.441µs from start of poll, 168.75344ms from cluster start. Timeout at 2021-07-10 01:05:02.990997006 +0000 UTC m=+8.379154996, 105.053807ms after stability testLoggerAdapter verbose: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.991Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: updating configuration: command=RemoveServer server-id=server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab server-addr= servers="[{Suffrage:Voter ID:server-caae0ef8-5ef3-3038-2a73-a05d64d77146 Address:caae0ef8-5ef3-3038-2a73-a05d64d77146} {Suffrage:Voter ID:server-2bbd6b80-00c9-8a45-10e8-db43464eb80a Address:2bbd6b80-00c9-8a45-10e8-db43464eb80a}]" testing.go:149: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.991Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: updating configuration: command=RemoveServer server-id=server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab server-addr= servers="[{Suffrage:Voter ID:server-caae0ef8-5ef3-3038-2a73-a05d64d77146 Address:caae0ef8-5ef3-3038-2a73-a05d64d77146} {Suffrage:Voter ID:server-2bbd6b80-00c9-8a45-10e8-db43464eb80a Address:2bbd6b80-00c9-8a45-10e8-db43464eb80a}]" testLoggerAdapter verbose: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.991Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: removed ourself, shutting down testing.go:149: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.991Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: removed ourself, shutting down testLoggerAdapter verbose: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.991Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: aborting pipeline replication: peer="{Voter server-2bbd6b80-00c9-8a45-10e8-db43464eb80a 2bbd6b80-00c9-8a45-10e8-db43464eb80a}" testing.go:149: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.991Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: aborting pipeline replication: peer="{Voter server-2bbd6b80-00c9-8a45-10e8-db43464eb80a 2bbd6b80-00c9-8a45-10e8-db43464eb80a}" testLoggerAdapter verbose: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.991Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: aborting pipeline replication: peer="{Voter server-caae0ef8-5ef3-3038-2a73-a05d64d77146 caae0ef8-5ef3-3038-2a73-a05d64d77146}" testing.go:149: server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: 2021-07-10T01:05:02.991Z [INFO] server-2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab: aborting pipeline replication: peer="{Voter server-caae0ef8-5ef3-3038-2a73-a05d64d77146 caae0ef8-5ef3-3038-2a73-a05d64d77146}" testLoggerAdapter verbose: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:03.041Z [WARN] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: heartbeat timeout reached, starting election: last-leader=2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab testing.go:149: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:03.041Z [WARN] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: heartbeat timeout reached, starting election: last-leader=2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab testLoggerAdapter verbose: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:03.042Z [INFO] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: entering candidate state: node="Node at caae0ef8-5ef3-3038-2a73-a05d64d77146 [Candidate]" term=3 testing.go:149: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:03.042Z [INFO] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: entering candidate state: node="Node at caae0ef8-5ef3-3038-2a73-a05d64d77146 [Candidate]" term=3 testLoggerAdapter verbose: server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: 2021-07-10T01:05:03.042Z [WARN] server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: rejecting vote request since we have a leader: from=caae0ef8-5ef3-3038-2a73-a05d64d77146 leader=2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab testing.go:149: server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: 2021-07-10T01:05:03.042Z [WARN] server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: rejecting vote request since we have a leader: from=caae0ef8-5ef3-3038-2a73-a05d64d77146 leader=2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab testLoggerAdapter verbose: server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: 2021-07-10T01:05:03.105Z [WARN] server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: heartbeat timeout reached, starting election: last-leader=2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab testLoggerAdapter verbose: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:03.105Z [WARN] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: Election timeout reached, restarting election testing.go:149: server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: 2021-07-10T01:05:03.105Z [WARN] server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: heartbeat timeout reached, starting election: last-leader=2c7d69cf-10ca-36e4-6f35-c429e1e5b3ab testLoggerAdapter verbose: server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: 2021-07-10T01:05:03.105Z [INFO] server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: entering candidate state: node="Node at 2bbd6b80-00c9-8a45-10e8-db43464eb80a [Candidate]" term=3 testing.go:149: server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: 2021-07-10T01:05:03.105Z [INFO] server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: entering candidate state: node="Node at 2bbd6b80-00c9-8a45-10e8-db43464eb80a [Candidate]" term=3 testing.go:149: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:03.105Z [WARN] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: Election timeout reached, restarting election testLoggerAdapter verbose: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:03.105Z [INFO] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: entering candidate state: node="Node at caae0ef8-5ef3-3038-2a73-a05d64d77146 [Candidate]" term=4 testing.go:149: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:03.105Z [INFO] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: entering candidate state: node="Node at caae0ef8-5ef3-3038-2a73-a05d64d77146 [Candidate]" term=4 testLoggerAdapter verbose: server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: 2021-07-10T01:05:03.105Z [INFO] server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: entering follower state: follower="Node at 2bbd6b80-00c9-8a45-10e8-db43464eb80a [Follower]" leader= testing.go:149: server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: 2021-07-10T01:05:03.105Z [INFO] server-2bbd6b80-00c9-8a45-10e8-db43464eb80a: entering follower state: follower="Node at 2bbd6b80-00c9-8a45-10e8-db43464eb80a [Follower]" leader= testLoggerAdapter verbose: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:03.106Z [INFO] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: election won: tally=2 testing.go:149: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:03.106Z [INFO] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: election won: tally=2 testLoggerAdapter verbose: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:03.106Z [INFO] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: entering leader state: leader="Node at caae0ef8-5ef3-3038-2a73-a05d64d77146 [Leader]" testing.go:149: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:03.106Z [INFO] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: entering leader state: leader="Node at caae0ef8-5ef3-3038-2a73-a05d64d77146 [Leader]" testLoggerAdapter verbose: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:03.106Z [INFO] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: added peer, starting replication: peer=server-2bbd6b80-00c9-8a45-10e8-db43464eb80a testing.go:149: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:03.106Z [INFO] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: added peer, starting replication: peer=server-2bbd6b80-00c9-8a45-10e8-db43464eb80a testLoggerAdapter verbose: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:03.106Z [INFO] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: pipelining replication: peer="{Voter server-2bbd6b80-00c9-8a45-10e8-db43464eb80a 2bbd6b80-00c9-8a45-10e8-db43464eb80a}" testing.go:149: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:03.106Z [INFO] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: pipelining replication: peer="{Voter server-2bbd6b80-00c9-8a45-10e8-db43464eb80a 2bbd6b80-00c9-8a45-10e8-db43464eb80a}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:03.201Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:03.201Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-07-10T01:05:03.307Z [INFO] stable state for Leader reached at 2021-07-10 01:05:03.202199253 +0000 UTC m=+8.590357203 (1 nodes), 61.521µs from start of poll, 485.009454ms from cluster start. Timeout at 2021-07-10 01:05:03.30727416 +0000 UTC m=+8.695432150, 105.074947ms after stability testing.go:149: cluster: 2021-07-10T01:05:03.307Z [INFO] stable state for Leader reached at 2021-07-10 01:05:03.202199253 +0000 UTC m=+8.590357203 (1 nodes), 61.521µs from start of poll, 485.009454ms from cluster start. Timeout at 2021-07-10 01:05:03.30727416 +0000 UTC m=+8.695432150, 105.074947ms after stability testLoggerAdapter verbose: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:03.307Z [INFO] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: aborting pipeline replication: peer="{Voter server-2bbd6b80-00c9-8a45-10e8-db43464eb80a 2bbd6b80-00c9-8a45-10e8-db43464eb80a}" testing.go:149: server-caae0ef8-5ef3-3038-2a73-a05d64d77146: 2021-07-10T01:05:03.307Z [INFO] server-caae0ef8-5ef3-3038-2a73-a05d64d77146: aborting pipeline replication: peer="{Voter server-2bbd6b80-00c9-8a45-10e8-db43464eb80a 2bbd6b80-00c9-8a45-10e8-db43464eb80a}" --- PASS: TestRaft_RemoveLeader (0.59s) === RUN TestRaft_RemoveLeader_NoShutdown testLoggerAdapter verbose: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.309Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6 Address:8a0b128e-a408-73d9-bbe2-b4581e28fdb6} {Suffrage:Voter ID:server-ce7230b8-5299-7647-eb53-198e68cf4494 Address:ce7230b8-5299-7647-eb53-198e68cf4494} {Suffrage:Voter ID:server-af31b31c-daea-1470-20d1-b0245fe34199 Address:af31b31c-daea-1470-20d1-b0245fe34199}]" testing.go:149: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.309Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6 Address:8a0b128e-a408-73d9-bbe2-b4581e28fdb6} {Suffrage:Voter ID:server-ce7230b8-5299-7647-eb53-198e68cf4494 Address:ce7230b8-5299-7647-eb53-198e68cf4494} {Suffrage:Voter ID:server-af31b31c-daea-1470-20d1-b0245fe34199 Address:af31b31c-daea-1470-20d1-b0245fe34199}]" testLoggerAdapter verbose: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.309Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: entering follower state: follower="Node at 8a0b128e-a408-73d9-bbe2-b4581e28fdb6 [Follower]" leader= testing.go:149: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.309Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: entering follower state: follower="Node at 8a0b128e-a408-73d9-bbe2-b4581e28fdb6 [Follower]" leader= testLoggerAdapter verbose: server-ce7230b8-5299-7647-eb53-198e68cf4494: 2021-07-10T01:05:03.309Z [INFO] server-ce7230b8-5299-7647-eb53-198e68cf4494: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6 Address:8a0b128e-a408-73d9-bbe2-b4581e28fdb6} {Suffrage:Voter ID:server-ce7230b8-5299-7647-eb53-198e68cf4494 Address:ce7230b8-5299-7647-eb53-198e68cf4494} {Suffrage:Voter ID:server-af31b31c-daea-1470-20d1-b0245fe34199 Address:af31b31c-daea-1470-20d1-b0245fe34199}]" testing.go:149: server-ce7230b8-5299-7647-eb53-198e68cf4494: 2021-07-10T01:05:03.309Z [INFO] server-ce7230b8-5299-7647-eb53-198e68cf4494: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6 Address:8a0b128e-a408-73d9-bbe2-b4581e28fdb6} {Suffrage:Voter ID:server-ce7230b8-5299-7647-eb53-198e68cf4494 Address:ce7230b8-5299-7647-eb53-198e68cf4494} {Suffrage:Voter ID:server-af31b31c-daea-1470-20d1-b0245fe34199 Address:af31b31c-daea-1470-20d1-b0245fe34199}]" testLoggerAdapter verbose: server-ce7230b8-5299-7647-eb53-198e68cf4494: 2021-07-10T01:05:03.309Z [INFO] server-ce7230b8-5299-7647-eb53-198e68cf4494: entering follower state: follower="Node at ce7230b8-5299-7647-eb53-198e68cf4494 [Follower]" leader= testing.go:149: server-ce7230b8-5299-7647-eb53-198e68cf4494: 2021-07-10T01:05:03.309Z [INFO] server-ce7230b8-5299-7647-eb53-198e68cf4494: entering follower state: follower="Node at ce7230b8-5299-7647-eb53-198e68cf4494 [Follower]" leader= testLoggerAdapter verbose: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.309Z [INFO] server-af31b31c-daea-1470-20d1-b0245fe34199: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6 Address:8a0b128e-a408-73d9-bbe2-b4581e28fdb6} {Suffrage:Voter ID:server-ce7230b8-5299-7647-eb53-198e68cf4494 Address:ce7230b8-5299-7647-eb53-198e68cf4494} {Suffrage:Voter ID:server-af31b31c-daea-1470-20d1-b0245fe34199 Address:af31b31c-daea-1470-20d1-b0245fe34199}]" testing.go:149: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.309Z [INFO] server-af31b31c-daea-1470-20d1-b0245fe34199: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6 Address:8a0b128e-a408-73d9-bbe2-b4581e28fdb6} {Suffrage:Voter ID:server-ce7230b8-5299-7647-eb53-198e68cf4494 Address:ce7230b8-5299-7647-eb53-198e68cf4494} {Suffrage:Voter ID:server-af31b31c-daea-1470-20d1-b0245fe34199 Address:af31b31c-daea-1470-20d1-b0245fe34199}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:03.309Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-07-10T01:05:03.309Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.309Z [INFO] server-af31b31c-daea-1470-20d1-b0245fe34199: entering follower state: follower="Node at af31b31c-daea-1470-20d1-b0245fe34199 [Follower]" leader= testing.go:149: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.309Z [INFO] server-af31b31c-daea-1470-20d1-b0245fe34199: entering follower state: follower="Node at af31b31c-daea-1470-20d1-b0245fe34199 [Follower]" leader= testLoggerAdapter verbose: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.366Z [WARN] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.366Z [WARN] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.366Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: entering candidate state: node="Node at 8a0b128e-a408-73d9-bbe2-b4581e28fdb6 [Candidate]" term=2 testing.go:149: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.366Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: entering candidate state: node="Node at 8a0b128e-a408-73d9-bbe2-b4581e28fdb6 [Candidate]" term=2 testLoggerAdapter verbose: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.366Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: election won: tally=2 testing.go:149: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.366Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: election won: tally=2 testLoggerAdapter verbose: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.366Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: entering leader state: leader="Node at 8a0b128e-a408-73d9-bbe2-b4581e28fdb6 [Leader]" testing.go:149: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.366Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: entering leader state: leader="Node at 8a0b128e-a408-73d9-bbe2-b4581e28fdb6 [Leader]" testLoggerAdapter verbose: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.366Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: added peer, starting replication: peer=server-ce7230b8-5299-7647-eb53-198e68cf4494 testing.go:149: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.366Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: added peer, starting replication: peer=server-ce7230b8-5299-7647-eb53-198e68cf4494 testLoggerAdapter verbose: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.366Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: added peer, starting replication: peer=server-af31b31c-daea-1470-20d1-b0245fe34199 testing.go:149: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.366Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: added peer, starting replication: peer=server-af31b31c-daea-1470-20d1-b0245fe34199 testLoggerAdapter verbose: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.366Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: pipelining replication: peer="{Voter server-ce7230b8-5299-7647-eb53-198e68cf4494 ce7230b8-5299-7647-eb53-198e68cf4494}" testing.go:149: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.366Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: pipelining replication: peer="{Voter server-ce7230b8-5299-7647-eb53-198e68cf4494 ce7230b8-5299-7647-eb53-198e68cf4494}" testLoggerAdapter verbose: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.366Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: pipelining replication: peer="{Voter server-af31b31c-daea-1470-20d1-b0245fe34199 af31b31c-daea-1470-20d1-b0245fe34199}" testing.go:149: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.366Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: pipelining replication: peer="{Voter server-af31b31c-daea-1470-20d1-b0245fe34199 af31b31c-daea-1470-20d1-b0245fe34199}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:03.471Z [INFO] stable state for Follower reached at 2021-07-10 01:05:03.366643486 +0000 UTC m=+8.754801476 (2 nodes), 56.89888ms from start of poll, 57.83321ms from cluster start. Timeout at 2021-07-10 01:05:03.471708894 +0000 UTC m=+8.859866864, 105.065388ms after stability testing.go:149: cluster: 2021-07-10T01:05:03.471Z [INFO] stable state for Follower reached at 2021-07-10 01:05:03.366643486 +0000 UTC m=+8.754801476 (2 nodes), 56.89888ms from start of poll, 57.83321ms from cluster start. Timeout at 2021-07-10 01:05:03.471708894 +0000 UTC m=+8.859866864, 105.065388ms after stability testLoggerAdapter verbose: cluster: 2021-07-10T01:05:03.471Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:03.471Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-07-10T01:05:03.577Z [INFO] stable state for Leader reached at 2021-07-10 01:05:03.471979617 +0000 UTC m=+8.860137587 (1 nodes), 1.82µs from start of poll, 163.169321ms from cluster start. Timeout at 2021-07-10 01:05:03.577049764 +0000 UTC m=+8.965207754, 105.070167ms after stability testing.go:149: cluster: 2021-07-10T01:05:03.577Z [INFO] stable state for Leader reached at 2021-07-10 01:05:03.471979617 +0000 UTC m=+8.860137587 (1 nodes), 1.82µs from start of poll, 163.169321ms from cluster start. Timeout at 2021-07-10 01:05:03.577049764 +0000 UTC m=+8.965207754, 105.070167ms after stability testLoggerAdapter verbose: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.579Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: updating configuration: command=RemoveServer server-id=server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6 server-addr= servers="[{Suffrage:Voter ID:server-ce7230b8-5299-7647-eb53-198e68cf4494 Address:ce7230b8-5299-7647-eb53-198e68cf4494} {Suffrage:Voter ID:server-af31b31c-daea-1470-20d1-b0245fe34199 Address:af31b31c-daea-1470-20d1-b0245fe34199}]" testing.go:149: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.579Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: updating configuration: command=RemoveServer server-id=server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6 server-addr= servers="[{Suffrage:Voter ID:server-ce7230b8-5299-7647-eb53-198e68cf4494 Address:ce7230b8-5299-7647-eb53-198e68cf4494} {Suffrage:Voter ID:server-af31b31c-daea-1470-20d1-b0245fe34199 Address:af31b31c-daea-1470-20d1-b0245fe34199}]" testLoggerAdapter verbose: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.579Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: removed ourself, transitioning to follower testing.go:149: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.579Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: removed ourself, transitioning to follower testLoggerAdapter verbose: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.579Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: entering follower state: follower="Node at 8a0b128e-a408-73d9-bbe2-b4581e28fdb6 [Follower]" leader= testing.go:149: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.579Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: entering follower state: follower="Node at 8a0b128e-a408-73d9-bbe2-b4581e28fdb6 [Follower]" leader= testLoggerAdapter verbose: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.579Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: aborting pipeline replication: peer="{Voter server-af31b31c-daea-1470-20d1-b0245fe34199 af31b31c-daea-1470-20d1-b0245fe34199}" testing.go:149: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.579Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: aborting pipeline replication: peer="{Voter server-af31b31c-daea-1470-20d1-b0245fe34199 af31b31c-daea-1470-20d1-b0245fe34199}" testLoggerAdapter verbose: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.579Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: aborting pipeline replication: peer="{Voter server-ce7230b8-5299-7647-eb53-198e68cf4494 ce7230b8-5299-7647-eb53-198e68cf4494}" testing.go:149: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.579Z [INFO] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: aborting pipeline replication: peer="{Voter server-ce7230b8-5299-7647-eb53-198e68cf4494 ce7230b8-5299-7647-eb53-198e68cf4494}" testLoggerAdapter verbose: server-ce7230b8-5299-7647-eb53-198e68cf4494: 2021-07-10T01:05:03.651Z [WARN] server-ce7230b8-5299-7647-eb53-198e68cf4494: heartbeat timeout reached, starting election: last-leader=8a0b128e-a408-73d9-bbe2-b4581e28fdb6 testing.go:149: server-ce7230b8-5299-7647-eb53-198e68cf4494: 2021-07-10T01:05:03.651Z [WARN] server-ce7230b8-5299-7647-eb53-198e68cf4494: heartbeat timeout reached, starting election: last-leader=8a0b128e-a408-73d9-bbe2-b4581e28fdb6 testLoggerAdapter verbose: server-ce7230b8-5299-7647-eb53-198e68cf4494: 2021-07-10T01:05:03.651Z [INFO] server-ce7230b8-5299-7647-eb53-198e68cf4494: entering candidate state: node="Node at ce7230b8-5299-7647-eb53-198e68cf4494 [Candidate]" term=3 testing.go:149: server-ce7230b8-5299-7647-eb53-198e68cf4494: 2021-07-10T01:05:03.651Z [INFO] server-ce7230b8-5299-7647-eb53-198e68cf4494: entering candidate state: node="Node at ce7230b8-5299-7647-eb53-198e68cf4494 [Candidate]" term=3 testLoggerAdapter verbose: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.652Z [WARN] server-af31b31c-daea-1470-20d1-b0245fe34199: rejecting vote request since we have a leader: from=ce7230b8-5299-7647-eb53-198e68cf4494 leader=8a0b128e-a408-73d9-bbe2-b4581e28fdb6 testing.go:149: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.652Z [WARN] server-af31b31c-daea-1470-20d1-b0245fe34199: rejecting vote request since we have a leader: from=ce7230b8-5299-7647-eb53-198e68cf4494 leader=8a0b128e-a408-73d9-bbe2-b4581e28fdb6 testLoggerAdapter verbose: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.659Z [WARN] server-af31b31c-daea-1470-20d1-b0245fe34199: heartbeat timeout reached, starting election: last-leader=8a0b128e-a408-73d9-bbe2-b4581e28fdb6 testing.go:149: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.659Z [WARN] server-af31b31c-daea-1470-20d1-b0245fe34199: heartbeat timeout reached, starting election: last-leader=8a0b128e-a408-73d9-bbe2-b4581e28fdb6 testLoggerAdapter verbose: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.659Z [INFO] server-af31b31c-daea-1470-20d1-b0245fe34199: entering candidate state: node="Node at af31b31c-daea-1470-20d1-b0245fe34199 [Candidate]" term=3 testing.go:149: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.659Z [INFO] server-af31b31c-daea-1470-20d1-b0245fe34199: entering candidate state: node="Node at af31b31c-daea-1470-20d1-b0245fe34199 [Candidate]" term=3 testLoggerAdapter verbose: server-ce7230b8-5299-7647-eb53-198e68cf4494: 2021-07-10T01:05:03.659Z [INFO] server-ce7230b8-5299-7647-eb53-198e68cf4494: duplicate requestVote for same term: term=3 testing.go:149: server-ce7230b8-5299-7647-eb53-198e68cf4494: 2021-07-10T01:05:03.659Z [INFO] server-ce7230b8-5299-7647-eb53-198e68cf4494: duplicate requestVote for same term: term=3 testLoggerAdapter verbose: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.677Z [WARN] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: not part of stable configuration, aborting election testing.go:149: server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: 2021-07-10T01:05:03.677Z [WARN] server-8a0b128e-a408-73d9-bbe2-b4581e28fdb6: not part of stable configuration, aborting election testLoggerAdapter verbose: cluster: 2021-07-10T01:05:03.684Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:03.684Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.722Z [WARN] server-af31b31c-daea-1470-20d1-b0245fe34199: Election timeout reached, restarting election testing.go:149: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.722Z [WARN] server-af31b31c-daea-1470-20d1-b0245fe34199: Election timeout reached, restarting election testLoggerAdapter verbose: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.723Z [INFO] server-af31b31c-daea-1470-20d1-b0245fe34199: entering candidate state: node="Node at af31b31c-daea-1470-20d1-b0245fe34199 [Candidate]" term=4 testing.go:149: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.723Z [INFO] server-af31b31c-daea-1470-20d1-b0245fe34199: entering candidate state: node="Node at af31b31c-daea-1470-20d1-b0245fe34199 [Candidate]" term=4 testLoggerAdapter verbose: server-ce7230b8-5299-7647-eb53-198e68cf4494: 2021-07-10T01:05:03.723Z [INFO] server-ce7230b8-5299-7647-eb53-198e68cf4494: entering follower state: follower="Node at ce7230b8-5299-7647-eb53-198e68cf4494 [Follower]" leader= testing.go:149: server-ce7230b8-5299-7647-eb53-198e68cf4494: 2021-07-10T01:05:03.723Z [INFO] server-ce7230b8-5299-7647-eb53-198e68cf4494: entering follower state: follower="Node at ce7230b8-5299-7647-eb53-198e68cf4494 [Follower]" leader= testLoggerAdapter verbose: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.723Z [INFO] server-af31b31c-daea-1470-20d1-b0245fe34199: election won: tally=2 testing.go:149: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.723Z [INFO] server-af31b31c-daea-1470-20d1-b0245fe34199: election won: tally=2 testLoggerAdapter verbose: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.723Z [INFO] server-af31b31c-daea-1470-20d1-b0245fe34199: entering leader state: leader="Node at af31b31c-daea-1470-20d1-b0245fe34199 [Leader]" testing.go:149: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.723Z [INFO] server-af31b31c-daea-1470-20d1-b0245fe34199: entering leader state: leader="Node at af31b31c-daea-1470-20d1-b0245fe34199 [Leader]" testLoggerAdapter verbose: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.723Z [INFO] server-af31b31c-daea-1470-20d1-b0245fe34199: added peer, starting replication: peer=server-ce7230b8-5299-7647-eb53-198e68cf4494 testing.go:149: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.723Z [INFO] server-af31b31c-daea-1470-20d1-b0245fe34199: added peer, starting replication: peer=server-ce7230b8-5299-7647-eb53-198e68cf4494 testLoggerAdapter verbose: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.723Z [INFO] server-af31b31c-daea-1470-20d1-b0245fe34199: pipelining replication: peer="{Voter server-ce7230b8-5299-7647-eb53-198e68cf4494 ce7230b8-5299-7647-eb53-198e68cf4494}" testing.go:149: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.723Z [INFO] server-af31b31c-daea-1470-20d1-b0245fe34199: pipelining replication: peer="{Voter server-ce7230b8-5299-7647-eb53-198e68cf4494 ce7230b8-5299-7647-eb53-198e68cf4494}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:03.828Z [INFO] stable state for Leader reached at 2021-07-10 01:05:03.723543569 +0000 UTC m=+9.111701559 (1 nodes), 38.499726ms from start of poll, 414.733293ms from cluster start. Timeout at 2021-07-10 01:05:03.828617076 +0000 UTC m=+9.216775046, 105.073487ms after stability testing.go:149: cluster: 2021-07-10T01:05:03.828Z [INFO] stable state for Leader reached at 2021-07-10 01:05:03.723543569 +0000 UTC m=+9.111701559 (1 nodes), 38.499726ms from start of poll, 414.733293ms from cluster start. Timeout at 2021-07-10 01:05:03.828617076 +0000 UTC m=+9.216775046, 105.073487ms after stability testLoggerAdapter verbose: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.933Z [INFO] server-af31b31c-daea-1470-20d1-b0245fe34199: aborting pipeline replication: peer="{Voter server-ce7230b8-5299-7647-eb53-198e68cf4494 ce7230b8-5299-7647-eb53-198e68cf4494}" testing.go:149: server-af31b31c-daea-1470-20d1-b0245fe34199: 2021-07-10T01:05:03.933Z [INFO] server-af31b31c-daea-1470-20d1-b0245fe34199: aborting pipeline replication: peer="{Voter server-ce7230b8-5299-7647-eb53-198e68cf4494 ce7230b8-5299-7647-eb53-198e68cf4494}" --- PASS: TestRaft_RemoveLeader_NoShutdown (0.63s) === RUN TestRaft_RemoveFollower_SplitCluster testLoggerAdapter verbose: server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8: 2021-07-10T01:05:03.935Z [INFO] server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 Address:e2dbfe11-9d56-e8dc-9b73-209a54540fb8} {Suffrage:Voter ID:server-7c957af8-88bd-0379-90bd-080eb1a1f963 Address:7c957af8-88bd-0379-90bd-080eb1a1f963} {Suffrage:Voter ID:server-16924e6a-4df3-d71d-c931-45a38d6620a1 Address:16924e6a-4df3-d71d-c931-45a38d6620a1} {Suffrage:Voter ID:server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f Address:c30fad15-bc9e-dcf4-1045-c40bf71e2b1f}]" testing.go:149: server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8: 2021-07-10T01:05:03.935Z [INFO] server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 Address:e2dbfe11-9d56-e8dc-9b73-209a54540fb8} {Suffrage:Voter ID:server-7c957af8-88bd-0379-90bd-080eb1a1f963 Address:7c957af8-88bd-0379-90bd-080eb1a1f963} {Suffrage:Voter ID:server-16924e6a-4df3-d71d-c931-45a38d6620a1 Address:16924e6a-4df3-d71d-c931-45a38d6620a1} {Suffrage:Voter ID:server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f Address:c30fad15-bc9e-dcf4-1045-c40bf71e2b1f}]" testLoggerAdapter verbose: server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8: 2021-07-10T01:05:03.936Z [INFO] server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8: entering follower state: follower="Node at e2dbfe11-9d56-e8dc-9b73-209a54540fb8 [Follower]" leader= testing.go:149: server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8: 2021-07-10T01:05:03.936Z [INFO] server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8: entering follower state: follower="Node at e2dbfe11-9d56-e8dc-9b73-209a54540fb8 [Follower]" leader= testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:03.936Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 Address:e2dbfe11-9d56-e8dc-9b73-209a54540fb8} {Suffrage:Voter ID:server-7c957af8-88bd-0379-90bd-080eb1a1f963 Address:7c957af8-88bd-0379-90bd-080eb1a1f963} {Suffrage:Voter ID:server-16924e6a-4df3-d71d-c931-45a38d6620a1 Address:16924e6a-4df3-d71d-c931-45a38d6620a1} {Suffrage:Voter ID:server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f Address:c30fad15-bc9e-dcf4-1045-c40bf71e2b1f}]" testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:03.936Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 Address:e2dbfe11-9d56-e8dc-9b73-209a54540fb8} {Suffrage:Voter ID:server-7c957af8-88bd-0379-90bd-080eb1a1f963 Address:7c957af8-88bd-0379-90bd-080eb1a1f963} {Suffrage:Voter ID:server-16924e6a-4df3-d71d-c931-45a38d6620a1 Address:16924e6a-4df3-d71d-c931-45a38d6620a1} {Suffrage:Voter ID:server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f Address:c30fad15-bc9e-dcf4-1045-c40bf71e2b1f}]" testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:03.936Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: entering follower state: follower="Node at 7c957af8-88bd-0379-90bd-080eb1a1f963 [Follower]" leader= testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:03.936Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: entering follower state: follower="Node at 7c957af8-88bd-0379-90bd-080eb1a1f963 [Follower]" leader= testLoggerAdapter verbose: server-16924e6a-4df3-d71d-c931-45a38d6620a1: 2021-07-10T01:05:03.936Z [INFO] server-16924e6a-4df3-d71d-c931-45a38d6620a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 Address:e2dbfe11-9d56-e8dc-9b73-209a54540fb8} {Suffrage:Voter ID:server-7c957af8-88bd-0379-90bd-080eb1a1f963 Address:7c957af8-88bd-0379-90bd-080eb1a1f963} {Suffrage:Voter ID:server-16924e6a-4df3-d71d-c931-45a38d6620a1 Address:16924e6a-4df3-d71d-c931-45a38d6620a1} {Suffrage:Voter ID:server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f Address:c30fad15-bc9e-dcf4-1045-c40bf71e2b1f}]" testing.go:149: server-16924e6a-4df3-d71d-c931-45a38d6620a1: 2021-07-10T01:05:03.936Z [INFO] server-16924e6a-4df3-d71d-c931-45a38d6620a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 Address:e2dbfe11-9d56-e8dc-9b73-209a54540fb8} {Suffrage:Voter ID:server-7c957af8-88bd-0379-90bd-080eb1a1f963 Address:7c957af8-88bd-0379-90bd-080eb1a1f963} {Suffrage:Voter ID:server-16924e6a-4df3-d71d-c931-45a38d6620a1 Address:16924e6a-4df3-d71d-c931-45a38d6620a1} {Suffrage:Voter ID:server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f Address:c30fad15-bc9e-dcf4-1045-c40bf71e2b1f}]" testLoggerAdapter verbose: server-16924e6a-4df3-d71d-c931-45a38d6620a1: 2021-07-10T01:05:03.936Z [INFO] server-16924e6a-4df3-d71d-c931-45a38d6620a1: entering follower state: follower="Node at 16924e6a-4df3-d71d-c931-45a38d6620a1 [Follower]" leader= testing.go:149: server-16924e6a-4df3-d71d-c931-45a38d6620a1: 2021-07-10T01:05:03.936Z [INFO] server-16924e6a-4df3-d71d-c931-45a38d6620a1: entering follower state: follower="Node at 16924e6a-4df3-d71d-c931-45a38d6620a1 [Follower]" leader= testLoggerAdapter verbose: server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f: 2021-07-10T01:05:03.936Z [INFO] server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 Address:e2dbfe11-9d56-e8dc-9b73-209a54540fb8} {Suffrage:Voter ID:server-7c957af8-88bd-0379-90bd-080eb1a1f963 Address:7c957af8-88bd-0379-90bd-080eb1a1f963} {Suffrage:Voter ID:server-16924e6a-4df3-d71d-c931-45a38d6620a1 Address:16924e6a-4df3-d71d-c931-45a38d6620a1} {Suffrage:Voter ID:server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f Address:c30fad15-bc9e-dcf4-1045-c40bf71e2b1f}]" testing.go:149: server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f: 2021-07-10T01:05:03.936Z [INFO] server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 Address:e2dbfe11-9d56-e8dc-9b73-209a54540fb8} {Suffrage:Voter ID:server-7c957af8-88bd-0379-90bd-080eb1a1f963 Address:7c957af8-88bd-0379-90bd-080eb1a1f963} {Suffrage:Voter ID:server-16924e6a-4df3-d71d-c931-45a38d6620a1 Address:16924e6a-4df3-d71d-c931-45a38d6620a1} {Suffrage:Voter ID:server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f Address:c30fad15-bc9e-dcf4-1045-c40bf71e2b1f}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:03.936Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:03.936Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f: 2021-07-10T01:05:03.936Z [INFO] server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f: entering follower state: follower="Node at c30fad15-bc9e-dcf4-1045-c40bf71e2b1f [Follower]" leader= testing.go:149: server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f: 2021-07-10T01:05:03.936Z [INFO] server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f: entering follower state: follower="Node at c30fad15-bc9e-dcf4-1045-c40bf71e2b1f [Follower]" leader= testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.001Z [WARN] server-7c957af8-88bd-0379-90bd-080eb1a1f963: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.001Z [WARN] server-7c957af8-88bd-0379-90bd-080eb1a1f963: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.001Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: entering candidate state: node="Node at 7c957af8-88bd-0379-90bd-080eb1a1f963 [Candidate]" term=2 testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.001Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: entering candidate state: node="Node at 7c957af8-88bd-0379-90bd-080eb1a1f963 [Candidate]" term=2 testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.001Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: election won: tally=3 testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.001Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: election won: tally=3 testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.002Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: entering leader state: leader="Node at 7c957af8-88bd-0379-90bd-080eb1a1f963 [Leader]" testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.002Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: entering leader state: leader="Node at 7c957af8-88bd-0379-90bd-080eb1a1f963 [Leader]" testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.002Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: added peer, starting replication: peer=server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.002Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: added peer, starting replication: peer=server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.002Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: added peer, starting replication: peer=server-16924e6a-4df3-d71d-c931-45a38d6620a1 testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.002Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: added peer, starting replication: peer=server-16924e6a-4df3-d71d-c931-45a38d6620a1 testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.002Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: added peer, starting replication: peer=server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.002Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: added peer, starting replication: peer=server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.002Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: pipelining replication: peer="{Voter server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 e2dbfe11-9d56-e8dc-9b73-209a54540fb8}" testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.002Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: pipelining replication: peer="{Voter server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 e2dbfe11-9d56-e8dc-9b73-209a54540fb8}" testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.002Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: pipelining replication: peer="{Voter server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f c30fad15-bc9e-dcf4-1045-c40bf71e2b1f}" testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.002Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: pipelining replication: peer="{Voter server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f c30fad15-bc9e-dcf4-1045-c40bf71e2b1f}" testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.002Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: pipelining replication: peer="{Voter server-16924e6a-4df3-d71d-c931-45a38d6620a1 16924e6a-4df3-d71d-c931-45a38d6620a1}" testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.002Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: pipelining replication: peer="{Voter server-16924e6a-4df3-d71d-c931-45a38d6620a1 16924e6a-4df3-d71d-c931-45a38d6620a1}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:04.107Z [INFO] stable state for Leader reached at 2021-07-10 01:05:04.002091485 +0000 UTC m=+9.390249475 (1 nodes), 65.159827ms from start of poll, 66.494101ms from cluster start. Timeout at 2021-07-10 01:05:04.107159293 +0000 UTC m=+9.495317263, 105.067788ms after stability testing.go:149: cluster: 2021-07-10T01:05:04.107Z [INFO] stable state for Leader reached at 2021-07-10 01:05:04.002091485 +0000 UTC m=+9.390249475 (1 nodes), 65.159827ms from start of poll, 66.494101ms from cluster start. Timeout at 2021-07-10 01:05:04.107159293 +0000 UTC m=+9.495317263, 105.067788ms after stability testLoggerAdapter verbose: cluster: 2021-07-10T01:05:04.212Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-07-10T01:05:04.212Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-07-10T01:05:04.317Z [INFO] stable state for Follower reached at 2021-07-10 01:05:04.212622505 +0000 UTC m=+9.600780475 (3 nodes), 2.58µs from start of poll, 277.025101ms from cluster start. Timeout at 2021-07-10 01:05:04.317686472 +0000 UTC m=+9.705844442, 105.063967ms after stability testing.go:149: cluster: 2021-07-10T01:05:04.317Z [INFO] stable state for Follower reached at 2021-07-10 01:05:04.212622505 +0000 UTC m=+9.600780475 (3 nodes), 2.58µs from start of poll, 277.025101ms from cluster start. Timeout at 2021-07-10 01:05:04.317686472 +0000 UTC m=+9.705844442, 105.063967ms after stability testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.317Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: updating configuration: command=RemoveServer server-id=server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f server-addr= servers="[{Suffrage:Voter ID:server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 Address:e2dbfe11-9d56-e8dc-9b73-209a54540fb8} {Suffrage:Voter ID:server-7c957af8-88bd-0379-90bd-080eb1a1f963 Address:7c957af8-88bd-0379-90bd-080eb1a1f963} {Suffrage:Voter ID:server-16924e6a-4df3-d71d-c931-45a38d6620a1 Address:16924e6a-4df3-d71d-c931-45a38d6620a1}]" testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.317Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: updating configuration: command=RemoveServer server-id=server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f server-addr= servers="[{Suffrage:Voter ID:server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 Address:e2dbfe11-9d56-e8dc-9b73-209a54540fb8} {Suffrage:Voter ID:server-7c957af8-88bd-0379-90bd-080eb1a1f963 Address:7c957af8-88bd-0379-90bd-080eb1a1f963} {Suffrage:Voter ID:server-16924e6a-4df3-d71d-c931-45a38d6620a1 Address:16924e6a-4df3-d71d-c931-45a38d6620a1}]" testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.318Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: removed peer, stopping replication: peer=server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f last-index=3 testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.318Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: removed peer, stopping replication: peer=server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f last-index=3 testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.318Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to pipeline appendEntries: peer="{Voter server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 e2dbfe11-9d56-e8dc-9b73-209a54540fb8}" error="append pipeline closed" testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.318Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to pipeline appendEntries: peer="{Voter server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 e2dbfe11-9d56-e8dc-9b73-209a54540fb8}" error="append pipeline closed" testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.318Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: aborting pipeline replication: peer="{Voter server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 e2dbfe11-9d56-e8dc-9b73-209a54540fb8}" testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.318Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: aborting pipeline replication: peer="{Voter server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 e2dbfe11-9d56-e8dc-9b73-209a54540fb8}" testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.318Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to pipeline appendEntries: peer="{Voter server-16924e6a-4df3-d71d-c931-45a38d6620a1 16924e6a-4df3-d71d-c931-45a38d6620a1}" error="append pipeline closed" testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.318Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to pipeline appendEntries: peer="{Voter server-16924e6a-4df3-d71d-c931-45a38d6620a1 16924e6a-4df3-d71d-c931-45a38d6620a1}" error="append pipeline closed" testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.318Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: aborting pipeline replication: peer="{Voter server-16924e6a-4df3-d71d-c931-45a38d6620a1 16924e6a-4df3-d71d-c931-45a38d6620a1}" testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.318Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: aborting pipeline replication: peer="{Voter server-16924e6a-4df3-d71d-c931-45a38d6620a1 16924e6a-4df3-d71d-c931-45a38d6620a1}" testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.318Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: aborting pipeline replication: peer="{Voter server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f c30fad15-bc9e-dcf4-1045-c40bf71e2b1f}" testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.318Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: aborting pipeline replication: peer="{Voter server-c30fad15-bc9e-dcf4-1045-c40bf71e2b1f c30fad15-bc9e-dcf4-1045-c40bf71e2b1f}" testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.321Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to heartbeat to: peer=e2dbfe11-9d56-e8dc-9b73-209a54540fb8 error="failed to connect to peer: e2dbfe11-9d56-e8dc-9b73-209a54540fb8" testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.321Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to heartbeat to: peer=e2dbfe11-9d56-e8dc-9b73-209a54540fb8 error="failed to connect to peer: e2dbfe11-9d56-e8dc-9b73-209a54540fb8" testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.327Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to appendEntries to: peer="{Voter server-16924e6a-4df3-d71d-c931-45a38d6620a1 16924e6a-4df3-d71d-c931-45a38d6620a1}" error="failed to connect to peer: 16924e6a-4df3-d71d-c931-45a38d6620a1" testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.327Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to appendEntries to: peer="{Voter server-16924e6a-4df3-d71d-c931-45a38d6620a1 16924e6a-4df3-d71d-c931-45a38d6620a1}" error="failed to connect to peer: 16924e6a-4df3-d71d-c931-45a38d6620a1" testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.327Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to appendEntries to: peer="{Voter server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 e2dbfe11-9d56-e8dc-9b73-209a54540fb8}" error="failed to connect to peer: e2dbfe11-9d56-e8dc-9b73-209a54540fb8" testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.327Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to appendEntries to: peer="{Voter server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 e2dbfe11-9d56-e8dc-9b73-209a54540fb8}" error="failed to connect to peer: e2dbfe11-9d56-e8dc-9b73-209a54540fb8" testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.327Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to heartbeat to: peer=16924e6a-4df3-d71d-c931-45a38d6620a1 error="failed to connect to peer: 16924e6a-4df3-d71d-c931-45a38d6620a1" testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.327Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to heartbeat to: peer=16924e6a-4df3-d71d-c931-45a38d6620a1 error="failed to connect to peer: 16924e6a-4df3-d71d-c931-45a38d6620a1" testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.338Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to heartbeat to: peer=e2dbfe11-9d56-e8dc-9b73-209a54540fb8 error="failed to connect to peer: e2dbfe11-9d56-e8dc-9b73-209a54540fb8" testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.338Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to heartbeat to: peer=e2dbfe11-9d56-e8dc-9b73-209a54540fb8 error="failed to connect to peer: e2dbfe11-9d56-e8dc-9b73-209a54540fb8" testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.345Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to appendEntries to: peer="{Voter server-16924e6a-4df3-d71d-c931-45a38d6620a1 16924e6a-4df3-d71d-c931-45a38d6620a1}" error="failed to connect to peer: 16924e6a-4df3-d71d-c931-45a38d6620a1" testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.345Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to appendEntries to: peer="{Voter server-16924e6a-4df3-d71d-c931-45a38d6620a1 16924e6a-4df3-d71d-c931-45a38d6620a1}" error="failed to connect to peer: 16924e6a-4df3-d71d-c931-45a38d6620a1" testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.346Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to appendEntries to: peer="{Voter server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 e2dbfe11-9d56-e8dc-9b73-209a54540fb8}" error="failed to connect to peer: e2dbfe11-9d56-e8dc-9b73-209a54540fb8" testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.346Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to appendEntries to: peer="{Voter server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 e2dbfe11-9d56-e8dc-9b73-209a54540fb8}" error="failed to connect to peer: e2dbfe11-9d56-e8dc-9b73-209a54540fb8" testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.347Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to heartbeat to: peer=16924e6a-4df3-d71d-c931-45a38d6620a1 error="failed to connect to peer: 16924e6a-4df3-d71d-c931-45a38d6620a1" testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.347Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to heartbeat to: peer=16924e6a-4df3-d71d-c931-45a38d6620a1 error="failed to connect to peer: 16924e6a-4df3-d71d-c931-45a38d6620a1" testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.355Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to heartbeat to: peer=e2dbfe11-9d56-e8dc-9b73-209a54540fb8 error="failed to connect to peer: e2dbfe11-9d56-e8dc-9b73-209a54540fb8" testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.355Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to heartbeat to: peer=e2dbfe11-9d56-e8dc-9b73-209a54540fb8 error="failed to connect to peer: e2dbfe11-9d56-e8dc-9b73-209a54540fb8" testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.361Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to appendEntries to: peer="{Voter server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 e2dbfe11-9d56-e8dc-9b73-209a54540fb8}" error="failed to connect to peer: e2dbfe11-9d56-e8dc-9b73-209a54540fb8" testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.361Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to appendEntries to: peer="{Voter server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 e2dbfe11-9d56-e8dc-9b73-209a54540fb8}" error="failed to connect to peer: e2dbfe11-9d56-e8dc-9b73-209a54540fb8" testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.364Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to appendEntries to: peer="{Voter server-16924e6a-4df3-d71d-c931-45a38d6620a1 16924e6a-4df3-d71d-c931-45a38d6620a1}" error="failed to connect to peer: 16924e6a-4df3-d71d-c931-45a38d6620a1" testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.364Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to appendEntries to: peer="{Voter server-16924e6a-4df3-d71d-c931-45a38d6620a1 16924e6a-4df3-d71d-c931-45a38d6620a1}" error="failed to connect to peer: 16924e6a-4df3-d71d-c931-45a38d6620a1" testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.365Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to heartbeat to: peer=16924e6a-4df3-d71d-c931-45a38d6620a1 error="failed to connect to peer: 16924e6a-4df3-d71d-c931-45a38d6620a1" testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.365Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to heartbeat to: peer=16924e6a-4df3-d71d-c931-45a38d6620a1 error="failed to connect to peer: 16924e6a-4df3-d71d-c931-45a38d6620a1" testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.366Z [WARN] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to contact: server-id=server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 time=50.073528ms testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.366Z [WARN] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to contact: server-id=server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 time=50.073528ms testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.376Z [WARN] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to contact: server-id=server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 time=60.268396ms testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.376Z [WARN] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to contact: server-id=server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 time=60.268396ms testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.376Z [WARN] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to contact: server-id=server-16924e6a-4df3-d71d-c931-45a38d6620a1 time=59.294765ms testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.376Z [WARN] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to contact: server-id=server-16924e6a-4df3-d71d-c931-45a38d6620a1 time=59.294765ms testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.376Z [WARN] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to contact quorum of nodes, stepping down testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.376Z [WARN] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to contact quorum of nodes, stepping down testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.376Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: entering follower state: follower="Node at 7c957af8-88bd-0379-90bd-080eb1a1f963 [Follower]" leader= testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.376Z [INFO] server-7c957af8-88bd-0379-90bd-080eb1a1f963: entering follower state: follower="Node at 7c957af8-88bd-0379-90bd-080eb1a1f963 [Follower]" leader= testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.376Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to appendEntries to: peer="{Voter server-16924e6a-4df3-d71d-c931-45a38d6620a1 16924e6a-4df3-d71d-c931-45a38d6620a1}" error="failed to connect to peer: 16924e6a-4df3-d71d-c931-45a38d6620a1" testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.376Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to appendEntries to: peer="{Voter server-16924e6a-4df3-d71d-c931-45a38d6620a1 16924e6a-4df3-d71d-c931-45a38d6620a1}" error="failed to connect to peer: 16924e6a-4df3-d71d-c931-45a38d6620a1" testLoggerAdapter verbose: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.376Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to appendEntries to: peer="{Voter server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 e2dbfe11-9d56-e8dc-9b73-209a54540fb8}" error="failed to connect to peer: e2dbfe11-9d56-e8dc-9b73-209a54540fb8" testing.go:149: server-7c957af8-88bd-0379-90bd-080eb1a1f963: 2021-07-10T01:05:04.376Z [ERROR] server-7c957af8-88bd-0379-90bd-080eb1a1f963: failed to appendEntries to: peer="{Voter server-e2dbfe11-9d56-e8dc-9b73-209a54540fb8 e2dbfe11-9d56-e8dc-9b73-209a54540fb8}" error="failed to connect to peer: e2dbfe11-9d56-e8dc-9b73-209a54540fb8" --- PASS: TestRaft_RemoveFollower_SplitCluster (0.44s) === RUN TestRaft_AddKnownPeer testLoggerAdapter verbose: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.378Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5 Address:46f9ebe0-96b0-0420-9cd5-c5b4d85585d5} {Suffrage:Voter ID:server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412 Address:1cd998d1-cbc4-6fe4-cd42-c8958b23e412} {Suffrage:Voter ID:server-8f4b8e8e-6a05-8718-184c-cd2e47306775 Address:8f4b8e8e-6a05-8718-184c-cd2e47306775}]" testing.go:149: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.378Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5 Address:46f9ebe0-96b0-0420-9cd5-c5b4d85585d5} {Suffrage:Voter ID:server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412 Address:1cd998d1-cbc4-6fe4-cd42-c8958b23e412} {Suffrage:Voter ID:server-8f4b8e8e-6a05-8718-184c-cd2e47306775 Address:8f4b8e8e-6a05-8718-184c-cd2e47306775}]" testLoggerAdapter verbose: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.378Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: entering follower state: follower="Node at 46f9ebe0-96b0-0420-9cd5-c5b4d85585d5 [Follower]" leader= testing.go:149: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.378Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: entering follower state: follower="Node at 46f9ebe0-96b0-0420-9cd5-c5b4d85585d5 [Follower]" leader= testLoggerAdapter verbose: server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412: 2021-07-10T01:05:04.378Z [INFO] server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5 Address:46f9ebe0-96b0-0420-9cd5-c5b4d85585d5} {Suffrage:Voter ID:server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412 Address:1cd998d1-cbc4-6fe4-cd42-c8958b23e412} {Suffrage:Voter ID:server-8f4b8e8e-6a05-8718-184c-cd2e47306775 Address:8f4b8e8e-6a05-8718-184c-cd2e47306775}]" testing.go:149: server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412: 2021-07-10T01:05:04.378Z [INFO] server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5 Address:46f9ebe0-96b0-0420-9cd5-c5b4d85585d5} {Suffrage:Voter ID:server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412 Address:1cd998d1-cbc4-6fe4-cd42-c8958b23e412} {Suffrage:Voter ID:server-8f4b8e8e-6a05-8718-184c-cd2e47306775 Address:8f4b8e8e-6a05-8718-184c-cd2e47306775}]" testLoggerAdapter verbose: server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412: 2021-07-10T01:05:04.378Z [INFO] server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412: entering follower state: follower="Node at 1cd998d1-cbc4-6fe4-cd42-c8958b23e412 [Follower]" leader= testing.go:149: server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412: 2021-07-10T01:05:04.378Z [INFO] server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412: entering follower state: follower="Node at 1cd998d1-cbc4-6fe4-cd42-c8958b23e412 [Follower]" leader= testLoggerAdapter verbose: server-8f4b8e8e-6a05-8718-184c-cd2e47306775: 2021-07-10T01:05:04.378Z [INFO] server-8f4b8e8e-6a05-8718-184c-cd2e47306775: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5 Address:46f9ebe0-96b0-0420-9cd5-c5b4d85585d5} {Suffrage:Voter ID:server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412 Address:1cd998d1-cbc4-6fe4-cd42-c8958b23e412} {Suffrage:Voter ID:server-8f4b8e8e-6a05-8718-184c-cd2e47306775 Address:8f4b8e8e-6a05-8718-184c-cd2e47306775}]" testing.go:149: server-8f4b8e8e-6a05-8718-184c-cd2e47306775: 2021-07-10T01:05:04.378Z [INFO] server-8f4b8e8e-6a05-8718-184c-cd2e47306775: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5 Address:46f9ebe0-96b0-0420-9cd5-c5b4d85585d5} {Suffrage:Voter ID:server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412 Address:1cd998d1-cbc4-6fe4-cd42-c8958b23e412} {Suffrage:Voter ID:server-8f4b8e8e-6a05-8718-184c-cd2e47306775 Address:8f4b8e8e-6a05-8718-184c-cd2e47306775}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:04.378Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:04.378Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-8f4b8e8e-6a05-8718-184c-cd2e47306775: 2021-07-10T01:05:04.379Z [INFO] server-8f4b8e8e-6a05-8718-184c-cd2e47306775: entering follower state: follower="Node at 8f4b8e8e-6a05-8718-184c-cd2e47306775 [Follower]" leader= testing.go:149: server-8f4b8e8e-6a05-8718-184c-cd2e47306775: 2021-07-10T01:05:04.379Z [INFO] server-8f4b8e8e-6a05-8718-184c-cd2e47306775: entering follower state: follower="Node at 8f4b8e8e-6a05-8718-184c-cd2e47306775 [Follower]" leader= testLoggerAdapter verbose: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.439Z [WARN] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.439Z [WARN] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.440Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: entering candidate state: node="Node at 46f9ebe0-96b0-0420-9cd5-c5b4d85585d5 [Candidate]" term=2 testing.go:149: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.440Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: entering candidate state: node="Node at 46f9ebe0-96b0-0420-9cd5-c5b4d85585d5 [Candidate]" term=2 testLoggerAdapter verbose: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.440Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: election won: tally=2 testing.go:149: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.440Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: election won: tally=2 testLoggerAdapter verbose: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.440Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: entering leader state: leader="Node at 46f9ebe0-96b0-0420-9cd5-c5b4d85585d5 [Leader]" testing.go:149: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.440Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: entering leader state: leader="Node at 46f9ebe0-96b0-0420-9cd5-c5b4d85585d5 [Leader]" testLoggerAdapter verbose: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.440Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: added peer, starting replication: peer=server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412 testing.go:149: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.440Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: added peer, starting replication: peer=server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412 testLoggerAdapter verbose: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.440Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: added peer, starting replication: peer=server-8f4b8e8e-6a05-8718-184c-cd2e47306775 testing.go:149: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.440Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: added peer, starting replication: peer=server-8f4b8e8e-6a05-8718-184c-cd2e47306775 testLoggerAdapter verbose: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.440Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: pipelining replication: peer="{Voter server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412 1cd998d1-cbc4-6fe4-cd42-c8958b23e412}" testing.go:149: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.440Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: pipelining replication: peer="{Voter server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412 1cd998d1-cbc4-6fe4-cd42-c8958b23e412}" testLoggerAdapter verbose: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.440Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: pipelining replication: peer="{Voter server-8f4b8e8e-6a05-8718-184c-cd2e47306775 8f4b8e8e-6a05-8718-184c-cd2e47306775}" testing.go:149: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.440Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: pipelining replication: peer="{Voter server-8f4b8e8e-6a05-8718-184c-cd2e47306775 8f4b8e8e-6a05-8718-184c-cd2e47306775}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:04.545Z [INFO] stable state for Leader reached at 2021-07-10 01:05:04.440455387 +0000 UTC m=+9.828613377 (1 nodes), 61.399648ms from start of poll, 62.375178ms from cluster start. Timeout at 2021-07-10 01:05:04.545477674 +0000 UTC m=+9.933635644, 105.022267ms after stability testing.go:149: cluster: 2021-07-10T01:05:04.545Z [INFO] stable state for Leader reached at 2021-07-10 01:05:04.440455387 +0000 UTC m=+9.828613377 (1 nodes), 61.399648ms from start of poll, 62.375178ms from cluster start. Timeout at 2021-07-10 01:05:04.545477674 +0000 UTC m=+9.933635644, 105.022267ms after stability testLoggerAdapter verbose: cluster: 2021-07-10T01:05:04.545Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-07-10T01:05:04.545Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-07-10T01:05:04.650Z [INFO] stable state for Follower reached at 2021-07-10 01:05:04.545705256 +0000 UTC m=+9.933863226 (2 nodes), 2.54µs from start of poll, 167.625027ms from cluster start. Timeout at 2021-07-10 01:05:04.650770704 +0000 UTC m=+10.038928674, 105.065448ms after stability testing.go:149: cluster: 2021-07-10T01:05:04.650Z [INFO] stable state for Follower reached at 2021-07-10 01:05:04.545705256 +0000 UTC m=+9.933863226 (2 nodes), 2.54µs from start of poll, 167.625027ms from cluster start. Timeout at 2021-07-10 01:05:04.650770704 +0000 UTC m=+10.038928674, 105.065448ms after stability testLoggerAdapter verbose: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.651Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: updating configuration: command=AddStaging server-id=server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412 server-addr=1cd998d1-cbc4-6fe4-cd42-c8958b23e412 servers="[{Suffrage:Voter ID:server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5 Address:46f9ebe0-96b0-0420-9cd5-c5b4d85585d5} {Suffrage:Voter ID:server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412 Address:1cd998d1-cbc4-6fe4-cd42-c8958b23e412} {Suffrage:Voter ID:server-8f4b8e8e-6a05-8718-184c-cd2e47306775 Address:8f4b8e8e-6a05-8718-184c-cd2e47306775}]" testing.go:149: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.651Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: updating configuration: command=AddStaging server-id=server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412 server-addr=1cd998d1-cbc4-6fe4-cd42-c8958b23e412 servers="[{Suffrage:Voter ID:server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5 Address:46f9ebe0-96b0-0420-9cd5-c5b4d85585d5} {Suffrage:Voter ID:server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412 Address:1cd998d1-cbc4-6fe4-cd42-c8958b23e412} {Suffrage:Voter ID:server-8f4b8e8e-6a05-8718-184c-cd2e47306775 Address:8f4b8e8e-6a05-8718-184c-cd2e47306775}]" testLoggerAdapter verbose: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.651Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: aborting pipeline replication: peer="{Voter server-8f4b8e8e-6a05-8718-184c-cd2e47306775 8f4b8e8e-6a05-8718-184c-cd2e47306775}" testing.go:149: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.651Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: aborting pipeline replication: peer="{Voter server-8f4b8e8e-6a05-8718-184c-cd2e47306775 8f4b8e8e-6a05-8718-184c-cd2e47306775}" testLoggerAdapter verbose: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.651Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: aborting pipeline replication: peer="{Voter server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412 1cd998d1-cbc4-6fe4-cd42-c8958b23e412}" testing.go:149: server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: 2021-07-10T01:05:04.651Z [INFO] server-46f9ebe0-96b0-0420-9cd5-c5b4d85585d5: aborting pipeline replication: peer="{Voter server-1cd998d1-cbc4-6fe4-cd42-c8958b23e412 1cd998d1-cbc4-6fe4-cd42-c8958b23e412}" --- PASS: TestRaft_AddKnownPeer (0.27s) === RUN TestRaft_RemoveUnknownPeer testLoggerAdapter verbose: server-46668501-dcf3-613d-4f54-fde5c2a55bcf: 2021-07-10T01:05:04.653Z [INFO] server-46668501-dcf3-613d-4f54-fde5c2a55bcf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-46668501-dcf3-613d-4f54-fde5c2a55bcf Address:46668501-dcf3-613d-4f54-fde5c2a55bcf} {Suffrage:Voter ID:server-3497e30e-884c-48a8-5653-12d1a38507a4 Address:3497e30e-884c-48a8-5653-12d1a38507a4} {Suffrage:Voter ID:server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4 Address:9aa15ca3-275d-8bac-133b-09b2ee61c7d4}]" testing.go:149: server-46668501-dcf3-613d-4f54-fde5c2a55bcf: 2021-07-10T01:05:04.653Z [INFO] server-46668501-dcf3-613d-4f54-fde5c2a55bcf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-46668501-dcf3-613d-4f54-fde5c2a55bcf Address:46668501-dcf3-613d-4f54-fde5c2a55bcf} {Suffrage:Voter ID:server-3497e30e-884c-48a8-5653-12d1a38507a4 Address:3497e30e-884c-48a8-5653-12d1a38507a4} {Suffrage:Voter ID:server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4 Address:9aa15ca3-275d-8bac-133b-09b2ee61c7d4}]" testLoggerAdapter verbose: server-46668501-dcf3-613d-4f54-fde5c2a55bcf: 2021-07-10T01:05:04.653Z [INFO] server-46668501-dcf3-613d-4f54-fde5c2a55bcf: entering follower state: follower="Node at 46668501-dcf3-613d-4f54-fde5c2a55bcf [Follower]" leader= testing.go:149: server-46668501-dcf3-613d-4f54-fde5c2a55bcf: 2021-07-10T01:05:04.653Z [INFO] server-46668501-dcf3-613d-4f54-fde5c2a55bcf: entering follower state: follower="Node at 46668501-dcf3-613d-4f54-fde5c2a55bcf [Follower]" leader= testLoggerAdapter verbose: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.653Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-46668501-dcf3-613d-4f54-fde5c2a55bcf Address:46668501-dcf3-613d-4f54-fde5c2a55bcf} {Suffrage:Voter ID:server-3497e30e-884c-48a8-5653-12d1a38507a4 Address:3497e30e-884c-48a8-5653-12d1a38507a4} {Suffrage:Voter ID:server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4 Address:9aa15ca3-275d-8bac-133b-09b2ee61c7d4}]" testing.go:149: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.653Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-46668501-dcf3-613d-4f54-fde5c2a55bcf Address:46668501-dcf3-613d-4f54-fde5c2a55bcf} {Suffrage:Voter ID:server-3497e30e-884c-48a8-5653-12d1a38507a4 Address:3497e30e-884c-48a8-5653-12d1a38507a4} {Suffrage:Voter ID:server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4 Address:9aa15ca3-275d-8bac-133b-09b2ee61c7d4}]" testLoggerAdapter verbose: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.653Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: entering follower state: follower="Node at 3497e30e-884c-48a8-5653-12d1a38507a4 [Follower]" leader= testing.go:149: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.653Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: entering follower state: follower="Node at 3497e30e-884c-48a8-5653-12d1a38507a4 [Follower]" leader= testLoggerAdapter verbose: server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4: 2021-07-10T01:05:04.653Z [INFO] server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-46668501-dcf3-613d-4f54-fde5c2a55bcf Address:46668501-dcf3-613d-4f54-fde5c2a55bcf} {Suffrage:Voter ID:server-3497e30e-884c-48a8-5653-12d1a38507a4 Address:3497e30e-884c-48a8-5653-12d1a38507a4} {Suffrage:Voter ID:server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4 Address:9aa15ca3-275d-8bac-133b-09b2ee61c7d4}]" testing.go:149: server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4: 2021-07-10T01:05:04.653Z [INFO] server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-46668501-dcf3-613d-4f54-fde5c2a55bcf Address:46668501-dcf3-613d-4f54-fde5c2a55bcf} {Suffrage:Voter ID:server-3497e30e-884c-48a8-5653-12d1a38507a4 Address:3497e30e-884c-48a8-5653-12d1a38507a4} {Suffrage:Voter ID:server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4 Address:9aa15ca3-275d-8bac-133b-09b2ee61c7d4}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:04.653Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:04.653Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4: 2021-07-10T01:05:04.653Z [INFO] server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4: entering follower state: follower="Node at 9aa15ca3-275d-8bac-133b-09b2ee61c7d4 [Follower]" leader= testing.go:149: server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4: 2021-07-10T01:05:04.653Z [INFO] server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4: entering follower state: follower="Node at 9aa15ca3-275d-8bac-133b-09b2ee61c7d4 [Follower]" leader= testLoggerAdapter verbose: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.704Z [WARN] server-3497e30e-884c-48a8-5653-12d1a38507a4: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.704Z [WARN] server-3497e30e-884c-48a8-5653-12d1a38507a4: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.704Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: entering candidate state: node="Node at 3497e30e-884c-48a8-5653-12d1a38507a4 [Candidate]" term=2 testing.go:149: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.704Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: entering candidate state: node="Node at 3497e30e-884c-48a8-5653-12d1a38507a4 [Candidate]" term=2 testLoggerAdapter verbose: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.705Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: election won: tally=2 testing.go:149: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.705Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: election won: tally=2 testLoggerAdapter verbose: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.705Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: entering leader state: leader="Node at 3497e30e-884c-48a8-5653-12d1a38507a4 [Leader]" testing.go:149: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.705Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: entering leader state: leader="Node at 3497e30e-884c-48a8-5653-12d1a38507a4 [Leader]" testLoggerAdapter verbose: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.705Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: added peer, starting replication: peer=server-46668501-dcf3-613d-4f54-fde5c2a55bcf testing.go:149: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.705Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: added peer, starting replication: peer=server-46668501-dcf3-613d-4f54-fde5c2a55bcf testLoggerAdapter verbose: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.705Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: added peer, starting replication: peer=server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4 testing.go:149: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.705Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: added peer, starting replication: peer=server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4 testLoggerAdapter verbose: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.705Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: pipelining replication: peer="{Voter server-46668501-dcf3-613d-4f54-fde5c2a55bcf 46668501-dcf3-613d-4f54-fde5c2a55bcf}" testing.go:149: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.705Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: pipelining replication: peer="{Voter server-46668501-dcf3-613d-4f54-fde5c2a55bcf 46668501-dcf3-613d-4f54-fde5c2a55bcf}" testLoggerAdapter verbose: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.705Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: pipelining replication: peer="{Voter server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4 9aa15ca3-275d-8bac-133b-09b2ee61c7d4}" testing.go:149: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.705Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: pipelining replication: peer="{Voter server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4 9aa15ca3-275d-8bac-133b-09b2ee61c7d4}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:04.810Z [INFO] stable state for Leader reached at 2021-07-10 01:05:04.705224358 +0000 UTC m=+10.093382328 (1 nodes), 51.26126ms from start of poll, 52.254571ms from cluster start. Timeout at 2021-07-10 01:05:04.810289046 +0000 UTC m=+10.198447036, 105.064708ms after stability testing.go:149: cluster: 2021-07-10T01:05:04.810Z [INFO] stable state for Leader reached at 2021-07-10 01:05:04.705224358 +0000 UTC m=+10.093382328 (1 nodes), 51.26126ms from start of poll, 52.254571ms from cluster start. Timeout at 2021-07-10 01:05:04.810289046 +0000 UTC m=+10.198447036, 105.064708ms after stability testLoggerAdapter verbose: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.810Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: updating configuration: command=RemoveServer server-id=9b1b8d02-0f87-2fe1-81fa-12f8626b104b server-addr= servers="[{Suffrage:Voter ID:server-46668501-dcf3-613d-4f54-fde5c2a55bcf Address:46668501-dcf3-613d-4f54-fde5c2a55bcf} {Suffrage:Voter ID:server-3497e30e-884c-48a8-5653-12d1a38507a4 Address:3497e30e-884c-48a8-5653-12d1a38507a4} {Suffrage:Voter ID:server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4 Address:9aa15ca3-275d-8bac-133b-09b2ee61c7d4}]" testing.go:149: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.810Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: updating configuration: command=RemoveServer server-id=9b1b8d02-0f87-2fe1-81fa-12f8626b104b server-addr= servers="[{Suffrage:Voter ID:server-46668501-dcf3-613d-4f54-fde5c2a55bcf Address:46668501-dcf3-613d-4f54-fde5c2a55bcf} {Suffrage:Voter ID:server-3497e30e-884c-48a8-5653-12d1a38507a4 Address:3497e30e-884c-48a8-5653-12d1a38507a4} {Suffrage:Voter ID:server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4 Address:9aa15ca3-275d-8bac-133b-09b2ee61c7d4}]" testLoggerAdapter verbose: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.811Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: aborting pipeline replication: peer="{Voter server-46668501-dcf3-613d-4f54-fde5c2a55bcf 46668501-dcf3-613d-4f54-fde5c2a55bcf}" testing.go:149: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.811Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: aborting pipeline replication: peer="{Voter server-46668501-dcf3-613d-4f54-fde5c2a55bcf 46668501-dcf3-613d-4f54-fde5c2a55bcf}" testLoggerAdapter verbose: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.811Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: aborting pipeline replication: peer="{Voter server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4 9aa15ca3-275d-8bac-133b-09b2ee61c7d4}" testing.go:149: server-3497e30e-884c-48a8-5653-12d1a38507a4: 2021-07-10T01:05:04.811Z [INFO] server-3497e30e-884c-48a8-5653-12d1a38507a4: aborting pipeline replication: peer="{Voter server-9aa15ca3-275d-8bac-133b-09b2ee61c7d4 9aa15ca3-275d-8bac-133b-09b2ee61c7d4}" --- PASS: TestRaft_RemoveUnknownPeer (0.16s) === RUN TestRaft_SnapshotRestore testLoggerAdapter verbose: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:04.812Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-72ddfcd9-c408-849c-f04f-07e68de80a4b Address:72ddfcd9-c408-849c-f04f-07e68de80a4b}]" testing.go:149: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:04.812Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-72ddfcd9-c408-849c-f04f-07e68de80a4b Address:72ddfcd9-c408-849c-f04f-07e68de80a4b}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:04.812Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:04.812Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:04.812Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: entering follower state: follower="Node at 72ddfcd9-c408-849c-f04f-07e68de80a4b [Follower]" leader= testing.go:149: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:04.812Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: entering follower state: follower="Node at 72ddfcd9-c408-849c-f04f-07e68de80a4b [Follower]" leader= testLoggerAdapter verbose: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:04.911Z [WARN] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:04.911Z [WARN] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:04.911Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: entering candidate state: node="Node at 72ddfcd9-c408-849c-f04f-07e68de80a4b [Candidate]" term=2 testing.go:149: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:04.911Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: entering candidate state: node="Node at 72ddfcd9-c408-849c-f04f-07e68de80a4b [Candidate]" term=2 testLoggerAdapter verbose: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:04.911Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: election won: tally=1 testing.go:149: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:04.911Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: election won: tally=1 testLoggerAdapter verbose: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:04.911Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: entering leader state: leader="Node at 72ddfcd9-c408-849c-f04f-07e68de80a4b [Leader]" testing.go:149: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:04.911Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: entering leader state: leader="Node at 72ddfcd9-c408-849c-f04f-07e68de80a4b [Leader]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:05.017Z [INFO] stable state for Leader reached at 2021-07-10 01:05:04.911998018 +0000 UTC m=+10.300156068 (1 nodes), 99.60725ms from start of poll, 99.965774ms from cluster start. Timeout at 2021-07-10 01:05:05.017103486 +0000 UTC m=+10.405261456, 105.105388ms after stability testing.go:149: cluster: 2021-07-10T01:05:05.017Z [INFO] stable state for Leader reached at 2021-07-10 01:05:04.911998018 +0000 UTC m=+10.300156068 (1 nodes), 99.60725ms from start of poll, 99.965774ms from cluster start. Timeout at 2021-07-10 01:05:05.017103486 +0000 UTC m=+10.405261456, 105.105388ms after stability testLoggerAdapter verbose: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:05.019Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: starting snapshot up to: index=102 testing.go:149: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:05.019Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: starting snapshot up to: index=102 testing.go:153: 2021-07-10T01:05:05.019Z [INFO] creating new snapshot: path=/tmp/raft700143239/snapshots/2-102-1625879105019.tmp testLoggerAdapter verbose: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:05.020Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: compacting logs: from=1 to=92 testing.go:149: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:05.020Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:05.020Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: snapshot complete up to: index=102 testing.go:149: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:05.020Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: snapshot complete up to: index=102 testLoggerAdapter verbose: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:05.020Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: restored from snapshot: id=2-102-1625879105019 testing.go:149: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:05.020Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: restored from snapshot: id=2-102-1625879105019 testLoggerAdapter verbose: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:05.021Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-72ddfcd9-c408-849c-f04f-07e68de80a4b Address:72ddfcd9-c408-849c-f04f-07e68de80a4b}]" testing.go:149: server-72ddfcd9-c408-849c-f04f-07e68de80a4b: 2021-07-10T01:05:05.021Z [INFO] server-72ddfcd9-c408-849c-f04f-07e68de80a4b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-72ddfcd9-c408-849c-f04f-07e68de80a4b Address:72ddfcd9-c408-849c-f04f-07e68de80a4b}]" --- PASS: TestRaft_SnapshotRestore (0.21s) === RUN TestRaft_NoRestoreOnStart testLoggerAdapter verbose: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.021Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1 Address:9de1fb4d-94f3-599a-7324-aa4df2d8ddf1}]" testing.go:149: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.021Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1 Address:9de1fb4d-94f3-599a-7324-aa4df2d8ddf1}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:05.022Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:05.022Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.022Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: entering follower state: follower="Node at 9de1fb4d-94f3-599a-7324-aa4df2d8ddf1 [Follower]" leader= testing.go:149: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.022Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: entering follower state: follower="Node at 9de1fb4d-94f3-599a-7324-aa4df2d8ddf1 [Follower]" leader= testLoggerAdapter verbose: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.077Z [WARN] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.077Z [WARN] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.077Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: entering candidate state: node="Node at 9de1fb4d-94f3-599a-7324-aa4df2d8ddf1 [Candidate]" term=2 testing.go:149: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.077Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: entering candidate state: node="Node at 9de1fb4d-94f3-599a-7324-aa4df2d8ddf1 [Candidate]" term=2 testLoggerAdapter verbose: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.077Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: election won: tally=1 testing.go:149: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.077Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: election won: tally=1 testLoggerAdapter verbose: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.077Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: entering leader state: leader="Node at 9de1fb4d-94f3-599a-7324-aa4df2d8ddf1 [Leader]" testing.go:149: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.077Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: entering leader state: leader="Node at 9de1fb4d-94f3-599a-7324-aa4df2d8ddf1 [Leader]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:05.183Z [INFO] stable state for Leader reached at 2021-07-10 01:05:05.077914387 +0000 UTC m=+10.466072397 (1 nodes), 55.796328ms from start of poll, 56.162772ms from cluster start. Timeout at 2021-07-10 01:05:05.183029495 +0000 UTC m=+10.571187465, 105.115068ms after stability testing.go:149: cluster: 2021-07-10T01:05:05.183Z [INFO] stable state for Leader reached at 2021-07-10 01:05:05.077914387 +0000 UTC m=+10.466072397 (1 nodes), 55.796328ms from start of poll, 56.162772ms from cluster start. Timeout at 2021-07-10 01:05:05.183029495 +0000 UTC m=+10.571187465, 105.115068ms after stability testLoggerAdapter verbose: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.185Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: starting snapshot up to: index=102 testing.go:149: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.185Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: starting snapshot up to: index=102 testing.go:153: 2021-07-10T01:05:05.185Z [INFO] creating new snapshot: path=/tmp/raft600398929/snapshots/2-102-1625879105185.tmp testLoggerAdapter verbose: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.186Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: compacting logs: from=1 to=92 testing.go:149: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.186Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.186Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: snapshot complete up to: index=102 testing.go:149: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.186Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: snapshot complete up to: index=102 testLoggerAdapter verbose: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.186Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1 Address:9de1fb4d-94f3-599a-7324-aa4df2d8ddf1}]" testing.go:149: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.186Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1 Address:9de1fb4d-94f3-599a-7324-aa4df2d8ddf1}]" --- PASS: TestRaft_NoRestoreOnStart (0.17s) === RUN TestRaft_SnapshotRestore_PeerChange raft_test.go:1026: Disabled unreliable test --- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s) === RUN TestRaft_AutoSnapshot raft_test.go:1132: Disabled unreliable test --- SKIP: TestRaft_AutoSnapshot (0.00s) === RUN TestRaft_UserSnapshot testLoggerAdapter verbose: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.187Z [INFO] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5 Address:9f7bc916-22a0-854d-b0ec-12d138c3cdf5}]" testing.go:149: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.187Z [INFO] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5 Address:9f7bc916-22a0-854d-b0ec-12d138c3cdf5}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:05.187Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:05.187Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.187Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: entering follower state: follower="Node at 9de1fb4d-94f3-599a-7324-aa4df2d8ddf1 [Follower]" leader= testLoggerAdapter verbose: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.187Z [INFO] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: entering follower state: follower="Node at 9f7bc916-22a0-854d-b0ec-12d138c3cdf5 [Follower]" leader= testing.go:149: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.187Z [INFO] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: entering follower state: follower="Node at 9f7bc916-22a0-854d-b0ec-12d138c3cdf5 [Follower]" leader= testLoggerAdapter verbose: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.244Z [WARN] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.244Z [WARN] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.244Z [INFO] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: entering candidate state: node="Node at 9f7bc916-22a0-854d-b0ec-12d138c3cdf5 [Candidate]" term=2 testing.go:149: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.244Z [INFO] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: entering candidate state: node="Node at 9f7bc916-22a0-854d-b0ec-12d138c3cdf5 [Candidate]" term=2 testLoggerAdapter verbose: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.244Z [INFO] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: election won: tally=1 testing.go:149: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.244Z [INFO] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: election won: tally=1 testLoggerAdapter verbose: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.244Z [INFO] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: entering leader state: leader="Node at 9f7bc916-22a0-854d-b0ec-12d138c3cdf5 [Leader]" testing.go:149: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.244Z [INFO] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: entering leader state: leader="Node at 9f7bc916-22a0-854d-b0ec-12d138c3cdf5 [Leader]" testLoggerAdapter verbose: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.250Z [WARN] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.251Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: entering candidate state: node="Node at 9de1fb4d-94f3-599a-7324-aa4df2d8ddf1 [Candidate]" term=3 testLoggerAdapter verbose: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.251Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: election won: tally=1 testLoggerAdapter verbose: server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: 2021-07-10T01:05:05.251Z [INFO] server-9de1fb4d-94f3-599a-7324-aa4df2d8ddf1: entering leader state: leader="Node at 9de1fb4d-94f3-599a-7324-aa4df2d8ddf1 [Leader]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:05.349Z [INFO] stable state for Leader reached at 2021-07-10 01:05:05.244475743 +0000 UTC m=+10.632633733 (1 nodes), 56.769659ms from start of poll, 57.178683ms from cluster start. Timeout at 2021-07-10 01:05:05.349577791 +0000 UTC m=+10.737735761, 105.102028ms after stability testing.go:149: cluster: 2021-07-10T01:05:05.349Z [INFO] stable state for Leader reached at 2021-07-10 01:05:05.244475743 +0000 UTC m=+10.632633733 (1 nodes), 56.769659ms from start of poll, 57.178683ms from cluster start. Timeout at 2021-07-10 01:05:05.349577791 +0000 UTC m=+10.737735761, 105.102028ms after stability testLoggerAdapter verbose: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.350Z [ERROR] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: failed to take snapshot: error="nothing new to snapshot" testing.go:149: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.350Z [ERROR] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: failed to take snapshot: error="nothing new to snapshot" testLoggerAdapter verbose: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.350Z [INFO] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: starting snapshot up to: index=12 testing.go:149: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.350Z [INFO] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: starting snapshot up to: index=12 testing.go:153: 2021-07-10T01:05:05.350Z [INFO] creating new snapshot: path=/tmp/raft874132139/snapshots/2-12-1625879105350.tmp testLoggerAdapter verbose: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.351Z [INFO] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: compacting logs: from=1 to=2 testing.go:149: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.351Z [INFO] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: compacting logs: from=1 to=2 testLoggerAdapter verbose: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.351Z [INFO] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: snapshot complete up to: index=12 testing.go:149: server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: 2021-07-10T01:05:05.351Z [INFO] server-9f7bc916-22a0-854d-b0ec-12d138c3cdf5: snapshot complete up to: index=12 --- PASS: TestRaft_UserSnapshot (0.16s) === RUN TestRaft_UserRestore testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.352Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3d894380-81d6-593f-a2b3-516cefadd7e7 Address:3d894380-81d6-593f-a2b3-516cefadd7e7} {Suffrage:Voter ID:server-e818da0e-30b8-095b-47df-8156b2526404 Address:e818da0e-30b8-095b-47df-8156b2526404} {Suffrage:Voter ID:server-e9b07343-5cdf-0560-bed6-36501b160906 Address:e9b07343-5cdf-0560-bed6-36501b160906}]" testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.352Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3d894380-81d6-593f-a2b3-516cefadd7e7 Address:3d894380-81d6-593f-a2b3-516cefadd7e7} {Suffrage:Voter ID:server-e818da0e-30b8-095b-47df-8156b2526404 Address:e818da0e-30b8-095b-47df-8156b2526404} {Suffrage:Voter ID:server-e9b07343-5cdf-0560-bed6-36501b160906 Address:e9b07343-5cdf-0560-bed6-36501b160906}]" testLoggerAdapter verbose: server-e818da0e-30b8-095b-47df-8156b2526404: 2021-07-10T01:05:05.353Z [INFO] server-e818da0e-30b8-095b-47df-8156b2526404: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3d894380-81d6-593f-a2b3-516cefadd7e7 Address:3d894380-81d6-593f-a2b3-516cefadd7e7} {Suffrage:Voter ID:server-e818da0e-30b8-095b-47df-8156b2526404 Address:e818da0e-30b8-095b-47df-8156b2526404} {Suffrage:Voter ID:server-e9b07343-5cdf-0560-bed6-36501b160906 Address:e9b07343-5cdf-0560-bed6-36501b160906}]" testing.go:149: server-e818da0e-30b8-095b-47df-8156b2526404: 2021-07-10T01:05:05.353Z [INFO] server-e818da0e-30b8-095b-47df-8156b2526404: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3d894380-81d6-593f-a2b3-516cefadd7e7 Address:3d894380-81d6-593f-a2b3-516cefadd7e7} {Suffrage:Voter ID:server-e818da0e-30b8-095b-47df-8156b2526404 Address:e818da0e-30b8-095b-47df-8156b2526404} {Suffrage:Voter ID:server-e9b07343-5cdf-0560-bed6-36501b160906 Address:e9b07343-5cdf-0560-bed6-36501b160906}]" testLoggerAdapter verbose: server-e9b07343-5cdf-0560-bed6-36501b160906: 2021-07-10T01:05:05.353Z [INFO] server-e9b07343-5cdf-0560-bed6-36501b160906: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3d894380-81d6-593f-a2b3-516cefadd7e7 Address:3d894380-81d6-593f-a2b3-516cefadd7e7} {Suffrage:Voter ID:server-e818da0e-30b8-095b-47df-8156b2526404 Address:e818da0e-30b8-095b-47df-8156b2526404} {Suffrage:Voter ID:server-e9b07343-5cdf-0560-bed6-36501b160906 Address:e9b07343-5cdf-0560-bed6-36501b160906}]" testing.go:149: server-e9b07343-5cdf-0560-bed6-36501b160906: 2021-07-10T01:05:05.353Z [INFO] server-e9b07343-5cdf-0560-bed6-36501b160906: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3d894380-81d6-593f-a2b3-516cefadd7e7 Address:3d894380-81d6-593f-a2b3-516cefadd7e7} {Suffrage:Voter ID:server-e818da0e-30b8-095b-47df-8156b2526404 Address:e818da0e-30b8-095b-47df-8156b2526404} {Suffrage:Voter ID:server-e9b07343-5cdf-0560-bed6-36501b160906 Address:e9b07343-5cdf-0560-bed6-36501b160906}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:05.353Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:05.353Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.353Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: entering follower state: follower="Node at 3d894380-81d6-593f-a2b3-516cefadd7e7 [Follower]" leader= testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.353Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: entering follower state: follower="Node at 3d894380-81d6-593f-a2b3-516cefadd7e7 [Follower]" leader= testLoggerAdapter verbose: server-e818da0e-30b8-095b-47df-8156b2526404: 2021-07-10T01:05:05.353Z [INFO] server-e818da0e-30b8-095b-47df-8156b2526404: entering follower state: follower="Node at e818da0e-30b8-095b-47df-8156b2526404 [Follower]" leader= testing.go:149: server-e818da0e-30b8-095b-47df-8156b2526404: 2021-07-10T01:05:05.353Z [INFO] server-e818da0e-30b8-095b-47df-8156b2526404: entering follower state: follower="Node at e818da0e-30b8-095b-47df-8156b2526404 [Follower]" leader= testLoggerAdapter verbose: server-e9b07343-5cdf-0560-bed6-36501b160906: 2021-07-10T01:05:05.353Z [INFO] server-e9b07343-5cdf-0560-bed6-36501b160906: entering follower state: follower="Node at e9b07343-5cdf-0560-bed6-36501b160906 [Follower]" leader= testing.go:149: server-e9b07343-5cdf-0560-bed6-36501b160906: 2021-07-10T01:05:05.353Z [INFO] server-e9b07343-5cdf-0560-bed6-36501b160906: entering follower state: follower="Node at e9b07343-5cdf-0560-bed6-36501b160906 [Follower]" leader= testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.438Z [WARN] server-3d894380-81d6-593f-a2b3-516cefadd7e7: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.438Z [WARN] server-3d894380-81d6-593f-a2b3-516cefadd7e7: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.439Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: entering candidate state: node="Node at 3d894380-81d6-593f-a2b3-516cefadd7e7 [Candidate]" term=2 testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.439Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: entering candidate state: node="Node at 3d894380-81d6-593f-a2b3-516cefadd7e7 [Candidate]" term=2 testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.439Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: election won: tally=2 testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.439Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: election won: tally=2 testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.439Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: entering leader state: leader="Node at 3d894380-81d6-593f-a2b3-516cefadd7e7 [Leader]" testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.439Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: entering leader state: leader="Node at 3d894380-81d6-593f-a2b3-516cefadd7e7 [Leader]" testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.439Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: added peer, starting replication: peer=server-e818da0e-30b8-095b-47df-8156b2526404 testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.439Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: added peer, starting replication: peer=server-e818da0e-30b8-095b-47df-8156b2526404 testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.439Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: added peer, starting replication: peer=server-e9b07343-5cdf-0560-bed6-36501b160906 testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.439Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: added peer, starting replication: peer=server-e9b07343-5cdf-0560-bed6-36501b160906 testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.439Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: pipelining replication: peer="{Voter server-e9b07343-5cdf-0560-bed6-36501b160906 e9b07343-5cdf-0560-bed6-36501b160906}" testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.439Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: pipelining replication: peer="{Voter server-e9b07343-5cdf-0560-bed6-36501b160906 e9b07343-5cdf-0560-bed6-36501b160906}" testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.439Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: pipelining replication: peer="{Voter server-e818da0e-30b8-095b-47df-8156b2526404 e818da0e-30b8-095b-47df-8156b2526404}" testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.439Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: pipelining replication: peer="{Voter server-e818da0e-30b8-095b-47df-8156b2526404 e818da0e-30b8-095b-47df-8156b2526404}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:05.544Z [INFO] stable state for Leader reached at 2021-07-10 01:05:05.43957486 +0000 UTC m=+10.827732850 (1 nodes), 85.932266ms from start of poll, 86.935197ms from cluster start. Timeout at 2021-07-10 01:05:05.544666968 +0000 UTC m=+10.932824938, 105.092088ms after stability testing.go:149: cluster: 2021-07-10T01:05:05.544Z [INFO] stable state for Leader reached at 2021-07-10 01:05:05.43957486 +0000 UTC m=+10.827732850 (1 nodes), 85.932266ms from start of poll, 86.935197ms from cluster start. Timeout at 2021-07-10 01:05:05.544666968 +0000 UTC m=+10.932824938, 105.092088ms after stability testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.545Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: starting snapshot up to: index=12 testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.545Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: starting snapshot up to: index=12 testing.go:153: 2021-07-10T01:05:05.545Z [INFO] creating new snapshot: path=/tmp/raft110042645/snapshots/2-12-1625879105545.tmp testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.546Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: snapshot complete up to: index=12 testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.546Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: snapshot complete up to: index=12 testing.go:153: 2021-07-10T01:05:05.547Z [INFO] creating new snapshot: path=/tmp/raft110042645/snapshots/2-23-1625879105547.tmp testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.548Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: copied to local snapshot: bytes=71 testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.548Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.549Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: restored user snapshot: index=1 testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.549Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: restored user snapshot: index=1 testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.549Z [ERROR] server-3d894380-81d6-593f-a2b3-516cefadd7e7: failed to get log: index=23 error="log not found" testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.549Z [ERROR] server-3d894380-81d6-593f-a2b3-516cefadd7e7: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.549Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: aborting pipeline replication: peer="{Voter server-e818da0e-30b8-095b-47df-8156b2526404 e818da0e-30b8-095b-47df-8156b2526404}" testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.549Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: aborting pipeline replication: peer="{Voter server-e818da0e-30b8-095b-47df-8156b2526404 e818da0e-30b8-095b-47df-8156b2526404}" testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.549Z [ERROR] server-3d894380-81d6-593f-a2b3-516cefadd7e7: failed to get log: index=23 error="log not found" testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.549Z [ERROR] server-3d894380-81d6-593f-a2b3-516cefadd7e7: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.549Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: aborting pipeline replication: peer="{Voter server-e9b07343-5cdf-0560-bed6-36501b160906 e9b07343-5cdf-0560-bed6-36501b160906}" testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.549Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: aborting pipeline replication: peer="{Voter server-e9b07343-5cdf-0560-bed6-36501b160906 e9b07343-5cdf-0560-bed6-36501b160906}" testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.557Z [ERROR] server-3d894380-81d6-593f-a2b3-516cefadd7e7: failed to get log: index=23 error="log not found" testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.557Z [ERROR] server-3d894380-81d6-593f-a2b3-516cefadd7e7: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.558Z [ERROR] server-3d894380-81d6-593f-a2b3-516cefadd7e7: failed to get log: index=23 error="log not found" testing.go:153: 2021-07-10T01:05:05.558Z [INFO] creating new snapshot: path=/tmp/raft184352015/snapshots/2-23-1625879105558.tmp testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.558Z [ERROR] server-3d894380-81d6-593f-a2b3-516cefadd7e7: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-e818da0e-30b8-095b-47df-8156b2526404: 2021-07-10T01:05:05.558Z [INFO] server-e818da0e-30b8-095b-47df-8156b2526404: copied to local snapshot: bytes=71 testing.go:153: 2021-07-10T01:05:05.558Z [INFO] creating new snapshot: path=/tmp/raft065387545/snapshots/2-23-1625879105558.tmp testing.go:149: server-e818da0e-30b8-095b-47df-8156b2526404: 2021-07-10T01:05:05.558Z [INFO] server-e818da0e-30b8-095b-47df-8156b2526404: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-e818da0e-30b8-095b-47df-8156b2526404: 2021-07-10T01:05:05.559Z [INFO] server-e818da0e-30b8-095b-47df-8156b2526404: Installed remote snapshot testing.go:149: server-e818da0e-30b8-095b-47df-8156b2526404: 2021-07-10T01:05:05.559Z [INFO] server-e818da0e-30b8-095b-47df-8156b2526404: Installed remote snapshot testLoggerAdapter verbose: server-e818da0e-30b8-095b-47df-8156b2526404: 2021-07-10T01:05:05.559Z [WARN] server-e818da0e-30b8-095b-47df-8156b2526404: skipping application of old log: index=22 testing.go:149: server-e818da0e-30b8-095b-47df-8156b2526404: 2021-07-10T01:05:05.559Z [WARN] server-e818da0e-30b8-095b-47df-8156b2526404: skipping application of old log: index=22 testLoggerAdapter verbose: server-e9b07343-5cdf-0560-bed6-36501b160906: 2021-07-10T01:05:05.559Z [INFO] server-e9b07343-5cdf-0560-bed6-36501b160906: copied to local snapshot: bytes=71 testing.go:149: server-e9b07343-5cdf-0560-bed6-36501b160906: 2021-07-10T01:05:05.559Z [INFO] server-e9b07343-5cdf-0560-bed6-36501b160906: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.559Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: pipelining replication: peer="{Voter server-e818da0e-30b8-095b-47df-8156b2526404 e818da0e-30b8-095b-47df-8156b2526404}" testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.559Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: pipelining replication: peer="{Voter server-e818da0e-30b8-095b-47df-8156b2526404 e818da0e-30b8-095b-47df-8156b2526404}" testLoggerAdapter verbose: server-e9b07343-5cdf-0560-bed6-36501b160906: 2021-07-10T01:05:05.559Z [INFO] server-e9b07343-5cdf-0560-bed6-36501b160906: Installed remote snapshot testing.go:149: server-e9b07343-5cdf-0560-bed6-36501b160906: 2021-07-10T01:05:05.559Z [INFO] server-e9b07343-5cdf-0560-bed6-36501b160906: Installed remote snapshot testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.559Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: pipelining replication: peer="{Voter server-e9b07343-5cdf-0560-bed6-36501b160906 e9b07343-5cdf-0560-bed6-36501b160906}" testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.559Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: pipelining replication: peer="{Voter server-e9b07343-5cdf-0560-bed6-36501b160906 e9b07343-5cdf-0560-bed6-36501b160906}" testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.575Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: aborting pipeline replication: peer="{Voter server-e818da0e-30b8-095b-47df-8156b2526404 e818da0e-30b8-095b-47df-8156b2526404}" testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.575Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: aborting pipeline replication: peer="{Voter server-e818da0e-30b8-095b-47df-8156b2526404 e818da0e-30b8-095b-47df-8156b2526404}" testLoggerAdapter verbose: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.575Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: aborting pipeline replication: peer="{Voter server-e9b07343-5cdf-0560-bed6-36501b160906 e9b07343-5cdf-0560-bed6-36501b160906}" testing.go:149: server-3d894380-81d6-593f-a2b3-516cefadd7e7: 2021-07-10T01:05:05.575Z [INFO] server-3d894380-81d6-593f-a2b3-516cefadd7e7: aborting pipeline replication: peer="{Voter server-e9b07343-5cdf-0560-bed6-36501b160906 e9b07343-5cdf-0560-bed6-36501b160906}" testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.577Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5df398f-3349-4628-2f92-4e243963cc61 Address:d5df398f-3349-4628-2f92-4e243963cc61} {Suffrage:Voter ID:server-6a233ea6-9f29-cb26-8f90-22b24de85079 Address:6a233ea6-9f29-cb26-8f90-22b24de85079} {Suffrage:Voter ID:server-4b63ed8c-a916-e473-8b4c-baa46c99b91b Address:4b63ed8c-a916-e473-8b4c-baa46c99b91b}]" testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.577Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5df398f-3349-4628-2f92-4e243963cc61 Address:d5df398f-3349-4628-2f92-4e243963cc61} {Suffrage:Voter ID:server-6a233ea6-9f29-cb26-8f90-22b24de85079 Address:6a233ea6-9f29-cb26-8f90-22b24de85079} {Suffrage:Voter ID:server-4b63ed8c-a916-e473-8b4c-baa46c99b91b Address:4b63ed8c-a916-e473-8b4c-baa46c99b91b}]" testLoggerAdapter verbose: server-6a233ea6-9f29-cb26-8f90-22b24de85079: 2021-07-10T01:05:05.577Z [INFO] server-6a233ea6-9f29-cb26-8f90-22b24de85079: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5df398f-3349-4628-2f92-4e243963cc61 Address:d5df398f-3349-4628-2f92-4e243963cc61} {Suffrage:Voter ID:server-6a233ea6-9f29-cb26-8f90-22b24de85079 Address:6a233ea6-9f29-cb26-8f90-22b24de85079} {Suffrage:Voter ID:server-4b63ed8c-a916-e473-8b4c-baa46c99b91b Address:4b63ed8c-a916-e473-8b4c-baa46c99b91b}]" testing.go:149: server-6a233ea6-9f29-cb26-8f90-22b24de85079: 2021-07-10T01:05:05.577Z [INFO] server-6a233ea6-9f29-cb26-8f90-22b24de85079: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5df398f-3349-4628-2f92-4e243963cc61 Address:d5df398f-3349-4628-2f92-4e243963cc61} {Suffrage:Voter ID:server-6a233ea6-9f29-cb26-8f90-22b24de85079 Address:6a233ea6-9f29-cb26-8f90-22b24de85079} {Suffrage:Voter ID:server-4b63ed8c-a916-e473-8b4c-baa46c99b91b Address:4b63ed8c-a916-e473-8b4c-baa46c99b91b}]" testLoggerAdapter verbose: server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: 2021-07-10T01:05:05.577Z [INFO] server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5df398f-3349-4628-2f92-4e243963cc61 Address:d5df398f-3349-4628-2f92-4e243963cc61} {Suffrage:Voter ID:server-6a233ea6-9f29-cb26-8f90-22b24de85079 Address:6a233ea6-9f29-cb26-8f90-22b24de85079} {Suffrage:Voter ID:server-4b63ed8c-a916-e473-8b4c-baa46c99b91b Address:4b63ed8c-a916-e473-8b4c-baa46c99b91b}]" testing.go:149: server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: 2021-07-10T01:05:05.577Z [INFO] server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5df398f-3349-4628-2f92-4e243963cc61 Address:d5df398f-3349-4628-2f92-4e243963cc61} {Suffrage:Voter ID:server-6a233ea6-9f29-cb26-8f90-22b24de85079 Address:6a233ea6-9f29-cb26-8f90-22b24de85079} {Suffrage:Voter ID:server-4b63ed8c-a916-e473-8b4c-baa46c99b91b Address:4b63ed8c-a916-e473-8b4c-baa46c99b91b}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:05.577Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:05.577Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-6a233ea6-9f29-cb26-8f90-22b24de85079: 2021-07-10T01:05:05.577Z [INFO] server-6a233ea6-9f29-cb26-8f90-22b24de85079: entering follower state: follower="Node at 6a233ea6-9f29-cb26-8f90-22b24de85079 [Follower]" leader= testLoggerAdapter verbose: server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: 2021-07-10T01:05:05.577Z [INFO] server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: entering follower state: follower="Node at 4b63ed8c-a916-e473-8b4c-baa46c99b91b [Follower]" leader= testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.577Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: entering follower state: follower="Node at d5df398f-3349-4628-2f92-4e243963cc61 [Follower]" leader= testing.go:149: server-6a233ea6-9f29-cb26-8f90-22b24de85079: 2021-07-10T01:05:05.577Z [INFO] server-6a233ea6-9f29-cb26-8f90-22b24de85079: entering follower state: follower="Node at 6a233ea6-9f29-cb26-8f90-22b24de85079 [Follower]" leader= testing.go:149: server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: 2021-07-10T01:05:05.577Z [INFO] server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: entering follower state: follower="Node at 4b63ed8c-a916-e473-8b4c-baa46c99b91b [Follower]" leader= testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.577Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: entering follower state: follower="Node at d5df398f-3349-4628-2f92-4e243963cc61 [Follower]" leader= testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.629Z [WARN] server-d5df398f-3349-4628-2f92-4e243963cc61: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.629Z [WARN] server-d5df398f-3349-4628-2f92-4e243963cc61: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.629Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: entering candidate state: node="Node at d5df398f-3349-4628-2f92-4e243963cc61 [Candidate]" term=2 testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.629Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: entering candidate state: node="Node at d5df398f-3349-4628-2f92-4e243963cc61 [Candidate]" term=2 testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.629Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: election won: tally=2 testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.629Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: election won: tally=2 testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.629Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: entering leader state: leader="Node at d5df398f-3349-4628-2f92-4e243963cc61 [Leader]" testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.629Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: entering leader state: leader="Node at d5df398f-3349-4628-2f92-4e243963cc61 [Leader]" testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.629Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: added peer, starting replication: peer=server-6a233ea6-9f29-cb26-8f90-22b24de85079 testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.629Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: added peer, starting replication: peer=server-6a233ea6-9f29-cb26-8f90-22b24de85079 testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.629Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: added peer, starting replication: peer=server-4b63ed8c-a916-e473-8b4c-baa46c99b91b testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.629Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: added peer, starting replication: peer=server-4b63ed8c-a916-e473-8b4c-baa46c99b91b testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.629Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: pipelining replication: peer="{Voter server-4b63ed8c-a916-e473-8b4c-baa46c99b91b 4b63ed8c-a916-e473-8b4c-baa46c99b91b}" testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.629Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: pipelining replication: peer="{Voter server-4b63ed8c-a916-e473-8b4c-baa46c99b91b 4b63ed8c-a916-e473-8b4c-baa46c99b91b}" testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.629Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: pipelining replication: peer="{Voter server-6a233ea6-9f29-cb26-8f90-22b24de85079 6a233ea6-9f29-cb26-8f90-22b24de85079}" testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.629Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: pipelining replication: peer="{Voter server-6a233ea6-9f29-cb26-8f90-22b24de85079 6a233ea6-9f29-cb26-8f90-22b24de85079}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:05.734Z [INFO] stable state for Leader reached at 2021-07-10 01:05:05.629807685 +0000 UTC m=+11.017965655 (1 nodes), 52.079169ms from start of poll, 52.951478ms from cluster start. Timeout at 2021-07-10 01:05:05.734879353 +0000 UTC m=+11.123037303, 105.071648ms after stability testing.go:149: cluster: 2021-07-10T01:05:05.734Z [INFO] stable state for Leader reached at 2021-07-10 01:05:05.629807685 +0000 UTC m=+11.017965655 (1 nodes), 52.079169ms from start of poll, 52.951478ms from cluster start. Timeout at 2021-07-10 01:05:05.734879353 +0000 UTC m=+11.123037303, 105.071648ms after stability testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.735Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: starting snapshot up to: index=12 testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.735Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: starting snapshot up to: index=12 testing.go:153: 2021-07-10T01:05:05.735Z [INFO] creating new snapshot: path=/tmp/raft563676595/snapshots/2-12-1625879105735.tmp testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.736Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: snapshot complete up to: index=12 testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.736Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: snapshot complete up to: index=12 testing.go:153: 2021-07-10T01:05:05.736Z [INFO] creating new snapshot: path=/tmp/raft563676595/snapshots/2-23-1625879105736.tmp testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.737Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: copied to local snapshot: bytes=71 testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.737Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.738Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: restored user snapshot: index=1 testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.738Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: restored user snapshot: index=1 testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.738Z [ERROR] server-d5df398f-3349-4628-2f92-4e243963cc61: failed to get log: index=23 error="log not found" testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.738Z [ERROR] server-d5df398f-3349-4628-2f92-4e243963cc61: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.738Z [ERROR] server-d5df398f-3349-4628-2f92-4e243963cc61: failed to get log: index=23 error="log not found" testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.738Z [ERROR] server-d5df398f-3349-4628-2f92-4e243963cc61: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.738Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: aborting pipeline replication: peer="{Voter server-4b63ed8c-a916-e473-8b4c-baa46c99b91b 4b63ed8c-a916-e473-8b4c-baa46c99b91b}" testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.738Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: aborting pipeline replication: peer="{Voter server-4b63ed8c-a916-e473-8b4c-baa46c99b91b 4b63ed8c-a916-e473-8b4c-baa46c99b91b}" testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.738Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: aborting pipeline replication: peer="{Voter server-6a233ea6-9f29-cb26-8f90-22b24de85079 6a233ea6-9f29-cb26-8f90-22b24de85079}" testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.738Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: aborting pipeline replication: peer="{Voter server-6a233ea6-9f29-cb26-8f90-22b24de85079 6a233ea6-9f29-cb26-8f90-22b24de85079}" testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.743Z [ERROR] server-d5df398f-3349-4628-2f92-4e243963cc61: failed to get log: index=23 error="log not found" testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.743Z [ERROR] server-d5df398f-3349-4628-2f92-4e243963cc61: failed to get log: index=23 error="log not found" testing.go:153: 2021-07-10T01:05:05.744Z [INFO] creating new snapshot: path=/tmp/raft278181015/snapshots/2-23-1625879105744.tmp testLoggerAdapter verbose: server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: 2021-07-10T01:05:05.744Z [INFO] server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: copied to local snapshot: bytes=71 testing.go:149: server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: 2021-07-10T01:05:05.744Z [INFO] server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: 2021-07-10T01:05:05.745Z [INFO] server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: Installed remote snapshot testing.go:149: server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: 2021-07-10T01:05:05.745Z [INFO] server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: Installed remote snapshot testLoggerAdapter verbose: server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: 2021-07-10T01:05:05.745Z [WARN] server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: skipping application of old log: index=22 testing.go:149: server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: 2021-07-10T01:05:05.745Z [WARN] server-4b63ed8c-a916-e473-8b4c-baa46c99b91b: skipping application of old log: index=22 testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.745Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: pipelining replication: peer="{Voter server-4b63ed8c-a916-e473-8b4c-baa46c99b91b 4b63ed8c-a916-e473-8b4c-baa46c99b91b}" testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.745Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: pipelining replication: peer="{Voter server-4b63ed8c-a916-e473-8b4c-baa46c99b91b 4b63ed8c-a916-e473-8b4c-baa46c99b91b}" testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.746Z [ERROR] server-d5df398f-3349-4628-2f92-4e243963cc61: failed to get log: index=23 error="log not found" testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.746Z [ERROR] server-d5df398f-3349-4628-2f92-4e243963cc61: failed to get log: index=23 error="log not found" testing.go:153: 2021-07-10T01:05:05.747Z [INFO] creating new snapshot: path=/tmp/raft347660893/snapshots/2-23-1625879105747.tmp testLoggerAdapter verbose: server-6a233ea6-9f29-cb26-8f90-22b24de85079: 2021-07-10T01:05:05.747Z [INFO] server-6a233ea6-9f29-cb26-8f90-22b24de85079: copied to local snapshot: bytes=71 testing.go:149: server-6a233ea6-9f29-cb26-8f90-22b24de85079: 2021-07-10T01:05:05.747Z [INFO] server-6a233ea6-9f29-cb26-8f90-22b24de85079: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-6a233ea6-9f29-cb26-8f90-22b24de85079: 2021-07-10T01:05:05.748Z [INFO] server-6a233ea6-9f29-cb26-8f90-22b24de85079: Installed remote snapshot testing.go:149: server-6a233ea6-9f29-cb26-8f90-22b24de85079: 2021-07-10T01:05:05.748Z [INFO] server-6a233ea6-9f29-cb26-8f90-22b24de85079: Installed remote snapshot testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.748Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: pipelining replication: peer="{Voter server-6a233ea6-9f29-cb26-8f90-22b24de85079 6a233ea6-9f29-cb26-8f90-22b24de85079}" testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.748Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: pipelining replication: peer="{Voter server-6a233ea6-9f29-cb26-8f90-22b24de85079 6a233ea6-9f29-cb26-8f90-22b24de85079}" testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.761Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: aborting pipeline replication: peer="{Voter server-4b63ed8c-a916-e473-8b4c-baa46c99b91b 4b63ed8c-a916-e473-8b4c-baa46c99b91b}" testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.761Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: aborting pipeline replication: peer="{Voter server-4b63ed8c-a916-e473-8b4c-baa46c99b91b 4b63ed8c-a916-e473-8b4c-baa46c99b91b}" testLoggerAdapter verbose: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.761Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: aborting pipeline replication: peer="{Voter server-6a233ea6-9f29-cb26-8f90-22b24de85079 6a233ea6-9f29-cb26-8f90-22b24de85079}" testing.go:149: server-d5df398f-3349-4628-2f92-4e243963cc61: 2021-07-10T01:05:05.761Z [INFO] server-d5df398f-3349-4628-2f92-4e243963cc61: aborting pipeline replication: peer="{Voter server-6a233ea6-9f29-cb26-8f90-22b24de85079 6a233ea6-9f29-cb26-8f90-22b24de85079}" testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.763Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8471d207-4de9-94a9-5793-b1487409cf54 Address:8471d207-4de9-94a9-5793-b1487409cf54} {Suffrage:Voter ID:server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0 Address:3ee9cfc8-34d3-969a-a04e-92671f5ff0d0} {Suffrage:Voter ID:server-f2ca7868-cf8e-471c-8066-09c8d7be878a Address:f2ca7868-cf8e-471c-8066-09c8d7be878a}]" testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.763Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8471d207-4de9-94a9-5793-b1487409cf54 Address:8471d207-4de9-94a9-5793-b1487409cf54} {Suffrage:Voter ID:server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0 Address:3ee9cfc8-34d3-969a-a04e-92671f5ff0d0} {Suffrage:Voter ID:server-f2ca7868-cf8e-471c-8066-09c8d7be878a Address:f2ca7868-cf8e-471c-8066-09c8d7be878a}]" testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.764Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: entering follower state: follower="Node at 8471d207-4de9-94a9-5793-b1487409cf54 [Follower]" leader= testLoggerAdapter verbose: server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0: 2021-07-10T01:05:05.763Z [INFO] server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8471d207-4de9-94a9-5793-b1487409cf54 Address:8471d207-4de9-94a9-5793-b1487409cf54} {Suffrage:Voter ID:server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0 Address:3ee9cfc8-34d3-969a-a04e-92671f5ff0d0} {Suffrage:Voter ID:server-f2ca7868-cf8e-471c-8066-09c8d7be878a Address:f2ca7868-cf8e-471c-8066-09c8d7be878a}]" testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.764Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: entering follower state: follower="Node at 8471d207-4de9-94a9-5793-b1487409cf54 [Follower]" leader= testing.go:149: server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0: 2021-07-10T01:05:05.763Z [INFO] server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8471d207-4de9-94a9-5793-b1487409cf54 Address:8471d207-4de9-94a9-5793-b1487409cf54} {Suffrage:Voter ID:server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0 Address:3ee9cfc8-34d3-969a-a04e-92671f5ff0d0} {Suffrage:Voter ID:server-f2ca7868-cf8e-471c-8066-09c8d7be878a Address:f2ca7868-cf8e-471c-8066-09c8d7be878a}]" testLoggerAdapter verbose: server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0: 2021-07-10T01:05:05.764Z [INFO] server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0: entering follower state: follower="Node at 3ee9cfc8-34d3-969a-a04e-92671f5ff0d0 [Follower]" leader= testing.go:149: server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0: 2021-07-10T01:05:05.764Z [INFO] server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0: entering follower state: follower="Node at 3ee9cfc8-34d3-969a-a04e-92671f5ff0d0 [Follower]" leader= testLoggerAdapter verbose: server-f2ca7868-cf8e-471c-8066-09c8d7be878a: 2021-07-10T01:05:05.764Z [INFO] server-f2ca7868-cf8e-471c-8066-09c8d7be878a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8471d207-4de9-94a9-5793-b1487409cf54 Address:8471d207-4de9-94a9-5793-b1487409cf54} {Suffrage:Voter ID:server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0 Address:3ee9cfc8-34d3-969a-a04e-92671f5ff0d0} {Suffrage:Voter ID:server-f2ca7868-cf8e-471c-8066-09c8d7be878a Address:f2ca7868-cf8e-471c-8066-09c8d7be878a}]" testing.go:149: server-f2ca7868-cf8e-471c-8066-09c8d7be878a: 2021-07-10T01:05:05.764Z [INFO] server-f2ca7868-cf8e-471c-8066-09c8d7be878a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8471d207-4de9-94a9-5793-b1487409cf54 Address:8471d207-4de9-94a9-5793-b1487409cf54} {Suffrage:Voter ID:server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0 Address:3ee9cfc8-34d3-969a-a04e-92671f5ff0d0} {Suffrage:Voter ID:server-f2ca7868-cf8e-471c-8066-09c8d7be878a Address:f2ca7868-cf8e-471c-8066-09c8d7be878a}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:05.765Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:05.765Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-f2ca7868-cf8e-471c-8066-09c8d7be878a: 2021-07-10T01:05:05.765Z [INFO] server-f2ca7868-cf8e-471c-8066-09c8d7be878a: entering follower state: follower="Node at f2ca7868-cf8e-471c-8066-09c8d7be878a [Follower]" leader= testing.go:149: server-f2ca7868-cf8e-471c-8066-09c8d7be878a: 2021-07-10T01:05:05.765Z [INFO] server-f2ca7868-cf8e-471c-8066-09c8d7be878a: entering follower state: follower="Node at f2ca7868-cf8e-471c-8066-09c8d7be878a [Follower]" leader= testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.831Z [WARN] server-8471d207-4de9-94a9-5793-b1487409cf54: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.831Z [WARN] server-8471d207-4de9-94a9-5793-b1487409cf54: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.832Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: entering candidate state: node="Node at 8471d207-4de9-94a9-5793-b1487409cf54 [Candidate]" term=2 testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.832Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: entering candidate state: node="Node at 8471d207-4de9-94a9-5793-b1487409cf54 [Candidate]" term=2 testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.832Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: election won: tally=2 testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.832Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: election won: tally=2 testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.832Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: entering leader state: leader="Node at 8471d207-4de9-94a9-5793-b1487409cf54 [Leader]" testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.832Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: entering leader state: leader="Node at 8471d207-4de9-94a9-5793-b1487409cf54 [Leader]" testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.832Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: added peer, starting replication: peer=server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0 testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.832Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: added peer, starting replication: peer=server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0 testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.832Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: added peer, starting replication: peer=server-f2ca7868-cf8e-471c-8066-09c8d7be878a testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.832Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: added peer, starting replication: peer=server-f2ca7868-cf8e-471c-8066-09c8d7be878a testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.832Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: pipelining replication: peer="{Voter server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0 3ee9cfc8-34d3-969a-a04e-92671f5ff0d0}" testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.832Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: pipelining replication: peer="{Voter server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0 3ee9cfc8-34d3-969a-a04e-92671f5ff0d0}" testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.833Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: pipelining replication: peer="{Voter server-f2ca7868-cf8e-471c-8066-09c8d7be878a f2ca7868-cf8e-471c-8066-09c8d7be878a}" testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.833Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: pipelining replication: peer="{Voter server-f2ca7868-cf8e-471c-8066-09c8d7be878a f2ca7868-cf8e-471c-8066-09c8d7be878a}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:05.937Z [INFO] stable state for Leader reached at 2021-07-10 01:05:05.832632884 +0000 UTC m=+11.220790874 (1 nodes), 67.511572ms from start of poll, 69.789296ms from cluster start. Timeout at 2021-07-10 01:05:05.937696771 +0000 UTC m=+11.325854741, 105.063867ms after stability testing.go:149: cluster: 2021-07-10T01:05:05.937Z [INFO] stable state for Leader reached at 2021-07-10 01:05:05.832632884 +0000 UTC m=+11.220790874 (1 nodes), 67.511572ms from start of poll, 69.789296ms from cluster start. Timeout at 2021-07-10 01:05:05.937696771 +0000 UTC m=+11.325854741, 105.063867ms after stability testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.938Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: starting snapshot up to: index=12 testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.938Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: starting snapshot up to: index=12 testing.go:153: 2021-07-10T01:05:05.938Z [INFO] creating new snapshot: path=/tmp/raft968864481/snapshots/2-12-1625879105938.tmp testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.939Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: snapshot complete up to: index=12 testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.939Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: snapshot complete up to: index=12 testing.go:153: 2021-07-10T01:05:05.940Z [INFO] creating new snapshot: path=/tmp/raft968864481/snapshots/2-23-1625879105940.tmp testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.940Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: copied to local snapshot: bytes=71 testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.940Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.941Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: restored user snapshot: index=1 testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.941Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: restored user snapshot: index=1 testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.941Z [ERROR] server-8471d207-4de9-94a9-5793-b1487409cf54: failed to get log: index=23 error="log not found" testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.941Z [ERROR] server-8471d207-4de9-94a9-5793-b1487409cf54: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.941Z [ERROR] server-8471d207-4de9-94a9-5793-b1487409cf54: failed to get log: index=23 error="log not found" testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.941Z [ERROR] server-8471d207-4de9-94a9-5793-b1487409cf54: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.941Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: aborting pipeline replication: peer="{Voter server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0 3ee9cfc8-34d3-969a-a04e-92671f5ff0d0}" testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.941Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: aborting pipeline replication: peer="{Voter server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0 3ee9cfc8-34d3-969a-a04e-92671f5ff0d0}" testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.941Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: aborting pipeline replication: peer="{Voter server-f2ca7868-cf8e-471c-8066-09c8d7be878a f2ca7868-cf8e-471c-8066-09c8d7be878a}" testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.941Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: aborting pipeline replication: peer="{Voter server-f2ca7868-cf8e-471c-8066-09c8d7be878a f2ca7868-cf8e-471c-8066-09c8d7be878a}" testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.949Z [ERROR] server-8471d207-4de9-94a9-5793-b1487409cf54: failed to get log: index=23 error="log not found" testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.949Z [ERROR] server-8471d207-4de9-94a9-5793-b1487409cf54: failed to get log: index=23 error="log not found" testing.go:153: 2021-07-10T01:05:05.950Z [INFO] creating new snapshot: path=/tmp/raft494882213/snapshots/2-23-1625879105950.tmp testLoggerAdapter verbose: server-f2ca7868-cf8e-471c-8066-09c8d7be878a: 2021-07-10T01:05:05.951Z [INFO] server-f2ca7868-cf8e-471c-8066-09c8d7be878a: copied to local snapshot: bytes=71 testing.go:149: server-f2ca7868-cf8e-471c-8066-09c8d7be878a: 2021-07-10T01:05:05.951Z [INFO] server-f2ca7868-cf8e-471c-8066-09c8d7be878a: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.951Z [ERROR] server-8471d207-4de9-94a9-5793-b1487409cf54: failed to get log: index=23 error="log not found" testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.951Z [ERROR] server-8471d207-4de9-94a9-5793-b1487409cf54: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-f2ca7868-cf8e-471c-8066-09c8d7be878a: 2021-07-10T01:05:05.951Z [INFO] server-f2ca7868-cf8e-471c-8066-09c8d7be878a: Installed remote snapshot testing.go:149: server-f2ca7868-cf8e-471c-8066-09c8d7be878a: 2021-07-10T01:05:05.951Z [INFO] server-f2ca7868-cf8e-471c-8066-09c8d7be878a: Installed remote snapshot testLoggerAdapter verbose: server-f2ca7868-cf8e-471c-8066-09c8d7be878a: 2021-07-10T01:05:05.951Z [WARN] server-f2ca7868-cf8e-471c-8066-09c8d7be878a: skipping application of old log: index=22 testing.go:149: server-f2ca7868-cf8e-471c-8066-09c8d7be878a: 2021-07-10T01:05:05.951Z [WARN] server-f2ca7868-cf8e-471c-8066-09c8d7be878a: skipping application of old log: index=22 testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.951Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: pipelining replication: peer="{Voter server-f2ca7868-cf8e-471c-8066-09c8d7be878a f2ca7868-cf8e-471c-8066-09c8d7be878a}" testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.951Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: pipelining replication: peer="{Voter server-f2ca7868-cf8e-471c-8066-09c8d7be878a f2ca7868-cf8e-471c-8066-09c8d7be878a}" testing.go:153: 2021-07-10T01:05:05.951Z [INFO] creating new snapshot: path=/tmp/raft740853691/snapshots/2-23-1625879105951.tmp testLoggerAdapter verbose: server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0: 2021-07-10T01:05:05.952Z [INFO] server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0: copied to local snapshot: bytes=71 testing.go:149: server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0: 2021-07-10T01:05:05.952Z [INFO] server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0: 2021-07-10T01:05:05.952Z [INFO] server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0: Installed remote snapshot testing.go:149: server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0: 2021-07-10T01:05:05.952Z [INFO] server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0: Installed remote snapshot testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.952Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: pipelining replication: peer="{Voter server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0 3ee9cfc8-34d3-969a-a04e-92671f5ff0d0}" testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.952Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: pipelining replication: peer="{Voter server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0 3ee9cfc8-34d3-969a-a04e-92671f5ff0d0}" testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.967Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: aborting pipeline replication: peer="{Voter server-f2ca7868-cf8e-471c-8066-09c8d7be878a f2ca7868-cf8e-471c-8066-09c8d7be878a}" testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.967Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: aborting pipeline replication: peer="{Voter server-f2ca7868-cf8e-471c-8066-09c8d7be878a f2ca7868-cf8e-471c-8066-09c8d7be878a}" testLoggerAdapter verbose: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.967Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: aborting pipeline replication: peer="{Voter server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0 3ee9cfc8-34d3-969a-a04e-92671f5ff0d0}" testing.go:149: server-8471d207-4de9-94a9-5793-b1487409cf54: 2021-07-10T01:05:05.967Z [INFO] server-8471d207-4de9-94a9-5793-b1487409cf54: aborting pipeline replication: peer="{Voter server-3ee9cfc8-34d3-969a-a04e-92671f5ff0d0 3ee9cfc8-34d3-969a-a04e-92671f5ff0d0}" testLoggerAdapter verbose: server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: 2021-07-10T01:05:05.969Z [INFO] server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-48eacc89-1ba5-1136-f5b7-64accdcd20c6 Address:48eacc89-1ba5-1136-f5b7-64accdcd20c6} {Suffrage:Voter ID:server-ca2f2293-543c-d2e7-acdd-df93c434c65b Address:ca2f2293-543c-d2e7-acdd-df93c434c65b} {Suffrage:Voter ID:server-7d0636ee-f718-97e6-3e8b-077575c1f1fb Address:7d0636ee-f718-97e6-3e8b-077575c1f1fb}]" testing.go:149: server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: 2021-07-10T01:05:05.969Z [INFO] server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-48eacc89-1ba5-1136-f5b7-64accdcd20c6 Address:48eacc89-1ba5-1136-f5b7-64accdcd20c6} {Suffrage:Voter ID:server-ca2f2293-543c-d2e7-acdd-df93c434c65b Address:ca2f2293-543c-d2e7-acdd-df93c434c65b} {Suffrage:Voter ID:server-7d0636ee-f718-97e6-3e8b-077575c1f1fb Address:7d0636ee-f718-97e6-3e8b-077575c1f1fb}]" testLoggerAdapter verbose: server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: 2021-07-10T01:05:05.969Z [INFO] server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: entering follower state: follower="Node at 48eacc89-1ba5-1136-f5b7-64accdcd20c6 [Follower]" leader= testing.go:149: server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: 2021-07-10T01:05:05.969Z [INFO] server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: entering follower state: follower="Node at 48eacc89-1ba5-1136-f5b7-64accdcd20c6 [Follower]" leader= testLoggerAdapter verbose: server-ca2f2293-543c-d2e7-acdd-df93c434c65b: 2021-07-10T01:05:05.969Z [INFO] server-ca2f2293-543c-d2e7-acdd-df93c434c65b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-48eacc89-1ba5-1136-f5b7-64accdcd20c6 Address:48eacc89-1ba5-1136-f5b7-64accdcd20c6} {Suffrage:Voter ID:server-ca2f2293-543c-d2e7-acdd-df93c434c65b Address:ca2f2293-543c-d2e7-acdd-df93c434c65b} {Suffrage:Voter ID:server-7d0636ee-f718-97e6-3e8b-077575c1f1fb Address:7d0636ee-f718-97e6-3e8b-077575c1f1fb}]" testing.go:149: server-ca2f2293-543c-d2e7-acdd-df93c434c65b: 2021-07-10T01:05:05.969Z [INFO] server-ca2f2293-543c-d2e7-acdd-df93c434c65b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-48eacc89-1ba5-1136-f5b7-64accdcd20c6 Address:48eacc89-1ba5-1136-f5b7-64accdcd20c6} {Suffrage:Voter ID:server-ca2f2293-543c-d2e7-acdd-df93c434c65b Address:ca2f2293-543c-d2e7-acdd-df93c434c65b} {Suffrage:Voter ID:server-7d0636ee-f718-97e6-3e8b-077575c1f1fb Address:7d0636ee-f718-97e6-3e8b-077575c1f1fb}]" testLoggerAdapter verbose: server-ca2f2293-543c-d2e7-acdd-df93c434c65b: 2021-07-10T01:05:05.969Z [INFO] server-ca2f2293-543c-d2e7-acdd-df93c434c65b: entering follower state: follower="Node at ca2f2293-543c-d2e7-acdd-df93c434c65b [Follower]" leader= testing.go:149: server-ca2f2293-543c-d2e7-acdd-df93c434c65b: 2021-07-10T01:05:05.969Z [INFO] server-ca2f2293-543c-d2e7-acdd-df93c434c65b: entering follower state: follower="Node at ca2f2293-543c-d2e7-acdd-df93c434c65b [Follower]" leader= testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:05.969Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-48eacc89-1ba5-1136-f5b7-64accdcd20c6 Address:48eacc89-1ba5-1136-f5b7-64accdcd20c6} {Suffrage:Voter ID:server-ca2f2293-543c-d2e7-acdd-df93c434c65b Address:ca2f2293-543c-d2e7-acdd-df93c434c65b} {Suffrage:Voter ID:server-7d0636ee-f718-97e6-3e8b-077575c1f1fb Address:7d0636ee-f718-97e6-3e8b-077575c1f1fb}]" testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:05.969Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-48eacc89-1ba5-1136-f5b7-64accdcd20c6 Address:48eacc89-1ba5-1136-f5b7-64accdcd20c6} {Suffrage:Voter ID:server-ca2f2293-543c-d2e7-acdd-df93c434c65b Address:ca2f2293-543c-d2e7-acdd-df93c434c65b} {Suffrage:Voter ID:server-7d0636ee-f718-97e6-3e8b-077575c1f1fb Address:7d0636ee-f718-97e6-3e8b-077575c1f1fb}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:05.969Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:05.969Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:05.969Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: entering follower state: follower="Node at 7d0636ee-f718-97e6-3e8b-077575c1f1fb [Follower]" leader= testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:05.969Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: entering follower state: follower="Node at 7d0636ee-f718-97e6-3e8b-077575c1f1fb [Follower]" leader= testLoggerAdapter verbose: server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: 2021-07-10T01:05:06.045Z [WARN] server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: 2021-07-10T01:05:06.045Z [WARN] server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.046Z [WARN] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.046Z [WARN] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.046Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: entering candidate state: node="Node at 7d0636ee-f718-97e6-3e8b-077575c1f1fb [Candidate]" term=2 testLoggerAdapter verbose: server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: 2021-07-10T01:05:06.046Z [INFO] server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: entering candidate state: node="Node at 48eacc89-1ba5-1136-f5b7-64accdcd20c6 [Candidate]" term=2 testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.046Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: entering candidate state: node="Node at 7d0636ee-f718-97e6-3e8b-077575c1f1fb [Candidate]" term=2 testing.go:149: server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: 2021-07-10T01:05:06.046Z [INFO] server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: entering candidate state: node="Node at 48eacc89-1ba5-1136-f5b7-64accdcd20c6 [Candidate]" term=2 testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.046Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: election won: tally=2 testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.046Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: election won: tally=2 testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.046Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: entering leader state: leader="Node at 7d0636ee-f718-97e6-3e8b-077575c1f1fb [Leader]" testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.046Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: entering leader state: leader="Node at 7d0636ee-f718-97e6-3e8b-077575c1f1fb [Leader]" testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.046Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: added peer, starting replication: peer=server-48eacc89-1ba5-1136-f5b7-64accdcd20c6 testLoggerAdapter verbose: server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: 2021-07-10T01:05:06.046Z [INFO] server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: duplicate requestVote for same term: term=2 testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.046Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: added peer, starting replication: peer=server-48eacc89-1ba5-1136-f5b7-64accdcd20c6 testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.046Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: added peer, starting replication: peer=server-ca2f2293-543c-d2e7-acdd-df93c434c65b testLoggerAdapter verbose: server-ca2f2293-543c-d2e7-acdd-df93c434c65b: 2021-07-10T01:05:06.046Z [INFO] server-ca2f2293-543c-d2e7-acdd-df93c434c65b: duplicate requestVote for same term: term=2 testing.go:149: server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: 2021-07-10T01:05:06.046Z [INFO] server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: duplicate requestVote for same term: term=2 testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.046Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: added peer, starting replication: peer=server-ca2f2293-543c-d2e7-acdd-df93c434c65b testLoggerAdapter verbose: server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: 2021-07-10T01:05:06.046Z [INFO] server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: entering follower state: follower="Node at 48eacc89-1ba5-1136-f5b7-64accdcd20c6 [Follower]" leader=7d0636ee-f718-97e6-3e8b-077575c1f1fb testing.go:149: server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: 2021-07-10T01:05:06.046Z [INFO] server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: entering follower state: follower="Node at 48eacc89-1ba5-1136-f5b7-64accdcd20c6 [Follower]" leader=7d0636ee-f718-97e6-3e8b-077575c1f1fb testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.046Z [WARN] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: rejecting vote request since we have a leader: from=48eacc89-1ba5-1136-f5b7-64accdcd20c6 leader=7d0636ee-f718-97e6-3e8b-077575c1f1fb testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.046Z [WARN] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: rejecting vote request since we have a leader: from=48eacc89-1ba5-1136-f5b7-64accdcd20c6 leader=7d0636ee-f718-97e6-3e8b-077575c1f1fb testing.go:149: server-ca2f2293-543c-d2e7-acdd-df93c434c65b: 2021-07-10T01:05:06.046Z [INFO] server-ca2f2293-543c-d2e7-acdd-df93c434c65b: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.046Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: pipelining replication: peer="{Voter server-48eacc89-1ba5-1136-f5b7-64accdcd20c6 48eacc89-1ba5-1136-f5b7-64accdcd20c6}" testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.046Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: pipelining replication: peer="{Voter server-48eacc89-1ba5-1136-f5b7-64accdcd20c6 48eacc89-1ba5-1136-f5b7-64accdcd20c6}" testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.046Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: pipelining replication: peer="{Voter server-ca2f2293-543c-d2e7-acdd-df93c434c65b ca2f2293-543c-d2e7-acdd-df93c434c65b}" testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.046Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: pipelining replication: peer="{Voter server-ca2f2293-543c-d2e7-acdd-df93c434c65b ca2f2293-543c-d2e7-acdd-df93c434c65b}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:06.151Z [INFO] stable state for Leader reached at 2021-07-10 01:05:06.04671056 +0000 UTC m=+11.434868530 (1 nodes), 76.718808ms from start of poll, 77.650498ms from cluster start. Timeout at 2021-07-10 01:05:06.151791588 +0000 UTC m=+11.539949558, 105.081028ms after stability testing.go:149: cluster: 2021-07-10T01:05:06.151Z [INFO] stable state for Leader reached at 2021-07-10 01:05:06.04671056 +0000 UTC m=+11.434868530 (1 nodes), 76.718808ms from start of poll, 77.650498ms from cluster start. Timeout at 2021-07-10 01:05:06.151791588 +0000 UTC m=+11.539949558, 105.081028ms after stability testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.152Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: starting snapshot up to: index=12 testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.152Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: starting snapshot up to: index=12 testing.go:153: 2021-07-10T01:05:06.152Z [INFO] creating new snapshot: path=/tmp/raft014180035/snapshots/2-12-1625879106152.tmp testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.153Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: snapshot complete up to: index=12 testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.153Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: snapshot complete up to: index=12 testing.go:153: 2021-07-10T01:05:06.154Z [INFO] creating new snapshot: path=/tmp/raft014180035/snapshots/2-113-1625879106154.tmp testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.155Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: copied to local snapshot: bytes=71 testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.155Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.155Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: restored user snapshot: index=1 testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.155Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: restored user snapshot: index=1 testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.155Z [ERROR] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: failed to get log: index=23 error="log not found" testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.155Z [ERROR] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.155Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: aborting pipeline replication: peer="{Voter server-ca2f2293-543c-d2e7-acdd-df93c434c65b ca2f2293-543c-d2e7-acdd-df93c434c65b}" testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.155Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: aborting pipeline replication: peer="{Voter server-ca2f2293-543c-d2e7-acdd-df93c434c65b ca2f2293-543c-d2e7-acdd-df93c434c65b}" testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.155Z [ERROR] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: failed to get log: index=23 error="log not found" testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.155Z [ERROR] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.155Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: aborting pipeline replication: peer="{Voter server-48eacc89-1ba5-1136-f5b7-64accdcd20c6 48eacc89-1ba5-1136-f5b7-64accdcd20c6}" testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.155Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: aborting pipeline replication: peer="{Voter server-48eacc89-1ba5-1136-f5b7-64accdcd20c6 48eacc89-1ba5-1136-f5b7-64accdcd20c6}" testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.161Z [ERROR] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: failed to get log: index=23 error="log not found" testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.161Z [ERROR] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.161Z [ERROR] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: failed to get log: index=23 error="log not found" testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.161Z [ERROR] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: failed to get log: index=23 error="log not found" testing.go:153: 2021-07-10T01:05:06.161Z [INFO] creating new snapshot: path=/tmp/raft046324393/snapshots/2-113-1625879106161.tmp testing.go:153: 2021-07-10T01:05:06.162Z [INFO] creating new snapshot: path=/tmp/raft434071327/snapshots/2-113-1625879106161.tmp testLoggerAdapter verbose: server-ca2f2293-543c-d2e7-acdd-df93c434c65b: 2021-07-10T01:05:06.162Z [INFO] server-ca2f2293-543c-d2e7-acdd-df93c434c65b: copied to local snapshot: bytes=71 testing.go:149: server-ca2f2293-543c-d2e7-acdd-df93c434c65b: 2021-07-10T01:05:06.162Z [INFO] server-ca2f2293-543c-d2e7-acdd-df93c434c65b: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: 2021-07-10T01:05:06.163Z [INFO] server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-ca2f2293-543c-d2e7-acdd-df93c434c65b: 2021-07-10T01:05:06.163Z [INFO] server-ca2f2293-543c-d2e7-acdd-df93c434c65b: Installed remote snapshot testing.go:149: server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: 2021-07-10T01:05:06.163Z [INFO] server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: copied to local snapshot: bytes=71 testing.go:149: server-ca2f2293-543c-d2e7-acdd-df93c434c65b: 2021-07-10T01:05:06.163Z [INFO] server-ca2f2293-543c-d2e7-acdd-df93c434c65b: Installed remote snapshot testLoggerAdapter verbose: server-ca2f2293-543c-d2e7-acdd-df93c434c65b: 2021-07-10T01:05:06.164Z [WARN] server-ca2f2293-543c-d2e7-acdd-df93c434c65b: skipping application of old log: index=22 testing.go:149: server-ca2f2293-543c-d2e7-acdd-df93c434c65b: 2021-07-10T01:05:06.164Z [WARN] server-ca2f2293-543c-d2e7-acdd-df93c434c65b: skipping application of old log: index=22 testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.164Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: pipelining replication: peer="{Voter server-ca2f2293-543c-d2e7-acdd-df93c434c65b ca2f2293-543c-d2e7-acdd-df93c434c65b}" testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.164Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: pipelining replication: peer="{Voter server-ca2f2293-543c-d2e7-acdd-df93c434c65b ca2f2293-543c-d2e7-acdd-df93c434c65b}" testLoggerAdapter verbose: server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: 2021-07-10T01:05:06.164Z [INFO] server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: Installed remote snapshot testing.go:149: server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: 2021-07-10T01:05:06.164Z [INFO] server-48eacc89-1ba5-1136-f5b7-64accdcd20c6: Installed remote snapshot testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.164Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: pipelining replication: peer="{Voter server-48eacc89-1ba5-1136-f5b7-64accdcd20c6 48eacc89-1ba5-1136-f5b7-64accdcd20c6}" testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.164Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: pipelining replication: peer="{Voter server-48eacc89-1ba5-1136-f5b7-64accdcd20c6 48eacc89-1ba5-1136-f5b7-64accdcd20c6}" testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.175Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: aborting pipeline replication: peer="{Voter server-ca2f2293-543c-d2e7-acdd-df93c434c65b ca2f2293-543c-d2e7-acdd-df93c434c65b}" testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.175Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: aborting pipeline replication: peer="{Voter server-ca2f2293-543c-d2e7-acdd-df93c434c65b ca2f2293-543c-d2e7-acdd-df93c434c65b}" testLoggerAdapter verbose: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.175Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: aborting pipeline replication: peer="{Voter server-48eacc89-1ba5-1136-f5b7-64accdcd20c6 48eacc89-1ba5-1136-f5b7-64accdcd20c6}" testing.go:149: server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: 2021-07-10T01:05:06.175Z [INFO] server-7d0636ee-f718-97e6-3e8b-077575c1f1fb: aborting pipeline replication: peer="{Voter server-48eacc89-1ba5-1136-f5b7-64accdcd20c6 48eacc89-1ba5-1136-f5b7-64accdcd20c6}" testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.177Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8516e85f-e82e-9856-b696-b2add172e417 Address:8516e85f-e82e-9856-b696-b2add172e417} {Suffrage:Voter ID:server-186970eb-04e2-8b42-4a61-7c4501527a25 Address:186970eb-04e2-8b42-4a61-7c4501527a25} {Suffrage:Voter ID:server-79f52308-c464-7454-e3f1-a5714823a1b1 Address:79f52308-c464-7454-e3f1-a5714823a1b1}]" testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.177Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8516e85f-e82e-9856-b696-b2add172e417 Address:8516e85f-e82e-9856-b696-b2add172e417} {Suffrage:Voter ID:server-186970eb-04e2-8b42-4a61-7c4501527a25 Address:186970eb-04e2-8b42-4a61-7c4501527a25} {Suffrage:Voter ID:server-79f52308-c464-7454-e3f1-a5714823a1b1 Address:79f52308-c464-7454-e3f1-a5714823a1b1}]" testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.177Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: entering follower state: follower="Node at 8516e85f-e82e-9856-b696-b2add172e417 [Follower]" leader= testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.177Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: entering follower state: follower="Node at 8516e85f-e82e-9856-b696-b2add172e417 [Follower]" leader= testLoggerAdapter verbose: server-186970eb-04e2-8b42-4a61-7c4501527a25: 2021-07-10T01:05:06.177Z [INFO] server-186970eb-04e2-8b42-4a61-7c4501527a25: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8516e85f-e82e-9856-b696-b2add172e417 Address:8516e85f-e82e-9856-b696-b2add172e417} {Suffrage:Voter ID:server-186970eb-04e2-8b42-4a61-7c4501527a25 Address:186970eb-04e2-8b42-4a61-7c4501527a25} {Suffrage:Voter ID:server-79f52308-c464-7454-e3f1-a5714823a1b1 Address:79f52308-c464-7454-e3f1-a5714823a1b1}]" testing.go:149: server-186970eb-04e2-8b42-4a61-7c4501527a25: 2021-07-10T01:05:06.177Z [INFO] server-186970eb-04e2-8b42-4a61-7c4501527a25: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8516e85f-e82e-9856-b696-b2add172e417 Address:8516e85f-e82e-9856-b696-b2add172e417} {Suffrage:Voter ID:server-186970eb-04e2-8b42-4a61-7c4501527a25 Address:186970eb-04e2-8b42-4a61-7c4501527a25} {Suffrage:Voter ID:server-79f52308-c464-7454-e3f1-a5714823a1b1 Address:79f52308-c464-7454-e3f1-a5714823a1b1}]" testLoggerAdapter verbose: server-186970eb-04e2-8b42-4a61-7c4501527a25: 2021-07-10T01:05:06.177Z [INFO] server-186970eb-04e2-8b42-4a61-7c4501527a25: entering follower state: follower="Node at 186970eb-04e2-8b42-4a61-7c4501527a25 [Follower]" leader= testing.go:149: server-186970eb-04e2-8b42-4a61-7c4501527a25: 2021-07-10T01:05:06.177Z [INFO] server-186970eb-04e2-8b42-4a61-7c4501527a25: entering follower state: follower="Node at 186970eb-04e2-8b42-4a61-7c4501527a25 [Follower]" leader= testLoggerAdapter verbose: server-79f52308-c464-7454-e3f1-a5714823a1b1: 2021-07-10T01:05:06.177Z [INFO] server-79f52308-c464-7454-e3f1-a5714823a1b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8516e85f-e82e-9856-b696-b2add172e417 Address:8516e85f-e82e-9856-b696-b2add172e417} {Suffrage:Voter ID:server-186970eb-04e2-8b42-4a61-7c4501527a25 Address:186970eb-04e2-8b42-4a61-7c4501527a25} {Suffrage:Voter ID:server-79f52308-c464-7454-e3f1-a5714823a1b1 Address:79f52308-c464-7454-e3f1-a5714823a1b1}]" testing.go:149: server-79f52308-c464-7454-e3f1-a5714823a1b1: 2021-07-10T01:05:06.177Z [INFO] server-79f52308-c464-7454-e3f1-a5714823a1b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8516e85f-e82e-9856-b696-b2add172e417 Address:8516e85f-e82e-9856-b696-b2add172e417} {Suffrage:Voter ID:server-186970eb-04e2-8b42-4a61-7c4501527a25 Address:186970eb-04e2-8b42-4a61-7c4501527a25} {Suffrage:Voter ID:server-79f52308-c464-7454-e3f1-a5714823a1b1 Address:79f52308-c464-7454-e3f1-a5714823a1b1}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:06.177Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:06.177Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-79f52308-c464-7454-e3f1-a5714823a1b1: 2021-07-10T01:05:06.177Z [INFO] server-79f52308-c464-7454-e3f1-a5714823a1b1: entering follower state: follower="Node at 79f52308-c464-7454-e3f1-a5714823a1b1 [Follower]" leader= testing.go:149: server-79f52308-c464-7454-e3f1-a5714823a1b1: 2021-07-10T01:05:06.177Z [INFO] server-79f52308-c464-7454-e3f1-a5714823a1b1: entering follower state: follower="Node at 79f52308-c464-7454-e3f1-a5714823a1b1 [Follower]" leader= testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.239Z [WARN] server-8516e85f-e82e-9856-b696-b2add172e417: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.239Z [WARN] server-8516e85f-e82e-9856-b696-b2add172e417: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.239Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: entering candidate state: node="Node at 8516e85f-e82e-9856-b696-b2add172e417 [Candidate]" term=2 testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.239Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: entering candidate state: node="Node at 8516e85f-e82e-9856-b696-b2add172e417 [Candidate]" term=2 testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.240Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: election won: tally=2 testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.240Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: election won: tally=2 testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.240Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: entering leader state: leader="Node at 8516e85f-e82e-9856-b696-b2add172e417 [Leader]" testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.240Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: entering leader state: leader="Node at 8516e85f-e82e-9856-b696-b2add172e417 [Leader]" testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.240Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: added peer, starting replication: peer=server-186970eb-04e2-8b42-4a61-7c4501527a25 testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.240Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: added peer, starting replication: peer=server-186970eb-04e2-8b42-4a61-7c4501527a25 testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.240Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: added peer, starting replication: peer=server-79f52308-c464-7454-e3f1-a5714823a1b1 testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.240Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: added peer, starting replication: peer=server-79f52308-c464-7454-e3f1-a5714823a1b1 testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.240Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: pipelining replication: peer="{Voter server-186970eb-04e2-8b42-4a61-7c4501527a25 186970eb-04e2-8b42-4a61-7c4501527a25}" testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.240Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: pipelining replication: peer="{Voter server-186970eb-04e2-8b42-4a61-7c4501527a25 186970eb-04e2-8b42-4a61-7c4501527a25}" testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.240Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: pipelining replication: peer="{Voter server-79f52308-c464-7454-e3f1-a5714823a1b1 79f52308-c464-7454-e3f1-a5714823a1b1}" testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.240Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: pipelining replication: peer="{Voter server-79f52308-c464-7454-e3f1-a5714823a1b1 79f52308-c464-7454-e3f1-a5714823a1b1}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:06.345Z [INFO] stable state for Leader reached at 2021-07-10 01:05:06.240317321 +0000 UTC m=+11.628475311 (1 nodes), 62.323917ms from start of poll, 63.189406ms from cluster start. Timeout at 2021-07-10 01:05:06.345397809 +0000 UTC m=+11.733555799, 105.080488ms after stability testing.go:149: cluster: 2021-07-10T01:05:06.345Z [INFO] stable state for Leader reached at 2021-07-10 01:05:06.240317321 +0000 UTC m=+11.628475311 (1 nodes), 62.323917ms from start of poll, 63.189406ms from cluster start. Timeout at 2021-07-10 01:05:06.345397809 +0000 UTC m=+11.733555799, 105.080488ms after stability testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.346Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: starting snapshot up to: index=12 testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.346Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: starting snapshot up to: index=12 testing.go:153: 2021-07-10T01:05:06.346Z [INFO] creating new snapshot: path=/tmp/raft728465389/snapshots/2-12-1625879106346.tmp testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.346Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: snapshot complete up to: index=12 testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.346Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: snapshot complete up to: index=12 testing.go:153: 2021-07-10T01:05:06.347Z [INFO] creating new snapshot: path=/tmp/raft728465389/snapshots/2-1013-1625879106347.tmp testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.348Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: copied to local snapshot: bytes=71 testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.348Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.348Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: restored user snapshot: index=1 testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.348Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: restored user snapshot: index=1 testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.348Z [ERROR] server-8516e85f-e82e-9856-b696-b2add172e417: failed to get log: index=23 error="log not found" testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.348Z [ERROR] server-8516e85f-e82e-9856-b696-b2add172e417: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.348Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: aborting pipeline replication: peer="{Voter server-79f52308-c464-7454-e3f1-a5714823a1b1 79f52308-c464-7454-e3f1-a5714823a1b1}" testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.348Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: aborting pipeline replication: peer="{Voter server-79f52308-c464-7454-e3f1-a5714823a1b1 79f52308-c464-7454-e3f1-a5714823a1b1}" testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.348Z [ERROR] server-8516e85f-e82e-9856-b696-b2add172e417: failed to get log: index=23 error="log not found" testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.348Z [ERROR] server-8516e85f-e82e-9856-b696-b2add172e417: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.349Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: aborting pipeline replication: peer="{Voter server-186970eb-04e2-8b42-4a61-7c4501527a25 186970eb-04e2-8b42-4a61-7c4501527a25}" testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.349Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: aborting pipeline replication: peer="{Voter server-186970eb-04e2-8b42-4a61-7c4501527a25 186970eb-04e2-8b42-4a61-7c4501527a25}" testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.357Z [ERROR] server-8516e85f-e82e-9856-b696-b2add172e417: failed to get log: index=23 error="log not found" testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.357Z [ERROR] server-8516e85f-e82e-9856-b696-b2add172e417: failed to get log: index=23 error="log not found" testing.go:153: 2021-07-10T01:05:06.357Z [INFO] creating new snapshot: path=/tmp/raft256942961/snapshots/2-1013-1625879106357.tmp testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.357Z [ERROR] server-8516e85f-e82e-9856-b696-b2add172e417: failed to get log: index=23 error="log not found" testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.357Z [ERROR] server-8516e85f-e82e-9856-b696-b2add172e417: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-79f52308-c464-7454-e3f1-a5714823a1b1: 2021-07-10T01:05:06.358Z [INFO] server-79f52308-c464-7454-e3f1-a5714823a1b1: copied to local snapshot: bytes=71 testing.go:149: server-79f52308-c464-7454-e3f1-a5714823a1b1: 2021-07-10T01:05:06.358Z [INFO] server-79f52308-c464-7454-e3f1-a5714823a1b1: copied to local snapshot: bytes=71 testing.go:153: 2021-07-10T01:05:06.358Z [INFO] creating new snapshot: path=/tmp/raft659057319/snapshots/2-1013-1625879106358.tmp testLoggerAdapter verbose: server-79f52308-c464-7454-e3f1-a5714823a1b1: 2021-07-10T01:05:06.358Z [INFO] server-79f52308-c464-7454-e3f1-a5714823a1b1: Installed remote snapshot testing.go:149: server-79f52308-c464-7454-e3f1-a5714823a1b1: 2021-07-10T01:05:06.358Z [INFO] server-79f52308-c464-7454-e3f1-a5714823a1b1: Installed remote snapshot testLoggerAdapter verbose: server-79f52308-c464-7454-e3f1-a5714823a1b1: 2021-07-10T01:05:06.358Z [WARN] server-79f52308-c464-7454-e3f1-a5714823a1b1: skipping application of old log: index=22 testing.go:149: server-79f52308-c464-7454-e3f1-a5714823a1b1: 2021-07-10T01:05:06.358Z [WARN] server-79f52308-c464-7454-e3f1-a5714823a1b1: skipping application of old log: index=22 testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.358Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: pipelining replication: peer="{Voter server-79f52308-c464-7454-e3f1-a5714823a1b1 79f52308-c464-7454-e3f1-a5714823a1b1}" testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.358Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: pipelining replication: peer="{Voter server-79f52308-c464-7454-e3f1-a5714823a1b1 79f52308-c464-7454-e3f1-a5714823a1b1}" testLoggerAdapter verbose: server-186970eb-04e2-8b42-4a61-7c4501527a25: 2021-07-10T01:05:06.359Z [INFO] server-186970eb-04e2-8b42-4a61-7c4501527a25: copied to local snapshot: bytes=71 testing.go:149: server-186970eb-04e2-8b42-4a61-7c4501527a25: 2021-07-10T01:05:06.359Z [INFO] server-186970eb-04e2-8b42-4a61-7c4501527a25: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-186970eb-04e2-8b42-4a61-7c4501527a25: 2021-07-10T01:05:06.359Z [INFO] server-186970eb-04e2-8b42-4a61-7c4501527a25: Installed remote snapshot testing.go:149: server-186970eb-04e2-8b42-4a61-7c4501527a25: 2021-07-10T01:05:06.359Z [INFO] server-186970eb-04e2-8b42-4a61-7c4501527a25: Installed remote snapshot testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.359Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: pipelining replication: peer="{Voter server-186970eb-04e2-8b42-4a61-7c4501527a25 186970eb-04e2-8b42-4a61-7c4501527a25}" testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.359Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: pipelining replication: peer="{Voter server-186970eb-04e2-8b42-4a61-7c4501527a25 186970eb-04e2-8b42-4a61-7c4501527a25}" testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.374Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: aborting pipeline replication: peer="{Voter server-79f52308-c464-7454-e3f1-a5714823a1b1 79f52308-c464-7454-e3f1-a5714823a1b1}" testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.374Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: aborting pipeline replication: peer="{Voter server-79f52308-c464-7454-e3f1-a5714823a1b1 79f52308-c464-7454-e3f1-a5714823a1b1}" testLoggerAdapter verbose: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.374Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: aborting pipeline replication: peer="{Voter server-186970eb-04e2-8b42-4a61-7c4501527a25 186970eb-04e2-8b42-4a61-7c4501527a25}" testing.go:149: server-8516e85f-e82e-9856-b696-b2add172e417: 2021-07-10T01:05:06.374Z [INFO] server-8516e85f-e82e-9856-b696-b2add172e417: aborting pipeline replication: peer="{Voter server-186970eb-04e2-8b42-4a61-7c4501527a25 186970eb-04e2-8b42-4a61-7c4501527a25}" testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.376Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8 Address:4b242baa-3ccf-bf7a-add2-3bc14bba42b8} {Suffrage:Voter ID:server-c04fd653-66f5-6ead-e982-aafeff94b195 Address:c04fd653-66f5-6ead-e982-aafeff94b195} {Suffrage:Voter ID:server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2 Address:7e2b8bf7-5150-d051-2dbd-f464df97f8a2}]" testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.376Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8 Address:4b242baa-3ccf-bf7a-add2-3bc14bba42b8} {Suffrage:Voter ID:server-c04fd653-66f5-6ead-e982-aafeff94b195 Address:c04fd653-66f5-6ead-e982-aafeff94b195} {Suffrage:Voter ID:server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2 Address:7e2b8bf7-5150-d051-2dbd-f464df97f8a2}]" testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.376Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: entering follower state: follower="Node at 4b242baa-3ccf-bf7a-add2-3bc14bba42b8 [Follower]" leader= testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.376Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: entering follower state: follower="Node at 4b242baa-3ccf-bf7a-add2-3bc14bba42b8 [Follower]" leader= testLoggerAdapter verbose: server-c04fd653-66f5-6ead-e982-aafeff94b195: 2021-07-10T01:05:06.376Z [INFO] server-c04fd653-66f5-6ead-e982-aafeff94b195: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8 Address:4b242baa-3ccf-bf7a-add2-3bc14bba42b8} {Suffrage:Voter ID:server-c04fd653-66f5-6ead-e982-aafeff94b195 Address:c04fd653-66f5-6ead-e982-aafeff94b195} {Suffrage:Voter ID:server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2 Address:7e2b8bf7-5150-d051-2dbd-f464df97f8a2}]" testing.go:149: server-c04fd653-66f5-6ead-e982-aafeff94b195: 2021-07-10T01:05:06.376Z [INFO] server-c04fd653-66f5-6ead-e982-aafeff94b195: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8 Address:4b242baa-3ccf-bf7a-add2-3bc14bba42b8} {Suffrage:Voter ID:server-c04fd653-66f5-6ead-e982-aafeff94b195 Address:c04fd653-66f5-6ead-e982-aafeff94b195} {Suffrage:Voter ID:server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2 Address:7e2b8bf7-5150-d051-2dbd-f464df97f8a2}]" testLoggerAdapter verbose: server-c04fd653-66f5-6ead-e982-aafeff94b195: 2021-07-10T01:05:06.376Z [INFO] server-c04fd653-66f5-6ead-e982-aafeff94b195: entering follower state: follower="Node at c04fd653-66f5-6ead-e982-aafeff94b195 [Follower]" leader= testing.go:149: server-c04fd653-66f5-6ead-e982-aafeff94b195: 2021-07-10T01:05:06.376Z [INFO] server-c04fd653-66f5-6ead-e982-aafeff94b195: entering follower state: follower="Node at c04fd653-66f5-6ead-e982-aafeff94b195 [Follower]" leader= testLoggerAdapter verbose: server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: 2021-07-10T01:05:06.376Z [INFO] server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8 Address:4b242baa-3ccf-bf7a-add2-3bc14bba42b8} {Suffrage:Voter ID:server-c04fd653-66f5-6ead-e982-aafeff94b195 Address:c04fd653-66f5-6ead-e982-aafeff94b195} {Suffrage:Voter ID:server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2 Address:7e2b8bf7-5150-d051-2dbd-f464df97f8a2}]" testing.go:149: server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: 2021-07-10T01:05:06.376Z [INFO] server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8 Address:4b242baa-3ccf-bf7a-add2-3bc14bba42b8} {Suffrage:Voter ID:server-c04fd653-66f5-6ead-e982-aafeff94b195 Address:c04fd653-66f5-6ead-e982-aafeff94b195} {Suffrage:Voter ID:server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2 Address:7e2b8bf7-5150-d051-2dbd-f464df97f8a2}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:06.377Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:06.377Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: 2021-07-10T01:05:06.377Z [INFO] server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: entering follower state: follower="Node at 7e2b8bf7-5150-d051-2dbd-f464df97f8a2 [Follower]" leader= testing.go:149: server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: 2021-07-10T01:05:06.377Z [INFO] server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: entering follower state: follower="Node at 7e2b8bf7-5150-d051-2dbd-f464df97f8a2 [Follower]" leader= testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.447Z [WARN] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.447Z [WARN] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.447Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: entering candidate state: node="Node at 4b242baa-3ccf-bf7a-add2-3bc14bba42b8 [Candidate]" term=2 testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.447Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: entering candidate state: node="Node at 4b242baa-3ccf-bf7a-add2-3bc14bba42b8 [Candidate]" term=2 testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.448Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: election won: tally=2 testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.448Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: election won: tally=2 testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.448Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: entering leader state: leader="Node at 4b242baa-3ccf-bf7a-add2-3bc14bba42b8 [Leader]" testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.448Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: entering leader state: leader="Node at 4b242baa-3ccf-bf7a-add2-3bc14bba42b8 [Leader]" testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.448Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: added peer, starting replication: peer=server-c04fd653-66f5-6ead-e982-aafeff94b195 testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.448Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: added peer, starting replication: peer=server-c04fd653-66f5-6ead-e982-aafeff94b195 testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.448Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: added peer, starting replication: peer=server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2 testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.448Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: added peer, starting replication: peer=server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2 testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.448Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: pipelining replication: peer="{Voter server-c04fd653-66f5-6ead-e982-aafeff94b195 c04fd653-66f5-6ead-e982-aafeff94b195}" testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.448Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: pipelining replication: peer="{Voter server-c04fd653-66f5-6ead-e982-aafeff94b195 c04fd653-66f5-6ead-e982-aafeff94b195}" testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.448Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: pipelining replication: peer="{Voter server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2 7e2b8bf7-5150-d051-2dbd-f464df97f8a2}" testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.448Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: pipelining replication: peer="{Voter server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2 7e2b8bf7-5150-d051-2dbd-f464df97f8a2}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:06.553Z [INFO] stable state for Leader reached at 2021-07-10 01:05:06.448239753 +0000 UTC m=+11.836397763 (1 nodes), 71.121669ms from start of poll, 72.09672ms from cluster start. Timeout at 2021-07-10 01:05:06.553318281 +0000 UTC m=+11.941476291, 105.078528ms after stability testing.go:149: cluster: 2021-07-10T01:05:06.553Z [INFO] stable state for Leader reached at 2021-07-10 01:05:06.448239753 +0000 UTC m=+11.836397763 (1 nodes), 71.121669ms from start of poll, 72.09672ms from cluster start. Timeout at 2021-07-10 01:05:06.553318281 +0000 UTC m=+11.941476291, 105.078528ms after stability testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.554Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: starting snapshot up to: index=12 testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.554Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: starting snapshot up to: index=12 testing.go:153: 2021-07-10T01:05:06.554Z [INFO] creating new snapshot: path=/tmp/raft174448843/snapshots/2-12-1625879106554.tmp testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.556Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: snapshot complete up to: index=12 testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.556Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: snapshot complete up to: index=12 testing.go:153: 2021-07-10T01:05:06.557Z [INFO] creating new snapshot: path=/tmp/raft174448843/snapshots/2-10013-1625879106557.tmp testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.558Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: copied to local snapshot: bytes=71 testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.558Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.558Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: restored user snapshot: index=1 testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.558Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: restored user snapshot: index=1 testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.558Z [ERROR] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: failed to get log: index=23 error="log not found" testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.558Z [ERROR] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.558Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: aborting pipeline replication: peer="{Voter server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2 7e2b8bf7-5150-d051-2dbd-f464df97f8a2}" testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.558Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: aborting pipeline replication: peer="{Voter server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2 7e2b8bf7-5150-d051-2dbd-f464df97f8a2}" testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.559Z [ERROR] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: failed to get log: index=23 error="log not found" testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.559Z [ERROR] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.559Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: aborting pipeline replication: peer="{Voter server-c04fd653-66f5-6ead-e982-aafeff94b195 c04fd653-66f5-6ead-e982-aafeff94b195}" testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.559Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: aborting pipeline replication: peer="{Voter server-c04fd653-66f5-6ead-e982-aafeff94b195 c04fd653-66f5-6ead-e982-aafeff94b195}" testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.565Z [ERROR] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: failed to get log: index=23 error="log not found" testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.565Z [ERROR] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: failed to get log: index=23 error="log not found" testing.go:153: 2021-07-10T01:05:06.565Z [INFO] creating new snapshot: path=/tmp/raft362405679/snapshots/2-10013-1625879106565.tmp testLoggerAdapter verbose: server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: 2021-07-10T01:05:06.566Z [INFO] server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: copied to local snapshot: bytes=71 testing.go:149: server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: 2021-07-10T01:05:06.566Z [INFO] server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.566Z [ERROR] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: failed to get log: index=23 error="log not found" testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.566Z [ERROR] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: 2021-07-10T01:05:06.567Z [INFO] server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: Installed remote snapshot testing.go:149: server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: 2021-07-10T01:05:06.567Z [INFO] server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: Installed remote snapshot testLoggerAdapter verbose: server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: 2021-07-10T01:05:06.567Z [WARN] server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: skipping application of old log: index=22 testing.go:149: server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: 2021-07-10T01:05:06.567Z [WARN] server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2: skipping application of old log: index=22 testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.567Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: pipelining replication: peer="{Voter server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2 7e2b8bf7-5150-d051-2dbd-f464df97f8a2}" testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.567Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: pipelining replication: peer="{Voter server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2 7e2b8bf7-5150-d051-2dbd-f464df97f8a2}" testing.go:153: 2021-07-10T01:05:06.567Z [INFO] creating new snapshot: path=/tmp/raft418321205/snapshots/2-10013-1625879106567.tmp testLoggerAdapter verbose: server-c04fd653-66f5-6ead-e982-aafeff94b195: 2021-07-10T01:05:06.568Z [INFO] server-c04fd653-66f5-6ead-e982-aafeff94b195: copied to local snapshot: bytes=71 testing.go:149: server-c04fd653-66f5-6ead-e982-aafeff94b195: 2021-07-10T01:05:06.568Z [INFO] server-c04fd653-66f5-6ead-e982-aafeff94b195: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-c04fd653-66f5-6ead-e982-aafeff94b195: 2021-07-10T01:05:06.568Z [INFO] server-c04fd653-66f5-6ead-e982-aafeff94b195: Installed remote snapshot testing.go:149: server-c04fd653-66f5-6ead-e982-aafeff94b195: 2021-07-10T01:05:06.568Z [INFO] server-c04fd653-66f5-6ead-e982-aafeff94b195: Installed remote snapshot testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.568Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: pipelining replication: peer="{Voter server-c04fd653-66f5-6ead-e982-aafeff94b195 c04fd653-66f5-6ead-e982-aafeff94b195}" testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.568Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: pipelining replication: peer="{Voter server-c04fd653-66f5-6ead-e982-aafeff94b195 c04fd653-66f5-6ead-e982-aafeff94b195}" testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.578Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: aborting pipeline replication: peer="{Voter server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2 7e2b8bf7-5150-d051-2dbd-f464df97f8a2}" testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.578Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: aborting pipeline replication: peer="{Voter server-7e2b8bf7-5150-d051-2dbd-f464df97f8a2 7e2b8bf7-5150-d051-2dbd-f464df97f8a2}" testLoggerAdapter verbose: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.578Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: aborting pipeline replication: peer="{Voter server-c04fd653-66f5-6ead-e982-aafeff94b195 c04fd653-66f5-6ead-e982-aafeff94b195}" testing.go:149: server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: 2021-07-10T01:05:06.578Z [INFO] server-4b242baa-3ccf-bf7a-add2-3bc14bba42b8: aborting pipeline replication: peer="{Voter server-c04fd653-66f5-6ead-e982-aafeff94b195 c04fd653-66f5-6ead-e982-aafeff94b195}" --- PASS: TestRaft_UserRestore (1.23s) === RUN TestRaft_SendSnapshotFollower testLoggerAdapter verbose: server-833f9884-bbe3-cac2-0483-712a20845245: 2021-07-10T01:05:06.580Z [INFO] server-833f9884-bbe3-cac2-0483-712a20845245: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-833f9884-bbe3-cac2-0483-712a20845245 Address:833f9884-bbe3-cac2-0483-712a20845245} {Suffrage:Voter ID:server-aff53f15-3429-aec5-85fc-802ba07ee431 Address:aff53f15-3429-aec5-85fc-802ba07ee431} {Suffrage:Voter ID:server-6f109fbf-935a-84c3-6933-fe72fe24c98a Address:6f109fbf-935a-84c3-6933-fe72fe24c98a}]" testing.go:149: server-833f9884-bbe3-cac2-0483-712a20845245: 2021-07-10T01:05:06.580Z [INFO] server-833f9884-bbe3-cac2-0483-712a20845245: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-833f9884-bbe3-cac2-0483-712a20845245 Address:833f9884-bbe3-cac2-0483-712a20845245} {Suffrage:Voter ID:server-aff53f15-3429-aec5-85fc-802ba07ee431 Address:aff53f15-3429-aec5-85fc-802ba07ee431} {Suffrage:Voter ID:server-6f109fbf-935a-84c3-6933-fe72fe24c98a Address:6f109fbf-935a-84c3-6933-fe72fe24c98a}]" testLoggerAdapter verbose: server-833f9884-bbe3-cac2-0483-712a20845245: 2021-07-10T01:05:06.580Z [INFO] server-833f9884-bbe3-cac2-0483-712a20845245: entering follower state: follower="Node at 833f9884-bbe3-cac2-0483-712a20845245 [Follower]" leader= testing.go:149: server-833f9884-bbe3-cac2-0483-712a20845245: 2021-07-10T01:05:06.580Z [INFO] server-833f9884-bbe3-cac2-0483-712a20845245: entering follower state: follower="Node at 833f9884-bbe3-cac2-0483-712a20845245 [Follower]" leader= testLoggerAdapter verbose: server-aff53f15-3429-aec5-85fc-802ba07ee431: 2021-07-10T01:05:06.580Z [INFO] server-aff53f15-3429-aec5-85fc-802ba07ee431: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-833f9884-bbe3-cac2-0483-712a20845245 Address:833f9884-bbe3-cac2-0483-712a20845245} {Suffrage:Voter ID:server-aff53f15-3429-aec5-85fc-802ba07ee431 Address:aff53f15-3429-aec5-85fc-802ba07ee431} {Suffrage:Voter ID:server-6f109fbf-935a-84c3-6933-fe72fe24c98a Address:6f109fbf-935a-84c3-6933-fe72fe24c98a}]" testing.go:149: server-aff53f15-3429-aec5-85fc-802ba07ee431: 2021-07-10T01:05:06.580Z [INFO] server-aff53f15-3429-aec5-85fc-802ba07ee431: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-833f9884-bbe3-cac2-0483-712a20845245 Address:833f9884-bbe3-cac2-0483-712a20845245} {Suffrage:Voter ID:server-aff53f15-3429-aec5-85fc-802ba07ee431 Address:aff53f15-3429-aec5-85fc-802ba07ee431} {Suffrage:Voter ID:server-6f109fbf-935a-84c3-6933-fe72fe24c98a Address:6f109fbf-935a-84c3-6933-fe72fe24c98a}]" testLoggerAdapter verbose: server-aff53f15-3429-aec5-85fc-802ba07ee431: 2021-07-10T01:05:06.580Z [INFO] server-aff53f15-3429-aec5-85fc-802ba07ee431: entering follower state: follower="Node at aff53f15-3429-aec5-85fc-802ba07ee431 [Follower]" leader= testing.go:149: server-aff53f15-3429-aec5-85fc-802ba07ee431: 2021-07-10T01:05:06.580Z [INFO] server-aff53f15-3429-aec5-85fc-802ba07ee431: entering follower state: follower="Node at aff53f15-3429-aec5-85fc-802ba07ee431 [Follower]" leader= testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.581Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-833f9884-bbe3-cac2-0483-712a20845245 Address:833f9884-bbe3-cac2-0483-712a20845245} {Suffrage:Voter ID:server-aff53f15-3429-aec5-85fc-802ba07ee431 Address:aff53f15-3429-aec5-85fc-802ba07ee431} {Suffrage:Voter ID:server-6f109fbf-935a-84c3-6933-fe72fe24c98a Address:6f109fbf-935a-84c3-6933-fe72fe24c98a}]" testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.581Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-833f9884-bbe3-cac2-0483-712a20845245 Address:833f9884-bbe3-cac2-0483-712a20845245} {Suffrage:Voter ID:server-aff53f15-3429-aec5-85fc-802ba07ee431 Address:aff53f15-3429-aec5-85fc-802ba07ee431} {Suffrage:Voter ID:server-6f109fbf-935a-84c3-6933-fe72fe24c98a Address:6f109fbf-935a-84c3-6933-fe72fe24c98a}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:06.581Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-07-10T01:05:06.581Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.581Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: entering follower state: follower="Node at 6f109fbf-935a-84c3-6933-fe72fe24c98a [Follower]" leader= testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.581Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: entering follower state: follower="Node at 6f109fbf-935a-84c3-6933-fe72fe24c98a [Follower]" leader= testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.645Z [WARN] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.645Z [WARN] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.645Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: entering candidate state: node="Node at 6f109fbf-935a-84c3-6933-fe72fe24c98a [Candidate]" term=2 testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.645Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: entering candidate state: node="Node at 6f109fbf-935a-84c3-6933-fe72fe24c98a [Candidate]" term=2 testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.645Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: election won: tally=2 testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.645Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: election won: tally=2 testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.645Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: entering leader state: leader="Node at 6f109fbf-935a-84c3-6933-fe72fe24c98a [Leader]" testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.645Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: entering leader state: leader="Node at 6f109fbf-935a-84c3-6933-fe72fe24c98a [Leader]" testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.645Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: added peer, starting replication: peer=server-833f9884-bbe3-cac2-0483-712a20845245 testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.645Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: added peer, starting replication: peer=server-833f9884-bbe3-cac2-0483-712a20845245 testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.645Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: added peer, starting replication: peer=server-aff53f15-3429-aec5-85fc-802ba07ee431 testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.645Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: added peer, starting replication: peer=server-aff53f15-3429-aec5-85fc-802ba07ee431 testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.645Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: pipelining replication: peer="{Voter server-833f9884-bbe3-cac2-0483-712a20845245 833f9884-bbe3-cac2-0483-712a20845245}" testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.645Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: pipelining replication: peer="{Voter server-833f9884-bbe3-cac2-0483-712a20845245 833f9884-bbe3-cac2-0483-712a20845245}" testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.645Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: pipelining replication: peer="{Voter server-aff53f15-3429-aec5-85fc-802ba07ee431 aff53f15-3429-aec5-85fc-802ba07ee431}" testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.645Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: pipelining replication: peer="{Voter server-aff53f15-3429-aec5-85fc-802ba07ee431 aff53f15-3429-aec5-85fc-802ba07ee431}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:06.750Z [INFO] stable state for Follower reached at 2021-07-10 01:05:06.645610134 +0000 UTC m=+12.033768124 (2 nodes), 64.416979ms from start of poll, 65.44569ms from cluster start. Timeout at 2021-07-10 01:05:06.750701142 +0000 UTC m=+12.138859112, 105.090988ms after stability testing.go:149: cluster: 2021-07-10T01:05:06.750Z [INFO] stable state for Follower reached at 2021-07-10 01:05:06.645610134 +0000 UTC m=+12.033768124 (2 nodes), 64.416979ms from start of poll, 65.44569ms from cluster start. Timeout at 2021-07-10 01:05:06.750701142 +0000 UTC m=+12.138859112, 105.090988ms after stability testLoggerAdapter verbose: cluster: 2021-07-10T01:05:06.750Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:06.750Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-07-10T01:05:06.856Z [INFO] stable state for Leader reached at 2021-07-10 01:05:06.750958985 +0000 UTC m=+12.139116935 (1 nodes), 2.64µs from start of poll, 170.794501ms from cluster start. Timeout at 2021-07-10 01:05:06.856041593 +0000 UTC m=+12.244199583, 105.082648ms after stability testing.go:149: cluster: 2021-07-10T01:05:06.856Z [INFO] stable state for Leader reached at 2021-07-10 01:05:06.750958985 +0000 UTC m=+12.139116935 (1 nodes), 2.64µs from start of poll, 170.794501ms from cluster start. Timeout at 2021-07-10 01:05:06.856041593 +0000 UTC m=+12.244199583, 105.082648ms after stability testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.856Z [ERROR] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: failed to pipeline appendEntries: peer="{Voter server-833f9884-bbe3-cac2-0483-712a20845245 833f9884-bbe3-cac2-0483-712a20845245}" error="append pipeline closed" testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.856Z [ERROR] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: failed to pipeline appendEntries: peer="{Voter server-833f9884-bbe3-cac2-0483-712a20845245 833f9884-bbe3-cac2-0483-712a20845245}" error="append pipeline closed" testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.856Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: aborting pipeline replication: peer="{Voter server-833f9884-bbe3-cac2-0483-712a20845245 833f9884-bbe3-cac2-0483-712a20845245}" testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.856Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: aborting pipeline replication: peer="{Voter server-833f9884-bbe3-cac2-0483-712a20845245 833f9884-bbe3-cac2-0483-712a20845245}" testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.856Z [ERROR] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: failed to appendEntries to: peer="{Voter server-833f9884-bbe3-cac2-0483-712a20845245 833f9884-bbe3-cac2-0483-712a20845245}" error="failed to connect to peer: 833f9884-bbe3-cac2-0483-712a20845245" testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.856Z [ERROR] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: failed to appendEntries to: peer="{Voter server-833f9884-bbe3-cac2-0483-712a20845245 833f9884-bbe3-cac2-0483-712a20845245}" error="failed to connect to peer: 833f9884-bbe3-cac2-0483-712a20845245" testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.858Z [ERROR] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: failed to heartbeat to: peer=833f9884-bbe3-cac2-0483-712a20845245 error="failed to connect to peer: 833f9884-bbe3-cac2-0483-712a20845245" testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.858Z [ERROR] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: failed to heartbeat to: peer=833f9884-bbe3-cac2-0483-712a20845245 error="failed to connect to peer: 833f9884-bbe3-cac2-0483-712a20845245" raft_test.go:1320: [INFO] Finished apply without behind follower testLoggerAdapter verbose: server-833f9884-bbe3-cac2-0483-712a20845245: 2021-07-10T01:05:06.859Z [ERROR] server-833f9884-bbe3-cac2-0483-712a20845245: failed to take snapshot: error="nothing new to snapshot" testing.go:149: server-833f9884-bbe3-cac2-0483-712a20845245: 2021-07-10T01:05:06.859Z [ERROR] server-833f9884-bbe3-cac2-0483-712a20845245: failed to take snapshot: error="nothing new to snapshot" testLoggerAdapter verbose: server-aff53f15-3429-aec5-85fc-802ba07ee431: 2021-07-10T01:05:06.859Z [INFO] server-aff53f15-3429-aec5-85fc-802ba07ee431: starting snapshot up to: index=99 testing.go:149: server-aff53f15-3429-aec5-85fc-802ba07ee431: 2021-07-10T01:05:06.859Z [INFO] server-aff53f15-3429-aec5-85fc-802ba07ee431: starting snapshot up to: index=99 testing.go:153: 2021-07-10T01:05:06.859Z [INFO] creating new snapshot: path=/tmp/raft660393939/snapshots/2-99-1625879106859.tmp testLoggerAdapter verbose: server-aff53f15-3429-aec5-85fc-802ba07ee431: 2021-07-10T01:05:06.860Z [INFO] server-aff53f15-3429-aec5-85fc-802ba07ee431: compacting logs: from=1 to=92 testing.go:149: server-aff53f15-3429-aec5-85fc-802ba07ee431: 2021-07-10T01:05:06.860Z [INFO] server-aff53f15-3429-aec5-85fc-802ba07ee431: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-aff53f15-3429-aec5-85fc-802ba07ee431: 2021-07-10T01:05:06.860Z [INFO] server-aff53f15-3429-aec5-85fc-802ba07ee431: snapshot complete up to: index=99 testing.go:149: server-aff53f15-3429-aec5-85fc-802ba07ee431: 2021-07-10T01:05:06.860Z [INFO] server-aff53f15-3429-aec5-85fc-802ba07ee431: snapshot complete up to: index=99 testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.860Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: starting snapshot up to: index=102 testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.860Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: starting snapshot up to: index=102 testing.go:153: 2021-07-10T01:05:06.861Z [INFO] creating new snapshot: path=/tmp/raft560011645/snapshots/2-102-1625879106861.tmp testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.861Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: compacting logs: from=1 to=92 testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.861Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.861Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: snapshot complete up to: index=102 testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.861Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: snapshot complete up to: index=102 testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.867Z [ERROR] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: failed to get log: index=2 error="log not found" testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.867Z [ERROR] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: failed to get log: index=2 error="log not found" testing.go:153: 2021-07-10T01:05:06.867Z [INFO] creating new snapshot: path=/tmp/raft589080377/snapshots/2-102-1625879106867.tmp testLoggerAdapter verbose: server-833f9884-bbe3-cac2-0483-712a20845245: 2021-07-10T01:05:06.868Z [INFO] server-833f9884-bbe3-cac2-0483-712a20845245: copied to local snapshot: bytes=693 testing.go:149: server-833f9884-bbe3-cac2-0483-712a20845245: 2021-07-10T01:05:06.868Z [INFO] server-833f9884-bbe3-cac2-0483-712a20845245: copied to local snapshot: bytes=693 testLoggerAdapter verbose: server-833f9884-bbe3-cac2-0483-712a20845245: 2021-07-10T01:05:06.868Z [INFO] server-833f9884-bbe3-cac2-0483-712a20845245: Installed remote snapshot testing.go:149: server-833f9884-bbe3-cac2-0483-712a20845245: 2021-07-10T01:05:06.868Z [INFO] server-833f9884-bbe3-cac2-0483-712a20845245: Installed remote snapshot testLoggerAdapter verbose: server-833f9884-bbe3-cac2-0483-712a20845245: 2021-07-10T01:05:06.868Z [WARN] server-833f9884-bbe3-cac2-0483-712a20845245: skipping application of old log: index=102 testing.go:149: server-833f9884-bbe3-cac2-0483-712a20845245: 2021-07-10T01:05:06.868Z [WARN] server-833f9884-bbe3-cac2-0483-712a20845245: skipping application of old log: index=102 testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.869Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: pipelining replication: peer="{Voter server-833f9884-bbe3-cac2-0483-712a20845245 833f9884-bbe3-cac2-0483-712a20845245}" testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.869Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: pipelining replication: peer="{Voter server-833f9884-bbe3-cac2-0483-712a20845245 833f9884-bbe3-cac2-0483-712a20845245}" testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.872Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: aborting pipeline replication: peer="{Voter server-833f9884-bbe3-cac2-0483-712a20845245 833f9884-bbe3-cac2-0483-712a20845245}" testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.872Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: aborting pipeline replication: peer="{Voter server-833f9884-bbe3-cac2-0483-712a20845245 833f9884-bbe3-cac2-0483-712a20845245}" testLoggerAdapter verbose: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.872Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: aborting pipeline replication: peer="{Voter server-aff53f15-3429-aec5-85fc-802ba07ee431 aff53f15-3429-aec5-85fc-802ba07ee431}" testing.go:149: server-6f109fbf-935a-84c3-6933-fe72fe24c98a: 2021-07-10T01:05:06.872Z [INFO] server-6f109fbf-935a-84c3-6933-fe72fe24c98a: aborting pipeline replication: peer="{Voter server-aff53f15-3429-aec5-85fc-802ba07ee431 aff53f15-3429-aec5-85fc-802ba07ee431}" --- PASS: TestRaft_SendSnapshotFollower (0.29s) === RUN TestRaft_SendSnapshotAndLogsFollower testLoggerAdapter verbose: server-6f4c103a-d978-ffc1-4b9d-66afe690d185: 2021-07-10T01:05:06.874Z [INFO] server-6f4c103a-d978-ffc1-4b9d-66afe690d185: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f4c103a-d978-ffc1-4b9d-66afe690d185 Address:6f4c103a-d978-ffc1-4b9d-66afe690d185} {Suffrage:Voter ID:server-1305ff47-0b5c-245e-8636-f3e7778aeda0 Address:1305ff47-0b5c-245e-8636-f3e7778aeda0} {Suffrage:Voter ID:server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8 Address:ab77d2b0-4d6f-3860-1a02-75b1f4032ac8}]" testing.go:149: server-6f4c103a-d978-ffc1-4b9d-66afe690d185: 2021-07-10T01:05:06.874Z [INFO] server-6f4c103a-d978-ffc1-4b9d-66afe690d185: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f4c103a-d978-ffc1-4b9d-66afe690d185 Address:6f4c103a-d978-ffc1-4b9d-66afe690d185} {Suffrage:Voter ID:server-1305ff47-0b5c-245e-8636-f3e7778aeda0 Address:1305ff47-0b5c-245e-8636-f3e7778aeda0} {Suffrage:Voter ID:server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8 Address:ab77d2b0-4d6f-3860-1a02-75b1f4032ac8}]" testLoggerAdapter verbose: server-6f4c103a-d978-ffc1-4b9d-66afe690d185: 2021-07-10T01:05:06.875Z [INFO] server-6f4c103a-d978-ffc1-4b9d-66afe690d185: entering follower state: follower="Node at 6f4c103a-d978-ffc1-4b9d-66afe690d185 [Follower]" leader= testing.go:149: server-6f4c103a-d978-ffc1-4b9d-66afe690d185: 2021-07-10T01:05:06.875Z [INFO] server-6f4c103a-d978-ffc1-4b9d-66afe690d185: entering follower state: follower="Node at 6f4c103a-d978-ffc1-4b9d-66afe690d185 [Follower]" leader= testLoggerAdapter verbose: server-1305ff47-0b5c-245e-8636-f3e7778aeda0: 2021-07-10T01:05:06.875Z [INFO] server-1305ff47-0b5c-245e-8636-f3e7778aeda0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f4c103a-d978-ffc1-4b9d-66afe690d185 Address:6f4c103a-d978-ffc1-4b9d-66afe690d185} {Suffrage:Voter ID:server-1305ff47-0b5c-245e-8636-f3e7778aeda0 Address:1305ff47-0b5c-245e-8636-f3e7778aeda0} {Suffrage:Voter ID:server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8 Address:ab77d2b0-4d6f-3860-1a02-75b1f4032ac8}]" testing.go:149: server-1305ff47-0b5c-245e-8636-f3e7778aeda0: 2021-07-10T01:05:06.875Z [INFO] server-1305ff47-0b5c-245e-8636-f3e7778aeda0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f4c103a-d978-ffc1-4b9d-66afe690d185 Address:6f4c103a-d978-ffc1-4b9d-66afe690d185} {Suffrage:Voter ID:server-1305ff47-0b5c-245e-8636-f3e7778aeda0 Address:1305ff47-0b5c-245e-8636-f3e7778aeda0} {Suffrage:Voter ID:server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8 Address:ab77d2b0-4d6f-3860-1a02-75b1f4032ac8}]" testLoggerAdapter verbose: server-1305ff47-0b5c-245e-8636-f3e7778aeda0: 2021-07-10T01:05:06.875Z [INFO] server-1305ff47-0b5c-245e-8636-f3e7778aeda0: entering follower state: follower="Node at 1305ff47-0b5c-245e-8636-f3e7778aeda0 [Follower]" leader= testing.go:149: server-1305ff47-0b5c-245e-8636-f3e7778aeda0: 2021-07-10T01:05:06.875Z [INFO] server-1305ff47-0b5c-245e-8636-f3e7778aeda0: entering follower state: follower="Node at 1305ff47-0b5c-245e-8636-f3e7778aeda0 [Follower]" leader= testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.875Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f4c103a-d978-ffc1-4b9d-66afe690d185 Address:6f4c103a-d978-ffc1-4b9d-66afe690d185} {Suffrage:Voter ID:server-1305ff47-0b5c-245e-8636-f3e7778aeda0 Address:1305ff47-0b5c-245e-8636-f3e7778aeda0} {Suffrage:Voter ID:server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8 Address:ab77d2b0-4d6f-3860-1a02-75b1f4032ac8}]" testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.875Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f4c103a-d978-ffc1-4b9d-66afe690d185 Address:6f4c103a-d978-ffc1-4b9d-66afe690d185} {Suffrage:Voter ID:server-1305ff47-0b5c-245e-8636-f3e7778aeda0 Address:1305ff47-0b5c-245e-8636-f3e7778aeda0} {Suffrage:Voter ID:server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8 Address:ab77d2b0-4d6f-3860-1a02-75b1f4032ac8}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:06.875Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-07-10T01:05:06.875Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.875Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: entering follower state: follower="Node at ab77d2b0-4d6f-3860-1a02-75b1f4032ac8 [Follower]" leader= testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.875Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: entering follower state: follower="Node at ab77d2b0-4d6f-3860-1a02-75b1f4032ac8 [Follower]" leader= testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.933Z [WARN] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.933Z [WARN] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.933Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: entering candidate state: node="Node at ab77d2b0-4d6f-3860-1a02-75b1f4032ac8 [Candidate]" term=2 testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.933Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: entering candidate state: node="Node at ab77d2b0-4d6f-3860-1a02-75b1f4032ac8 [Candidate]" term=2 testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.933Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: election won: tally=2 testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.933Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: election won: tally=2 testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.933Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: entering leader state: leader="Node at ab77d2b0-4d6f-3860-1a02-75b1f4032ac8 [Leader]" testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.933Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: entering leader state: leader="Node at ab77d2b0-4d6f-3860-1a02-75b1f4032ac8 [Leader]" testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.933Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: added peer, starting replication: peer=server-6f4c103a-d978-ffc1-4b9d-66afe690d185 testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.933Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: added peer, starting replication: peer=server-6f4c103a-d978-ffc1-4b9d-66afe690d185 testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.933Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: added peer, starting replication: peer=server-1305ff47-0b5c-245e-8636-f3e7778aeda0 testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.933Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: added peer, starting replication: peer=server-1305ff47-0b5c-245e-8636-f3e7778aeda0 testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.933Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: pipelining replication: peer="{Voter server-6f4c103a-d978-ffc1-4b9d-66afe690d185 6f4c103a-d978-ffc1-4b9d-66afe690d185}" testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.933Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: pipelining replication: peer="{Voter server-6f4c103a-d978-ffc1-4b9d-66afe690d185 6f4c103a-d978-ffc1-4b9d-66afe690d185}" testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.933Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: pipelining replication: peer="{Voter server-1305ff47-0b5c-245e-8636-f3e7778aeda0 1305ff47-0b5c-245e-8636-f3e7778aeda0}" testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:06.933Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: pipelining replication: peer="{Voter server-1305ff47-0b5c-245e-8636-f3e7778aeda0 1305ff47-0b5c-245e-8636-f3e7778aeda0}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:07.038Z [INFO] stable state for Follower reached at 2021-07-10 01:05:06.933776372 +0000 UTC m=+12.321934322 (2 nodes), 57.942631ms from start of poll, 59.008802ms from cluster start. Timeout at 2021-07-10 01:05:07.03885572 +0000 UTC m=+12.427013750, 105.079428ms after stability testing.go:149: cluster: 2021-07-10T01:05:07.038Z [INFO] stable state for Follower reached at 2021-07-10 01:05:06.933776372 +0000 UTC m=+12.321934322 (2 nodes), 57.942631ms from start of poll, 59.008802ms from cluster start. Timeout at 2021-07-10 01:05:07.03885572 +0000 UTC m=+12.427013750, 105.079428ms after stability testLoggerAdapter verbose: cluster: 2021-07-10T01:05:07.039Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:07.039Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-07-10T01:05:07.144Z [INFO] stable state for Leader reached at 2021-07-10 01:05:07.039121863 +0000 UTC m=+12.427279913 (1 nodes), 2.44µs from start of poll, 164.354393ms from cluster start. Timeout at 2021-07-10 01:05:07.14419145 +0000 UTC m=+12.532349440, 105.069527ms after stability testing.go:149: cluster: 2021-07-10T01:05:07.144Z [INFO] stable state for Leader reached at 2021-07-10 01:05:07.039121863 +0000 UTC m=+12.427279913 (1 nodes), 2.44µs from start of poll, 164.354393ms from cluster start. Timeout at 2021-07-10 01:05:07.14419145 +0000 UTC m=+12.532349440, 105.069527ms after stability testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.144Z [ERROR] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: failed to pipeline appendEntries: peer="{Voter server-6f4c103a-d978-ffc1-4b9d-66afe690d185 6f4c103a-d978-ffc1-4b9d-66afe690d185}" error="append pipeline closed" testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.144Z [ERROR] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: failed to pipeline appendEntries: peer="{Voter server-6f4c103a-d978-ffc1-4b9d-66afe690d185 6f4c103a-d978-ffc1-4b9d-66afe690d185}" error="append pipeline closed" testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.144Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: aborting pipeline replication: peer="{Voter server-6f4c103a-d978-ffc1-4b9d-66afe690d185 6f4c103a-d978-ffc1-4b9d-66afe690d185}" testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.144Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: aborting pipeline replication: peer="{Voter server-6f4c103a-d978-ffc1-4b9d-66afe690d185 6f4c103a-d978-ffc1-4b9d-66afe690d185}" testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.144Z [ERROR] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: failed to appendEntries to: peer="{Voter server-6f4c103a-d978-ffc1-4b9d-66afe690d185 6f4c103a-d978-ffc1-4b9d-66afe690d185}" error="failed to connect to peer: 6f4c103a-d978-ffc1-4b9d-66afe690d185" testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.144Z [ERROR] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: failed to appendEntries to: peer="{Voter server-6f4c103a-d978-ffc1-4b9d-66afe690d185 6f4c103a-d978-ffc1-4b9d-66afe690d185}" error="failed to connect to peer: 6f4c103a-d978-ffc1-4b9d-66afe690d185" raft_test.go:1362: [INFO] Finished apply without behind follower testLoggerAdapter verbose: server-6f4c103a-d978-ffc1-4b9d-66afe690d185: 2021-07-10T01:05:07.147Z [ERROR] server-6f4c103a-d978-ffc1-4b9d-66afe690d185: failed to take snapshot: error="nothing new to snapshot" testing.go:149: server-6f4c103a-d978-ffc1-4b9d-66afe690d185: 2021-07-10T01:05:07.147Z [ERROR] server-6f4c103a-d978-ffc1-4b9d-66afe690d185: failed to take snapshot: error="nothing new to snapshot" testLoggerAdapter verbose: server-1305ff47-0b5c-245e-8636-f3e7778aeda0: 2021-07-10T01:05:07.147Z [INFO] server-1305ff47-0b5c-245e-8636-f3e7778aeda0: starting snapshot up to: index=100 testing.go:149: server-1305ff47-0b5c-245e-8636-f3e7778aeda0: 2021-07-10T01:05:07.147Z [INFO] server-1305ff47-0b5c-245e-8636-f3e7778aeda0: starting snapshot up to: index=100 testing.go:153: 2021-07-10T01:05:07.147Z [INFO] creating new snapshot: path=/tmp/raft332030977/snapshots/2-100-1625879107147.tmp testLoggerAdapter verbose: server-1305ff47-0b5c-245e-8636-f3e7778aeda0: 2021-07-10T01:05:07.148Z [INFO] server-1305ff47-0b5c-245e-8636-f3e7778aeda0: compacting logs: from=1 to=92 testing.go:149: server-1305ff47-0b5c-245e-8636-f3e7778aeda0: 2021-07-10T01:05:07.148Z [INFO] server-1305ff47-0b5c-245e-8636-f3e7778aeda0: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-1305ff47-0b5c-245e-8636-f3e7778aeda0: 2021-07-10T01:05:07.148Z [INFO] server-1305ff47-0b5c-245e-8636-f3e7778aeda0: snapshot complete up to: index=100 testing.go:149: server-1305ff47-0b5c-245e-8636-f3e7778aeda0: 2021-07-10T01:05:07.148Z [INFO] server-1305ff47-0b5c-245e-8636-f3e7778aeda0: snapshot complete up to: index=100 testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.148Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: starting snapshot up to: index=102 testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.148Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: starting snapshot up to: index=102 testing.go:153: 2021-07-10T01:05:07.148Z [INFO] creating new snapshot: path=/tmp/raft587756507/snapshots/2-102-1625879107148.tmp testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.149Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: compacting logs: from=1 to=92 testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.149Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.149Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: snapshot complete up to: index=102 testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.149Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: snapshot complete up to: index=102 testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.150Z [ERROR] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: failed to heartbeat to: peer=6f4c103a-d978-ffc1-4b9d-66afe690d185 error="failed to connect to peer: 6f4c103a-d978-ffc1-4b9d-66afe690d185" testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.150Z [ERROR] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: failed to heartbeat to: peer=6f4c103a-d978-ffc1-4b9d-66afe690d185 error="failed to connect to peer: 6f4c103a-d978-ffc1-4b9d-66afe690d185" raft_test.go:1383: [INFO] Finished apply without behind follower testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.154Z [ERROR] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: failed to get log: index=2 error="log not found" testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.154Z [ERROR] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: failed to get log: index=2 error="log not found" testing.go:153: 2021-07-10T01:05:07.155Z [INFO] creating new snapshot: path=/tmp/raft766657207/snapshots/2-102-1625879107155.tmp testLoggerAdapter verbose: server-6f4c103a-d978-ffc1-4b9d-66afe690d185: 2021-07-10T01:05:07.155Z [INFO] server-6f4c103a-d978-ffc1-4b9d-66afe690d185: copied to local snapshot: bytes=693 testing.go:149: server-6f4c103a-d978-ffc1-4b9d-66afe690d185: 2021-07-10T01:05:07.155Z [INFO] server-6f4c103a-d978-ffc1-4b9d-66afe690d185: copied to local snapshot: bytes=693 testLoggerAdapter verbose: server-6f4c103a-d978-ffc1-4b9d-66afe690d185: 2021-07-10T01:05:07.156Z [INFO] server-6f4c103a-d978-ffc1-4b9d-66afe690d185: Installed remote snapshot testing.go:149: server-6f4c103a-d978-ffc1-4b9d-66afe690d185: 2021-07-10T01:05:07.156Z [INFO] server-6f4c103a-d978-ffc1-4b9d-66afe690d185: Installed remote snapshot testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.156Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: pipelining replication: peer="{Voter server-6f4c103a-d978-ffc1-4b9d-66afe690d185 6f4c103a-d978-ffc1-4b9d-66afe690d185}" testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.156Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: pipelining replication: peer="{Voter server-6f4c103a-d978-ffc1-4b9d-66afe690d185 6f4c103a-d978-ffc1-4b9d-66afe690d185}" testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.161Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: aborting pipeline replication: peer="{Voter server-1305ff47-0b5c-245e-8636-f3e7778aeda0 1305ff47-0b5c-245e-8636-f3e7778aeda0}" testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.161Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: aborting pipeline replication: peer="{Voter server-1305ff47-0b5c-245e-8636-f3e7778aeda0 1305ff47-0b5c-245e-8636-f3e7778aeda0}" testLoggerAdapter verbose: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.161Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: aborting pipeline replication: peer="{Voter server-6f4c103a-d978-ffc1-4b9d-66afe690d185 6f4c103a-d978-ffc1-4b9d-66afe690d185}" testing.go:149: server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: 2021-07-10T01:05:07.161Z [INFO] server-ab77d2b0-4d6f-3860-1a02-75b1f4032ac8: aborting pipeline replication: peer="{Voter server-6f4c103a-d978-ffc1-4b9d-66afe690d185 6f4c103a-d978-ffc1-4b9d-66afe690d185}" --- PASS: TestRaft_SendSnapshotAndLogsFollower (0.29s) === RUN TestRaft_ReJoinFollower testLoggerAdapter verbose: server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: 2021-07-10T01:05:07.163Z [INFO] server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 Address:0d2493c3-115c-992a-5a45-e72b1f0a85b1} {Suffrage:Voter ID:server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 Address:b76e5ee7-3f04-6b7d-2af0-4a98745184b4} {Suffrage:Voter ID:server-99962721-dd8b-44ac-1b09-7ed704d78cc5 Address:99962721-dd8b-44ac-1b09-7ed704d78cc5}]" testing.go:149: server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: 2021-07-10T01:05:07.163Z [INFO] server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 Address:0d2493c3-115c-992a-5a45-e72b1f0a85b1} {Suffrage:Voter ID:server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 Address:b76e5ee7-3f04-6b7d-2af0-4a98745184b4} {Suffrage:Voter ID:server-99962721-dd8b-44ac-1b09-7ed704d78cc5 Address:99962721-dd8b-44ac-1b09-7ed704d78cc5}]" testLoggerAdapter verbose: server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: 2021-07-10T01:05:07.163Z [INFO] server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: entering follower state: follower="Node at 0d2493c3-115c-992a-5a45-e72b1f0a85b1 [Follower]" leader= testing.go:149: server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: 2021-07-10T01:05:07.163Z [INFO] server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: entering follower state: follower="Node at 0d2493c3-115c-992a-5a45-e72b1f0a85b1 [Follower]" leader= testLoggerAdapter verbose: server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4: 2021-07-10T01:05:07.164Z [INFO] server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 Address:0d2493c3-115c-992a-5a45-e72b1f0a85b1} {Suffrage:Voter ID:server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 Address:b76e5ee7-3f04-6b7d-2af0-4a98745184b4} {Suffrage:Voter ID:server-99962721-dd8b-44ac-1b09-7ed704d78cc5 Address:99962721-dd8b-44ac-1b09-7ed704d78cc5}]" testing.go:149: server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4: 2021-07-10T01:05:07.164Z [INFO] server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 Address:0d2493c3-115c-992a-5a45-e72b1f0a85b1} {Suffrage:Voter ID:server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 Address:b76e5ee7-3f04-6b7d-2af0-4a98745184b4} {Suffrage:Voter ID:server-99962721-dd8b-44ac-1b09-7ed704d78cc5 Address:99962721-dd8b-44ac-1b09-7ed704d78cc5}]" testLoggerAdapter verbose: server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4: 2021-07-10T01:05:07.164Z [INFO] server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4: entering follower state: follower="Node at b76e5ee7-3f04-6b7d-2af0-4a98745184b4 [Follower]" leader= testing.go:149: server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4: 2021-07-10T01:05:07.164Z [INFO] server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4: entering follower state: follower="Node at b76e5ee7-3f04-6b7d-2af0-4a98745184b4 [Follower]" leader= testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.164Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 Address:0d2493c3-115c-992a-5a45-e72b1f0a85b1} {Suffrage:Voter ID:server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 Address:b76e5ee7-3f04-6b7d-2af0-4a98745184b4} {Suffrage:Voter ID:server-99962721-dd8b-44ac-1b09-7ed704d78cc5 Address:99962721-dd8b-44ac-1b09-7ed704d78cc5}]" testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.164Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 Address:0d2493c3-115c-992a-5a45-e72b1f0a85b1} {Suffrage:Voter ID:server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 Address:b76e5ee7-3f04-6b7d-2af0-4a98745184b4} {Suffrage:Voter ID:server-99962721-dd8b-44ac-1b09-7ed704d78cc5 Address:99962721-dd8b-44ac-1b09-7ed704d78cc5}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:07.164Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:07.164Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.164Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: entering follower state: follower="Node at 99962721-dd8b-44ac-1b09-7ed704d78cc5 [Follower]" leader= testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.164Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: entering follower state: follower="Node at 99962721-dd8b-44ac-1b09-7ed704d78cc5 [Follower]" leader= testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.219Z [WARN] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.219Z [WARN] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.219Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: entering candidate state: node="Node at 99962721-dd8b-44ac-1b09-7ed704d78cc5 [Candidate]" term=2 testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.219Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: entering candidate state: node="Node at 99962721-dd8b-44ac-1b09-7ed704d78cc5 [Candidate]" term=2 testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.219Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: election won: tally=2 testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.219Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: election won: tally=2 testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.219Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: entering leader state: leader="Node at 99962721-dd8b-44ac-1b09-7ed704d78cc5 [Leader]" testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.219Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: entering leader state: leader="Node at 99962721-dd8b-44ac-1b09-7ed704d78cc5 [Leader]" testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.220Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: added peer, starting replication: peer=server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.220Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: added peer, starting replication: peer=server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.220Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: added peer, starting replication: peer=server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.220Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: added peer, starting replication: peer=server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.220Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: pipelining replication: peer="{Voter server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 0d2493c3-115c-992a-5a45-e72b1f0a85b1}" testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.220Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: pipelining replication: peer="{Voter server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 0d2493c3-115c-992a-5a45-e72b1f0a85b1}" testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.220Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: pipelining replication: peer="{Voter server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 b76e5ee7-3f04-6b7d-2af0-4a98745184b4}" testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.220Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: pipelining replication: peer="{Voter server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 b76e5ee7-3f04-6b7d-2af0-4a98745184b4}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:07.325Z [INFO] stable state for Leader reached at 2021-07-10 01:05:07.22005621 +0000 UTC m=+12.608214200 (1 nodes), 55.512885ms from start of poll, 56.496055ms from cluster start. Timeout at 2021-07-10 01:05:07.325127738 +0000 UTC m=+12.713285728, 105.071528ms after stability testing.go:149: cluster: 2021-07-10T01:05:07.325Z [INFO] stable state for Leader reached at 2021-07-10 01:05:07.22005621 +0000 UTC m=+12.608214200 (1 nodes), 55.512885ms from start of poll, 56.496055ms from cluster start. Timeout at 2021-07-10 01:05:07.325127738 +0000 UTC m=+12.713285728, 105.071528ms after stability testLoggerAdapter verbose: cluster: 2021-07-10T01:05:07.330Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-07-10T01:05:07.330Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-07-10T01:05:07.435Z [INFO] stable state for Follower reached at 2021-07-10 01:05:07.330558955 +0000 UTC m=+12.718716905 (2 nodes), 3.2µs from start of poll, 166.99876ms from cluster start. Timeout at 2021-07-10 01:05:07.435623403 +0000 UTC m=+12.823781373, 105.064468ms after stability testing.go:149: cluster: 2021-07-10T01:05:07.435Z [INFO] stable state for Follower reached at 2021-07-10 01:05:07.330558955 +0000 UTC m=+12.718716905 (2 nodes), 3.2µs from start of poll, 166.99876ms from cluster start. Timeout at 2021-07-10 01:05:07.435623403 +0000 UTC m=+12.823781373, 105.064468ms after stability testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.435Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: updating configuration: command=RemoveServer server-id=server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 server-addr= servers="[{Suffrage:Voter ID:server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 Address:b76e5ee7-3f04-6b7d-2af0-4a98745184b4} {Suffrage:Voter ID:server-99962721-dd8b-44ac-1b09-7ed704d78cc5 Address:99962721-dd8b-44ac-1b09-7ed704d78cc5}]" testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.435Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: updating configuration: command=RemoveServer server-id=server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 server-addr= servers="[{Suffrage:Voter ID:server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 Address:b76e5ee7-3f04-6b7d-2af0-4a98745184b4} {Suffrage:Voter ID:server-99962721-dd8b-44ac-1b09-7ed704d78cc5 Address:99962721-dd8b-44ac-1b09-7ed704d78cc5}]" testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.436Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: removed peer, stopping replication: peer=server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 last-index=3 testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.436Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: removed peer, stopping replication: peer=server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 last-index=3 testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.436Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: aborting pipeline replication: peer="{Voter server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 0d2493c3-115c-992a-5a45-e72b1f0a85b1}" testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.436Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: aborting pipeline replication: peer="{Voter server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 0d2493c3-115c-992a-5a45-e72b1f0a85b1}" testLoggerAdapter verbose: server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: 2021-07-10T01:05:07.494Z [WARN] server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: heartbeat timeout reached, starting election: last-leader=99962721-dd8b-44ac-1b09-7ed704d78cc5 testing.go:149: server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: 2021-07-10T01:05:07.494Z [WARN] server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: heartbeat timeout reached, starting election: last-leader=99962721-dd8b-44ac-1b09-7ed704d78cc5 testLoggerAdapter verbose: server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: 2021-07-10T01:05:07.494Z [INFO] server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: entering candidate state: node="Node at 0d2493c3-115c-992a-5a45-e72b1f0a85b1 [Candidate]" term=3 testing.go:149: server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: 2021-07-10T01:05:07.494Z [INFO] server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: entering candidate state: node="Node at 0d2493c3-115c-992a-5a45-e72b1f0a85b1 [Candidate]" term=3 testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.495Z [WARN] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: rejecting vote request since we have a leader: from=0d2493c3-115c-992a-5a45-e72b1f0a85b1 leader=99962721-dd8b-44ac-1b09-7ed704d78cc5 testLoggerAdapter verbose: server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4: 2021-07-10T01:05:07.495Z [WARN] server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4: rejecting vote request since we have a leader: from=0d2493c3-115c-992a-5a45-e72b1f0a85b1 leader=99962721-dd8b-44ac-1b09-7ed704d78cc5 testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.495Z [WARN] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: rejecting vote request since we have a leader: from=0d2493c3-115c-992a-5a45-e72b1f0a85b1 leader=99962721-dd8b-44ac-1b09-7ed704d78cc5 testing.go:149: server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4: 2021-07-10T01:05:07.495Z [WARN] server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4: rejecting vote request since we have a leader: from=0d2493c3-115c-992a-5a45-e72b1f0a85b1 leader=99962721-dd8b-44ac-1b09-7ed704d78cc5 testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.541Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: updating configuration: command=AddStaging server-id=server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 server-addr=0d2493c3-115c-992a-5a45-e72b1f0a85b1 servers="[{Suffrage:Voter ID:server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 Address:b76e5ee7-3f04-6b7d-2af0-4a98745184b4} {Suffrage:Voter ID:server-99962721-dd8b-44ac-1b09-7ed704d78cc5 Address:99962721-dd8b-44ac-1b09-7ed704d78cc5} {Suffrage:Voter ID:server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 Address:0d2493c3-115c-992a-5a45-e72b1f0a85b1}]" testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.541Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: updating configuration: command=AddStaging server-id=server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 server-addr=0d2493c3-115c-992a-5a45-e72b1f0a85b1 servers="[{Suffrage:Voter ID:server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 Address:b76e5ee7-3f04-6b7d-2af0-4a98745184b4} {Suffrage:Voter ID:server-99962721-dd8b-44ac-1b09-7ed704d78cc5 Address:99962721-dd8b-44ac-1b09-7ed704d78cc5} {Suffrage:Voter ID:server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 Address:0d2493c3-115c-992a-5a45-e72b1f0a85b1}]" testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.541Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: added peer, starting replication: peer=server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.541Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: added peer, starting replication: peer=server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.541Z [ERROR] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: peer has newer term, stopping replication: peer="{Voter server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 0d2493c3-115c-992a-5a45-e72b1f0a85b1}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:07.541Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.541Z [ERROR] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: peer has newer term, stopping replication: peer="{Voter server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 0d2493c3-115c-992a-5a45-e72b1f0a85b1}" testing.go:149: cluster: 2021-07-10T01:05:07.541Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.542Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: entering follower state: follower="Node at 99962721-dd8b-44ac-1b09-7ed704d78cc5 [Follower]" leader= testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.542Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: entering follower state: follower="Node at 99962721-dd8b-44ac-1b09-7ed704d78cc5 [Follower]" leader= testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.542Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: aborting pipeline replication: peer="{Voter server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 b76e5ee7-3f04-6b7d-2af0-4a98745184b4}" testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.542Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: aborting pipeline replication: peer="{Voter server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 b76e5ee7-3f04-6b7d-2af0-4a98745184b4}" testLoggerAdapter verbose: server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: 2021-07-10T01:05:07.594Z [WARN] server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: Election timeout reached, restarting election testing.go:149: server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: 2021-07-10T01:05:07.594Z [WARN] server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: Election timeout reached, restarting election testLoggerAdapter verbose: server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: 2021-07-10T01:05:07.594Z [INFO] server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: entering candidate state: node="Node at 0d2493c3-115c-992a-5a45-e72b1f0a85b1 [Candidate]" term=4 testing.go:149: server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: 2021-07-10T01:05:07.594Z [INFO] server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: entering candidate state: node="Node at 0d2493c3-115c-992a-5a45-e72b1f0a85b1 [Candidate]" term=4 testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.594Z [WARN] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: rejecting vote request since our last index is greater: candidate=0d2493c3-115c-992a-5a45-e72b1f0a85b1 last-index=4 last-candidate-index=3 testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.594Z [WARN] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: rejecting vote request since our last index is greater: candidate=0d2493c3-115c-992a-5a45-e72b1f0a85b1 last-index=4 last-candidate-index=3 testLoggerAdapter verbose: server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4: 2021-07-10T01:05:07.594Z [WARN] server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4: rejecting vote request since we have a leader: from=0d2493c3-115c-992a-5a45-e72b1f0a85b1 leader=99962721-dd8b-44ac-1b09-7ed704d78cc5 testing.go:149: server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4: 2021-07-10T01:05:07.594Z [WARN] server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4: rejecting vote request since we have a leader: from=0d2493c3-115c-992a-5a45-e72b1f0a85b1 leader=99962721-dd8b-44ac-1b09-7ed704d78cc5 testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.617Z [WARN] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.617Z [WARN] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.617Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: entering candidate state: node="Node at 99962721-dd8b-44ac-1b09-7ed704d78cc5 [Candidate]" term=5 testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.617Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: entering candidate state: node="Node at 99962721-dd8b-44ac-1b09-7ed704d78cc5 [Candidate]" term=5 testLoggerAdapter verbose: server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: 2021-07-10T01:05:07.617Z [INFO] server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: entering follower state: follower="Node at 0d2493c3-115c-992a-5a45-e72b1f0a85b1 [Follower]" leader= testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.617Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: election won: tally=2 testing.go:149: server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: 2021-07-10T01:05:07.617Z [INFO] server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: entering follower state: follower="Node at 0d2493c3-115c-992a-5a45-e72b1f0a85b1 [Follower]" leader= testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.617Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: election won: tally=2 testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.617Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: entering leader state: leader="Node at 99962721-dd8b-44ac-1b09-7ed704d78cc5 [Leader]" testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.617Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: entering leader state: leader="Node at 99962721-dd8b-44ac-1b09-7ed704d78cc5 [Leader]" testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.618Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: added peer, starting replication: peer=server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.618Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: added peer, starting replication: peer=server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.618Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: added peer, starting replication: peer=server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.618Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: added peer, starting replication: peer=server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 testLoggerAdapter verbose: server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: 2021-07-10T01:05:07.618Z [WARN] server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: failed to get previous log: previous-index=4 last-index=3 error="log not found" testing.go:149: server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: 2021-07-10T01:05:07.618Z [WARN] server-0d2493c3-115c-992a-5a45-e72b1f0a85b1: failed to get previous log: previous-index=4 last-index=3 error="log not found" testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.618Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: pipelining replication: peer="{Voter server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 b76e5ee7-3f04-6b7d-2af0-4a98745184b4}" testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.618Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: pipelining replication: peer="{Voter server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 b76e5ee7-3f04-6b7d-2af0-4a98745184b4}" testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.618Z [WARN] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: appendEntries rejected, sending older logs: peer="{Voter server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 0d2493c3-115c-992a-5a45-e72b1f0a85b1}" next=4 testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.618Z [WARN] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: appendEntries rejected, sending older logs: peer="{Voter server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 0d2493c3-115c-992a-5a45-e72b1f0a85b1}" next=4 testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.619Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: pipelining replication: peer="{Voter server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 0d2493c3-115c-992a-5a45-e72b1f0a85b1}" testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.619Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: pipelining replication: peer="{Voter server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 0d2493c3-115c-992a-5a45-e72b1f0a85b1}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:07.723Z [INFO] stable state for Leader reached at 2021-07-10 01:05:07.618001026 +0000 UTC m=+13.006159016 (1 nodes), 75.918241ms from start of poll, 454.440871ms from cluster start. Timeout at 2021-07-10 01:05:07.723069513 +0000 UTC m=+13.111227483, 105.068467ms after stability testing.go:149: cluster: 2021-07-10T01:05:07.723Z [INFO] stable state for Leader reached at 2021-07-10 01:05:07.618001026 +0000 UTC m=+13.006159016 (1 nodes), 75.918241ms from start of poll, 454.440871ms from cluster start. Timeout at 2021-07-10 01:05:07.723069513 +0000 UTC m=+13.111227483, 105.068467ms after stability testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.723Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: aborting pipeline replication: peer="{Voter server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 0d2493c3-115c-992a-5a45-e72b1f0a85b1}" testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.723Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: aborting pipeline replication: peer="{Voter server-0d2493c3-115c-992a-5a45-e72b1f0a85b1 0d2493c3-115c-992a-5a45-e72b1f0a85b1}" testLoggerAdapter verbose: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.723Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: aborting pipeline replication: peer="{Voter server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 b76e5ee7-3f04-6b7d-2af0-4a98745184b4}" testing.go:149: server-99962721-dd8b-44ac-1b09-7ed704d78cc5: 2021-07-10T01:05:07.723Z [INFO] server-99962721-dd8b-44ac-1b09-7ed704d78cc5: aborting pipeline replication: peer="{Voter server-b76e5ee7-3f04-6b7d-2af0-4a98745184b4 b76e5ee7-3f04-6b7d-2af0-4a98745184b4}" --- PASS: TestRaft_ReJoinFollower (0.56s) === RUN TestRaft_LeaderLeaseExpire testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:07.724Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7f0943e4-17b9-a000-0b75-5cba262cdcb7 Address:7f0943e4-17b9-a000-0b75-5cba262cdcb7} {Suffrage:Voter ID:server-edf8725a-05c2-da08-91e3-f04f36db93c4 Address:edf8725a-05c2-da08-91e3-f04f36db93c4}]" testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:07.724Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7f0943e4-17b9-a000-0b75-5cba262cdcb7 Address:7f0943e4-17b9-a000-0b75-5cba262cdcb7} {Suffrage:Voter ID:server-edf8725a-05c2-da08-91e3-f04f36db93c4 Address:edf8725a-05c2-da08-91e3-f04f36db93c4}]" testLoggerAdapter verbose: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:07.724Z [INFO] server-edf8725a-05c2-da08-91e3-f04f36db93c4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7f0943e4-17b9-a000-0b75-5cba262cdcb7 Address:7f0943e4-17b9-a000-0b75-5cba262cdcb7} {Suffrage:Voter ID:server-edf8725a-05c2-da08-91e3-f04f36db93c4 Address:edf8725a-05c2-da08-91e3-f04f36db93c4}]" testing.go:149: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:07.724Z [INFO] server-edf8725a-05c2-da08-91e3-f04f36db93c4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7f0943e4-17b9-a000-0b75-5cba262cdcb7 Address:7f0943e4-17b9-a000-0b75-5cba262cdcb7} {Suffrage:Voter ID:server-edf8725a-05c2-da08-91e3-f04f36db93c4 Address:edf8725a-05c2-da08-91e3-f04f36db93c4}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:07.724Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:07.724Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:07.725Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: entering follower state: follower="Node at 7f0943e4-17b9-a000-0b75-5cba262cdcb7 [Follower]" leader= testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:07.725Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: entering follower state: follower="Node at 7f0943e4-17b9-a000-0b75-5cba262cdcb7 [Follower]" leader= testLoggerAdapter verbose: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:07.725Z [INFO] server-edf8725a-05c2-da08-91e3-f04f36db93c4: entering follower state: follower="Node at edf8725a-05c2-da08-91e3-f04f36db93c4 [Follower]" leader= testing.go:149: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:07.725Z [INFO] server-edf8725a-05c2-da08-91e3-f04f36db93c4: entering follower state: follower="Node at edf8725a-05c2-da08-91e3-f04f36db93c4 [Follower]" leader= testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:07.803Z [WARN] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:07.803Z [WARN] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:07.803Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: entering candidate state: node="Node at 7f0943e4-17b9-a000-0b75-5cba262cdcb7 [Candidate]" term=2 testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:07.803Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: entering candidate state: node="Node at 7f0943e4-17b9-a000-0b75-5cba262cdcb7 [Candidate]" term=2 testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:07.803Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: election won: tally=2 testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:07.803Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: election won: tally=2 testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:07.804Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: entering leader state: leader="Node at 7f0943e4-17b9-a000-0b75-5cba262cdcb7 [Leader]" testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:07.804Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: entering leader state: leader="Node at 7f0943e4-17b9-a000-0b75-5cba262cdcb7 [Leader]" testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:07.804Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: added peer, starting replication: peer=server-edf8725a-05c2-da08-91e3-f04f36db93c4 testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:07.804Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: added peer, starting replication: peer=server-edf8725a-05c2-da08-91e3-f04f36db93c4 testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:07.804Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: pipelining replication: peer="{Voter server-edf8725a-05c2-da08-91e3-f04f36db93c4 edf8725a-05c2-da08-91e3-f04f36db93c4}" testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:07.804Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: pipelining replication: peer="{Voter server-edf8725a-05c2-da08-91e3-f04f36db93c4 edf8725a-05c2-da08-91e3-f04f36db93c4}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:07.909Z [INFO] stable state for Leader reached at 2021-07-10 01:05:07.804114068 +0000 UTC m=+13.192272038 (1 nodes), 79.088494ms from start of poll, 79.691761ms from cluster start. Timeout at 2021-07-10 01:05:07.909205636 +0000 UTC m=+13.297363606, 105.091568ms after stability testing.go:149: cluster: 2021-07-10T01:05:07.909Z [INFO] stable state for Leader reached at 2021-07-10 01:05:07.804114068 +0000 UTC m=+13.192272038 (1 nodes), 79.088494ms from start of poll, 79.691761ms from cluster start. Timeout at 2021-07-10 01:05:07.909205636 +0000 UTC m=+13.297363606, 105.091568ms after stability testLoggerAdapter verbose: cluster: 2021-07-10T01:05:07.914Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-07-10T01:05:07.914Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-07-10T01:05:08.019Z [INFO] stable state for Follower reached at 2021-07-10 01:05:07.914683793 +0000 UTC m=+13.302841763 (1 nodes), 2.32µs from start of poll, 190.261486ms from cluster start. Timeout at 2021-07-10 01:05:08.019766581 +0000 UTC m=+13.407924551, 105.082788ms after stability testing.go:149: cluster: 2021-07-10T01:05:08.019Z [INFO] stable state for Follower reached at 2021-07-10 01:05:07.914683793 +0000 UTC m=+13.302841763 (1 nodes), 2.32µs from start of poll, 190.261486ms from cluster start. Timeout at 2021-07-10 01:05:08.019766581 +0000 UTC m=+13.407924551, 105.082788ms after stability raft_test.go:1492: [INFO] Disconnecting Node at edf8725a-05c2-da08-91e3-f04f36db93c4 [Follower] testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.023Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to heartbeat to: peer=edf8725a-05c2-da08-91e3-f04f36db93c4 error="failed to connect to peer: edf8725a-05c2-da08-91e3-f04f36db93c4" testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.023Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to heartbeat to: peer=edf8725a-05c2-da08-91e3-f04f36db93c4 error="failed to connect to peer: edf8725a-05c2-da08-91e3-f04f36db93c4" testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.025Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to pipeline appendEntries: peer="{Voter server-edf8725a-05c2-da08-91e3-f04f36db93c4 edf8725a-05c2-da08-91e3-f04f36db93c4}" error="append pipeline closed" testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.025Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to pipeline appendEntries: peer="{Voter server-edf8725a-05c2-da08-91e3-f04f36db93c4 edf8725a-05c2-da08-91e3-f04f36db93c4}" error="append pipeline closed" testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.025Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: aborting pipeline replication: peer="{Voter server-edf8725a-05c2-da08-91e3-f04f36db93c4 edf8725a-05c2-da08-91e3-f04f36db93c4}" testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.025Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: aborting pipeline replication: peer="{Voter server-edf8725a-05c2-da08-91e3-f04f36db93c4 edf8725a-05c2-da08-91e3-f04f36db93c4}" testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.033Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to appendEntries to: peer="{Voter server-edf8725a-05c2-da08-91e3-f04f36db93c4 edf8725a-05c2-da08-91e3-f04f36db93c4}" error="failed to connect to peer: edf8725a-05c2-da08-91e3-f04f36db93c4" testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.033Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to appendEntries to: peer="{Voter server-edf8725a-05c2-da08-91e3-f04f36db93c4 edf8725a-05c2-da08-91e3-f04f36db93c4}" error="failed to connect to peer: edf8725a-05c2-da08-91e3-f04f36db93c4" testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.038Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to heartbeat to: peer=edf8725a-05c2-da08-91e3-f04f36db93c4 error="failed to connect to peer: edf8725a-05c2-da08-91e3-f04f36db93c4" testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.038Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to heartbeat to: peer=edf8725a-05c2-da08-91e3-f04f36db93c4 error="failed to connect to peer: edf8725a-05c2-da08-91e3-f04f36db93c4" testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.051Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to appendEntries to: peer="{Voter server-edf8725a-05c2-da08-91e3-f04f36db93c4 edf8725a-05c2-da08-91e3-f04f36db93c4}" error="failed to connect to peer: edf8725a-05c2-da08-91e3-f04f36db93c4" testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.051Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to appendEntries to: peer="{Voter server-edf8725a-05c2-da08-91e3-f04f36db93c4 edf8725a-05c2-da08-91e3-f04f36db93c4}" error="failed to connect to peer: edf8725a-05c2-da08-91e3-f04f36db93c4" testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.056Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to heartbeat to: peer=edf8725a-05c2-da08-91e3-f04f36db93c4 error="failed to connect to peer: edf8725a-05c2-da08-91e3-f04f36db93c4" testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.056Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to heartbeat to: peer=edf8725a-05c2-da08-91e3-f04f36db93c4 error="failed to connect to peer: edf8725a-05c2-da08-91e3-f04f36db93c4" testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.066Z [WARN] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to contact: server-id=server-edf8725a-05c2-da08-91e3-f04f36db93c4 time=50.059728ms testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.066Z [WARN] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to contact: server-id=server-edf8725a-05c2-da08-91e3-f04f36db93c4 time=50.059728ms testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.066Z [WARN] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to contact quorum of nodes, stepping down testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.066Z [WARN] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to contact quorum of nodes, stepping down testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.066Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: entering follower state: follower="Node at 7f0943e4-17b9-a000-0b75-5cba262cdcb7 [Follower]" leader= testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.066Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: entering follower state: follower="Node at 7f0943e4-17b9-a000-0b75-5cba262cdcb7 [Follower]" leader= testLoggerAdapter verbose: cluster: 2021-07-10T01:05:08.066Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:08.066Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.071Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to appendEntries to: peer="{Voter server-edf8725a-05c2-da08-91e3-f04f36db93c4 edf8725a-05c2-da08-91e3-f04f36db93c4}" error="failed to connect to peer: edf8725a-05c2-da08-91e3-f04f36db93c4" testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.071Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to appendEntries to: peer="{Voter server-edf8725a-05c2-da08-91e3-f04f36db93c4 edf8725a-05c2-da08-91e3-f04f36db93c4}" error="failed to connect to peer: edf8725a-05c2-da08-91e3-f04f36db93c4" testLoggerAdapter verbose: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:08.109Z [WARN] server-edf8725a-05c2-da08-91e3-f04f36db93c4: heartbeat timeout reached, starting election: last-leader=7f0943e4-17b9-a000-0b75-5cba262cdcb7 testing.go:149: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:08.109Z [WARN] server-edf8725a-05c2-da08-91e3-f04f36db93c4: heartbeat timeout reached, starting election: last-leader=7f0943e4-17b9-a000-0b75-5cba262cdcb7 testLoggerAdapter verbose: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:08.109Z [INFO] server-edf8725a-05c2-da08-91e3-f04f36db93c4: entering candidate state: node="Node at edf8725a-05c2-da08-91e3-f04f36db93c4 [Candidate]" term=3 testing.go:149: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:08.109Z [INFO] server-edf8725a-05c2-da08-91e3-f04f36db93c4: entering candidate state: node="Node at edf8725a-05c2-da08-91e3-f04f36db93c4 [Candidate]" term=3 testLoggerAdapter verbose: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:08.109Z [ERROR] server-edf8725a-05c2-da08-91e3-f04f36db93c4: failed to make requestVote RPC: target="{Voter server-7f0943e4-17b9-a000-0b75-5cba262cdcb7 7f0943e4-17b9-a000-0b75-5cba262cdcb7}" error="failed to connect to peer: 7f0943e4-17b9-a000-0b75-5cba262cdcb7" testing.go:149: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:08.109Z [ERROR] server-edf8725a-05c2-da08-91e3-f04f36db93c4: failed to make requestVote RPC: target="{Voter server-7f0943e4-17b9-a000-0b75-5cba262cdcb7 7f0943e4-17b9-a000-0b75-5cba262cdcb7}" error="failed to connect to peer: 7f0943e4-17b9-a000-0b75-5cba262cdcb7" testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.159Z [WARN] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.159Z [WARN] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.159Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: entering candidate state: node="Node at 7f0943e4-17b9-a000-0b75-5cba262cdcb7 [Candidate]" term=3 testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.159Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: entering candidate state: node="Node at 7f0943e4-17b9-a000-0b75-5cba262cdcb7 [Candidate]" term=3 testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.159Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to make requestVote RPC: target="{Voter server-edf8725a-05c2-da08-91e3-f04f36db93c4 edf8725a-05c2-da08-91e3-f04f36db93c4}" error="failed to connect to peer: edf8725a-05c2-da08-91e3-f04f36db93c4" testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.159Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to make requestVote RPC: target="{Voter server-edf8725a-05c2-da08-91e3-f04f36db93c4 edf8725a-05c2-da08-91e3-f04f36db93c4}" error="failed to connect to peer: edf8725a-05c2-da08-91e3-f04f36db93c4" testLoggerAdapter verbose: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:08.166Z [WARN] server-edf8725a-05c2-da08-91e3-f04f36db93c4: Election timeout reached, restarting election testing.go:149: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:08.166Z [WARN] server-edf8725a-05c2-da08-91e3-f04f36db93c4: Election timeout reached, restarting election testLoggerAdapter verbose: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:08.166Z [INFO] server-edf8725a-05c2-da08-91e3-f04f36db93c4: entering candidate state: node="Node at edf8725a-05c2-da08-91e3-f04f36db93c4 [Candidate]" term=4 testing.go:149: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:08.166Z [INFO] server-edf8725a-05c2-da08-91e3-f04f36db93c4: entering candidate state: node="Node at edf8725a-05c2-da08-91e3-f04f36db93c4 [Candidate]" term=4 testLoggerAdapter verbose: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:08.166Z [ERROR] server-edf8725a-05c2-da08-91e3-f04f36db93c4: failed to make requestVote RPC: target="{Voter server-7f0943e4-17b9-a000-0b75-5cba262cdcb7 7f0943e4-17b9-a000-0b75-5cba262cdcb7}" error="failed to connect to peer: 7f0943e4-17b9-a000-0b75-5cba262cdcb7" testing.go:149: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:08.166Z [ERROR] server-edf8725a-05c2-da08-91e3-f04f36db93c4: failed to make requestVote RPC: target="{Voter server-7f0943e4-17b9-a000-0b75-5cba262cdcb7 7f0943e4-17b9-a000-0b75-5cba262cdcb7}" error="failed to connect to peer: 7f0943e4-17b9-a000-0b75-5cba262cdcb7" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:08.215Z [INFO] stable state for Leader reached at 2021-07-10 01:05:08.109883391 +0000 UTC m=+13.498041381 (0 nodes), 43.147035ms from start of poll, 385.461104ms from cluster start. Timeout at 2021-07-10 01:05:08.214978619 +0000 UTC m=+13.603136589, 105.095208ms after stability testing.go:149: cluster: 2021-07-10T01:05:08.215Z [INFO] stable state for Leader reached at 2021-07-10 01:05:08.109883391 +0000 UTC m=+13.498041381 (0 nodes), 43.147035ms from start of poll, 385.461104ms from cluster start. Timeout at 2021-07-10 01:05:08.214978619 +0000 UTC m=+13.603136589, 105.095208ms after stability testLoggerAdapter verbose: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:08.244Z [WARN] server-edf8725a-05c2-da08-91e3-f04f36db93c4: Election timeout reached, restarting election testing.go:149: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:08.244Z [WARN] server-edf8725a-05c2-da08-91e3-f04f36db93c4: Election timeout reached, restarting election testLoggerAdapter verbose: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:08.244Z [INFO] server-edf8725a-05c2-da08-91e3-f04f36db93c4: entering candidate state: node="Node at edf8725a-05c2-da08-91e3-f04f36db93c4 [Candidate]" term=5 testing.go:149: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:08.244Z [INFO] server-edf8725a-05c2-da08-91e3-f04f36db93c4: entering candidate state: node="Node at edf8725a-05c2-da08-91e3-f04f36db93c4 [Candidate]" term=5 testLoggerAdapter verbose: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:08.245Z [ERROR] server-edf8725a-05c2-da08-91e3-f04f36db93c4: failed to make requestVote RPC: target="{Voter server-7f0943e4-17b9-a000-0b75-5cba262cdcb7 7f0943e4-17b9-a000-0b75-5cba262cdcb7}" error="failed to connect to peer: 7f0943e4-17b9-a000-0b75-5cba262cdcb7" testing.go:149: server-edf8725a-05c2-da08-91e3-f04f36db93c4: 2021-07-10T01:05:08.245Z [ERROR] server-edf8725a-05c2-da08-91e3-f04f36db93c4: failed to make requestVote RPC: target="{Voter server-7f0943e4-17b9-a000-0b75-5cba262cdcb7 7f0943e4-17b9-a000-0b75-5cba262cdcb7}" error="failed to connect to peer: 7f0943e4-17b9-a000-0b75-5cba262cdcb7" testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.251Z [WARN] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: Election timeout reached, restarting election testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.251Z [WARN] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: Election timeout reached, restarting election testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.251Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: entering candidate state: node="Node at 7f0943e4-17b9-a000-0b75-5cba262cdcb7 [Candidate]" term=4 testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.251Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: entering candidate state: node="Node at 7f0943e4-17b9-a000-0b75-5cba262cdcb7 [Candidate]" term=4 testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.251Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to make requestVote RPC: target="{Voter server-edf8725a-05c2-da08-91e3-f04f36db93c4 edf8725a-05c2-da08-91e3-f04f36db93c4}" error="failed to connect to peer: edf8725a-05c2-da08-91e3-f04f36db93c4" testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.251Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to make requestVote RPC: target="{Voter server-edf8725a-05c2-da08-91e3-f04f36db93c4 edf8725a-05c2-da08-91e3-f04f36db93c4}" error="failed to connect to peer: edf8725a-05c2-da08-91e3-f04f36db93c4" testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.313Z [WARN] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: Election timeout reached, restarting election testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.313Z [WARN] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: Election timeout reached, restarting election testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.314Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: entering candidate state: node="Node at 7f0943e4-17b9-a000-0b75-5cba262cdcb7 [Candidate]" term=5 testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.314Z [INFO] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: entering candidate state: node="Node at 7f0943e4-17b9-a000-0b75-5cba262cdcb7 [Candidate]" term=5 testLoggerAdapter verbose: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.314Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to make requestVote RPC: target="{Voter server-edf8725a-05c2-da08-91e3-f04f36db93c4 edf8725a-05c2-da08-91e3-f04f36db93c4}" error="failed to connect to peer: edf8725a-05c2-da08-91e3-f04f36db93c4" testing.go:149: server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: 2021-07-10T01:05:08.314Z [ERROR] server-7f0943e4-17b9-a000-0b75-5cba262cdcb7: failed to make requestVote RPC: target="{Voter server-edf8725a-05c2-da08-91e3-f04f36db93c4 edf8725a-05c2-da08-91e3-f04f36db93c4}" error="failed to connect to peer: edf8725a-05c2-da08-91e3-f04f36db93c4" --- PASS: TestRaft_LeaderLeaseExpire (0.60s) === RUN TestRaft_Barrier testLoggerAdapter verbose: server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a: 2021-07-10T01:05:08.321Z [INFO] server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a Address:6c11a2fe-1efc-a2f9-acb4-068a2c4b024a} {Suffrage:Voter ID:server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7 Address:3ade38d3-0204-eb9e-f3db-53d78f60bfe7} {Suffrage:Voter ID:server-a01aed58-07aa-0ace-493c-e9efea909c1a Address:a01aed58-07aa-0ace-493c-e9efea909c1a}]" testing.go:149: server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a: 2021-07-10T01:05:08.321Z [INFO] server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a Address:6c11a2fe-1efc-a2f9-acb4-068a2c4b024a} {Suffrage:Voter ID:server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7 Address:3ade38d3-0204-eb9e-f3db-53d78f60bfe7} {Suffrage:Voter ID:server-a01aed58-07aa-0ace-493c-e9efea909c1a Address:a01aed58-07aa-0ace-493c-e9efea909c1a}]" testLoggerAdapter verbose: server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a: 2021-07-10T01:05:08.321Z [INFO] server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a: entering follower state: follower="Node at 6c11a2fe-1efc-a2f9-acb4-068a2c4b024a [Follower]" leader= testing.go:149: server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a: 2021-07-10T01:05:08.321Z [INFO] server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a: entering follower state: follower="Node at 6c11a2fe-1efc-a2f9-acb4-068a2c4b024a [Follower]" leader= testLoggerAdapter verbose: server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7: 2021-07-10T01:05:08.321Z [INFO] server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a Address:6c11a2fe-1efc-a2f9-acb4-068a2c4b024a} {Suffrage:Voter ID:server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7 Address:3ade38d3-0204-eb9e-f3db-53d78f60bfe7} {Suffrage:Voter ID:server-a01aed58-07aa-0ace-493c-e9efea909c1a Address:a01aed58-07aa-0ace-493c-e9efea909c1a}]" testing.go:149: server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7: 2021-07-10T01:05:08.321Z [INFO] server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a Address:6c11a2fe-1efc-a2f9-acb4-068a2c4b024a} {Suffrage:Voter ID:server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7 Address:3ade38d3-0204-eb9e-f3db-53d78f60bfe7} {Suffrage:Voter ID:server-a01aed58-07aa-0ace-493c-e9efea909c1a Address:a01aed58-07aa-0ace-493c-e9efea909c1a}]" testLoggerAdapter verbose: server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7: 2021-07-10T01:05:08.321Z [INFO] server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7: entering follower state: follower="Node at 3ade38d3-0204-eb9e-f3db-53d78f60bfe7 [Follower]" leader= testing.go:149: server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7: 2021-07-10T01:05:08.321Z [INFO] server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7: entering follower state: follower="Node at 3ade38d3-0204-eb9e-f3db-53d78f60bfe7 [Follower]" leader= testLoggerAdapter verbose: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.322Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a Address:6c11a2fe-1efc-a2f9-acb4-068a2c4b024a} {Suffrage:Voter ID:server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7 Address:3ade38d3-0204-eb9e-f3db-53d78f60bfe7} {Suffrage:Voter ID:server-a01aed58-07aa-0ace-493c-e9efea909c1a Address:a01aed58-07aa-0ace-493c-e9efea909c1a}]" testing.go:149: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.322Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a Address:6c11a2fe-1efc-a2f9-acb4-068a2c4b024a} {Suffrage:Voter ID:server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7 Address:3ade38d3-0204-eb9e-f3db-53d78f60bfe7} {Suffrage:Voter ID:server-a01aed58-07aa-0ace-493c-e9efea909c1a Address:a01aed58-07aa-0ace-493c-e9efea909c1a}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:08.322Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:08.322Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.322Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: entering follower state: follower="Node at a01aed58-07aa-0ace-493c-e9efea909c1a [Follower]" leader= testing.go:149: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.322Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: entering follower state: follower="Node at a01aed58-07aa-0ace-493c-e9efea909c1a [Follower]" leader= testLoggerAdapter verbose: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.383Z [WARN] server-a01aed58-07aa-0ace-493c-e9efea909c1a: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.383Z [WARN] server-a01aed58-07aa-0ace-493c-e9efea909c1a: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.383Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: entering candidate state: node="Node at a01aed58-07aa-0ace-493c-e9efea909c1a [Candidate]" term=2 testing.go:149: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.383Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: entering candidate state: node="Node at a01aed58-07aa-0ace-493c-e9efea909c1a [Candidate]" term=2 testLoggerAdapter verbose: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.383Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: election won: tally=2 testing.go:149: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.383Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: election won: tally=2 testLoggerAdapter verbose: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.383Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: entering leader state: leader="Node at a01aed58-07aa-0ace-493c-e9efea909c1a [Leader]" testing.go:149: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.383Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: entering leader state: leader="Node at a01aed58-07aa-0ace-493c-e9efea909c1a [Leader]" testLoggerAdapter verbose: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.383Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: added peer, starting replication: peer=server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a testing.go:149: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.383Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: added peer, starting replication: peer=server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a testLoggerAdapter verbose: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.383Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: added peer, starting replication: peer=server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7 testing.go:149: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.383Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: added peer, starting replication: peer=server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7 testLoggerAdapter verbose: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.383Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: pipelining replication: peer="{Voter server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a 6c11a2fe-1efc-a2f9-acb4-068a2c4b024a}" testing.go:149: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.383Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: pipelining replication: peer="{Voter server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a 6c11a2fe-1efc-a2f9-acb4-068a2c4b024a}" testLoggerAdapter verbose: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.383Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: pipelining replication: peer="{Voter server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7 3ade38d3-0204-eb9e-f3db-53d78f60bfe7}" testing.go:149: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.383Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: pipelining replication: peer="{Voter server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7 3ade38d3-0204-eb9e-f3db-53d78f60bfe7}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:08.488Z [INFO] stable state for Leader reached at 2021-07-10 01:05:08.383663857 +0000 UTC m=+13.771821807 (1 nodes), 61.413707ms from start of poll, 62.306136ms from cluster start. Timeout at 2021-07-10 01:05:08.488728565 +0000 UTC m=+13.876886535, 105.064728ms after stability testing.go:149: cluster: 2021-07-10T01:05:08.488Z [INFO] stable state for Leader reached at 2021-07-10 01:05:08.383663857 +0000 UTC m=+13.771821807 (1 nodes), 61.413707ms from start of poll, 62.306136ms from cluster start. Timeout at 2021-07-10 01:05:08.488728565 +0000 UTC m=+13.876886535, 105.064728ms after stability testLoggerAdapter verbose: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.501Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: aborting pipeline replication: peer="{Voter server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7 3ade38d3-0204-eb9e-f3db-53d78f60bfe7}" testing.go:149: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.501Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: aborting pipeline replication: peer="{Voter server-3ade38d3-0204-eb9e-f3db-53d78f60bfe7 3ade38d3-0204-eb9e-f3db-53d78f60bfe7}" testLoggerAdapter verbose: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.501Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: aborting pipeline replication: peer="{Voter server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a 6c11a2fe-1efc-a2f9-acb4-068a2c4b024a}" testing.go:149: server-a01aed58-07aa-0ace-493c-e9efea909c1a: 2021-07-10T01:05:08.501Z [INFO] server-a01aed58-07aa-0ace-493c-e9efea909c1a: aborting pipeline replication: peer="{Voter server-6c11a2fe-1efc-a2f9-acb4-068a2c4b024a 6c11a2fe-1efc-a2f9-acb4-068a2c4b024a}" --- PASS: TestRaft_Barrier (0.18s) === RUN TestRaft_VerifyLeader testLoggerAdapter verbose: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.503Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f80ef73e-f1d6-f137-0763-2550381591d3 Address:f80ef73e-f1d6-f137-0763-2550381591d3} {Suffrage:Voter ID:server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f Address:d7aee95d-d803-bb31-5fe2-03f7fc53ce0f} {Suffrage:Voter ID:server-6533e748-739f-3530-cef8-f8240bfb4c4d Address:6533e748-739f-3530-cef8-f8240bfb4c4d}]" testing.go:149: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.503Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f80ef73e-f1d6-f137-0763-2550381591d3 Address:f80ef73e-f1d6-f137-0763-2550381591d3} {Suffrage:Voter ID:server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f Address:d7aee95d-d803-bb31-5fe2-03f7fc53ce0f} {Suffrage:Voter ID:server-6533e748-739f-3530-cef8-f8240bfb4c4d Address:6533e748-739f-3530-cef8-f8240bfb4c4d}]" testLoggerAdapter verbose: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.503Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: entering follower state: follower="Node at f80ef73e-f1d6-f137-0763-2550381591d3 [Follower]" leader= testing.go:149: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.503Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: entering follower state: follower="Node at f80ef73e-f1d6-f137-0763-2550381591d3 [Follower]" leader= testLoggerAdapter verbose: server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f: 2021-07-10T01:05:08.503Z [INFO] server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f80ef73e-f1d6-f137-0763-2550381591d3 Address:f80ef73e-f1d6-f137-0763-2550381591d3} {Suffrage:Voter ID:server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f Address:d7aee95d-d803-bb31-5fe2-03f7fc53ce0f} {Suffrage:Voter ID:server-6533e748-739f-3530-cef8-f8240bfb4c4d Address:6533e748-739f-3530-cef8-f8240bfb4c4d}]" testing.go:149: server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f: 2021-07-10T01:05:08.503Z [INFO] server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f80ef73e-f1d6-f137-0763-2550381591d3 Address:f80ef73e-f1d6-f137-0763-2550381591d3} {Suffrage:Voter ID:server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f Address:d7aee95d-d803-bb31-5fe2-03f7fc53ce0f} {Suffrage:Voter ID:server-6533e748-739f-3530-cef8-f8240bfb4c4d Address:6533e748-739f-3530-cef8-f8240bfb4c4d}]" testLoggerAdapter verbose: server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f: 2021-07-10T01:05:08.503Z [INFO] server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f: entering follower state: follower="Node at d7aee95d-d803-bb31-5fe2-03f7fc53ce0f [Follower]" leader= testing.go:149: server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f: 2021-07-10T01:05:08.503Z [INFO] server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f: entering follower state: follower="Node at d7aee95d-d803-bb31-5fe2-03f7fc53ce0f [Follower]" leader= testLoggerAdapter verbose: server-6533e748-739f-3530-cef8-f8240bfb4c4d: 2021-07-10T01:05:08.503Z [INFO] server-6533e748-739f-3530-cef8-f8240bfb4c4d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f80ef73e-f1d6-f137-0763-2550381591d3 Address:f80ef73e-f1d6-f137-0763-2550381591d3} {Suffrage:Voter ID:server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f Address:d7aee95d-d803-bb31-5fe2-03f7fc53ce0f} {Suffrage:Voter ID:server-6533e748-739f-3530-cef8-f8240bfb4c4d Address:6533e748-739f-3530-cef8-f8240bfb4c4d}]" testing.go:149: server-6533e748-739f-3530-cef8-f8240bfb4c4d: 2021-07-10T01:05:08.503Z [INFO] server-6533e748-739f-3530-cef8-f8240bfb4c4d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f80ef73e-f1d6-f137-0763-2550381591d3 Address:f80ef73e-f1d6-f137-0763-2550381591d3} {Suffrage:Voter ID:server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f Address:d7aee95d-d803-bb31-5fe2-03f7fc53ce0f} {Suffrage:Voter ID:server-6533e748-739f-3530-cef8-f8240bfb4c4d Address:6533e748-739f-3530-cef8-f8240bfb4c4d}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:08.504Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:08.504Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-6533e748-739f-3530-cef8-f8240bfb4c4d: 2021-07-10T01:05:08.504Z [INFO] server-6533e748-739f-3530-cef8-f8240bfb4c4d: entering follower state: follower="Node at 6533e748-739f-3530-cef8-f8240bfb4c4d [Follower]" leader= testing.go:149: server-6533e748-739f-3530-cef8-f8240bfb4c4d: 2021-07-10T01:05:08.504Z [INFO] server-6533e748-739f-3530-cef8-f8240bfb4c4d: entering follower state: follower="Node at 6533e748-739f-3530-cef8-f8240bfb4c4d [Follower]" leader= testLoggerAdapter verbose: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.556Z [WARN] server-f80ef73e-f1d6-f137-0763-2550381591d3: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.556Z [WARN] server-f80ef73e-f1d6-f137-0763-2550381591d3: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.556Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: entering candidate state: node="Node at f80ef73e-f1d6-f137-0763-2550381591d3 [Candidate]" term=2 testing.go:149: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.556Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: entering candidate state: node="Node at f80ef73e-f1d6-f137-0763-2550381591d3 [Candidate]" term=2 testLoggerAdapter verbose: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.556Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: election won: tally=2 testing.go:149: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.556Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: election won: tally=2 testLoggerAdapter verbose: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.557Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: entering leader state: leader="Node at f80ef73e-f1d6-f137-0763-2550381591d3 [Leader]" testing.go:149: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.557Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: entering leader state: leader="Node at f80ef73e-f1d6-f137-0763-2550381591d3 [Leader]" testLoggerAdapter verbose: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.557Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: added peer, starting replication: peer=server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f testing.go:149: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.557Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: added peer, starting replication: peer=server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f testLoggerAdapter verbose: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.557Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: added peer, starting replication: peer=server-6533e748-739f-3530-cef8-f8240bfb4c4d testing.go:149: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.557Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: added peer, starting replication: peer=server-6533e748-739f-3530-cef8-f8240bfb4c4d testLoggerAdapter verbose: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.557Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: pipelining replication: peer="{Voter server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f d7aee95d-d803-bb31-5fe2-03f7fc53ce0f}" testing.go:149: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.557Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: pipelining replication: peer="{Voter server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f d7aee95d-d803-bb31-5fe2-03f7fc53ce0f}" testLoggerAdapter verbose: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.557Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: pipelining replication: peer="{Voter server-6533e748-739f-3530-cef8-f8240bfb4c4d 6533e748-739f-3530-cef8-f8240bfb4c4d}" testing.go:149: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.557Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: pipelining replication: peer="{Voter server-6533e748-739f-3530-cef8-f8240bfb4c4d 6533e748-739f-3530-cef8-f8240bfb4c4d}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:08.662Z [INFO] stable state for Leader reached at 2021-07-10 01:05:08.557147306 +0000 UTC m=+13.945305296 (1 nodes), 53.033919ms from start of poll, 54.022849ms from cluster start. Timeout at 2021-07-10 01:05:08.662232834 +0000 UTC m=+14.050390824, 105.085528ms after stability testing.go:149: cluster: 2021-07-10T01:05:08.662Z [INFO] stable state for Leader reached at 2021-07-10 01:05:08.557147306 +0000 UTC m=+13.945305296 (1 nodes), 53.033919ms from start of poll, 54.022849ms from cluster start. Timeout at 2021-07-10 01:05:08.662232834 +0000 UTC m=+14.050390824, 105.085528ms after stability testLoggerAdapter verbose: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.662Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: aborting pipeline replication: peer="{Voter server-6533e748-739f-3530-cef8-f8240bfb4c4d 6533e748-739f-3530-cef8-f8240bfb4c4d}" testing.go:149: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.662Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: aborting pipeline replication: peer="{Voter server-6533e748-739f-3530-cef8-f8240bfb4c4d 6533e748-739f-3530-cef8-f8240bfb4c4d}" testLoggerAdapter verbose: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.662Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: aborting pipeline replication: peer="{Voter server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f d7aee95d-d803-bb31-5fe2-03f7fc53ce0f}" testing.go:149: server-f80ef73e-f1d6-f137-0763-2550381591d3: 2021-07-10T01:05:08.662Z [INFO] server-f80ef73e-f1d6-f137-0763-2550381591d3: aborting pipeline replication: peer="{Voter server-d7aee95d-d803-bb31-5fe2-03f7fc53ce0f d7aee95d-d803-bb31-5fe2-03f7fc53ce0f}" --- PASS: TestRaft_VerifyLeader (0.16s) === RUN TestRaft_VerifyLeader_Single testLoggerAdapter verbose: server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: 2021-07-10T01:05:08.664Z [INFO] server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572 Address:5a7c4091-dc6f-8fb9-8854-c30bb1eaf572}]" testing.go:149: server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: 2021-07-10T01:05:08.664Z [INFO] server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572 Address:5a7c4091-dc6f-8fb9-8854-c30bb1eaf572}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:08.664Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:08.664Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: 2021-07-10T01:05:08.664Z [INFO] server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: entering follower state: follower="Node at 5a7c4091-dc6f-8fb9-8854-c30bb1eaf572 [Follower]" leader= testing.go:149: server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: 2021-07-10T01:05:08.664Z [INFO] server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: entering follower state: follower="Node at 5a7c4091-dc6f-8fb9-8854-c30bb1eaf572 [Follower]" leader= testLoggerAdapter verbose: server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: 2021-07-10T01:05:08.720Z [WARN] server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: 2021-07-10T01:05:08.720Z [WARN] server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: 2021-07-10T01:05:08.721Z [INFO] server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: entering candidate state: node="Node at 5a7c4091-dc6f-8fb9-8854-c30bb1eaf572 [Candidate]" term=2 testing.go:149: server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: 2021-07-10T01:05:08.721Z [INFO] server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: entering candidate state: node="Node at 5a7c4091-dc6f-8fb9-8854-c30bb1eaf572 [Candidate]" term=2 testLoggerAdapter verbose: server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: 2021-07-10T01:05:08.721Z [INFO] server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: election won: tally=1 testing.go:149: server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: 2021-07-10T01:05:08.721Z [INFO] server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: election won: tally=1 testLoggerAdapter verbose: server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: 2021-07-10T01:05:08.721Z [INFO] server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: entering leader state: leader="Node at 5a7c4091-dc6f-8fb9-8854-c30bb1eaf572 [Leader]" testing.go:149: server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: 2021-07-10T01:05:08.721Z [INFO] server-5a7c4091-dc6f-8fb9-8854-c30bb1eaf572: entering leader state: leader="Node at 5a7c4091-dc6f-8fb9-8854-c30bb1eaf572 [Leader]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:08.826Z [INFO] stable state for Leader reached at 2021-07-10 01:05:08.721306097 +0000 UTC m=+14.109464127 (1 nodes), 57.096002ms from start of poll, 57.588787ms from cluster start. Timeout at 2021-07-10 01:05:08.826391765 +0000 UTC m=+14.214549735, 105.085608ms after stability testing.go:149: cluster: 2021-07-10T01:05:08.826Z [INFO] stable state for Leader reached at 2021-07-10 01:05:08.721306097 +0000 UTC m=+14.109464127 (1 nodes), 57.096002ms from start of poll, 57.588787ms from cluster start. Timeout at 2021-07-10 01:05:08.826391765 +0000 UTC m=+14.214549735, 105.085608ms after stability --- PASS: TestRaft_VerifyLeader_Single (0.16s) === RUN TestRaft_VerifyLeader_Fail testLoggerAdapter verbose: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:08.828Z [INFO] server-49248667-6b34-68b4-1b72-125153ca0cd9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49248667-6b34-68b4-1b72-125153ca0cd9 Address:49248667-6b34-68b4-1b72-125153ca0cd9} {Suffrage:Voter ID:server-5d8b8708-f49b-4d49-a364-61238cdaf4ea Address:5d8b8708-f49b-4d49-a364-61238cdaf4ea}]" testing.go:149: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:08.828Z [INFO] server-49248667-6b34-68b4-1b72-125153ca0cd9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49248667-6b34-68b4-1b72-125153ca0cd9 Address:49248667-6b34-68b4-1b72-125153ca0cd9} {Suffrage:Voter ID:server-5d8b8708-f49b-4d49-a364-61238cdaf4ea Address:5d8b8708-f49b-4d49-a364-61238cdaf4ea}]" testLoggerAdapter verbose: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:08.829Z [INFO] server-49248667-6b34-68b4-1b72-125153ca0cd9: entering follower state: follower="Node at 49248667-6b34-68b4-1b72-125153ca0cd9 [Follower]" leader= testing.go:149: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:08.829Z [INFO] server-49248667-6b34-68b4-1b72-125153ca0cd9: entering follower state: follower="Node at 49248667-6b34-68b4-1b72-125153ca0cd9 [Follower]" leader= testLoggerAdapter verbose: server-5d8b8708-f49b-4d49-a364-61238cdaf4ea: 2021-07-10T01:05:08.829Z [INFO] server-5d8b8708-f49b-4d49-a364-61238cdaf4ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49248667-6b34-68b4-1b72-125153ca0cd9 Address:49248667-6b34-68b4-1b72-125153ca0cd9} {Suffrage:Voter ID:server-5d8b8708-f49b-4d49-a364-61238cdaf4ea Address:5d8b8708-f49b-4d49-a364-61238cdaf4ea}]" testing.go:149: server-5d8b8708-f49b-4d49-a364-61238cdaf4ea: 2021-07-10T01:05:08.829Z [INFO] server-5d8b8708-f49b-4d49-a364-61238cdaf4ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49248667-6b34-68b4-1b72-125153ca0cd9 Address:49248667-6b34-68b4-1b72-125153ca0cd9} {Suffrage:Voter ID:server-5d8b8708-f49b-4d49-a364-61238cdaf4ea Address:5d8b8708-f49b-4d49-a364-61238cdaf4ea}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:08.829Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:08.829Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-5d8b8708-f49b-4d49-a364-61238cdaf4ea: 2021-07-10T01:05:08.829Z [INFO] server-5d8b8708-f49b-4d49-a364-61238cdaf4ea: entering follower state: follower="Node at 5d8b8708-f49b-4d49-a364-61238cdaf4ea [Follower]" leader= testing.go:149: server-5d8b8708-f49b-4d49-a364-61238cdaf4ea: 2021-07-10T01:05:08.829Z [INFO] server-5d8b8708-f49b-4d49-a364-61238cdaf4ea: entering follower state: follower="Node at 5d8b8708-f49b-4d49-a364-61238cdaf4ea [Follower]" leader= testLoggerAdapter verbose: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:08.898Z [WARN] server-49248667-6b34-68b4-1b72-125153ca0cd9: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:08.898Z [WARN] server-49248667-6b34-68b4-1b72-125153ca0cd9: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:08.898Z [INFO] server-49248667-6b34-68b4-1b72-125153ca0cd9: entering candidate state: node="Node at 49248667-6b34-68b4-1b72-125153ca0cd9 [Candidate]" term=2 testing.go:149: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:08.898Z [INFO] server-49248667-6b34-68b4-1b72-125153ca0cd9: entering candidate state: node="Node at 49248667-6b34-68b4-1b72-125153ca0cd9 [Candidate]" term=2 testLoggerAdapter verbose: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:08.898Z [INFO] server-49248667-6b34-68b4-1b72-125153ca0cd9: election won: tally=2 testing.go:149: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:08.898Z [INFO] server-49248667-6b34-68b4-1b72-125153ca0cd9: election won: tally=2 testLoggerAdapter verbose: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:08.898Z [INFO] server-49248667-6b34-68b4-1b72-125153ca0cd9: entering leader state: leader="Node at 49248667-6b34-68b4-1b72-125153ca0cd9 [Leader]" testing.go:149: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:08.898Z [INFO] server-49248667-6b34-68b4-1b72-125153ca0cd9: entering leader state: leader="Node at 49248667-6b34-68b4-1b72-125153ca0cd9 [Leader]" testLoggerAdapter verbose: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:08.898Z [INFO] server-49248667-6b34-68b4-1b72-125153ca0cd9: added peer, starting replication: peer=server-5d8b8708-f49b-4d49-a364-61238cdaf4ea testing.go:149: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:08.898Z [INFO] server-49248667-6b34-68b4-1b72-125153ca0cd9: added peer, starting replication: peer=server-5d8b8708-f49b-4d49-a364-61238cdaf4ea testLoggerAdapter verbose: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:08.899Z [INFO] server-49248667-6b34-68b4-1b72-125153ca0cd9: pipelining replication: peer="{Voter server-5d8b8708-f49b-4d49-a364-61238cdaf4ea 5d8b8708-f49b-4d49-a364-61238cdaf4ea}" testing.go:149: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:08.899Z [INFO] server-49248667-6b34-68b4-1b72-125153ca0cd9: pipelining replication: peer="{Voter server-5d8b8708-f49b-4d49-a364-61238cdaf4ea 5d8b8708-f49b-4d49-a364-61238cdaf4ea}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:09.004Z [INFO] stable state for Leader reached at 2021-07-10 01:05:08.89892045 +0000 UTC m=+14.287078440 (1 nodes), 69.635135ms from start of poll, 70.353762ms from cluster start. Timeout at 2021-07-10 01:05:09.003994117 +0000 UTC m=+14.392152087, 105.073647ms after stability testing.go:149: cluster: 2021-07-10T01:05:09.004Z [INFO] stable state for Leader reached at 2021-07-10 01:05:08.89892045 +0000 UTC m=+14.287078440 (1 nodes), 69.635135ms from start of poll, 70.353762ms from cluster start. Timeout at 2021-07-10 01:05:09.003994117 +0000 UTC m=+14.392152087, 105.073647ms after stability testLoggerAdapter verbose: cluster: 2021-07-10T01:05:09.004Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-07-10T01:05:09.004Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-07-10T01:05:09.109Z [INFO] stable state for Follower reached at 2021-07-10 01:05:09.004318801 +0000 UTC m=+14.392476751 (1 nodes), 1.86µs from start of poll, 175.752073ms from cluster start. Timeout at 2021-07-10 01:05:09.109388488 +0000 UTC m=+14.497546458, 105.069707ms after stability testing.go:149: cluster: 2021-07-10T01:05:09.109Z [INFO] stable state for Follower reached at 2021-07-10 01:05:09.004318801 +0000 UTC m=+14.392476751 (1 nodes), 1.86µs from start of poll, 175.752073ms from cluster start. Timeout at 2021-07-10 01:05:09.109388488 +0000 UTC m=+14.497546458, 105.069707ms after stability testLoggerAdapter verbose: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:09.109Z [WARN] server-49248667-6b34-68b4-1b72-125153ca0cd9: new leader elected, stepping down testing.go:149: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:09.109Z [WARN] server-49248667-6b34-68b4-1b72-125153ca0cd9: new leader elected, stepping down testLoggerAdapter verbose: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:09.109Z [INFO] server-49248667-6b34-68b4-1b72-125153ca0cd9: entering follower state: follower="Node at 49248667-6b34-68b4-1b72-125153ca0cd9 [Shutdown]" leader= testing.go:149: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:09.109Z [INFO] server-49248667-6b34-68b4-1b72-125153ca0cd9: entering follower state: follower="Node at 49248667-6b34-68b4-1b72-125153ca0cd9 [Shutdown]" leader= testLoggerAdapter verbose: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:09.109Z [INFO] server-49248667-6b34-68b4-1b72-125153ca0cd9: aborting pipeline replication: peer="{Voter server-5d8b8708-f49b-4d49-a364-61238cdaf4ea 5d8b8708-f49b-4d49-a364-61238cdaf4ea}" testing.go:149: server-49248667-6b34-68b4-1b72-125153ca0cd9: 2021-07-10T01:05:09.109Z [INFO] server-49248667-6b34-68b4-1b72-125153ca0cd9: aborting pipeline replication: peer="{Voter server-5d8b8708-f49b-4d49-a364-61238cdaf4ea 5d8b8708-f49b-4d49-a364-61238cdaf4ea}" --- PASS: TestRaft_VerifyLeader_Fail (0.28s) === RUN TestRaft_VerifyLeader_PartialConnect testLoggerAdapter verbose: server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0: 2021-07-10T01:05:09.111Z [INFO] server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0 Address:0296557a-4dbe-3b4d-b2d5-4c199b97d6d0} {Suffrage:Voter ID:server-463bc25b-73bf-1f88-9ff5-b53a151968a8 Address:463bc25b-73bf-1f88-9ff5-b53a151968a8} {Suffrage:Voter ID:server-a21e9604-f93b-7718-8a1d-7c0f031126d4 Address:a21e9604-f93b-7718-8a1d-7c0f031126d4}]" testing.go:149: server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0: 2021-07-10T01:05:09.111Z [INFO] server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0 Address:0296557a-4dbe-3b4d-b2d5-4c199b97d6d0} {Suffrage:Voter ID:server-463bc25b-73bf-1f88-9ff5-b53a151968a8 Address:463bc25b-73bf-1f88-9ff5-b53a151968a8} {Suffrage:Voter ID:server-a21e9604-f93b-7718-8a1d-7c0f031126d4 Address:a21e9604-f93b-7718-8a1d-7c0f031126d4}]" testLoggerAdapter verbose: server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0: 2021-07-10T01:05:09.111Z [INFO] server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0: entering follower state: follower="Node at 0296557a-4dbe-3b4d-b2d5-4c199b97d6d0 [Follower]" leader= testing.go:149: server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0: 2021-07-10T01:05:09.111Z [INFO] server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0: entering follower state: follower="Node at 0296557a-4dbe-3b4d-b2d5-4c199b97d6d0 [Follower]" leader= testLoggerAdapter verbose: server-463bc25b-73bf-1f88-9ff5-b53a151968a8: 2021-07-10T01:05:09.111Z [INFO] server-463bc25b-73bf-1f88-9ff5-b53a151968a8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0 Address:0296557a-4dbe-3b4d-b2d5-4c199b97d6d0} {Suffrage:Voter ID:server-463bc25b-73bf-1f88-9ff5-b53a151968a8 Address:463bc25b-73bf-1f88-9ff5-b53a151968a8} {Suffrage:Voter ID:server-a21e9604-f93b-7718-8a1d-7c0f031126d4 Address:a21e9604-f93b-7718-8a1d-7c0f031126d4}]" testing.go:149: server-463bc25b-73bf-1f88-9ff5-b53a151968a8: 2021-07-10T01:05:09.111Z [INFO] server-463bc25b-73bf-1f88-9ff5-b53a151968a8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0 Address:0296557a-4dbe-3b4d-b2d5-4c199b97d6d0} {Suffrage:Voter ID:server-463bc25b-73bf-1f88-9ff5-b53a151968a8 Address:463bc25b-73bf-1f88-9ff5-b53a151968a8} {Suffrage:Voter ID:server-a21e9604-f93b-7718-8a1d-7c0f031126d4 Address:a21e9604-f93b-7718-8a1d-7c0f031126d4}]" testLoggerAdapter verbose: server-463bc25b-73bf-1f88-9ff5-b53a151968a8: 2021-07-10T01:05:09.111Z [INFO] server-463bc25b-73bf-1f88-9ff5-b53a151968a8: entering follower state: follower="Node at 463bc25b-73bf-1f88-9ff5-b53a151968a8 [Follower]" leader= testing.go:149: server-463bc25b-73bf-1f88-9ff5-b53a151968a8: 2021-07-10T01:05:09.111Z [INFO] server-463bc25b-73bf-1f88-9ff5-b53a151968a8: entering follower state: follower="Node at 463bc25b-73bf-1f88-9ff5-b53a151968a8 [Follower]" leader= testLoggerAdapter verbose: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.112Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0 Address:0296557a-4dbe-3b4d-b2d5-4c199b97d6d0} {Suffrage:Voter ID:server-463bc25b-73bf-1f88-9ff5-b53a151968a8 Address:463bc25b-73bf-1f88-9ff5-b53a151968a8} {Suffrage:Voter ID:server-a21e9604-f93b-7718-8a1d-7c0f031126d4 Address:a21e9604-f93b-7718-8a1d-7c0f031126d4}]" testing.go:149: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.112Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0 Address:0296557a-4dbe-3b4d-b2d5-4c199b97d6d0} {Suffrage:Voter ID:server-463bc25b-73bf-1f88-9ff5-b53a151968a8 Address:463bc25b-73bf-1f88-9ff5-b53a151968a8} {Suffrage:Voter ID:server-a21e9604-f93b-7718-8a1d-7c0f031126d4 Address:a21e9604-f93b-7718-8a1d-7c0f031126d4}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:09.112Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:09.112Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.112Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: entering follower state: follower="Node at a21e9604-f93b-7718-8a1d-7c0f031126d4 [Follower]" leader= testing.go:149: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.112Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: entering follower state: follower="Node at a21e9604-f93b-7718-8a1d-7c0f031126d4 [Follower]" leader= testLoggerAdapter verbose: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.164Z [WARN] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.164Z [WARN] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.164Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: entering candidate state: node="Node at a21e9604-f93b-7718-8a1d-7c0f031126d4 [Candidate]" term=2 testing.go:149: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.164Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: entering candidate state: node="Node at a21e9604-f93b-7718-8a1d-7c0f031126d4 [Candidate]" term=2 testLoggerAdapter verbose: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.164Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: election won: tally=2 testing.go:149: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.164Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: election won: tally=2 testLoggerAdapter verbose: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.164Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: entering leader state: leader="Node at a21e9604-f93b-7718-8a1d-7c0f031126d4 [Leader]" testing.go:149: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.164Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: entering leader state: leader="Node at a21e9604-f93b-7718-8a1d-7c0f031126d4 [Leader]" testLoggerAdapter verbose: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.164Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: added peer, starting replication: peer=server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0 testing.go:149: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.164Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: added peer, starting replication: peer=server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0 testLoggerAdapter verbose: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.164Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: added peer, starting replication: peer=server-463bc25b-73bf-1f88-9ff5-b53a151968a8 testing.go:149: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.164Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: added peer, starting replication: peer=server-463bc25b-73bf-1f88-9ff5-b53a151968a8 testLoggerAdapter verbose: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.164Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: pipelining replication: peer="{Voter server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0 0296557a-4dbe-3b4d-b2d5-4c199b97d6d0}" testing.go:149: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.164Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: pipelining replication: peer="{Voter server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0 0296557a-4dbe-3b4d-b2d5-4c199b97d6d0}" testLoggerAdapter verbose: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.164Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: pipelining replication: peer="{Voter server-463bc25b-73bf-1f88-9ff5-b53a151968a8 463bc25b-73bf-1f88-9ff5-b53a151968a8}" testing.go:149: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.164Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: pipelining replication: peer="{Voter server-463bc25b-73bf-1f88-9ff5-b53a151968a8 463bc25b-73bf-1f88-9ff5-b53a151968a8}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:09.269Z [INFO] stable state for Leader reached at 2021-07-10 01:05:09.164669171 +0000 UTC m=+14.552827181 (1 nodes), 52.480873ms from start of poll, 53.532344ms from cluster start. Timeout at 2021-07-10 01:05:09.269750999 +0000 UTC m=+14.657908969, 105.081788ms after stability testing.go:149: cluster: 2021-07-10T01:05:09.269Z [INFO] stable state for Leader reached at 2021-07-10 01:05:09.164669171 +0000 UTC m=+14.552827181 (1 nodes), 52.480873ms from start of poll, 53.532344ms from cluster start. Timeout at 2021-07-10 01:05:09.269750999 +0000 UTC m=+14.657908969, 105.081788ms after stability testLoggerAdapter verbose: cluster: 2021-07-10T01:05:09.275Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-07-10T01:05:09.275Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-07-10T01:05:09.380Z [INFO] stable state for Follower reached at 2021-07-10 01:05:09.275163356 +0000 UTC m=+14.663321306 (2 nodes), 2.86µs from start of poll, 164.026469ms from cluster start. Timeout at 2021-07-10 01:05:09.380229664 +0000 UTC m=+14.768387634, 105.066328ms after stability testing.go:149: cluster: 2021-07-10T01:05:09.380Z [INFO] stable state for Follower reached at 2021-07-10 01:05:09.275163356 +0000 UTC m=+14.663321306 (2 nodes), 2.86µs from start of poll, 164.026469ms from cluster start. Timeout at 2021-07-10 01:05:09.380229664 +0000 UTC m=+14.768387634, 105.066328ms after stability raft_test.go:1647: [INFO] Disconnecting Node at 0296557a-4dbe-3b4d-b2d5-4c199b97d6d0 [Follower] testLoggerAdapter verbose: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.380Z [ERROR] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: failed to heartbeat to: peer=0296557a-4dbe-3b4d-b2d5-4c199b97d6d0 error="failed to connect to peer: 0296557a-4dbe-3b4d-b2d5-4c199b97d6d0" testing.go:149: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.380Z [ERROR] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: failed to heartbeat to: peer=0296557a-4dbe-3b4d-b2d5-4c199b97d6d0 error="failed to connect to peer: 0296557a-4dbe-3b4d-b2d5-4c199b97d6d0" testLoggerAdapter verbose: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.380Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: aborting pipeline replication: peer="{Voter server-463bc25b-73bf-1f88-9ff5-b53a151968a8 463bc25b-73bf-1f88-9ff5-b53a151968a8}" testing.go:149: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.380Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: aborting pipeline replication: peer="{Voter server-463bc25b-73bf-1f88-9ff5-b53a151968a8 463bc25b-73bf-1f88-9ff5-b53a151968a8}" testLoggerAdapter verbose: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.380Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: aborting pipeline replication: peer="{Voter server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0 0296557a-4dbe-3b4d-b2d5-4c199b97d6d0}" testing.go:149: server-a21e9604-f93b-7718-8a1d-7c0f031126d4: 2021-07-10T01:05:09.380Z [INFO] server-a21e9604-f93b-7718-8a1d-7c0f031126d4: aborting pipeline replication: peer="{Voter server-0296557a-4dbe-3b4d-b2d5-4c199b97d6d0 0296557a-4dbe-3b4d-b2d5-4c199b97d6d0}" --- PASS: TestRaft_VerifyLeader_PartialConnect (0.27s) === RUN TestRaft_StartAsLeader testLoggerAdapter verbose: server-7b8f9ddf-b93c-2668-7308-bbed0d0d90d5: 2021-07-10T01:05:09.382Z [INFO] server-7b8f9ddf-b93c-2668-7308-bbed0d0d90d5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b8f9ddf-b93c-2668-7308-bbed0d0d90d5 Address:7b8f9ddf-b93c-2668-7308-bbed0d0d90d5}]" testing.go:149: server-7b8f9ddf-b93c-2668-7308-bbed0d0d90d5: 2021-07-10T01:05:09.382Z [INFO] server-7b8f9ddf-b93c-2668-7308-bbed0d0d90d5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b8f9ddf-b93c-2668-7308-bbed0d0d90d5 Address:7b8f9ddf-b93c-2668-7308-bbed0d0d90d5}]" testLoggerAdapter verbose: server-7b8f9ddf-b93c-2668-7308-bbed0d0d90d5: 2021-07-10T01:05:09.382Z [INFO] server-7b8f9ddf-b93c-2668-7308-bbed0d0d90d5: entering leader state: leader="Node at 7b8f9ddf-b93c-2668-7308-bbed0d0d90d5 [Leader]" testing.go:149: server-7b8f9ddf-b93c-2668-7308-bbed0d0d90d5: 2021-07-10T01:05:09.382Z [INFO] server-7b8f9ddf-b93c-2668-7308-bbed0d0d90d5: entering leader state: leader="Node at 7b8f9ddf-b93c-2668-7308-bbed0d0d90d5 [Leader]" --- PASS: TestRaft_StartAsLeader (0.00s) === RUN TestRaft_NotifyCh testLoggerAdapter verbose: server-f8896627-a809-21db-6ab5-129eacf80e8e: 2021-07-10T01:05:09.382Z [INFO] server-f8896627-a809-21db-6ab5-129eacf80e8e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f8896627-a809-21db-6ab5-129eacf80e8e Address:f8896627-a809-21db-6ab5-129eacf80e8e}]" testing.go:149: server-f8896627-a809-21db-6ab5-129eacf80e8e: 2021-07-10T01:05:09.382Z [INFO] server-f8896627-a809-21db-6ab5-129eacf80e8e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f8896627-a809-21db-6ab5-129eacf80e8e Address:f8896627-a809-21db-6ab5-129eacf80e8e}]" testLoggerAdapter verbose: server-f8896627-a809-21db-6ab5-129eacf80e8e: 2021-07-10T01:05:09.383Z [INFO] server-f8896627-a809-21db-6ab5-129eacf80e8e: entering follower state: follower="Node at f8896627-a809-21db-6ab5-129eacf80e8e [Follower]" leader= testing.go:149: server-f8896627-a809-21db-6ab5-129eacf80e8e: 2021-07-10T01:05:09.383Z [INFO] server-f8896627-a809-21db-6ab5-129eacf80e8e: entering follower state: follower="Node at f8896627-a809-21db-6ab5-129eacf80e8e [Follower]" leader= testLoggerAdapter verbose: server-f8896627-a809-21db-6ab5-129eacf80e8e: 2021-07-10T01:05:09.481Z [WARN] server-f8896627-a809-21db-6ab5-129eacf80e8e: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-f8896627-a809-21db-6ab5-129eacf80e8e: 2021-07-10T01:05:09.481Z [WARN] server-f8896627-a809-21db-6ab5-129eacf80e8e: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-f8896627-a809-21db-6ab5-129eacf80e8e: 2021-07-10T01:05:09.481Z [INFO] server-f8896627-a809-21db-6ab5-129eacf80e8e: entering candidate state: node="Node at f8896627-a809-21db-6ab5-129eacf80e8e [Candidate]" term=2 testing.go:149: server-f8896627-a809-21db-6ab5-129eacf80e8e: 2021-07-10T01:05:09.481Z [INFO] server-f8896627-a809-21db-6ab5-129eacf80e8e: entering candidate state: node="Node at f8896627-a809-21db-6ab5-129eacf80e8e [Candidate]" term=2 testLoggerAdapter verbose: server-f8896627-a809-21db-6ab5-129eacf80e8e: 2021-07-10T01:05:09.481Z [INFO] server-f8896627-a809-21db-6ab5-129eacf80e8e: election won: tally=1 testing.go:149: server-f8896627-a809-21db-6ab5-129eacf80e8e: 2021-07-10T01:05:09.481Z [INFO] server-f8896627-a809-21db-6ab5-129eacf80e8e: election won: tally=1 testLoggerAdapter verbose: server-f8896627-a809-21db-6ab5-129eacf80e8e: 2021-07-10T01:05:09.481Z [INFO] server-f8896627-a809-21db-6ab5-129eacf80e8e: entering leader state: leader="Node at f8896627-a809-21db-6ab5-129eacf80e8e [Leader]" testing.go:149: server-f8896627-a809-21db-6ab5-129eacf80e8e: 2021-07-10T01:05:09.481Z [INFO] server-f8896627-a809-21db-6ab5-129eacf80e8e: entering leader state: leader="Node at f8896627-a809-21db-6ab5-129eacf80e8e [Leader]" --- PASS: TestRaft_NotifyCh (0.10s) === RUN TestRaft_Voting testLoggerAdapter verbose: server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d: 2021-07-10T01:05:09.483Z [INFO] server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d Address:6316c959-88d4-e67a-a5ff-cd3975fa8d2d} {Suffrage:Voter ID:server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e Address:0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e} {Suffrage:Voter ID:server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460 Address:77a1df6c-8e57-68f7-a83d-e1f8b2f67460}]" testing.go:149: server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d: 2021-07-10T01:05:09.483Z [INFO] server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d Address:6316c959-88d4-e67a-a5ff-cd3975fa8d2d} {Suffrage:Voter ID:server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e Address:0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e} {Suffrage:Voter ID:server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460 Address:77a1df6c-8e57-68f7-a83d-e1f8b2f67460}]" testLoggerAdapter verbose: server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d: 2021-07-10T01:05:09.483Z [INFO] server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d: entering follower state: follower="Node at 6316c959-88d4-e67a-a5ff-cd3975fa8d2d [Follower]" leader= testing.go:149: server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d: 2021-07-10T01:05:09.483Z [INFO] server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d: entering follower state: follower="Node at 6316c959-88d4-e67a-a5ff-cd3975fa8d2d [Follower]" leader= testLoggerAdapter verbose: server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e: 2021-07-10T01:05:09.483Z [INFO] server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d Address:6316c959-88d4-e67a-a5ff-cd3975fa8d2d} {Suffrage:Voter ID:server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e Address:0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e} {Suffrage:Voter ID:server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460 Address:77a1df6c-8e57-68f7-a83d-e1f8b2f67460}]" testing.go:149: server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e: 2021-07-10T01:05:09.483Z [INFO] server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d Address:6316c959-88d4-e67a-a5ff-cd3975fa8d2d} {Suffrage:Voter ID:server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e Address:0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e} {Suffrage:Voter ID:server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460 Address:77a1df6c-8e57-68f7-a83d-e1f8b2f67460}]" testLoggerAdapter verbose: server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e: 2021-07-10T01:05:09.483Z [INFO] server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e: entering follower state: follower="Node at 0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e [Follower]" leader= testing.go:149: server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e: 2021-07-10T01:05:09.483Z [INFO] server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e: entering follower state: follower="Node at 0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e [Follower]" leader= testLoggerAdapter verbose: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.483Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d Address:6316c959-88d4-e67a-a5ff-cd3975fa8d2d} {Suffrage:Voter ID:server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e Address:0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e} {Suffrage:Voter ID:server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460 Address:77a1df6c-8e57-68f7-a83d-e1f8b2f67460}]" testing.go:149: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.483Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d Address:6316c959-88d4-e67a-a5ff-cd3975fa8d2d} {Suffrage:Voter ID:server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e Address:0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e} {Suffrage:Voter ID:server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460 Address:77a1df6c-8e57-68f7-a83d-e1f8b2f67460}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:09.483Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.483Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: entering follower state: follower="Node at 77a1df6c-8e57-68f7-a83d-e1f8b2f67460 [Follower]" leader= testing.go:149: cluster: 2021-07-10T01:05:09.483Z [INFO] starting stability test: raft-state=Follower testing.go:149: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.483Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: entering follower state: follower="Node at 77a1df6c-8e57-68f7-a83d-e1f8b2f67460 [Follower]" leader= testLoggerAdapter verbose: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.544Z [WARN] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.544Z [WARN] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.544Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: entering candidate state: node="Node at 77a1df6c-8e57-68f7-a83d-e1f8b2f67460 [Candidate]" term=2 testing.go:149: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.544Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: entering candidate state: node="Node at 77a1df6c-8e57-68f7-a83d-e1f8b2f67460 [Candidate]" term=2 testLoggerAdapter verbose: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.544Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: election won: tally=2 testing.go:149: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.544Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: election won: tally=2 testLoggerAdapter verbose: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.545Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: entering leader state: leader="Node at 77a1df6c-8e57-68f7-a83d-e1f8b2f67460 [Leader]" testing.go:149: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.545Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: entering leader state: leader="Node at 77a1df6c-8e57-68f7-a83d-e1f8b2f67460 [Leader]" testLoggerAdapter verbose: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.545Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: added peer, starting replication: peer=server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d testing.go:149: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.545Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: added peer, starting replication: peer=server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d testLoggerAdapter verbose: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.545Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: added peer, starting replication: peer=server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e testing.go:149: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.545Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: added peer, starting replication: peer=server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e testLoggerAdapter verbose: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.545Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: pipelining replication: peer="{Voter server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d 6316c959-88d4-e67a-a5ff-cd3975fa8d2d}" testing.go:149: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.545Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: pipelining replication: peer="{Voter server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d 6316c959-88d4-e67a-a5ff-cd3975fa8d2d}" testLoggerAdapter verbose: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.545Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: pipelining replication: peer="{Voter server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e 0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e}" testing.go:149: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.545Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: pipelining replication: peer="{Voter server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e 0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:09.650Z [INFO] stable state for Follower reached at 2021-07-10 01:05:09.545132642 +0000 UTC m=+14.933290612 (2 nodes), 61.118424ms from start of poll, 62.175515ms from cluster start. Timeout at 2021-07-10 01:05:09.65019925 +0000 UTC m=+15.038357240, 105.066628ms after stability testing.go:149: cluster: 2021-07-10T01:05:09.650Z [INFO] stable state for Follower reached at 2021-07-10 01:05:09.545132642 +0000 UTC m=+14.933290612 (2 nodes), 61.118424ms from start of poll, 62.175515ms from cluster start. Timeout at 2021-07-10 01:05:09.65019925 +0000 UTC m=+15.038357240, 105.066628ms after stability testLoggerAdapter verbose: cluster: 2021-07-10T01:05:09.650Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:09.650Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-07-10T01:05:09.755Z [INFO] stable state for Leader reached at 2021-07-10 01:05:09.650469493 +0000 UTC m=+15.038627463 (1 nodes), 2.12µs from start of poll, 167.512366ms from cluster start. Timeout at 2021-07-10 01:05:09.75548068 +0000 UTC m=+15.143638650, 105.011187ms after stability testing.go:149: cluster: 2021-07-10T01:05:09.755Z [INFO] stable state for Leader reached at 2021-07-10 01:05:09.650469493 +0000 UTC m=+15.038627463 (1 nodes), 2.12µs from start of poll, 167.512366ms from cluster start. Timeout at 2021-07-10 01:05:09.75548068 +0000 UTC m=+15.143638650, 105.011187ms after stability testLoggerAdapter verbose: server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e: 2021-07-10T01:05:09.755Z [WARN] server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e: rejecting vote request since we have a leader: from=6316c959-88d4-e67a-a5ff-cd3975fa8d2d leader=77a1df6c-8e57-68f7-a83d-e1f8b2f67460 testing.go:149: server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e: 2021-07-10T01:05:09.755Z [WARN] server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e: rejecting vote request since we have a leader: from=6316c959-88d4-e67a-a5ff-cd3975fa8d2d leader=77a1df6c-8e57-68f7-a83d-e1f8b2f67460 testLoggerAdapter verbose: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.755Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: aborting pipeline replication: peer="{Voter server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e 0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e}" testing.go:149: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.755Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: aborting pipeline replication: peer="{Voter server-0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e 0d1940ef-2d85-e0c3-c501-3c7c61ddcd6e}" testLoggerAdapter verbose: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.756Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: aborting pipeline replication: peer="{Voter server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d 6316c959-88d4-e67a-a5ff-cd3975fa8d2d}" testing.go:149: server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: 2021-07-10T01:05:09.756Z [INFO] server-77a1df6c-8e57-68f7-a83d-e1f8b2f67460: aborting pipeline replication: peer="{Voter server-6316c959-88d4-e67a-a5ff-cd3975fa8d2d 6316c959-88d4-e67a-a5ff-cd3975fa8d2d}" --- PASS: TestRaft_Voting (0.27s) === RUN TestRaft_ProtocolVersion_RejectRPC testLoggerAdapter verbose: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.757Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31 Address:5f6ce37a-9c3a-c951-74b3-d324f3ed3e31} {Suffrage:Voter ID:server-1000e46c-dd80-5b3d-08c7-02868a2f867c Address:1000e46c-dd80-5b3d-08c7-02868a2f867c} {Suffrage:Voter ID:server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0 Address:3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0}]" testing.go:149: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.757Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31 Address:5f6ce37a-9c3a-c951-74b3-d324f3ed3e31} {Suffrage:Voter ID:server-1000e46c-dd80-5b3d-08c7-02868a2f867c Address:1000e46c-dd80-5b3d-08c7-02868a2f867c} {Suffrage:Voter ID:server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0 Address:3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0}]" testLoggerAdapter verbose: server-1000e46c-dd80-5b3d-08c7-02868a2f867c: 2021-07-10T01:05:09.757Z [INFO] server-1000e46c-dd80-5b3d-08c7-02868a2f867c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31 Address:5f6ce37a-9c3a-c951-74b3-d324f3ed3e31} {Suffrage:Voter ID:server-1000e46c-dd80-5b3d-08c7-02868a2f867c Address:1000e46c-dd80-5b3d-08c7-02868a2f867c} {Suffrage:Voter ID:server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0 Address:3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0}]" testing.go:149: server-1000e46c-dd80-5b3d-08c7-02868a2f867c: 2021-07-10T01:05:09.757Z [INFO] server-1000e46c-dd80-5b3d-08c7-02868a2f867c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31 Address:5f6ce37a-9c3a-c951-74b3-d324f3ed3e31} {Suffrage:Voter ID:server-1000e46c-dd80-5b3d-08c7-02868a2f867c Address:1000e46c-dd80-5b3d-08c7-02868a2f867c} {Suffrage:Voter ID:server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0 Address:3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0}]" testLoggerAdapter verbose: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.757Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: entering follower state: follower="Node at 5f6ce37a-9c3a-c951-74b3-d324f3ed3e31 [Follower]" leader= testing.go:149: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.757Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: entering follower state: follower="Node at 5f6ce37a-9c3a-c951-74b3-d324f3ed3e31 [Follower]" leader= testLoggerAdapter verbose: server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0: 2021-07-10T01:05:09.757Z [INFO] server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31 Address:5f6ce37a-9c3a-c951-74b3-d324f3ed3e31} {Suffrage:Voter ID:server-1000e46c-dd80-5b3d-08c7-02868a2f867c Address:1000e46c-dd80-5b3d-08c7-02868a2f867c} {Suffrage:Voter ID:server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0 Address:3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0}]" testLoggerAdapter verbose: server-1000e46c-dd80-5b3d-08c7-02868a2f867c: 2021-07-10T01:05:09.757Z [INFO] server-1000e46c-dd80-5b3d-08c7-02868a2f867c: entering follower state: follower="Node at 1000e46c-dd80-5b3d-08c7-02868a2f867c [Follower]" leader= testing.go:149: server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0: 2021-07-10T01:05:09.757Z [INFO] server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31 Address:5f6ce37a-9c3a-c951-74b3-d324f3ed3e31} {Suffrage:Voter ID:server-1000e46c-dd80-5b3d-08c7-02868a2f867c Address:1000e46c-dd80-5b3d-08c7-02868a2f867c} {Suffrage:Voter ID:server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0 Address:3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:09.757Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-07-10T01:05:09.757Z [INFO] starting stability test: raft-state=Follower testing.go:149: server-1000e46c-dd80-5b3d-08c7-02868a2f867c: 2021-07-10T01:05:09.757Z [INFO] server-1000e46c-dd80-5b3d-08c7-02868a2f867c: entering follower state: follower="Node at 1000e46c-dd80-5b3d-08c7-02868a2f867c [Follower]" leader= testLoggerAdapter verbose: server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0: 2021-07-10T01:05:09.757Z [INFO] server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0: entering follower state: follower="Node at 3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0 [Follower]" leader= testing.go:149: server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0: 2021-07-10T01:05:09.757Z [INFO] server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0: entering follower state: follower="Node at 3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0 [Follower]" leader= testLoggerAdapter verbose: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.843Z [WARN] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.843Z [WARN] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.843Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: entering candidate state: node="Node at 5f6ce37a-9c3a-c951-74b3-d324f3ed3e31 [Candidate]" term=2 testing.go:149: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.843Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: entering candidate state: node="Node at 5f6ce37a-9c3a-c951-74b3-d324f3ed3e31 [Candidate]" term=2 testLoggerAdapter verbose: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.844Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: election won: tally=2 testing.go:149: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.844Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: election won: tally=2 testLoggerAdapter verbose: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.844Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: entering leader state: leader="Node at 5f6ce37a-9c3a-c951-74b3-d324f3ed3e31 [Leader]" testing.go:149: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.844Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: entering leader state: leader="Node at 5f6ce37a-9c3a-c951-74b3-d324f3ed3e31 [Leader]" testLoggerAdapter verbose: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.844Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: added peer, starting replication: peer=server-1000e46c-dd80-5b3d-08c7-02868a2f867c testing.go:149: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.844Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: added peer, starting replication: peer=server-1000e46c-dd80-5b3d-08c7-02868a2f867c testLoggerAdapter verbose: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.844Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: added peer, starting replication: peer=server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0 testing.go:149: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.844Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: added peer, starting replication: peer=server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0 testLoggerAdapter verbose: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.844Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: pipelining replication: peer="{Voter server-1000e46c-dd80-5b3d-08c7-02868a2f867c 1000e46c-dd80-5b3d-08c7-02868a2f867c}" testing.go:149: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.844Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: pipelining replication: peer="{Voter server-1000e46c-dd80-5b3d-08c7-02868a2f867c 1000e46c-dd80-5b3d-08c7-02868a2f867c}" testLoggerAdapter verbose: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.844Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: pipelining replication: peer="{Voter server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0 3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0}" testing.go:149: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:09.844Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: pipelining replication: peer="{Voter server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0 3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:09.949Z [INFO] stable state for Follower reached at 2021-07-10 01:05:09.844171775 +0000 UTC m=+15.232329865 (2 nodes), 86.197449ms from start of poll, 87.025518ms from cluster start. Timeout at 2021-07-10 01:05:09.949229882 +0000 UTC m=+15.337387852, 105.057987ms after stability testing.go:149: cluster: 2021-07-10T01:05:09.949Z [INFO] stable state for Follower reached at 2021-07-10 01:05:09.844171775 +0000 UTC m=+15.232329865 (2 nodes), 86.197449ms from start of poll, 87.025518ms from cluster start. Timeout at 2021-07-10 01:05:09.949229882 +0000 UTC m=+15.337387852, 105.057987ms after stability testLoggerAdapter verbose: cluster: 2021-07-10T01:05:09.949Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:09.949Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-07-10T01:05:10.054Z [INFO] stable state for Leader reached at 2021-07-10 01:05:09.949474185 +0000 UTC m=+15.337632155 (1 nodes), 2.06µs from start of poll, 192.327808ms from cluster start. Timeout at 2021-07-10 01:05:10.054542133 +0000 UTC m=+15.442700103, 105.067948ms after stability testing.go:149: cluster: 2021-07-10T01:05:10.054Z [INFO] stable state for Leader reached at 2021-07-10 01:05:09.949474185 +0000 UTC m=+15.337632155 (1 nodes), 2.06µs from start of poll, 192.327808ms from cluster start. Timeout at 2021-07-10 01:05:10.054542133 +0000 UTC m=+15.442700103, 105.067948ms after stability testLoggerAdapter verbose: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:10.054Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: aborting pipeline replication: peer="{Voter server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0 3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0}" testing.go:149: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:10.054Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: aborting pipeline replication: peer="{Voter server-3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0 3aed6cf6-fbb7-2648-16e3-bf9a94f1b8a0}" testLoggerAdapter verbose: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:10.054Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: aborting pipeline replication: peer="{Voter server-1000e46c-dd80-5b3d-08c7-02868a2f867c 1000e46c-dd80-5b3d-08c7-02868a2f867c}" testing.go:149: server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: 2021-07-10T01:05:10.054Z [INFO] server-5f6ce37a-9c3a-c951-74b3-d324f3ed3e31: aborting pipeline replication: peer="{Voter server-1000e46c-dd80-5b3d-08c7-02868a2f867c 1000e46c-dd80-5b3d-08c7-02868a2f867c}" --- PASS: TestRaft_ProtocolVersion_RejectRPC (0.30s) === RUN TestRaft_ProtocolVersion_Upgrade_1_2 testLoggerAdapter verbose: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.056Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: initial configuration: index=1 servers="[{Suffrage:Voter ID:9a872e48-d9d3-d627-f6d2-461f9c91991c Address:9a872e48-d9d3-d627-f6d2-461f9c91991c} {Suffrage:Voter ID:020322e8-db03-a3eb-6c7f-aa0385507418 Address:020322e8-db03-a3eb-6c7f-aa0385507418}]" testing.go:149: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.056Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: initial configuration: index=1 servers="[{Suffrage:Voter ID:9a872e48-d9d3-d627-f6d2-461f9c91991c Address:9a872e48-d9d3-d627-f6d2-461f9c91991c} {Suffrage:Voter ID:020322e8-db03-a3eb-6c7f-aa0385507418 Address:020322e8-db03-a3eb-6c7f-aa0385507418}]" testLoggerAdapter verbose: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.056Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: entering follower state: follower="Node at 9a872e48-d9d3-d627-f6d2-461f9c91991c [Follower]" leader= testing.go:149: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.056Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: entering follower state: follower="Node at 9a872e48-d9d3-d627-f6d2-461f9c91991c [Follower]" leader= testLoggerAdapter verbose: 020322e8-db03-a3eb-6c7f-aa0385507418: 2021-07-10T01:05:10.056Z [INFO] 020322e8-db03-a3eb-6c7f-aa0385507418: initial configuration: index=1 servers="[{Suffrage:Voter ID:9a872e48-d9d3-d627-f6d2-461f9c91991c Address:9a872e48-d9d3-d627-f6d2-461f9c91991c} {Suffrage:Voter ID:020322e8-db03-a3eb-6c7f-aa0385507418 Address:020322e8-db03-a3eb-6c7f-aa0385507418}]" testing.go:149: 020322e8-db03-a3eb-6c7f-aa0385507418: 2021-07-10T01:05:10.056Z [INFO] 020322e8-db03-a3eb-6c7f-aa0385507418: initial configuration: index=1 servers="[{Suffrage:Voter ID:9a872e48-d9d3-d627-f6d2-461f9c91991c Address:9a872e48-d9d3-d627-f6d2-461f9c91991c} {Suffrage:Voter ID:020322e8-db03-a3eb-6c7f-aa0385507418 Address:020322e8-db03-a3eb-6c7f-aa0385507418}]" testLoggerAdapter verbose: 020322e8-db03-a3eb-6c7f-aa0385507418: 2021-07-10T01:05:10.056Z [INFO] 020322e8-db03-a3eb-6c7f-aa0385507418: entering follower state: follower="Node at 020322e8-db03-a3eb-6c7f-aa0385507418 [Follower]" leader= testing.go:149: 020322e8-db03-a3eb-6c7f-aa0385507418: 2021-07-10T01:05:10.056Z [INFO] 020322e8-db03-a3eb-6c7f-aa0385507418: entering follower state: follower="Node at 020322e8-db03-a3eb-6c7f-aa0385507418 [Follower]" leader= testLoggerAdapter verbose: 363d840e-a128-8f51-7ba8-2e437195bfa9: 2021-07-10T01:05:10.056Z [INFO] 363d840e-a128-8f51-7ba8-2e437195bfa9: initial configuration: index=0 servers=[] testing.go:149: 363d840e-a128-8f51-7ba8-2e437195bfa9: 2021-07-10T01:05:10.056Z [INFO] 363d840e-a128-8f51-7ba8-2e437195bfa9: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2021-07-10T01:05:10.056Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: 363d840e-a128-8f51-7ba8-2e437195bfa9: 2021-07-10T01:05:10.056Z [INFO] 363d840e-a128-8f51-7ba8-2e437195bfa9: entering follower state: follower="Node at 363d840e-a128-8f51-7ba8-2e437195bfa9 [Follower]" leader= testing.go:149: cluster: 2021-07-10T01:05:10.056Z [INFO] starting stability test: raft-state=Leader testing.go:149: 363d840e-a128-8f51-7ba8-2e437195bfa9: 2021-07-10T01:05:10.056Z [INFO] 363d840e-a128-8f51-7ba8-2e437195bfa9: entering follower state: follower="Node at 363d840e-a128-8f51-7ba8-2e437195bfa9 [Follower]" leader= testLoggerAdapter verbose: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.129Z [WARN] 9a872e48-d9d3-d627-f6d2-461f9c91991c: heartbeat timeout reached, starting election: last-leader= testing.go:149: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.129Z [WARN] 9a872e48-d9d3-d627-f6d2-461f9c91991c: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.130Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: entering candidate state: node="Node at 9a872e48-d9d3-d627-f6d2-461f9c91991c [Candidate]" term=2 testing.go:149: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.130Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: entering candidate state: node="Node at 9a872e48-d9d3-d627-f6d2-461f9c91991c [Candidate]" term=2 testLoggerAdapter verbose: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.130Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: election won: tally=2 testing.go:149: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.130Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: election won: tally=2 testLoggerAdapter verbose: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.130Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: entering leader state: leader="Node at 9a872e48-d9d3-d627-f6d2-461f9c91991c [Leader]" testing.go:149: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.130Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: entering leader state: leader="Node at 9a872e48-d9d3-d627-f6d2-461f9c91991c [Leader]" testLoggerAdapter verbose: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.130Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: added peer, starting replication: peer=020322e8-db03-a3eb-6c7f-aa0385507418 testing.go:149: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.130Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: added peer, starting replication: peer=020322e8-db03-a3eb-6c7f-aa0385507418 testLoggerAdapter verbose: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.130Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: pipelining replication: peer="{Voter 020322e8-db03-a3eb-6c7f-aa0385507418 020322e8-db03-a3eb-6c7f-aa0385507418}" testing.go:149: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.130Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: pipelining replication: peer="{Voter 020322e8-db03-a3eb-6c7f-aa0385507418 020322e8-db03-a3eb-6c7f-aa0385507418}" testLoggerAdapter verbose: 363d840e-a128-8f51-7ba8-2e437195bfa9: 2021-07-10T01:05:10.135Z [WARN] 363d840e-a128-8f51-7ba8-2e437195bfa9: no known peers, aborting election testing.go:149: 363d840e-a128-8f51-7ba8-2e437195bfa9: 2021-07-10T01:05:10.135Z [WARN] 363d840e-a128-8f51-7ba8-2e437195bfa9: no known peers, aborting election testLoggerAdapter verbose: cluster: 2021-07-10T01:05:10.235Z [INFO] stable state for Leader reached at 2021-07-10 01:05:10.130416533 +0000 UTC m=+15.518574563 (1 nodes), 73.355534ms from start of poll, 74.363264ms from cluster start. Timeout at 2021-07-10 01:05:10.23550362 +0000 UTC m=+15.623661610, 105.087047ms after stability testing.go:149: cluster: 2021-07-10T01:05:10.235Z [INFO] stable state for Leader reached at 2021-07-10 01:05:10.130416533 +0000 UTC m=+15.518574563 (1 nodes), 73.355534ms from start of poll, 74.363264ms from cluster start. Timeout at 2021-07-10 01:05:10.23550362 +0000 UTC m=+15.623661610, 105.087047ms after stability testLoggerAdapter verbose: cluster: 2021-07-10T01:05:10.235Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:10.235Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-07-10T01:05:10.340Z [INFO] stable state for Leader reached at 2021-07-10 01:05:10.235777443 +0000 UTC m=+15.623935413 (1 nodes), 2.06µs from start of poll, 179.724114ms from cluster start. Timeout at 2021-07-10 01:05:10.340861471 +0000 UTC m=+15.729019441, 105.084028ms after stability testing.go:149: cluster: 2021-07-10T01:05:10.340Z [INFO] stable state for Leader reached at 2021-07-10 01:05:10.235777443 +0000 UTC m=+15.623935413 (1 nodes), 2.06µs from start of poll, 179.724114ms from cluster start. Timeout at 2021-07-10 01:05:10.340861471 +0000 UTC m=+15.729019441, 105.084028ms after stability testLoggerAdapter verbose: cluster: 2021-07-10T01:05:10.341Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:10.341Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-07-10T01:05:10.446Z [INFO] stable state for Leader reached at 2021-07-10 01:05:10.341137394 +0000 UTC m=+15.729295364 (1 nodes), 1.92µs from start of poll, 285.084065ms from cluster start. Timeout at 2021-07-10 01:05:10.446239282 +0000 UTC m=+15.834397252, 105.101888ms after stability testing.go:149: cluster: 2021-07-10T01:05:10.446Z [INFO] stable state for Leader reached at 2021-07-10 01:05:10.341137394 +0000 UTC m=+15.729295364 (1 nodes), 1.92µs from start of poll, 285.084065ms from cluster start. Timeout at 2021-07-10 01:05:10.446239282 +0000 UTC m=+15.834397252, 105.101888ms after stability testLoggerAdapter verbose: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.446Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: updating configuration: command=AddStaging server-id=363d840e-a128-8f51-7ba8-2e437195bfa9 server-addr=363d840e-a128-8f51-7ba8-2e437195bfa9 servers="[{Suffrage:Voter ID:9a872e48-d9d3-d627-f6d2-461f9c91991c Address:9a872e48-d9d3-d627-f6d2-461f9c91991c} {Suffrage:Voter ID:020322e8-db03-a3eb-6c7f-aa0385507418 Address:020322e8-db03-a3eb-6c7f-aa0385507418} {Suffrage:Voter ID:363d840e-a128-8f51-7ba8-2e437195bfa9 Address:363d840e-a128-8f51-7ba8-2e437195bfa9}]" testing.go:149: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.446Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: updating configuration: command=AddStaging server-id=363d840e-a128-8f51-7ba8-2e437195bfa9 server-addr=363d840e-a128-8f51-7ba8-2e437195bfa9 servers="[{Suffrage:Voter ID:9a872e48-d9d3-d627-f6d2-461f9c91991c Address:9a872e48-d9d3-d627-f6d2-461f9c91991c} {Suffrage:Voter ID:020322e8-db03-a3eb-6c7f-aa0385507418 Address:020322e8-db03-a3eb-6c7f-aa0385507418} {Suffrage:Voter ID:363d840e-a128-8f51-7ba8-2e437195bfa9 Address:363d840e-a128-8f51-7ba8-2e437195bfa9}]" testLoggerAdapter verbose: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.446Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: added peer, starting replication: peer=363d840e-a128-8f51-7ba8-2e437195bfa9 testing.go:149: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.446Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: added peer, starting replication: peer=363d840e-a128-8f51-7ba8-2e437195bfa9 testLoggerAdapter verbose: 363d840e-a128-8f51-7ba8-2e437195bfa9: 2021-07-10T01:05:10.446Z [WARN] 363d840e-a128-8f51-7ba8-2e437195bfa9: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: 363d840e-a128-8f51-7ba8-2e437195bfa9: 2021-07-10T01:05:10.446Z [WARN] 363d840e-a128-8f51-7ba8-2e437195bfa9: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.447Z [WARN] 9a872e48-d9d3-d627-f6d2-461f9c91991c: appendEntries rejected, sending older logs: peer="{Voter 363d840e-a128-8f51-7ba8-2e437195bfa9 363d840e-a128-8f51-7ba8-2e437195bfa9}" next=1 testing.go:149: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.447Z [WARN] 9a872e48-d9d3-d627-f6d2-461f9c91991c: appendEntries rejected, sending older logs: peer="{Voter 363d840e-a128-8f51-7ba8-2e437195bfa9 363d840e-a128-8f51-7ba8-2e437195bfa9}" next=1 testLoggerAdapter verbose: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.447Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: pipelining replication: peer="{Voter 363d840e-a128-8f51-7ba8-2e437195bfa9 363d840e-a128-8f51-7ba8-2e437195bfa9}" testing.go:149: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.447Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: pipelining replication: peer="{Voter 363d840e-a128-8f51-7ba8-2e437195bfa9 363d840e-a128-8f51-7ba8-2e437195bfa9}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:10.452Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:10.452Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-07-10T01:05:10.557Z [INFO] stable state for Leader reached at 2021-07-10 01:05:10.452377007 +0000 UTC m=+15.840534977 (1 nodes), 4.32µs from start of poll, 396.323678ms from cluster start. Timeout at 2021-07-10 01:05:10.557445414 +0000 UTC m=+15.945603384, 105.068407ms after stability testing.go:149: cluster: 2021-07-10T01:05:10.557Z [INFO] stable state for Leader reached at 2021-07-10 01:05:10.452377007 +0000 UTC m=+15.840534977 (1 nodes), 4.32µs from start of poll, 396.323678ms from cluster start. Timeout at 2021-07-10 01:05:10.557445414 +0000 UTC m=+15.945603384, 105.068407ms after stability testLoggerAdapter verbose: cluster: 2021-07-10T01:05:10.557Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:10.557Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-07-10T01:05:10.662Z [INFO] stable state for Leader reached at 2021-07-10 01:05:10.557672477 +0000 UTC m=+15.945830447 (1 nodes), 2.4µs from start of poll, 501.619148ms from cluster start. Timeout at 2021-07-10 01:05:10.662688684 +0000 UTC m=+16.050846654, 105.016207ms after stability testing.go:149: cluster: 2021-07-10T01:05:10.662Z [INFO] stable state for Leader reached at 2021-07-10 01:05:10.557672477 +0000 UTC m=+15.945830447 (1 nodes), 2.4µs from start of poll, 501.619148ms from cluster start. Timeout at 2021-07-10 01:05:10.662688684 +0000 UTC m=+16.050846654, 105.016207ms after stability testLoggerAdapter verbose: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.663Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: updating configuration: command=RemoveServer server-id=363d840e-a128-8f51-7ba8-2e437195bfa9 server-addr= servers="[{Suffrage:Voter ID:9a872e48-d9d3-d627-f6d2-461f9c91991c Address:9a872e48-d9d3-d627-f6d2-461f9c91991c} {Suffrage:Voter ID:020322e8-db03-a3eb-6c7f-aa0385507418 Address:020322e8-db03-a3eb-6c7f-aa0385507418}]" testing.go:149: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.663Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: updating configuration: command=RemoveServer server-id=363d840e-a128-8f51-7ba8-2e437195bfa9 server-addr= servers="[{Suffrage:Voter ID:9a872e48-d9d3-d627-f6d2-461f9c91991c Address:9a872e48-d9d3-d627-f6d2-461f9c91991c} {Suffrage:Voter ID:020322e8-db03-a3eb-6c7f-aa0385507418 Address:020322e8-db03-a3eb-6c7f-aa0385507418}]" testLoggerAdapter verbose: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.663Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: removed peer, stopping replication: peer=363d840e-a128-8f51-7ba8-2e437195bfa9 last-index=4 testing.go:149: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.663Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: removed peer, stopping replication: peer=363d840e-a128-8f51-7ba8-2e437195bfa9 last-index=4 testLoggerAdapter verbose: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.663Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: aborting pipeline replication: peer="{Voter 363d840e-a128-8f51-7ba8-2e437195bfa9 363d840e-a128-8f51-7ba8-2e437195bfa9}" testing.go:149: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.663Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: aborting pipeline replication: peer="{Voter 363d840e-a128-8f51-7ba8-2e437195bfa9 363d840e-a128-8f51-7ba8-2e437195bfa9}" testLoggerAdapter verbose: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.663Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: aborting pipeline replication: peer="{Voter 020322e8-db03-a3eb-6c7f-aa0385507418 020322e8-db03-a3eb-6c7f-aa0385507418}" testing.go:149: 9a872e48-d9d3-d627-f6d2-461f9c91991c: 2021-07-10T01:05:10.663Z [INFO] 9a872e48-d9d3-d627-f6d2-461f9c91991c: aborting pipeline replication: peer="{Voter 020322e8-db03-a3eb-6c7f-aa0385507418 020322e8-db03-a3eb-6c7f-aa0385507418}" --- PASS: TestRaft_ProtocolVersion_Upgrade_1_2 (0.61s) === RUN TestRaft_ProtocolVersion_Upgrade_2_3 testLoggerAdapter verbose: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.664Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: initial configuration: index=1 servers="[{Suffrage:Voter ID:1305795e-8f52-6a93-4f5f-2c58c5b5d119 Address:1305795e-8f52-6a93-4f5f-2c58c5b5d119} {Suffrage:Voter ID:a0850083-6008-3bdc-77a7-b2a2091c4613 Address:a0850083-6008-3bdc-77a7-b2a2091c4613}]" testing.go:149: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.664Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: initial configuration: index=1 servers="[{Suffrage:Voter ID:1305795e-8f52-6a93-4f5f-2c58c5b5d119 Address:1305795e-8f52-6a93-4f5f-2c58c5b5d119} {Suffrage:Voter ID:a0850083-6008-3bdc-77a7-b2a2091c4613 Address:a0850083-6008-3bdc-77a7-b2a2091c4613}]" testLoggerAdapter verbose: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.664Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: entering follower state: follower="Node at 1305795e-8f52-6a93-4f5f-2c58c5b5d119 [Follower]" leader= testing.go:149: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.664Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: entering follower state: follower="Node at 1305795e-8f52-6a93-4f5f-2c58c5b5d119 [Follower]" leader= testLoggerAdapter verbose: a0850083-6008-3bdc-77a7-b2a2091c4613: 2021-07-10T01:05:10.664Z [INFO] a0850083-6008-3bdc-77a7-b2a2091c4613: initial configuration: index=1 servers="[{Suffrage:Voter ID:1305795e-8f52-6a93-4f5f-2c58c5b5d119 Address:1305795e-8f52-6a93-4f5f-2c58c5b5d119} {Suffrage:Voter ID:a0850083-6008-3bdc-77a7-b2a2091c4613 Address:a0850083-6008-3bdc-77a7-b2a2091c4613}]" testing.go:149: a0850083-6008-3bdc-77a7-b2a2091c4613: 2021-07-10T01:05:10.664Z [INFO] a0850083-6008-3bdc-77a7-b2a2091c4613: initial configuration: index=1 servers="[{Suffrage:Voter ID:1305795e-8f52-6a93-4f5f-2c58c5b5d119 Address:1305795e-8f52-6a93-4f5f-2c58c5b5d119} {Suffrage:Voter ID:a0850083-6008-3bdc-77a7-b2a2091c4613 Address:a0850083-6008-3bdc-77a7-b2a2091c4613}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:10.665Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-07-10T01:05:10.665Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: a0850083-6008-3bdc-77a7-b2a2091c4613: 2021-07-10T01:05:10.665Z [INFO] a0850083-6008-3bdc-77a7-b2a2091c4613: entering follower state: follower="Node at a0850083-6008-3bdc-77a7-b2a2091c4613 [Follower]" leader= testing.go:149: a0850083-6008-3bdc-77a7-b2a2091c4613: 2021-07-10T01:05:10.665Z [INFO] a0850083-6008-3bdc-77a7-b2a2091c4613: entering follower state: follower="Node at a0850083-6008-3bdc-77a7-b2a2091c4613 [Follower]" leader= testLoggerAdapter verbose: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.750Z [WARN] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: heartbeat timeout reached, starting election: last-leader= testing.go:149: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.750Z [WARN] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.750Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: entering candidate state: node="Node at 1305795e-8f52-6a93-4f5f-2c58c5b5d119 [Candidate]" term=2 testing.go:149: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.750Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: entering candidate state: node="Node at 1305795e-8f52-6a93-4f5f-2c58c5b5d119 [Candidate]" term=2 testLoggerAdapter verbose: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.751Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: election won: tally=2 testing.go:149: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.751Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: election won: tally=2 testLoggerAdapter verbose: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.751Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: entering leader state: leader="Node at 1305795e-8f52-6a93-4f5f-2c58c5b5d119 [Leader]" testing.go:149: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.751Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: entering leader state: leader="Node at 1305795e-8f52-6a93-4f5f-2c58c5b5d119 [Leader]" testLoggerAdapter verbose: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.751Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: added peer, starting replication: peer=a0850083-6008-3bdc-77a7-b2a2091c4613 testing.go:149: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.751Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: added peer, starting replication: peer=a0850083-6008-3bdc-77a7-b2a2091c4613 testLoggerAdapter verbose: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.751Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: pipelining replication: peer="{Voter a0850083-6008-3bdc-77a7-b2a2091c4613 a0850083-6008-3bdc-77a7-b2a2091c4613}" testing.go:149: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.751Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: pipelining replication: peer="{Voter a0850083-6008-3bdc-77a7-b2a2091c4613 a0850083-6008-3bdc-77a7-b2a2091c4613}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:10.856Z [INFO] stable state for Follower reached at 2021-07-10 01:05:10.751230797 +0000 UTC m=+16.139388787 (1 nodes), 86.169048ms from start of poll, 86.741934ms from cluster start. Timeout at 2021-07-10 01:05:10.856306805 +0000 UTC m=+16.244464775, 105.075988ms after stability testing.go:149: cluster: 2021-07-10T01:05:10.856Z [INFO] stable state for Follower reached at 2021-07-10 01:05:10.751230797 +0000 UTC m=+16.139388787 (1 nodes), 86.169048ms from start of poll, 86.741934ms from cluster start. Timeout at 2021-07-10 01:05:10.856306805 +0000 UTC m=+16.244464775, 105.075988ms after stability testLoggerAdapter verbose: server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d: 2021-07-10T01:05:10.856Z [INFO] server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d: initial configuration: index=0 servers=[] testing.go:149: server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d: 2021-07-10T01:05:10.856Z [INFO] server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2021-07-10T01:05:10.857Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:10.857Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d: 2021-07-10T01:05:10.857Z [INFO] server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d: entering follower state: follower="Node at a8678b99-61f6-a3e5-6f78-5df9e6bafe9d [Follower]" leader= testing.go:149: server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d: 2021-07-10T01:05:10.857Z [INFO] server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d: entering follower state: follower="Node at a8678b99-61f6-a3e5-6f78-5df9e6bafe9d [Follower]" leader= testLoggerAdapter verbose: server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d: 2021-07-10T01:05:10.919Z [WARN] server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d: no known peers, aborting election testing.go:149: server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d: 2021-07-10T01:05:10.919Z [WARN] server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d: no known peers, aborting election testLoggerAdapter verbose: cluster: 2021-07-10T01:05:10.962Z [INFO] stable state for Leader reached at 2021-07-10 01:05:10.857127914 +0000 UTC m=+16.245285884 (1 nodes), 1.46µs from start of poll, 192.639031ms from cluster start. Timeout at 2021-07-10 01:05:10.962197782 +0000 UTC m=+16.350355752, 105.069868ms after stability testing.go:149: cluster: 2021-07-10T01:05:10.962Z [INFO] stable state for Leader reached at 2021-07-10 01:05:10.857127914 +0000 UTC m=+16.245285884 (1 nodes), 1.46µs from start of poll, 192.639031ms from cluster start. Timeout at 2021-07-10 01:05:10.962197782 +0000 UTC m=+16.350355752, 105.069868ms after stability testLoggerAdapter verbose: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.962Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: updating configuration: command=AddStaging server-id=server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d server-addr=a8678b99-61f6-a3e5-6f78-5df9e6bafe9d servers="[{Suffrage:Voter ID:1305795e-8f52-6a93-4f5f-2c58c5b5d119 Address:1305795e-8f52-6a93-4f5f-2c58c5b5d119} {Suffrage:Voter ID:a0850083-6008-3bdc-77a7-b2a2091c4613 Address:a0850083-6008-3bdc-77a7-b2a2091c4613} {Suffrage:Voter ID:server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d Address:a8678b99-61f6-a3e5-6f78-5df9e6bafe9d}]" testing.go:149: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.962Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: updating configuration: command=AddStaging server-id=server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d server-addr=a8678b99-61f6-a3e5-6f78-5df9e6bafe9d servers="[{Suffrage:Voter ID:1305795e-8f52-6a93-4f5f-2c58c5b5d119 Address:1305795e-8f52-6a93-4f5f-2c58c5b5d119} {Suffrage:Voter ID:a0850083-6008-3bdc-77a7-b2a2091c4613 Address:a0850083-6008-3bdc-77a7-b2a2091c4613} {Suffrage:Voter ID:server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d Address:a8678b99-61f6-a3e5-6f78-5df9e6bafe9d}]" testLoggerAdapter verbose: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.962Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: added peer, starting replication: peer=server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d testing.go:149: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.962Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: added peer, starting replication: peer=server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d testLoggerAdapter verbose: server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d: 2021-07-10T01:05:10.962Z [WARN] server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d: 2021-07-10T01:05:10.962Z [WARN] server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.962Z [WARN] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: appendEntries rejected, sending older logs: peer="{Voter server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d a8678b99-61f6-a3e5-6f78-5df9e6bafe9d}" next=1 testing.go:149: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.962Z [WARN] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: appendEntries rejected, sending older logs: peer="{Voter server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d a8678b99-61f6-a3e5-6f78-5df9e6bafe9d}" next=1 testLoggerAdapter verbose: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.963Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: pipelining replication: peer="{Voter server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d a8678b99-61f6-a3e5-6f78-5df9e6bafe9d}" testing.go:149: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:10.963Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: pipelining replication: peer="{Voter server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d a8678b99-61f6-a3e5-6f78-5df9e6bafe9d}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:10.968Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:10.968Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-07-10T01:05:11.073Z [INFO] stable state for Leader reached at 2021-07-10 01:05:10.968225845 +0000 UTC m=+16.356383815 (1 nodes), 2.02µs from start of poll, 303.736962ms from cluster start. Timeout at 2021-07-10 01:05:11.073302193 +0000 UTC m=+16.461460143, 105.076328ms after stability testing.go:149: cluster: 2021-07-10T01:05:11.073Z [INFO] stable state for Leader reached at 2021-07-10 01:05:10.968225845 +0000 UTC m=+16.356383815 (1 nodes), 2.02µs from start of poll, 303.736962ms from cluster start. Timeout at 2021-07-10 01:05:11.073302193 +0000 UTC m=+16.461460143, 105.076328ms after stability testLoggerAdapter verbose: cluster: 2021-07-10T01:05:11.073Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:11.073Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-07-10T01:05:11.178Z [INFO] stable state for Leader reached at 2021-07-10 01:05:11.073530575 +0000 UTC m=+16.461688545 (1 nodes), 2.7µs from start of poll, 409.041692ms from cluster start. Timeout at 2021-07-10 01:05:11.178608463 +0000 UTC m=+16.566766433, 105.077888ms after stability testing.go:149: cluster: 2021-07-10T01:05:11.178Z [INFO] stable state for Leader reached at 2021-07-10 01:05:11.073530575 +0000 UTC m=+16.461688545 (1 nodes), 2.7µs from start of poll, 409.041692ms from cluster start. Timeout at 2021-07-10 01:05:11.178608463 +0000 UTC m=+16.566766433, 105.077888ms after stability testLoggerAdapter verbose: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:11.178Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: updating configuration: command=RemoveServer server-id=a0850083-6008-3bdc-77a7-b2a2091c4613 server-addr= servers="[{Suffrage:Voter ID:1305795e-8f52-6a93-4f5f-2c58c5b5d119 Address:1305795e-8f52-6a93-4f5f-2c58c5b5d119} {Suffrage:Voter ID:server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d Address:a8678b99-61f6-a3e5-6f78-5df9e6bafe9d}]" testing.go:149: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:11.178Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: updating configuration: command=RemoveServer server-id=a0850083-6008-3bdc-77a7-b2a2091c4613 server-addr= servers="[{Suffrage:Voter ID:1305795e-8f52-6a93-4f5f-2c58c5b5d119 Address:1305795e-8f52-6a93-4f5f-2c58c5b5d119} {Suffrage:Voter ID:server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d Address:a8678b99-61f6-a3e5-6f78-5df9e6bafe9d}]" testLoggerAdapter verbose: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:11.179Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: removed peer, stopping replication: peer=a0850083-6008-3bdc-77a7-b2a2091c4613 last-index=4 testing.go:149: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:11.179Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: removed peer, stopping replication: peer=a0850083-6008-3bdc-77a7-b2a2091c4613 last-index=4 testLoggerAdapter verbose: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:11.179Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: aborting pipeline replication: peer="{Voter a0850083-6008-3bdc-77a7-b2a2091c4613 a0850083-6008-3bdc-77a7-b2a2091c4613}" testing.go:149: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:11.179Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: aborting pipeline replication: peer="{Voter a0850083-6008-3bdc-77a7-b2a2091c4613 a0850083-6008-3bdc-77a7-b2a2091c4613}" testLoggerAdapter verbose: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:11.179Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: aborting pipeline replication: peer="{Voter server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d a8678b99-61f6-a3e5-6f78-5df9e6bafe9d}" testing.go:149: 1305795e-8f52-6a93-4f5f-2c58c5b5d119: 2021-07-10T01:05:11.179Z [INFO] 1305795e-8f52-6a93-4f5f-2c58c5b5d119: aborting pipeline replication: peer="{Voter server-a8678b99-61f6-a3e5-6f78-5df9e6bafe9d a8678b99-61f6-a3e5-6f78-5df9e6bafe9d}" --- PASS: TestRaft_ProtocolVersion_Upgrade_2_3 (0.52s) === RUN TestRaft_LeadershipTransferInProgress --- PASS: TestRaft_LeadershipTransferInProgress (0.00s) === RUN TestRaft_LeadershipTransferPickServer --- PASS: TestRaft_LeadershipTransferPickServer (0.00s) === RUN TestRaft_LeadershipTransfer testLoggerAdapter verbose: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.180Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a76f879-c778-63b4-3230-37e80df5f9a7 Address:6a76f879-c778-63b4-3230-37e80df5f9a7} {Suffrage:Voter ID:server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d Address:de91a26b-f14a-bd86-fc3c-9dd7e80a495d} {Suffrage:Voter ID:server-f6290c41-6e88-8e2c-a14b-87f717de5dee Address:f6290c41-6e88-8e2c-a14b-87f717de5dee}]" testing.go:149: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.180Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a76f879-c778-63b4-3230-37e80df5f9a7 Address:6a76f879-c778-63b4-3230-37e80df5f9a7} {Suffrage:Voter ID:server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d Address:de91a26b-f14a-bd86-fc3c-9dd7e80a495d} {Suffrage:Voter ID:server-f6290c41-6e88-8e2c-a14b-87f717de5dee Address:f6290c41-6e88-8e2c-a14b-87f717de5dee}]" testLoggerAdapter verbose: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.181Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: entering follower state: follower="Node at 6a76f879-c778-63b4-3230-37e80df5f9a7 [Follower]" leader= testing.go:149: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.181Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: entering follower state: follower="Node at 6a76f879-c778-63b4-3230-37e80df5f9a7 [Follower]" leader= testLoggerAdapter verbose: server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d: 2021-07-10T01:05:11.181Z [INFO] server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a76f879-c778-63b4-3230-37e80df5f9a7 Address:6a76f879-c778-63b4-3230-37e80df5f9a7} {Suffrage:Voter ID:server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d Address:de91a26b-f14a-bd86-fc3c-9dd7e80a495d} {Suffrage:Voter ID:server-f6290c41-6e88-8e2c-a14b-87f717de5dee Address:f6290c41-6e88-8e2c-a14b-87f717de5dee}]" testing.go:149: server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d: 2021-07-10T01:05:11.181Z [INFO] server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a76f879-c778-63b4-3230-37e80df5f9a7 Address:6a76f879-c778-63b4-3230-37e80df5f9a7} {Suffrage:Voter ID:server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d Address:de91a26b-f14a-bd86-fc3c-9dd7e80a495d} {Suffrage:Voter ID:server-f6290c41-6e88-8e2c-a14b-87f717de5dee Address:f6290c41-6e88-8e2c-a14b-87f717de5dee}]" testLoggerAdapter verbose: server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d: 2021-07-10T01:05:11.181Z [INFO] server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d: entering follower state: follower="Node at de91a26b-f14a-bd86-fc3c-9dd7e80a495d [Follower]" leader= testing.go:149: server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d: 2021-07-10T01:05:11.181Z [INFO] server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d: entering follower state: follower="Node at de91a26b-f14a-bd86-fc3c-9dd7e80a495d [Follower]" leader= testLoggerAdapter verbose: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.181Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a76f879-c778-63b4-3230-37e80df5f9a7 Address:6a76f879-c778-63b4-3230-37e80df5f9a7} {Suffrage:Voter ID:server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d Address:de91a26b-f14a-bd86-fc3c-9dd7e80a495d} {Suffrage:Voter ID:server-f6290c41-6e88-8e2c-a14b-87f717de5dee Address:f6290c41-6e88-8e2c-a14b-87f717de5dee}]" testing.go:149: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.181Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a76f879-c778-63b4-3230-37e80df5f9a7 Address:6a76f879-c778-63b4-3230-37e80df5f9a7} {Suffrage:Voter ID:server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d Address:de91a26b-f14a-bd86-fc3c-9dd7e80a495d} {Suffrage:Voter ID:server-f6290c41-6e88-8e2c-a14b-87f717de5dee Address:f6290c41-6e88-8e2c-a14b-87f717de5dee}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:11.181Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:11.181Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.181Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: entering follower state: follower="Node at f6290c41-6e88-8e2c-a14b-87f717de5dee [Follower]" leader= testing.go:149: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.181Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: entering follower state: follower="Node at f6290c41-6e88-8e2c-a14b-87f717de5dee [Follower]" leader= testLoggerAdapter verbose: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.245Z [WARN] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.245Z [WARN] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.245Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: entering candidate state: node="Node at f6290c41-6e88-8e2c-a14b-87f717de5dee [Candidate]" term=2 testing.go:149: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.245Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: entering candidate state: node="Node at f6290c41-6e88-8e2c-a14b-87f717de5dee [Candidate]" term=2 testLoggerAdapter verbose: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.246Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: election won: tally=2 testing.go:149: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.246Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: election won: tally=2 testLoggerAdapter verbose: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.246Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: entering leader state: leader="Node at f6290c41-6e88-8e2c-a14b-87f717de5dee [Leader]" testing.go:149: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.246Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: entering leader state: leader="Node at f6290c41-6e88-8e2c-a14b-87f717de5dee [Leader]" testLoggerAdapter verbose: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.246Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: added peer, starting replication: peer=server-6a76f879-c778-63b4-3230-37e80df5f9a7 testing.go:149: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.246Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: added peer, starting replication: peer=server-6a76f879-c778-63b4-3230-37e80df5f9a7 testLoggerAdapter verbose: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.246Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: added peer, starting replication: peer=server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d testing.go:149: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.246Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: added peer, starting replication: peer=server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d testLoggerAdapter verbose: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.246Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: pipelining replication: peer="{Voter server-6a76f879-c778-63b4-3230-37e80df5f9a7 6a76f879-c778-63b4-3230-37e80df5f9a7}" testing.go:149: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.246Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: pipelining replication: peer="{Voter server-6a76f879-c778-63b4-3230-37e80df5f9a7 6a76f879-c778-63b4-3230-37e80df5f9a7}" testLoggerAdapter verbose: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.246Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: pipelining replication: peer="{Voter server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d de91a26b-f14a-bd86-fc3c-9dd7e80a495d}" testing.go:149: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.246Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: pipelining replication: peer="{Voter server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d de91a26b-f14a-bd86-fc3c-9dd7e80a495d}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:11.351Z [INFO] stable state for Leader reached at 2021-07-10 01:05:11.246184535 +0000 UTC m=+16.634342505 (1 nodes), 64.51988ms from start of poll, 65.438049ms from cluster start. Timeout at 2021-07-10 01:05:11.351278343 +0000 UTC m=+16.739436293, 105.093788ms after stability testing.go:149: cluster: 2021-07-10T01:05:11.351Z [INFO] stable state for Leader reached at 2021-07-10 01:05:11.246184535 +0000 UTC m=+16.634342505 (1 nodes), 64.51988ms from start of poll, 65.438049ms from cluster start. Timeout at 2021-07-10 01:05:11.351278343 +0000 UTC m=+16.739436293, 105.093788ms after stability testLoggerAdapter verbose: cluster: 2021-07-10T01:05:11.351Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:11.351Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-07-10T01:05:11.456Z [INFO] stable state for Leader reached at 2021-07-10 01:05:11.351594147 +0000 UTC m=+16.739752117 (1 nodes), 2.4µs from start of poll, 170.847661ms from cluster start. Timeout at 2021-07-10 01:05:11.456649034 +0000 UTC m=+16.844807004, 105.054887ms after stability testing.go:149: cluster: 2021-07-10T01:05:11.456Z [INFO] stable state for Leader reached at 2021-07-10 01:05:11.351594147 +0000 UTC m=+16.739752117 (1 nodes), 2.4µs from start of poll, 170.847661ms from cluster start. Timeout at 2021-07-10 01:05:11.456649034 +0000 UTC m=+16.844807004, 105.054887ms after stability testLoggerAdapter verbose: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.456Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: entering candidate state: node="Node at 6a76f879-c778-63b4-3230-37e80df5f9a7 [Candidate]" term=3 testLoggerAdapter verbose: cluster: 2021-07-10T01:05:11.457Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.456Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: entering candidate state: node="Node at 6a76f879-c778-63b4-3230-37e80df5f9a7 [Candidate]" term=3 testing.go:149: cluster: 2021-07-10T01:05:11.457Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.457Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: entering follower state: follower="Node at f6290c41-6e88-8e2c-a14b-87f717de5dee [Follower]" leader= testing.go:149: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.457Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: entering follower state: follower="Node at f6290c41-6e88-8e2c-a14b-87f717de5dee [Follower]" leader= testLoggerAdapter verbose: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.457Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: aborting pipeline replication: peer="{Voter server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d de91a26b-f14a-bd86-fc3c-9dd7e80a495d}" testing.go:149: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.457Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: aborting pipeline replication: peer="{Voter server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d de91a26b-f14a-bd86-fc3c-9dd7e80a495d}" testLoggerAdapter verbose: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.457Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: election won: tally=2 testLoggerAdapter verbose: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.457Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: aborting pipeline replication: peer="{Voter server-6a76f879-c778-63b4-3230-37e80df5f9a7 6a76f879-c778-63b4-3230-37e80df5f9a7}" testing.go:149: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.457Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: election won: tally=2 testLoggerAdapter verbose: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.457Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: entering leader state: leader="Node at 6a76f879-c778-63b4-3230-37e80df5f9a7 [Leader]" testing.go:149: server-f6290c41-6e88-8e2c-a14b-87f717de5dee: 2021-07-10T01:05:11.457Z [INFO] server-f6290c41-6e88-8e2c-a14b-87f717de5dee: aborting pipeline replication: peer="{Voter server-6a76f879-c778-63b4-3230-37e80df5f9a7 6a76f879-c778-63b4-3230-37e80df5f9a7}" testing.go:149: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.457Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: entering leader state: leader="Node at 6a76f879-c778-63b4-3230-37e80df5f9a7 [Leader]" testLoggerAdapter verbose: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.457Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: added peer, starting replication: peer=server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d testing.go:149: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.457Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: added peer, starting replication: peer=server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d testLoggerAdapter verbose: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.457Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: added peer, starting replication: peer=server-f6290c41-6e88-8e2c-a14b-87f717de5dee testing.go:149: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.457Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: added peer, starting replication: peer=server-f6290c41-6e88-8e2c-a14b-87f717de5dee testLoggerAdapter verbose: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.457Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: pipelining replication: peer="{Voter server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d de91a26b-f14a-bd86-fc3c-9dd7e80a495d}" testing.go:149: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.457Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: pipelining replication: peer="{Voter server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d de91a26b-f14a-bd86-fc3c-9dd7e80a495d}" testLoggerAdapter verbose: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.457Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: pipelining replication: peer="{Voter server-f6290c41-6e88-8e2c-a14b-87f717de5dee f6290c41-6e88-8e2c-a14b-87f717de5dee}" testing.go:149: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.457Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: pipelining replication: peer="{Voter server-f6290c41-6e88-8e2c-a14b-87f717de5dee f6290c41-6e88-8e2c-a14b-87f717de5dee}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:11.562Z [INFO] stable state for Leader reached at 2021-07-10 01:05:11.457546824 +0000 UTC m=+16.845704814 (1 nodes), 320.504µs from start of poll, 276.800358ms from cluster start. Timeout at 2021-07-10 01:05:11.562643112 +0000 UTC m=+16.950801102, 105.096288ms after stability testing.go:149: cluster: 2021-07-10T01:05:11.562Z [INFO] stable state for Leader reached at 2021-07-10 01:05:11.457546824 +0000 UTC m=+16.845704814 (1 nodes), 320.504µs from start of poll, 276.800358ms from cluster start. Timeout at 2021-07-10 01:05:11.562643112 +0000 UTC m=+16.950801102, 105.096288ms after stability testLoggerAdapter verbose: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.562Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: aborting pipeline replication: peer="{Voter server-f6290c41-6e88-8e2c-a14b-87f717de5dee f6290c41-6e88-8e2c-a14b-87f717de5dee}" testing.go:149: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.562Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: aborting pipeline replication: peer="{Voter server-f6290c41-6e88-8e2c-a14b-87f717de5dee f6290c41-6e88-8e2c-a14b-87f717de5dee}" testLoggerAdapter verbose: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.563Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: aborting pipeline replication: peer="{Voter server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d de91a26b-f14a-bd86-fc3c-9dd7e80a495d}" testing.go:149: server-6a76f879-c778-63b4-3230-37e80df5f9a7: 2021-07-10T01:05:11.563Z [INFO] server-6a76f879-c778-63b4-3230-37e80df5f9a7: aborting pipeline replication: peer="{Voter server-de91a26b-f14a-bd86-fc3c-9dd7e80a495d de91a26b-f14a-bd86-fc3c-9dd7e80a495d}" --- PASS: TestRaft_LeadershipTransfer (0.38s) === RUN TestRaft_LeadershipTransferWithOneNode testLoggerAdapter verbose: server-674e2b05-a148-c0de-3845-668cd7394e8e: 2021-07-10T01:05:11.564Z [INFO] server-674e2b05-a148-c0de-3845-668cd7394e8e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-674e2b05-a148-c0de-3845-668cd7394e8e Address:674e2b05-a148-c0de-3845-668cd7394e8e}]" testing.go:149: server-674e2b05-a148-c0de-3845-668cd7394e8e: 2021-07-10T01:05:11.564Z [INFO] server-674e2b05-a148-c0de-3845-668cd7394e8e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-674e2b05-a148-c0de-3845-668cd7394e8e Address:674e2b05-a148-c0de-3845-668cd7394e8e}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:11.564Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:11.564Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-674e2b05-a148-c0de-3845-668cd7394e8e: 2021-07-10T01:05:11.564Z [INFO] server-674e2b05-a148-c0de-3845-668cd7394e8e: entering follower state: follower="Node at 674e2b05-a148-c0de-3845-668cd7394e8e [Follower]" leader= testing.go:149: server-674e2b05-a148-c0de-3845-668cd7394e8e: 2021-07-10T01:05:11.564Z [INFO] server-674e2b05-a148-c0de-3845-668cd7394e8e: entering follower state: follower="Node at 674e2b05-a148-c0de-3845-668cd7394e8e [Follower]" leader= testLoggerAdapter verbose: server-674e2b05-a148-c0de-3845-668cd7394e8e: 2021-07-10T01:05:11.628Z [WARN] server-674e2b05-a148-c0de-3845-668cd7394e8e: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-674e2b05-a148-c0de-3845-668cd7394e8e: 2021-07-10T01:05:11.628Z [WARN] server-674e2b05-a148-c0de-3845-668cd7394e8e: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-674e2b05-a148-c0de-3845-668cd7394e8e: 2021-07-10T01:05:11.628Z [INFO] server-674e2b05-a148-c0de-3845-668cd7394e8e: entering candidate state: node="Node at 674e2b05-a148-c0de-3845-668cd7394e8e [Candidate]" term=2 testing.go:149: server-674e2b05-a148-c0de-3845-668cd7394e8e: 2021-07-10T01:05:11.628Z [INFO] server-674e2b05-a148-c0de-3845-668cd7394e8e: entering candidate state: node="Node at 674e2b05-a148-c0de-3845-668cd7394e8e [Candidate]" term=2 testLoggerAdapter verbose: server-674e2b05-a148-c0de-3845-668cd7394e8e: 2021-07-10T01:05:11.628Z [INFO] server-674e2b05-a148-c0de-3845-668cd7394e8e: election won: tally=1 testing.go:149: server-674e2b05-a148-c0de-3845-668cd7394e8e: 2021-07-10T01:05:11.628Z [INFO] server-674e2b05-a148-c0de-3845-668cd7394e8e: election won: tally=1 testLoggerAdapter verbose: server-674e2b05-a148-c0de-3845-668cd7394e8e: 2021-07-10T01:05:11.628Z [INFO] server-674e2b05-a148-c0de-3845-668cd7394e8e: entering leader state: leader="Node at 674e2b05-a148-c0de-3845-668cd7394e8e [Leader]" testing.go:149: server-674e2b05-a148-c0de-3845-668cd7394e8e: 2021-07-10T01:05:11.628Z [INFO] server-674e2b05-a148-c0de-3845-668cd7394e8e: entering leader state: leader="Node at 674e2b05-a148-c0de-3845-668cd7394e8e [Leader]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:11.734Z [INFO] stable state for Leader reached at 2021-07-10 01:05:11.628818329 +0000 UTC m=+17.016976439 (1 nodes), 64.133956ms from start of poll, 64.632401ms from cluster start. Timeout at 2021-07-10 01:05:11.733911817 +0000 UTC m=+17.122069807, 105.093368ms after stability testing.go:149: cluster: 2021-07-10T01:05:11.734Z [INFO] stable state for Leader reached at 2021-07-10 01:05:11.628818329 +0000 UTC m=+17.016976439 (1 nodes), 64.133956ms from start of poll, 64.632401ms from cluster start. Timeout at 2021-07-10 01:05:11.733911817 +0000 UTC m=+17.122069807, 105.093368ms after stability --- PASS: TestRaft_LeadershipTransferWithOneNode (0.17s) === RUN TestRaft_LeadershipTransferWithSevenNodes testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.736Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace9227e-5165-dd6e-c019-1d2527aa2089 Address:ace9227e-5165-dd6e-c019-1d2527aa2089} {Suffrage:Voter ID:server-1568b52b-331a-882f-74a0-d70b659fb2e4 Address:1568b52b-331a-882f-74a0-d70b659fb2e4} {Suffrage:Voter ID:server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 Address:f55d7e90-36a2-ee2a-f65d-95c1e28cdb58} {Suffrage:Voter ID:server-95d157c1-a440-c46a-0d3e-40707d4563ce Address:95d157c1-a440-c46a-0d3e-40707d4563ce} {Suffrage:Voter ID:server-20ce517a-3433-6266-3126-63cfd16a1a27 Address:20ce517a-3433-6266-3126-63cfd16a1a27} {Suffrage:Voter ID:server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 Address:a33a7d16-94dc-ff82-86ab-a4333f4b91d3} {Suffrage:Voter ID:server-744fc635-8d74-914d-af27-318108d034bd Address:744fc635-8d74-914d-af27-318108d034bd}]" testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.736Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace9227e-5165-dd6e-c019-1d2527aa2089 Address:ace9227e-5165-dd6e-c019-1d2527aa2089} {Suffrage:Voter ID:server-1568b52b-331a-882f-74a0-d70b659fb2e4 Address:1568b52b-331a-882f-74a0-d70b659fb2e4} {Suffrage:Voter ID:server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 Address:f55d7e90-36a2-ee2a-f65d-95c1e28cdb58} {Suffrage:Voter ID:server-95d157c1-a440-c46a-0d3e-40707d4563ce Address:95d157c1-a440-c46a-0d3e-40707d4563ce} {Suffrage:Voter ID:server-20ce517a-3433-6266-3126-63cfd16a1a27 Address:20ce517a-3433-6266-3126-63cfd16a1a27} {Suffrage:Voter ID:server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 Address:a33a7d16-94dc-ff82-86ab-a4333f4b91d3} {Suffrage:Voter ID:server-744fc635-8d74-914d-af27-318108d034bd Address:744fc635-8d74-914d-af27-318108d034bd}]" testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.736Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: entering follower state: follower="Node at ace9227e-5165-dd6e-c019-1d2527aa2089 [Follower]" leader= testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.736Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: entering follower state: follower="Node at ace9227e-5165-dd6e-c019-1d2527aa2089 [Follower]" leader= testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:11.737Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace9227e-5165-dd6e-c019-1d2527aa2089 Address:ace9227e-5165-dd6e-c019-1d2527aa2089} {Suffrage:Voter ID:server-1568b52b-331a-882f-74a0-d70b659fb2e4 Address:1568b52b-331a-882f-74a0-d70b659fb2e4} {Suffrage:Voter ID:server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 Address:f55d7e90-36a2-ee2a-f65d-95c1e28cdb58} {Suffrage:Voter ID:server-95d157c1-a440-c46a-0d3e-40707d4563ce Address:95d157c1-a440-c46a-0d3e-40707d4563ce} {Suffrage:Voter ID:server-20ce517a-3433-6266-3126-63cfd16a1a27 Address:20ce517a-3433-6266-3126-63cfd16a1a27} {Suffrage:Voter ID:server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 Address:a33a7d16-94dc-ff82-86ab-a4333f4b91d3} {Suffrage:Voter ID:server-744fc635-8d74-914d-af27-318108d034bd Address:744fc635-8d74-914d-af27-318108d034bd}]" testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:11.737Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace9227e-5165-dd6e-c019-1d2527aa2089 Address:ace9227e-5165-dd6e-c019-1d2527aa2089} {Suffrage:Voter ID:server-1568b52b-331a-882f-74a0-d70b659fb2e4 Address:1568b52b-331a-882f-74a0-d70b659fb2e4} {Suffrage:Voter ID:server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 Address:f55d7e90-36a2-ee2a-f65d-95c1e28cdb58} {Suffrage:Voter ID:server-95d157c1-a440-c46a-0d3e-40707d4563ce Address:95d157c1-a440-c46a-0d3e-40707d4563ce} {Suffrage:Voter ID:server-20ce517a-3433-6266-3126-63cfd16a1a27 Address:20ce517a-3433-6266-3126-63cfd16a1a27} {Suffrage:Voter ID:server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 Address:a33a7d16-94dc-ff82-86ab-a4333f4b91d3} {Suffrage:Voter ID:server-744fc635-8d74-914d-af27-318108d034bd Address:744fc635-8d74-914d-af27-318108d034bd}]" testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:11.737Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: entering follower state: follower="Node at 1568b52b-331a-882f-74a0-d70b659fb2e4 [Follower]" leader= testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:11.737Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: entering follower state: follower="Node at 1568b52b-331a-882f-74a0-d70b659fb2e4 [Follower]" leader= testLoggerAdapter verbose: server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58: 2021-07-10T01:05:11.737Z [INFO] server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace9227e-5165-dd6e-c019-1d2527aa2089 Address:ace9227e-5165-dd6e-c019-1d2527aa2089} {Suffrage:Voter ID:server-1568b52b-331a-882f-74a0-d70b659fb2e4 Address:1568b52b-331a-882f-74a0-d70b659fb2e4} {Suffrage:Voter ID:server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 Address:f55d7e90-36a2-ee2a-f65d-95c1e28cdb58} {Suffrage:Voter ID:server-95d157c1-a440-c46a-0d3e-40707d4563ce Address:95d157c1-a440-c46a-0d3e-40707d4563ce} {Suffrage:Voter ID:server-20ce517a-3433-6266-3126-63cfd16a1a27 Address:20ce517a-3433-6266-3126-63cfd16a1a27} {Suffrage:Voter ID:server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 Address:a33a7d16-94dc-ff82-86ab-a4333f4b91d3} {Suffrage:Voter ID:server-744fc635-8d74-914d-af27-318108d034bd Address:744fc635-8d74-914d-af27-318108d034bd}]" testing.go:149: server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58: 2021-07-10T01:05:11.737Z [INFO] server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace9227e-5165-dd6e-c019-1d2527aa2089 Address:ace9227e-5165-dd6e-c019-1d2527aa2089} {Suffrage:Voter ID:server-1568b52b-331a-882f-74a0-d70b659fb2e4 Address:1568b52b-331a-882f-74a0-d70b659fb2e4} {Suffrage:Voter ID:server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 Address:f55d7e90-36a2-ee2a-f65d-95c1e28cdb58} {Suffrage:Voter ID:server-95d157c1-a440-c46a-0d3e-40707d4563ce Address:95d157c1-a440-c46a-0d3e-40707d4563ce} {Suffrage:Voter ID:server-20ce517a-3433-6266-3126-63cfd16a1a27 Address:20ce517a-3433-6266-3126-63cfd16a1a27} {Suffrage:Voter ID:server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 Address:a33a7d16-94dc-ff82-86ab-a4333f4b91d3} {Suffrage:Voter ID:server-744fc635-8d74-914d-af27-318108d034bd Address:744fc635-8d74-914d-af27-318108d034bd}]" testLoggerAdapter verbose: server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58: 2021-07-10T01:05:11.737Z [INFO] server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58: entering follower state: follower="Node at f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 [Follower]" leader= testing.go:149: server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58: 2021-07-10T01:05:11.737Z [INFO] server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58: entering follower state: follower="Node at f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 [Follower]" leader= testLoggerAdapter verbose: server-95d157c1-a440-c46a-0d3e-40707d4563ce: 2021-07-10T01:05:11.738Z [INFO] server-95d157c1-a440-c46a-0d3e-40707d4563ce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace9227e-5165-dd6e-c019-1d2527aa2089 Address:ace9227e-5165-dd6e-c019-1d2527aa2089} {Suffrage:Voter ID:server-1568b52b-331a-882f-74a0-d70b659fb2e4 Address:1568b52b-331a-882f-74a0-d70b659fb2e4} {Suffrage:Voter ID:server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 Address:f55d7e90-36a2-ee2a-f65d-95c1e28cdb58} {Suffrage:Voter ID:server-95d157c1-a440-c46a-0d3e-40707d4563ce Address:95d157c1-a440-c46a-0d3e-40707d4563ce} {Suffrage:Voter ID:server-20ce517a-3433-6266-3126-63cfd16a1a27 Address:20ce517a-3433-6266-3126-63cfd16a1a27} {Suffrage:Voter ID:server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 Address:a33a7d16-94dc-ff82-86ab-a4333f4b91d3} {Suffrage:Voter ID:server-744fc635-8d74-914d-af27-318108d034bd Address:744fc635-8d74-914d-af27-318108d034bd}]" testing.go:149: server-95d157c1-a440-c46a-0d3e-40707d4563ce: 2021-07-10T01:05:11.738Z [INFO] server-95d157c1-a440-c46a-0d3e-40707d4563ce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace9227e-5165-dd6e-c019-1d2527aa2089 Address:ace9227e-5165-dd6e-c019-1d2527aa2089} {Suffrage:Voter ID:server-1568b52b-331a-882f-74a0-d70b659fb2e4 Address:1568b52b-331a-882f-74a0-d70b659fb2e4} {Suffrage:Voter ID:server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 Address:f55d7e90-36a2-ee2a-f65d-95c1e28cdb58} {Suffrage:Voter ID:server-95d157c1-a440-c46a-0d3e-40707d4563ce Address:95d157c1-a440-c46a-0d3e-40707d4563ce} {Suffrage:Voter ID:server-20ce517a-3433-6266-3126-63cfd16a1a27 Address:20ce517a-3433-6266-3126-63cfd16a1a27} {Suffrage:Voter ID:server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 Address:a33a7d16-94dc-ff82-86ab-a4333f4b91d3} {Suffrage:Voter ID:server-744fc635-8d74-914d-af27-318108d034bd Address:744fc635-8d74-914d-af27-318108d034bd}]" testLoggerAdapter verbose: server-95d157c1-a440-c46a-0d3e-40707d4563ce: 2021-07-10T01:05:11.738Z [INFO] server-95d157c1-a440-c46a-0d3e-40707d4563ce: entering follower state: follower="Node at 95d157c1-a440-c46a-0d3e-40707d4563ce [Follower]" leader= testing.go:149: server-95d157c1-a440-c46a-0d3e-40707d4563ce: 2021-07-10T01:05:11.738Z [INFO] server-95d157c1-a440-c46a-0d3e-40707d4563ce: entering follower state: follower="Node at 95d157c1-a440-c46a-0d3e-40707d4563ce [Follower]" leader= testLoggerAdapter verbose: server-20ce517a-3433-6266-3126-63cfd16a1a27: 2021-07-10T01:05:11.738Z [INFO] server-20ce517a-3433-6266-3126-63cfd16a1a27: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace9227e-5165-dd6e-c019-1d2527aa2089 Address:ace9227e-5165-dd6e-c019-1d2527aa2089} {Suffrage:Voter ID:server-1568b52b-331a-882f-74a0-d70b659fb2e4 Address:1568b52b-331a-882f-74a0-d70b659fb2e4} {Suffrage:Voter ID:server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 Address:f55d7e90-36a2-ee2a-f65d-95c1e28cdb58} {Suffrage:Voter ID:server-95d157c1-a440-c46a-0d3e-40707d4563ce Address:95d157c1-a440-c46a-0d3e-40707d4563ce} {Suffrage:Voter ID:server-20ce517a-3433-6266-3126-63cfd16a1a27 Address:20ce517a-3433-6266-3126-63cfd16a1a27} {Suffrage:Voter ID:server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 Address:a33a7d16-94dc-ff82-86ab-a4333f4b91d3} {Suffrage:Voter ID:server-744fc635-8d74-914d-af27-318108d034bd Address:744fc635-8d74-914d-af27-318108d034bd}]" testing.go:149: server-20ce517a-3433-6266-3126-63cfd16a1a27: 2021-07-10T01:05:11.738Z [INFO] server-20ce517a-3433-6266-3126-63cfd16a1a27: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace9227e-5165-dd6e-c019-1d2527aa2089 Address:ace9227e-5165-dd6e-c019-1d2527aa2089} {Suffrage:Voter ID:server-1568b52b-331a-882f-74a0-d70b659fb2e4 Address:1568b52b-331a-882f-74a0-d70b659fb2e4} {Suffrage:Voter ID:server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 Address:f55d7e90-36a2-ee2a-f65d-95c1e28cdb58} {Suffrage:Voter ID:server-95d157c1-a440-c46a-0d3e-40707d4563ce Address:95d157c1-a440-c46a-0d3e-40707d4563ce} {Suffrage:Voter ID:server-20ce517a-3433-6266-3126-63cfd16a1a27 Address:20ce517a-3433-6266-3126-63cfd16a1a27} {Suffrage:Voter ID:server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 Address:a33a7d16-94dc-ff82-86ab-a4333f4b91d3} {Suffrage:Voter ID:server-744fc635-8d74-914d-af27-318108d034bd Address:744fc635-8d74-914d-af27-318108d034bd}]" testLoggerAdapter verbose: server-20ce517a-3433-6266-3126-63cfd16a1a27: 2021-07-10T01:05:11.738Z [INFO] server-20ce517a-3433-6266-3126-63cfd16a1a27: entering follower state: follower="Node at 20ce517a-3433-6266-3126-63cfd16a1a27 [Follower]" leader= testing.go:149: server-20ce517a-3433-6266-3126-63cfd16a1a27: 2021-07-10T01:05:11.738Z [INFO] server-20ce517a-3433-6266-3126-63cfd16a1a27: entering follower state: follower="Node at 20ce517a-3433-6266-3126-63cfd16a1a27 [Follower]" leader= testLoggerAdapter verbose: server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3: 2021-07-10T01:05:11.738Z [INFO] server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace9227e-5165-dd6e-c019-1d2527aa2089 Address:ace9227e-5165-dd6e-c019-1d2527aa2089} {Suffrage:Voter ID:server-1568b52b-331a-882f-74a0-d70b659fb2e4 Address:1568b52b-331a-882f-74a0-d70b659fb2e4} {Suffrage:Voter ID:server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 Address:f55d7e90-36a2-ee2a-f65d-95c1e28cdb58} {Suffrage:Voter ID:server-95d157c1-a440-c46a-0d3e-40707d4563ce Address:95d157c1-a440-c46a-0d3e-40707d4563ce} {Suffrage:Voter ID:server-20ce517a-3433-6266-3126-63cfd16a1a27 Address:20ce517a-3433-6266-3126-63cfd16a1a27} {Suffrage:Voter ID:server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 Address:a33a7d16-94dc-ff82-86ab-a4333f4b91d3} {Suffrage:Voter ID:server-744fc635-8d74-914d-af27-318108d034bd Address:744fc635-8d74-914d-af27-318108d034bd}]" testing.go:149: server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3: 2021-07-10T01:05:11.738Z [INFO] server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace9227e-5165-dd6e-c019-1d2527aa2089 Address:ace9227e-5165-dd6e-c019-1d2527aa2089} {Suffrage:Voter ID:server-1568b52b-331a-882f-74a0-d70b659fb2e4 Address:1568b52b-331a-882f-74a0-d70b659fb2e4} {Suffrage:Voter ID:server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 Address:f55d7e90-36a2-ee2a-f65d-95c1e28cdb58} {Suffrage:Voter ID:server-95d157c1-a440-c46a-0d3e-40707d4563ce Address:95d157c1-a440-c46a-0d3e-40707d4563ce} {Suffrage:Voter ID:server-20ce517a-3433-6266-3126-63cfd16a1a27 Address:20ce517a-3433-6266-3126-63cfd16a1a27} {Suffrage:Voter ID:server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 Address:a33a7d16-94dc-ff82-86ab-a4333f4b91d3} {Suffrage:Voter ID:server-744fc635-8d74-914d-af27-318108d034bd Address:744fc635-8d74-914d-af27-318108d034bd}]" testLoggerAdapter verbose: server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3: 2021-07-10T01:05:11.738Z [INFO] server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3: entering follower state: follower="Node at a33a7d16-94dc-ff82-86ab-a4333f4b91d3 [Follower]" leader= testing.go:149: server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3: 2021-07-10T01:05:11.738Z [INFO] server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3: entering follower state: follower="Node at a33a7d16-94dc-ff82-86ab-a4333f4b91d3 [Follower]" leader= testLoggerAdapter verbose: server-744fc635-8d74-914d-af27-318108d034bd: 2021-07-10T01:05:11.738Z [INFO] server-744fc635-8d74-914d-af27-318108d034bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace9227e-5165-dd6e-c019-1d2527aa2089 Address:ace9227e-5165-dd6e-c019-1d2527aa2089} {Suffrage:Voter ID:server-1568b52b-331a-882f-74a0-d70b659fb2e4 Address:1568b52b-331a-882f-74a0-d70b659fb2e4} {Suffrage:Voter ID:server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 Address:f55d7e90-36a2-ee2a-f65d-95c1e28cdb58} {Suffrage:Voter ID:server-95d157c1-a440-c46a-0d3e-40707d4563ce Address:95d157c1-a440-c46a-0d3e-40707d4563ce} {Suffrage:Voter ID:server-20ce517a-3433-6266-3126-63cfd16a1a27 Address:20ce517a-3433-6266-3126-63cfd16a1a27} {Suffrage:Voter ID:server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 Address:a33a7d16-94dc-ff82-86ab-a4333f4b91d3} {Suffrage:Voter ID:server-744fc635-8d74-914d-af27-318108d034bd Address:744fc635-8d74-914d-af27-318108d034bd}]" testing.go:149: server-744fc635-8d74-914d-af27-318108d034bd: 2021-07-10T01:05:11.738Z [INFO] server-744fc635-8d74-914d-af27-318108d034bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ace9227e-5165-dd6e-c019-1d2527aa2089 Address:ace9227e-5165-dd6e-c019-1d2527aa2089} {Suffrage:Voter ID:server-1568b52b-331a-882f-74a0-d70b659fb2e4 Address:1568b52b-331a-882f-74a0-d70b659fb2e4} {Suffrage:Voter ID:server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 Address:f55d7e90-36a2-ee2a-f65d-95c1e28cdb58} {Suffrage:Voter ID:server-95d157c1-a440-c46a-0d3e-40707d4563ce Address:95d157c1-a440-c46a-0d3e-40707d4563ce} {Suffrage:Voter ID:server-20ce517a-3433-6266-3126-63cfd16a1a27 Address:20ce517a-3433-6266-3126-63cfd16a1a27} {Suffrage:Voter ID:server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 Address:a33a7d16-94dc-ff82-86ab-a4333f4b91d3} {Suffrage:Voter ID:server-744fc635-8d74-914d-af27-318108d034bd Address:744fc635-8d74-914d-af27-318108d034bd}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:11.739Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:11.739Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-744fc635-8d74-914d-af27-318108d034bd: 2021-07-10T01:05:11.739Z [INFO] server-744fc635-8d74-914d-af27-318108d034bd: entering follower state: follower="Node at 744fc635-8d74-914d-af27-318108d034bd [Follower]" leader= testing.go:149: server-744fc635-8d74-914d-af27-318108d034bd: 2021-07-10T01:05:11.739Z [INFO] server-744fc635-8d74-914d-af27-318108d034bd: entering follower state: follower="Node at 744fc635-8d74-914d-af27-318108d034bd [Follower]" leader= testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.796Z [WARN] server-ace9227e-5165-dd6e-c019-1d2527aa2089: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.796Z [WARN] server-ace9227e-5165-dd6e-c019-1d2527aa2089: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: entering candidate state: node="Node at ace9227e-5165-dd6e-c019-1d2527aa2089 [Candidate]" term=2 testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: entering candidate state: node="Node at ace9227e-5165-dd6e-c019-1d2527aa2089 [Candidate]" term=2 testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: election won: tally=4 testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: election won: tally=4 testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: entering leader state: leader="Node at ace9227e-5165-dd6e-c019-1d2527aa2089 [Leader]" testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: entering leader state: leader="Node at ace9227e-5165-dd6e-c019-1d2527aa2089 [Leader]" testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: added peer, starting replication: peer=server-1568b52b-331a-882f-74a0-d70b659fb2e4 testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: added peer, starting replication: peer=server-1568b52b-331a-882f-74a0-d70b659fb2e4 testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: added peer, starting replication: peer=server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: added peer, starting replication: peer=server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: added peer, starting replication: peer=server-95d157c1-a440-c46a-0d3e-40707d4563ce testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: added peer, starting replication: peer=server-95d157c1-a440-c46a-0d3e-40707d4563ce testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: added peer, starting replication: peer=server-20ce517a-3433-6266-3126-63cfd16a1a27 testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: added peer, starting replication: peer=server-20ce517a-3433-6266-3126-63cfd16a1a27 testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: added peer, starting replication: peer=server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: added peer, starting replication: peer=server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: pipelining replication: peer="{Voter server-95d157c1-a440-c46a-0d3e-40707d4563ce 95d157c1-a440-c46a-0d3e-40707d4563ce}" testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: pipelining replication: peer="{Voter server-95d157c1-a440-c46a-0d3e-40707d4563ce 95d157c1-a440-c46a-0d3e-40707d4563ce}" testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: pipelining replication: peer="{Voter server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 f55d7e90-36a2-ee2a-f65d-95c1e28cdb58}" testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: pipelining replication: peer="{Voter server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 f55d7e90-36a2-ee2a-f65d-95c1e28cdb58}" testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: added peer, starting replication: peer=server-744fc635-8d74-914d-af27-318108d034bd testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: added peer, starting replication: peer=server-744fc635-8d74-914d-af27-318108d034bd testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: pipelining replication: peer="{Voter server-1568b52b-331a-882f-74a0-d70b659fb2e4 1568b52b-331a-882f-74a0-d70b659fb2e4}" testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: pipelining replication: peer="{Voter server-1568b52b-331a-882f-74a0-d70b659fb2e4 1568b52b-331a-882f-74a0-d70b659fb2e4}" testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.798Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: pipelining replication: peer="{Voter server-744fc635-8d74-914d-af27-318108d034bd 744fc635-8d74-914d-af27-318108d034bd}" testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.798Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: pipelining replication: peer="{Voter server-744fc635-8d74-914d-af27-318108d034bd 744fc635-8d74-914d-af27-318108d034bd}" testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: pipelining replication: peer="{Voter server-20ce517a-3433-6266-3126-63cfd16a1a27 20ce517a-3433-6266-3126-63cfd16a1a27}" testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: pipelining replication: peer="{Voter server-20ce517a-3433-6266-3126-63cfd16a1a27 20ce517a-3433-6266-3126-63cfd16a1a27}" testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: pipelining replication: peer="{Voter server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 a33a7d16-94dc-ff82-86ab-a4333f4b91d3}" testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:11.797Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: pipelining replication: peer="{Voter server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 a33a7d16-94dc-ff82-86ab-a4333f4b91d3}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:11.902Z [INFO] stable state for Leader reached at 2021-07-10 01:05:11.797518348 +0000 UTC m=+17.185676318 (1 nodes), 58.388076ms from start of poll, 61.897213ms from cluster start. Timeout at 2021-07-10 01:05:11.902583956 +0000 UTC m=+17.290741926, 105.065608ms after stability testing.go:149: cluster: 2021-07-10T01:05:11.902Z [INFO] stable state for Leader reached at 2021-07-10 01:05:11.797518348 +0000 UTC m=+17.185676318 (1 nodes), 58.388076ms from start of poll, 61.897213ms from cluster start. Timeout at 2021-07-10 01:05:11.902583956 +0000 UTC m=+17.290741926, 105.065608ms after stability testLoggerAdapter verbose: cluster: 2021-07-10T01:05:11.902Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-07-10T01:05:11.902Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-07-10T01:05:12.008Z [INFO] stable state for Follower reached at 2021-07-10 01:05:11.902839378 +0000 UTC m=+17.290997328 (6 nodes), 4.7µs from start of poll, 167.218223ms from cluster start. Timeout at 2021-07-10 01:05:12.007918026 +0000 UTC m=+17.396076016, 105.078688ms after stability testing.go:149: cluster: 2021-07-10T01:05:12.008Z [INFO] stable state for Follower reached at 2021-07-10 01:05:11.902839378 +0000 UTC m=+17.290997328 (6 nodes), 4.7µs from start of poll, 167.218223ms from cluster start. Timeout at 2021-07-10 01:05:12.007918026 +0000 UTC m=+17.396076016, 105.078688ms after stability testLoggerAdapter verbose: cluster: 2021-07-10T01:05:12.008Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:12.008Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-07-10T01:05:12.113Z [INFO] stable state for Leader reached at 2021-07-10 01:05:12.008201189 +0000 UTC m=+17.396359139 (1 nodes), 2.78µs from start of poll, 272.580034ms from cluster start. Timeout at 2021-07-10 01:05:12.113270557 +0000 UTC m=+17.501428547, 105.069408ms after stability testing.go:149: cluster: 2021-07-10T01:05:12.113Z [INFO] stable state for Leader reached at 2021-07-10 01:05:12.008201189 +0000 UTC m=+17.396359139 (1 nodes), 2.78µs from start of poll, 272.580034ms from cluster start. Timeout at 2021-07-10 01:05:12.113270557 +0000 UTC m=+17.501428547, 105.069408ms after stability testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.113Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: entering candidate state: node="Node at 1568b52b-331a-882f-74a0-d70b659fb2e4 [Candidate]" term=3 testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.113Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: entering candidate state: node="Node at 1568b52b-331a-882f-74a0-d70b659fb2e4 [Candidate]" term=3 testLoggerAdapter verbose: cluster: 2021-07-10T01:05:12.113Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:12.113Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:12.113Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: entering follower state: follower="Node at ace9227e-5165-dd6e-c019-1d2527aa2089 [Follower]" leader= testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:12.113Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: entering follower state: follower="Node at ace9227e-5165-dd6e-c019-1d2527aa2089 [Follower]" leader= testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.113Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: election won: tally=4 testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.113Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: election won: tally=4 testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:12.113Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: aborting pipeline replication: peer="{Voter server-744fc635-8d74-914d-af27-318108d034bd 744fc635-8d74-914d-af27-318108d034bd}" testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.113Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: entering leader state: leader="Node at 1568b52b-331a-882f-74a0-d70b659fb2e4 [Leader]" testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:12.113Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: aborting pipeline replication: peer="{Voter server-744fc635-8d74-914d-af27-318108d034bd 744fc635-8d74-914d-af27-318108d034bd}" testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:12.114Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: aborting pipeline replication: peer="{Voter server-95d157c1-a440-c46a-0d3e-40707d4563ce 95d157c1-a440-c46a-0d3e-40707d4563ce}" testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.113Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: entering leader state: leader="Node at 1568b52b-331a-882f-74a0-d70b659fb2e4 [Leader]" testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: added peer, starting replication: peer=server-ace9227e-5165-dd6e-c019-1d2527aa2089 testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: added peer, starting replication: peer=server-ace9227e-5165-dd6e-c019-1d2527aa2089 testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: added peer, starting replication: peer=server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:12.114Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: aborting pipeline replication: peer="{Voter server-95d157c1-a440-c46a-0d3e-40707d4563ce 95d157c1-a440-c46a-0d3e-40707d4563ce}" testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: added peer, starting replication: peer=server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: added peer, starting replication: peer=server-95d157c1-a440-c46a-0d3e-40707d4563ce testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:12.114Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: aborting pipeline replication: peer="{Voter server-20ce517a-3433-6266-3126-63cfd16a1a27 20ce517a-3433-6266-3126-63cfd16a1a27}" testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: added peer, starting replication: peer=server-95d157c1-a440-c46a-0d3e-40707d4563ce testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: added peer, starting replication: peer=server-20ce517a-3433-6266-3126-63cfd16a1a27 testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: added peer, starting replication: peer=server-20ce517a-3433-6266-3126-63cfd16a1a27 testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: added peer, starting replication: peer=server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: added peer, starting replication: peer=server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:12.114Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: aborting pipeline replication: peer="{Voter server-20ce517a-3433-6266-3126-63cfd16a1a27 20ce517a-3433-6266-3126-63cfd16a1a27}" testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: pipelining replication: peer="{Voter server-ace9227e-5165-dd6e-c019-1d2527aa2089 ace9227e-5165-dd6e-c019-1d2527aa2089}" testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:12.114Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: aborting pipeline replication: peer="{Voter server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 f55d7e90-36a2-ee2a-f65d-95c1e28cdb58}" testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: pipelining replication: peer="{Voter server-ace9227e-5165-dd6e-c019-1d2527aa2089 ace9227e-5165-dd6e-c019-1d2527aa2089}" testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:12.114Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: aborting pipeline replication: peer="{Voter server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 f55d7e90-36a2-ee2a-f65d-95c1e28cdb58}" testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: pipelining replication: peer="{Voter server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 f55d7e90-36a2-ee2a-f65d-95c1e28cdb58}" testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: pipelining replication: peer="{Voter server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 f55d7e90-36a2-ee2a-f65d-95c1e28cdb58}" testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:12.114Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: aborting pipeline replication: peer="{Voter server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 a33a7d16-94dc-ff82-86ab-a4333f4b91d3}" testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: pipelining replication: peer="{Voter server-95d157c1-a440-c46a-0d3e-40707d4563ce 95d157c1-a440-c46a-0d3e-40707d4563ce}" testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:12.114Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: aborting pipeline replication: peer="{Voter server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 a33a7d16-94dc-ff82-86ab-a4333f4b91d3}" testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: pipelining replication: peer="{Voter server-95d157c1-a440-c46a-0d3e-40707d4563ce 95d157c1-a440-c46a-0d3e-40707d4563ce}" testLoggerAdapter verbose: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:12.114Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: aborting pipeline replication: peer="{Voter server-1568b52b-331a-882f-74a0-d70b659fb2e4 1568b52b-331a-882f-74a0-d70b659fb2e4}" testing.go:149: server-ace9227e-5165-dd6e-c019-1d2527aa2089: 2021-07-10T01:05:12.114Z [INFO] server-ace9227e-5165-dd6e-c019-1d2527aa2089: aborting pipeline replication: peer="{Voter server-1568b52b-331a-882f-74a0-d70b659fb2e4 1568b52b-331a-882f-74a0-d70b659fb2e4}" testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: added peer, starting replication: peer=server-744fc635-8d74-914d-af27-318108d034bd testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: added peer, starting replication: peer=server-744fc635-8d74-914d-af27-318108d034bd testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: pipelining replication: peer="{Voter server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 a33a7d16-94dc-ff82-86ab-a4333f4b91d3}" testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: pipelining replication: peer="{Voter server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 a33a7d16-94dc-ff82-86ab-a4333f4b91d3}" testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: pipelining replication: peer="{Voter server-20ce517a-3433-6266-3126-63cfd16a1a27 20ce517a-3433-6266-3126-63cfd16a1a27}" testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.114Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: pipelining replication: peer="{Voter server-20ce517a-3433-6266-3126-63cfd16a1a27 20ce517a-3433-6266-3126-63cfd16a1a27}" testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.115Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: pipelining replication: peer="{Voter server-744fc635-8d74-914d-af27-318108d034bd 744fc635-8d74-914d-af27-318108d034bd}" testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.115Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: pipelining replication: peer="{Voter server-744fc635-8d74-914d-af27-318108d034bd 744fc635-8d74-914d-af27-318108d034bd}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:12.219Z [INFO] stable state for Leader reached at 2021-07-10 01:05:12.114044125 +0000 UTC m=+17.502202075 (1 nodes), 198.922µs from start of poll, 378.42297ms from cluster start. Timeout at 2021-07-10 01:05:12.219131533 +0000 UTC m=+17.607289523, 105.087448ms after stability testing.go:149: cluster: 2021-07-10T01:05:12.219Z [INFO] stable state for Leader reached at 2021-07-10 01:05:12.114044125 +0000 UTC m=+17.502202075 (1 nodes), 198.922µs from start of poll, 378.42297ms from cluster start. Timeout at 2021-07-10 01:05:12.219131533 +0000 UTC m=+17.607289523, 105.087448ms after stability testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.219Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: aborting pipeline replication: peer="{Voter server-744fc635-8d74-914d-af27-318108d034bd 744fc635-8d74-914d-af27-318108d034bd}" testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.219Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: aborting pipeline replication: peer="{Voter server-744fc635-8d74-914d-af27-318108d034bd 744fc635-8d74-914d-af27-318108d034bd}" testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.219Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: aborting pipeline replication: peer="{Voter server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 f55d7e90-36a2-ee2a-f65d-95c1e28cdb58}" testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.219Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: aborting pipeline replication: peer="{Voter server-f55d7e90-36a2-ee2a-f65d-95c1e28cdb58 f55d7e90-36a2-ee2a-f65d-95c1e28cdb58}" testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.219Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: aborting pipeline replication: peer="{Voter server-20ce517a-3433-6266-3126-63cfd16a1a27 20ce517a-3433-6266-3126-63cfd16a1a27}" testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.219Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: aborting pipeline replication: peer="{Voter server-20ce517a-3433-6266-3126-63cfd16a1a27 20ce517a-3433-6266-3126-63cfd16a1a27}" testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.219Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: aborting pipeline replication: peer="{Voter server-95d157c1-a440-c46a-0d3e-40707d4563ce 95d157c1-a440-c46a-0d3e-40707d4563ce}" testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.219Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: aborting pipeline replication: peer="{Voter server-95d157c1-a440-c46a-0d3e-40707d4563ce 95d157c1-a440-c46a-0d3e-40707d4563ce}" testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.219Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: aborting pipeline replication: peer="{Voter server-ace9227e-5165-dd6e-c019-1d2527aa2089 ace9227e-5165-dd6e-c019-1d2527aa2089}" testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.219Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: aborting pipeline replication: peer="{Voter server-ace9227e-5165-dd6e-c019-1d2527aa2089 ace9227e-5165-dd6e-c019-1d2527aa2089}" testLoggerAdapter verbose: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.219Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: aborting pipeline replication: peer="{Voter server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 a33a7d16-94dc-ff82-86ab-a4333f4b91d3}" testing.go:149: server-1568b52b-331a-882f-74a0-d70b659fb2e4: 2021-07-10T01:05:12.219Z [INFO] server-1568b52b-331a-882f-74a0-d70b659fb2e4: aborting pipeline replication: peer="{Voter server-a33a7d16-94dc-ff82-86ab-a4333f4b91d3 a33a7d16-94dc-ff82-86ab-a4333f4b91d3}" --- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.49s) === RUN TestRaft_LeadershipTransferToInvalidID testLoggerAdapter verbose: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.222Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c1bf599e-fdef-6653-c143-6898145d4e80 Address:c1bf599e-fdef-6653-c143-6898145d4e80} {Suffrage:Voter ID:server-fe706442-30ee-16de-924a-a53418948471 Address:fe706442-30ee-16de-924a-a53418948471} {Suffrage:Voter ID:server-6b9e4268-2f95-1adc-e18f-a6e74f441426 Address:6b9e4268-2f95-1adc-e18f-a6e74f441426}]" testing.go:149: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.222Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c1bf599e-fdef-6653-c143-6898145d4e80 Address:c1bf599e-fdef-6653-c143-6898145d4e80} {Suffrage:Voter ID:server-fe706442-30ee-16de-924a-a53418948471 Address:fe706442-30ee-16de-924a-a53418948471} {Suffrage:Voter ID:server-6b9e4268-2f95-1adc-e18f-a6e74f441426 Address:6b9e4268-2f95-1adc-e18f-a6e74f441426}]" testLoggerAdapter verbose: server-fe706442-30ee-16de-924a-a53418948471: 2021-07-10T01:05:12.222Z [INFO] server-fe706442-30ee-16de-924a-a53418948471: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c1bf599e-fdef-6653-c143-6898145d4e80 Address:c1bf599e-fdef-6653-c143-6898145d4e80} {Suffrage:Voter ID:server-fe706442-30ee-16de-924a-a53418948471 Address:fe706442-30ee-16de-924a-a53418948471} {Suffrage:Voter ID:server-6b9e4268-2f95-1adc-e18f-a6e74f441426 Address:6b9e4268-2f95-1adc-e18f-a6e74f441426}]" testing.go:149: server-fe706442-30ee-16de-924a-a53418948471: 2021-07-10T01:05:12.222Z [INFO] server-fe706442-30ee-16de-924a-a53418948471: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c1bf599e-fdef-6653-c143-6898145d4e80 Address:c1bf599e-fdef-6653-c143-6898145d4e80} {Suffrage:Voter ID:server-fe706442-30ee-16de-924a-a53418948471 Address:fe706442-30ee-16de-924a-a53418948471} {Suffrage:Voter ID:server-6b9e4268-2f95-1adc-e18f-a6e74f441426 Address:6b9e4268-2f95-1adc-e18f-a6e74f441426}]" testLoggerAdapter verbose: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.222Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: entering follower state: follower="Node at c1bf599e-fdef-6653-c143-6898145d4e80 [Follower]" leader= testLoggerAdapter verbose: server-6b9e4268-2f95-1adc-e18f-a6e74f441426: 2021-07-10T01:05:12.222Z [INFO] server-6b9e4268-2f95-1adc-e18f-a6e74f441426: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c1bf599e-fdef-6653-c143-6898145d4e80 Address:c1bf599e-fdef-6653-c143-6898145d4e80} {Suffrage:Voter ID:server-fe706442-30ee-16de-924a-a53418948471 Address:fe706442-30ee-16de-924a-a53418948471} {Suffrage:Voter ID:server-6b9e4268-2f95-1adc-e18f-a6e74f441426 Address:6b9e4268-2f95-1adc-e18f-a6e74f441426}]" testing.go:149: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.222Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: entering follower state: follower="Node at c1bf599e-fdef-6653-c143-6898145d4e80 [Follower]" leader= testing.go:149: server-6b9e4268-2f95-1adc-e18f-a6e74f441426: 2021-07-10T01:05:12.222Z [INFO] server-6b9e4268-2f95-1adc-e18f-a6e74f441426: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c1bf599e-fdef-6653-c143-6898145d4e80 Address:c1bf599e-fdef-6653-c143-6898145d4e80} {Suffrage:Voter ID:server-fe706442-30ee-16de-924a-a53418948471 Address:fe706442-30ee-16de-924a-a53418948471} {Suffrage:Voter ID:server-6b9e4268-2f95-1adc-e18f-a6e74f441426 Address:6b9e4268-2f95-1adc-e18f-a6e74f441426}]" testLoggerAdapter verbose: server-fe706442-30ee-16de-924a-a53418948471: 2021-07-10T01:05:12.223Z [INFO] server-fe706442-30ee-16de-924a-a53418948471: entering follower state: follower="Node at fe706442-30ee-16de-924a-a53418948471 [Follower]" leader= testing.go:149: server-fe706442-30ee-16de-924a-a53418948471: 2021-07-10T01:05:12.223Z [INFO] server-fe706442-30ee-16de-924a-a53418948471: entering follower state: follower="Node at fe706442-30ee-16de-924a-a53418948471 [Follower]" leader= testLoggerAdapter verbose: cluster: 2021-07-10T01:05:12.223Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:12.223Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-6b9e4268-2f95-1adc-e18f-a6e74f441426: 2021-07-10T01:05:12.223Z [INFO] server-6b9e4268-2f95-1adc-e18f-a6e74f441426: entering follower state: follower="Node at 6b9e4268-2f95-1adc-e18f-a6e74f441426 [Follower]" leader= testing.go:149: server-6b9e4268-2f95-1adc-e18f-a6e74f441426: 2021-07-10T01:05:12.223Z [INFO] server-6b9e4268-2f95-1adc-e18f-a6e74f441426: entering follower state: follower="Node at 6b9e4268-2f95-1adc-e18f-a6e74f441426 [Follower]" leader= testLoggerAdapter verbose: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.273Z [WARN] server-c1bf599e-fdef-6653-c143-6898145d4e80: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.273Z [WARN] server-c1bf599e-fdef-6653-c143-6898145d4e80: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.274Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: entering candidate state: node="Node at c1bf599e-fdef-6653-c143-6898145d4e80 [Candidate]" term=2 testing.go:149: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.274Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: entering candidate state: node="Node at c1bf599e-fdef-6653-c143-6898145d4e80 [Candidate]" term=2 testLoggerAdapter verbose: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.274Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: election won: tally=2 testing.go:149: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.274Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: election won: tally=2 testLoggerAdapter verbose: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.274Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: entering leader state: leader="Node at c1bf599e-fdef-6653-c143-6898145d4e80 [Leader]" testing.go:149: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.274Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: entering leader state: leader="Node at c1bf599e-fdef-6653-c143-6898145d4e80 [Leader]" testLoggerAdapter verbose: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.274Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: added peer, starting replication: peer=server-fe706442-30ee-16de-924a-a53418948471 testing.go:149: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.274Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: added peer, starting replication: peer=server-fe706442-30ee-16de-924a-a53418948471 testLoggerAdapter verbose: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.274Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: added peer, starting replication: peer=server-6b9e4268-2f95-1adc-e18f-a6e74f441426 testing.go:149: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.274Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: added peer, starting replication: peer=server-6b9e4268-2f95-1adc-e18f-a6e74f441426 testLoggerAdapter verbose: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.274Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: pipelining replication: peer="{Voter server-fe706442-30ee-16de-924a-a53418948471 fe706442-30ee-16de-924a-a53418948471}" testing.go:149: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.274Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: pipelining replication: peer="{Voter server-fe706442-30ee-16de-924a-a53418948471 fe706442-30ee-16de-924a-a53418948471}" testLoggerAdapter verbose: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.274Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: pipelining replication: peer="{Voter server-6b9e4268-2f95-1adc-e18f-a6e74f441426 6b9e4268-2f95-1adc-e18f-a6e74f441426}" testing.go:149: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.274Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: pipelining replication: peer="{Voter server-6b9e4268-2f95-1adc-e18f-a6e74f441426 6b9e4268-2f95-1adc-e18f-a6e74f441426}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:12.379Z [INFO] stable state for Leader reached at 2021-07-10 01:05:12.274413376 +0000 UTC m=+17.662571366 (1 nodes), 51.287941ms from start of poll, 52.409333ms from cluster start. Timeout at 2021-07-10 01:05:12.379420583 +0000 UTC m=+17.767578533, 105.007167ms after stability testing.go:149: cluster: 2021-07-10T01:05:12.379Z [INFO] stable state for Leader reached at 2021-07-10 01:05:12.274413376 +0000 UTC m=+17.662571366 (1 nodes), 51.287941ms from start of poll, 52.409333ms from cluster start. Timeout at 2021-07-10 01:05:12.379420583 +0000 UTC m=+17.767578533, 105.007167ms after stability testLoggerAdapter verbose: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.379Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: aborting pipeline replication: peer="{Voter server-6b9e4268-2f95-1adc-e18f-a6e74f441426 6b9e4268-2f95-1adc-e18f-a6e74f441426}" testing.go:149: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.379Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: aborting pipeline replication: peer="{Voter server-6b9e4268-2f95-1adc-e18f-a6e74f441426 6b9e4268-2f95-1adc-e18f-a6e74f441426}" testLoggerAdapter verbose: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.379Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: aborting pipeline replication: peer="{Voter server-fe706442-30ee-16de-924a-a53418948471 fe706442-30ee-16de-924a-a53418948471}" testing.go:149: server-c1bf599e-fdef-6653-c143-6898145d4e80: 2021-07-10T01:05:12.379Z [INFO] server-c1bf599e-fdef-6653-c143-6898145d4e80: aborting pipeline replication: peer="{Voter server-fe706442-30ee-16de-924a-a53418948471 fe706442-30ee-16de-924a-a53418948471}" --- PASS: TestRaft_LeadershipTransferToInvalidID (0.16s) === RUN TestRaft_LeadershipTransferToInvalidAddress testLoggerAdapter verbose: server-b1772040-3133-b00b-7350-12856ccfada7: 2021-07-10T01:05:12.381Z [INFO] server-b1772040-3133-b00b-7350-12856ccfada7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b1772040-3133-b00b-7350-12856ccfada7 Address:b1772040-3133-b00b-7350-12856ccfada7} {Suffrage:Voter ID:server-d339639b-0c34-3129-0da2-ac4a7d72b2e9 Address:d339639b-0c34-3129-0da2-ac4a7d72b2e9} {Suffrage:Voter ID:server-5ee30373-c40a-9687-1f27-9b105599e38a Address:5ee30373-c40a-9687-1f27-9b105599e38a}]" testing.go:149: server-b1772040-3133-b00b-7350-12856ccfada7: 2021-07-10T01:05:12.381Z [INFO] server-b1772040-3133-b00b-7350-12856ccfada7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b1772040-3133-b00b-7350-12856ccfada7 Address:b1772040-3133-b00b-7350-12856ccfada7} {Suffrage:Voter ID:server-d339639b-0c34-3129-0da2-ac4a7d72b2e9 Address:d339639b-0c34-3129-0da2-ac4a7d72b2e9} {Suffrage:Voter ID:server-5ee30373-c40a-9687-1f27-9b105599e38a Address:5ee30373-c40a-9687-1f27-9b105599e38a}]" testLoggerAdapter verbose: server-b1772040-3133-b00b-7350-12856ccfada7: 2021-07-10T01:05:12.382Z [INFO] server-b1772040-3133-b00b-7350-12856ccfada7: entering follower state: follower="Node at b1772040-3133-b00b-7350-12856ccfada7 [Follower]" leader= testing.go:149: server-b1772040-3133-b00b-7350-12856ccfada7: 2021-07-10T01:05:12.382Z [INFO] server-b1772040-3133-b00b-7350-12856ccfada7: entering follower state: follower="Node at b1772040-3133-b00b-7350-12856ccfada7 [Follower]" leader= testLoggerAdapter verbose: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.382Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b1772040-3133-b00b-7350-12856ccfada7 Address:b1772040-3133-b00b-7350-12856ccfada7} {Suffrage:Voter ID:server-d339639b-0c34-3129-0da2-ac4a7d72b2e9 Address:d339639b-0c34-3129-0da2-ac4a7d72b2e9} {Suffrage:Voter ID:server-5ee30373-c40a-9687-1f27-9b105599e38a Address:5ee30373-c40a-9687-1f27-9b105599e38a}]" testing.go:149: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.382Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b1772040-3133-b00b-7350-12856ccfada7 Address:b1772040-3133-b00b-7350-12856ccfada7} {Suffrage:Voter ID:server-d339639b-0c34-3129-0da2-ac4a7d72b2e9 Address:d339639b-0c34-3129-0da2-ac4a7d72b2e9} {Suffrage:Voter ID:server-5ee30373-c40a-9687-1f27-9b105599e38a Address:5ee30373-c40a-9687-1f27-9b105599e38a}]" testLoggerAdapter verbose: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.382Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: entering follower state: follower="Node at d339639b-0c34-3129-0da2-ac4a7d72b2e9 [Follower]" leader= testing.go:149: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.382Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: entering follower state: follower="Node at d339639b-0c34-3129-0da2-ac4a7d72b2e9 [Follower]" leader= testLoggerAdapter verbose: server-5ee30373-c40a-9687-1f27-9b105599e38a: 2021-07-10T01:05:12.382Z [INFO] server-5ee30373-c40a-9687-1f27-9b105599e38a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b1772040-3133-b00b-7350-12856ccfada7 Address:b1772040-3133-b00b-7350-12856ccfada7} {Suffrage:Voter ID:server-d339639b-0c34-3129-0da2-ac4a7d72b2e9 Address:d339639b-0c34-3129-0da2-ac4a7d72b2e9} {Suffrage:Voter ID:server-5ee30373-c40a-9687-1f27-9b105599e38a Address:5ee30373-c40a-9687-1f27-9b105599e38a}]" testing.go:149: server-5ee30373-c40a-9687-1f27-9b105599e38a: 2021-07-10T01:05:12.382Z [INFO] server-5ee30373-c40a-9687-1f27-9b105599e38a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b1772040-3133-b00b-7350-12856ccfada7 Address:b1772040-3133-b00b-7350-12856ccfada7} {Suffrage:Voter ID:server-d339639b-0c34-3129-0da2-ac4a7d72b2e9 Address:d339639b-0c34-3129-0da2-ac4a7d72b2e9} {Suffrage:Voter ID:server-5ee30373-c40a-9687-1f27-9b105599e38a Address:5ee30373-c40a-9687-1f27-9b105599e38a}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:12.382Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-07-10T01:05:12.382Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-5ee30373-c40a-9687-1f27-9b105599e38a: 2021-07-10T01:05:12.382Z [INFO] server-5ee30373-c40a-9687-1f27-9b105599e38a: entering follower state: follower="Node at 5ee30373-c40a-9687-1f27-9b105599e38a [Follower]" leader= testing.go:149: server-5ee30373-c40a-9687-1f27-9b105599e38a: 2021-07-10T01:05:12.382Z [INFO] server-5ee30373-c40a-9687-1f27-9b105599e38a: entering follower state: follower="Node at 5ee30373-c40a-9687-1f27-9b105599e38a [Follower]" leader= testLoggerAdapter verbose: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.460Z [WARN] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.460Z [WARN] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.460Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: entering candidate state: node="Node at d339639b-0c34-3129-0da2-ac4a7d72b2e9 [Candidate]" term=2 testing.go:149: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.460Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: entering candidate state: node="Node at d339639b-0c34-3129-0da2-ac4a7d72b2e9 [Candidate]" term=2 testLoggerAdapter verbose: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.460Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: election won: tally=2 testing.go:149: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.460Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: election won: tally=2 testLoggerAdapter verbose: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.460Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: entering leader state: leader="Node at d339639b-0c34-3129-0da2-ac4a7d72b2e9 [Leader]" testing.go:149: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.460Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: entering leader state: leader="Node at d339639b-0c34-3129-0da2-ac4a7d72b2e9 [Leader]" testLoggerAdapter verbose: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.460Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: added peer, starting replication: peer=server-b1772040-3133-b00b-7350-12856ccfada7 testing.go:149: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.460Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: added peer, starting replication: peer=server-b1772040-3133-b00b-7350-12856ccfada7 testLoggerAdapter verbose: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.460Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: added peer, starting replication: peer=server-5ee30373-c40a-9687-1f27-9b105599e38a testing.go:149: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.460Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: added peer, starting replication: peer=server-5ee30373-c40a-9687-1f27-9b105599e38a testLoggerAdapter verbose: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.460Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: pipelining replication: peer="{Voter server-b1772040-3133-b00b-7350-12856ccfada7 b1772040-3133-b00b-7350-12856ccfada7}" testing.go:149: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.460Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: pipelining replication: peer="{Voter server-b1772040-3133-b00b-7350-12856ccfada7 b1772040-3133-b00b-7350-12856ccfada7}" testLoggerAdapter verbose: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.460Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: pipelining replication: peer="{Voter server-5ee30373-c40a-9687-1f27-9b105599e38a 5ee30373-c40a-9687-1f27-9b105599e38a}" testing.go:149: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.460Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: pipelining replication: peer="{Voter server-5ee30373-c40a-9687-1f27-9b105599e38a 5ee30373-c40a-9687-1f27-9b105599e38a}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:12.565Z [INFO] stable state for Follower reached at 2021-07-10 01:05:12.4607527 +0000 UTC m=+17.848910650 (2 nodes), 77.884761ms from start of poll, 79.077434ms from cluster start. Timeout at 2021-07-10 01:05:12.565821748 +0000 UTC m=+17.953979718, 105.069068ms after stability testing.go:149: cluster: 2021-07-10T01:05:12.565Z [INFO] stable state for Follower reached at 2021-07-10 01:05:12.4607527 +0000 UTC m=+17.848910650 (2 nodes), 77.884761ms from start of poll, 79.077434ms from cluster start. Timeout at 2021-07-10 01:05:12.565821748 +0000 UTC m=+17.953979718, 105.069068ms after stability testLoggerAdapter verbose: cluster: 2021-07-10T01:05:12.566Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:12.566Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-07-10T01:05:12.671Z [INFO] stable state for Leader reached at 2021-07-10 01:05:12.56606609 +0000 UTC m=+17.954224060 (1 nodes), 3.3µs from start of poll, 184.390844ms from cluster start. Timeout at 2021-07-10 01:05:12.671101018 +0000 UTC m=+18.059259008, 105.034948ms after stability testing.go:149: cluster: 2021-07-10T01:05:12.671Z [INFO] stable state for Leader reached at 2021-07-10 01:05:12.56606609 +0000 UTC m=+17.954224060 (1 nodes), 3.3µs from start of poll, 184.390844ms from cluster start. Timeout at 2021-07-10 01:05:12.671101018 +0000 UTC m=+18.059259008, 105.034948ms after stability testLoggerAdapter verbose: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.671Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: aborting pipeline replication: peer="{Voter server-5ee30373-c40a-9687-1f27-9b105599e38a 5ee30373-c40a-9687-1f27-9b105599e38a}" testing.go:149: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.671Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: aborting pipeline replication: peer="{Voter server-5ee30373-c40a-9687-1f27-9b105599e38a 5ee30373-c40a-9687-1f27-9b105599e38a}" testLoggerAdapter verbose: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.671Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: aborting pipeline replication: peer="{Voter server-b1772040-3133-b00b-7350-12856ccfada7 b1772040-3133-b00b-7350-12856ccfada7}" testing.go:149: server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: 2021-07-10T01:05:12.671Z [INFO] server-d339639b-0c34-3129-0da2-ac4a7d72b2e9: aborting pipeline replication: peer="{Voter server-b1772040-3133-b00b-7350-12856ccfada7 b1772040-3133-b00b-7350-12856ccfada7}" --- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.29s) === RUN TestRaft_LeadershipTransferToBehindServer testLoggerAdapter verbose: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.673Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1dd37b1b-6d57-d208-7177-96f51751af2c Address:1dd37b1b-6d57-d208-7177-96f51751af2c} {Suffrage:Voter ID:server-5be7c512-6ae8-11d8-6ad2-be45b2bec328 Address:5be7c512-6ae8-11d8-6ad2-be45b2bec328} {Suffrage:Voter ID:server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f Address:c5a447e6-f091-7f22-7c30-c48d1dbbf37f}]" testing.go:149: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.673Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1dd37b1b-6d57-d208-7177-96f51751af2c Address:1dd37b1b-6d57-d208-7177-96f51751af2c} {Suffrage:Voter ID:server-5be7c512-6ae8-11d8-6ad2-be45b2bec328 Address:5be7c512-6ae8-11d8-6ad2-be45b2bec328} {Suffrage:Voter ID:server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f Address:c5a447e6-f091-7f22-7c30-c48d1dbbf37f}]" testLoggerAdapter verbose: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.673Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: entering follower state: follower="Node at 1dd37b1b-6d57-d208-7177-96f51751af2c [Follower]" leader= testing.go:149: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.673Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: entering follower state: follower="Node at 1dd37b1b-6d57-d208-7177-96f51751af2c [Follower]" leader= testLoggerAdapter verbose: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:12.673Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1dd37b1b-6d57-d208-7177-96f51751af2c Address:1dd37b1b-6d57-d208-7177-96f51751af2c} {Suffrage:Voter ID:server-5be7c512-6ae8-11d8-6ad2-be45b2bec328 Address:5be7c512-6ae8-11d8-6ad2-be45b2bec328} {Suffrage:Voter ID:server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f Address:c5a447e6-f091-7f22-7c30-c48d1dbbf37f}]" testing.go:149: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:12.673Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1dd37b1b-6d57-d208-7177-96f51751af2c Address:1dd37b1b-6d57-d208-7177-96f51751af2c} {Suffrage:Voter ID:server-5be7c512-6ae8-11d8-6ad2-be45b2bec328 Address:5be7c512-6ae8-11d8-6ad2-be45b2bec328} {Suffrage:Voter ID:server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f Address:c5a447e6-f091-7f22-7c30-c48d1dbbf37f}]" testLoggerAdapter verbose: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:12.673Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: entering follower state: follower="Node at 5be7c512-6ae8-11d8-6ad2-be45b2bec328 [Follower]" leader= testing.go:149: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:12.673Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: entering follower state: follower="Node at 5be7c512-6ae8-11d8-6ad2-be45b2bec328 [Follower]" leader= testLoggerAdapter verbose: server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f: 2021-07-10T01:05:12.673Z [INFO] server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1dd37b1b-6d57-d208-7177-96f51751af2c Address:1dd37b1b-6d57-d208-7177-96f51751af2c} {Suffrage:Voter ID:server-5be7c512-6ae8-11d8-6ad2-be45b2bec328 Address:5be7c512-6ae8-11d8-6ad2-be45b2bec328} {Suffrage:Voter ID:server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f Address:c5a447e6-f091-7f22-7c30-c48d1dbbf37f}]" testing.go:149: server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f: 2021-07-10T01:05:12.673Z [INFO] server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1dd37b1b-6d57-d208-7177-96f51751af2c Address:1dd37b1b-6d57-d208-7177-96f51751af2c} {Suffrage:Voter ID:server-5be7c512-6ae8-11d8-6ad2-be45b2bec328 Address:5be7c512-6ae8-11d8-6ad2-be45b2bec328} {Suffrage:Voter ID:server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f Address:c5a447e6-f091-7f22-7c30-c48d1dbbf37f}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:12.673Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:12.673Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f: 2021-07-10T01:05:12.673Z [INFO] server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f: entering follower state: follower="Node at c5a447e6-f091-7f22-7c30-c48d1dbbf37f [Follower]" leader= testing.go:149: server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f: 2021-07-10T01:05:12.673Z [INFO] server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f: entering follower state: follower="Node at c5a447e6-f091-7f22-7c30-c48d1dbbf37f [Follower]" leader= testLoggerAdapter verbose: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.748Z [WARN] server-1dd37b1b-6d57-d208-7177-96f51751af2c: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.748Z [WARN] server-1dd37b1b-6d57-d208-7177-96f51751af2c: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.748Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: entering candidate state: node="Node at 1dd37b1b-6d57-d208-7177-96f51751af2c [Candidate]" term=2 testing.go:149: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.748Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: entering candidate state: node="Node at 1dd37b1b-6d57-d208-7177-96f51751af2c [Candidate]" term=2 testLoggerAdapter verbose: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.749Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: election won: tally=2 testing.go:149: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.749Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: election won: tally=2 testLoggerAdapter verbose: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.749Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: entering leader state: leader="Node at 1dd37b1b-6d57-d208-7177-96f51751af2c [Leader]" testing.go:149: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.749Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: entering leader state: leader="Node at 1dd37b1b-6d57-d208-7177-96f51751af2c [Leader]" testLoggerAdapter verbose: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.749Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: added peer, starting replication: peer=server-5be7c512-6ae8-11d8-6ad2-be45b2bec328 testing.go:149: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.749Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: added peer, starting replication: peer=server-5be7c512-6ae8-11d8-6ad2-be45b2bec328 testLoggerAdapter verbose: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.749Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: added peer, starting replication: peer=server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f testing.go:149: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.749Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: added peer, starting replication: peer=server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f testLoggerAdapter verbose: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.749Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: pipelining replication: peer="{Voter server-5be7c512-6ae8-11d8-6ad2-be45b2bec328 5be7c512-6ae8-11d8-6ad2-be45b2bec328}" testing.go:149: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.749Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: pipelining replication: peer="{Voter server-5be7c512-6ae8-11d8-6ad2-be45b2bec328 5be7c512-6ae8-11d8-6ad2-be45b2bec328}" testLoggerAdapter verbose: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.749Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: pipelining replication: peer="{Voter server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f c5a447e6-f091-7f22-7c30-c48d1dbbf37f}" testing.go:149: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.749Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: pipelining replication: peer="{Voter server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f c5a447e6-f091-7f22-7c30-c48d1dbbf37f}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:12.854Z [INFO] stable state for Leader reached at 2021-07-10 01:05:12.74914728 +0000 UTC m=+18.137305230 (1 nodes), 75.171892ms from start of poll, 76.190763ms from cluster start. Timeout at 2021-07-10 01:05:12.854235728 +0000 UTC m=+18.242393698, 105.088468ms after stability testing.go:149: cluster: 2021-07-10T01:05:12.854Z [INFO] stable state for Leader reached at 2021-07-10 01:05:12.74914728 +0000 UTC m=+18.137305230 (1 nodes), 75.171892ms from start of poll, 76.190763ms from cluster start. Timeout at 2021-07-10 01:05:12.854235728 +0000 UTC m=+18.242393698, 105.088468ms after stability testLoggerAdapter verbose: cluster: 2021-07-10T01:05:12.854Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-07-10T01:05:12.854Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-07-10T01:05:12.959Z [INFO] stable state for Follower reached at 2021-07-10 01:05:12.854476171 +0000 UTC m=+18.242634161 (2 nodes), 3.04µs from start of poll, 181.519694ms from cluster start. Timeout at 2021-07-10 01:05:12.959553339 +0000 UTC m=+18.347711309, 105.077148ms after stability testing.go:149: cluster: 2021-07-10T01:05:12.959Z [INFO] stable state for Follower reached at 2021-07-10 01:05:12.854476171 +0000 UTC m=+18.242634161 (2 nodes), 3.04µs from start of poll, 181.519694ms from cluster start. Timeout at 2021-07-10 01:05:12.959553339 +0000 UTC m=+18.347711309, 105.077148ms after stability testLoggerAdapter verbose: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:12.995Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: entering candidate state: node="Node at 5be7c512-6ae8-11d8-6ad2-be45b2bec328 [Candidate]" term=3 testLoggerAdapter verbose: cluster: 2021-07-10T01:05:12.996Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:12.995Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: entering candidate state: node="Node at 5be7c512-6ae8-11d8-6ad2-be45b2bec328 [Candidate]" term=3 testing.go:149: cluster: 2021-07-10T01:05:12.996Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.996Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: entering follower state: follower="Node at 1dd37b1b-6d57-d208-7177-96f51751af2c [Follower]" leader= testing.go:149: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.996Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: entering follower state: follower="Node at 1dd37b1b-6d57-d208-7177-96f51751af2c [Follower]" leader= testLoggerAdapter verbose: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:12.996Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: election won: tally=2 testLoggerAdapter verbose: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.996Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: aborting pipeline replication: peer="{Voter server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f c5a447e6-f091-7f22-7c30-c48d1dbbf37f}" testing.go:149: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.996Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: aborting pipeline replication: peer="{Voter server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f c5a447e6-f091-7f22-7c30-c48d1dbbf37f}" testing.go:149: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:12.996Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: election won: tally=2 testLoggerAdapter verbose: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:12.996Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: entering leader state: leader="Node at 5be7c512-6ae8-11d8-6ad2-be45b2bec328 [Leader]" testLoggerAdapter verbose: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.996Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: aborting pipeline replication: peer="{Voter server-5be7c512-6ae8-11d8-6ad2-be45b2bec328 5be7c512-6ae8-11d8-6ad2-be45b2bec328}" testing.go:149: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:12.996Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: entering leader state: leader="Node at 5be7c512-6ae8-11d8-6ad2-be45b2bec328 [Leader]" testLoggerAdapter verbose: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:12.997Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: added peer, starting replication: peer=server-1dd37b1b-6d57-d208-7177-96f51751af2c testing.go:149: server-1dd37b1b-6d57-d208-7177-96f51751af2c: 2021-07-10T01:05:12.996Z [INFO] server-1dd37b1b-6d57-d208-7177-96f51751af2c: aborting pipeline replication: peer="{Voter server-5be7c512-6ae8-11d8-6ad2-be45b2bec328 5be7c512-6ae8-11d8-6ad2-be45b2bec328}" testing.go:149: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:12.997Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: added peer, starting replication: peer=server-1dd37b1b-6d57-d208-7177-96f51751af2c testLoggerAdapter verbose: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:12.997Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: added peer, starting replication: peer=server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f testing.go:149: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:12.997Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: added peer, starting replication: peer=server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f testLoggerAdapter verbose: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:12.997Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: pipelining replication: peer="{Voter server-1dd37b1b-6d57-d208-7177-96f51751af2c 1dd37b1b-6d57-d208-7177-96f51751af2c}" testing.go:149: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:12.997Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: pipelining replication: peer="{Voter server-1dd37b1b-6d57-d208-7177-96f51751af2c 1dd37b1b-6d57-d208-7177-96f51751af2c}" testLoggerAdapter verbose: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:12.997Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: pipelining replication: peer="{Voter server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f c5a447e6-f091-7f22-7c30-c48d1dbbf37f}" testing.go:149: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:12.997Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: pipelining replication: peer="{Voter server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f c5a447e6-f091-7f22-7c30-c48d1dbbf37f}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:13.101Z [INFO] stable state for Leader reached at 2021-07-10 01:05:12.99670791 +0000 UTC m=+18.384865880 (1 nodes), 430.184µs from start of poll, 323.751413ms from cluster start. Timeout at 2021-07-10 01:05:13.101790738 +0000 UTC m=+18.489948728, 105.082848ms after stability testing.go:149: cluster: 2021-07-10T01:05:13.101Z [INFO] stable state for Leader reached at 2021-07-10 01:05:12.99670791 +0000 UTC m=+18.384865880 (1 nodes), 430.184µs from start of poll, 323.751413ms from cluster start. Timeout at 2021-07-10 01:05:13.101790738 +0000 UTC m=+18.489948728, 105.082848ms after stability testLoggerAdapter verbose: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:13.102Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: aborting pipeline replication: peer="{Voter server-1dd37b1b-6d57-d208-7177-96f51751af2c 1dd37b1b-6d57-d208-7177-96f51751af2c}" testing.go:149: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:13.102Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: aborting pipeline replication: peer="{Voter server-1dd37b1b-6d57-d208-7177-96f51751af2c 1dd37b1b-6d57-d208-7177-96f51751af2c}" testLoggerAdapter verbose: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:13.102Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: aborting pipeline replication: peer="{Voter server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f c5a447e6-f091-7f22-7c30-c48d1dbbf37f}" testing.go:149: server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: 2021-07-10T01:05:13.102Z [INFO] server-5be7c512-6ae8-11d8-6ad2-be45b2bec328: aborting pipeline replication: peer="{Voter server-c5a447e6-f091-7f22-7c30-c48d1dbbf37f c5a447e6-f091-7f22-7c30-c48d1dbbf37f}" --- PASS: TestRaft_LeadershipTransferToBehindServer (0.43s) === RUN TestRaft_LeadershipTransferToItself testLoggerAdapter verbose: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.103Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2d5700b-f740-36ed-03c4-d6cd984af401 Address:c2d5700b-f740-36ed-03c4-d6cd984af401} {Suffrage:Voter ID:server-489b32cf-416b-083b-fbd8-739b930ddf7f Address:489b32cf-416b-083b-fbd8-739b930ddf7f} {Suffrage:Voter ID:server-c3c7356b-3f6f-c121-b973-8e4356c8576a Address:c3c7356b-3f6f-c121-b973-8e4356c8576a}]" testing.go:149: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.103Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2d5700b-f740-36ed-03c4-d6cd984af401 Address:c2d5700b-f740-36ed-03c4-d6cd984af401} {Suffrage:Voter ID:server-489b32cf-416b-083b-fbd8-739b930ddf7f Address:489b32cf-416b-083b-fbd8-739b930ddf7f} {Suffrage:Voter ID:server-c3c7356b-3f6f-c121-b973-8e4356c8576a Address:c3c7356b-3f6f-c121-b973-8e4356c8576a}]" testLoggerAdapter verbose: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.103Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: entering follower state: follower="Node at c2d5700b-f740-36ed-03c4-d6cd984af401 [Follower]" leader= testing.go:149: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.103Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: entering follower state: follower="Node at c2d5700b-f740-36ed-03c4-d6cd984af401 [Follower]" leader= testLoggerAdapter verbose: server-489b32cf-416b-083b-fbd8-739b930ddf7f: 2021-07-10T01:05:13.103Z [INFO] server-489b32cf-416b-083b-fbd8-739b930ddf7f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2d5700b-f740-36ed-03c4-d6cd984af401 Address:c2d5700b-f740-36ed-03c4-d6cd984af401} {Suffrage:Voter ID:server-489b32cf-416b-083b-fbd8-739b930ddf7f Address:489b32cf-416b-083b-fbd8-739b930ddf7f} {Suffrage:Voter ID:server-c3c7356b-3f6f-c121-b973-8e4356c8576a Address:c3c7356b-3f6f-c121-b973-8e4356c8576a}]" testing.go:149: server-489b32cf-416b-083b-fbd8-739b930ddf7f: 2021-07-10T01:05:13.103Z [INFO] server-489b32cf-416b-083b-fbd8-739b930ddf7f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2d5700b-f740-36ed-03c4-d6cd984af401 Address:c2d5700b-f740-36ed-03c4-d6cd984af401} {Suffrage:Voter ID:server-489b32cf-416b-083b-fbd8-739b930ddf7f Address:489b32cf-416b-083b-fbd8-739b930ddf7f} {Suffrage:Voter ID:server-c3c7356b-3f6f-c121-b973-8e4356c8576a Address:c3c7356b-3f6f-c121-b973-8e4356c8576a}]" testLoggerAdapter verbose: server-489b32cf-416b-083b-fbd8-739b930ddf7f: 2021-07-10T01:05:13.103Z [INFO] server-489b32cf-416b-083b-fbd8-739b930ddf7f: entering follower state: follower="Node at 489b32cf-416b-083b-fbd8-739b930ddf7f [Follower]" leader= testing.go:149: server-489b32cf-416b-083b-fbd8-739b930ddf7f: 2021-07-10T01:05:13.103Z [INFO] server-489b32cf-416b-083b-fbd8-739b930ddf7f: entering follower state: follower="Node at 489b32cf-416b-083b-fbd8-739b930ddf7f [Follower]" leader= testLoggerAdapter verbose: server-c3c7356b-3f6f-c121-b973-8e4356c8576a: 2021-07-10T01:05:13.104Z [INFO] server-c3c7356b-3f6f-c121-b973-8e4356c8576a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2d5700b-f740-36ed-03c4-d6cd984af401 Address:c2d5700b-f740-36ed-03c4-d6cd984af401} {Suffrage:Voter ID:server-489b32cf-416b-083b-fbd8-739b930ddf7f Address:489b32cf-416b-083b-fbd8-739b930ddf7f} {Suffrage:Voter ID:server-c3c7356b-3f6f-c121-b973-8e4356c8576a Address:c3c7356b-3f6f-c121-b973-8e4356c8576a}]" testing.go:149: server-c3c7356b-3f6f-c121-b973-8e4356c8576a: 2021-07-10T01:05:13.104Z [INFO] server-c3c7356b-3f6f-c121-b973-8e4356c8576a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2d5700b-f740-36ed-03c4-d6cd984af401 Address:c2d5700b-f740-36ed-03c4-d6cd984af401} {Suffrage:Voter ID:server-489b32cf-416b-083b-fbd8-739b930ddf7f Address:489b32cf-416b-083b-fbd8-739b930ddf7f} {Suffrage:Voter ID:server-c3c7356b-3f6f-c121-b973-8e4356c8576a Address:c3c7356b-3f6f-c121-b973-8e4356c8576a}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:13.104Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:13.104Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-c3c7356b-3f6f-c121-b973-8e4356c8576a: 2021-07-10T01:05:13.104Z [INFO] server-c3c7356b-3f6f-c121-b973-8e4356c8576a: entering follower state: follower="Node at c3c7356b-3f6f-c121-b973-8e4356c8576a [Follower]" leader= testing.go:149: server-c3c7356b-3f6f-c121-b973-8e4356c8576a: 2021-07-10T01:05:13.104Z [INFO] server-c3c7356b-3f6f-c121-b973-8e4356c8576a: entering follower state: follower="Node at c3c7356b-3f6f-c121-b973-8e4356c8576a [Follower]" leader= testLoggerAdapter verbose: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.162Z [WARN] server-c2d5700b-f740-36ed-03c4-d6cd984af401: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.162Z [WARN] server-c2d5700b-f740-36ed-03c4-d6cd984af401: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.163Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: entering candidate state: node="Node at c2d5700b-f740-36ed-03c4-d6cd984af401 [Candidate]" term=2 testing.go:149: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.163Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: entering candidate state: node="Node at c2d5700b-f740-36ed-03c4-d6cd984af401 [Candidate]" term=2 testLoggerAdapter verbose: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.163Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: election won: tally=2 testing.go:149: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.163Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: election won: tally=2 testLoggerAdapter verbose: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.163Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: entering leader state: leader="Node at c2d5700b-f740-36ed-03c4-d6cd984af401 [Leader]" testing.go:149: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.163Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: entering leader state: leader="Node at c2d5700b-f740-36ed-03c4-d6cd984af401 [Leader]" testLoggerAdapter verbose: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.163Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: added peer, starting replication: peer=server-489b32cf-416b-083b-fbd8-739b930ddf7f testing.go:149: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.163Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: added peer, starting replication: peer=server-489b32cf-416b-083b-fbd8-739b930ddf7f testLoggerAdapter verbose: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.163Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: added peer, starting replication: peer=server-c3c7356b-3f6f-c121-b973-8e4356c8576a testing.go:149: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.163Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: added peer, starting replication: peer=server-c3c7356b-3f6f-c121-b973-8e4356c8576a testLoggerAdapter verbose: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.163Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: pipelining replication: peer="{Voter server-489b32cf-416b-083b-fbd8-739b930ddf7f 489b32cf-416b-083b-fbd8-739b930ddf7f}" testing.go:149: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.163Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: pipelining replication: peer="{Voter server-489b32cf-416b-083b-fbd8-739b930ddf7f 489b32cf-416b-083b-fbd8-739b930ddf7f}" testLoggerAdapter verbose: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.163Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: pipelining replication: peer="{Voter server-c3c7356b-3f6f-c121-b973-8e4356c8576a c3c7356b-3f6f-c121-b973-8e4356c8576a}" testing.go:149: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.163Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: pipelining replication: peer="{Voter server-c3c7356b-3f6f-c121-b973-8e4356c8576a c3c7356b-3f6f-c121-b973-8e4356c8576a}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:13.268Z [INFO] stable state for Leader reached at 2021-07-10 01:05:13.163538589 +0000 UTC m=+18.551696579 (1 nodes), 59.264925ms from start of poll, 60.266435ms from cluster start. Timeout at 2021-07-10 01:05:13.268617797 +0000 UTC m=+18.656775767, 105.079188ms after stability testing.go:149: cluster: 2021-07-10T01:05:13.268Z [INFO] stable state for Leader reached at 2021-07-10 01:05:13.163538589 +0000 UTC m=+18.551696579 (1 nodes), 59.264925ms from start of poll, 60.266435ms from cluster start. Timeout at 2021-07-10 01:05:13.268617797 +0000 UTC m=+18.656775767, 105.079188ms after stability testLoggerAdapter verbose: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.268Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: cannot transfer leadership to itself testing.go:149: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.268Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: cannot transfer leadership to itself testLoggerAdapter verbose: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.268Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: aborting pipeline replication: peer="{Voter server-c3c7356b-3f6f-c121-b973-8e4356c8576a c3c7356b-3f6f-c121-b973-8e4356c8576a}" testing.go:149: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.268Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: aborting pipeline replication: peer="{Voter server-c3c7356b-3f6f-c121-b973-8e4356c8576a c3c7356b-3f6f-c121-b973-8e4356c8576a}" testLoggerAdapter verbose: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.268Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: aborting pipeline replication: peer="{Voter server-489b32cf-416b-083b-fbd8-739b930ddf7f 489b32cf-416b-083b-fbd8-739b930ddf7f}" testing.go:149: server-c2d5700b-f740-36ed-03c4-d6cd984af401: 2021-07-10T01:05:13.268Z [INFO] server-c2d5700b-f740-36ed-03c4-d6cd984af401: aborting pipeline replication: peer="{Voter server-489b32cf-416b-083b-fbd8-739b930ddf7f 489b32cf-416b-083b-fbd8-739b930ddf7f}" --- PASS: TestRaft_LeadershipTransferToItself (0.17s) === RUN TestRaft_LeadershipTransferLeaderRejectsClientRequests raft_test.go:2064: Disabled unreliable test --- SKIP: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.00s) === RUN TestRaft_LeadershipTransferLeaderReplicationTimeout testLoggerAdapter verbose: server-e9bae396-860a-aac1-bfb4-a462c2172ee0: 2021-07-10T01:05:13.270Z [INFO] server-e9bae396-860a-aac1-bfb4-a462c2172ee0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e9bae396-860a-aac1-bfb4-a462c2172ee0 Address:e9bae396-860a-aac1-bfb4-a462c2172ee0} {Suffrage:Voter ID:server-6d4f2788-2b13-a8b9-e435-a0774f871ab8 Address:6d4f2788-2b13-a8b9-e435-a0774f871ab8} {Suffrage:Voter ID:server-57b5d042-8833-3178-5c09-e3f830dfd7b4 Address:57b5d042-8833-3178-5c09-e3f830dfd7b4}]" testing.go:149: server-e9bae396-860a-aac1-bfb4-a462c2172ee0: 2021-07-10T01:05:13.270Z [INFO] server-e9bae396-860a-aac1-bfb4-a462c2172ee0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e9bae396-860a-aac1-bfb4-a462c2172ee0 Address:e9bae396-860a-aac1-bfb4-a462c2172ee0} {Suffrage:Voter ID:server-6d4f2788-2b13-a8b9-e435-a0774f871ab8 Address:6d4f2788-2b13-a8b9-e435-a0774f871ab8} {Suffrage:Voter ID:server-57b5d042-8833-3178-5c09-e3f830dfd7b4 Address:57b5d042-8833-3178-5c09-e3f830dfd7b4}]" testLoggerAdapter verbose: server-e9bae396-860a-aac1-bfb4-a462c2172ee0: 2021-07-10T01:05:13.271Z [INFO] server-e9bae396-860a-aac1-bfb4-a462c2172ee0: entering follower state: follower="Node at e9bae396-860a-aac1-bfb4-a462c2172ee0 [Follower]" leader= testing.go:149: server-e9bae396-860a-aac1-bfb4-a462c2172ee0: 2021-07-10T01:05:13.271Z [INFO] server-e9bae396-860a-aac1-bfb4-a462c2172ee0: entering follower state: follower="Node at e9bae396-860a-aac1-bfb4-a462c2172ee0 [Follower]" leader= testLoggerAdapter verbose: server-6d4f2788-2b13-a8b9-e435-a0774f871ab8: 2021-07-10T01:05:13.271Z [INFO] server-6d4f2788-2b13-a8b9-e435-a0774f871ab8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e9bae396-860a-aac1-bfb4-a462c2172ee0 Address:e9bae396-860a-aac1-bfb4-a462c2172ee0} {Suffrage:Voter ID:server-6d4f2788-2b13-a8b9-e435-a0774f871ab8 Address:6d4f2788-2b13-a8b9-e435-a0774f871ab8} {Suffrage:Voter ID:server-57b5d042-8833-3178-5c09-e3f830dfd7b4 Address:57b5d042-8833-3178-5c09-e3f830dfd7b4}]" testing.go:149: server-6d4f2788-2b13-a8b9-e435-a0774f871ab8: 2021-07-10T01:05:13.271Z [INFO] server-6d4f2788-2b13-a8b9-e435-a0774f871ab8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e9bae396-860a-aac1-bfb4-a462c2172ee0 Address:e9bae396-860a-aac1-bfb4-a462c2172ee0} {Suffrage:Voter ID:server-6d4f2788-2b13-a8b9-e435-a0774f871ab8 Address:6d4f2788-2b13-a8b9-e435-a0774f871ab8} {Suffrage:Voter ID:server-57b5d042-8833-3178-5c09-e3f830dfd7b4 Address:57b5d042-8833-3178-5c09-e3f830dfd7b4}]" testLoggerAdapter verbose: server-6d4f2788-2b13-a8b9-e435-a0774f871ab8: 2021-07-10T01:05:13.271Z [INFO] server-6d4f2788-2b13-a8b9-e435-a0774f871ab8: entering follower state: follower="Node at 6d4f2788-2b13-a8b9-e435-a0774f871ab8 [Follower]" leader= testing.go:149: server-6d4f2788-2b13-a8b9-e435-a0774f871ab8: 2021-07-10T01:05:13.271Z [INFO] server-6d4f2788-2b13-a8b9-e435-a0774f871ab8: entering follower state: follower="Node at 6d4f2788-2b13-a8b9-e435-a0774f871ab8 [Follower]" leader= testLoggerAdapter verbose: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.271Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e9bae396-860a-aac1-bfb4-a462c2172ee0 Address:e9bae396-860a-aac1-bfb4-a462c2172ee0} {Suffrage:Voter ID:server-6d4f2788-2b13-a8b9-e435-a0774f871ab8 Address:6d4f2788-2b13-a8b9-e435-a0774f871ab8} {Suffrage:Voter ID:server-57b5d042-8833-3178-5c09-e3f830dfd7b4 Address:57b5d042-8833-3178-5c09-e3f830dfd7b4}]" testing.go:149: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.271Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e9bae396-860a-aac1-bfb4-a462c2172ee0 Address:e9bae396-860a-aac1-bfb4-a462c2172ee0} {Suffrage:Voter ID:server-6d4f2788-2b13-a8b9-e435-a0774f871ab8 Address:6d4f2788-2b13-a8b9-e435-a0774f871ab8} {Suffrage:Voter ID:server-57b5d042-8833-3178-5c09-e3f830dfd7b4 Address:57b5d042-8833-3178-5c09-e3f830dfd7b4}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:13.271Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:13.271Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.271Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: entering follower state: follower="Node at 57b5d042-8833-3178-5c09-e3f830dfd7b4 [Follower]" leader= testing.go:149: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.271Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: entering follower state: follower="Node at 57b5d042-8833-3178-5c09-e3f830dfd7b4 [Follower]" leader= testLoggerAdapter verbose: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.350Z [WARN] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.350Z [WARN] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.350Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: entering candidate state: node="Node at 57b5d042-8833-3178-5c09-e3f830dfd7b4 [Candidate]" term=2 testing.go:149: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.350Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: entering candidate state: node="Node at 57b5d042-8833-3178-5c09-e3f830dfd7b4 [Candidate]" term=2 testLoggerAdapter verbose: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.350Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: election won: tally=2 testing.go:149: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.350Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: election won: tally=2 testLoggerAdapter verbose: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.350Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: entering leader state: leader="Node at 57b5d042-8833-3178-5c09-e3f830dfd7b4 [Leader]" testing.go:149: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.350Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: entering leader state: leader="Node at 57b5d042-8833-3178-5c09-e3f830dfd7b4 [Leader]" testLoggerAdapter verbose: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.350Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: added peer, starting replication: peer=server-e9bae396-860a-aac1-bfb4-a462c2172ee0 testing.go:149: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.350Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: added peer, starting replication: peer=server-e9bae396-860a-aac1-bfb4-a462c2172ee0 testLoggerAdapter verbose: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.350Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: added peer, starting replication: peer=server-6d4f2788-2b13-a8b9-e435-a0774f871ab8 testing.go:149: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.350Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: added peer, starting replication: peer=server-6d4f2788-2b13-a8b9-e435-a0774f871ab8 testLoggerAdapter verbose: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.351Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: pipelining replication: peer="{Voter server-e9bae396-860a-aac1-bfb4-a462c2172ee0 e9bae396-860a-aac1-bfb4-a462c2172ee0}" testing.go:149: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.351Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: pipelining replication: peer="{Voter server-e9bae396-860a-aac1-bfb4-a462c2172ee0 e9bae396-860a-aac1-bfb4-a462c2172ee0}" testLoggerAdapter verbose: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.351Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: pipelining replication: peer="{Voter server-6d4f2788-2b13-a8b9-e435-a0774f871ab8 6d4f2788-2b13-a8b9-e435-a0774f871ab8}" testing.go:149: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.351Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: pipelining replication: peer="{Voter server-6d4f2788-2b13-a8b9-e435-a0774f871ab8 6d4f2788-2b13-a8b9-e435-a0774f871ab8}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:13.455Z [INFO] stable state for Leader reached at 2021-07-10 01:05:13.350814004 +0000 UTC m=+18.738971994 (1 nodes), 78.927913ms from start of poll, 80.119405ms from cluster start. Timeout at 2021-07-10 01:05:13.455822931 +0000 UTC m=+18.843980921, 105.008927ms after stability testing.go:149: cluster: 2021-07-10T01:05:13.455Z [INFO] stable state for Leader reached at 2021-07-10 01:05:13.350814004 +0000 UTC m=+18.738971994 (1 nodes), 78.927913ms from start of poll, 80.119405ms from cluster start. Timeout at 2021-07-10 01:05:13.455822931 +0000 UTC m=+18.843980921, 105.008927ms after stability testLoggerAdapter verbose: cluster: 2021-07-10T01:05:13.456Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-07-10T01:05:13.456Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-07-10T01:05:13.561Z [INFO] stable state for Follower reached at 2021-07-10 01:05:13.456101314 +0000 UTC m=+18.844259304 (2 nodes), 3.321µs from start of poll, 185.406715ms from cluster start. Timeout at 2021-07-10 01:05:13.561180501 +0000 UTC m=+18.949338491, 105.079187ms after stability testing.go:149: cluster: 2021-07-10T01:05:13.561Z [INFO] stable state for Follower reached at 2021-07-10 01:05:13.456101314 +0000 UTC m=+18.844259304 (2 nodes), 3.321µs from start of poll, 185.406715ms from cluster start. Timeout at 2021-07-10 01:05:13.561180501 +0000 UTC m=+18.949338491, 105.079187ms after stability testLoggerAdapter verbose: server-e9bae396-860a-aac1-bfb4-a462c2172ee0: 2021-07-10T01:05:13.934Z [INFO] server-e9bae396-860a-aac1-bfb4-a462c2172ee0: entering candidate state: node="Node at e9bae396-860a-aac1-bfb4-a462c2172ee0 [Shutdown]" term=3 testLoggerAdapter verbose: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.934Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: aborting pipeline replication: peer="{Voter server-6d4f2788-2b13-a8b9-e435-a0774f871ab8 6d4f2788-2b13-a8b9-e435-a0774f871ab8}" testing.go:149: server-e9bae396-860a-aac1-bfb4-a462c2172ee0: 2021-07-10T01:05:13.934Z [INFO] server-e9bae396-860a-aac1-bfb4-a462c2172ee0: entering candidate state: node="Node at e9bae396-860a-aac1-bfb4-a462c2172ee0 [Shutdown]" term=3 testing.go:149: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.934Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: aborting pipeline replication: peer="{Voter server-6d4f2788-2b13-a8b9-e435-a0774f871ab8 6d4f2788-2b13-a8b9-e435-a0774f871ab8}" testLoggerAdapter verbose: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.934Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: aborting pipeline replication: peer="{Voter server-e9bae396-860a-aac1-bfb4-a462c2172ee0 e9bae396-860a-aac1-bfb4-a462c2172ee0}" testing.go:149: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.934Z [INFO] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: aborting pipeline replication: peer="{Voter server-e9bae396-860a-aac1-bfb4-a462c2172ee0 e9bae396-860a-aac1-bfb4-a462c2172ee0}" testLoggerAdapter verbose: server-e9bae396-860a-aac1-bfb4-a462c2172ee0: 2021-07-10T01:05:13.986Z [ERROR] server-e9bae396-860a-aac1-bfb4-a462c2172ee0: failed to make requestVote RPC: target="{Voter server-6d4f2788-2b13-a8b9-e435-a0774f871ab8 6d4f2788-2b13-a8b9-e435-a0774f871ab8}" error="command timed out" testing.go:149: server-e9bae396-860a-aac1-bfb4-a462c2172ee0: 2021-07-10T01:05:13.986Z [ERROR] server-e9bae396-860a-aac1-bfb4-a462c2172ee0: failed to make requestVote RPC: target="{Voter server-6d4f2788-2b13-a8b9-e435-a0774f871ab8 6d4f2788-2b13-a8b9-e435-a0774f871ab8}" error="command timed out" testLoggerAdapter verbose: server-e9bae396-860a-aac1-bfb4-a462c2172ee0: 2021-07-10T01:05:13.986Z [ERROR] server-e9bae396-860a-aac1-bfb4-a462c2172ee0: failed to make requestVote RPC: target="{Voter server-57b5d042-8833-3178-5c09-e3f830dfd7b4 57b5d042-8833-3178-5c09-e3f830dfd7b4}" error="command timed out" testing.go:149: server-e9bae396-860a-aac1-bfb4-a462c2172ee0: 2021-07-10T01:05:13.986Z [ERROR] server-e9bae396-860a-aac1-bfb4-a462c2172ee0: failed to make requestVote RPC: target="{Voter server-57b5d042-8833-3178-5c09-e3f830dfd7b4 57b5d042-8833-3178-5c09-e3f830dfd7b4}" error="command timed out" testLoggerAdapter verbose: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.989Z [ERROR] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: failed to heartbeat to: peer=6d4f2788-2b13-a8b9-e435-a0774f871ab8 error="command timed out" testing.go:149: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.989Z [ERROR] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: failed to heartbeat to: peer=6d4f2788-2b13-a8b9-e435-a0774f871ab8 error="command timed out" testLoggerAdapter verbose: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.989Z [ERROR] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: failed to heartbeat to: peer=e9bae396-860a-aac1-bfb4-a462c2172ee0 error="command timed out" testing.go:149: server-57b5d042-8833-3178-5c09-e3f830dfd7b4: 2021-07-10T01:05:13.989Z [ERROR] server-57b5d042-8833-3178-5c09-e3f830dfd7b4: failed to heartbeat to: peer=e9bae396-860a-aac1-bfb4-a462c2172ee0 error="command timed out" --- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (0.72s) === RUN TestRaft_LeadershipTransferStopRightAway --- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s) === RUN TestRaft_GetConfigurationNoBootstrap testLoggerAdapter verbose: server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42: 2021-07-10T01:05:13.991Z [INFO] server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42 Address:6acaff73-6bcc-8805-4ba3-6fcb9e528f42} {Suffrage:Voter ID:server-7546905d-7105-f2b6-e4b1-acbde4cf5589 Address:7546905d-7105-f2b6-e4b1-acbde4cf5589}]" testing.go:149: server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42: 2021-07-10T01:05:13.991Z [INFO] server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42 Address:6acaff73-6bcc-8805-4ba3-6fcb9e528f42} {Suffrage:Voter ID:server-7546905d-7105-f2b6-e4b1-acbde4cf5589 Address:7546905d-7105-f2b6-e4b1-acbde4cf5589}]" testLoggerAdapter verbose: server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42: 2021-07-10T01:05:13.991Z [INFO] server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42: entering follower state: follower="Node at 6acaff73-6bcc-8805-4ba3-6fcb9e528f42 [Follower]" leader= testing.go:149: server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42: 2021-07-10T01:05:13.991Z [INFO] server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42: entering follower state: follower="Node at 6acaff73-6bcc-8805-4ba3-6fcb9e528f42 [Follower]" leader= testLoggerAdapter verbose: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:13.991Z [INFO] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42 Address:6acaff73-6bcc-8805-4ba3-6fcb9e528f42} {Suffrage:Voter ID:server-7546905d-7105-f2b6-e4b1-acbde4cf5589 Address:7546905d-7105-f2b6-e4b1-acbde4cf5589}]" testing.go:149: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:13.991Z [INFO] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42 Address:6acaff73-6bcc-8805-4ba3-6fcb9e528f42} {Suffrage:Voter ID:server-7546905d-7105-f2b6-e4b1-acbde4cf5589 Address:7546905d-7105-f2b6-e4b1-acbde4cf5589}]" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:13.991Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-07-10T01:05:13.991Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:13.991Z [INFO] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: entering follower state: follower="Node at 7546905d-7105-f2b6-e4b1-acbde4cf5589 [Follower]" leader= testing.go:149: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:13.991Z [INFO] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: entering follower state: follower="Node at 7546905d-7105-f2b6-e4b1-acbde4cf5589 [Follower]" leader= testLoggerAdapter verbose: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:14.067Z [WARN] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:14.067Z [WARN] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:14.067Z [INFO] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: entering candidate state: node="Node at 7546905d-7105-f2b6-e4b1-acbde4cf5589 [Candidate]" term=2 testing.go:149: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:14.067Z [INFO] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: entering candidate state: node="Node at 7546905d-7105-f2b6-e4b1-acbde4cf5589 [Candidate]" term=2 testLoggerAdapter verbose: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:14.067Z [INFO] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: election won: tally=2 testing.go:149: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:14.067Z [INFO] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: election won: tally=2 testLoggerAdapter verbose: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:14.067Z [INFO] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: entering leader state: leader="Node at 7546905d-7105-f2b6-e4b1-acbde4cf5589 [Leader]" testing.go:149: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:14.067Z [INFO] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: entering leader state: leader="Node at 7546905d-7105-f2b6-e4b1-acbde4cf5589 [Leader]" testLoggerAdapter verbose: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:14.067Z [INFO] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: added peer, starting replication: peer=server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42 testing.go:149: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:14.067Z [INFO] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: added peer, starting replication: peer=server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42 testLoggerAdapter verbose: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:14.068Z [INFO] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: pipelining replication: peer="{Voter server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42 6acaff73-6bcc-8805-4ba3-6fcb9e528f42}" testing.go:149: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:14.068Z [INFO] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: pipelining replication: peer="{Voter server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42 6acaff73-6bcc-8805-4ba3-6fcb9e528f42}" testLoggerAdapter verbose: cluster: 2021-07-10T01:05:14.173Z [INFO] stable state for Follower reached at 2021-07-10 01:05:14.067887503 +0000 UTC m=+19.456045573 (1 nodes), 76.211883ms from start of poll, 76.924631ms from cluster start. Timeout at 2021-07-10 01:05:14.172974631 +0000 UTC m=+19.561132621, 105.087048ms after stability testing.go:149: cluster: 2021-07-10T01:05:14.173Z [INFO] stable state for Follower reached at 2021-07-10 01:05:14.067887503 +0000 UTC m=+19.456045573 (1 nodes), 76.211883ms from start of poll, 76.924631ms from cluster start. Timeout at 2021-07-10 01:05:14.172974631 +0000 UTC m=+19.561132621, 105.087048ms after stability testLoggerAdapter verbose: cluster: 2021-07-10T01:05:14.173Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-07-10T01:05:14.173Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-07-10T01:05:14.278Z [INFO] stable state for Leader reached at 2021-07-10 01:05:14.173273334 +0000 UTC m=+19.561431304 (1 nodes), 2.28µs from start of poll, 182.310362ms from cluster start. Timeout at 2021-07-10 01:05:14.278350662 +0000 UTC m=+19.666508632, 105.077328ms after stability testing.go:149: cluster: 2021-07-10T01:05:14.278Z [INFO] stable state for Leader reached at 2021-07-10 01:05:14.173273334 +0000 UTC m=+19.561431304 (1 nodes), 2.28µs from start of poll, 182.310362ms from cluster start. Timeout at 2021-07-10 01:05:14.278350662 +0000 UTC m=+19.666508632, 105.077328ms after stability testLoggerAdapter verbose: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:14.289Z [INFO] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42 Address:6acaff73-6bcc-8805-4ba3-6fcb9e528f42} {Suffrage:Voter ID:server-7546905d-7105-f2b6-e4b1-acbde4cf5589 Address:7546905d-7105-f2b6-e4b1-acbde4cf5589}]" testing.go:149: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:14.289Z [INFO] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42 Address:6acaff73-6bcc-8805-4ba3-6fcb9e528f42} {Suffrage:Voter ID:server-7546905d-7105-f2b6-e4b1-acbde4cf5589 Address:7546905d-7105-f2b6-e4b1-acbde4cf5589}]" testLoggerAdapter verbose: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:14.289Z [INFO] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: aborting pipeline replication: peer="{Voter server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42 6acaff73-6bcc-8805-4ba3-6fcb9e528f42}" testing.go:149: server-7546905d-7105-f2b6-e4b1-acbde4cf5589: 2021-07-10T01:05:14.289Z [INFO] server-7546905d-7105-f2b6-e4b1-acbde4cf5589: aborting pipeline replication: peer="{Voter server-6acaff73-6bcc-8805-4ba3-6fcb9e528f42 6acaff73-6bcc-8805-4ba3-6fcb9e528f42}" --- PASS: TestRaft_GetConfigurationNoBootstrap (0.30s) === RUN TestTCPTransport_BadAddr --- PASS: TestTCPTransport_BadAddr (0.00s) === RUN TestTCPTransport_WithAdvertise --- PASS: TestTCPTransport_WithAdvertise (0.00s) === RUN TestTransport_StartStop --- PASS: TestTransport_StartStop (0.00s) === RUN TestTransport_AppendEntries --- PASS: TestTransport_AppendEntries (0.00s) === RUN TestTransport_AppendEntriesPipeline --- PASS: TestTransport_AppendEntriesPipeline (0.00s) === RUN TestTransport_RequestVote --- PASS: TestTransport_RequestVote (0.00s) === RUN TestTransport_InstallSnapshot --- PASS: TestTransport_InstallSnapshot (0.00s) === RUN TestTransport_EncodeDecode --- PASS: TestTransport_EncodeDecode (0.00s) === RUN TestRandomTimeout --- PASS: TestRandomTimeout (0.00s) === RUN TestNewSeed --- PASS: TestNewSeed (0.00s) === RUN TestRandomTimeout_NoTime --- PASS: TestRandomTimeout_NoTime (0.00s) === RUN TestMin --- PASS: TestMin (0.00s) === RUN TestMax --- PASS: TestMax (0.00s) === RUN TestGenerateUUID --- PASS: TestGenerateUUID (0.00s) === RUN TestBackoff --- PASS: TestBackoff (0.00s) PASS ok github.com/hashicorp/raft 19.697s ? github.com/hashicorp/raft/bench [no test files] create-stamp debian/debhelper-build-stamp fakeroot debian/rules binary dh binary --buildsystem=golang --with=golang dh_testroot -O--buildsystem=golang dh_prep -O--buildsystem=golang dh_auto_install -O--buildsystem=golang dh_installdocs -O--buildsystem=golang dh_installchangelogs -O--buildsystem=golang dh_installinit -O--buildsystem=golang dh_installsystemduser -O--buildsystem=golang dh_perl -O--buildsystem=golang dh_link -O--buildsystem=golang dh_strip_nondeterminism -O--buildsystem=golang dh_compress -O--buildsystem=golang dh_fixperms -O--buildsystem=golang dh_missing -O--buildsystem=golang dh_installdeb -O--buildsystem=golang dh_golang -O--buildsystem=golang dh_gencontrol -O--buildsystem=golang dpkg-gencontrol: warning: Depends field of package golang-github-hashicorp-raft-dev: substitution variable ${shlibs:Depends} used, but is not defined dh_md5sums -O--buildsystem=golang dh_builddeb -O--buildsystem=golang dpkg-deb: building package 'golang-github-hashicorp-raft-dev' in '../golang-github-hashicorp-raft-dev_1.1.2-1_all.deb'. dpkg-genbuildinfo --build=binary dpkg-genchanges --build=binary >../golang-github-hashicorp-raft_1.1.2-1_arm64.changes dpkg-genchanges: info: binary-only upload (no source code included) dpkg-source --after-build . dpkg-buildpackage: info: binary-only upload (no source included) dpkg-genchanges: info: including full source code in upload I: copying local configuration I: unmounting dev/ptmx filesystem I: unmounting dev/pts filesystem I: unmounting dev/shm filesystem I: unmounting proc filesystem I: unmounting sys filesystem I: cleaning the build env I: removing directory /srv/workspace/pbuilder/15724 and its subdirectories I: Current time: Fri Jul 9 13:05:22 -12 2021 I: pbuilder-time-stamp: 1625879122 Sat Jul 10 01:05:24 UTC 2021 I: 1st build successful. Starting 2nd build on remote node codethink11-arm64.debian.net. Sat Jul 10 01:05:24 UTC 2021 I: Preparing to do remote build '2' on codethink11-arm64.debian.net. Sat Jul 10 01:07:28 UTC 2021 I: Deleting $TMPDIR on codethink11-arm64.debian.net. Sat Jul 10 01:07:29 UTC 2021 I: golang-github-hashicorp-raft_1.1.2-1_arm64.changes: Format: 1.8 Date: Sun, 19 Jan 2020 18:15:42 +1100 Source: golang-github-hashicorp-raft Binary: golang-github-hashicorp-raft-dev Architecture: all Version: 1.1.2-1 Distribution: unstable Urgency: medium Maintainer: Debian Go Packaging Team Changed-By: Dmitry Smirnov Description: golang-github-hashicorp-raft-dev - implementation of the Raft consensus protocol Changes: golang-github-hashicorp-raft (1.1.2-1) unstable; urgency=medium . * New upstream release. Checksums-Sha1: fd2f48e7cbf9067cdcc7171212e616e619053c70 90896 golang-github-hashicorp-raft-dev_1.1.2-1_all.deb ad020564414a74165ee438c246126c95748cf71e 8653 golang-github-hashicorp-raft_1.1.2-1_arm64.buildinfo Checksums-Sha256: f34877bf2e6d361137752e477ed94bb92556fd43a6ed29fa6135cc010f7d1954 90896 golang-github-hashicorp-raft-dev_1.1.2-1_all.deb 78072b24cd0e2809336f21d1a57051ac554f6cddc3686ff27b7ac4a341bbfe84 8653 golang-github-hashicorp-raft_1.1.2-1_arm64.buildinfo Files: 9217d71a882e9745b40d8acfe32cc432 90896 devel optional golang-github-hashicorp-raft-dev_1.1.2-1_all.deb 0bbfd5f41df23886f6a6dc188ace033c 8653 devel optional golang-github-hashicorp-raft_1.1.2-1_arm64.buildinfo Sat Jul 10 01:07:30 UTC 2021 I: diffoscope 177 will be used to compare the two builds: # Profiling output for: /usr/bin/diffoscope --html /srv/reproducible-results/rbuild-debian/tmp.tE7qwSSjNo/golang-github-hashicorp-raft_1.1.2-1.diffoscope.html --text /srv/reproducible-results/rbuild-debian/tmp.tE7qwSSjNo/golang-github-hashicorp-raft_1.1.2-1.diffoscope.txt --json /srv/reproducible-results/rbuild-debian/tmp.tE7qwSSjNo/golang-github-hashicorp-raft_1.1.2-1.diffoscope.json --profile=- /srv/reproducible-results/rbuild-debian/tmp.tE7qwSSjNo/b1/golang-github-hashicorp-raft_1.1.2-1_arm64.changes /srv/reproducible-results/rbuild-debian/tmp.tE7qwSSjNo/b2/golang-github-hashicorp-raft_1.1.2-1_arm64.changes ## command (total time: 0.000s) 0.000s 1 call cmp (internal) ## has_same_content_as (total time: 0.000s) 0.000s 1 call abc.DotChangesFile ## main (total time: 0.266s) 0.266s 2 calls outputs 0.000s 1 call cleanup ## recognizes (total time: 0.018s) 0.018s 10 calls diffoscope.comparators.binary.FilesystemFile 0.000s 8 calls abc.DotChangesFile Sat Jul 10 01:07:32 UTC 2021 I: diffoscope 177 found no differences in the changes files, and a .buildinfo file also exists. Sat Jul 10 01:07:32 UTC 2021 I: golang-github-hashicorp-raft from bullseye built successfully and reproducibly on arm64. Sat Jul 10 01:07:33 UTC 2021 I: Submitting .buildinfo files to external archives: Sat Jul 10 01:07:33 UTC 2021 I: Submitting 12K b1/golang-github-hashicorp-raft_1.1.2-1_arm64.buildinfo.asc Sat Jul 10 01:07:35 UTC 2021 I: Submitting 12K b2/golang-github-hashicorp-raft_1.1.2-1_arm64.buildinfo.asc Sat Jul 10 01:07:36 UTC 2021 I: Done submitting .buildinfo files to http://buildinfo.debian.net/api/submit. Sat Jul 10 01:07:36 UTC 2021 I: Done submitting .buildinfo files. Sat Jul 10 01:07:36 UTC 2021 I: Removing signed golang-github-hashicorp-raft_1.1.2-1_arm64.buildinfo.asc files: removed './b1/golang-github-hashicorp-raft_1.1.2-1_arm64.buildinfo.asc' removed './b2/golang-github-hashicorp-raft_1.1.2-1_arm64.buildinfo.asc'