I: pbuilder: network access will be disabled during build I: Current time: Wed Jun 12 04:30:07 +14 2024 I: pbuilder-time-stamp: 1718116207 I: Building the build Environment I: extracting base tarball [/var/cache/pbuilder/bookworm-reproducible-base.tgz] I: copying local configuration W: --override-config is not set; not updating apt.conf Read the manpage for details. I: mounting /proc filesystem I: mounting /sys filesystem I: creating /{dev,run}/shm I: mounting /dev/pts filesystem I: redirecting /dev/ptmx to /dev/pts/ptmx I: policy-rc.d already exists I: Copying source file I: copying [golang-github-hashicorp-raft_1.1.2-1.dsc] I: copying [./golang-github-hashicorp-raft_1.1.2.orig.tar.xz] I: copying [./golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz] I: Extracting source gpgv: Signature made Sun Jan 19 21:24:28 2020 +14 gpgv: using RSA key 50BC7CF939D20C272A6B065652B6BBD953968D1B gpgv: Can't check signature: No public key dpkg-source: warning: cannot verify inline signature for ./golang-github-hashicorp-raft_1.1.2-1.dsc: no acceptable signature found dpkg-source: info: extracting golang-github-hashicorp-raft in golang-github-hashicorp-raft-1.1.2 dpkg-source: info: unpacking golang-github-hashicorp-raft_1.1.2.orig.tar.xz dpkg-source: info: unpacking golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz dpkg-source: info: using patch list from debian/patches/series dpkg-source: info: applying t-skip--TestNetworkTransport_AppendEntriesPipeline_CloseStreams.patch dpkg-source: info: applying t-skip--TestRaft_AutoSnapshot.patch dpkg-source: info: applying t-skip--TestRaft_LeadershipTransferLeaderRejectsClientRequests.patch dpkg-source: info: applying t-skip--TestRaft_RecoverCluster.patch dpkg-source: info: applying t-skip--TestRaft_SnapshotRestore_PeerChange.patch I: using fakeroot in build. I: Installing the build-deps I: user script /srv/workspace/pbuilder/18637/tmp/hooks/D01_modify_environment starting debug: Running on virt64z. I: Changing host+domainname to test build reproducibility I: Adding a custom variable just for the fun of it... I: Changing /bin/sh to bash '/bin/sh' -> '/bin/bash' lrwxrwxrwx 1 root root 9 Jun 12 04:30 /bin/sh -> /bin/bash I: Setting pbuilder2's login shell to /bin/bash I: Setting pbuilder2's GECOS to second user,second room,second work-phone,second home-phone,second other I: user script /srv/workspace/pbuilder/18637/tmp/hooks/D01_modify_environment finished I: user script /srv/workspace/pbuilder/18637/tmp/hooks/D02_print_environment starting I: set BASH=/bin/sh BASHOPTS=checkwinsize:cmdhist:complete_fullquote:extquote:force_fignore:globasciiranges:globskipdots:hostcomplete:interactive_comments:patsub_replacement:progcomp:promptvars:sourcepath BASH_ALIASES=() BASH_ARGC=() BASH_ARGV=() BASH_CMDS=() BASH_LINENO=([0]="12" [1]="0") BASH_LOADABLES_PATH=/usr/local/lib/bash:/usr/lib/bash:/opt/local/lib/bash:/usr/pkg/lib/bash:/opt/pkg/lib/bash:. BASH_SOURCE=([0]="/tmp/hooks/D02_print_environment" [1]="/tmp/hooks/D02_print_environment") BASH_VERSINFO=([0]="5" [1]="2" [2]="15" [3]="1" [4]="release" [5]="arm-unknown-linux-gnueabihf") BASH_VERSION='5.2.15(1)-release' BUILDDIR=/build/reproducible-path BUILDUSERGECOS='second user,second room,second work-phone,second home-phone,second other' BUILDUSERNAME=pbuilder2 BUILD_ARCH=armhf DEBIAN_FRONTEND=noninteractive DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all parallel=4 ' DIRSTACK=() DISTRIBUTION=bookworm EUID=0 FUNCNAME=([0]="Echo" [1]="main") GROUPS=() HOME=/root HOSTNAME=i-capture-the-hostname HOSTTYPE=arm HOST_ARCH=armhf IFS=' ' INVOCATION_ID=13f7f702a825411da853c99fbca43e55 LANG=C LANGUAGE=it_CH:it LC_ALL=C MACHTYPE=arm-unknown-linux-gnueabihf MAIL=/var/mail/root OPTERR=1 OPTIND=1 OSTYPE=linux-gnueabihf PATH=/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path PBCURRENTCOMMANDLINEOPERATION=build PBUILDER_OPERATION=build PBUILDER_PKGDATADIR=/usr/share/pbuilder PBUILDER_PKGLIBDIR=/usr/lib/pbuilder PBUILDER_SYSCONFDIR=/etc PIPESTATUS=([0]="0") POSIXLY_CORRECT=y PPID=18637 PS4='+ ' PWD=/ SHELL=/bin/bash SHELLOPTS=braceexpand:errexit:hashall:interactive-comments:posix SHLVL=3 SUDO_COMMAND='/usr/bin/timeout -k 24.1h 24h /usr/bin/ionice -c 3 /usr/bin/nice -n 11 /usr/bin/unshare --uts -- /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.avc5eAbN/pbuilderrc_0Ywj --distribution bookworm --hookdir /etc/pbuilder/rebuild-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/bookworm-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.avc5eAbN/b2 --logfile b2/build.log golang-github-hashicorp-raft_1.1.2-1.dsc' SUDO_GID=110 SUDO_UID=103 SUDO_USER=jenkins TERM=unknown TZ=/usr/share/zoneinfo/Etc/GMT-14 UID=0 USER=root _='I: set' http_proxy=http://10.0.0.15:3142/ I: uname -a Linux i-capture-the-hostname 6.1.0-21-arm64 #1 SMP Debian 6.1.90-1 (2024-05-03) aarch64 GNU/Linux I: ls -l /bin total 4964 -rwxr-xr-x 1 root root 838488 Apr 24 2023 bash -rwxr-xr-x 3 root root 67144 Sep 19 2022 bunzip2 -rwxr-xr-x 3 root root 67144 Sep 19 2022 bzcat lrwxrwxrwx 1 root root 6 Sep 19 2022 bzcmp -> bzdiff -rwxr-xr-x 1 root root 2225 Sep 19 2022 bzdiff lrwxrwxrwx 1 root root 6 Sep 19 2022 bzegrep -> bzgrep -rwxr-xr-x 1 root root 4893 Nov 28 2021 bzexe lrwxrwxrwx 1 root root 6 Sep 19 2022 bzfgrep -> bzgrep -rwxr-xr-x 1 root root 3775 Sep 19 2022 bzgrep -rwxr-xr-x 3 root root 67144 Sep 19 2022 bzip2 -rwxr-xr-x 1 root root 67112 Sep 19 2022 bzip2recover lrwxrwxrwx 1 root root 6 Sep 19 2022 bzless -> bzmore -rwxr-xr-x 1 root root 1297 Sep 19 2022 bzmore -rwxr-xr-x 1 root root 67632 Sep 21 2022 cat -rwxr-xr-x 1 root root 67676 Sep 21 2022 chgrp -rwxr-xr-x 1 root root 67644 Sep 21 2022 chmod -rwxr-xr-x 1 root root 67684 Sep 21 2022 chown -rwxr-xr-x 1 root root 133532 Sep 21 2022 cp -rwxr-xr-x 1 root root 132868 Jan 6 2023 dash -rwxr-xr-x 1 root root 133220 Sep 21 2022 date -rwxr-xr-x 1 root root 67732 Sep 21 2022 dd -rwxr-xr-x 1 root root 68104 Sep 21 2022 df -rwxr-xr-x 1 root root 133632 Sep 21 2022 dir -rwxr-xr-x 1 root root 59128 Mar 23 2023 dmesg lrwxrwxrwx 1 root root 8 Dec 20 2022 dnsdomainname -> hostname lrwxrwxrwx 1 root root 8 Dec 20 2022 domainname -> hostname -rwxr-xr-x 1 root root 67560 Sep 21 2022 echo -rwxr-xr-x 1 root root 41 Jan 25 2023 egrep -rwxr-xr-x 1 root root 67548 Sep 21 2022 false -rwxr-xr-x 1 root root 41 Jan 25 2023 fgrep -rwxr-xr-x 1 root root 55748 Mar 23 2023 findmnt -rwsr-xr-x 1 root root 26208 Mar 23 2023 fusermount -rwxr-xr-x 1 root root 128608 Jan 25 2023 grep -rwxr-xr-x 2 root root 2346 Apr 10 2022 gunzip -rwxr-xr-x 1 root root 6447 Apr 10 2022 gzexe -rwxr-xr-x 1 root root 64220 Apr 10 2022 gzip -rwxr-xr-x 1 root root 67032 Dec 20 2022 hostname -rwxr-xr-x 1 root root 67720 Sep 21 2022 ln -rwxr-xr-x 1 root root 35132 Mar 23 2023 login -rwxr-xr-x 1 root root 133632 Sep 21 2022 ls -rwxr-xr-x 1 root root 136808 Mar 23 2023 lsblk -rwxr-xr-x 1 root root 67800 Sep 21 2022 mkdir -rwxr-xr-x 1 root root 67764 Sep 21 2022 mknod -rwxr-xr-x 1 root root 67596 Sep 21 2022 mktemp -rwxr-xr-x 1 root root 38504 Mar 23 2023 more -rwsr-xr-x 1 root root 38496 Mar 23 2023 mount -rwxr-xr-x 1 root root 9824 Mar 23 2023 mountpoint -rwxr-xr-x 1 root root 133532 Sep 21 2022 mv lrwxrwxrwx 1 root root 8 Dec 20 2022 nisdomainname -> hostname lrwxrwxrwx 1 root root 14 Apr 3 2023 pidof -> /sbin/killall5 -rwxr-xr-x 1 root root 67608 Sep 21 2022 pwd lrwxrwxrwx 1 root root 4 Apr 24 2023 rbash -> bash -rwxr-xr-x 1 root root 67600 Sep 21 2022 readlink -rwxr-xr-x 1 root root 67672 Sep 21 2022 rm -rwxr-xr-x 1 root root 67600 Sep 21 2022 rmdir -rwxr-xr-x 1 root root 14152 Jul 29 2023 run-parts -rwxr-xr-x 1 root root 133372 Jan 6 2023 sed lrwxrwxrwx 1 root root 9 Jun 12 04:30 sh -> /bin/bash -rwxr-xr-x 1 root root 67584 Sep 21 2022 sleep -rwxr-xr-x 1 root root 67644 Sep 21 2022 stty -rwsr-xr-x 1 root root 50800 Mar 23 2023 su -rwxr-xr-x 1 root root 67584 Sep 21 2022 sync -rwxr-xr-x 1 root root 336764 Jan 20 23:27 tar -rwxr-xr-x 1 root root 9800 Jul 29 2023 tempfile -rwxr-xr-x 1 root root 133224 Sep 21 2022 touch -rwxr-xr-x 1 root root 67548 Sep 21 2022 true -rwxr-xr-x 1 root root 9768 Mar 23 2023 ulockmgr_server -rwsr-xr-x 1 root root 22108 Mar 23 2023 umount -rwxr-xr-x 1 root root 67572 Sep 21 2022 uname -rwxr-xr-x 2 root root 2346 Apr 10 2022 uncompress -rwxr-xr-x 1 root root 133632 Sep 21 2022 vdir -rwxr-xr-x 1 root root 42608 Mar 23 2023 wdctl lrwxrwxrwx 1 root root 8 Dec 20 2022 ypdomainname -> hostname -rwxr-xr-x 1 root root 1984 Apr 10 2022 zcat -rwxr-xr-x 1 root root 1678 Apr 10 2022 zcmp -rwxr-xr-x 1 root root 6460 Apr 10 2022 zdiff -rwxr-xr-x 1 root root 29 Apr 10 2022 zegrep -rwxr-xr-x 1 root root 29 Apr 10 2022 zfgrep -rwxr-xr-x 1 root root 2081 Apr 10 2022 zforce -rwxr-xr-x 1 root root 8103 Apr 10 2022 zgrep -rwxr-xr-x 1 root root 2206 Apr 10 2022 zless -rwxr-xr-x 1 root root 1842 Apr 10 2022 zmore -rwxr-xr-x 1 root root 4577 Apr 10 2022 znew I: user script /srv/workspace/pbuilder/18637/tmp/hooks/D02_print_environment finished -> Attempting to satisfy build-dependencies -> Creating pbuilder-satisfydepends-dummy package Package: pbuilder-satisfydepends-dummy Version: 0.invalid.0 Architecture: armhf Maintainer: Debian Pbuilder Team Description: Dummy package to satisfy dependencies with aptitude - created by pbuilder This package was created automatically by pbuilder to satisfy the build-dependencies of the package being currently built. Depends: debhelper (>= 12~), dh-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~) dpkg-deb: building package 'pbuilder-satisfydepends-dummy' in '/tmp/satisfydepends-aptitude/pbuilder-satisfydepends-dummy.deb'. Selecting previously unselected package pbuilder-satisfydepends-dummy. (Reading database ... 19288 files and directories currently installed.) Preparing to unpack .../pbuilder-satisfydepends-dummy.deb ... Unpacking pbuilder-satisfydepends-dummy (0.invalid.0) ... dpkg: pbuilder-satisfydepends-dummy: dependency problems, but configuring anyway as you requested: pbuilder-satisfydepends-dummy depends on debhelper (>= 12~); however: Package debhelper is not installed. pbuilder-satisfydepends-dummy depends on dh-golang; however: Package dh-golang is not installed. pbuilder-satisfydepends-dummy depends on golang-any; however: Package golang-any is not installed. pbuilder-satisfydepends-dummy depends on golang-github-armon-go-metrics-dev; however: Package golang-github-armon-go-metrics-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-hclog-dev (>= 0.9.2~); however: Package golang-github-hashicorp-go-hclog-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~); however: Package golang-github-hashicorp-go-msgpack-dev is not installed. Setting up pbuilder-satisfydepends-dummy (0.invalid.0) ... Reading package lists... Building dependency tree... Reading state information... Initializing package states... Writing extended state information... Building tag database... pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0) pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0) The following NEW packages will be installed: autoconf{a} automake{a} autopoint{a} autotools-dev{a} bsdextrautils{a} debhelper{a} dh-autoreconf{a} dh-golang{a} dh-strip-nondeterminism{a} dwz{a} file{a} gettext{a} gettext-base{a} golang-1.19-go{a} golang-1.19-src{a} golang-any{a} golang-github-alecthomas-units-dev{a} golang-github-armon-go-metrics-dev{a} golang-github-aws-aws-sdk-go-dev{a} golang-github-beorn7-perks-dev{a} golang-github-cespare-xxhash-dev{a} golang-github-circonus-labs-circonus-gometrics-dev{a} golang-github-circonus-labs-circonusllhist-dev{a} golang-github-creack-pty-dev{a} golang-github-datadog-datadog-go-dev{a} golang-github-davecgh-go-spew-dev{a} golang-github-fatih-color-dev{a} golang-github-go-kit-log-dev{a} golang-github-go-logfmt-logfmt-dev{a} golang-github-golang-protobuf-1-3-dev{a} golang-github-hashicorp-go-cleanhttp-dev{a} golang-github-hashicorp-go-hclog-dev{a} golang-github-hashicorp-go-immutable-radix-dev{a} golang-github-hashicorp-go-msgpack-dev{a} golang-github-hashicorp-go-retryablehttp-dev{a} golang-github-hashicorp-go-uuid-dev{a} golang-github-hashicorp-golang-lru-dev{a} golang-github-jmespath-go-jmespath-dev{a} golang-github-jpillora-backoff-dev{a} golang-github-json-iterator-go-dev{a} golang-github-julienschmidt-httprouter-dev{a} golang-github-kr-pretty-dev{a} golang-github-kr-text-dev{a} golang-github-mattn-go-colorable-dev{a} golang-github-mattn-go-isatty-dev{a} golang-github-modern-go-concurrent-dev{a} golang-github-modern-go-reflect2-dev{a} golang-github-mwitkow-go-conntrack-dev{a} golang-github-pascaldekloe-goe-dev{a} golang-github-pkg-diff-dev{a} golang-github-pkg-errors-dev{a} golang-github-pmezard-go-difflib-dev{a} golang-github-prometheus-client-golang-dev{a} golang-github-prometheus-client-model-dev{a} golang-github-prometheus-common-dev{a} golang-github-prometheus-procfs-dev{a} golang-github-rogpeppe-go-internal-dev{a} golang-github-stretchr-objx-dev{a} golang-github-stretchr-testify-dev{a} golang-github-tv42-httpunix-dev{a} golang-github-vmihailenco-tagparser-dev{a} golang-go{a} golang-golang-x-net-dev{a} golang-golang-x-oauth2-dev{a} golang-golang-x-sync-dev{a} golang-golang-x-sys-dev{a} golang-golang-x-text-dev{a} golang-google-protobuf-dev{a} golang-gopkg-alecthomas-kingpin.v2-dev{a} golang-gopkg-check.v1-dev{a} golang-gopkg-mgo.v2-dev{a} golang-gopkg-tomb.v2-dev{a} golang-gopkg-vmihailenco-msgpack.v2-dev{a} golang-gopkg-yaml.v2-dev{a} golang-gopkg-yaml.v3-dev{a} golang-protobuf-extensions-dev{a} golang-src{a} groff-base{a} intltool-debian{a} libarchive-zip-perl{a} libdebhelper-perl{a} libelf1{a} libfile-stripnondeterminism-perl{a} libicu72{a} libmagic-mgc{a} libmagic1{a} libpipeline1{a} libsasl2-2{a} libsasl2-dev{a} libsasl2-modules-db{a} libsub-override-perl{a} libtool{a} libuchardet0{a} libxml2{a} m4{a} man-db{a} po-debconf{a} sensible-utils{a} The following packages are RECOMMENDED but will NOT be installed: curl libarchive-cpio-perl libltdl-dev libmail-sendmail-perl libsasl2-modules lynx pkg-config pkgconf wget 0 packages upgraded, 98 newly installed, 0 to remove and 0 not upgraded. Need to get 112 MB of archives. After unpacking 751 MB will be used. Writing extended state information... Get: 1 http://deb.debian.org/debian bookworm/main armhf sensible-utils all 0.0.17+nmu1 [19.0 kB] Get: 2 http://deb.debian.org/debian bookworm/main armhf libmagic-mgc armhf 1:5.44-3 [305 kB] Get: 3 http://deb.debian.org/debian bookworm/main armhf libmagic1 armhf 1:5.44-3 [96.5 kB] Get: 4 http://deb.debian.org/debian bookworm/main armhf file armhf 1:5.44-3 [41.6 kB] Get: 5 http://deb.debian.org/debian bookworm/main armhf gettext-base armhf 0.21-12 [157 kB] Get: 6 http://deb.debian.org/debian bookworm/main armhf libuchardet0 armhf 0.0.7-1 [65.0 kB] Get: 7 http://deb.debian.org/debian bookworm/main armhf groff-base armhf 1.22.4-10 [825 kB] Get: 8 http://deb.debian.org/debian bookworm/main armhf bsdextrautils armhf 2.38.1-5+b1 [78.6 kB] Get: 9 http://deb.debian.org/debian bookworm/main armhf libpipeline1 armhf 1.5.7-1 [33.6 kB] Get: 10 http://deb.debian.org/debian bookworm/main armhf man-db armhf 2.11.2-2 [1351 kB] Get: 11 http://deb.debian.org/debian bookworm/main armhf m4 armhf 1.4.19-3 [265 kB] Get: 12 http://deb.debian.org/debian bookworm/main armhf autoconf all 2.71-3 [332 kB] Get: 13 http://deb.debian.org/debian bookworm/main armhf autotools-dev all 20220109.1 [51.6 kB] Get: 14 http://deb.debian.org/debian bookworm/main armhf automake all 1:1.16.5-1.3 [823 kB] Get: 15 http://deb.debian.org/debian bookworm/main armhf autopoint all 0.21-12 [495 kB] Get: 16 http://deb.debian.org/debian bookworm/main armhf libdebhelper-perl all 13.11.4 [81.2 kB] Get: 17 http://deb.debian.org/debian bookworm/main armhf libtool all 2.4.7-5 [517 kB] Get: 18 http://deb.debian.org/debian bookworm/main armhf dh-autoreconf all 20 [17.1 kB] Get: 19 http://deb.debian.org/debian bookworm/main armhf libarchive-zip-perl all 1.68-1 [104 kB] Get: 20 http://deb.debian.org/debian bookworm/main armhf libsub-override-perl all 0.09-4 [9304 B] Get: 21 http://deb.debian.org/debian bookworm/main armhf libfile-stripnondeterminism-perl all 1.13.1-1 [19.4 kB] Get: 22 http://deb.debian.org/debian bookworm/main armhf dh-strip-nondeterminism all 1.13.1-1 [8620 B] Get: 23 http://deb.debian.org/debian bookworm/main armhf libelf1 armhf 0.188-2.1 [170 kB] Get: 24 http://deb.debian.org/debian bookworm/main armhf dwz armhf 0.15-1 [101 kB] Get: 25 http://deb.debian.org/debian bookworm/main armhf libicu72 armhf 72.1-3 [9048 kB] Get: 26 http://deb.debian.org/debian bookworm/main armhf libxml2 armhf 2.9.14+dfsg-1.3~deb12u1 [591 kB] Get: 27 http://deb.debian.org/debian bookworm/main armhf gettext armhf 0.21-12 [1229 kB] Get: 28 http://deb.debian.org/debian bookworm/main armhf intltool-debian all 0.35.0+20060710.6 [22.9 kB] Get: 29 http://deb.debian.org/debian bookworm/main armhf po-debconf all 1.0.21+nmu1 [248 kB] Get: 30 http://deb.debian.org/debian bookworm/main armhf debhelper all 13.11.4 [942 kB] Get: 31 http://deb.debian.org/debian bookworm/main armhf dh-golang all 1.59 [27.8 kB] Get: 32 http://deb.debian.org/debian bookworm/main armhf golang-1.19-src all 1.19.8-2 [18.3 MB] Get: 33 http://deb.debian.org/debian bookworm/main armhf golang-1.19-go armhf 1.19.8-2 [56.9 MB] Get: 34 http://deb.debian.org/debian bookworm/main armhf golang-src all 2:1.19~1 [5756 B] Get: 35 http://deb.debian.org/debian bookworm/main armhf golang-go armhf 2:1.19~1 [45.0 kB] Get: 36 http://deb.debian.org/debian bookworm/main armhf golang-any armhf 2:1.19~1 [5876 B] Get: 37 http://deb.debian.org/debian bookworm/main armhf golang-github-alecthomas-units-dev all 0.0~git20211218.b94a6e3-1 [8776 B] Get: 38 http://deb.debian.org/debian bookworm/main armhf golang-github-circonus-labs-circonusllhist-dev all 0.0~git20191022.ec08cde-1 [11.1 kB] Get: 39 http://deb.debian.org/debian bookworm/main armhf golang-github-hashicorp-go-cleanhttp-dev all 0.5.2-1 [10.8 kB] Get: 40 http://deb.debian.org/debian bookworm/main armhf golang-golang-x-sys-dev all 0.3.0-1 [393 kB] Get: 41 http://deb.debian.org/debian bookworm/main armhf golang-github-mattn-go-isatty-dev all 0.0.17-1 [6016 B] Get: 42 http://deb.debian.org/debian bookworm/main armhf golang-github-mattn-go-colorable-dev all 0.1.13-1 [10.3 kB] Get: 43 http://deb.debian.org/debian bookworm/main armhf golang-github-fatih-color-dev all 1.7.0-1 [11.4 kB] Get: 44 http://deb.debian.org/debian bookworm/main armhf golang-github-davecgh-go-spew-dev all 1.1.1-3 [30.1 kB] Get: 45 http://deb.debian.org/debian bookworm/main armhf golang-github-pmezard-go-difflib-dev all 1.0.0-3 [12.3 kB] Get: 46 http://deb.debian.org/debian bookworm/main armhf golang-github-stretchr-objx-dev all 0.5.0-1 [25.5 kB] Get: 47 http://deb.debian.org/debian bookworm/main armhf golang-gopkg-yaml.v3-dev all 3.0.1-3 [74.3 kB] Get: 48 http://deb.debian.org/debian bookworm/main armhf golang-github-stretchr-testify-dev all 1.8.1-1 [67.9 kB] Get: 49 http://deb.debian.org/debian bookworm/main armhf golang-github-hashicorp-go-hclog-dev all 0.11.0-1 [17.9 kB] Get: 50 http://deb.debian.org/debian bookworm/main armhf golang-github-hashicorp-go-retryablehttp-dev all 0.7.1-1 [21.3 kB] Get: 51 http://deb.debian.org/debian bookworm/main armhf golang-github-pkg-errors-dev all 0.9.1-2 [13.2 kB] Get: 52 http://deb.debian.org/debian bookworm/main armhf golang-github-tv42-httpunix-dev all 0.0~git20150427.b75d861-3 [3956 B] Get: 53 http://deb.debian.org/debian bookworm/main armhf golang-github-circonus-labs-circonus-gometrics-dev all 2.3.1-4 [65.7 kB] Get: 54 http://deb.debian.org/debian bookworm/main armhf golang-github-datadog-datadog-go-dev all 2.1.0-3 [14.9 kB] Get: 55 http://deb.debian.org/debian bookworm/main armhf golang-github-hashicorp-go-uuid-dev all 1.0.2-2 [8480 B] Get: 56 http://deb.debian.org/debian bookworm/main armhf golang-github-hashicorp-golang-lru-dev all 0.5.4-2 [14.7 kB] Get: 57 http://deb.debian.org/debian bookworm/main armhf golang-github-hashicorp-go-immutable-radix-dev all 1.3.1-3 [26.2 kB] Get: 58 http://deb.debian.org/debian bookworm/main armhf golang-github-pascaldekloe-goe-dev all 0.1.0-4 [10.9 kB] Get: 59 http://deb.debian.org/debian bookworm/main armhf golang-github-beorn7-perks-dev all 1.0.1-1 [12.1 kB] Get: 60 http://deb.debian.org/debian bookworm/main armhf golang-github-cespare-xxhash-dev all 2.1.1-2 [8888 B] Get: 61 http://deb.debian.org/debian bookworm/main armhf golang-github-golang-protobuf-1-3-dev all 1.3.5-4 [234 kB] Get: 62 http://deb.debian.org/debian bookworm/main armhf golang-github-modern-go-concurrent-dev all 1.0.3-1.1 [4668 B] Get: 63 http://deb.debian.org/debian bookworm/main armhf golang-github-modern-go-reflect2-dev all 1.0.2-2 [10.9 kB] Get: 64 http://deb.debian.org/debian bookworm/main armhf golang-github-json-iterator-go-dev all 1.1.12-1 [62.2 kB] Get: 65 http://deb.debian.org/debian bookworm/main armhf golang-github-prometheus-client-model-dev all 0.3.0-3 [10.5 kB] Get: 66 http://deb.debian.org/debian bookworm/main armhf golang-gopkg-yaml.v2-dev all 2.4.0-4 [61.5 kB] Get: 67 http://deb.debian.org/debian bookworm/main armhf golang-github-jmespath-go-jmespath-dev all 0.4.0-2 [91.6 kB] Get: 68 http://deb.debian.org/debian bookworm/main armhf golang-golang-x-text-dev all 0.7.0-1 [3863 kB] Get: 69 http://deb.debian.org/debian bookworm/main armhf golang-golang-x-net-dev all 1:0.7.0+dfsg-1 [731 kB] Get: 70 http://deb.debian.org/debian bookworm/main armhf golang-github-aws-aws-sdk-go-dev all 1.44.133-1 [10.5 MB] Get: 71 http://deb.debian.org/debian bookworm/main armhf golang-github-go-logfmt-logfmt-dev all 0.5.0-2 [13.4 kB] Get: 72 http://deb.debian.org/debian bookworm/main armhf golang-github-go-kit-log-dev all 0.2.1-1 [26.9 kB] Get: 73 http://deb.debian.org/debian bookworm/main armhf golang-github-julienschmidt-httprouter-dev all 1.3.0-2 [19.3 kB] Get: 74 http://deb.debian.org/debian bookworm/main armhf golang-github-jpillora-backoff-dev all 1.0.0-1.1 [3712 B] Get: 75 http://deb.debian.org/debian bookworm/main armhf golang-github-mwitkow-go-conntrack-dev all 0.0~git20190716.2f06839-3 [14.6 kB] Get: 76 http://deb.debian.org/debian bookworm/main armhf golang-golang-x-oauth2-dev all 0.3.0-1 [35.0 kB] Get: 77 http://deb.debian.org/debian bookworm/main armhf golang-google-protobuf-dev all 1.28.1-3 [542 kB] Get: 78 http://deb.debian.org/debian bookworm/main armhf golang-gopkg-alecthomas-kingpin.v2-dev all 2.2.6-4 [42.8 kB] Get: 79 http://deb.debian.org/debian bookworm/main armhf golang-protobuf-extensions-dev all 1.0.1-3 [29.5 kB] Get: 80 http://deb.debian.org/debian bookworm/main armhf golang-github-prometheus-common-dev all 0.39.0-2 [110 kB] Get: 81 http://deb.debian.org/debian bookworm/main armhf golang-golang-x-sync-dev all 0.1.0-1 [19.7 kB] Get: 82 http://deb.debian.org/debian bookworm/main armhf golang-github-prometheus-procfs-dev all 0.8.0-3 [148 kB] Get: 83 http://deb.debian.org/debian bookworm/main armhf golang-github-prometheus-client-golang-dev all 1.14.0-3 [174 kB] Get: 84 http://deb.debian.org/debian bookworm/main armhf golang-github-armon-go-metrics-dev all 0.3.4-1 [30.5 kB] Get: 85 http://deb.debian.org/debian bookworm/main armhf golang-github-creack-pty-dev all 1.1.18-1 [9796 B] Get: 86 http://deb.debian.org/debian bookworm/main armhf golang-github-vmihailenco-tagparser-dev all 0.1.1-4 [4684 B] Get: 87 http://deb.debian.org/debian bookworm/main armhf golang-gopkg-vmihailenco-msgpack.v2-dev all 4.3.1-2 [27.5 kB] Get: 88 http://deb.debian.org/debian bookworm/main armhf golang-gopkg-tomb.v2-dev all 0.0~git20161208.d5d1b58-3 [6840 B] Get: 89 http://deb.debian.org/debian bookworm/main armhf libsasl2-modules-db armhf 2.1.28+dfsg-10 [19.0 kB] Get: 90 http://deb.debian.org/debian bookworm/main armhf libsasl2-2 armhf 2.1.28+dfsg-10 [52.3 kB] Get: 91 http://deb.debian.org/debian bookworm/main armhf libsasl2-dev armhf 2.1.28+dfsg-10 [219 kB] Get: 92 http://deb.debian.org/debian bookworm/main armhf golang-github-kr-text-dev all 0.2.0-2 [10.2 kB] Get: 93 http://deb.debian.org/debian bookworm/main armhf golang-github-pkg-diff-dev all 0.0~git20210226.20ebb0f-1 [93.6 kB] Get: 94 http://deb.debian.org/debian bookworm/main armhf golang-github-rogpeppe-go-internal-dev all 1.9.0-1 [117 kB] Get: 95 http://deb.debian.org/debian bookworm/main armhf golang-github-kr-pretty-dev all 0.3.0-1 [14.9 kB] Get: 96 http://deb.debian.org/debian bookworm/main armhf golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-1 [32.1 kB] Get: 97 http://deb.debian.org/debian bookworm/main armhf golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB] Get: 98 http://deb.debian.org/debian bookworm/main armhf golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB] Fetched 112 MB in 5s (23.0 MB/s) debconf: delaying package configuration, since apt-utils is not installed Selecting previously unselected package sensible-utils. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 19288 files and directories currently installed.) Preparing to unpack .../00-sensible-utils_0.0.17+nmu1_all.deb ... Unpacking sensible-utils (0.0.17+nmu1) ... Selecting previously unselected package libmagic-mgc. Preparing to unpack .../01-libmagic-mgc_1%3a5.44-3_armhf.deb ... Unpacking libmagic-mgc (1:5.44-3) ... Selecting previously unselected package libmagic1:armhf. Preparing to unpack .../02-libmagic1_1%3a5.44-3_armhf.deb ... Unpacking libmagic1:armhf (1:5.44-3) ... Selecting previously unselected package file. Preparing to unpack .../03-file_1%3a5.44-3_armhf.deb ... Unpacking file (1:5.44-3) ... Selecting previously unselected package gettext-base. Preparing to unpack .../04-gettext-base_0.21-12_armhf.deb ... Unpacking gettext-base (0.21-12) ... Selecting previously unselected package libuchardet0:armhf. Preparing to unpack .../05-libuchardet0_0.0.7-1_armhf.deb ... Unpacking libuchardet0:armhf (0.0.7-1) ... Selecting previously unselected package groff-base. Preparing to unpack .../06-groff-base_1.22.4-10_armhf.deb ... Unpacking groff-base (1.22.4-10) ... Selecting previously unselected package bsdextrautils. Preparing to unpack .../07-bsdextrautils_2.38.1-5+b1_armhf.deb ... Unpacking bsdextrautils (2.38.1-5+b1) ... Selecting previously unselected package libpipeline1:armhf. Preparing to unpack .../08-libpipeline1_1.5.7-1_armhf.deb ... Unpacking libpipeline1:armhf (1.5.7-1) ... Selecting previously unselected package man-db. Preparing to unpack .../09-man-db_2.11.2-2_armhf.deb ... Unpacking man-db (2.11.2-2) ... Selecting previously unselected package m4. Preparing to unpack .../10-m4_1.4.19-3_armhf.deb ... Unpacking m4 (1.4.19-3) ... Selecting previously unselected package autoconf. Preparing to unpack .../11-autoconf_2.71-3_all.deb ... Unpacking autoconf (2.71-3) ... Selecting previously unselected package autotools-dev. Preparing to unpack .../12-autotools-dev_20220109.1_all.deb ... Unpacking autotools-dev (20220109.1) ... Selecting previously unselected package automake. Preparing to unpack .../13-automake_1%3a1.16.5-1.3_all.deb ... Unpacking automake (1:1.16.5-1.3) ... Selecting previously unselected package autopoint. Preparing to unpack .../14-autopoint_0.21-12_all.deb ... Unpacking autopoint (0.21-12) ... Selecting previously unselected package libdebhelper-perl. Preparing to unpack .../15-libdebhelper-perl_13.11.4_all.deb ... Unpacking libdebhelper-perl (13.11.4) ... Selecting previously unselected package libtool. Preparing to unpack .../16-libtool_2.4.7-5_all.deb ... Unpacking libtool (2.4.7-5) ... Selecting previously unselected package dh-autoreconf. Preparing to unpack .../17-dh-autoreconf_20_all.deb ... Unpacking dh-autoreconf (20) ... Selecting previously unselected package libarchive-zip-perl. Preparing to unpack .../18-libarchive-zip-perl_1.68-1_all.deb ... Unpacking libarchive-zip-perl (1.68-1) ... Selecting previously unselected package libsub-override-perl. Preparing to unpack .../19-libsub-override-perl_0.09-4_all.deb ... Unpacking libsub-override-perl (0.09-4) ... Selecting previously unselected package libfile-stripnondeterminism-perl. Preparing to unpack .../20-libfile-stripnondeterminism-perl_1.13.1-1_all.deb ... Unpacking libfile-stripnondeterminism-perl (1.13.1-1) ... Selecting previously unselected package dh-strip-nondeterminism. Preparing to unpack .../21-dh-strip-nondeterminism_1.13.1-1_all.deb ... Unpacking dh-strip-nondeterminism (1.13.1-1) ... Selecting previously unselected package libelf1:armhf. Preparing to unpack .../22-libelf1_0.188-2.1_armhf.deb ... Unpacking libelf1:armhf (0.188-2.1) ... Selecting previously unselected package dwz. Preparing to unpack .../23-dwz_0.15-1_armhf.deb ... Unpacking dwz (0.15-1) ... Selecting previously unselected package libicu72:armhf. Preparing to unpack .../24-libicu72_72.1-3_armhf.deb ... Unpacking libicu72:armhf (72.1-3) ... Selecting previously unselected package libxml2:armhf. Preparing to unpack .../25-libxml2_2.9.14+dfsg-1.3~deb12u1_armhf.deb ... Unpacking libxml2:armhf (2.9.14+dfsg-1.3~deb12u1) ... Selecting previously unselected package gettext. Preparing to unpack .../26-gettext_0.21-12_armhf.deb ... Unpacking gettext (0.21-12) ... Selecting previously unselected package intltool-debian. Preparing to unpack .../27-intltool-debian_0.35.0+20060710.6_all.deb ... Unpacking intltool-debian (0.35.0+20060710.6) ... Selecting previously unselected package po-debconf. Preparing to unpack .../28-po-debconf_1.0.21+nmu1_all.deb ... Unpacking po-debconf (1.0.21+nmu1) ... Selecting previously unselected package debhelper. Preparing to unpack .../29-debhelper_13.11.4_all.deb ... Unpacking debhelper (13.11.4) ... Selecting previously unselected package dh-golang. Preparing to unpack .../30-dh-golang_1.59_all.deb ... Unpacking dh-golang (1.59) ... Selecting previously unselected package golang-1.19-src. Preparing to unpack .../31-golang-1.19-src_1.19.8-2_all.deb ... Unpacking golang-1.19-src (1.19.8-2) ... Selecting previously unselected package golang-1.19-go. Preparing to unpack .../32-golang-1.19-go_1.19.8-2_armhf.deb ... Unpacking golang-1.19-go (1.19.8-2) ... Selecting previously unselected package golang-src. Preparing to unpack .../33-golang-src_2%3a1.19~1_all.deb ... Unpacking golang-src (2:1.19~1) ... Selecting previously unselected package golang-go:armhf. Preparing to unpack .../34-golang-go_2%3a1.19~1_armhf.deb ... Unpacking golang-go:armhf (2:1.19~1) ... Selecting previously unselected package golang-any:armhf. Preparing to unpack .../35-golang-any_2%3a1.19~1_armhf.deb ... Unpacking golang-any:armhf (2:1.19~1) ... Selecting previously unselected package golang-github-alecthomas-units-dev. Preparing to unpack .../36-golang-github-alecthomas-units-dev_0.0~git20211218.b94a6e3-1_all.deb ... Unpacking golang-github-alecthomas-units-dev (0.0~git20211218.b94a6e3-1) ... Selecting previously unselected package golang-github-circonus-labs-circonusllhist-dev. Preparing to unpack .../37-golang-github-circonus-labs-circonusllhist-dev_0.0~git20191022.ec08cde-1_all.deb ... Unpacking golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ... Selecting previously unselected package golang-github-hashicorp-go-cleanhttp-dev. Preparing to unpack .../38-golang-github-hashicorp-go-cleanhttp-dev_0.5.2-1_all.deb ... Unpacking golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ... Selecting previously unselected package golang-golang-x-sys-dev. Preparing to unpack .../39-golang-golang-x-sys-dev_0.3.0-1_all.deb ... Unpacking golang-golang-x-sys-dev (0.3.0-1) ... Selecting previously unselected package golang-github-mattn-go-isatty-dev. Preparing to unpack .../40-golang-github-mattn-go-isatty-dev_0.0.17-1_all.deb ... Unpacking golang-github-mattn-go-isatty-dev (0.0.17-1) ... Selecting previously unselected package golang-github-mattn-go-colorable-dev. Preparing to unpack .../41-golang-github-mattn-go-colorable-dev_0.1.13-1_all.deb ... Unpacking golang-github-mattn-go-colorable-dev (0.1.13-1) ... Selecting previously unselected package golang-github-fatih-color-dev. Preparing to unpack .../42-golang-github-fatih-color-dev_1.7.0-1_all.deb ... Unpacking golang-github-fatih-color-dev (1.7.0-1) ... Selecting previously unselected package golang-github-davecgh-go-spew-dev. Preparing to unpack .../43-golang-github-davecgh-go-spew-dev_1.1.1-3_all.deb ... Unpacking golang-github-davecgh-go-spew-dev (1.1.1-3) ... Selecting previously unselected package golang-github-pmezard-go-difflib-dev. Preparing to unpack .../44-golang-github-pmezard-go-difflib-dev_1.0.0-3_all.deb ... Unpacking golang-github-pmezard-go-difflib-dev (1.0.0-3) ... Selecting previously unselected package golang-github-stretchr-objx-dev. Preparing to unpack .../45-golang-github-stretchr-objx-dev_0.5.0-1_all.deb ... Unpacking golang-github-stretchr-objx-dev (0.5.0-1) ... Selecting previously unselected package golang-gopkg-yaml.v3-dev. Preparing to unpack .../46-golang-gopkg-yaml.v3-dev_3.0.1-3_all.deb ... Unpacking golang-gopkg-yaml.v3-dev (3.0.1-3) ... Selecting previously unselected package golang-github-stretchr-testify-dev. Preparing to unpack .../47-golang-github-stretchr-testify-dev_1.8.1-1_all.deb ... Unpacking golang-github-stretchr-testify-dev (1.8.1-1) ... Selecting previously unselected package golang-github-hashicorp-go-hclog-dev. Preparing to unpack .../48-golang-github-hashicorp-go-hclog-dev_0.11.0-1_all.deb ... Unpacking golang-github-hashicorp-go-hclog-dev (0.11.0-1) ... Selecting previously unselected package golang-github-hashicorp-go-retryablehttp-dev. Preparing to unpack .../49-golang-github-hashicorp-go-retryablehttp-dev_0.7.1-1_all.deb ... Unpacking golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ... Selecting previously unselected package golang-github-pkg-errors-dev. Preparing to unpack .../50-golang-github-pkg-errors-dev_0.9.1-2_all.deb ... Unpacking golang-github-pkg-errors-dev (0.9.1-2) ... Selecting previously unselected package golang-github-tv42-httpunix-dev. Preparing to unpack .../51-golang-github-tv42-httpunix-dev_0.0~git20150427.b75d861-3_all.deb ... Unpacking golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ... Selecting previously unselected package golang-github-circonus-labs-circonus-gometrics-dev. Preparing to unpack .../52-golang-github-circonus-labs-circonus-gometrics-dev_2.3.1-4_all.deb ... Unpacking golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ... Selecting previously unselected package golang-github-datadog-datadog-go-dev. Preparing to unpack .../53-golang-github-datadog-datadog-go-dev_2.1.0-3_all.deb ... Unpacking golang-github-datadog-datadog-go-dev (2.1.0-3) ... Selecting previously unselected package golang-github-hashicorp-go-uuid-dev. Preparing to unpack .../54-golang-github-hashicorp-go-uuid-dev_1.0.2-2_all.deb ... Unpacking golang-github-hashicorp-go-uuid-dev (1.0.2-2) ... Selecting previously unselected package golang-github-hashicorp-golang-lru-dev. Preparing to unpack .../55-golang-github-hashicorp-golang-lru-dev_0.5.4-2_all.deb ... Unpacking golang-github-hashicorp-golang-lru-dev (0.5.4-2) ... Selecting previously unselected package golang-github-hashicorp-go-immutable-radix-dev. Preparing to unpack .../56-golang-github-hashicorp-go-immutable-radix-dev_1.3.1-3_all.deb ... Unpacking golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ... Selecting previously unselected package golang-github-pascaldekloe-goe-dev. Preparing to unpack .../57-golang-github-pascaldekloe-goe-dev_0.1.0-4_all.deb ... Unpacking golang-github-pascaldekloe-goe-dev (0.1.0-4) ... Selecting previously unselected package golang-github-beorn7-perks-dev. Preparing to unpack .../58-golang-github-beorn7-perks-dev_1.0.1-1_all.deb ... Unpacking golang-github-beorn7-perks-dev (1.0.1-1) ... Selecting previously unselected package golang-github-cespare-xxhash-dev. Preparing to unpack .../59-golang-github-cespare-xxhash-dev_2.1.1-2_all.deb ... Unpacking golang-github-cespare-xxhash-dev (2.1.1-2) ... Selecting previously unselected package golang-github-golang-protobuf-1-3-dev. Preparing to unpack .../60-golang-github-golang-protobuf-1-3-dev_1.3.5-4_all.deb ... Unpacking golang-github-golang-protobuf-1-3-dev (1.3.5-4) ... Selecting previously unselected package golang-github-modern-go-concurrent-dev. Preparing to unpack .../61-golang-github-modern-go-concurrent-dev_1.0.3-1.1_all.deb ... Unpacking golang-github-modern-go-concurrent-dev (1.0.3-1.1) ... Selecting previously unselected package golang-github-modern-go-reflect2-dev. Preparing to unpack .../62-golang-github-modern-go-reflect2-dev_1.0.2-2_all.deb ... Unpacking golang-github-modern-go-reflect2-dev (1.0.2-2) ... Selecting previously unselected package golang-github-json-iterator-go-dev. Preparing to unpack .../63-golang-github-json-iterator-go-dev_1.1.12-1_all.deb ... Unpacking golang-github-json-iterator-go-dev (1.1.12-1) ... Selecting previously unselected package golang-github-prometheus-client-model-dev. Preparing to unpack .../64-golang-github-prometheus-client-model-dev_0.3.0-3_all.deb ... Unpacking golang-github-prometheus-client-model-dev (0.3.0-3) ... Selecting previously unselected package golang-gopkg-yaml.v2-dev. Preparing to unpack .../65-golang-gopkg-yaml.v2-dev_2.4.0-4_all.deb ... Unpacking golang-gopkg-yaml.v2-dev (2.4.0-4) ... Selecting previously unselected package golang-github-jmespath-go-jmespath-dev. Preparing to unpack .../66-golang-github-jmespath-go-jmespath-dev_0.4.0-2_all.deb ... Unpacking golang-github-jmespath-go-jmespath-dev (0.4.0-2) ... Selecting previously unselected package golang-golang-x-text-dev. Preparing to unpack .../67-golang-golang-x-text-dev_0.7.0-1_all.deb ... Unpacking golang-golang-x-text-dev (0.7.0-1) ... Selecting previously unselected package golang-golang-x-net-dev. Preparing to unpack .../68-golang-golang-x-net-dev_1%3a0.7.0+dfsg-1_all.deb ... Unpacking golang-golang-x-net-dev (1:0.7.0+dfsg-1) ... Selecting previously unselected package golang-github-aws-aws-sdk-go-dev. Preparing to unpack .../69-golang-github-aws-aws-sdk-go-dev_1.44.133-1_all.deb ... Unpacking golang-github-aws-aws-sdk-go-dev (1.44.133-1) ... Selecting previously unselected package golang-github-go-logfmt-logfmt-dev. Preparing to unpack .../70-golang-github-go-logfmt-logfmt-dev_0.5.0-2_all.deb ... Unpacking golang-github-go-logfmt-logfmt-dev (0.5.0-2) ... Selecting previously unselected package golang-github-go-kit-log-dev. Preparing to unpack .../71-golang-github-go-kit-log-dev_0.2.1-1_all.deb ... Unpacking golang-github-go-kit-log-dev (0.2.1-1) ... Selecting previously unselected package golang-github-julienschmidt-httprouter-dev. Preparing to unpack .../72-golang-github-julienschmidt-httprouter-dev_1.3.0-2_all.deb ... Unpacking golang-github-julienschmidt-httprouter-dev (1.3.0-2) ... Selecting previously unselected package golang-github-jpillora-backoff-dev. Preparing to unpack .../73-golang-github-jpillora-backoff-dev_1.0.0-1.1_all.deb ... Unpacking golang-github-jpillora-backoff-dev (1.0.0-1.1) ... Selecting previously unselected package golang-github-mwitkow-go-conntrack-dev. Preparing to unpack .../74-golang-github-mwitkow-go-conntrack-dev_0.0~git20190716.2f06839-3_all.deb ... Unpacking golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ... Selecting previously unselected package golang-golang-x-oauth2-dev. Preparing to unpack .../75-golang-golang-x-oauth2-dev_0.3.0-1_all.deb ... Unpacking golang-golang-x-oauth2-dev (0.3.0-1) ... Selecting previously unselected package golang-google-protobuf-dev. Preparing to unpack .../76-golang-google-protobuf-dev_1.28.1-3_all.deb ... Unpacking golang-google-protobuf-dev (1.28.1-3) ... Selecting previously unselected package golang-gopkg-alecthomas-kingpin.v2-dev. Preparing to unpack .../77-golang-gopkg-alecthomas-kingpin.v2-dev_2.2.6-4_all.deb ... Unpacking golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ... Selecting previously unselected package golang-protobuf-extensions-dev. Preparing to unpack .../78-golang-protobuf-extensions-dev_1.0.1-3_all.deb ... Unpacking golang-protobuf-extensions-dev (1.0.1-3) ... Selecting previously unselected package golang-github-prometheus-common-dev. Preparing to unpack .../79-golang-github-prometheus-common-dev_0.39.0-2_all.deb ... Unpacking golang-github-prometheus-common-dev (0.39.0-2) ... Selecting previously unselected package golang-golang-x-sync-dev. Preparing to unpack .../80-golang-golang-x-sync-dev_0.1.0-1_all.deb ... Unpacking golang-golang-x-sync-dev (0.1.0-1) ... Selecting previously unselected package golang-github-prometheus-procfs-dev. Preparing to unpack .../81-golang-github-prometheus-procfs-dev_0.8.0-3_all.deb ... Unpacking golang-github-prometheus-procfs-dev (0.8.0-3) ... Selecting previously unselected package golang-github-prometheus-client-golang-dev. Preparing to unpack .../82-golang-github-prometheus-client-golang-dev_1.14.0-3_all.deb ... Unpacking golang-github-prometheus-client-golang-dev (1.14.0-3) ... Selecting previously unselected package golang-github-armon-go-metrics-dev. Preparing to unpack .../83-golang-github-armon-go-metrics-dev_0.3.4-1_all.deb ... Unpacking golang-github-armon-go-metrics-dev (0.3.4-1) ... Selecting previously unselected package golang-github-creack-pty-dev. Preparing to unpack .../84-golang-github-creack-pty-dev_1.1.18-1_all.deb ... Unpacking golang-github-creack-pty-dev (1.1.18-1) ... Selecting previously unselected package golang-github-vmihailenco-tagparser-dev. Preparing to unpack .../85-golang-github-vmihailenco-tagparser-dev_0.1.1-4_all.deb ... Unpacking golang-github-vmihailenco-tagparser-dev (0.1.1-4) ... Selecting previously unselected package golang-gopkg-vmihailenco-msgpack.v2-dev. Preparing to unpack .../86-golang-gopkg-vmihailenco-msgpack.v2-dev_4.3.1-2_all.deb ... Unpacking golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ... Selecting previously unselected package golang-gopkg-tomb.v2-dev. Preparing to unpack .../87-golang-gopkg-tomb.v2-dev_0.0~git20161208.d5d1b58-3_all.deb ... Unpacking golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ... Selecting previously unselected package libsasl2-modules-db:armhf. Preparing to unpack .../88-libsasl2-modules-db_2.1.28+dfsg-10_armhf.deb ... Unpacking libsasl2-modules-db:armhf (2.1.28+dfsg-10) ... Selecting previously unselected package libsasl2-2:armhf. Preparing to unpack .../89-libsasl2-2_2.1.28+dfsg-10_armhf.deb ... Unpacking libsasl2-2:armhf (2.1.28+dfsg-10) ... Selecting previously unselected package libsasl2-dev. Preparing to unpack .../90-libsasl2-dev_2.1.28+dfsg-10_armhf.deb ... Unpacking libsasl2-dev (2.1.28+dfsg-10) ... Selecting previously unselected package golang-github-kr-text-dev. Preparing to unpack .../91-golang-github-kr-text-dev_0.2.0-2_all.deb ... Unpacking golang-github-kr-text-dev (0.2.0-2) ... Selecting previously unselected package golang-github-pkg-diff-dev. Preparing to unpack .../92-golang-github-pkg-diff-dev_0.0~git20210226.20ebb0f-1_all.deb ... Unpacking golang-github-pkg-diff-dev (0.0~git20210226.20ebb0f-1) ... Selecting previously unselected package golang-github-rogpeppe-go-internal-dev. Preparing to unpack .../93-golang-github-rogpeppe-go-internal-dev_1.9.0-1_all.deb ... Unpacking golang-github-rogpeppe-go-internal-dev (1.9.0-1) ... Selecting previously unselected package golang-github-kr-pretty-dev. Preparing to unpack .../94-golang-github-kr-pretty-dev_0.3.0-1_all.deb ... Unpacking golang-github-kr-pretty-dev (0.3.0-1) ... Selecting previously unselected package golang-gopkg-check.v1-dev. Preparing to unpack .../95-golang-gopkg-check.v1-dev_0.0+git20200902.038fdea-1_all.deb ... Unpacking golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ... Selecting previously unselected package golang-gopkg-mgo.v2-dev. Preparing to unpack .../96-golang-gopkg-mgo.v2-dev_2016.08.01-7_all.deb ... Unpacking golang-gopkg-mgo.v2-dev (2016.08.01-7) ... Selecting previously unselected package golang-github-hashicorp-go-msgpack-dev. Preparing to unpack .../97-golang-github-hashicorp-go-msgpack-dev_0.5.5-1_all.deb ... Unpacking golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ... Setting up libpipeline1:armhf (1.5.7-1) ... Setting up golang-gopkg-yaml.v2-dev (2.4.0-4) ... Setting up golang-github-hashicorp-go-uuid-dev (1.0.2-2) ... Setting up libicu72:armhf (72.1-3) ... Setting up bsdextrautils (2.38.1-5+b1) ... Setting up golang-github-creack-pty-dev (1.1.18-1) ... Setting up golang-github-pkg-diff-dev (0.0~git20210226.20ebb0f-1) ... Setting up golang-github-stretchr-objx-dev (0.5.0-1) ... Setting up libmagic-mgc (1:5.44-3) ... Setting up golang-github-pkg-errors-dev (0.9.1-2) ... Setting up golang-github-hashicorp-golang-lru-dev (0.5.4-2) ... Setting up libarchive-zip-perl (1.68-1) ... Setting up golang-gopkg-yaml.v3-dev (3.0.1-3) ... Setting up libdebhelper-perl (13.11.4) ... Setting up golang-golang-x-sys-dev (0.3.0-1) ... Setting up libmagic1:armhf (1:5.44-3) ... Setting up golang-github-pmezard-go-difflib-dev (1.0.0-3) ... Setting up golang-github-modern-go-concurrent-dev (1.0.3-1.1) ... Setting up gettext-base (0.21-12) ... Setting up golang-1.19-src (1.19.8-2) ... Setting up m4 (1.4.19-3) ... Setting up golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ... Setting up golang-github-golang-protobuf-1-3-dev (1.3.5-4) ... Setting up file (1:5.44-3) ... Setting up golang-github-beorn7-perks-dev (1.0.1-1) ... Setting up golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ... Setting up libsasl2-modules-db:armhf (2.1.28+dfsg-10) ... Setting up golang-github-cespare-xxhash-dev (2.1.1-2) ... Setting up golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ... Setting up golang-github-jpillora-backoff-dev (1.0.0-1.1) ... Setting up golang-github-davecgh-go-spew-dev (1.1.1-3) ... Setting up autotools-dev (20220109.1) ... Setting up golang-github-pascaldekloe-goe-dev (0.1.0-4) ... Setting up golang-github-go-logfmt-logfmt-dev (0.5.0-2) ... Setting up golang-google-protobuf-dev (1.28.1-3) ... Setting up golang-github-kr-text-dev (0.2.0-2) ... Setting up golang-github-stretchr-testify-dev (1.8.1-1) ... Setting up golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ... Setting up golang-golang-x-sync-dev (0.1.0-1) ... Setting up autopoint (0.21-12) ... Setting up libsasl2-2:armhf (2.1.28+dfsg-10) ... Setting up golang-github-julienschmidt-httprouter-dev (1.3.0-2) ... Setting up autoconf (2.71-3) ... Setting up golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ... Setting up sensible-utils (0.0.17+nmu1) ... Setting up golang-golang-x-text-dev (0.7.0-1) ... Setting up libuchardet0:armhf (0.0.7-1) ... Setting up golang-github-prometheus-procfs-dev (0.8.0-3) ... Setting up golang-github-alecthomas-units-dev (0.0~git20211218.b94a6e3-1) ... Setting up golang-github-modern-go-reflect2-dev (1.0.2-2) ... Setting up libsub-override-perl (0.09-4) ... Setting up golang-github-prometheus-client-model-dev (0.3.0-3) ... Setting up golang-github-datadog-datadog-go-dev (2.1.0-3) ... Setting up golang-github-rogpeppe-go-internal-dev (1.9.0-1) ... Setting up golang-src (2:1.19~1) ... Setting up libelf1:armhf (0.188-2.1) ... Setting up golang-github-vmihailenco-tagparser-dev (0.1.1-4) ... Setting up libxml2:armhf (2.9.14+dfsg-1.3~deb12u1) ... Setting up golang-github-jmespath-go-jmespath-dev (0.4.0-2) ... Setting up golang-1.19-go (1.19.8-2) ... Setting up automake (1:1.16.5-1.3) ... update-alternatives: using /usr/bin/automake-1.16 to provide /usr/bin/automake (automake) in auto mode Setting up golang-github-kr-pretty-dev (0.3.0-1) ... Setting up libfile-stripnondeterminism-perl (1.13.1-1) ... Setting up golang-github-mattn-go-isatty-dev (0.0.17-1) ... Setting up golang-github-go-kit-log-dev (0.2.1-1) ... Setting up gettext (0.21-12) ... Setting up golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ... Setting up golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ... Setting up libtool (2.4.7-5) ... Setting up golang-go:armhf (2:1.19~1) ... Setting up golang-github-mattn-go-colorable-dev (0.1.13-1) ... Setting up golang-protobuf-extensions-dev (1.0.1-3) ... Setting up golang-github-json-iterator-go-dev (1.1.12-1) ... Setting up golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ... Setting up golang-any:armhf (2:1.19~1) ... Setting up intltool-debian (0.35.0+20060710.6) ... Setting up dh-autoreconf (20) ... Setting up libsasl2-dev (2.1.28+dfsg-10) ... Setting up golang-gopkg-mgo.v2-dev (2016.08.01-7) ... Setting up golang-golang-x-net-dev (1:0.7.0+dfsg-1) ... Setting up dh-strip-nondeterminism (1.13.1-1) ... Setting up dwz (0.15-1) ... Setting up groff-base (1.22.4-10) ... Setting up golang-golang-x-oauth2-dev (0.3.0-1) ... Setting up golang-github-aws-aws-sdk-go-dev (1.44.133-1) ... Setting up golang-github-fatih-color-dev (1.7.0-1) ... Setting up golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ... Setting up po-debconf (1.0.21+nmu1) ... Setting up man-db (2.11.2-2) ... Not building database; man-db/auto-update is not 'true'. Setting up golang-github-hashicorp-go-hclog-dev (0.11.0-1) ... Setting up debhelper (13.11.4) ... Setting up golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ... Setting up dh-golang (1.59) ... Setting up golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ... Setting up golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ... Setting up golang-github-prometheus-common-dev (0.39.0-2) ... Setting up golang-github-prometheus-client-golang-dev (1.14.0-3) ... Setting up golang-github-armon-go-metrics-dev (0.3.4-1) ... Processing triggers for libc-bin (2.36-9+deb12u4) ... Reading package lists... Building dependency tree... Reading state information... Reading extended state information... Initializing package states... Writing extended state information... Building tag database... -> Finished parsing the build-deps Reading package lists... Building dependency tree... Reading state information... fakeroot is already the newest version (1.31-1.2). 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. I: Building the package I: user script /srv/workspace/pbuilder/18637/tmp/hooks/A99_set_merged_usr starting Not re-configuring usrmerge for bookworm I: user script /srv/workspace/pbuilder/18637/tmp/hooks/A99_set_merged_usr finished hostname: Name or service not known I: Running cd /build/reproducible-path/golang-github-hashicorp-raft-1.1.2/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-genchanges -S > ../golang-github-hashicorp-raft_1.1.2-1_source.changes dpkg-buildpackage: info: source package golang-github-hashicorp-raft dpkg-buildpackage: info: source version 1.1.2-1 dpkg-buildpackage: info: source distribution unstable dpkg-buildpackage: info: source changed by Dmitry Smirnov dpkg-source --before-build . dpkg-buildpackage: info: host architecture armhf fakeroot debian/rules clean dh clean --buildsystem=golang --with=golang dh_auto_clean -O--buildsystem=golang dh_autoreconf_clean -O--buildsystem=golang dh_clean -O--buildsystem=golang debian/rules build dh build --buildsystem=golang --with=golang dh_update_autotools_config -O--buildsystem=golang dh_autoreconf -O--buildsystem=golang dh_auto_configure -O--buildsystem=golang dh_auto_build -O--buildsystem=golang cd obj-arm-linux-gnueabihf && go install -trimpath -v -p 4 github.com/hashicorp/raft github.com/hashicorp/raft/bench internal/goarch internal/goexperiment internal/unsafeheader internal/goos internal/cpu runtime/internal/math internal/abi runtime/internal/sys runtime/internal/syscall internal/race sync/atomic unicode/utf8 unicode internal/bytealg runtime/internal/atomic container/list internal/itoa math/bits crypto/internal/subtle crypto/subtle crypto/internal/boring/sig math encoding unicode/utf16 runtime internal/nettrace vendor/golang.org/x/crypto/cryptobyte/asn1 vendor/golang.org/x/crypto/internal/subtle internal/reflectlite sync internal/testlog internal/singleflight math/rand internal/sysinfo runtime/cgo errors sort io internal/oserror syscall strconv bytes strings hash crypto reflect bufio time crypto/internal/randutil internal/syscall/unix internal/safefilepath internal/syscall/execenv path github.com/hashicorp/golang-lru/simplelru vendor/golang.org/x/net/dns/dnsmessage github.com/hashicorp/go-immutable-radix hash/crc32 context internal/poll io/fs crypto/internal/nistec/fiat crypto/rc4 vendor/golang.org/x/text/transform os encoding/binary internal/fmtsort net/http/internal/ascii regexp/syntax crypto/cipher crypto/internal/boring fmt encoding/base64 internal/godebug internal/intern crypto/aes net/netip crypto/des crypto/sha512 crypto/internal/edwards25519/field math/big encoding/json log net compress/flate compress/gzip crypto/internal/nistec crypto/internal/edwards25519 crypto/hmac crypto/md5 crypto/rand crypto/elliptic crypto/internal/boring/bbig encoding/asn1 crypto/ed25519 crypto/rsa crypto/sha1 crypto/sha256 crypto/dsa vendor/golang.org/x/crypto/cryptobyte encoding/hex encoding/pem crypto/x509/pkix net/url path/filepath crypto/ecdsa vendor/golang.org/x/crypto/chacha20 vendor/golang.org/x/crypto/internal/poly1305 vendor/golang.org/x/crypto/curve25519/internal/field vendor/golang.org/x/crypto/hkdf vendor/golang.org/x/crypto/chacha20poly1305 vendor/golang.org/x/text/unicode/bidi vendor/golang.org/x/crypto/curve25519 vendor/golang.org/x/text/unicode/norm vendor/golang.org/x/net/http2/hpack mime vendor/golang.org/x/text/secure/bidirule mime/quotedprintable net/http/internal os/signal golang.org/x/sys/unix io/ioutil regexp vendor/golang.org/x/net/idna encoding/gob go/token html text/template/parse crypto/x509 net/textproto vendor/golang.org/x/net/http/httpguts vendor/golang.org/x/net/http/httpproxy mime/multipart github.com/mattn/go-isatty text/template github.com/mattn/go-colorable hash/crc64 github.com/fatih/color flag github.com/hashicorp/go-hclog crypto/tls runtime/debug runtime/trace testing html/template net/http/httptrace net/http net/rpc github.com/armon/go-metrics github.com/hashicorp/go-msgpack/codec github.com/hashicorp/raft github.com/hashicorp/raft/bench dh_auto_test -O--buildsystem=golang cd obj-arm-linux-gnueabihf && go test -vet=off -v -p 4 github.com/hashicorp/raft github.com/hashicorp/raft/bench === RUN TestCommitment_setVoters --- PASS: TestCommitment_setVoters (0.00s) === RUN TestCommitment_match_max --- PASS: TestCommitment_match_max (0.00s) === RUN TestCommitment_match_nonVoting --- PASS: TestCommitment_match_nonVoting (0.00s) === RUN TestCommitment_recalculate --- PASS: TestCommitment_recalculate (0.00s) === RUN TestCommitment_recalculate_startIndex --- PASS: TestCommitment_recalculate_startIndex (0.00s) === RUN TestCommitment_noVoterSanity --- PASS: TestCommitment_noVoterSanity (0.00s) === RUN TestCommitment_singleVoter --- PASS: TestCommitment_singleVoter (0.00s) === RUN TestConfiguration_Configuration_Clone --- PASS: TestConfiguration_Configuration_Clone (0.00s) === RUN TestConfiguration_configurations_Clone --- PASS: TestConfiguration_configurations_Clone (0.00s) === RUN TestConfiguration_hasVote --- PASS: TestConfiguration_hasVote (0.00s) === RUN TestConfiguration_checkConfiguration --- PASS: TestConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_nextConfiguration_table --- PASS: TestConfiguration_nextConfiguration_table (0.00s) === RUN TestConfiguration_nextConfiguration_prevIndex --- PASS: TestConfiguration_nextConfiguration_prevIndex (0.00s) === RUN TestConfiguration_nextConfiguration_checkConfiguration --- PASS: TestConfiguration_nextConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_encodeDecodePeers --- PASS: TestConfiguration_encodeDecodePeers (0.00s) === RUN TestConfiguration_encodeDecodeConfiguration --- PASS: TestConfiguration_encodeDecodeConfiguration (0.00s) === RUN TestDiscardSnapshotStoreImpl --- PASS: TestDiscardSnapshotStoreImpl (0.00s) === RUN TestDiscardSnapshotSinkImpl --- PASS: TestDiscardSnapshotSinkImpl (0.00s) === RUN TestFileSnapshotStoreImpl --- PASS: TestFileSnapshotStoreImpl (0.00s) === RUN TestFileSnapshotSinkImpl --- PASS: TestFileSnapshotSinkImpl (0.00s) === RUN TestFileSS_CreateSnapshotMissingParentDir testing.go:153: 2024-06-12T04:32:23.101+1400 [INFO] creating new snapshot: path=/tmp/raft2739472815/raft441183824/snapshots/3-10-1718116343101.tmp --- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.01s) === RUN TestFileSS_CreateSnapshot testing.go:153: 2024-06-12T04:32:23.111+1400 [INFO] creating new snapshot: path=/tmp/raft350919335/snapshots/3-10-1718116343111.tmp testing.go:153: 2024-06-12T04:32:23.115+1400 [WARN] found temporary snapshot: name=3-10-1718116343111.tmp --- PASS: TestFileSS_CreateSnapshot (0.02s) === RUN TestFileSS_CancelSnapshot testing.go:153: 2024-06-12T04:32:23.126+1400 [INFO] creating new snapshot: path=/tmp/raft702078745/snapshots/3-10-1718116343126.tmp --- PASS: TestFileSS_CancelSnapshot (0.01s) === RUN TestFileSS_Retention testing.go:153: 2024-06-12T04:32:23.134+1400 [INFO] creating new snapshot: path=/tmp/raft2677625755/snapshots/3-10-1718116343134.tmp testing.go:153: 2024-06-12T04:32:23.148+1400 [INFO] creating new snapshot: path=/tmp/raft2677625755/snapshots/3-11-1718116343148.tmp testing.go:153: 2024-06-12T04:32:23.161+1400 [INFO] creating new snapshot: path=/tmp/raft2677625755/snapshots/3-12-1718116343161.tmp testing.go:153: 2024-06-12T04:32:23.175+1400 [INFO] reaping snapshot: path=/tmp/raft2677625755/snapshots/3-10-1718116343134 testing.go:153: 2024-06-12T04:32:23.175+1400 [INFO] creating new snapshot: path=/tmp/raft2677625755/snapshots/3-13-1718116343175.tmp testing.go:153: 2024-06-12T04:32:23.188+1400 [INFO] reaping snapshot: path=/tmp/raft2677625755/snapshots/3-11-1718116343148 testing.go:153: 2024-06-12T04:32:23.188+1400 [INFO] creating new snapshot: path=/tmp/raft2677625755/snapshots/3-14-1718116343188.tmp testing.go:153: 2024-06-12T04:32:23.202+1400 [INFO] reaping snapshot: path=/tmp/raft2677625755/snapshots/3-12-1718116343161 --- PASS: TestFileSS_Retention (0.07s) === RUN TestFileSS_BadPerm --- PASS: TestFileSS_BadPerm (0.00s) === RUN TestFileSS_MissingParentDir --- PASS: TestFileSS_MissingParentDir (0.00s) === RUN TestFileSS_Ordering testing.go:153: 2024-06-12T04:32:23.206+1400 [INFO] creating new snapshot: path=/tmp/raft3603578058/snapshots/5-130350-1718116343206.tmp testing.go:153: 2024-06-12T04:32:23.218+1400 [INFO] creating new snapshot: path=/tmp/raft3603578058/snapshots/36-204917-1718116343218.tmp --- PASS: TestFileSS_Ordering (0.03s) === RUN TestDeferFutureSuccess --- PASS: TestDeferFutureSuccess (0.00s) === RUN TestDeferFutureError --- PASS: TestDeferFutureError (0.00s) === RUN TestDeferFutureConcurrent --- PASS: TestDeferFutureConcurrent (0.00s) === RUN TestInmemSnapshotStoreImpl --- PASS: TestInmemSnapshotStoreImpl (0.00s) === RUN TestInmemSnapshotSinkImpl --- PASS: TestInmemSnapshotSinkImpl (0.00s) === RUN TestInmemSS_CreateSnapshot --- PASS: TestInmemSS_CreateSnapshot (0.00s) === RUN TestInmemSS_OpenSnapshotTwice --- PASS: TestInmemSS_OpenSnapshotTwice (0.00s) === RUN TestInmemTransportImpl --- PASS: TestInmemTransportImpl (0.00s) === RUN TestInmemTransportWriteTimeout --- PASS: TestInmemTransportWriteTimeout (0.01s) === RUN TestRaft_Integ --- SKIP: TestRaft_Integ (0.00s) === RUN TestLogCache --- PASS: TestLogCache (0.00s) === RUN TestNetworkTransport_CloseStreams --- PASS: TestNetworkTransport_CloseStreams (0.00s) === RUN TestNetworkTransport_StartStop --- PASS: TestNetworkTransport_StartStop (0.00s) === RUN TestNetworkTransport_Heartbeat_FastPath --- PASS: TestNetworkTransport_Heartbeat_FastPath (0.00s) === RUN TestNetworkTransport_AppendEntries --- PASS: TestNetworkTransport_AppendEntries (0.00s) === RUN TestNetworkTransport_AppendEntriesPipeline --- PASS: TestNetworkTransport_AppendEntriesPipeline (0.00s) === RUN TestNetworkTransport_AppendEntriesPipeline_CloseStreams net_transport_test.go:339: Disabled unreliable test --- SKIP: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.00s) === RUN TestNetworkTransport_RequestVote --- PASS: TestNetworkTransport_RequestVote (0.00s) === RUN TestNetworkTransport_InstallSnapshot --- PASS: TestNetworkTransport_InstallSnapshot (0.00s) === RUN TestNetworkTransport_EncodeDecode --- PASS: TestNetworkTransport_EncodeDecode (0.00s) === RUN TestNetworkTransport_EncodeDecode_AddressProvider --- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s) === RUN TestNetworkTransport_PooledConn --- PASS: TestNetworkTransport_PooledConn (0.00s) === RUN TestNetworkTransport_ListenBackoff net_transport_test.go:718: countingWriter: 2024-06-12T04:32:23.257+1400 [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2024-06-12T04:32:23.263+1400 [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2024-06-12T04:32:23.275+1400 [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2024-06-12T04:32:23.296+1400 [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2024-06-12T04:32:23.336+1400 [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2024-06-12T04:32:23.417+1400 [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2024-06-12T04:32:23.579+1400 [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2024-06-12T04:32:23.900+1400 [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2024-06-12T04:32:24.543+1400 [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2024-06-12T04:32:25.547+1400 [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2024-06-12T04:32:26.551+1400 [ERROR] test: failed to accept connection: error="intentional error in test" --- PASS: TestNetworkTransport_ListenBackoff (4.00s) === RUN TestPeersJSON_BadConfiguration --- PASS: TestPeersJSON_BadConfiguration (0.00s) === RUN TestPeersJSON_ReadPeersJSON --- PASS: TestPeersJSON_ReadPeersJSON (0.00s) === RUN TestPeersJSON_ReadConfigJSON --- PASS: TestPeersJSON_ReadConfigJSON (0.00s) === RUN TestRaft_StartStop testLoggerAdapter verbose: server-0eae6cf0-ea81-0a58-e22b-5697f6c319fc: 2024-06-12T04:32:27.263+1400 [INFO] server-0eae6cf0-ea81-0a58-e22b-5697f6c319fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0eae6cf0-ea81-0a58-e22b-5697f6c319fc Address:0eae6cf0-ea81-0a58-e22b-5697f6c319fc}]" testing.go:149: server-0eae6cf0-ea81-0a58-e22b-5697f6c319fc: 2024-06-12T04:32:27.263+1400 [INFO] server-0eae6cf0-ea81-0a58-e22b-5697f6c319fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0eae6cf0-ea81-0a58-e22b-5697f6c319fc Address:0eae6cf0-ea81-0a58-e22b-5697f6c319fc}]" --- PASS: TestRaft_StartStop (0.00s) === RUN TestRaft_AfterShutdown testLoggerAdapter verbose: server-eeb511d0-5d7f-63c2-affe-091a8d78d2d2: 2024-06-12T04:32:27.264+1400 [INFO] server-eeb511d0-5d7f-63c2-affe-091a8d78d2d2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eeb511d0-5d7f-63c2-affe-091a8d78d2d2 Address:eeb511d0-5d7f-63c2-affe-091a8d78d2d2}]" testing.go:149: server-eeb511d0-5d7f-63c2-affe-091a8d78d2d2: 2024-06-12T04:32:27.264+1400 [INFO] server-eeb511d0-5d7f-63c2-affe-091a8d78d2d2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eeb511d0-5d7f-63c2-affe-091a8d78d2d2 Address:eeb511d0-5d7f-63c2-affe-091a8d78d2d2}]" --- PASS: TestRaft_AfterShutdown (0.00s) === RUN TestRaft_LiveBootstrap testLoggerAdapter verbose: server-9959a38a-d08f-ce06-fc11-6245c7647547: 2024-06-12T04:32:27.266+1400 [INFO] server-9959a38a-d08f-ce06-fc11-6245c7647547: initial configuration: index=0 servers=[] testing.go:149: server-9959a38a-d08f-ce06-fc11-6245c7647547: 2024-06-12T04:32:27.266+1400 [INFO] server-9959a38a-d08f-ce06-fc11-6245c7647547: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-3cd6135d-68ac-7c7b-1f7f-06c68177f1a3: 2024-06-12T04:32:27.266+1400 [INFO] server-3cd6135d-68ac-7c7b-1f7f-06c68177f1a3: initial configuration: index=0 servers=[] testing.go:149: server-3cd6135d-68ac-7c7b-1f7f-06c68177f1a3: 2024-06-12T04:32:27.266+1400 [INFO] server-3cd6135d-68ac-7c7b-1f7f-06c68177f1a3: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-524b994e-d4c7-6910-bf4d-515087b45e85: 2024-06-12T04:32:27.266+1400 [INFO] server-524b994e-d4c7-6910-bf4d-515087b45e85: initial configuration: index=0 servers=[] testing.go:149: server-524b994e-d4c7-6910-bf4d-515087b45e85: 2024-06-12T04:32:27.266+1400 [INFO] server-524b994e-d4c7-6910-bf4d-515087b45e85: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-9959a38a-d08f-ce06-fc11-6245c7647547: 2024-06-12T04:32:27.266+1400 [INFO] server-9959a38a-d08f-ce06-fc11-6245c7647547: entering follower state: follower="Node at 9959a38a-d08f-ce06-fc11-6245c7647547 [Follower]" leader= testing.go:149: server-9959a38a-d08f-ce06-fc11-6245c7647547: 2024-06-12T04:32:27.266+1400 [INFO] server-9959a38a-d08f-ce06-fc11-6245c7647547: entering follower state: follower="Node at 9959a38a-d08f-ce06-fc11-6245c7647547 [Follower]" leader= testLoggerAdapter verbose: cluster: 2024-06-12T04:32:27.266+1400 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-06-12T04:32:27.266+1400 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-3cd6135d-68ac-7c7b-1f7f-06c68177f1a3: 2024-06-12T04:32:27.266+1400 [INFO] server-3cd6135d-68ac-7c7b-1f7f-06c68177f1a3: entering follower state: follower="Node at 3cd6135d-68ac-7c7b-1f7f-06c68177f1a3 [Follower]" leader= testing.go:149: server-3cd6135d-68ac-7c7b-1f7f-06c68177f1a3: 2024-06-12T04:32:27.266+1400 [INFO] server-3cd6135d-68ac-7c7b-1f7f-06c68177f1a3: entering follower state: follower="Node at 3cd6135d-68ac-7c7b-1f7f-06c68177f1a3 [Follower]" leader= testLoggerAdapter verbose: server-524b994e-d4c7-6910-bf4d-515087b45e85: 2024-06-12T04:32:27.266+1400 [INFO] server-524b994e-d4c7-6910-bf4d-515087b45e85: entering follower state: follower="Node at 524b994e-d4c7-6910-bf4d-515087b45e85 [Follower]" leader= testing.go:149: server-524b994e-d4c7-6910-bf4d-515087b45e85: 2024-06-12T04:32:27.266+1400 [INFO] server-524b994e-d4c7-6910-bf4d-515087b45e85: entering follower state: follower="Node at 524b994e-d4c7-6910-bf4d-515087b45e85 [Follower]" leader= testLoggerAdapter verbose: server-524b994e-d4c7-6910-bf4d-515087b45e85: 2024-06-12T04:32:27.320+1400 [WARN] server-524b994e-d4c7-6910-bf4d-515087b45e85: no known peers, aborting election testing.go:149: server-524b994e-d4c7-6910-bf4d-515087b45e85: 2024-06-12T04:32:27.320+1400 [WARN] server-524b994e-d4c7-6910-bf4d-515087b45e85: no known peers, aborting election testLoggerAdapter verbose: server-3cd6135d-68ac-7c7b-1f7f-06c68177f1a3: 2024-06-12T04:32:27.327+1400 [WARN] server-3cd6135d-68ac-7c7b-1f7f-06c68177f1a3: no known peers, aborting election testing.go:149: server-3cd6135d-68ac-7c7b-1f7f-06c68177f1a3: 2024-06-12T04:32:27.327+1400 [WARN] server-3cd6135d-68ac-7c7b-1f7f-06c68177f1a3: no known peers, aborting election testLoggerAdapter verbose: server-9959a38a-d08f-ce06-fc11-6245c7647547: 2024-06-12T04:32:27.329+1400 [WARN] server-9959a38a-d08f-ce06-fc11-6245c7647547: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-9959a38a-d08f-ce06-fc11-6245c7647547: 2024-06-12T04:32:27.329+1400 [WARN] server-9959a38a-d08f-ce06-fc11-6245c7647547: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-9959a38a-d08f-ce06-fc11-6245c7647547: 2024-06-12T04:32:27.329+1400 [INFO] server-9959a38a-d08f-ce06-fc11-6245c7647547: entering candidate state: node="Node at 9959a38a-d08f-ce06-fc11-6245c7647547 [Candidate]" term=2 testing.go:149: server-9959a38a-d08f-ce06-fc11-6245c7647547: 2024-06-12T04:32:27.329+1400 [INFO] server-9959a38a-d08f-ce06-fc11-6245c7647547: entering candidate state: node="Node at 9959a38a-d08f-ce06-fc11-6245c7647547 [Candidate]" term=2 testLoggerAdapter verbose: server-9959a38a-d08f-ce06-fc11-6245c7647547: 2024-06-12T04:32:27.329+1400 [INFO] server-9959a38a-d08f-ce06-fc11-6245c7647547: election won: tally=2 testing.go:149: server-9959a38a-d08f-ce06-fc11-6245c7647547: 2024-06-12T04:32:27.329+1400 [INFO] server-9959a38a-d08f-ce06-fc11-6245c7647547: election won: tally=2 testLoggerAdapter verbose: server-9959a38a-d08f-ce06-fc11-6245c7647547: 2024-06-12T04:32:27.329+1400 [INFO] server-9959a38a-d08f-ce06-fc11-6245c7647547: entering leader state: leader="Node at 9959a38a-d08f-ce06-fc11-6245c7647547 [Leader]" testing.go:149: server-9959a38a-d08f-ce06-fc11-6245c7647547: 2024-06-12T04:32:27.329+1400 [INFO] server-9959a38a-d08f-ce06-fc11-6245c7647547: entering leader state: leader="Node at 9959a38a-d08f-ce06-fc11-6245c7647547 [Leader]" testLoggerAdapter verbose: server-9959a38a-d08f-ce06-fc11-6245c7647547: 2024-06-12T04:32:27.329+1400 [INFO] server-9959a38a-d08f-ce06-fc11-6245c7647547: added peer, starting replication: peer=server-3cd6135d-68ac-7c7b-1f7f-06c68177f1a3 testing.go:149: server-9959a38a-d08f-ce06-fc11-6245c7647547: 2024-06-12T04:32:27.329+1400 [INFO] server-9959a38a-d08f-ce06-fc11-6245c7647547: added peer, starting replication: peer=server-3cd6135d-68ac-7c7b-1f7f-06c68177f1a3 testLoggerAdapter verbose: server-9959a38a-d08f-ce06-fc11-6245c7647547: 2024-06-12T04:32:27.329+1400 [INFO] server-9959a38a-d08f-ce06-fc11-6245c7647547: added peer, starting replication: peer=server-524b994e-d4c7-6910-bf4d-515087b45e85 testing.go:149: server-9959a38a-d08f-ce06-fc11-6245c7647547: 2024-06-12T04:32:27.329+1400 [INFO] server-9959a38a-d08f-ce06-fc11-6245c7647547: added peer, starting replication: peer=server-524b994e-d4c7-6910-bf4d-515087b45e85 testLoggerAdapter verbose: server-524b994e-d4c7-6910-bf4d-515087b45e85: 2024-06-12T04:32:27.329+1400 [WARN] server-524b994e-d4c7-6910-bf4d-515087b45e85: failed to get previous log: previous-index=1 last-index=0 error="log not found" testing.go:149: server-524b994e-d4c7-6910-bf4d-515087b45e85: 2024-06-12T04:32:27.329+1400 [WARN] server-524b994e-d4c7-6910-bf4d-515087b45e85: failed to get previous log: previous-index=1 last-index=0 error="log not found" testLoggerAdapter verbose: server-9959a38a-d08f-ce06-fc11-6245c7647547: 2024-06-12T04:32:27.329+1400 [WARN] server-9959a38a-d08f-ce06-fc11-6245c7647547: appendEntries rejected, sending older logs: peer="{Voter server-524b994e-d4c7-6910-bf4d-515087b45e85 524b994e-d4c7-6910-bf4d-515087b45e85}" next=1 testing.go:149: server-9959a38a-d08f-ce06-fc11-6245c7647547: 2024-06-12T04:32:27.329+1400 [WARN] server-9959a38a-d08f-ce06-fc11-6245c7647547: appendEntries rejected, sending older logs: peer="{Voter server-524b994e-d4c7-6910-bf4d-515087b45e85 524b994e-d4c7-6910-bf4d-515087b45e85}" next=1 testLoggerAdapter verbose: server-9959a38a-d08f-ce06-fc11-6245c7647547: 2024-06-12T04:32:27.329+1400 [INFO] server-9959a38a-d08f-ce06-fc11-6245c7647547: pipelining replication: peer="{Voter server-524b994e-d4c7-6910-bf4d-515087b45e85 524b994e-d4c7-6910-bf4d-515087b45e85}" testing.go:149: server-9959a38a-d08f-ce06-fc11-6245c7647547: 2024-06-12T04:32:27.329+1400 [INFO] server-9959a38a-d08f-ce06-fc11-6245c7647547: pipelining replication: peer="{Voter server-524b994e-d4c7-6910-bf4d-515087b45e85 524b994e-d4c7-6910-bf4d-515087b45e85}" testLoggerAdapter verbose: server-3cd6135d-68ac-7c7b-1f7f-06c68177f1a3: 2024-06-12T04:32:27.329+1400 [WARN] server-3cd6135d-68ac-7c7b-1f7f-06c68177f1a3: failed to get previous log: previous-index=1 last-index=0 error="log not found" testing.go:149: server-3cd6135d-68ac-7c7b-1f7f-06c68177f1a3: 2024-06-12T04:32:27.329+1400 [WARN] server-3cd6135d-68ac-7c7b-1f7f-06c68177f1a3: failed to get previous log: previous-index=1 last-index=0 error="log not found" testLoggerAdapter verbose: server-9959a38a-d08f-ce06-fc11-6245c7647547: 2024-06-12T04:32:27.329+1400 [WARN] server-9959a38a-d08f-ce06-fc11-6245c7647547: appendEntries rejected, sending older logs: peer="{Voter server-3cd6135d-68ac-7c7b-1f7f-06c68177f1a3 3cd6135d-68ac-7c7b-1f7f-06c68177f1a3}" next=1 testing.go:149: server-9959a38a-d08f-ce06-fc11-6245c7647547: 2024-06-12T04:32:27.329+1400 [WARN] server-9959a38a-d08f-ce06-fc11-6245c7647547: appendEntries rejected, sending older logs: peer="{Voter server-3cd6135d-68ac-7c7b-1f7f-06c68177f1a3 3cd6135d-68ac-7c7b-1f7f-06c68177f1a3}" next=1 testLoggerAdapter verbose: server-9959a38a-d08f-ce06-fc11-6245c7647547: 2024-06-12T04:32:27.330+1400 [INFO] server-9959a38a-d08f-ce06-fc11-6245c7647547: pipelining replication: peer="{Voter server-3cd6135d-68ac-7c7b-1f7f-06c68177f1a3 3cd6135d-68ac-7c7b-1f7f-06c68177f1a3}" testing.go:149: server-9959a38a-d08f-ce06-fc11-6245c7647547: 2024-06-12T04:32:27.330+1400 [INFO] server-9959a38a-d08f-ce06-fc11-6245c7647547: pipelining replication: peer="{Voter server-3cd6135d-68ac-7c7b-1f7f-06c68177f1a3 3cd6135d-68ac-7c7b-1f7f-06c68177f1a3}" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:27.436+1400 [INFO] stable state for Follower reached at 2024-06-12 04:32:27.330109675 +1400 +14 m=+4.237628254 (2 nodes), 63.525564ms from start of poll, 64.099384ms from cluster start. Timeout at 2024-06-12 04:32:27.436099279 +1400 +14 m=+4.343617898, 105.989644ms after stability testing.go:149: cluster: 2024-06-12T04:32:27.436+1400 [INFO] stable state for Follower reached at 2024-06-12 04:32:27.330109675 +1400 +14 m=+4.237628254 (2 nodes), 63.525564ms from start of poll, 64.099384ms from cluster start. Timeout at 2024-06-12 04:32:27.436099279 +1400 +14 m=+4.343617898, 105.989644ms after stability testLoggerAdapter verbose: cluster: 2024-06-12T04:32:27.436+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-12T04:32:27.436+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-06-12T04:32:27.541+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:27.436316431 +1400 +14 m=+4.343835010 (1 nodes), 1.48µs from start of poll, 170.30614ms from cluster start. Timeout at 2024-06-12 04:32:27.541465985 +1400 +14 m=+4.448984564, 105.149554ms after stability testing.go:149: cluster: 2024-06-12T04:32:27.541+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:27.436316431 +1400 +14 m=+4.343835010 (1 nodes), 1.48µs from start of poll, 170.30614ms from cluster start. Timeout at 2024-06-12 04:32:27.541465985 +1400 +14 m=+4.448984564, 105.149554ms after stability testLoggerAdapter verbose: server-9959a38a-d08f-ce06-fc11-6245c7647547: 2024-06-12T04:32:27.553+1400 [INFO] server-9959a38a-d08f-ce06-fc11-6245c7647547: aborting pipeline replication: peer="{Voter server-524b994e-d4c7-6910-bf4d-515087b45e85 524b994e-d4c7-6910-bf4d-515087b45e85}" testing.go:149: server-9959a38a-d08f-ce06-fc11-6245c7647547: 2024-06-12T04:32:27.553+1400 [INFO] server-9959a38a-d08f-ce06-fc11-6245c7647547: aborting pipeline replication: peer="{Voter server-524b994e-d4c7-6910-bf4d-515087b45e85 524b994e-d4c7-6910-bf4d-515087b45e85}" testLoggerAdapter verbose: server-9959a38a-d08f-ce06-fc11-6245c7647547: 2024-06-12T04:32:27.553+1400 [INFO] server-9959a38a-d08f-ce06-fc11-6245c7647547: aborting pipeline replication: peer="{Voter server-3cd6135d-68ac-7c7b-1f7f-06c68177f1a3 3cd6135d-68ac-7c7b-1f7f-06c68177f1a3}" testing.go:149: server-9959a38a-d08f-ce06-fc11-6245c7647547: 2024-06-12T04:32:27.553+1400 [INFO] server-9959a38a-d08f-ce06-fc11-6245c7647547: aborting pipeline replication: peer="{Voter server-3cd6135d-68ac-7c7b-1f7f-06c68177f1a3 3cd6135d-68ac-7c7b-1f7f-06c68177f1a3}" --- PASS: TestRaft_LiveBootstrap (0.29s) === RUN TestRaft_RecoverCluster_NoState testLoggerAdapter verbose: server-3cca2780-e302-a7ca-0212-14e74e467f4e: 2024-06-12T04:32:27.554+1400 [INFO] server-3cca2780-e302-a7ca-0212-14e74e467f4e: initial configuration: index=0 servers=[] testing.go:149: server-3cca2780-e302-a7ca-0212-14e74e467f4e: 2024-06-12T04:32:27.554+1400 [INFO] server-3cca2780-e302-a7ca-0212-14e74e467f4e: initial configuration: index=0 servers=[] --- PASS: TestRaft_RecoverCluster_NoState (0.00s) === RUN TestRaft_RecoverCluster raft_test.go:119: Disabled unreliable test --- SKIP: TestRaft_RecoverCluster (0.00s) === RUN TestRaft_HasExistingState testLoggerAdapter verbose: server-458cf6b8-fd2f-3e20-0e83-adcf4b9f63b8: 2024-06-12T04:32:27.556+1400 [INFO] server-458cf6b8-fd2f-3e20-0e83-adcf4b9f63b8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-458cf6b8-fd2f-3e20-0e83-adcf4b9f63b8 Address:458cf6b8-fd2f-3e20-0e83-adcf4b9f63b8} {Suffrage:Voter ID:server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09 Address:edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09}]" testing.go:149: server-458cf6b8-fd2f-3e20-0e83-adcf4b9f63b8: 2024-06-12T04:32:27.556+1400 [INFO] server-458cf6b8-fd2f-3e20-0e83-adcf4b9f63b8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-458cf6b8-fd2f-3e20-0e83-adcf4b9f63b8 Address:458cf6b8-fd2f-3e20-0e83-adcf4b9f63b8} {Suffrage:Voter ID:server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09 Address:edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09}]" testLoggerAdapter verbose: server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: 2024-06-12T04:32:27.556+1400 [INFO] server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-458cf6b8-fd2f-3e20-0e83-adcf4b9f63b8 Address:458cf6b8-fd2f-3e20-0e83-adcf4b9f63b8} {Suffrage:Voter ID:server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09 Address:edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09}]" testing.go:149: server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: 2024-06-12T04:32:27.556+1400 [INFO] server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-458cf6b8-fd2f-3e20-0e83-adcf4b9f63b8 Address:458cf6b8-fd2f-3e20-0e83-adcf4b9f63b8} {Suffrage:Voter ID:server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09 Address:edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09}]" testLoggerAdapter verbose: server-a478c75e-aa84-b1b6-2c10-4f09a6bcd070: 2024-06-12T04:32:27.556+1400 [INFO] server-a478c75e-aa84-b1b6-2c10-4f09a6bcd070: initial configuration: index=0 servers=[] testing.go:149: server-a478c75e-aa84-b1b6-2c10-4f09a6bcd070: 2024-06-12T04:32:27.556+1400 [INFO] server-a478c75e-aa84-b1b6-2c10-4f09a6bcd070: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2024-06-12T04:32:27.557+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-12T04:32:27.557+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-458cf6b8-fd2f-3e20-0e83-adcf4b9f63b8: 2024-06-12T04:32:27.557+1400 [INFO] server-458cf6b8-fd2f-3e20-0e83-adcf4b9f63b8: entering follower state: follower="Node at 458cf6b8-fd2f-3e20-0e83-adcf4b9f63b8 [Follower]" leader= testing.go:149: server-458cf6b8-fd2f-3e20-0e83-adcf4b9f63b8: 2024-06-12T04:32:27.557+1400 [INFO] server-458cf6b8-fd2f-3e20-0e83-adcf4b9f63b8: entering follower state: follower="Node at 458cf6b8-fd2f-3e20-0e83-adcf4b9f63b8 [Follower]" leader= testLoggerAdapter verbose: server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: 2024-06-12T04:32:27.557+1400 [INFO] server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: entering follower state: follower="Node at edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09 [Follower]" leader= testing.go:149: server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: 2024-06-12T04:32:27.557+1400 [INFO] server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: entering follower state: follower="Node at edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09 [Follower]" leader= testLoggerAdapter verbose: server-a478c75e-aa84-b1b6-2c10-4f09a6bcd070: 2024-06-12T04:32:27.557+1400 [INFO] server-a478c75e-aa84-b1b6-2c10-4f09a6bcd070: entering follower state: follower="Node at a478c75e-aa84-b1b6-2c10-4f09a6bcd070 [Follower]" leader= testing.go:149: server-a478c75e-aa84-b1b6-2c10-4f09a6bcd070: 2024-06-12T04:32:27.557+1400 [INFO] server-a478c75e-aa84-b1b6-2c10-4f09a6bcd070: entering follower state: follower="Node at a478c75e-aa84-b1b6-2c10-4f09a6bcd070 [Follower]" leader= testLoggerAdapter verbose: server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: 2024-06-12T04:32:27.624+1400 [WARN] server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: 2024-06-12T04:32:27.624+1400 [WARN] server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: 2024-06-12T04:32:27.624+1400 [INFO] server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: entering candidate state: node="Node at edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09 [Candidate]" term=2 testing.go:149: server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: 2024-06-12T04:32:27.624+1400 [INFO] server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: entering candidate state: node="Node at edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09 [Candidate]" term=2 testLoggerAdapter verbose: server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: 2024-06-12T04:32:27.624+1400 [INFO] server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: election won: tally=2 testing.go:149: server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: 2024-06-12T04:32:27.624+1400 [INFO] server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: election won: tally=2 testLoggerAdapter verbose: server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: 2024-06-12T04:32:27.624+1400 [INFO] server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: entering leader state: leader="Node at edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09 [Leader]" testing.go:149: server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: 2024-06-12T04:32:27.624+1400 [INFO] server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: entering leader state: leader="Node at edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09 [Leader]" testLoggerAdapter verbose: server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: 2024-06-12T04:32:27.624+1400 [INFO] server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: added peer, starting replication: peer=server-458cf6b8-fd2f-3e20-0e83-adcf4b9f63b8 testing.go:149: server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: 2024-06-12T04:32:27.624+1400 [INFO] server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: added peer, starting replication: peer=server-458cf6b8-fd2f-3e20-0e83-adcf4b9f63b8 testLoggerAdapter verbose: server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: 2024-06-12T04:32:27.624+1400 [INFO] server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: pipelining replication: peer="{Voter server-458cf6b8-fd2f-3e20-0e83-adcf4b9f63b8 458cf6b8-fd2f-3e20-0e83-adcf4b9f63b8}" testing.go:149: server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: 2024-06-12T04:32:27.624+1400 [INFO] server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: pipelining replication: peer="{Voter server-458cf6b8-fd2f-3e20-0e83-adcf4b9f63b8 458cf6b8-fd2f-3e20-0e83-adcf4b9f63b8}" testLoggerAdapter verbose: server-a478c75e-aa84-b1b6-2c10-4f09a6bcd070: 2024-06-12T04:32:27.639+1400 [WARN] server-a478c75e-aa84-b1b6-2c10-4f09a6bcd070: no known peers, aborting election testing.go:149: server-a478c75e-aa84-b1b6-2c10-4f09a6bcd070: 2024-06-12T04:32:27.639+1400 [WARN] server-a478c75e-aa84-b1b6-2c10-4f09a6bcd070: no known peers, aborting election testLoggerAdapter verbose: cluster: 2024-06-12T04:32:27.730+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:27.624694284 +1400 +14 m=+4.532212863 (1 nodes), 67.598459ms from start of poll, 68.553985ms from cluster start. Timeout at 2024-06-12 04:32:27.729974352 +1400 +14 m=+4.637492931, 105.280068ms after stability testing.go:149: cluster: 2024-06-12T04:32:27.730+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:27.624694284 +1400 +14 m=+4.532212863 (1 nodes), 67.598459ms from start of poll, 68.553985ms from cluster start. Timeout at 2024-06-12 04:32:27.729974352 +1400 +14 m=+4.637492931, 105.280068ms after stability testLoggerAdapter verbose: server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: 2024-06-12T04:32:27.730+1400 [INFO] server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: updating configuration: command=AddStaging server-id=server-a478c75e-aa84-b1b6-2c10-4f09a6bcd070 server-addr=a478c75e-aa84-b1b6-2c10-4f09a6bcd070 servers="[{Suffrage:Voter ID:server-458cf6b8-fd2f-3e20-0e83-adcf4b9f63b8 Address:458cf6b8-fd2f-3e20-0e83-adcf4b9f63b8} {Suffrage:Voter ID:server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09 Address:edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09} {Suffrage:Voter ID:server-a478c75e-aa84-b1b6-2c10-4f09a6bcd070 Address:a478c75e-aa84-b1b6-2c10-4f09a6bcd070}]" testing.go:149: server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: 2024-06-12T04:32:27.730+1400 [INFO] server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: updating configuration: command=AddStaging server-id=server-a478c75e-aa84-b1b6-2c10-4f09a6bcd070 server-addr=a478c75e-aa84-b1b6-2c10-4f09a6bcd070 servers="[{Suffrage:Voter ID:server-458cf6b8-fd2f-3e20-0e83-adcf4b9f63b8 Address:458cf6b8-fd2f-3e20-0e83-adcf4b9f63b8} {Suffrage:Voter ID:server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09 Address:edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09} {Suffrage:Voter ID:server-a478c75e-aa84-b1b6-2c10-4f09a6bcd070 Address:a478c75e-aa84-b1b6-2c10-4f09a6bcd070}]" testLoggerAdapter verbose: server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: 2024-06-12T04:32:27.730+1400 [INFO] server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: added peer, starting replication: peer=server-a478c75e-aa84-b1b6-2c10-4f09a6bcd070 testing.go:149: server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: 2024-06-12T04:32:27.730+1400 [INFO] server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: added peer, starting replication: peer=server-a478c75e-aa84-b1b6-2c10-4f09a6bcd070 testLoggerAdapter verbose: server-a478c75e-aa84-b1b6-2c10-4f09a6bcd070: 2024-06-12T04:32:27.730+1400 [WARN] server-a478c75e-aa84-b1b6-2c10-4f09a6bcd070: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: server-a478c75e-aa84-b1b6-2c10-4f09a6bcd070: 2024-06-12T04:32:27.730+1400 [WARN] server-a478c75e-aa84-b1b6-2c10-4f09a6bcd070: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: 2024-06-12T04:32:27.730+1400 [WARN] server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: appendEntries rejected, sending older logs: peer="{Voter server-a478c75e-aa84-b1b6-2c10-4f09a6bcd070 a478c75e-aa84-b1b6-2c10-4f09a6bcd070}" next=1 testing.go:149: server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: 2024-06-12T04:32:27.730+1400 [WARN] server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: appendEntries rejected, sending older logs: peer="{Voter server-a478c75e-aa84-b1b6-2c10-4f09a6bcd070 a478c75e-aa84-b1b6-2c10-4f09a6bcd070}" next=1 testLoggerAdapter verbose: server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: 2024-06-12T04:32:27.731+1400 [INFO] server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: pipelining replication: peer="{Voter server-a478c75e-aa84-b1b6-2c10-4f09a6bcd070 a478c75e-aa84-b1b6-2c10-4f09a6bcd070}" testing.go:149: server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: 2024-06-12T04:32:27.731+1400 [INFO] server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: pipelining replication: peer="{Voter server-a478c75e-aa84-b1b6-2c10-4f09a6bcd070 a478c75e-aa84-b1b6-2c10-4f09a6bcd070}" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:27.731+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-12T04:32:27.731+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-06-12T04:32:27.836+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:27.731392342 +1400 +14 m=+4.638910921 (1 nodes), 1.32µs from start of poll, 175.252043ms from cluster start. Timeout at 2024-06-12 04:32:27.836715969 +1400 +14 m=+4.744234548, 105.323627ms after stability testing.go:149: cluster: 2024-06-12T04:32:27.836+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:27.731392342 +1400 +14 m=+4.638910921 (1 nodes), 1.32µs from start of poll, 175.252043ms from cluster start. Timeout at 2024-06-12 04:32:27.836715969 +1400 +14 m=+4.744234548, 105.323627ms after stability testLoggerAdapter verbose: server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: 2024-06-12T04:32:27.836+1400 [INFO] server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: aborting pipeline replication: peer="{Voter server-a478c75e-aa84-b1b6-2c10-4f09a6bcd070 a478c75e-aa84-b1b6-2c10-4f09a6bcd070}" testing.go:149: server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: 2024-06-12T04:32:27.836+1400 [INFO] server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: aborting pipeline replication: peer="{Voter server-a478c75e-aa84-b1b6-2c10-4f09a6bcd070 a478c75e-aa84-b1b6-2c10-4f09a6bcd070}" testLoggerAdapter verbose: server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: 2024-06-12T04:32:27.836+1400 [INFO] server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: aborting pipeline replication: peer="{Voter server-458cf6b8-fd2f-3e20-0e83-adcf4b9f63b8 458cf6b8-fd2f-3e20-0e83-adcf4b9f63b8}" testing.go:149: server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: 2024-06-12T04:32:27.836+1400 [INFO] server-edc3da8f-77a9-1a3d-b6e9-cf6da7a7ac09: aborting pipeline replication: peer="{Voter server-458cf6b8-fd2f-3e20-0e83-adcf4b9f63b8 458cf6b8-fd2f-3e20-0e83-adcf4b9f63b8}" --- PASS: TestRaft_HasExistingState (0.28s) === RUN TestRaft_SingleNode testLoggerAdapter verbose: server-0704f701-febd-f2a2-f84b-7c6907c370da: 2024-06-12T04:32:27.838+1400 [INFO] server-0704f701-febd-f2a2-f84b-7c6907c370da: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0704f701-febd-f2a2-f84b-7c6907c370da Address:0704f701-febd-f2a2-f84b-7c6907c370da}]" testing.go:149: server-0704f701-febd-f2a2-f84b-7c6907c370da: 2024-06-12T04:32:27.838+1400 [INFO] server-0704f701-febd-f2a2-f84b-7c6907c370da: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0704f701-febd-f2a2-f84b-7c6907c370da Address:0704f701-febd-f2a2-f84b-7c6907c370da}]" testLoggerAdapter verbose: server-0704f701-febd-f2a2-f84b-7c6907c370da: 2024-06-12T04:32:27.838+1400 [INFO] server-0704f701-febd-f2a2-f84b-7c6907c370da: entering follower state: follower="Node at 0704f701-febd-f2a2-f84b-7c6907c370da [Follower]" leader= testing.go:149: server-0704f701-febd-f2a2-f84b-7c6907c370da: 2024-06-12T04:32:27.838+1400 [INFO] server-0704f701-febd-f2a2-f84b-7c6907c370da: entering follower state: follower="Node at 0704f701-febd-f2a2-f84b-7c6907c370da [Follower]" leader= testLoggerAdapter verbose: server-0704f701-febd-f2a2-f84b-7c6907c370da: 2024-06-12T04:32:27.903+1400 [WARN] server-0704f701-febd-f2a2-f84b-7c6907c370da: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-0704f701-febd-f2a2-f84b-7c6907c370da: 2024-06-12T04:32:27.903+1400 [WARN] server-0704f701-febd-f2a2-f84b-7c6907c370da: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-0704f701-febd-f2a2-f84b-7c6907c370da: 2024-06-12T04:32:27.903+1400 [INFO] server-0704f701-febd-f2a2-f84b-7c6907c370da: entering candidate state: node="Node at 0704f701-febd-f2a2-f84b-7c6907c370da [Candidate]" term=2 testing.go:149: server-0704f701-febd-f2a2-f84b-7c6907c370da: 2024-06-12T04:32:27.903+1400 [INFO] server-0704f701-febd-f2a2-f84b-7c6907c370da: entering candidate state: node="Node at 0704f701-febd-f2a2-f84b-7c6907c370da [Candidate]" term=2 testLoggerAdapter verbose: server-0704f701-febd-f2a2-f84b-7c6907c370da: 2024-06-12T04:32:27.904+1400 [INFO] server-0704f701-febd-f2a2-f84b-7c6907c370da: election won: tally=1 testing.go:149: server-0704f701-febd-f2a2-f84b-7c6907c370da: 2024-06-12T04:32:27.904+1400 [INFO] server-0704f701-febd-f2a2-f84b-7c6907c370da: election won: tally=1 testLoggerAdapter verbose: server-0704f701-febd-f2a2-f84b-7c6907c370da: 2024-06-12T04:32:27.904+1400 [INFO] server-0704f701-febd-f2a2-f84b-7c6907c370da: entering leader state: leader="Node at 0704f701-febd-f2a2-f84b-7c6907c370da [Leader]" testing.go:149: server-0704f701-febd-f2a2-f84b-7c6907c370da: 2024-06-12T04:32:27.904+1400 [INFO] server-0704f701-febd-f2a2-f84b-7c6907c370da: entering leader state: leader="Node at 0704f701-febd-f2a2-f84b-7c6907c370da [Leader]" --- PASS: TestRaft_SingleNode (0.07s) === RUN TestRaft_TripleNode testLoggerAdapter verbose: server-8a314442-5589-2def-73fa-76ef09960360: 2024-06-12T04:32:27.905+1400 [INFO] server-8a314442-5589-2def-73fa-76ef09960360: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a314442-5589-2def-73fa-76ef09960360 Address:8a314442-5589-2def-73fa-76ef09960360} {Suffrage:Voter ID:server-b0aadc71-d749-6d6c-4fb2-82d3df3ba0f3 Address:b0aadc71-d749-6d6c-4fb2-82d3df3ba0f3} {Suffrage:Voter ID:server-021c45a8-3e1d-5a00-fe44-4e8eb773ffb6 Address:021c45a8-3e1d-5a00-fe44-4e8eb773ffb6}]" testing.go:149: server-8a314442-5589-2def-73fa-76ef09960360: 2024-06-12T04:32:27.905+1400 [INFO] server-8a314442-5589-2def-73fa-76ef09960360: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a314442-5589-2def-73fa-76ef09960360 Address:8a314442-5589-2def-73fa-76ef09960360} {Suffrage:Voter ID:server-b0aadc71-d749-6d6c-4fb2-82d3df3ba0f3 Address:b0aadc71-d749-6d6c-4fb2-82d3df3ba0f3} {Suffrage:Voter ID:server-021c45a8-3e1d-5a00-fe44-4e8eb773ffb6 Address:021c45a8-3e1d-5a00-fe44-4e8eb773ffb6}]" testLoggerAdapter verbose: server-b0aadc71-d749-6d6c-4fb2-82d3df3ba0f3: 2024-06-12T04:32:27.905+1400 [INFO] server-b0aadc71-d749-6d6c-4fb2-82d3df3ba0f3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a314442-5589-2def-73fa-76ef09960360 Address:8a314442-5589-2def-73fa-76ef09960360} {Suffrage:Voter ID:server-b0aadc71-d749-6d6c-4fb2-82d3df3ba0f3 Address:b0aadc71-d749-6d6c-4fb2-82d3df3ba0f3} {Suffrage:Voter ID:server-021c45a8-3e1d-5a00-fe44-4e8eb773ffb6 Address:021c45a8-3e1d-5a00-fe44-4e8eb773ffb6}]" testing.go:149: server-b0aadc71-d749-6d6c-4fb2-82d3df3ba0f3: 2024-06-12T04:32:27.905+1400 [INFO] server-b0aadc71-d749-6d6c-4fb2-82d3df3ba0f3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a314442-5589-2def-73fa-76ef09960360 Address:8a314442-5589-2def-73fa-76ef09960360} {Suffrage:Voter ID:server-b0aadc71-d749-6d6c-4fb2-82d3df3ba0f3 Address:b0aadc71-d749-6d6c-4fb2-82d3df3ba0f3} {Suffrage:Voter ID:server-021c45a8-3e1d-5a00-fe44-4e8eb773ffb6 Address:021c45a8-3e1d-5a00-fe44-4e8eb773ffb6}]" testLoggerAdapter verbose: server-021c45a8-3e1d-5a00-fe44-4e8eb773ffb6: 2024-06-12T04:32:27.906+1400 [INFO] server-021c45a8-3e1d-5a00-fe44-4e8eb773ffb6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a314442-5589-2def-73fa-76ef09960360 Address:8a314442-5589-2def-73fa-76ef09960360} {Suffrage:Voter ID:server-b0aadc71-d749-6d6c-4fb2-82d3df3ba0f3 Address:b0aadc71-d749-6d6c-4fb2-82d3df3ba0f3} {Suffrage:Voter ID:server-021c45a8-3e1d-5a00-fe44-4e8eb773ffb6 Address:021c45a8-3e1d-5a00-fe44-4e8eb773ffb6}]" testing.go:149: server-021c45a8-3e1d-5a00-fe44-4e8eb773ffb6: 2024-06-12T04:32:27.906+1400 [INFO] server-021c45a8-3e1d-5a00-fe44-4e8eb773ffb6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a314442-5589-2def-73fa-76ef09960360 Address:8a314442-5589-2def-73fa-76ef09960360} {Suffrage:Voter ID:server-b0aadc71-d749-6d6c-4fb2-82d3df3ba0f3 Address:b0aadc71-d749-6d6c-4fb2-82d3df3ba0f3} {Suffrage:Voter ID:server-021c45a8-3e1d-5a00-fe44-4e8eb773ffb6 Address:021c45a8-3e1d-5a00-fe44-4e8eb773ffb6}]" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:27.906+1400 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-06-12T04:32:27.906+1400 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-8a314442-5589-2def-73fa-76ef09960360: 2024-06-12T04:32:27.906+1400 [INFO] server-8a314442-5589-2def-73fa-76ef09960360: entering follower state: follower="Node at 8a314442-5589-2def-73fa-76ef09960360 [Follower]" leader= testing.go:149: server-8a314442-5589-2def-73fa-76ef09960360: 2024-06-12T04:32:27.906+1400 [INFO] server-8a314442-5589-2def-73fa-76ef09960360: entering follower state: follower="Node at 8a314442-5589-2def-73fa-76ef09960360 [Follower]" leader= testLoggerAdapter verbose: server-b0aadc71-d749-6d6c-4fb2-82d3df3ba0f3: 2024-06-12T04:32:27.906+1400 [INFO] server-b0aadc71-d749-6d6c-4fb2-82d3df3ba0f3: entering follower state: follower="Node at b0aadc71-d749-6d6c-4fb2-82d3df3ba0f3 [Follower]" leader= testing.go:149: server-b0aadc71-d749-6d6c-4fb2-82d3df3ba0f3: 2024-06-12T04:32:27.906+1400 [INFO] server-b0aadc71-d749-6d6c-4fb2-82d3df3ba0f3: entering follower state: follower="Node at b0aadc71-d749-6d6c-4fb2-82d3df3ba0f3 [Follower]" leader= testLoggerAdapter verbose: server-021c45a8-3e1d-5a00-fe44-4e8eb773ffb6: 2024-06-12T04:32:27.906+1400 [INFO] server-021c45a8-3e1d-5a00-fe44-4e8eb773ffb6: entering follower state: follower="Node at 021c45a8-3e1d-5a00-fe44-4e8eb773ffb6 [Follower]" leader= testing.go:149: server-021c45a8-3e1d-5a00-fe44-4e8eb773ffb6: 2024-06-12T04:32:27.906+1400 [INFO] server-021c45a8-3e1d-5a00-fe44-4e8eb773ffb6: entering follower state: follower="Node at 021c45a8-3e1d-5a00-fe44-4e8eb773ffb6 [Follower]" leader= testLoggerAdapter verbose: server-8a314442-5589-2def-73fa-76ef09960360: 2024-06-12T04:32:27.966+1400 [WARN] server-8a314442-5589-2def-73fa-76ef09960360: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-8a314442-5589-2def-73fa-76ef09960360: 2024-06-12T04:32:27.966+1400 [WARN] server-8a314442-5589-2def-73fa-76ef09960360: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-8a314442-5589-2def-73fa-76ef09960360: 2024-06-12T04:32:27.966+1400 [INFO] server-8a314442-5589-2def-73fa-76ef09960360: entering candidate state: node="Node at 8a314442-5589-2def-73fa-76ef09960360 [Candidate]" term=2 testing.go:149: server-8a314442-5589-2def-73fa-76ef09960360: 2024-06-12T04:32:27.966+1400 [INFO] server-8a314442-5589-2def-73fa-76ef09960360: entering candidate state: node="Node at 8a314442-5589-2def-73fa-76ef09960360 [Candidate]" term=2 testLoggerAdapter verbose: server-8a314442-5589-2def-73fa-76ef09960360: 2024-06-12T04:32:27.966+1400 [INFO] server-8a314442-5589-2def-73fa-76ef09960360: election won: tally=2 testing.go:149: server-8a314442-5589-2def-73fa-76ef09960360: 2024-06-12T04:32:27.966+1400 [INFO] server-8a314442-5589-2def-73fa-76ef09960360: election won: tally=2 testLoggerAdapter verbose: server-8a314442-5589-2def-73fa-76ef09960360: 2024-06-12T04:32:27.966+1400 [INFO] server-8a314442-5589-2def-73fa-76ef09960360: entering leader state: leader="Node at 8a314442-5589-2def-73fa-76ef09960360 [Leader]" testing.go:149: server-8a314442-5589-2def-73fa-76ef09960360: 2024-06-12T04:32:27.966+1400 [INFO] server-8a314442-5589-2def-73fa-76ef09960360: entering leader state: leader="Node at 8a314442-5589-2def-73fa-76ef09960360 [Leader]" testLoggerAdapter verbose: server-8a314442-5589-2def-73fa-76ef09960360: 2024-06-12T04:32:27.966+1400 [INFO] server-8a314442-5589-2def-73fa-76ef09960360: added peer, starting replication: peer=server-b0aadc71-d749-6d6c-4fb2-82d3df3ba0f3 testing.go:149: server-8a314442-5589-2def-73fa-76ef09960360: 2024-06-12T04:32:27.966+1400 [INFO] server-8a314442-5589-2def-73fa-76ef09960360: added peer, starting replication: peer=server-b0aadc71-d749-6d6c-4fb2-82d3df3ba0f3 testLoggerAdapter verbose: server-8a314442-5589-2def-73fa-76ef09960360: 2024-06-12T04:32:27.966+1400 [INFO] server-8a314442-5589-2def-73fa-76ef09960360: added peer, starting replication: peer=server-021c45a8-3e1d-5a00-fe44-4e8eb773ffb6 testing.go:149: server-8a314442-5589-2def-73fa-76ef09960360: 2024-06-12T04:32:27.966+1400 [INFO] server-8a314442-5589-2def-73fa-76ef09960360: added peer, starting replication: peer=server-021c45a8-3e1d-5a00-fe44-4e8eb773ffb6 testLoggerAdapter verbose: server-8a314442-5589-2def-73fa-76ef09960360: 2024-06-12T04:32:27.966+1400 [INFO] server-8a314442-5589-2def-73fa-76ef09960360: pipelining replication: peer="{Voter server-021c45a8-3e1d-5a00-fe44-4e8eb773ffb6 021c45a8-3e1d-5a00-fe44-4e8eb773ffb6}" testing.go:149: server-8a314442-5589-2def-73fa-76ef09960360: 2024-06-12T04:32:27.966+1400 [INFO] server-8a314442-5589-2def-73fa-76ef09960360: pipelining replication: peer="{Voter server-021c45a8-3e1d-5a00-fe44-4e8eb773ffb6 021c45a8-3e1d-5a00-fe44-4e8eb773ffb6}" testLoggerAdapter verbose: server-8a314442-5589-2def-73fa-76ef09960360: 2024-06-12T04:32:27.966+1400 [INFO] server-8a314442-5589-2def-73fa-76ef09960360: pipelining replication: peer="{Voter server-b0aadc71-d749-6d6c-4fb2-82d3df3ba0f3 b0aadc71-d749-6d6c-4fb2-82d3df3ba0f3}" testing.go:149: server-8a314442-5589-2def-73fa-76ef09960360: 2024-06-12T04:32:27.966+1400 [INFO] server-8a314442-5589-2def-73fa-76ef09960360: pipelining replication: peer="{Voter server-b0aadc71-d749-6d6c-4fb2-82d3df3ba0f3 b0aadc71-d749-6d6c-4fb2-82d3df3ba0f3}" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:28.072+1400 [INFO] stable state for Follower reached at 2024-06-12 04:32:27.966869347 +1400 +14 m=+4.874387926 (2 nodes), 60.618269ms from start of poll, 61.299644ms from cluster start. Timeout at 2024-06-12 04:32:28.072165216 +1400 +14 m=+4.979683835, 105.295909ms after stability testing.go:149: cluster: 2024-06-12T04:32:28.072+1400 [INFO] stable state for Follower reached at 2024-06-12 04:32:27.966869347 +1400 +14 m=+4.874387926 (2 nodes), 60.618269ms from start of poll, 61.299644ms from cluster start. Timeout at 2024-06-12 04:32:28.072165216 +1400 +14 m=+4.979683835, 105.295909ms after stability testLoggerAdapter verbose: cluster: 2024-06-12T04:32:28.072+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-12T04:32:28.072+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-06-12T04:32:28.177+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:28.072362169 +1400 +14 m=+4.979880748 (1 nodes), 1.52µs from start of poll, 166.792466ms from cluster start. Timeout at 2024-06-12 04:32:28.177877549 +1400 +14 m=+5.085396128, 105.51538ms after stability testing.go:149: cluster: 2024-06-12T04:32:28.177+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:28.072362169 +1400 +14 m=+4.979880748 (1 nodes), 1.52µs from start of poll, 166.792466ms from cluster start. Timeout at 2024-06-12 04:32:28.177877549 +1400 +14 m=+5.085396128, 105.51538ms after stability testLoggerAdapter verbose: server-8a314442-5589-2def-73fa-76ef09960360: 2024-06-12T04:32:28.189+1400 [INFO] server-8a314442-5589-2def-73fa-76ef09960360: aborting pipeline replication: peer="{Voter server-021c45a8-3e1d-5a00-fe44-4e8eb773ffb6 021c45a8-3e1d-5a00-fe44-4e8eb773ffb6}" testing.go:149: server-8a314442-5589-2def-73fa-76ef09960360: 2024-06-12T04:32:28.189+1400 [INFO] server-8a314442-5589-2def-73fa-76ef09960360: aborting pipeline replication: peer="{Voter server-021c45a8-3e1d-5a00-fe44-4e8eb773ffb6 021c45a8-3e1d-5a00-fe44-4e8eb773ffb6}" testLoggerAdapter verbose: server-8a314442-5589-2def-73fa-76ef09960360: 2024-06-12T04:32:28.189+1400 [INFO] server-8a314442-5589-2def-73fa-76ef09960360: aborting pipeline replication: peer="{Voter server-b0aadc71-d749-6d6c-4fb2-82d3df3ba0f3 b0aadc71-d749-6d6c-4fb2-82d3df3ba0f3}" testing.go:149: server-8a314442-5589-2def-73fa-76ef09960360: 2024-06-12T04:32:28.189+1400 [INFO] server-8a314442-5589-2def-73fa-76ef09960360: aborting pipeline replication: peer="{Voter server-b0aadc71-d749-6d6c-4fb2-82d3df3ba0f3 b0aadc71-d749-6d6c-4fb2-82d3df3ba0f3}" --- PASS: TestRaft_TripleNode (0.29s) === RUN TestRaft_LeaderFail testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.192+1400 [INFO] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b67a0c6-3830-c6e6-3008-ce525ea6796b Address:1b67a0c6-3830-c6e6-3008-ce525ea6796b} {Suffrage:Voter ID:server-b98bac58-4a84-edd5-1003-a3d0df30dc2c Address:b98bac58-4a84-edd5-1003-a3d0df30dc2c} {Suffrage:Voter ID:server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6 Address:e07a9304-4edc-3a14-d254-66c4a5bfcaf6}]" testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.192+1400 [INFO] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b67a0c6-3830-c6e6-3008-ce525ea6796b Address:1b67a0c6-3830-c6e6-3008-ce525ea6796b} {Suffrage:Voter ID:server-b98bac58-4a84-edd5-1003-a3d0df30dc2c Address:b98bac58-4a84-edd5-1003-a3d0df30dc2c} {Suffrage:Voter ID:server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6 Address:e07a9304-4edc-3a14-d254-66c4a5bfcaf6}]" testLoggerAdapter verbose: server-b98bac58-4a84-edd5-1003-a3d0df30dc2c: 2024-06-12T04:32:28.192+1400 [INFO] server-b98bac58-4a84-edd5-1003-a3d0df30dc2c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b67a0c6-3830-c6e6-3008-ce525ea6796b Address:1b67a0c6-3830-c6e6-3008-ce525ea6796b} {Suffrage:Voter ID:server-b98bac58-4a84-edd5-1003-a3d0df30dc2c Address:b98bac58-4a84-edd5-1003-a3d0df30dc2c} {Suffrage:Voter ID:server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6 Address:e07a9304-4edc-3a14-d254-66c4a5bfcaf6}]" testing.go:149: server-b98bac58-4a84-edd5-1003-a3d0df30dc2c: 2024-06-12T04:32:28.192+1400 [INFO] server-b98bac58-4a84-edd5-1003-a3d0df30dc2c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b67a0c6-3830-c6e6-3008-ce525ea6796b Address:1b67a0c6-3830-c6e6-3008-ce525ea6796b} {Suffrage:Voter ID:server-b98bac58-4a84-edd5-1003-a3d0df30dc2c Address:b98bac58-4a84-edd5-1003-a3d0df30dc2c} {Suffrage:Voter ID:server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6 Address:e07a9304-4edc-3a14-d254-66c4a5bfcaf6}]" testLoggerAdapter verbose: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.192+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b67a0c6-3830-c6e6-3008-ce525ea6796b Address:1b67a0c6-3830-c6e6-3008-ce525ea6796b} {Suffrage:Voter ID:server-b98bac58-4a84-edd5-1003-a3d0df30dc2c Address:b98bac58-4a84-edd5-1003-a3d0df30dc2c} {Suffrage:Voter ID:server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6 Address:e07a9304-4edc-3a14-d254-66c4a5bfcaf6}]" testing.go:149: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.192+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b67a0c6-3830-c6e6-3008-ce525ea6796b Address:1b67a0c6-3830-c6e6-3008-ce525ea6796b} {Suffrage:Voter ID:server-b98bac58-4a84-edd5-1003-a3d0df30dc2c Address:b98bac58-4a84-edd5-1003-a3d0df30dc2c} {Suffrage:Voter ID:server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6 Address:e07a9304-4edc-3a14-d254-66c4a5bfcaf6}]" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:28.192+1400 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-06-12T04:32:28.192+1400 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.192+1400 [INFO] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: entering follower state: follower="Node at 1b67a0c6-3830-c6e6-3008-ce525ea6796b [Follower]" leader= testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.192+1400 [INFO] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: entering follower state: follower="Node at 1b67a0c6-3830-c6e6-3008-ce525ea6796b [Follower]" leader= testLoggerAdapter verbose: server-b98bac58-4a84-edd5-1003-a3d0df30dc2c: 2024-06-12T04:32:28.192+1400 [INFO] server-b98bac58-4a84-edd5-1003-a3d0df30dc2c: entering follower state: follower="Node at b98bac58-4a84-edd5-1003-a3d0df30dc2c [Follower]" leader= testing.go:149: server-b98bac58-4a84-edd5-1003-a3d0df30dc2c: 2024-06-12T04:32:28.192+1400 [INFO] server-b98bac58-4a84-edd5-1003-a3d0df30dc2c: entering follower state: follower="Node at b98bac58-4a84-edd5-1003-a3d0df30dc2c [Follower]" leader= testLoggerAdapter verbose: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.193+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: entering follower state: follower="Node at e07a9304-4edc-3a14-d254-66c4a5bfcaf6 [Follower]" leader= testing.go:149: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.193+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: entering follower state: follower="Node at e07a9304-4edc-3a14-d254-66c4a5bfcaf6 [Follower]" leader= testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.271+1400 [WARN] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.271+1400 [WARN] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.271+1400 [INFO] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: entering candidate state: node="Node at 1b67a0c6-3830-c6e6-3008-ce525ea6796b [Candidate]" term=2 testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.271+1400 [INFO] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: entering candidate state: node="Node at 1b67a0c6-3830-c6e6-3008-ce525ea6796b [Candidate]" term=2 testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.271+1400 [INFO] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: election won: tally=2 testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.271+1400 [INFO] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: election won: tally=2 testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.271+1400 [INFO] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: entering leader state: leader="Node at 1b67a0c6-3830-c6e6-3008-ce525ea6796b [Leader]" testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.271+1400 [INFO] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: entering leader state: leader="Node at 1b67a0c6-3830-c6e6-3008-ce525ea6796b [Leader]" testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.271+1400 [INFO] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: added peer, starting replication: peer=server-b98bac58-4a84-edd5-1003-a3d0df30dc2c testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.271+1400 [INFO] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: added peer, starting replication: peer=server-b98bac58-4a84-edd5-1003-a3d0df30dc2c testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.271+1400 [INFO] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: added peer, starting replication: peer=server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6 testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.271+1400 [INFO] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: added peer, starting replication: peer=server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6 testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.272+1400 [INFO] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: pipelining replication: peer="{Voter server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6 e07a9304-4edc-3a14-d254-66c4a5bfcaf6}" testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.272+1400 [INFO] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: pipelining replication: peer="{Voter server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6 e07a9304-4edc-3a14-d254-66c4a5bfcaf6}" testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.272+1400 [INFO] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: pipelining replication: peer="{Voter server-b98bac58-4a84-edd5-1003-a3d0df30dc2c b98bac58-4a84-edd5-1003-a3d0df30dc2c}" testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.272+1400 [INFO] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: pipelining replication: peer="{Voter server-b98bac58-4a84-edd5-1003-a3d0df30dc2c b98bac58-4a84-edd5-1003-a3d0df30dc2c}" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:28.378+1400 [INFO] stable state for Follower reached at 2024-06-12 04:32:28.272239609 +1400 +14 m=+5.179758188 (2 nodes), 79.395196ms from start of poll, 80.052532ms from cluster start. Timeout at 2024-06-12 04:32:28.377977582 +1400 +14 m=+5.285496161, 105.737973ms after stability testing.go:149: cluster: 2024-06-12T04:32:28.378+1400 [INFO] stable state for Follower reached at 2024-06-12 04:32:28.272239609 +1400 +14 m=+5.179758188 (2 nodes), 79.395196ms from start of poll, 80.052532ms from cluster start. Timeout at 2024-06-12 04:32:28.377977582 +1400 +14 m=+5.285496161, 105.737973ms after stability testLoggerAdapter verbose: cluster: 2024-06-12T04:32:28.378+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-12T04:32:28.378+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-06-12T04:32:28.483+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:28.378172175 +1400 +14 m=+5.285690754 (1 nodes), 1.64µs from start of poll, 185.985098ms from cluster start. Timeout at 2024-06-12 04:32:28.48326889 +1400 +14 m=+5.390787509, 105.096755ms after stability testing.go:149: cluster: 2024-06-12T04:32:28.483+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:28.378172175 +1400 +14 m=+5.285690754 (1 nodes), 1.64µs from start of poll, 185.985098ms from cluster start. Timeout at 2024-06-12 04:32:28.48326889 +1400 +14 m=+5.390787509, 105.096755ms after stability raft_test.go:336: [INFO] Disconnecting Node at 1b67a0c6-3830-c6e6-3008-ce525ea6796b [Leader] testLoggerAdapter verbose: cluster: 2024-06-12T04:32:28.500+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-12T04:32:28.500+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.502+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to pipeline appendEntries: peer="{Voter server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6 e07a9304-4edc-3a14-d254-66c4a5bfcaf6}" error="append pipeline closed" testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.502+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to pipeline appendEntries: peer="{Voter server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6 e07a9304-4edc-3a14-d254-66c4a5bfcaf6}" error="append pipeline closed" testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.502+1400 [INFO] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: aborting pipeline replication: peer="{Voter server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6 e07a9304-4edc-3a14-d254-66c4a5bfcaf6}" testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.502+1400 [INFO] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: aborting pipeline replication: peer="{Voter server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6 e07a9304-4edc-3a14-d254-66c4a5bfcaf6}" testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.502+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to heartbeat to: peer=b98bac58-4a84-edd5-1003-a3d0df30dc2c error="failed to connect to peer: b98bac58-4a84-edd5-1003-a3d0df30dc2c" testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.502+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to heartbeat to: peer=b98bac58-4a84-edd5-1003-a3d0df30dc2c error="failed to connect to peer: b98bac58-4a84-edd5-1003-a3d0df30dc2c" testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.503+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to pipeline appendEntries: peer="{Voter server-b98bac58-4a84-edd5-1003-a3d0df30dc2c b98bac58-4a84-edd5-1003-a3d0df30dc2c}" error="append pipeline closed" testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.503+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to pipeline appendEntries: peer="{Voter server-b98bac58-4a84-edd5-1003-a3d0df30dc2c b98bac58-4a84-edd5-1003-a3d0df30dc2c}" error="append pipeline closed" testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.503+1400 [INFO] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: aborting pipeline replication: peer="{Voter server-b98bac58-4a84-edd5-1003-a3d0df30dc2c b98bac58-4a84-edd5-1003-a3d0df30dc2c}" testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.503+1400 [INFO] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: aborting pipeline replication: peer="{Voter server-b98bac58-4a84-edd5-1003-a3d0df30dc2c b98bac58-4a84-edd5-1003-a3d0df30dc2c}" testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.505+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to heartbeat to: peer=e07a9304-4edc-3a14-d254-66c4a5bfcaf6 error="failed to connect to peer: e07a9304-4edc-3a14-d254-66c4a5bfcaf6" testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.505+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to heartbeat to: peer=e07a9304-4edc-3a14-d254-66c4a5bfcaf6 error="failed to connect to peer: e07a9304-4edc-3a14-d254-66c4a5bfcaf6" testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.510+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to appendEntries to: peer="{Voter server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6 e07a9304-4edc-3a14-d254-66c4a5bfcaf6}" error="failed to connect to peer: e07a9304-4edc-3a14-d254-66c4a5bfcaf6" testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.510+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to appendEntries to: peer="{Voter server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6 e07a9304-4edc-3a14-d254-66c4a5bfcaf6}" error="failed to connect to peer: e07a9304-4edc-3a14-d254-66c4a5bfcaf6" testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.510+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to appendEntries to: peer="{Voter server-b98bac58-4a84-edd5-1003-a3d0df30dc2c b98bac58-4a84-edd5-1003-a3d0df30dc2c}" error="failed to connect to peer: b98bac58-4a84-edd5-1003-a3d0df30dc2c" testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.510+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to appendEntries to: peer="{Voter server-b98bac58-4a84-edd5-1003-a3d0df30dc2c b98bac58-4a84-edd5-1003-a3d0df30dc2c}" error="failed to connect to peer: b98bac58-4a84-edd5-1003-a3d0df30dc2c" testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.520+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to heartbeat to: peer=b98bac58-4a84-edd5-1003-a3d0df30dc2c error="failed to connect to peer: b98bac58-4a84-edd5-1003-a3d0df30dc2c" testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.520+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to heartbeat to: peer=b98bac58-4a84-edd5-1003-a3d0df30dc2c error="failed to connect to peer: b98bac58-4a84-edd5-1003-a3d0df30dc2c" testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.524+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to heartbeat to: peer=e07a9304-4edc-3a14-d254-66c4a5bfcaf6 error="failed to connect to peer: e07a9304-4edc-3a14-d254-66c4a5bfcaf6" testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.524+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to heartbeat to: peer=e07a9304-4edc-3a14-d254-66c4a5bfcaf6 error="failed to connect to peer: e07a9304-4edc-3a14-d254-66c4a5bfcaf6" testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.526+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to appendEntries to: peer="{Voter server-b98bac58-4a84-edd5-1003-a3d0df30dc2c b98bac58-4a84-edd5-1003-a3d0df30dc2c}" error="failed to connect to peer: b98bac58-4a84-edd5-1003-a3d0df30dc2c" testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.526+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to appendEntries to: peer="{Voter server-b98bac58-4a84-edd5-1003-a3d0df30dc2c b98bac58-4a84-edd5-1003-a3d0df30dc2c}" error="failed to connect to peer: b98bac58-4a84-edd5-1003-a3d0df30dc2c" testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.530+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to appendEntries to: peer="{Voter server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6 e07a9304-4edc-3a14-d254-66c4a5bfcaf6}" error="failed to connect to peer: e07a9304-4edc-3a14-d254-66c4a5bfcaf6" testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.530+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to appendEntries to: peer="{Voter server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6 e07a9304-4edc-3a14-d254-66c4a5bfcaf6}" error="failed to connect to peer: e07a9304-4edc-3a14-d254-66c4a5bfcaf6" testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.539+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to heartbeat to: peer=b98bac58-4a84-edd5-1003-a3d0df30dc2c error="failed to connect to peer: b98bac58-4a84-edd5-1003-a3d0df30dc2c" testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.539+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to heartbeat to: peer=b98bac58-4a84-edd5-1003-a3d0df30dc2c error="failed to connect to peer: b98bac58-4a84-edd5-1003-a3d0df30dc2c" testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.541+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to heartbeat to: peer=e07a9304-4edc-3a14-d254-66c4a5bfcaf6 error="failed to connect to peer: e07a9304-4edc-3a14-d254-66c4a5bfcaf6" testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.541+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to heartbeat to: peer=e07a9304-4edc-3a14-d254-66c4a5bfcaf6 error="failed to connect to peer: e07a9304-4edc-3a14-d254-66c4a5bfcaf6" testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.544+1400 [WARN] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to contact: server-id=server-b98bac58-4a84-edd5-1003-a3d0df30dc2c time=50.892417ms testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.544+1400 [WARN] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to contact: server-id=server-b98bac58-4a84-edd5-1003-a3d0df30dc2c time=50.892417ms testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.544+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to appendEntries to: peer="{Voter server-b98bac58-4a84-edd5-1003-a3d0df30dc2c b98bac58-4a84-edd5-1003-a3d0df30dc2c}" error="failed to connect to peer: b98bac58-4a84-edd5-1003-a3d0df30dc2c" testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.544+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to appendEntries to: peer="{Voter server-b98bac58-4a84-edd5-1003-a3d0df30dc2c b98bac58-4a84-edd5-1003-a3d0df30dc2c}" error="failed to connect to peer: b98bac58-4a84-edd5-1003-a3d0df30dc2c" testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.546+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to appendEntries to: peer="{Voter server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6 e07a9304-4edc-3a14-d254-66c4a5bfcaf6}" error="failed to connect to peer: e07a9304-4edc-3a14-d254-66c4a5bfcaf6" testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.546+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to appendEntries to: peer="{Voter server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6 e07a9304-4edc-3a14-d254-66c4a5bfcaf6}" error="failed to connect to peer: e07a9304-4edc-3a14-d254-66c4a5bfcaf6" testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.555+1400 [WARN] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to contact: server-id=server-b98bac58-4a84-edd5-1003-a3d0df30dc2c time=61.990219ms testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.555+1400 [WARN] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to contact: server-id=server-b98bac58-4a84-edd5-1003-a3d0df30dc2c time=61.990219ms testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.555+1400 [WARN] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to contact: server-id=server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6 time=60.83502ms testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.555+1400 [WARN] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to contact: server-id=server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6 time=60.83502ms testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.555+1400 [WARN] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to contact quorum of nodes, stepping down testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.555+1400 [WARN] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to contact quorum of nodes, stepping down testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.556+1400 [INFO] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: entering follower state: follower="Node at 1b67a0c6-3830-c6e6-3008-ce525ea6796b [Follower]" leader= testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.556+1400 [INFO] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: entering follower state: follower="Node at 1b67a0c6-3830-c6e6-3008-ce525ea6796b [Follower]" leader= testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.568+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to heartbeat to: peer=b98bac58-4a84-edd5-1003-a3d0df30dc2c error="failed to connect to peer: b98bac58-4a84-edd5-1003-a3d0df30dc2c" testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.568+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to heartbeat to: peer=b98bac58-4a84-edd5-1003-a3d0df30dc2c error="failed to connect to peer: b98bac58-4a84-edd5-1003-a3d0df30dc2c" testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.569+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to heartbeat to: peer=e07a9304-4edc-3a14-d254-66c4a5bfcaf6 error="failed to connect to peer: e07a9304-4edc-3a14-d254-66c4a5bfcaf6" testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.569+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to heartbeat to: peer=e07a9304-4edc-3a14-d254-66c4a5bfcaf6 error="failed to connect to peer: e07a9304-4edc-3a14-d254-66c4a5bfcaf6" testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.575+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to appendEntries to: peer="{Voter server-b98bac58-4a84-edd5-1003-a3d0df30dc2c b98bac58-4a84-edd5-1003-a3d0df30dc2c}" error="failed to connect to peer: b98bac58-4a84-edd5-1003-a3d0df30dc2c" testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.575+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to appendEntries to: peer="{Voter server-b98bac58-4a84-edd5-1003-a3d0df30dc2c b98bac58-4a84-edd5-1003-a3d0df30dc2c}" error="failed to connect to peer: b98bac58-4a84-edd5-1003-a3d0df30dc2c" testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.576+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to appendEntries to: peer="{Voter server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6 e07a9304-4edc-3a14-d254-66c4a5bfcaf6}" error="failed to connect to peer: e07a9304-4edc-3a14-d254-66c4a5bfcaf6" testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.576+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to appendEntries to: peer="{Voter server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6 e07a9304-4edc-3a14-d254-66c4a5bfcaf6}" error="failed to connect to peer: e07a9304-4edc-3a14-d254-66c4a5bfcaf6" testLoggerAdapter verbose: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.601+1400 [WARN] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: heartbeat timeout reached, starting election: last-leader=1b67a0c6-3830-c6e6-3008-ce525ea6796b testing.go:149: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.601+1400 [WARN] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: heartbeat timeout reached, starting election: last-leader=1b67a0c6-3830-c6e6-3008-ce525ea6796b testLoggerAdapter verbose: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.601+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: entering candidate state: node="Node at e07a9304-4edc-3a14-d254-66c4a5bfcaf6 [Candidate]" term=3 testing.go:149: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.601+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: entering candidate state: node="Node at e07a9304-4edc-3a14-d254-66c4a5bfcaf6 [Candidate]" term=3 testLoggerAdapter verbose: server-b98bac58-4a84-edd5-1003-a3d0df30dc2c: 2024-06-12T04:32:28.601+1400 [WARN] server-b98bac58-4a84-edd5-1003-a3d0df30dc2c: rejecting vote request since we have a leader: from=e07a9304-4edc-3a14-d254-66c4a5bfcaf6 leader=1b67a0c6-3830-c6e6-3008-ce525ea6796b testing.go:149: server-b98bac58-4a84-edd5-1003-a3d0df30dc2c: 2024-06-12T04:32:28.601+1400 [WARN] server-b98bac58-4a84-edd5-1003-a3d0df30dc2c: rejecting vote request since we have a leader: from=e07a9304-4edc-3a14-d254-66c4a5bfcaf6 leader=1b67a0c6-3830-c6e6-3008-ce525ea6796b testLoggerAdapter verbose: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.601+1400 [ERROR] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: failed to make requestVote RPC: target="{Voter server-1b67a0c6-3830-c6e6-3008-ce525ea6796b 1b67a0c6-3830-c6e6-3008-ce525ea6796b}" error="failed to connect to peer: 1b67a0c6-3830-c6e6-3008-ce525ea6796b" testing.go:149: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.601+1400 [ERROR] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: failed to make requestVote RPC: target="{Voter server-1b67a0c6-3830-c6e6-3008-ce525ea6796b 1b67a0c6-3830-c6e6-3008-ce525ea6796b}" error="failed to connect to peer: 1b67a0c6-3830-c6e6-3008-ce525ea6796b" testLoggerAdapter verbose: server-b98bac58-4a84-edd5-1003-a3d0df30dc2c: 2024-06-12T04:32:28.625+1400 [WARN] server-b98bac58-4a84-edd5-1003-a3d0df30dc2c: heartbeat timeout reached, starting election: last-leader=1b67a0c6-3830-c6e6-3008-ce525ea6796b testing.go:149: server-b98bac58-4a84-edd5-1003-a3d0df30dc2c: 2024-06-12T04:32:28.625+1400 [WARN] server-b98bac58-4a84-edd5-1003-a3d0df30dc2c: heartbeat timeout reached, starting election: last-leader=1b67a0c6-3830-c6e6-3008-ce525ea6796b testLoggerAdapter verbose: server-b98bac58-4a84-edd5-1003-a3d0df30dc2c: 2024-06-12T04:32:28.625+1400 [INFO] server-b98bac58-4a84-edd5-1003-a3d0df30dc2c: entering candidate state: node="Node at b98bac58-4a84-edd5-1003-a3d0df30dc2c [Candidate]" term=3 testing.go:149: server-b98bac58-4a84-edd5-1003-a3d0df30dc2c: 2024-06-12T04:32:28.625+1400 [INFO] server-b98bac58-4a84-edd5-1003-a3d0df30dc2c: entering candidate state: node="Node at b98bac58-4a84-edd5-1003-a3d0df30dc2c [Candidate]" term=3 testLoggerAdapter verbose: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.625+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: duplicate requestVote for same term: term=3 testing.go:149: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.625+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: duplicate requestVote for same term: term=3 testLoggerAdapter verbose: server-b98bac58-4a84-edd5-1003-a3d0df30dc2c: 2024-06-12T04:32:28.625+1400 [ERROR] server-b98bac58-4a84-edd5-1003-a3d0df30dc2c: failed to make requestVote RPC: target="{Voter server-1b67a0c6-3830-c6e6-3008-ce525ea6796b 1b67a0c6-3830-c6e6-3008-ce525ea6796b}" error="failed to connect to peer: 1b67a0c6-3830-c6e6-3008-ce525ea6796b" testing.go:149: server-b98bac58-4a84-edd5-1003-a3d0df30dc2c: 2024-06-12T04:32:28.625+1400 [ERROR] server-b98bac58-4a84-edd5-1003-a3d0df30dc2c: failed to make requestVote RPC: target="{Voter server-1b67a0c6-3830-c6e6-3008-ce525ea6796b 1b67a0c6-3830-c6e6-3008-ce525ea6796b}" error="failed to connect to peer: 1b67a0c6-3830-c6e6-3008-ce525ea6796b" testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.632+1400 [WARN] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.632+1400 [WARN] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.632+1400 [INFO] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: entering candidate state: node="Node at 1b67a0c6-3830-c6e6-3008-ce525ea6796b [Candidate]" term=3 testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.632+1400 [INFO] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: entering candidate state: node="Node at 1b67a0c6-3830-c6e6-3008-ce525ea6796b [Candidate]" term=3 testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.632+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to make requestVote RPC: target="{Voter server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6 e07a9304-4edc-3a14-d254-66c4a5bfcaf6}" error="failed to connect to peer: e07a9304-4edc-3a14-d254-66c4a5bfcaf6" testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.632+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to make requestVote RPC: target="{Voter server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6 e07a9304-4edc-3a14-d254-66c4a5bfcaf6}" error="failed to connect to peer: e07a9304-4edc-3a14-d254-66c4a5bfcaf6" testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.632+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to make requestVote RPC: target="{Voter server-b98bac58-4a84-edd5-1003-a3d0df30dc2c b98bac58-4a84-edd5-1003-a3d0df30dc2c}" error="failed to connect to peer: b98bac58-4a84-edd5-1003-a3d0df30dc2c" testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.632+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to make requestVote RPC: target="{Voter server-b98bac58-4a84-edd5-1003-a3d0df30dc2c b98bac58-4a84-edd5-1003-a3d0df30dc2c}" error="failed to connect to peer: b98bac58-4a84-edd5-1003-a3d0df30dc2c" testLoggerAdapter verbose: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.693+1400 [WARN] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: Election timeout reached, restarting election testing.go:149: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.693+1400 [WARN] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: Election timeout reached, restarting election testLoggerAdapter verbose: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.693+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: entering candidate state: node="Node at e07a9304-4edc-3a14-d254-66c4a5bfcaf6 [Candidate]" term=4 testing.go:149: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.693+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: entering candidate state: node="Node at e07a9304-4edc-3a14-d254-66c4a5bfcaf6 [Candidate]" term=4 testLoggerAdapter verbose: server-b98bac58-4a84-edd5-1003-a3d0df30dc2c: 2024-06-12T04:32:28.693+1400 [INFO] server-b98bac58-4a84-edd5-1003-a3d0df30dc2c: entering follower state: follower="Node at b98bac58-4a84-edd5-1003-a3d0df30dc2c [Follower]" leader= testing.go:149: server-b98bac58-4a84-edd5-1003-a3d0df30dc2c: 2024-06-12T04:32:28.693+1400 [INFO] server-b98bac58-4a84-edd5-1003-a3d0df30dc2c: entering follower state: follower="Node at b98bac58-4a84-edd5-1003-a3d0df30dc2c [Follower]" leader= testLoggerAdapter verbose: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.693+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: election won: tally=2 testing.go:149: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.693+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: election won: tally=2 testLoggerAdapter verbose: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.693+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: entering leader state: leader="Node at e07a9304-4edc-3a14-d254-66c4a5bfcaf6 [Leader]" testing.go:149: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.693+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: entering leader state: leader="Node at e07a9304-4edc-3a14-d254-66c4a5bfcaf6 [Leader]" testLoggerAdapter verbose: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.693+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: added peer, starting replication: peer=server-1b67a0c6-3830-c6e6-3008-ce525ea6796b testing.go:149: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.693+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: added peer, starting replication: peer=server-1b67a0c6-3830-c6e6-3008-ce525ea6796b testLoggerAdapter verbose: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.693+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: added peer, starting replication: peer=server-b98bac58-4a84-edd5-1003-a3d0df30dc2c testing.go:149: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.693+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: added peer, starting replication: peer=server-b98bac58-4a84-edd5-1003-a3d0df30dc2c testLoggerAdapter verbose: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.693+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: pipelining replication: peer="{Voter server-b98bac58-4a84-edd5-1003-a3d0df30dc2c b98bac58-4a84-edd5-1003-a3d0df30dc2c}" testing.go:149: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.693+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: pipelining replication: peer="{Voter server-b98bac58-4a84-edd5-1003-a3d0df30dc2c b98bac58-4a84-edd5-1003-a3d0df30dc2c}" testLoggerAdapter verbose: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.693+1400 [ERROR] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: failed to make requestVote RPC: target="{Voter server-1b67a0c6-3830-c6e6-3008-ce525ea6796b 1b67a0c6-3830-c6e6-3008-ce525ea6796b}" error="failed to connect to peer: 1b67a0c6-3830-c6e6-3008-ce525ea6796b" testing.go:149: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.693+1400 [ERROR] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: failed to make requestVote RPC: target="{Voter server-1b67a0c6-3830-c6e6-3008-ce525ea6796b 1b67a0c6-3830-c6e6-3008-ce525ea6796b}" error="failed to connect to peer: 1b67a0c6-3830-c6e6-3008-ce525ea6796b" testLoggerAdapter verbose: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.694+1400 [ERROR] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: failed to appendEntries to: peer="{Voter server-1b67a0c6-3830-c6e6-3008-ce525ea6796b 1b67a0c6-3830-c6e6-3008-ce525ea6796b}" error="failed to connect to peer: 1b67a0c6-3830-c6e6-3008-ce525ea6796b" testing.go:149: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.694+1400 [ERROR] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: failed to appendEntries to: peer="{Voter server-1b67a0c6-3830-c6e6-3008-ce525ea6796b 1b67a0c6-3830-c6e6-3008-ce525ea6796b}" error="failed to connect to peer: 1b67a0c6-3830-c6e6-3008-ce525ea6796b" testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.698+1400 [WARN] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: Election timeout reached, restarting election testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.698+1400 [WARN] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: Election timeout reached, restarting election testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.698+1400 [INFO] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: entering candidate state: node="Node at 1b67a0c6-3830-c6e6-3008-ce525ea6796b [Candidate]" term=4 testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.698+1400 [INFO] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: entering candidate state: node="Node at 1b67a0c6-3830-c6e6-3008-ce525ea6796b [Candidate]" term=4 testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.698+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to make requestVote RPC: target="{Voter server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6 e07a9304-4edc-3a14-d254-66c4a5bfcaf6}" error="failed to connect to peer: e07a9304-4edc-3a14-d254-66c4a5bfcaf6" testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.698+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to make requestVote RPC: target="{Voter server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6 e07a9304-4edc-3a14-d254-66c4a5bfcaf6}" error="failed to connect to peer: e07a9304-4edc-3a14-d254-66c4a5bfcaf6" testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.698+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to make requestVote RPC: target="{Voter server-b98bac58-4a84-edd5-1003-a3d0df30dc2c b98bac58-4a84-edd5-1003-a3d0df30dc2c}" error="failed to connect to peer: b98bac58-4a84-edd5-1003-a3d0df30dc2c" testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.698+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to make requestVote RPC: target="{Voter server-b98bac58-4a84-edd5-1003-a3d0df30dc2c b98bac58-4a84-edd5-1003-a3d0df30dc2c}" error="failed to connect to peer: b98bac58-4a84-edd5-1003-a3d0df30dc2c" testLoggerAdapter verbose: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.703+1400 [ERROR] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: failed to heartbeat to: peer=1b67a0c6-3830-c6e6-3008-ce525ea6796b error="failed to connect to peer: 1b67a0c6-3830-c6e6-3008-ce525ea6796b" testing.go:149: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.703+1400 [ERROR] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: failed to heartbeat to: peer=1b67a0c6-3830-c6e6-3008-ce525ea6796b error="failed to connect to peer: 1b67a0c6-3830-c6e6-3008-ce525ea6796b" testLoggerAdapter verbose: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.712+1400 [ERROR] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: failed to appendEntries to: peer="{Voter server-1b67a0c6-3830-c6e6-3008-ce525ea6796b 1b67a0c6-3830-c6e6-3008-ce525ea6796b}" error="failed to connect to peer: 1b67a0c6-3830-c6e6-3008-ce525ea6796b" testing.go:149: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.712+1400 [ERROR] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: failed to appendEntries to: peer="{Voter server-1b67a0c6-3830-c6e6-3008-ce525ea6796b 1b67a0c6-3830-c6e6-3008-ce525ea6796b}" error="failed to connect to peer: 1b67a0c6-3830-c6e6-3008-ce525ea6796b" testLoggerAdapter verbose: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.721+1400 [ERROR] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: failed to heartbeat to: peer=1b67a0c6-3830-c6e6-3008-ce525ea6796b error="failed to connect to peer: 1b67a0c6-3830-c6e6-3008-ce525ea6796b" testing.go:149: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.721+1400 [ERROR] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: failed to heartbeat to: peer=1b67a0c6-3830-c6e6-3008-ce525ea6796b error="failed to connect to peer: 1b67a0c6-3830-c6e6-3008-ce525ea6796b" testLoggerAdapter verbose: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.731+1400 [ERROR] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: failed to appendEntries to: peer="{Voter server-1b67a0c6-3830-c6e6-3008-ce525ea6796b 1b67a0c6-3830-c6e6-3008-ce525ea6796b}" error="failed to connect to peer: 1b67a0c6-3830-c6e6-3008-ce525ea6796b" testing.go:149: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.731+1400 [ERROR] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: failed to appendEntries to: peer="{Voter server-1b67a0c6-3830-c6e6-3008-ce525ea6796b 1b67a0c6-3830-c6e6-3008-ce525ea6796b}" error="failed to connect to peer: 1b67a0c6-3830-c6e6-3008-ce525ea6796b" testLoggerAdapter verbose: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.740+1400 [ERROR] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: failed to heartbeat to: peer=1b67a0c6-3830-c6e6-3008-ce525ea6796b error="failed to connect to peer: 1b67a0c6-3830-c6e6-3008-ce525ea6796b" testing.go:149: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.740+1400 [ERROR] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: failed to heartbeat to: peer=1b67a0c6-3830-c6e6-3008-ce525ea6796b error="failed to connect to peer: 1b67a0c6-3830-c6e6-3008-ce525ea6796b" testLoggerAdapter verbose: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.744+1400 [WARN] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: failed to contact: server-id=server-1b67a0c6-3830-c6e6-3008-ce525ea6796b time=50.422034ms testing.go:149: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.744+1400 [WARN] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: failed to contact: server-id=server-1b67a0c6-3830-c6e6-3008-ce525ea6796b time=50.422034ms testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.754+1400 [WARN] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: Election timeout reached, restarting election testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.754+1400 [WARN] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: Election timeout reached, restarting election testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.754+1400 [INFO] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: entering candidate state: node="Node at 1b67a0c6-3830-c6e6-3008-ce525ea6796b [Candidate]" term=5 testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.754+1400 [INFO] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: entering candidate state: node="Node at 1b67a0c6-3830-c6e6-3008-ce525ea6796b [Candidate]" term=5 testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.754+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to make requestVote RPC: target="{Voter server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6 e07a9304-4edc-3a14-d254-66c4a5bfcaf6}" error="failed to connect to peer: e07a9304-4edc-3a14-d254-66c4a5bfcaf6" testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.754+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to make requestVote RPC: target="{Voter server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6 e07a9304-4edc-3a14-d254-66c4a5bfcaf6}" error="failed to connect to peer: e07a9304-4edc-3a14-d254-66c4a5bfcaf6" testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.754+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to make requestVote RPC: target="{Voter server-b98bac58-4a84-edd5-1003-a3d0df30dc2c b98bac58-4a84-edd5-1003-a3d0df30dc2c}" error="failed to connect to peer: b98bac58-4a84-edd5-1003-a3d0df30dc2c" testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.754+1400 [ERROR] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to make requestVote RPC: target="{Voter server-b98bac58-4a84-edd5-1003-a3d0df30dc2c b98bac58-4a84-edd5-1003-a3d0df30dc2c}" error="failed to connect to peer: b98bac58-4a84-edd5-1003-a3d0df30dc2c" testLoggerAdapter verbose: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.761+1400 [ERROR] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: failed to appendEntries to: peer="{Voter server-1b67a0c6-3830-c6e6-3008-ce525ea6796b 1b67a0c6-3830-c6e6-3008-ce525ea6796b}" error="failed to connect to peer: 1b67a0c6-3830-c6e6-3008-ce525ea6796b" testing.go:149: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.761+1400 [ERROR] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: failed to appendEntries to: peer="{Voter server-1b67a0c6-3830-c6e6-3008-ce525ea6796b 1b67a0c6-3830-c6e6-3008-ce525ea6796b}" error="failed to connect to peer: 1b67a0c6-3830-c6e6-3008-ce525ea6796b" testLoggerAdapter verbose: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.770+1400 [ERROR] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: failed to heartbeat to: peer=1b67a0c6-3830-c6e6-3008-ce525ea6796b error="failed to connect to peer: 1b67a0c6-3830-c6e6-3008-ce525ea6796b" testing.go:149: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.770+1400 [ERROR] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: failed to heartbeat to: peer=1b67a0c6-3830-c6e6-3008-ce525ea6796b error="failed to connect to peer: 1b67a0c6-3830-c6e6-3008-ce525ea6796b" testLoggerAdapter verbose: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.793+1400 [WARN] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: failed to contact: server-id=server-1b67a0c6-3830-c6e6-3008-ce525ea6796b time=99.503796ms testing.go:149: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.793+1400 [WARN] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: failed to contact: server-id=server-1b67a0c6-3830-c6e6-3008-ce525ea6796b time=99.503796ms testLoggerAdapter verbose: cluster: 2024-06-12T04:32:28.800+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:28.694086379 +1400 +14 m=+5.601604998 (1 nodes), 193.316756ms from start of poll, 501.899342ms from cluster start. Timeout at 2024-06-12 04:32:28.800059463 +1400 +14 m=+5.707578082, 105.973084ms after stability testing.go:149: cluster: 2024-06-12T04:32:28.800+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:28.694086379 +1400 +14 m=+5.601604998 (1 nodes), 193.316756ms from start of poll, 501.899342ms from cluster start. Timeout at 2024-06-12 04:32:28.800059463 +1400 +14 m=+5.707578082, 105.973084ms after stability raft_test.go:371: [INFO] Reconnecting Node at 1b67a0c6-3830-c6e6-3008-ce525ea6796b [Candidate] testLoggerAdapter verbose: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.810+1400 [ERROR] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: peer has newer term, stopping replication: peer="{Voter server-1b67a0c6-3830-c6e6-3008-ce525ea6796b 1b67a0c6-3830-c6e6-3008-ce525ea6796b}" testing.go:149: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.810+1400 [ERROR] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: peer has newer term, stopping replication: peer="{Voter server-1b67a0c6-3830-c6e6-3008-ce525ea6796b 1b67a0c6-3830-c6e6-3008-ce525ea6796b}" testLoggerAdapter verbose: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.810+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: entering follower state: follower="Node at e07a9304-4edc-3a14-d254-66c4a5bfcaf6 [Follower]" leader= testing.go:149: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.810+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: entering follower state: follower="Node at e07a9304-4edc-3a14-d254-66c4a5bfcaf6 [Follower]" leader= testLoggerAdapter verbose: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.810+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: aborting pipeline replication: peer="{Voter server-b98bac58-4a84-edd5-1003-a3d0df30dc2c b98bac58-4a84-edd5-1003-a3d0df30dc2c}" testing.go:149: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.810+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: aborting pipeline replication: peer="{Voter server-b98bac58-4a84-edd5-1003-a3d0df30dc2c b98bac58-4a84-edd5-1003-a3d0df30dc2c}" testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.821+1400 [WARN] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: Election timeout reached, restarting election testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.821+1400 [WARN] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: Election timeout reached, restarting election testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.821+1400 [INFO] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: entering candidate state: node="Node at 1b67a0c6-3830-c6e6-3008-ce525ea6796b [Candidate]" term=6 testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.821+1400 [INFO] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: entering candidate state: node="Node at 1b67a0c6-3830-c6e6-3008-ce525ea6796b [Candidate]" term=6 testLoggerAdapter verbose: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.821+1400 [WARN] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: rejecting vote request since our last term is greater: candidate=1b67a0c6-3830-c6e6-3008-ce525ea6796b last-term=4 last-candidate-term=2 testing.go:149: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.821+1400 [WARN] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: rejecting vote request since our last term is greater: candidate=1b67a0c6-3830-c6e6-3008-ce525ea6796b last-term=4 last-candidate-term=2 testLoggerAdapter verbose: server-b98bac58-4a84-edd5-1003-a3d0df30dc2c: 2024-06-12T04:32:28.821+1400 [WARN] server-b98bac58-4a84-edd5-1003-a3d0df30dc2c: rejecting vote request since we have a leader: from=1b67a0c6-3830-c6e6-3008-ce525ea6796b leader=e07a9304-4edc-3a14-d254-66c4a5bfcaf6 testing.go:149: server-b98bac58-4a84-edd5-1003-a3d0df30dc2c: 2024-06-12T04:32:28.821+1400 [WARN] server-b98bac58-4a84-edd5-1003-a3d0df30dc2c: rejecting vote request since we have a leader: from=1b67a0c6-3830-c6e6-3008-ce525ea6796b leader=e07a9304-4edc-3a14-d254-66c4a5bfcaf6 testLoggerAdapter verbose: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.886+1400 [WARN] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.886+1400 [WARN] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.886+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: entering candidate state: node="Node at e07a9304-4edc-3a14-d254-66c4a5bfcaf6 [Candidate]" term=7 testing.go:149: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.886+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: entering candidate state: node="Node at e07a9304-4edc-3a14-d254-66c4a5bfcaf6 [Candidate]" term=7 testLoggerAdapter verbose: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.886+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: election won: tally=2 testing.go:149: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.886+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: election won: tally=2 testLoggerAdapter verbose: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.886+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: entering leader state: leader="Node at e07a9304-4edc-3a14-d254-66c4a5bfcaf6 [Leader]" testing.go:149: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.886+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: entering leader state: leader="Node at e07a9304-4edc-3a14-d254-66c4a5bfcaf6 [Leader]" testLoggerAdapter verbose: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.886+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: added peer, starting replication: peer=server-1b67a0c6-3830-c6e6-3008-ce525ea6796b testing.go:149: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.886+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: added peer, starting replication: peer=server-1b67a0c6-3830-c6e6-3008-ce525ea6796b testLoggerAdapter verbose: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.886+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: added peer, starting replication: peer=server-b98bac58-4a84-edd5-1003-a3d0df30dc2c testing.go:149: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.886+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: added peer, starting replication: peer=server-b98bac58-4a84-edd5-1003-a3d0df30dc2c testLoggerAdapter verbose: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.887+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: pipelining replication: peer="{Voter server-b98bac58-4a84-edd5-1003-a3d0df30dc2c b98bac58-4a84-edd5-1003-a3d0df30dc2c}" testing.go:149: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.887+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: pipelining replication: peer="{Voter server-b98bac58-4a84-edd5-1003-a3d0df30dc2c b98bac58-4a84-edd5-1003-a3d0df30dc2c}" testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.887+1400 [INFO] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: entering follower state: follower="Node at 1b67a0c6-3830-c6e6-3008-ce525ea6796b [Follower]" leader= testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.887+1400 [INFO] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: entering follower state: follower="Node at 1b67a0c6-3830-c6e6-3008-ce525ea6796b [Follower]" leader= testLoggerAdapter verbose: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.887+1400 [WARN] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to get previous log: previous-index=5 last-index=3 error="log not found" testing.go:149: server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: 2024-06-12T04:32:28.887+1400 [WARN] server-1b67a0c6-3830-c6e6-3008-ce525ea6796b: failed to get previous log: previous-index=5 last-index=3 error="log not found" testLoggerAdapter verbose: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.887+1400 [WARN] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: appendEntries rejected, sending older logs: peer="{Voter server-1b67a0c6-3830-c6e6-3008-ce525ea6796b 1b67a0c6-3830-c6e6-3008-ce525ea6796b}" next=4 testing.go:149: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.887+1400 [WARN] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: appendEntries rejected, sending older logs: peer="{Voter server-1b67a0c6-3830-c6e6-3008-ce525ea6796b 1b67a0c6-3830-c6e6-3008-ce525ea6796b}" next=4 testLoggerAdapter verbose: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.887+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: pipelining replication: peer="{Voter server-1b67a0c6-3830-c6e6-3008-ce525ea6796b 1b67a0c6-3830-c6e6-3008-ce525ea6796b}" testing.go:149: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.887+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: pipelining replication: peer="{Voter server-1b67a0c6-3830-c6e6-3008-ce525ea6796b 1b67a0c6-3830-c6e6-3008-ce525ea6796b}" testLoggerAdapter verbose: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.892+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: aborting pipeline replication: peer="{Voter server-b98bac58-4a84-edd5-1003-a3d0df30dc2c b98bac58-4a84-edd5-1003-a3d0df30dc2c}" testing.go:149: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.892+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: aborting pipeline replication: peer="{Voter server-b98bac58-4a84-edd5-1003-a3d0df30dc2c b98bac58-4a84-edd5-1003-a3d0df30dc2c}" testLoggerAdapter verbose: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.892+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: aborting pipeline replication: peer="{Voter server-1b67a0c6-3830-c6e6-3008-ce525ea6796b 1b67a0c6-3830-c6e6-3008-ce525ea6796b}" testing.go:149: server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: 2024-06-12T04:32:28.892+1400 [INFO] server-e07a9304-4edc-3a14-d254-66c4a5bfcaf6: aborting pipeline replication: peer="{Voter server-1b67a0c6-3830-c6e6-3008-ce525ea6796b 1b67a0c6-3830-c6e6-3008-ce525ea6796b}" --- PASS: TestRaft_LeaderFail (0.70s) === RUN TestRaft_BehindFollower testLoggerAdapter verbose: server-69be2739-89bd-c296-c04b-dfebbfb903ad: 2024-06-12T04:32:28.894+1400 [INFO] server-69be2739-89bd-c296-c04b-dfebbfb903ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-69be2739-89bd-c296-c04b-dfebbfb903ad Address:69be2739-89bd-c296-c04b-dfebbfb903ad} {Suffrage:Voter ID:server-48c20aa7-07e6-d893-c52d-be0ff43926c7 Address:48c20aa7-07e6-d893-c52d-be0ff43926c7} {Suffrage:Voter ID:server-97313dad-c6e1-76ce-4f67-c599c5c35287 Address:97313dad-c6e1-76ce-4f67-c599c5c35287}]" testing.go:149: server-69be2739-89bd-c296-c04b-dfebbfb903ad: 2024-06-12T04:32:28.894+1400 [INFO] server-69be2739-89bd-c296-c04b-dfebbfb903ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-69be2739-89bd-c296-c04b-dfebbfb903ad Address:69be2739-89bd-c296-c04b-dfebbfb903ad} {Suffrage:Voter ID:server-48c20aa7-07e6-d893-c52d-be0ff43926c7 Address:48c20aa7-07e6-d893-c52d-be0ff43926c7} {Suffrage:Voter ID:server-97313dad-c6e1-76ce-4f67-c599c5c35287 Address:97313dad-c6e1-76ce-4f67-c599c5c35287}]" testLoggerAdapter verbose: server-48c20aa7-07e6-d893-c52d-be0ff43926c7: 2024-06-12T04:32:28.895+1400 [INFO] server-48c20aa7-07e6-d893-c52d-be0ff43926c7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-69be2739-89bd-c296-c04b-dfebbfb903ad Address:69be2739-89bd-c296-c04b-dfebbfb903ad} {Suffrage:Voter ID:server-48c20aa7-07e6-d893-c52d-be0ff43926c7 Address:48c20aa7-07e6-d893-c52d-be0ff43926c7} {Suffrage:Voter ID:server-97313dad-c6e1-76ce-4f67-c599c5c35287 Address:97313dad-c6e1-76ce-4f67-c599c5c35287}]" testing.go:149: server-48c20aa7-07e6-d893-c52d-be0ff43926c7: 2024-06-12T04:32:28.895+1400 [INFO] server-48c20aa7-07e6-d893-c52d-be0ff43926c7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-69be2739-89bd-c296-c04b-dfebbfb903ad Address:69be2739-89bd-c296-c04b-dfebbfb903ad} {Suffrage:Voter ID:server-48c20aa7-07e6-d893-c52d-be0ff43926c7 Address:48c20aa7-07e6-d893-c52d-be0ff43926c7} {Suffrage:Voter ID:server-97313dad-c6e1-76ce-4f67-c599c5c35287 Address:97313dad-c6e1-76ce-4f67-c599c5c35287}]" testLoggerAdapter verbose: server-97313dad-c6e1-76ce-4f67-c599c5c35287: 2024-06-12T04:32:28.895+1400 [INFO] server-97313dad-c6e1-76ce-4f67-c599c5c35287: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-69be2739-89bd-c296-c04b-dfebbfb903ad Address:69be2739-89bd-c296-c04b-dfebbfb903ad} {Suffrage:Voter ID:server-48c20aa7-07e6-d893-c52d-be0ff43926c7 Address:48c20aa7-07e6-d893-c52d-be0ff43926c7} {Suffrage:Voter ID:server-97313dad-c6e1-76ce-4f67-c599c5c35287 Address:97313dad-c6e1-76ce-4f67-c599c5c35287}]" testing.go:149: server-97313dad-c6e1-76ce-4f67-c599c5c35287: 2024-06-12T04:32:28.895+1400 [INFO] server-97313dad-c6e1-76ce-4f67-c599c5c35287: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-69be2739-89bd-c296-c04b-dfebbfb903ad Address:69be2739-89bd-c296-c04b-dfebbfb903ad} {Suffrage:Voter ID:server-48c20aa7-07e6-d893-c52d-be0ff43926c7 Address:48c20aa7-07e6-d893-c52d-be0ff43926c7} {Suffrage:Voter ID:server-97313dad-c6e1-76ce-4f67-c599c5c35287 Address:97313dad-c6e1-76ce-4f67-c599c5c35287}]" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:28.895+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-12T04:32:28.895+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-69be2739-89bd-c296-c04b-dfebbfb903ad: 2024-06-12T04:32:28.895+1400 [INFO] server-69be2739-89bd-c296-c04b-dfebbfb903ad: entering follower state: follower="Node at 69be2739-89bd-c296-c04b-dfebbfb903ad [Follower]" leader= testing.go:149: server-69be2739-89bd-c296-c04b-dfebbfb903ad: 2024-06-12T04:32:28.895+1400 [INFO] server-69be2739-89bd-c296-c04b-dfebbfb903ad: entering follower state: follower="Node at 69be2739-89bd-c296-c04b-dfebbfb903ad [Follower]" leader= testLoggerAdapter verbose: server-48c20aa7-07e6-d893-c52d-be0ff43926c7: 2024-06-12T04:32:28.895+1400 [INFO] server-48c20aa7-07e6-d893-c52d-be0ff43926c7: entering follower state: follower="Node at 48c20aa7-07e6-d893-c52d-be0ff43926c7 [Follower]" leader= testing.go:149: server-48c20aa7-07e6-d893-c52d-be0ff43926c7: 2024-06-12T04:32:28.895+1400 [INFO] server-48c20aa7-07e6-d893-c52d-be0ff43926c7: entering follower state: follower="Node at 48c20aa7-07e6-d893-c52d-be0ff43926c7 [Follower]" leader= testLoggerAdapter verbose: server-97313dad-c6e1-76ce-4f67-c599c5c35287: 2024-06-12T04:32:28.895+1400 [INFO] server-97313dad-c6e1-76ce-4f67-c599c5c35287: entering follower state: follower="Node at 97313dad-c6e1-76ce-4f67-c599c5c35287 [Follower]" leader= testing.go:149: server-97313dad-c6e1-76ce-4f67-c599c5c35287: 2024-06-12T04:32:28.895+1400 [INFO] server-97313dad-c6e1-76ce-4f67-c599c5c35287: entering follower state: follower="Node at 97313dad-c6e1-76ce-4f67-c599c5c35287 [Follower]" leader= testLoggerAdapter verbose: server-97313dad-c6e1-76ce-4f67-c599c5c35287: 2024-06-12T04:32:28.980+1400 [WARN] server-97313dad-c6e1-76ce-4f67-c599c5c35287: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-97313dad-c6e1-76ce-4f67-c599c5c35287: 2024-06-12T04:32:28.980+1400 [WARN] server-97313dad-c6e1-76ce-4f67-c599c5c35287: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-97313dad-c6e1-76ce-4f67-c599c5c35287: 2024-06-12T04:32:28.980+1400 [INFO] server-97313dad-c6e1-76ce-4f67-c599c5c35287: entering candidate state: node="Node at 97313dad-c6e1-76ce-4f67-c599c5c35287 [Candidate]" term=2 testing.go:149: server-97313dad-c6e1-76ce-4f67-c599c5c35287: 2024-06-12T04:32:28.980+1400 [INFO] server-97313dad-c6e1-76ce-4f67-c599c5c35287: entering candidate state: node="Node at 97313dad-c6e1-76ce-4f67-c599c5c35287 [Candidate]" term=2 testLoggerAdapter verbose: server-97313dad-c6e1-76ce-4f67-c599c5c35287: 2024-06-12T04:32:28.980+1400 [INFO] server-97313dad-c6e1-76ce-4f67-c599c5c35287: election won: tally=2 testing.go:149: server-97313dad-c6e1-76ce-4f67-c599c5c35287: 2024-06-12T04:32:28.980+1400 [INFO] server-97313dad-c6e1-76ce-4f67-c599c5c35287: election won: tally=2 testLoggerAdapter verbose: server-97313dad-c6e1-76ce-4f67-c599c5c35287: 2024-06-12T04:32:28.980+1400 [INFO] server-97313dad-c6e1-76ce-4f67-c599c5c35287: entering leader state: leader="Node at 97313dad-c6e1-76ce-4f67-c599c5c35287 [Leader]" testing.go:149: server-97313dad-c6e1-76ce-4f67-c599c5c35287: 2024-06-12T04:32:28.980+1400 [INFO] server-97313dad-c6e1-76ce-4f67-c599c5c35287: entering leader state: leader="Node at 97313dad-c6e1-76ce-4f67-c599c5c35287 [Leader]" testLoggerAdapter verbose: server-97313dad-c6e1-76ce-4f67-c599c5c35287: 2024-06-12T04:32:28.980+1400 [INFO] server-97313dad-c6e1-76ce-4f67-c599c5c35287: added peer, starting replication: peer=server-69be2739-89bd-c296-c04b-dfebbfb903ad testing.go:149: server-97313dad-c6e1-76ce-4f67-c599c5c35287: 2024-06-12T04:32:28.980+1400 [INFO] server-97313dad-c6e1-76ce-4f67-c599c5c35287: added peer, starting replication: peer=server-69be2739-89bd-c296-c04b-dfebbfb903ad testLoggerAdapter verbose: server-97313dad-c6e1-76ce-4f67-c599c5c35287: 2024-06-12T04:32:28.981+1400 [INFO] server-97313dad-c6e1-76ce-4f67-c599c5c35287: added peer, starting replication: peer=server-48c20aa7-07e6-d893-c52d-be0ff43926c7 testing.go:149: server-97313dad-c6e1-76ce-4f67-c599c5c35287: 2024-06-12T04:32:28.981+1400 [INFO] server-97313dad-c6e1-76ce-4f67-c599c5c35287: added peer, starting replication: peer=server-48c20aa7-07e6-d893-c52d-be0ff43926c7 testLoggerAdapter verbose: server-97313dad-c6e1-76ce-4f67-c599c5c35287: 2024-06-12T04:32:28.981+1400 [INFO] server-97313dad-c6e1-76ce-4f67-c599c5c35287: pipelining replication: peer="{Voter server-48c20aa7-07e6-d893-c52d-be0ff43926c7 48c20aa7-07e6-d893-c52d-be0ff43926c7}" testing.go:149: server-97313dad-c6e1-76ce-4f67-c599c5c35287: 2024-06-12T04:32:28.981+1400 [INFO] server-97313dad-c6e1-76ce-4f67-c599c5c35287: pipelining replication: peer="{Voter server-48c20aa7-07e6-d893-c52d-be0ff43926c7 48c20aa7-07e6-d893-c52d-be0ff43926c7}" testLoggerAdapter verbose: server-97313dad-c6e1-76ce-4f67-c599c5c35287: 2024-06-12T04:32:28.981+1400 [INFO] server-97313dad-c6e1-76ce-4f67-c599c5c35287: pipelining replication: peer="{Voter server-69be2739-89bd-c296-c04b-dfebbfb903ad 69be2739-89bd-c296-c04b-dfebbfb903ad}" testing.go:149: server-97313dad-c6e1-76ce-4f67-c599c5c35287: 2024-06-12T04:32:28.981+1400 [INFO] server-97313dad-c6e1-76ce-4f67-c599c5c35287: pipelining replication: peer="{Voter server-69be2739-89bd-c296-c04b-dfebbfb903ad 69be2739-89bd-c296-c04b-dfebbfb903ad}" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:29.086+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:28.981399568 +1400 +14 m=+5.888918147 (1 nodes), 85.900323ms from start of poll, 86.606098ms from cluster start. Timeout at 2024-06-12 04:32:29.086414366 +1400 +14 m=+5.993932985, 105.014838ms after stability testing.go:149: cluster: 2024-06-12T04:32:29.086+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:28.981399568 +1400 +14 m=+5.888918147 (1 nodes), 85.900323ms from start of poll, 86.606098ms from cluster start. Timeout at 2024-06-12 04:32:29.086414366 +1400 +14 m=+5.993932985, 105.014838ms after stability testLoggerAdapter verbose: cluster: 2024-06-12T04:32:29.086+1400 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-06-12T04:32:29.086+1400 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2024-06-12T04:32:29.192+1400 [INFO] stable state for Follower reached at 2024-06-12 04:32:29.086612239 +1400 +14 m=+5.994130818 (2 nodes), 1.72µs from start of poll, 191.818769ms from cluster start. Timeout at 2024-06-12 04:32:29.192657641 +1400 +14 m=+6.100176260, 106.045442ms after stability testing.go:149: cluster: 2024-06-12T04:32:29.192+1400 [INFO] stable state for Follower reached at 2024-06-12 04:32:29.086612239 +1400 +14 m=+5.994130818 (2 nodes), 1.72µs from start of poll, 191.818769ms from cluster start. Timeout at 2024-06-12 04:32:29.192657641 +1400 +14 m=+6.100176260, 106.045442ms after stability testLoggerAdapter verbose: server-97313dad-c6e1-76ce-4f67-c599c5c35287: 2024-06-12T04:32:29.192+1400 [ERROR] server-97313dad-c6e1-76ce-4f67-c599c5c35287: failed to pipeline appendEntries: peer="{Voter server-69be2739-89bd-c296-c04b-dfebbfb903ad 69be2739-89bd-c296-c04b-dfebbfb903ad}" error="append pipeline closed" testing.go:149: server-97313dad-c6e1-76ce-4f67-c599c5c35287: 2024-06-12T04:32:29.192+1400 [ERROR] server-97313dad-c6e1-76ce-4f67-c599c5c35287: failed to pipeline appendEntries: peer="{Voter server-69be2739-89bd-c296-c04b-dfebbfb903ad 69be2739-89bd-c296-c04b-dfebbfb903ad}" error="append pipeline closed" testLoggerAdapter verbose: server-97313dad-c6e1-76ce-4f67-c599c5c35287: 2024-06-12T04:32:29.192+1400 [INFO] server-97313dad-c6e1-76ce-4f67-c599c5c35287: aborting pipeline replication: peer="{Voter server-69be2739-89bd-c296-c04b-dfebbfb903ad 69be2739-89bd-c296-c04b-dfebbfb903ad}" testing.go:149: server-97313dad-c6e1-76ce-4f67-c599c5c35287: 2024-06-12T04:32:29.192+1400 [INFO] server-97313dad-c6e1-76ce-4f67-c599c5c35287: aborting pipeline replication: peer="{Voter server-69be2739-89bd-c296-c04b-dfebbfb903ad 69be2739-89bd-c296-c04b-dfebbfb903ad}" testLoggerAdapter verbose: server-97313dad-c6e1-76ce-4f67-c599c5c35287: 2024-06-12T04:32:29.193+1400 [ERROR] server-97313dad-c6e1-76ce-4f67-c599c5c35287: failed to appendEntries to: peer="{Voter server-69be2739-89bd-c296-c04b-dfebbfb903ad 69be2739-89bd-c296-c04b-dfebbfb903ad}" error="failed to connect to peer: 69be2739-89bd-c296-c04b-dfebbfb903ad" testing.go:149: server-97313dad-c6e1-76ce-4f67-c599c5c35287: 2024-06-12T04:32:29.193+1400 [ERROR] server-97313dad-c6e1-76ce-4f67-c599c5c35287: failed to appendEntries to: peer="{Voter server-69be2739-89bd-c296-c04b-dfebbfb903ad 69be2739-89bd-c296-c04b-dfebbfb903ad}" error="failed to connect to peer: 69be2739-89bd-c296-c04b-dfebbfb903ad" raft_test.go:420: [INFO] Finished apply without behind follower testLoggerAdapter verbose: server-97313dad-c6e1-76ce-4f67-c599c5c35287: 2024-06-12T04:32:29.203+1400 [INFO] server-97313dad-c6e1-76ce-4f67-c599c5c35287: pipelining replication: peer="{Voter server-69be2739-89bd-c296-c04b-dfebbfb903ad 69be2739-89bd-c296-c04b-dfebbfb903ad}" testing.go:149: server-97313dad-c6e1-76ce-4f67-c599c5c35287: 2024-06-12T04:32:29.203+1400 [INFO] server-97313dad-c6e1-76ce-4f67-c599c5c35287: pipelining replication: peer="{Voter server-69be2739-89bd-c296-c04b-dfebbfb903ad 69be2739-89bd-c296-c04b-dfebbfb903ad}" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:29.213+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-12T04:32:29.213+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-06-12T04:32:29.319+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:29.213744845 +1400 +14 m=+6.121263464 (1 nodes), 1.44µs from start of poll, 318.951415ms from cluster start. Timeout at 2024-06-12 04:32:29.319592134 +1400 +14 m=+6.227110753, 105.847289ms after stability testing.go:149: cluster: 2024-06-12T04:32:29.319+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:29.213744845 +1400 +14 m=+6.121263464 (1 nodes), 1.44µs from start of poll, 318.951415ms from cluster start. Timeout at 2024-06-12 04:32:29.319592134 +1400 +14 m=+6.227110753, 105.847289ms after stability testLoggerAdapter verbose: server-97313dad-c6e1-76ce-4f67-c599c5c35287: 2024-06-12T04:32:29.319+1400 [INFO] server-97313dad-c6e1-76ce-4f67-c599c5c35287: aborting pipeline replication: peer="{Voter server-48c20aa7-07e6-d893-c52d-be0ff43926c7 48c20aa7-07e6-d893-c52d-be0ff43926c7}" testing.go:149: server-97313dad-c6e1-76ce-4f67-c599c5c35287: 2024-06-12T04:32:29.319+1400 [INFO] server-97313dad-c6e1-76ce-4f67-c599c5c35287: aborting pipeline replication: peer="{Voter server-48c20aa7-07e6-d893-c52d-be0ff43926c7 48c20aa7-07e6-d893-c52d-be0ff43926c7}" testLoggerAdapter verbose: server-97313dad-c6e1-76ce-4f67-c599c5c35287: 2024-06-12T04:32:29.319+1400 [INFO] server-97313dad-c6e1-76ce-4f67-c599c5c35287: aborting pipeline replication: peer="{Voter server-69be2739-89bd-c296-c04b-dfebbfb903ad 69be2739-89bd-c296-c04b-dfebbfb903ad}" testing.go:149: server-97313dad-c6e1-76ce-4f67-c599c5c35287: 2024-06-12T04:32:29.319+1400 [INFO] server-97313dad-c6e1-76ce-4f67-c599c5c35287: aborting pipeline replication: peer="{Voter server-69be2739-89bd-c296-c04b-dfebbfb903ad 69be2739-89bd-c296-c04b-dfebbfb903ad}" --- PASS: TestRaft_BehindFollower (0.43s) === RUN TestRaft_ApplyNonLeader testLoggerAdapter verbose: server-c09d30a2-a922-f086-95e2-c9d01912bed3: 2024-06-12T04:32:29.321+1400 [INFO] server-c09d30a2-a922-f086-95e2-c9d01912bed3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c09d30a2-a922-f086-95e2-c9d01912bed3 Address:c09d30a2-a922-f086-95e2-c9d01912bed3} {Suffrage:Voter ID:server-df5d237b-3e52-babb-d245-bbd701516719 Address:df5d237b-3e52-babb-d245-bbd701516719} {Suffrage:Voter ID:server-3b40fe9f-29c8-e8ac-af9d-c55d7b09c2f6 Address:3b40fe9f-29c8-e8ac-af9d-c55d7b09c2f6}]" testing.go:149: server-c09d30a2-a922-f086-95e2-c9d01912bed3: 2024-06-12T04:32:29.321+1400 [INFO] server-c09d30a2-a922-f086-95e2-c9d01912bed3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c09d30a2-a922-f086-95e2-c9d01912bed3 Address:c09d30a2-a922-f086-95e2-c9d01912bed3} {Suffrage:Voter ID:server-df5d237b-3e52-babb-d245-bbd701516719 Address:df5d237b-3e52-babb-d245-bbd701516719} {Suffrage:Voter ID:server-3b40fe9f-29c8-e8ac-af9d-c55d7b09c2f6 Address:3b40fe9f-29c8-e8ac-af9d-c55d7b09c2f6}]" testLoggerAdapter verbose: server-df5d237b-3e52-babb-d245-bbd701516719: 2024-06-12T04:32:29.322+1400 [INFO] server-df5d237b-3e52-babb-d245-bbd701516719: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c09d30a2-a922-f086-95e2-c9d01912bed3 Address:c09d30a2-a922-f086-95e2-c9d01912bed3} {Suffrage:Voter ID:server-df5d237b-3e52-babb-d245-bbd701516719 Address:df5d237b-3e52-babb-d245-bbd701516719} {Suffrage:Voter ID:server-3b40fe9f-29c8-e8ac-af9d-c55d7b09c2f6 Address:3b40fe9f-29c8-e8ac-af9d-c55d7b09c2f6}]" testing.go:149: server-df5d237b-3e52-babb-d245-bbd701516719: 2024-06-12T04:32:29.322+1400 [INFO] server-df5d237b-3e52-babb-d245-bbd701516719: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c09d30a2-a922-f086-95e2-c9d01912bed3 Address:c09d30a2-a922-f086-95e2-c9d01912bed3} {Suffrage:Voter ID:server-df5d237b-3e52-babb-d245-bbd701516719 Address:df5d237b-3e52-babb-d245-bbd701516719} {Suffrage:Voter ID:server-3b40fe9f-29c8-e8ac-af9d-c55d7b09c2f6 Address:3b40fe9f-29c8-e8ac-af9d-c55d7b09c2f6}]" testLoggerAdapter verbose: server-3b40fe9f-29c8-e8ac-af9d-c55d7b09c2f6: 2024-06-12T04:32:29.322+1400 [INFO] server-3b40fe9f-29c8-e8ac-af9d-c55d7b09c2f6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c09d30a2-a922-f086-95e2-c9d01912bed3 Address:c09d30a2-a922-f086-95e2-c9d01912bed3} {Suffrage:Voter ID:server-df5d237b-3e52-babb-d245-bbd701516719 Address:df5d237b-3e52-babb-d245-bbd701516719} {Suffrage:Voter ID:server-3b40fe9f-29c8-e8ac-af9d-c55d7b09c2f6 Address:3b40fe9f-29c8-e8ac-af9d-c55d7b09c2f6}]" testing.go:149: server-3b40fe9f-29c8-e8ac-af9d-c55d7b09c2f6: 2024-06-12T04:32:29.322+1400 [INFO] server-3b40fe9f-29c8-e8ac-af9d-c55d7b09c2f6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c09d30a2-a922-f086-95e2-c9d01912bed3 Address:c09d30a2-a922-f086-95e2-c9d01912bed3} {Suffrage:Voter ID:server-df5d237b-3e52-babb-d245-bbd701516719 Address:df5d237b-3e52-babb-d245-bbd701516719} {Suffrage:Voter ID:server-3b40fe9f-29c8-e8ac-af9d-c55d7b09c2f6 Address:3b40fe9f-29c8-e8ac-af9d-c55d7b09c2f6}]" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:29.322+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-12T04:32:29.322+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-c09d30a2-a922-f086-95e2-c9d01912bed3: 2024-06-12T04:32:29.322+1400 [INFO] server-c09d30a2-a922-f086-95e2-c9d01912bed3: entering follower state: follower="Node at c09d30a2-a922-f086-95e2-c9d01912bed3 [Follower]" leader= testing.go:149: server-c09d30a2-a922-f086-95e2-c9d01912bed3: 2024-06-12T04:32:29.322+1400 [INFO] server-c09d30a2-a922-f086-95e2-c9d01912bed3: entering follower state: follower="Node at c09d30a2-a922-f086-95e2-c9d01912bed3 [Follower]" leader= testLoggerAdapter verbose: server-3b40fe9f-29c8-e8ac-af9d-c55d7b09c2f6: 2024-06-12T04:32:29.322+1400 [INFO] server-3b40fe9f-29c8-e8ac-af9d-c55d7b09c2f6: entering follower state: follower="Node at 3b40fe9f-29c8-e8ac-af9d-c55d7b09c2f6 [Follower]" leader= testing.go:149: server-3b40fe9f-29c8-e8ac-af9d-c55d7b09c2f6: 2024-06-12T04:32:29.322+1400 [INFO] server-3b40fe9f-29c8-e8ac-af9d-c55d7b09c2f6: entering follower state: follower="Node at 3b40fe9f-29c8-e8ac-af9d-c55d7b09c2f6 [Follower]" leader= testLoggerAdapter verbose: server-df5d237b-3e52-babb-d245-bbd701516719: 2024-06-12T04:32:29.322+1400 [INFO] server-df5d237b-3e52-babb-d245-bbd701516719: entering follower state: follower="Node at df5d237b-3e52-babb-d245-bbd701516719 [Follower]" leader= testing.go:149: server-df5d237b-3e52-babb-d245-bbd701516719: 2024-06-12T04:32:29.322+1400 [INFO] server-df5d237b-3e52-babb-d245-bbd701516719: entering follower state: follower="Node at df5d237b-3e52-babb-d245-bbd701516719 [Follower]" leader= testLoggerAdapter verbose: server-c09d30a2-a922-f086-95e2-c9d01912bed3: 2024-06-12T04:32:29.387+1400 [WARN] server-c09d30a2-a922-f086-95e2-c9d01912bed3: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-c09d30a2-a922-f086-95e2-c9d01912bed3: 2024-06-12T04:32:29.387+1400 [WARN] server-c09d30a2-a922-f086-95e2-c9d01912bed3: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-c09d30a2-a922-f086-95e2-c9d01912bed3: 2024-06-12T04:32:29.387+1400 [INFO] server-c09d30a2-a922-f086-95e2-c9d01912bed3: entering candidate state: node="Node at c09d30a2-a922-f086-95e2-c9d01912bed3 [Candidate]" term=2 testing.go:149: server-c09d30a2-a922-f086-95e2-c9d01912bed3: 2024-06-12T04:32:29.387+1400 [INFO] server-c09d30a2-a922-f086-95e2-c9d01912bed3: entering candidate state: node="Node at c09d30a2-a922-f086-95e2-c9d01912bed3 [Candidate]" term=2 testLoggerAdapter verbose: server-c09d30a2-a922-f086-95e2-c9d01912bed3: 2024-06-12T04:32:29.387+1400 [INFO] server-c09d30a2-a922-f086-95e2-c9d01912bed3: election won: tally=2 testing.go:149: server-c09d30a2-a922-f086-95e2-c9d01912bed3: 2024-06-12T04:32:29.387+1400 [INFO] server-c09d30a2-a922-f086-95e2-c9d01912bed3: election won: tally=2 testLoggerAdapter verbose: server-c09d30a2-a922-f086-95e2-c9d01912bed3: 2024-06-12T04:32:29.387+1400 [INFO] server-c09d30a2-a922-f086-95e2-c9d01912bed3: entering leader state: leader="Node at c09d30a2-a922-f086-95e2-c9d01912bed3 [Leader]" testing.go:149: server-c09d30a2-a922-f086-95e2-c9d01912bed3: 2024-06-12T04:32:29.387+1400 [INFO] server-c09d30a2-a922-f086-95e2-c9d01912bed3: entering leader state: leader="Node at c09d30a2-a922-f086-95e2-c9d01912bed3 [Leader]" testLoggerAdapter verbose: server-c09d30a2-a922-f086-95e2-c9d01912bed3: 2024-06-12T04:32:29.387+1400 [INFO] server-c09d30a2-a922-f086-95e2-c9d01912bed3: added peer, starting replication: peer=server-df5d237b-3e52-babb-d245-bbd701516719 testing.go:149: server-c09d30a2-a922-f086-95e2-c9d01912bed3: 2024-06-12T04:32:29.387+1400 [INFO] server-c09d30a2-a922-f086-95e2-c9d01912bed3: added peer, starting replication: peer=server-df5d237b-3e52-babb-d245-bbd701516719 testLoggerAdapter verbose: server-c09d30a2-a922-f086-95e2-c9d01912bed3: 2024-06-12T04:32:29.387+1400 [INFO] server-c09d30a2-a922-f086-95e2-c9d01912bed3: added peer, starting replication: peer=server-3b40fe9f-29c8-e8ac-af9d-c55d7b09c2f6 testing.go:149: server-c09d30a2-a922-f086-95e2-c9d01912bed3: 2024-06-12T04:32:29.387+1400 [INFO] server-c09d30a2-a922-f086-95e2-c9d01912bed3: added peer, starting replication: peer=server-3b40fe9f-29c8-e8ac-af9d-c55d7b09c2f6 testLoggerAdapter verbose: server-c09d30a2-a922-f086-95e2-c9d01912bed3: 2024-06-12T04:32:29.387+1400 [INFO] server-c09d30a2-a922-f086-95e2-c9d01912bed3: pipelining replication: peer="{Voter server-df5d237b-3e52-babb-d245-bbd701516719 df5d237b-3e52-babb-d245-bbd701516719}" testing.go:149: server-c09d30a2-a922-f086-95e2-c9d01912bed3: 2024-06-12T04:32:29.387+1400 [INFO] server-c09d30a2-a922-f086-95e2-c9d01912bed3: pipelining replication: peer="{Voter server-df5d237b-3e52-babb-d245-bbd701516719 df5d237b-3e52-babb-d245-bbd701516719}" testLoggerAdapter verbose: server-c09d30a2-a922-f086-95e2-c9d01912bed3: 2024-06-12T04:32:29.388+1400 [INFO] server-c09d30a2-a922-f086-95e2-c9d01912bed3: pipelining replication: peer="{Voter server-3b40fe9f-29c8-e8ac-af9d-c55d7b09c2f6 3b40fe9f-29c8-e8ac-af9d-c55d7b09c2f6}" testing.go:149: server-c09d30a2-a922-f086-95e2-c9d01912bed3: 2024-06-12T04:32:29.388+1400 [INFO] server-c09d30a2-a922-f086-95e2-c9d01912bed3: pipelining replication: peer="{Voter server-3b40fe9f-29c8-e8ac-af9d-c55d7b09c2f6 3b40fe9f-29c8-e8ac-af9d-c55d7b09c2f6}" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:29.493+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:29.38781417 +1400 +14 m=+6.295332749 (1 nodes), 64.989352ms from start of poll, 66.110952ms from cluster start. Timeout at 2024-06-12 04:32:29.493551783 +1400 +14 m=+6.401070362, 105.737613ms after stability testing.go:149: cluster: 2024-06-12T04:32:29.493+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:29.38781417 +1400 +14 m=+6.295332749 (1 nodes), 64.989352ms from start of poll, 66.110952ms from cluster start. Timeout at 2024-06-12 04:32:29.493551783 +1400 +14 m=+6.401070362, 105.737613ms after stability testLoggerAdapter verbose: cluster: 2024-06-12T04:32:29.493+1400 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-06-12T04:32:29.493+1400 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2024-06-12T04:32:29.599+1400 [INFO] stable state for Follower reached at 2024-06-12 04:32:29.493748656 +1400 +14 m=+6.401267235 (2 nodes), 2.12µs from start of poll, 172.045438ms from cluster start. Timeout at 2024-06-12 04:32:29.599256717 +1400 +14 m=+6.506775336, 105.508101ms after stability testing.go:149: cluster: 2024-06-12T04:32:29.599+1400 [INFO] stable state for Follower reached at 2024-06-12 04:32:29.493748656 +1400 +14 m=+6.401267235 (2 nodes), 2.12µs from start of poll, 172.045438ms from cluster start. Timeout at 2024-06-12 04:32:29.599256717 +1400 +14 m=+6.506775336, 105.508101ms after stability testLoggerAdapter verbose: server-c09d30a2-a922-f086-95e2-c9d01912bed3: 2024-06-12T04:32:29.599+1400 [INFO] server-c09d30a2-a922-f086-95e2-c9d01912bed3: aborting pipeline replication: peer="{Voter server-df5d237b-3e52-babb-d245-bbd701516719 df5d237b-3e52-babb-d245-bbd701516719}" testing.go:149: server-c09d30a2-a922-f086-95e2-c9d01912bed3: 2024-06-12T04:32:29.599+1400 [INFO] server-c09d30a2-a922-f086-95e2-c9d01912bed3: aborting pipeline replication: peer="{Voter server-df5d237b-3e52-babb-d245-bbd701516719 df5d237b-3e52-babb-d245-bbd701516719}" testLoggerAdapter verbose: server-c09d30a2-a922-f086-95e2-c9d01912bed3: 2024-06-12T04:32:29.599+1400 [INFO] server-c09d30a2-a922-f086-95e2-c9d01912bed3: aborting pipeline replication: peer="{Voter server-3b40fe9f-29c8-e8ac-af9d-c55d7b09c2f6 3b40fe9f-29c8-e8ac-af9d-c55d7b09c2f6}" testing.go:149: server-c09d30a2-a922-f086-95e2-c9d01912bed3: 2024-06-12T04:32:29.599+1400 [INFO] server-c09d30a2-a922-f086-95e2-c9d01912bed3: aborting pipeline replication: peer="{Voter server-3b40fe9f-29c8-e8ac-af9d-c55d7b09c2f6 3b40fe9f-29c8-e8ac-af9d-c55d7b09c2f6}" --- PASS: TestRaft_ApplyNonLeader (0.28s) === RUN TestRaft_ApplyConcurrent testLoggerAdapter verbose: server-725f2e99-44df-9a1d-09a4-87a804953151: 2024-06-12T04:32:29.602+1400 [INFO] server-725f2e99-44df-9a1d-09a4-87a804953151: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-725f2e99-44df-9a1d-09a4-87a804953151 Address:725f2e99-44df-9a1d-09a4-87a804953151} {Suffrage:Voter ID:server-7c0e0a89-0af6-24b8-0327-bef42f52a8bb Address:7c0e0a89-0af6-24b8-0327-bef42f52a8bb} {Suffrage:Voter ID:server-68002a6e-c0fd-f70a-0a4d-d065e1609f17 Address:68002a6e-c0fd-f70a-0a4d-d065e1609f17}]" testing.go:149: server-725f2e99-44df-9a1d-09a4-87a804953151: 2024-06-12T04:32:29.602+1400 [INFO] server-725f2e99-44df-9a1d-09a4-87a804953151: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-725f2e99-44df-9a1d-09a4-87a804953151 Address:725f2e99-44df-9a1d-09a4-87a804953151} {Suffrage:Voter ID:server-7c0e0a89-0af6-24b8-0327-bef42f52a8bb Address:7c0e0a89-0af6-24b8-0327-bef42f52a8bb} {Suffrage:Voter ID:server-68002a6e-c0fd-f70a-0a4d-d065e1609f17 Address:68002a6e-c0fd-f70a-0a4d-d065e1609f17}]" testLoggerAdapter verbose: server-7c0e0a89-0af6-24b8-0327-bef42f52a8bb: 2024-06-12T04:32:29.603+1400 [INFO] server-7c0e0a89-0af6-24b8-0327-bef42f52a8bb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-725f2e99-44df-9a1d-09a4-87a804953151 Address:725f2e99-44df-9a1d-09a4-87a804953151} {Suffrage:Voter ID:server-7c0e0a89-0af6-24b8-0327-bef42f52a8bb Address:7c0e0a89-0af6-24b8-0327-bef42f52a8bb} {Suffrage:Voter ID:server-68002a6e-c0fd-f70a-0a4d-d065e1609f17 Address:68002a6e-c0fd-f70a-0a4d-d065e1609f17}]" testing.go:149: server-7c0e0a89-0af6-24b8-0327-bef42f52a8bb: 2024-06-12T04:32:29.603+1400 [INFO] server-7c0e0a89-0af6-24b8-0327-bef42f52a8bb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-725f2e99-44df-9a1d-09a4-87a804953151 Address:725f2e99-44df-9a1d-09a4-87a804953151} {Suffrage:Voter ID:server-7c0e0a89-0af6-24b8-0327-bef42f52a8bb Address:7c0e0a89-0af6-24b8-0327-bef42f52a8bb} {Suffrage:Voter ID:server-68002a6e-c0fd-f70a-0a4d-d065e1609f17 Address:68002a6e-c0fd-f70a-0a4d-d065e1609f17}]" testLoggerAdapter verbose: server-68002a6e-c0fd-f70a-0a4d-d065e1609f17: 2024-06-12T04:32:29.603+1400 [INFO] server-68002a6e-c0fd-f70a-0a4d-d065e1609f17: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-725f2e99-44df-9a1d-09a4-87a804953151 Address:725f2e99-44df-9a1d-09a4-87a804953151} {Suffrage:Voter ID:server-7c0e0a89-0af6-24b8-0327-bef42f52a8bb Address:7c0e0a89-0af6-24b8-0327-bef42f52a8bb} {Suffrage:Voter ID:server-68002a6e-c0fd-f70a-0a4d-d065e1609f17 Address:68002a6e-c0fd-f70a-0a4d-d065e1609f17}]" testing.go:149: server-68002a6e-c0fd-f70a-0a4d-d065e1609f17: 2024-06-12T04:32:29.603+1400 [INFO] server-68002a6e-c0fd-f70a-0a4d-d065e1609f17: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-725f2e99-44df-9a1d-09a4-87a804953151 Address:725f2e99-44df-9a1d-09a4-87a804953151} {Suffrage:Voter ID:server-7c0e0a89-0af6-24b8-0327-bef42f52a8bb Address:7c0e0a89-0af6-24b8-0327-bef42f52a8bb} {Suffrage:Voter ID:server-68002a6e-c0fd-f70a-0a4d-d065e1609f17 Address:68002a6e-c0fd-f70a-0a4d-d065e1609f17}]" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:29.603+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-12T04:32:29.603+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-7c0e0a89-0af6-24b8-0327-bef42f52a8bb: 2024-06-12T04:32:29.603+1400 [INFO] server-7c0e0a89-0af6-24b8-0327-bef42f52a8bb: entering follower state: follower="Node at 7c0e0a89-0af6-24b8-0327-bef42f52a8bb [Follower]" leader= testing.go:149: server-7c0e0a89-0af6-24b8-0327-bef42f52a8bb: 2024-06-12T04:32:29.603+1400 [INFO] server-7c0e0a89-0af6-24b8-0327-bef42f52a8bb: entering follower state: follower="Node at 7c0e0a89-0af6-24b8-0327-bef42f52a8bb [Follower]" leader= testLoggerAdapter verbose: server-725f2e99-44df-9a1d-09a4-87a804953151: 2024-06-12T04:32:29.603+1400 [INFO] server-725f2e99-44df-9a1d-09a4-87a804953151: entering follower state: follower="Node at 725f2e99-44df-9a1d-09a4-87a804953151 [Follower]" leader= testing.go:149: server-725f2e99-44df-9a1d-09a4-87a804953151: 2024-06-12T04:32:29.603+1400 [INFO] server-725f2e99-44df-9a1d-09a4-87a804953151: entering follower state: follower="Node at 725f2e99-44df-9a1d-09a4-87a804953151 [Follower]" leader= testLoggerAdapter verbose: server-68002a6e-c0fd-f70a-0a4d-d065e1609f17: 2024-06-12T04:32:29.604+1400 [INFO] server-68002a6e-c0fd-f70a-0a4d-d065e1609f17: entering follower state: follower="Node at 68002a6e-c0fd-f70a-0a4d-d065e1609f17 [Follower]" leader= testing.go:149: server-68002a6e-c0fd-f70a-0a4d-d065e1609f17: 2024-06-12T04:32:29.604+1400 [INFO] server-68002a6e-c0fd-f70a-0a4d-d065e1609f17: entering follower state: follower="Node at 68002a6e-c0fd-f70a-0a4d-d065e1609f17 [Follower]" leader= testLoggerAdapter verbose: server-725f2e99-44df-9a1d-09a4-87a804953151: 2024-06-12T04:32:29.713+1400 [WARN] server-725f2e99-44df-9a1d-09a4-87a804953151: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-725f2e99-44df-9a1d-09a4-87a804953151: 2024-06-12T04:32:29.713+1400 [WARN] server-725f2e99-44df-9a1d-09a4-87a804953151: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-725f2e99-44df-9a1d-09a4-87a804953151: 2024-06-12T04:32:29.713+1400 [INFO] server-725f2e99-44df-9a1d-09a4-87a804953151: entering candidate state: node="Node at 725f2e99-44df-9a1d-09a4-87a804953151 [Candidate]" term=2 testing.go:149: server-725f2e99-44df-9a1d-09a4-87a804953151: 2024-06-12T04:32:29.713+1400 [INFO] server-725f2e99-44df-9a1d-09a4-87a804953151: entering candidate state: node="Node at 725f2e99-44df-9a1d-09a4-87a804953151 [Candidate]" term=2 testLoggerAdapter verbose: server-725f2e99-44df-9a1d-09a4-87a804953151: 2024-06-12T04:32:29.713+1400 [INFO] server-725f2e99-44df-9a1d-09a4-87a804953151: election won: tally=2 testing.go:149: server-725f2e99-44df-9a1d-09a4-87a804953151: 2024-06-12T04:32:29.713+1400 [INFO] server-725f2e99-44df-9a1d-09a4-87a804953151: election won: tally=2 testLoggerAdapter verbose: server-725f2e99-44df-9a1d-09a4-87a804953151: 2024-06-12T04:32:29.713+1400 [INFO] server-725f2e99-44df-9a1d-09a4-87a804953151: entering leader state: leader="Node at 725f2e99-44df-9a1d-09a4-87a804953151 [Leader]" testing.go:149: server-725f2e99-44df-9a1d-09a4-87a804953151: 2024-06-12T04:32:29.713+1400 [INFO] server-725f2e99-44df-9a1d-09a4-87a804953151: entering leader state: leader="Node at 725f2e99-44df-9a1d-09a4-87a804953151 [Leader]" testLoggerAdapter verbose: server-725f2e99-44df-9a1d-09a4-87a804953151: 2024-06-12T04:32:29.713+1400 [INFO] server-725f2e99-44df-9a1d-09a4-87a804953151: added peer, starting replication: peer=server-7c0e0a89-0af6-24b8-0327-bef42f52a8bb testing.go:149: server-725f2e99-44df-9a1d-09a4-87a804953151: 2024-06-12T04:32:29.713+1400 [INFO] server-725f2e99-44df-9a1d-09a4-87a804953151: added peer, starting replication: peer=server-7c0e0a89-0af6-24b8-0327-bef42f52a8bb testLoggerAdapter verbose: server-725f2e99-44df-9a1d-09a4-87a804953151: 2024-06-12T04:32:29.713+1400 [INFO] server-725f2e99-44df-9a1d-09a4-87a804953151: added peer, starting replication: peer=server-68002a6e-c0fd-f70a-0a4d-d065e1609f17 testing.go:149: server-725f2e99-44df-9a1d-09a4-87a804953151: 2024-06-12T04:32:29.713+1400 [INFO] server-725f2e99-44df-9a1d-09a4-87a804953151: added peer, starting replication: peer=server-68002a6e-c0fd-f70a-0a4d-d065e1609f17 testLoggerAdapter verbose: server-725f2e99-44df-9a1d-09a4-87a804953151: 2024-06-12T04:32:29.713+1400 [INFO] server-725f2e99-44df-9a1d-09a4-87a804953151: pipelining replication: peer="{Voter server-68002a6e-c0fd-f70a-0a4d-d065e1609f17 68002a6e-c0fd-f70a-0a4d-d065e1609f17}" testing.go:149: server-725f2e99-44df-9a1d-09a4-87a804953151: 2024-06-12T04:32:29.713+1400 [INFO] server-725f2e99-44df-9a1d-09a4-87a804953151: pipelining replication: peer="{Voter server-68002a6e-c0fd-f70a-0a4d-d065e1609f17 68002a6e-c0fd-f70a-0a4d-d065e1609f17}" testLoggerAdapter verbose: server-725f2e99-44df-9a1d-09a4-87a804953151: 2024-06-12T04:32:29.713+1400 [INFO] server-725f2e99-44df-9a1d-09a4-87a804953151: pipelining replication: peer="{Voter server-7c0e0a89-0af6-24b8-0327-bef42f52a8bb 7c0e0a89-0af6-24b8-0327-bef42f52a8bb}" testing.go:149: server-725f2e99-44df-9a1d-09a4-87a804953151: 2024-06-12T04:32:29.713+1400 [INFO] server-725f2e99-44df-9a1d-09a4-87a804953151: pipelining replication: peer="{Voter server-7c0e0a89-0af6-24b8-0327-bef42f52a8bb 7c0e0a89-0af6-24b8-0327-bef42f52a8bb}" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:29.919+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:29.713865931 +1400 +14 m=+6.621384550 (1 nodes), 110.413245ms from start of poll, 111.325972ms from cluster start. Timeout at 2024-06-12 04:32:29.919301213 +1400 +14 m=+6.826819832, 205.435282ms after stability testing.go:149: cluster: 2024-06-12T04:32:29.919+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:29.713865931 +1400 +14 m=+6.621384550 (1 nodes), 110.413245ms from start of poll, 111.325972ms from cluster start. Timeout at 2024-06-12 04:32:29.919301213 +1400 +14 m=+6.826819832, 205.435282ms after stability testLoggerAdapter verbose: server-725f2e99-44df-9a1d-09a4-87a804953151: 2024-06-12T04:32:29.935+1400 [INFO] server-725f2e99-44df-9a1d-09a4-87a804953151: aborting pipeline replication: peer="{Voter server-68002a6e-c0fd-f70a-0a4d-d065e1609f17 68002a6e-c0fd-f70a-0a4d-d065e1609f17}" testing.go:149: server-725f2e99-44df-9a1d-09a4-87a804953151: 2024-06-12T04:32:29.935+1400 [INFO] server-725f2e99-44df-9a1d-09a4-87a804953151: aborting pipeline replication: peer="{Voter server-68002a6e-c0fd-f70a-0a4d-d065e1609f17 68002a6e-c0fd-f70a-0a4d-d065e1609f17}" testLoggerAdapter verbose: server-725f2e99-44df-9a1d-09a4-87a804953151: 2024-06-12T04:32:29.935+1400 [INFO] server-725f2e99-44df-9a1d-09a4-87a804953151: aborting pipeline replication: peer="{Voter server-7c0e0a89-0af6-24b8-0327-bef42f52a8bb 7c0e0a89-0af6-24b8-0327-bef42f52a8bb}" testing.go:149: server-725f2e99-44df-9a1d-09a4-87a804953151: 2024-06-12T04:32:29.935+1400 [INFO] server-725f2e99-44df-9a1d-09a4-87a804953151: aborting pipeline replication: peer="{Voter server-7c0e0a89-0af6-24b8-0327-bef42f52a8bb 7c0e0a89-0af6-24b8-0327-bef42f52a8bb}" --- PASS: TestRaft_ApplyConcurrent (0.34s) === RUN TestRaft_ApplyConcurrent_Timeout testLoggerAdapter verbose: server-8a8dab93-58bf-e07a-4145-b749878e27e4: 2024-06-12T04:32:29.937+1400 [INFO] server-8a8dab93-58bf-e07a-4145-b749878e27e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a8dab93-58bf-e07a-4145-b749878e27e4 Address:8a8dab93-58bf-e07a-4145-b749878e27e4}]" testing.go:149: server-8a8dab93-58bf-e07a-4145-b749878e27e4: 2024-06-12T04:32:29.937+1400 [INFO] server-8a8dab93-58bf-e07a-4145-b749878e27e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a8dab93-58bf-e07a-4145-b749878e27e4 Address:8a8dab93-58bf-e07a-4145-b749878e27e4}]" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:29.937+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-12T04:32:29.937+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-8a8dab93-58bf-e07a-4145-b749878e27e4: 2024-06-12T04:32:29.937+1400 [INFO] server-8a8dab93-58bf-e07a-4145-b749878e27e4: entering follower state: follower="Node at 8a8dab93-58bf-e07a-4145-b749878e27e4 [Follower]" leader= testing.go:149: server-8a8dab93-58bf-e07a-4145-b749878e27e4: 2024-06-12T04:32:29.937+1400 [INFO] server-8a8dab93-58bf-e07a-4145-b749878e27e4: entering follower state: follower="Node at 8a8dab93-58bf-e07a-4145-b749878e27e4 [Follower]" leader= testLoggerAdapter verbose: server-8a8dab93-58bf-e07a-4145-b749878e27e4: 2024-06-12T04:32:30.077+1400 [WARN] server-8a8dab93-58bf-e07a-4145-b749878e27e4: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-8a8dab93-58bf-e07a-4145-b749878e27e4: 2024-06-12T04:32:30.077+1400 [WARN] server-8a8dab93-58bf-e07a-4145-b749878e27e4: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-8a8dab93-58bf-e07a-4145-b749878e27e4: 2024-06-12T04:32:30.077+1400 [INFO] server-8a8dab93-58bf-e07a-4145-b749878e27e4: entering candidate state: node="Node at 8a8dab93-58bf-e07a-4145-b749878e27e4 [Candidate]" term=2 testing.go:149: server-8a8dab93-58bf-e07a-4145-b749878e27e4: 2024-06-12T04:32:30.077+1400 [INFO] server-8a8dab93-58bf-e07a-4145-b749878e27e4: entering candidate state: node="Node at 8a8dab93-58bf-e07a-4145-b749878e27e4 [Candidate]" term=2 testLoggerAdapter verbose: server-8a8dab93-58bf-e07a-4145-b749878e27e4: 2024-06-12T04:32:30.077+1400 [INFO] server-8a8dab93-58bf-e07a-4145-b749878e27e4: election won: tally=1 testing.go:149: server-8a8dab93-58bf-e07a-4145-b749878e27e4: 2024-06-12T04:32:30.077+1400 [INFO] server-8a8dab93-58bf-e07a-4145-b749878e27e4: election won: tally=1 testLoggerAdapter verbose: server-8a8dab93-58bf-e07a-4145-b749878e27e4: 2024-06-12T04:32:30.077+1400 [INFO] server-8a8dab93-58bf-e07a-4145-b749878e27e4: entering leader state: leader="Node at 8a8dab93-58bf-e07a-4145-b749878e27e4 [Leader]" testing.go:149: server-8a8dab93-58bf-e07a-4145-b749878e27e4: 2024-06-12T04:32:30.077+1400 [INFO] server-8a8dab93-58bf-e07a-4145-b749878e27e4: entering leader state: leader="Node at 8a8dab93-58bf-e07a-4145-b749878e27e4 [Leader]" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:30.278+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:30.077638262 +1400 +14 m=+6.985156841 (1 nodes), 140.120022ms from start of poll, 140.393692ms from cluster start. Timeout at 2024-06-12 04:32:30.278763577 +1400 +14 m=+7.186282156, 201.125315ms after stability testing.go:149: cluster: 2024-06-12T04:32:30.278+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:30.077638262 +1400 +14 m=+6.985156841 (1 nodes), 140.120022ms from start of poll, 140.393692ms from cluster start. Timeout at 2024-06-12 04:32:30.278763577 +1400 +14 m=+7.186282156, 201.125315ms after stability --- PASS: TestRaft_ApplyConcurrent_Timeout (0.34s) === RUN TestRaft_JoinNode testLoggerAdapter verbose: server-ba5ea0b5-da8f-f7a4-c2b0-6369b9df7d9d: 2024-06-12T04:32:30.283+1400 [INFO] server-ba5ea0b5-da8f-f7a4-c2b0-6369b9df7d9d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ba5ea0b5-da8f-f7a4-c2b0-6369b9df7d9d Address:ba5ea0b5-da8f-f7a4-c2b0-6369b9df7d9d} {Suffrage:Voter ID:server-adab6b73-3013-8559-f2d7-22591934104c Address:adab6b73-3013-8559-f2d7-22591934104c}]" testing.go:149: server-ba5ea0b5-da8f-f7a4-c2b0-6369b9df7d9d: 2024-06-12T04:32:30.283+1400 [INFO] server-ba5ea0b5-da8f-f7a4-c2b0-6369b9df7d9d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ba5ea0b5-da8f-f7a4-c2b0-6369b9df7d9d Address:ba5ea0b5-da8f-f7a4-c2b0-6369b9df7d9d} {Suffrage:Voter ID:server-adab6b73-3013-8559-f2d7-22591934104c Address:adab6b73-3013-8559-f2d7-22591934104c}]" testLoggerAdapter verbose: server-adab6b73-3013-8559-f2d7-22591934104c: 2024-06-12T04:32:30.284+1400 [INFO] server-adab6b73-3013-8559-f2d7-22591934104c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ba5ea0b5-da8f-f7a4-c2b0-6369b9df7d9d Address:ba5ea0b5-da8f-f7a4-c2b0-6369b9df7d9d} {Suffrage:Voter ID:server-adab6b73-3013-8559-f2d7-22591934104c Address:adab6b73-3013-8559-f2d7-22591934104c}]" testing.go:149: server-adab6b73-3013-8559-f2d7-22591934104c: 2024-06-12T04:32:30.284+1400 [INFO] server-adab6b73-3013-8559-f2d7-22591934104c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ba5ea0b5-da8f-f7a4-c2b0-6369b9df7d9d Address:ba5ea0b5-da8f-f7a4-c2b0-6369b9df7d9d} {Suffrage:Voter ID:server-adab6b73-3013-8559-f2d7-22591934104c Address:adab6b73-3013-8559-f2d7-22591934104c}]" testLoggerAdapter verbose: server-d56af706-964d-9927-fc35-25334e2b1781: 2024-06-12T04:32:30.284+1400 [INFO] server-d56af706-964d-9927-fc35-25334e2b1781: initial configuration: index=0 servers=[] testing.go:149: server-d56af706-964d-9927-fc35-25334e2b1781: 2024-06-12T04:32:30.284+1400 [INFO] server-d56af706-964d-9927-fc35-25334e2b1781: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2024-06-12T04:32:30.284+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-12T04:32:30.284+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-ba5ea0b5-da8f-f7a4-c2b0-6369b9df7d9d: 2024-06-12T04:32:30.284+1400 [INFO] server-ba5ea0b5-da8f-f7a4-c2b0-6369b9df7d9d: entering follower state: follower="Node at ba5ea0b5-da8f-f7a4-c2b0-6369b9df7d9d [Follower]" leader= testing.go:149: server-ba5ea0b5-da8f-f7a4-c2b0-6369b9df7d9d: 2024-06-12T04:32:30.284+1400 [INFO] server-ba5ea0b5-da8f-f7a4-c2b0-6369b9df7d9d: entering follower state: follower="Node at ba5ea0b5-da8f-f7a4-c2b0-6369b9df7d9d [Follower]" leader= testLoggerAdapter verbose: server-adab6b73-3013-8559-f2d7-22591934104c: 2024-06-12T04:32:30.284+1400 [INFO] server-adab6b73-3013-8559-f2d7-22591934104c: entering follower state: follower="Node at adab6b73-3013-8559-f2d7-22591934104c [Follower]" leader= testing.go:149: server-adab6b73-3013-8559-f2d7-22591934104c: 2024-06-12T04:32:30.284+1400 [INFO] server-adab6b73-3013-8559-f2d7-22591934104c: entering follower state: follower="Node at adab6b73-3013-8559-f2d7-22591934104c [Follower]" leader= testLoggerAdapter verbose: server-d56af706-964d-9927-fc35-25334e2b1781: 2024-06-12T04:32:30.284+1400 [INFO] server-d56af706-964d-9927-fc35-25334e2b1781: entering follower state: follower="Node at d56af706-964d-9927-fc35-25334e2b1781 [Follower]" leader= testing.go:149: server-d56af706-964d-9927-fc35-25334e2b1781: 2024-06-12T04:32:30.284+1400 [INFO] server-d56af706-964d-9927-fc35-25334e2b1781: entering follower state: follower="Node at d56af706-964d-9927-fc35-25334e2b1781 [Follower]" leader= testLoggerAdapter verbose: server-adab6b73-3013-8559-f2d7-22591934104c: 2024-06-12T04:32:30.370+1400 [WARN] server-adab6b73-3013-8559-f2d7-22591934104c: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-adab6b73-3013-8559-f2d7-22591934104c: 2024-06-12T04:32:30.370+1400 [WARN] server-adab6b73-3013-8559-f2d7-22591934104c: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-adab6b73-3013-8559-f2d7-22591934104c: 2024-06-12T04:32:30.370+1400 [INFO] server-adab6b73-3013-8559-f2d7-22591934104c: entering candidate state: node="Node at adab6b73-3013-8559-f2d7-22591934104c [Candidate]" term=2 testing.go:149: server-adab6b73-3013-8559-f2d7-22591934104c: 2024-06-12T04:32:30.370+1400 [INFO] server-adab6b73-3013-8559-f2d7-22591934104c: entering candidate state: node="Node at adab6b73-3013-8559-f2d7-22591934104c [Candidate]" term=2 testLoggerAdapter verbose: server-adab6b73-3013-8559-f2d7-22591934104c: 2024-06-12T04:32:30.370+1400 [INFO] server-adab6b73-3013-8559-f2d7-22591934104c: election won: tally=2 testing.go:149: server-adab6b73-3013-8559-f2d7-22591934104c: 2024-06-12T04:32:30.370+1400 [INFO] server-adab6b73-3013-8559-f2d7-22591934104c: election won: tally=2 testLoggerAdapter verbose: server-adab6b73-3013-8559-f2d7-22591934104c: 2024-06-12T04:32:30.370+1400 [INFO] server-adab6b73-3013-8559-f2d7-22591934104c: entering leader state: leader="Node at adab6b73-3013-8559-f2d7-22591934104c [Leader]" testing.go:149: server-adab6b73-3013-8559-f2d7-22591934104c: 2024-06-12T04:32:30.370+1400 [INFO] server-adab6b73-3013-8559-f2d7-22591934104c: entering leader state: leader="Node at adab6b73-3013-8559-f2d7-22591934104c [Leader]" testLoggerAdapter verbose: server-adab6b73-3013-8559-f2d7-22591934104c: 2024-06-12T04:32:30.370+1400 [INFO] server-adab6b73-3013-8559-f2d7-22591934104c: added peer, starting replication: peer=server-ba5ea0b5-da8f-f7a4-c2b0-6369b9df7d9d testing.go:149: server-adab6b73-3013-8559-f2d7-22591934104c: 2024-06-12T04:32:30.370+1400 [INFO] server-adab6b73-3013-8559-f2d7-22591934104c: added peer, starting replication: peer=server-ba5ea0b5-da8f-f7a4-c2b0-6369b9df7d9d testLoggerAdapter verbose: server-adab6b73-3013-8559-f2d7-22591934104c: 2024-06-12T04:32:30.370+1400 [INFO] server-adab6b73-3013-8559-f2d7-22591934104c: pipelining replication: peer="{Voter server-ba5ea0b5-da8f-f7a4-c2b0-6369b9df7d9d ba5ea0b5-da8f-f7a4-c2b0-6369b9df7d9d}" testing.go:149: server-adab6b73-3013-8559-f2d7-22591934104c: 2024-06-12T04:32:30.370+1400 [INFO] server-adab6b73-3013-8559-f2d7-22591934104c: pipelining replication: peer="{Voter server-ba5ea0b5-da8f-f7a4-c2b0-6369b9df7d9d ba5ea0b5-da8f-f7a4-c2b0-6369b9df7d9d}" testLoggerAdapter verbose: server-d56af706-964d-9927-fc35-25334e2b1781: 2024-06-12T04:32:30.377+1400 [WARN] server-d56af706-964d-9927-fc35-25334e2b1781: no known peers, aborting election testing.go:149: server-d56af706-964d-9927-fc35-25334e2b1781: 2024-06-12T04:32:30.377+1400 [WARN] server-d56af706-964d-9927-fc35-25334e2b1781: no known peers, aborting election testLoggerAdapter verbose: cluster: 2024-06-12T04:32:30.476+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:30.370872918 +1400 +14 m=+7.278391537 (1 nodes), 86.227631ms from start of poll, 87.128599ms from cluster start. Timeout at 2024-06-12 04:32:30.47690308 +1400 +14 m=+7.384421699, 106.030162ms after stability testing.go:149: cluster: 2024-06-12T04:32:30.476+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:30.370872918 +1400 +14 m=+7.278391537 (1 nodes), 86.227631ms from start of poll, 87.128599ms from cluster start. Timeout at 2024-06-12 04:32:30.47690308 +1400 +14 m=+7.384421699, 106.030162ms after stability testLoggerAdapter verbose: server-adab6b73-3013-8559-f2d7-22591934104c: 2024-06-12T04:32:30.477+1400 [INFO] server-adab6b73-3013-8559-f2d7-22591934104c: updating configuration: command=AddStaging server-id=server-d56af706-964d-9927-fc35-25334e2b1781 server-addr=d56af706-964d-9927-fc35-25334e2b1781 servers="[{Suffrage:Voter ID:server-ba5ea0b5-da8f-f7a4-c2b0-6369b9df7d9d Address:ba5ea0b5-da8f-f7a4-c2b0-6369b9df7d9d} {Suffrage:Voter ID:server-adab6b73-3013-8559-f2d7-22591934104c Address:adab6b73-3013-8559-f2d7-22591934104c} {Suffrage:Voter ID:server-d56af706-964d-9927-fc35-25334e2b1781 Address:d56af706-964d-9927-fc35-25334e2b1781}]" testing.go:149: server-adab6b73-3013-8559-f2d7-22591934104c: 2024-06-12T04:32:30.477+1400 [INFO] server-adab6b73-3013-8559-f2d7-22591934104c: updating configuration: command=AddStaging server-id=server-d56af706-964d-9927-fc35-25334e2b1781 server-addr=d56af706-964d-9927-fc35-25334e2b1781 servers="[{Suffrage:Voter ID:server-ba5ea0b5-da8f-f7a4-c2b0-6369b9df7d9d Address:ba5ea0b5-da8f-f7a4-c2b0-6369b9df7d9d} {Suffrage:Voter ID:server-adab6b73-3013-8559-f2d7-22591934104c Address:adab6b73-3013-8559-f2d7-22591934104c} {Suffrage:Voter ID:server-d56af706-964d-9927-fc35-25334e2b1781 Address:d56af706-964d-9927-fc35-25334e2b1781}]" testLoggerAdapter verbose: server-adab6b73-3013-8559-f2d7-22591934104c: 2024-06-12T04:32:30.477+1400 [INFO] server-adab6b73-3013-8559-f2d7-22591934104c: added peer, starting replication: peer=server-d56af706-964d-9927-fc35-25334e2b1781 testing.go:149: server-adab6b73-3013-8559-f2d7-22591934104c: 2024-06-12T04:32:30.477+1400 [INFO] server-adab6b73-3013-8559-f2d7-22591934104c: added peer, starting replication: peer=server-d56af706-964d-9927-fc35-25334e2b1781 testLoggerAdapter verbose: server-d56af706-964d-9927-fc35-25334e2b1781: 2024-06-12T04:32:30.477+1400 [WARN] server-d56af706-964d-9927-fc35-25334e2b1781: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: server-d56af706-964d-9927-fc35-25334e2b1781: 2024-06-12T04:32:30.477+1400 [WARN] server-d56af706-964d-9927-fc35-25334e2b1781: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: server-adab6b73-3013-8559-f2d7-22591934104c: 2024-06-12T04:32:30.477+1400 [WARN] server-adab6b73-3013-8559-f2d7-22591934104c: appendEntries rejected, sending older logs: peer="{Voter server-d56af706-964d-9927-fc35-25334e2b1781 d56af706-964d-9927-fc35-25334e2b1781}" next=1 testing.go:149: server-adab6b73-3013-8559-f2d7-22591934104c: 2024-06-12T04:32:30.477+1400 [WARN] server-adab6b73-3013-8559-f2d7-22591934104c: appendEntries rejected, sending older logs: peer="{Voter server-d56af706-964d-9927-fc35-25334e2b1781 d56af706-964d-9927-fc35-25334e2b1781}" next=1 testLoggerAdapter verbose: server-adab6b73-3013-8559-f2d7-22591934104c: 2024-06-12T04:32:30.477+1400 [INFO] server-adab6b73-3013-8559-f2d7-22591934104c: pipelining replication: peer="{Voter server-d56af706-964d-9927-fc35-25334e2b1781 d56af706-964d-9927-fc35-25334e2b1781}" testing.go:149: server-adab6b73-3013-8559-f2d7-22591934104c: 2024-06-12T04:32:30.477+1400 [INFO] server-adab6b73-3013-8559-f2d7-22591934104c: pipelining replication: peer="{Voter server-d56af706-964d-9927-fc35-25334e2b1781 d56af706-964d-9927-fc35-25334e2b1781}" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:30.477+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-12T04:32:30.477+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-06-12T04:32:30.582+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:30.477817807 +1400 +14 m=+7.385336426 (1 nodes), 1.959µs from start of poll, 194.073488ms from cluster start. Timeout at 2024-06-12 04:32:30.582849765 +1400 +14 m=+7.490368344, 105.031918ms after stability testing.go:149: cluster: 2024-06-12T04:32:30.582+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:30.477817807 +1400 +14 m=+7.385336426 (1 nodes), 1.959µs from start of poll, 194.073488ms from cluster start. Timeout at 2024-06-12 04:32:30.582849765 +1400 +14 m=+7.490368344, 105.031918ms after stability testLoggerAdapter verbose: server-adab6b73-3013-8559-f2d7-22591934104c: 2024-06-12T04:32:30.583+1400 [INFO] server-adab6b73-3013-8559-f2d7-22591934104c: aborting pipeline replication: peer="{Voter server-d56af706-964d-9927-fc35-25334e2b1781 d56af706-964d-9927-fc35-25334e2b1781}" testing.go:149: server-adab6b73-3013-8559-f2d7-22591934104c: 2024-06-12T04:32:30.583+1400 [INFO] server-adab6b73-3013-8559-f2d7-22591934104c: aborting pipeline replication: peer="{Voter server-d56af706-964d-9927-fc35-25334e2b1781 d56af706-964d-9927-fc35-25334e2b1781}" testLoggerAdapter verbose: server-adab6b73-3013-8559-f2d7-22591934104c: 2024-06-12T04:32:30.583+1400 [INFO] server-adab6b73-3013-8559-f2d7-22591934104c: aborting pipeline replication: peer="{Voter server-ba5ea0b5-da8f-f7a4-c2b0-6369b9df7d9d ba5ea0b5-da8f-f7a4-c2b0-6369b9df7d9d}" testing.go:149: server-adab6b73-3013-8559-f2d7-22591934104c: 2024-06-12T04:32:30.583+1400 [INFO] server-adab6b73-3013-8559-f2d7-22591934104c: aborting pipeline replication: peer="{Voter server-ba5ea0b5-da8f-f7a4-c2b0-6369b9df7d9d ba5ea0b5-da8f-f7a4-c2b0-6369b9df7d9d}" --- PASS: TestRaft_JoinNode (0.30s) === RUN TestRaft_JoinNode_ConfigStore testLoggerAdapter verbose: server-745c502e-38cb-1b1e-a943-bc158cafcce8: 2024-06-12T04:32:30.584+1400 [INFO] server-745c502e-38cb-1b1e-a943-bc158cafcce8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-745c502e-38cb-1b1e-a943-bc158cafcce8 Address:745c502e-38cb-1b1e-a943-bc158cafcce8}]" testing.go:149: server-745c502e-38cb-1b1e-a943-bc158cafcce8: 2024-06-12T04:32:30.584+1400 [INFO] server-745c502e-38cb-1b1e-a943-bc158cafcce8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-745c502e-38cb-1b1e-a943-bc158cafcce8 Address:745c502e-38cb-1b1e-a943-bc158cafcce8}]" testLoggerAdapter verbose: server-aecede68-0038-d0d2-a0bf-dfc29611e5d7: 2024-06-12T04:32:30.585+1400 [INFO] server-aecede68-0038-d0d2-a0bf-dfc29611e5d7: initial configuration: index=0 servers=[] testing.go:149: server-aecede68-0038-d0d2-a0bf-dfc29611e5d7: 2024-06-12T04:32:30.585+1400 [INFO] server-aecede68-0038-d0d2-a0bf-dfc29611e5d7: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-f3da6f6c-5d28-9436-09ca-c821425fade9: 2024-06-12T04:32:30.585+1400 [INFO] server-f3da6f6c-5d28-9436-09ca-c821425fade9: initial configuration: index=0 servers=[] testing.go:149: server-f3da6f6c-5d28-9436-09ca-c821425fade9: 2024-06-12T04:32:30.585+1400 [INFO] server-f3da6f6c-5d28-9436-09ca-c821425fade9: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2024-06-12T04:32:30.585+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-12T04:32:30.585+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-745c502e-38cb-1b1e-a943-bc158cafcce8: 2024-06-12T04:32:30.585+1400 [INFO] server-745c502e-38cb-1b1e-a943-bc158cafcce8: entering follower state: follower="Node at 745c502e-38cb-1b1e-a943-bc158cafcce8 [Follower]" leader= testing.go:149: server-745c502e-38cb-1b1e-a943-bc158cafcce8: 2024-06-12T04:32:30.585+1400 [INFO] server-745c502e-38cb-1b1e-a943-bc158cafcce8: entering follower state: follower="Node at 745c502e-38cb-1b1e-a943-bc158cafcce8 [Follower]" leader= testLoggerAdapter verbose: server-aecede68-0038-d0d2-a0bf-dfc29611e5d7: 2024-06-12T04:32:30.585+1400 [INFO] server-aecede68-0038-d0d2-a0bf-dfc29611e5d7: entering follower state: follower="Node at aecede68-0038-d0d2-a0bf-dfc29611e5d7 [Follower]" leader= testing.go:149: server-aecede68-0038-d0d2-a0bf-dfc29611e5d7: 2024-06-12T04:32:30.585+1400 [INFO] server-aecede68-0038-d0d2-a0bf-dfc29611e5d7: entering follower state: follower="Node at aecede68-0038-d0d2-a0bf-dfc29611e5d7 [Follower]" leader= testLoggerAdapter verbose: server-f3da6f6c-5d28-9436-09ca-c821425fade9: 2024-06-12T04:32:30.586+1400 [INFO] server-f3da6f6c-5d28-9436-09ca-c821425fade9: entering follower state: follower="Node at f3da6f6c-5d28-9436-09ca-c821425fade9 [Follower]" leader= testing.go:149: server-f3da6f6c-5d28-9436-09ca-c821425fade9: 2024-06-12T04:32:30.586+1400 [INFO] server-f3da6f6c-5d28-9436-09ca-c821425fade9: entering follower state: follower="Node at f3da6f6c-5d28-9436-09ca-c821425fade9 [Follower]" leader= testLoggerAdapter verbose: server-745c502e-38cb-1b1e-a943-bc158cafcce8: 2024-06-12T04:32:30.636+1400 [WARN] server-745c502e-38cb-1b1e-a943-bc158cafcce8: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-745c502e-38cb-1b1e-a943-bc158cafcce8: 2024-06-12T04:32:30.636+1400 [WARN] server-745c502e-38cb-1b1e-a943-bc158cafcce8: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-745c502e-38cb-1b1e-a943-bc158cafcce8: 2024-06-12T04:32:30.636+1400 [INFO] server-745c502e-38cb-1b1e-a943-bc158cafcce8: entering candidate state: node="Node at 745c502e-38cb-1b1e-a943-bc158cafcce8 [Candidate]" term=2 testing.go:149: server-745c502e-38cb-1b1e-a943-bc158cafcce8: 2024-06-12T04:32:30.636+1400 [INFO] server-745c502e-38cb-1b1e-a943-bc158cafcce8: entering candidate state: node="Node at 745c502e-38cb-1b1e-a943-bc158cafcce8 [Candidate]" term=2 testLoggerAdapter verbose: server-745c502e-38cb-1b1e-a943-bc158cafcce8: 2024-06-12T04:32:30.636+1400 [INFO] server-745c502e-38cb-1b1e-a943-bc158cafcce8: election won: tally=1 testing.go:149: server-745c502e-38cb-1b1e-a943-bc158cafcce8: 2024-06-12T04:32:30.636+1400 [INFO] server-745c502e-38cb-1b1e-a943-bc158cafcce8: election won: tally=1 testLoggerAdapter verbose: server-745c502e-38cb-1b1e-a943-bc158cafcce8: 2024-06-12T04:32:30.636+1400 [INFO] server-745c502e-38cb-1b1e-a943-bc158cafcce8: entering leader state: leader="Node at 745c502e-38cb-1b1e-a943-bc158cafcce8 [Leader]" testing.go:149: server-745c502e-38cb-1b1e-a943-bc158cafcce8: 2024-06-12T04:32:30.636+1400 [INFO] server-745c502e-38cb-1b1e-a943-bc158cafcce8: entering leader state: leader="Node at 745c502e-38cb-1b1e-a943-bc158cafcce8 [Leader]" testLoggerAdapter verbose: server-f3da6f6c-5d28-9436-09ca-c821425fade9: 2024-06-12T04:32:30.656+1400 [WARN] server-f3da6f6c-5d28-9436-09ca-c821425fade9: no known peers, aborting election testing.go:149: server-f3da6f6c-5d28-9436-09ca-c821425fade9: 2024-06-12T04:32:30.656+1400 [WARN] server-f3da6f6c-5d28-9436-09ca-c821425fade9: no known peers, aborting election testLoggerAdapter verbose: server-aecede68-0038-d0d2-a0bf-dfc29611e5d7: 2024-06-12T04:32:30.683+1400 [WARN] server-aecede68-0038-d0d2-a0bf-dfc29611e5d7: no known peers, aborting election testing.go:149: server-aecede68-0038-d0d2-a0bf-dfc29611e5d7: 2024-06-12T04:32:30.683+1400 [WARN] server-aecede68-0038-d0d2-a0bf-dfc29611e5d7: no known peers, aborting election testLoggerAdapter verbose: cluster: 2024-06-12T04:32:30.743+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:30.636897349 +1400 +14 m=+7.544415928 (1 nodes), 51.048771ms from start of poll, 52.229449ms from cluster start. Timeout at 2024-06-12 04:32:30.742935311 +1400 +14 m=+7.650453930, 106.038002ms after stability testing.go:149: cluster: 2024-06-12T04:32:30.743+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:30.636897349 +1400 +14 m=+7.544415928 (1 nodes), 51.048771ms from start of poll, 52.229449ms from cluster start. Timeout at 2024-06-12 04:32:30.742935311 +1400 +14 m=+7.650453930, 106.038002ms after stability testLoggerAdapter verbose: server-745c502e-38cb-1b1e-a943-bc158cafcce8: 2024-06-12T04:32:30.743+1400 [INFO] server-745c502e-38cb-1b1e-a943-bc158cafcce8: updating configuration: command=AddStaging server-id=server-aecede68-0038-d0d2-a0bf-dfc29611e5d7 server-addr=aecede68-0038-d0d2-a0bf-dfc29611e5d7 servers="[{Suffrage:Voter ID:server-745c502e-38cb-1b1e-a943-bc158cafcce8 Address:745c502e-38cb-1b1e-a943-bc158cafcce8} {Suffrage:Voter ID:server-aecede68-0038-d0d2-a0bf-dfc29611e5d7 Address:aecede68-0038-d0d2-a0bf-dfc29611e5d7}]" testing.go:149: server-745c502e-38cb-1b1e-a943-bc158cafcce8: 2024-06-12T04:32:30.743+1400 [INFO] server-745c502e-38cb-1b1e-a943-bc158cafcce8: updating configuration: command=AddStaging server-id=server-aecede68-0038-d0d2-a0bf-dfc29611e5d7 server-addr=aecede68-0038-d0d2-a0bf-dfc29611e5d7 servers="[{Suffrage:Voter ID:server-745c502e-38cb-1b1e-a943-bc158cafcce8 Address:745c502e-38cb-1b1e-a943-bc158cafcce8} {Suffrage:Voter ID:server-aecede68-0038-d0d2-a0bf-dfc29611e5d7 Address:aecede68-0038-d0d2-a0bf-dfc29611e5d7}]" testLoggerAdapter verbose: server-745c502e-38cb-1b1e-a943-bc158cafcce8: 2024-06-12T04:32:30.743+1400 [INFO] server-745c502e-38cb-1b1e-a943-bc158cafcce8: added peer, starting replication: peer=server-aecede68-0038-d0d2-a0bf-dfc29611e5d7 testing.go:149: server-745c502e-38cb-1b1e-a943-bc158cafcce8: 2024-06-12T04:32:30.743+1400 [INFO] server-745c502e-38cb-1b1e-a943-bc158cafcce8: added peer, starting replication: peer=server-aecede68-0038-d0d2-a0bf-dfc29611e5d7 testLoggerAdapter verbose: cluster: 2024-06-12T04:32:30.743+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-12T04:32:30.743+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-aecede68-0038-d0d2-a0bf-dfc29611e5d7: 2024-06-12T04:32:30.743+1400 [WARN] server-aecede68-0038-d0d2-a0bf-dfc29611e5d7: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: server-aecede68-0038-d0d2-a0bf-dfc29611e5d7: 2024-06-12T04:32:30.743+1400 [WARN] server-aecede68-0038-d0d2-a0bf-dfc29611e5d7: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: server-745c502e-38cb-1b1e-a943-bc158cafcce8: 2024-06-12T04:32:30.743+1400 [WARN] server-745c502e-38cb-1b1e-a943-bc158cafcce8: appendEntries rejected, sending older logs: peer="{Voter server-aecede68-0038-d0d2-a0bf-dfc29611e5d7 aecede68-0038-d0d2-a0bf-dfc29611e5d7}" next=1 testing.go:149: server-745c502e-38cb-1b1e-a943-bc158cafcce8: 2024-06-12T04:32:30.743+1400 [WARN] server-745c502e-38cb-1b1e-a943-bc158cafcce8: appendEntries rejected, sending older logs: peer="{Voter server-aecede68-0038-d0d2-a0bf-dfc29611e5d7 aecede68-0038-d0d2-a0bf-dfc29611e5d7}" next=1 testLoggerAdapter verbose: server-745c502e-38cb-1b1e-a943-bc158cafcce8: 2024-06-12T04:32:30.743+1400 [INFO] server-745c502e-38cb-1b1e-a943-bc158cafcce8: pipelining replication: peer="{Voter server-aecede68-0038-d0d2-a0bf-dfc29611e5d7 aecede68-0038-d0d2-a0bf-dfc29611e5d7}" testing.go:149: server-745c502e-38cb-1b1e-a943-bc158cafcce8: 2024-06-12T04:32:30.743+1400 [INFO] server-745c502e-38cb-1b1e-a943-bc158cafcce8: pipelining replication: peer="{Voter server-aecede68-0038-d0d2-a0bf-dfc29611e5d7 aecede68-0038-d0d2-a0bf-dfc29611e5d7}" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:30.849+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:30.743417534 +1400 +14 m=+7.650936113 (1 nodes), 2.44µs from start of poll, 158.749634ms from cluster start. Timeout at 2024-06-12 04:32:30.849172066 +1400 +14 m=+7.756690685, 105.754572ms after stability testing.go:149: cluster: 2024-06-12T04:32:30.849+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:30.743417534 +1400 +14 m=+7.650936113 (1 nodes), 2.44µs from start of poll, 158.749634ms from cluster start. Timeout at 2024-06-12 04:32:30.849172066 +1400 +14 m=+7.756690685, 105.754572ms after stability testLoggerAdapter verbose: server-745c502e-38cb-1b1e-a943-bc158cafcce8: 2024-06-12T04:32:30.849+1400 [INFO] server-745c502e-38cb-1b1e-a943-bc158cafcce8: updating configuration: command=AddStaging server-id=server-f3da6f6c-5d28-9436-09ca-c821425fade9 server-addr=f3da6f6c-5d28-9436-09ca-c821425fade9 servers="[{Suffrage:Voter ID:server-745c502e-38cb-1b1e-a943-bc158cafcce8 Address:745c502e-38cb-1b1e-a943-bc158cafcce8} {Suffrage:Voter ID:server-aecede68-0038-d0d2-a0bf-dfc29611e5d7 Address:aecede68-0038-d0d2-a0bf-dfc29611e5d7} {Suffrage:Voter ID:server-f3da6f6c-5d28-9436-09ca-c821425fade9 Address:f3da6f6c-5d28-9436-09ca-c821425fade9}]" testing.go:149: server-745c502e-38cb-1b1e-a943-bc158cafcce8: 2024-06-12T04:32:30.849+1400 [INFO] server-745c502e-38cb-1b1e-a943-bc158cafcce8: updating configuration: command=AddStaging server-id=server-f3da6f6c-5d28-9436-09ca-c821425fade9 server-addr=f3da6f6c-5d28-9436-09ca-c821425fade9 servers="[{Suffrage:Voter ID:server-745c502e-38cb-1b1e-a943-bc158cafcce8 Address:745c502e-38cb-1b1e-a943-bc158cafcce8} {Suffrage:Voter ID:server-aecede68-0038-d0d2-a0bf-dfc29611e5d7 Address:aecede68-0038-d0d2-a0bf-dfc29611e5d7} {Suffrage:Voter ID:server-f3da6f6c-5d28-9436-09ca-c821425fade9 Address:f3da6f6c-5d28-9436-09ca-c821425fade9}]" testLoggerAdapter verbose: server-745c502e-38cb-1b1e-a943-bc158cafcce8: 2024-06-12T04:32:30.849+1400 [INFO] server-745c502e-38cb-1b1e-a943-bc158cafcce8: added peer, starting replication: peer=server-f3da6f6c-5d28-9436-09ca-c821425fade9 testing.go:149: server-745c502e-38cb-1b1e-a943-bc158cafcce8: 2024-06-12T04:32:30.849+1400 [INFO] server-745c502e-38cb-1b1e-a943-bc158cafcce8: added peer, starting replication: peer=server-f3da6f6c-5d28-9436-09ca-c821425fade9 testLoggerAdapter verbose: server-f3da6f6c-5d28-9436-09ca-c821425fade9: 2024-06-12T04:32:30.849+1400 [WARN] server-f3da6f6c-5d28-9436-09ca-c821425fade9: failed to get previous log: previous-index=4 last-index=0 error="log not found" testing.go:149: server-f3da6f6c-5d28-9436-09ca-c821425fade9: 2024-06-12T04:32:30.849+1400 [WARN] server-f3da6f6c-5d28-9436-09ca-c821425fade9: failed to get previous log: previous-index=4 last-index=0 error="log not found" testLoggerAdapter verbose: server-745c502e-38cb-1b1e-a943-bc158cafcce8: 2024-06-12T04:32:30.849+1400 [WARN] server-745c502e-38cb-1b1e-a943-bc158cafcce8: appendEntries rejected, sending older logs: peer="{Voter server-f3da6f6c-5d28-9436-09ca-c821425fade9 f3da6f6c-5d28-9436-09ca-c821425fade9}" next=1 testing.go:149: server-745c502e-38cb-1b1e-a943-bc158cafcce8: 2024-06-12T04:32:30.849+1400 [WARN] server-745c502e-38cb-1b1e-a943-bc158cafcce8: appendEntries rejected, sending older logs: peer="{Voter server-f3da6f6c-5d28-9436-09ca-c821425fade9 f3da6f6c-5d28-9436-09ca-c821425fade9}" next=1 testLoggerAdapter verbose: server-745c502e-38cb-1b1e-a943-bc158cafcce8: 2024-06-12T04:32:30.849+1400 [INFO] server-745c502e-38cb-1b1e-a943-bc158cafcce8: pipelining replication: peer="{Voter server-f3da6f6c-5d28-9436-09ca-c821425fade9 f3da6f6c-5d28-9436-09ca-c821425fade9}" testing.go:149: server-745c502e-38cb-1b1e-a943-bc158cafcce8: 2024-06-12T04:32:30.849+1400 [INFO] server-745c502e-38cb-1b1e-a943-bc158cafcce8: pipelining replication: peer="{Voter server-f3da6f6c-5d28-9436-09ca-c821425fade9 f3da6f6c-5d28-9436-09ca-c821425fade9}" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:30.850+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-12T04:32:30.850+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-06-12T04:32:30.955+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:30.850102673 +1400 +14 m=+7.757621252 (1 nodes), 1.36µs from start of poll, 265.434773ms from cluster start. Timeout at 2024-06-12 04:32:30.955440459 +1400 +14 m=+7.862959038, 105.337786ms after stability testing.go:149: cluster: 2024-06-12T04:32:30.955+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:30.850102673 +1400 +14 m=+7.757621252 (1 nodes), 1.36µs from start of poll, 265.434773ms from cluster start. Timeout at 2024-06-12 04:32:30.955440459 +1400 +14 m=+7.862959038, 105.337786ms after stability testLoggerAdapter verbose: server-745c502e-38cb-1b1e-a943-bc158cafcce8: 2024-06-12T04:32:30.955+1400 [INFO] server-745c502e-38cb-1b1e-a943-bc158cafcce8: aborting pipeline replication: peer="{Voter server-f3da6f6c-5d28-9436-09ca-c821425fade9 f3da6f6c-5d28-9436-09ca-c821425fade9}" testing.go:149: server-745c502e-38cb-1b1e-a943-bc158cafcce8: 2024-06-12T04:32:30.955+1400 [INFO] server-745c502e-38cb-1b1e-a943-bc158cafcce8: aborting pipeline replication: peer="{Voter server-f3da6f6c-5d28-9436-09ca-c821425fade9 f3da6f6c-5d28-9436-09ca-c821425fade9}" testLoggerAdapter verbose: server-745c502e-38cb-1b1e-a943-bc158cafcce8: 2024-06-12T04:32:30.956+1400 [INFO] server-745c502e-38cb-1b1e-a943-bc158cafcce8: aborting pipeline replication: peer="{Voter server-aecede68-0038-d0d2-a0bf-dfc29611e5d7 aecede68-0038-d0d2-a0bf-dfc29611e5d7}" testing.go:149: server-745c502e-38cb-1b1e-a943-bc158cafcce8: 2024-06-12T04:32:30.956+1400 [INFO] server-745c502e-38cb-1b1e-a943-bc158cafcce8: aborting pipeline replication: peer="{Voter server-aecede68-0038-d0d2-a0bf-dfc29611e5d7 aecede68-0038-d0d2-a0bf-dfc29611e5d7}" --- PASS: TestRaft_JoinNode_ConfigStore (0.37s) === RUN TestRaft_RemoveFollower testLoggerAdapter verbose: server-26b9b025-eced-2e04-4877-6a2e92ef64a1: 2024-06-12T04:32:30.958+1400 [INFO] server-26b9b025-eced-2e04-4877-6a2e92ef64a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-26b9b025-eced-2e04-4877-6a2e92ef64a1 Address:26b9b025-eced-2e04-4877-6a2e92ef64a1} {Suffrage:Voter ID:server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328 Address:a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328} {Suffrage:Voter ID:server-3e3b0020-764a-ead7-384c-623cc6f4ae61 Address:3e3b0020-764a-ead7-384c-623cc6f4ae61}]" testing.go:149: server-26b9b025-eced-2e04-4877-6a2e92ef64a1: 2024-06-12T04:32:30.958+1400 [INFO] server-26b9b025-eced-2e04-4877-6a2e92ef64a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-26b9b025-eced-2e04-4877-6a2e92ef64a1 Address:26b9b025-eced-2e04-4877-6a2e92ef64a1} {Suffrage:Voter ID:server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328 Address:a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328} {Suffrage:Voter ID:server-3e3b0020-764a-ead7-384c-623cc6f4ae61 Address:3e3b0020-764a-ead7-384c-623cc6f4ae61}]" testLoggerAdapter verbose: server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: 2024-06-12T04:32:30.958+1400 [INFO] server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-26b9b025-eced-2e04-4877-6a2e92ef64a1 Address:26b9b025-eced-2e04-4877-6a2e92ef64a1} {Suffrage:Voter ID:server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328 Address:a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328} {Suffrage:Voter ID:server-3e3b0020-764a-ead7-384c-623cc6f4ae61 Address:3e3b0020-764a-ead7-384c-623cc6f4ae61}]" testing.go:149: server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: 2024-06-12T04:32:30.958+1400 [INFO] server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-26b9b025-eced-2e04-4877-6a2e92ef64a1 Address:26b9b025-eced-2e04-4877-6a2e92ef64a1} {Suffrage:Voter ID:server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328 Address:a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328} {Suffrage:Voter ID:server-3e3b0020-764a-ead7-384c-623cc6f4ae61 Address:3e3b0020-764a-ead7-384c-623cc6f4ae61}]" testLoggerAdapter verbose: server-3e3b0020-764a-ead7-384c-623cc6f4ae61: 2024-06-12T04:32:30.958+1400 [INFO] server-3e3b0020-764a-ead7-384c-623cc6f4ae61: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-26b9b025-eced-2e04-4877-6a2e92ef64a1 Address:26b9b025-eced-2e04-4877-6a2e92ef64a1} {Suffrage:Voter ID:server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328 Address:a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328} {Suffrage:Voter ID:server-3e3b0020-764a-ead7-384c-623cc6f4ae61 Address:3e3b0020-764a-ead7-384c-623cc6f4ae61}]" testing.go:149: server-3e3b0020-764a-ead7-384c-623cc6f4ae61: 2024-06-12T04:32:30.958+1400 [INFO] server-3e3b0020-764a-ead7-384c-623cc6f4ae61: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-26b9b025-eced-2e04-4877-6a2e92ef64a1 Address:26b9b025-eced-2e04-4877-6a2e92ef64a1} {Suffrage:Voter ID:server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328 Address:a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328} {Suffrage:Voter ID:server-3e3b0020-764a-ead7-384c-623cc6f4ae61 Address:3e3b0020-764a-ead7-384c-623cc6f4ae61}]" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:30.958+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-12T04:32:30.958+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-26b9b025-eced-2e04-4877-6a2e92ef64a1: 2024-06-12T04:32:30.958+1400 [INFO] server-26b9b025-eced-2e04-4877-6a2e92ef64a1: entering follower state: follower="Node at 26b9b025-eced-2e04-4877-6a2e92ef64a1 [Follower]" leader= testing.go:149: server-26b9b025-eced-2e04-4877-6a2e92ef64a1: 2024-06-12T04:32:30.958+1400 [INFO] server-26b9b025-eced-2e04-4877-6a2e92ef64a1: entering follower state: follower="Node at 26b9b025-eced-2e04-4877-6a2e92ef64a1 [Follower]" leader= testLoggerAdapter verbose: server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: 2024-06-12T04:32:30.959+1400 [INFO] server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: entering follower state: follower="Node at a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328 [Follower]" leader= testing.go:149: server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: 2024-06-12T04:32:30.959+1400 [INFO] server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: entering follower state: follower="Node at a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328 [Follower]" leader= testLoggerAdapter verbose: server-3e3b0020-764a-ead7-384c-623cc6f4ae61: 2024-06-12T04:32:30.959+1400 [INFO] server-3e3b0020-764a-ead7-384c-623cc6f4ae61: entering follower state: follower="Node at 3e3b0020-764a-ead7-384c-623cc6f4ae61 [Follower]" leader= testing.go:149: server-3e3b0020-764a-ead7-384c-623cc6f4ae61: 2024-06-12T04:32:30.959+1400 [INFO] server-3e3b0020-764a-ead7-384c-623cc6f4ae61: entering follower state: follower="Node at 3e3b0020-764a-ead7-384c-623cc6f4ae61 [Follower]" leader= testLoggerAdapter verbose: server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: 2024-06-12T04:32:31.015+1400 [WARN] server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: 2024-06-12T04:32:31.015+1400 [WARN] server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: 2024-06-12T04:32:31.015+1400 [INFO] server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: entering candidate state: node="Node at a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328 [Candidate]" term=2 testing.go:149: server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: 2024-06-12T04:32:31.015+1400 [INFO] server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: entering candidate state: node="Node at a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328 [Candidate]" term=2 testLoggerAdapter verbose: server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: 2024-06-12T04:32:31.015+1400 [INFO] server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: election won: tally=2 testing.go:149: server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: 2024-06-12T04:32:31.015+1400 [INFO] server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: election won: tally=2 testLoggerAdapter verbose: server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: 2024-06-12T04:32:31.015+1400 [INFO] server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: entering leader state: leader="Node at a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328 [Leader]" testing.go:149: server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: 2024-06-12T04:32:31.015+1400 [INFO] server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: entering leader state: leader="Node at a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328 [Leader]" testLoggerAdapter verbose: server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: 2024-06-12T04:32:31.015+1400 [INFO] server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: added peer, starting replication: peer=server-26b9b025-eced-2e04-4877-6a2e92ef64a1 testing.go:149: server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: 2024-06-12T04:32:31.015+1400 [INFO] server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: added peer, starting replication: peer=server-26b9b025-eced-2e04-4877-6a2e92ef64a1 testLoggerAdapter verbose: server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: 2024-06-12T04:32:31.015+1400 [INFO] server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: added peer, starting replication: peer=server-3e3b0020-764a-ead7-384c-623cc6f4ae61 testing.go:149: server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: 2024-06-12T04:32:31.015+1400 [INFO] server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: added peer, starting replication: peer=server-3e3b0020-764a-ead7-384c-623cc6f4ae61 testLoggerAdapter verbose: server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: 2024-06-12T04:32:31.015+1400 [INFO] server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: pipelining replication: peer="{Voter server-3e3b0020-764a-ead7-384c-623cc6f4ae61 3e3b0020-764a-ead7-384c-623cc6f4ae61}" testLoggerAdapter verbose: server-26b9b025-eced-2e04-4877-6a2e92ef64a1: 2024-06-12T04:32:31.015+1400 [WARN] server-26b9b025-eced-2e04-4877-6a2e92ef64a1: rejecting vote request since our last term is greater: candidate=a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328 last-term=2 last-candidate-term=1 testing.go:149: server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: 2024-06-12T04:32:31.015+1400 [INFO] server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: pipelining replication: peer="{Voter server-3e3b0020-764a-ead7-384c-623cc6f4ae61 3e3b0020-764a-ead7-384c-623cc6f4ae61}" testing.go:149: server-26b9b025-eced-2e04-4877-6a2e92ef64a1: 2024-06-12T04:32:31.015+1400 [WARN] server-26b9b025-eced-2e04-4877-6a2e92ef64a1: rejecting vote request since our last term is greater: candidate=a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328 last-term=2 last-candidate-term=1 testLoggerAdapter verbose: server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: 2024-06-12T04:32:31.015+1400 [INFO] server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: pipelining replication: peer="{Voter server-26b9b025-eced-2e04-4877-6a2e92ef64a1 26b9b025-eced-2e04-4877-6a2e92ef64a1}" testing.go:149: server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: 2024-06-12T04:32:31.015+1400 [INFO] server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: pipelining replication: peer="{Voter server-26b9b025-eced-2e04-4877-6a2e92ef64a1 26b9b025-eced-2e04-4877-6a2e92ef64a1}" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:31.120+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:31.015898454 +1400 +14 m=+7.923417033 (1 nodes), 56.934721ms from start of poll, 57.752372ms from cluster start. Timeout at 2024-06-12 04:32:31.120914012 +1400 +14 m=+8.028432591, 105.015558ms after stability testing.go:149: cluster: 2024-06-12T04:32:31.120+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:31.015898454 +1400 +14 m=+7.923417033 (1 nodes), 56.934721ms from start of poll, 57.752372ms from cluster start. Timeout at 2024-06-12 04:32:31.120914012 +1400 +14 m=+8.028432591, 105.015558ms after stability testLoggerAdapter verbose: cluster: 2024-06-12T04:32:31.127+1400 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-06-12T04:32:31.127+1400 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2024-06-12T04:32:31.232+1400 [INFO] stable state for Follower reached at 2024-06-12 04:32:31.127170708 +1400 +14 m=+8.034689287 (2 nodes), 1.96µs from start of poll, 169.024626ms from cluster start. Timeout at 2024-06-12 04:32:31.232233305 +1400 +14 m=+8.139751884, 105.062597ms after stability testing.go:149: cluster: 2024-06-12T04:32:31.232+1400 [INFO] stable state for Follower reached at 2024-06-12 04:32:31.127170708 +1400 +14 m=+8.034689287 (2 nodes), 1.96µs from start of poll, 169.024626ms from cluster start. Timeout at 2024-06-12 04:32:31.232233305 +1400 +14 m=+8.139751884, 105.062597ms after stability testLoggerAdapter verbose: server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: 2024-06-12T04:32:31.232+1400 [INFO] server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: updating configuration: command=RemoveServer server-id=server-26b9b025-eced-2e04-4877-6a2e92ef64a1 server-addr= servers="[{Suffrage:Voter ID:server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328 Address:a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328} {Suffrage:Voter ID:server-3e3b0020-764a-ead7-384c-623cc6f4ae61 Address:3e3b0020-764a-ead7-384c-623cc6f4ae61}]" testing.go:149: server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: 2024-06-12T04:32:31.232+1400 [INFO] server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: updating configuration: command=RemoveServer server-id=server-26b9b025-eced-2e04-4877-6a2e92ef64a1 server-addr= servers="[{Suffrage:Voter ID:server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328 Address:a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328} {Suffrage:Voter ID:server-3e3b0020-764a-ead7-384c-623cc6f4ae61 Address:3e3b0020-764a-ead7-384c-623cc6f4ae61}]" testLoggerAdapter verbose: server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: 2024-06-12T04:32:31.232+1400 [INFO] server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: removed peer, stopping replication: peer=server-26b9b025-eced-2e04-4877-6a2e92ef64a1 last-index=3 testing.go:149: server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: 2024-06-12T04:32:31.232+1400 [INFO] server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: removed peer, stopping replication: peer=server-26b9b025-eced-2e04-4877-6a2e92ef64a1 last-index=3 testLoggerAdapter verbose: server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: 2024-06-12T04:32:31.232+1400 [INFO] server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: aborting pipeline replication: peer="{Voter server-26b9b025-eced-2e04-4877-6a2e92ef64a1 26b9b025-eced-2e04-4877-6a2e92ef64a1}" testing.go:149: server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: 2024-06-12T04:32:31.232+1400 [INFO] server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: aborting pipeline replication: peer="{Voter server-26b9b025-eced-2e04-4877-6a2e92ef64a1 26b9b025-eced-2e04-4877-6a2e92ef64a1}" testLoggerAdapter verbose: server-26b9b025-eced-2e04-4877-6a2e92ef64a1: 2024-06-12T04:32:31.328+1400 [WARN] server-26b9b025-eced-2e04-4877-6a2e92ef64a1: heartbeat timeout reached, starting election: last-leader=a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328 testing.go:149: server-26b9b025-eced-2e04-4877-6a2e92ef64a1: 2024-06-12T04:32:31.328+1400 [WARN] server-26b9b025-eced-2e04-4877-6a2e92ef64a1: heartbeat timeout reached, starting election: last-leader=a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328 testLoggerAdapter verbose: server-26b9b025-eced-2e04-4877-6a2e92ef64a1: 2024-06-12T04:32:31.328+1400 [INFO] server-26b9b025-eced-2e04-4877-6a2e92ef64a1: entering candidate state: node="Node at 26b9b025-eced-2e04-4877-6a2e92ef64a1 [Candidate]" term=3 testing.go:149: server-26b9b025-eced-2e04-4877-6a2e92ef64a1: 2024-06-12T04:32:31.328+1400 [INFO] server-26b9b025-eced-2e04-4877-6a2e92ef64a1: entering candidate state: node="Node at 26b9b025-eced-2e04-4877-6a2e92ef64a1 [Candidate]" term=3 testLoggerAdapter verbose: server-3e3b0020-764a-ead7-384c-623cc6f4ae61: 2024-06-12T04:32:31.328+1400 [WARN] server-3e3b0020-764a-ead7-384c-623cc6f4ae61: rejecting vote request since we have a leader: from=26b9b025-eced-2e04-4877-6a2e92ef64a1 leader=a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328 testing.go:149: server-3e3b0020-764a-ead7-384c-623cc6f4ae61: 2024-06-12T04:32:31.328+1400 [WARN] server-3e3b0020-764a-ead7-384c-623cc6f4ae61: rejecting vote request since we have a leader: from=26b9b025-eced-2e04-4877-6a2e92ef64a1 leader=a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328 testLoggerAdapter verbose: server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: 2024-06-12T04:32:31.328+1400 [WARN] server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: rejecting vote request since we have a leader: from=26b9b025-eced-2e04-4877-6a2e92ef64a1 leader=a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328 testing.go:149: server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: 2024-06-12T04:32:31.328+1400 [WARN] server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: rejecting vote request since we have a leader: from=26b9b025-eced-2e04-4877-6a2e92ef64a1 leader=a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328 testLoggerAdapter verbose: server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: 2024-06-12T04:32:31.338+1400 [INFO] server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: aborting pipeline replication: peer="{Voter server-3e3b0020-764a-ead7-384c-623cc6f4ae61 3e3b0020-764a-ead7-384c-623cc6f4ae61}" testing.go:149: server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: 2024-06-12T04:32:31.338+1400 [INFO] server-a12bf9da-cfb7-9a5b-bb07-29b5d5fe7328: aborting pipeline replication: peer="{Voter server-3e3b0020-764a-ead7-384c-623cc6f4ae61 3e3b0020-764a-ead7-384c-623cc6f4ae61}" --- PASS: TestRaft_RemoveFollower (0.38s) === RUN TestRaft_RemoveLeader testLoggerAdapter verbose: server-22894afa-22a3-fb46-6be6-70213ddabb61: 2024-06-12T04:32:31.341+1400 [INFO] server-22894afa-22a3-fb46-6be6-70213ddabb61: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-22894afa-22a3-fb46-6be6-70213ddabb61 Address:22894afa-22a3-fb46-6be6-70213ddabb61} {Suffrage:Voter ID:server-063c306d-3b25-9ea2-cd1d-b631d74ac673 Address:063c306d-3b25-9ea2-cd1d-b631d74ac673} {Suffrage:Voter ID:server-35cc5789-32c0-509d-3472-243f4999d8a5 Address:35cc5789-32c0-509d-3472-243f4999d8a5}]" testing.go:149: server-22894afa-22a3-fb46-6be6-70213ddabb61: 2024-06-12T04:32:31.341+1400 [INFO] server-22894afa-22a3-fb46-6be6-70213ddabb61: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-22894afa-22a3-fb46-6be6-70213ddabb61 Address:22894afa-22a3-fb46-6be6-70213ddabb61} {Suffrage:Voter ID:server-063c306d-3b25-9ea2-cd1d-b631d74ac673 Address:063c306d-3b25-9ea2-cd1d-b631d74ac673} {Suffrage:Voter ID:server-35cc5789-32c0-509d-3472-243f4999d8a5 Address:35cc5789-32c0-509d-3472-243f4999d8a5}]" testLoggerAdapter verbose: server-063c306d-3b25-9ea2-cd1d-b631d74ac673: 2024-06-12T04:32:31.341+1400 [INFO] server-063c306d-3b25-9ea2-cd1d-b631d74ac673: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-22894afa-22a3-fb46-6be6-70213ddabb61 Address:22894afa-22a3-fb46-6be6-70213ddabb61} {Suffrage:Voter ID:server-063c306d-3b25-9ea2-cd1d-b631d74ac673 Address:063c306d-3b25-9ea2-cd1d-b631d74ac673} {Suffrage:Voter ID:server-35cc5789-32c0-509d-3472-243f4999d8a5 Address:35cc5789-32c0-509d-3472-243f4999d8a5}]" testing.go:149: server-063c306d-3b25-9ea2-cd1d-b631d74ac673: 2024-06-12T04:32:31.341+1400 [INFO] server-063c306d-3b25-9ea2-cd1d-b631d74ac673: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-22894afa-22a3-fb46-6be6-70213ddabb61 Address:22894afa-22a3-fb46-6be6-70213ddabb61} {Suffrage:Voter ID:server-063c306d-3b25-9ea2-cd1d-b631d74ac673 Address:063c306d-3b25-9ea2-cd1d-b631d74ac673} {Suffrage:Voter ID:server-35cc5789-32c0-509d-3472-243f4999d8a5 Address:35cc5789-32c0-509d-3472-243f4999d8a5}]" testLoggerAdapter verbose: server-35cc5789-32c0-509d-3472-243f4999d8a5: 2024-06-12T04:32:31.341+1400 [INFO] server-35cc5789-32c0-509d-3472-243f4999d8a5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-22894afa-22a3-fb46-6be6-70213ddabb61 Address:22894afa-22a3-fb46-6be6-70213ddabb61} {Suffrage:Voter ID:server-063c306d-3b25-9ea2-cd1d-b631d74ac673 Address:063c306d-3b25-9ea2-cd1d-b631d74ac673} {Suffrage:Voter ID:server-35cc5789-32c0-509d-3472-243f4999d8a5 Address:35cc5789-32c0-509d-3472-243f4999d8a5}]" testing.go:149: server-35cc5789-32c0-509d-3472-243f4999d8a5: 2024-06-12T04:32:31.341+1400 [INFO] server-35cc5789-32c0-509d-3472-243f4999d8a5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-22894afa-22a3-fb46-6be6-70213ddabb61 Address:22894afa-22a3-fb46-6be6-70213ddabb61} {Suffrage:Voter ID:server-063c306d-3b25-9ea2-cd1d-b631d74ac673 Address:063c306d-3b25-9ea2-cd1d-b631d74ac673} {Suffrage:Voter ID:server-35cc5789-32c0-509d-3472-243f4999d8a5 Address:35cc5789-32c0-509d-3472-243f4999d8a5}]" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:31.341+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-12T04:32:31.341+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-22894afa-22a3-fb46-6be6-70213ddabb61: 2024-06-12T04:32:31.341+1400 [INFO] server-22894afa-22a3-fb46-6be6-70213ddabb61: entering follower state: follower="Node at 22894afa-22a3-fb46-6be6-70213ddabb61 [Follower]" leader= testing.go:149: server-22894afa-22a3-fb46-6be6-70213ddabb61: 2024-06-12T04:32:31.341+1400 [INFO] server-22894afa-22a3-fb46-6be6-70213ddabb61: entering follower state: follower="Node at 22894afa-22a3-fb46-6be6-70213ddabb61 [Follower]" leader= testLoggerAdapter verbose: server-063c306d-3b25-9ea2-cd1d-b631d74ac673: 2024-06-12T04:32:31.341+1400 [INFO] server-063c306d-3b25-9ea2-cd1d-b631d74ac673: entering follower state: follower="Node at 063c306d-3b25-9ea2-cd1d-b631d74ac673 [Follower]" leader= testing.go:149: server-063c306d-3b25-9ea2-cd1d-b631d74ac673: 2024-06-12T04:32:31.341+1400 [INFO] server-063c306d-3b25-9ea2-cd1d-b631d74ac673: entering follower state: follower="Node at 063c306d-3b25-9ea2-cd1d-b631d74ac673 [Follower]" leader= testLoggerAdapter verbose: server-35cc5789-32c0-509d-3472-243f4999d8a5: 2024-06-12T04:32:31.341+1400 [INFO] server-35cc5789-32c0-509d-3472-243f4999d8a5: entering follower state: follower="Node at 35cc5789-32c0-509d-3472-243f4999d8a5 [Follower]" leader= testing.go:149: server-35cc5789-32c0-509d-3472-243f4999d8a5: 2024-06-12T04:32:31.341+1400 [INFO] server-35cc5789-32c0-509d-3472-243f4999d8a5: entering follower state: follower="Node at 35cc5789-32c0-509d-3472-243f4999d8a5 [Follower]" leader= testLoggerAdapter verbose: server-063c306d-3b25-9ea2-cd1d-b631d74ac673: 2024-06-12T04:32:31.399+1400 [WARN] server-063c306d-3b25-9ea2-cd1d-b631d74ac673: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-063c306d-3b25-9ea2-cd1d-b631d74ac673: 2024-06-12T04:32:31.399+1400 [WARN] server-063c306d-3b25-9ea2-cd1d-b631d74ac673: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-063c306d-3b25-9ea2-cd1d-b631d74ac673: 2024-06-12T04:32:31.400+1400 [INFO] server-063c306d-3b25-9ea2-cd1d-b631d74ac673: entering candidate state: node="Node at 063c306d-3b25-9ea2-cd1d-b631d74ac673 [Candidate]" term=2 testing.go:149: server-063c306d-3b25-9ea2-cd1d-b631d74ac673: 2024-06-12T04:32:31.400+1400 [INFO] server-063c306d-3b25-9ea2-cd1d-b631d74ac673: entering candidate state: node="Node at 063c306d-3b25-9ea2-cd1d-b631d74ac673 [Candidate]" term=2 testLoggerAdapter verbose: server-063c306d-3b25-9ea2-cd1d-b631d74ac673: 2024-06-12T04:32:31.400+1400 [INFO] server-063c306d-3b25-9ea2-cd1d-b631d74ac673: election won: tally=2 testing.go:149: server-063c306d-3b25-9ea2-cd1d-b631d74ac673: 2024-06-12T04:32:31.400+1400 [INFO] server-063c306d-3b25-9ea2-cd1d-b631d74ac673: election won: tally=2 testLoggerAdapter verbose: server-063c306d-3b25-9ea2-cd1d-b631d74ac673: 2024-06-12T04:32:31.400+1400 [INFO] server-063c306d-3b25-9ea2-cd1d-b631d74ac673: entering leader state: leader="Node at 063c306d-3b25-9ea2-cd1d-b631d74ac673 [Leader]" testing.go:149: server-063c306d-3b25-9ea2-cd1d-b631d74ac673: 2024-06-12T04:32:31.400+1400 [INFO] server-063c306d-3b25-9ea2-cd1d-b631d74ac673: entering leader state: leader="Node at 063c306d-3b25-9ea2-cd1d-b631d74ac673 [Leader]" testLoggerAdapter verbose: server-063c306d-3b25-9ea2-cd1d-b631d74ac673: 2024-06-12T04:32:31.400+1400 [INFO] server-063c306d-3b25-9ea2-cd1d-b631d74ac673: added peer, starting replication: peer=server-22894afa-22a3-fb46-6be6-70213ddabb61 testing.go:149: server-063c306d-3b25-9ea2-cd1d-b631d74ac673: 2024-06-12T04:32:31.400+1400 [INFO] server-063c306d-3b25-9ea2-cd1d-b631d74ac673: added peer, starting replication: peer=server-22894afa-22a3-fb46-6be6-70213ddabb61 testLoggerAdapter verbose: server-063c306d-3b25-9ea2-cd1d-b631d74ac673: 2024-06-12T04:32:31.400+1400 [INFO] server-063c306d-3b25-9ea2-cd1d-b631d74ac673: added peer, starting replication: peer=server-35cc5789-32c0-509d-3472-243f4999d8a5 testing.go:149: server-063c306d-3b25-9ea2-cd1d-b631d74ac673: 2024-06-12T04:32:31.400+1400 [INFO] server-063c306d-3b25-9ea2-cd1d-b631d74ac673: added peer, starting replication: peer=server-35cc5789-32c0-509d-3472-243f4999d8a5 testLoggerAdapter verbose: server-063c306d-3b25-9ea2-cd1d-b631d74ac673: 2024-06-12T04:32:31.400+1400 [INFO] server-063c306d-3b25-9ea2-cd1d-b631d74ac673: pipelining replication: peer="{Voter server-35cc5789-32c0-509d-3472-243f4999d8a5 35cc5789-32c0-509d-3472-243f4999d8a5}" testing.go:149: server-063c306d-3b25-9ea2-cd1d-b631d74ac673: 2024-06-12T04:32:31.400+1400 [INFO] server-063c306d-3b25-9ea2-cd1d-b631d74ac673: pipelining replication: peer="{Voter server-35cc5789-32c0-509d-3472-243f4999d8a5 35cc5789-32c0-509d-3472-243f4999d8a5}" testLoggerAdapter verbose: server-063c306d-3b25-9ea2-cd1d-b631d74ac673: 2024-06-12T04:32:31.400+1400 [INFO] server-063c306d-3b25-9ea2-cd1d-b631d74ac673: pipelining replication: peer="{Voter server-22894afa-22a3-fb46-6be6-70213ddabb61 22894afa-22a3-fb46-6be6-70213ddabb61}" testing.go:149: server-063c306d-3b25-9ea2-cd1d-b631d74ac673: 2024-06-12T04:32:31.400+1400 [INFO] server-063c306d-3b25-9ea2-cd1d-b631d74ac673: pipelining replication: peer="{Voter server-22894afa-22a3-fb46-6be6-70213ddabb61 22894afa-22a3-fb46-6be6-70213ddabb61}" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:31.505+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:31.400649392 +1400 +14 m=+8.308168011 (1 nodes), 59.048965ms from start of poll, 59.72258ms from cluster start. Timeout at 2024-06-12 04:32:31.505760947 +1400 +14 m=+8.413279526, 105.111515ms after stability testing.go:149: cluster: 2024-06-12T04:32:31.505+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:31.400649392 +1400 +14 m=+8.308168011 (1 nodes), 59.048965ms from start of poll, 59.72258ms from cluster start. Timeout at 2024-06-12 04:32:31.505760947 +1400 +14 m=+8.413279526, 105.111515ms after stability testLoggerAdapter verbose: cluster: 2024-06-12T04:32:31.511+1400 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-06-12T04:32:31.511+1400 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2024-06-12T04:32:31.616+1400 [INFO] stable state for Follower reached at 2024-06-12 04:32:31.51155074 +1400 +14 m=+8.419069319 (2 nodes), 1.92µs from start of poll, 170.623888ms from cluster start. Timeout at 2024-06-12 04:32:31.616901086 +1400 +14 m=+8.524419705, 105.350386ms after stability testing.go:149: cluster: 2024-06-12T04:32:31.616+1400 [INFO] stable state for Follower reached at 2024-06-12 04:32:31.51155074 +1400 +14 m=+8.419069319 (2 nodes), 1.92µs from start of poll, 170.623888ms from cluster start. Timeout at 2024-06-12 04:32:31.616901086 +1400 +14 m=+8.524419705, 105.350386ms after stability testLoggerAdapter verbose: server-063c306d-3b25-9ea2-cd1d-b631d74ac673: 2024-06-12T04:32:31.617+1400 [INFO] server-063c306d-3b25-9ea2-cd1d-b631d74ac673: updating configuration: command=RemoveServer server-id=server-063c306d-3b25-9ea2-cd1d-b631d74ac673 server-addr= servers="[{Suffrage:Voter ID:server-22894afa-22a3-fb46-6be6-70213ddabb61 Address:22894afa-22a3-fb46-6be6-70213ddabb61} {Suffrage:Voter ID:server-35cc5789-32c0-509d-3472-243f4999d8a5 Address:35cc5789-32c0-509d-3472-243f4999d8a5}]" testing.go:149: server-063c306d-3b25-9ea2-cd1d-b631d74ac673: 2024-06-12T04:32:31.617+1400 [INFO] server-063c306d-3b25-9ea2-cd1d-b631d74ac673: updating configuration: command=RemoveServer server-id=server-063c306d-3b25-9ea2-cd1d-b631d74ac673 server-addr= servers="[{Suffrage:Voter ID:server-22894afa-22a3-fb46-6be6-70213ddabb61 Address:22894afa-22a3-fb46-6be6-70213ddabb61} {Suffrage:Voter ID:server-35cc5789-32c0-509d-3472-243f4999d8a5 Address:35cc5789-32c0-509d-3472-243f4999d8a5}]" testLoggerAdapter verbose: server-063c306d-3b25-9ea2-cd1d-b631d74ac673: 2024-06-12T04:32:31.617+1400 [INFO] server-063c306d-3b25-9ea2-cd1d-b631d74ac673: removed ourself, shutting down testing.go:149: server-063c306d-3b25-9ea2-cd1d-b631d74ac673: 2024-06-12T04:32:31.617+1400 [INFO] server-063c306d-3b25-9ea2-cd1d-b631d74ac673: removed ourself, shutting down testLoggerAdapter verbose: server-063c306d-3b25-9ea2-cd1d-b631d74ac673: 2024-06-12T04:32:31.617+1400 [INFO] server-063c306d-3b25-9ea2-cd1d-b631d74ac673: aborting pipeline replication: peer="{Voter server-35cc5789-32c0-509d-3472-243f4999d8a5 35cc5789-32c0-509d-3472-243f4999d8a5}" testing.go:149: server-063c306d-3b25-9ea2-cd1d-b631d74ac673: 2024-06-12T04:32:31.617+1400 [INFO] server-063c306d-3b25-9ea2-cd1d-b631d74ac673: aborting pipeline replication: peer="{Voter server-35cc5789-32c0-509d-3472-243f4999d8a5 35cc5789-32c0-509d-3472-243f4999d8a5}" testLoggerAdapter verbose: server-063c306d-3b25-9ea2-cd1d-b631d74ac673: 2024-06-12T04:32:31.617+1400 [INFO] server-063c306d-3b25-9ea2-cd1d-b631d74ac673: aborting pipeline replication: peer="{Voter server-22894afa-22a3-fb46-6be6-70213ddabb61 22894afa-22a3-fb46-6be6-70213ddabb61}" testing.go:149: server-063c306d-3b25-9ea2-cd1d-b631d74ac673: 2024-06-12T04:32:31.617+1400 [INFO] server-063c306d-3b25-9ea2-cd1d-b631d74ac673: aborting pipeline replication: peer="{Voter server-22894afa-22a3-fb46-6be6-70213ddabb61 22894afa-22a3-fb46-6be6-70213ddabb61}" testLoggerAdapter verbose: server-22894afa-22a3-fb46-6be6-70213ddabb61: 2024-06-12T04:32:31.669+1400 [WARN] server-22894afa-22a3-fb46-6be6-70213ddabb61: heartbeat timeout reached, starting election: last-leader=063c306d-3b25-9ea2-cd1d-b631d74ac673 testing.go:149: server-22894afa-22a3-fb46-6be6-70213ddabb61: 2024-06-12T04:32:31.669+1400 [WARN] server-22894afa-22a3-fb46-6be6-70213ddabb61: heartbeat timeout reached, starting election: last-leader=063c306d-3b25-9ea2-cd1d-b631d74ac673 testLoggerAdapter verbose: server-22894afa-22a3-fb46-6be6-70213ddabb61: 2024-06-12T04:32:31.669+1400 [INFO] server-22894afa-22a3-fb46-6be6-70213ddabb61: entering candidate state: node="Node at 22894afa-22a3-fb46-6be6-70213ddabb61 [Candidate]" term=3 testing.go:149: server-22894afa-22a3-fb46-6be6-70213ddabb61: 2024-06-12T04:32:31.669+1400 [INFO] server-22894afa-22a3-fb46-6be6-70213ddabb61: entering candidate state: node="Node at 22894afa-22a3-fb46-6be6-70213ddabb61 [Candidate]" term=3 testLoggerAdapter verbose: server-35cc5789-32c0-509d-3472-243f4999d8a5: 2024-06-12T04:32:31.669+1400 [WARN] server-35cc5789-32c0-509d-3472-243f4999d8a5: rejecting vote request since we have a leader: from=22894afa-22a3-fb46-6be6-70213ddabb61 leader=063c306d-3b25-9ea2-cd1d-b631d74ac673 testing.go:149: server-35cc5789-32c0-509d-3472-243f4999d8a5: 2024-06-12T04:32:31.669+1400 [WARN] server-35cc5789-32c0-509d-3472-243f4999d8a5: rejecting vote request since we have a leader: from=22894afa-22a3-fb46-6be6-70213ddabb61 leader=063c306d-3b25-9ea2-cd1d-b631d74ac673 testLoggerAdapter verbose: server-35cc5789-32c0-509d-3472-243f4999d8a5: 2024-06-12T04:32:31.692+1400 [WARN] server-35cc5789-32c0-509d-3472-243f4999d8a5: heartbeat timeout reached, starting election: last-leader=063c306d-3b25-9ea2-cd1d-b631d74ac673 testing.go:149: server-35cc5789-32c0-509d-3472-243f4999d8a5: 2024-06-12T04:32:31.692+1400 [WARN] server-35cc5789-32c0-509d-3472-243f4999d8a5: heartbeat timeout reached, starting election: last-leader=063c306d-3b25-9ea2-cd1d-b631d74ac673 testLoggerAdapter verbose: server-35cc5789-32c0-509d-3472-243f4999d8a5: 2024-06-12T04:32:31.692+1400 [INFO] server-35cc5789-32c0-509d-3472-243f4999d8a5: entering candidate state: node="Node at 35cc5789-32c0-509d-3472-243f4999d8a5 [Candidate]" term=3 testing.go:149: server-35cc5789-32c0-509d-3472-243f4999d8a5: 2024-06-12T04:32:31.692+1400 [INFO] server-35cc5789-32c0-509d-3472-243f4999d8a5: entering candidate state: node="Node at 35cc5789-32c0-509d-3472-243f4999d8a5 [Candidate]" term=3 testLoggerAdapter verbose: server-22894afa-22a3-fb46-6be6-70213ddabb61: 2024-06-12T04:32:31.692+1400 [INFO] server-22894afa-22a3-fb46-6be6-70213ddabb61: duplicate requestVote for same term: term=3 testing.go:149: server-22894afa-22a3-fb46-6be6-70213ddabb61: 2024-06-12T04:32:31.692+1400 [INFO] server-22894afa-22a3-fb46-6be6-70213ddabb61: duplicate requestVote for same term: term=3 testLoggerAdapter verbose: server-22894afa-22a3-fb46-6be6-70213ddabb61: 2024-06-12T04:32:31.753+1400 [WARN] server-22894afa-22a3-fb46-6be6-70213ddabb61: Election timeout reached, restarting election testing.go:149: server-22894afa-22a3-fb46-6be6-70213ddabb61: 2024-06-12T04:32:31.753+1400 [WARN] server-22894afa-22a3-fb46-6be6-70213ddabb61: Election timeout reached, restarting election testLoggerAdapter verbose: server-22894afa-22a3-fb46-6be6-70213ddabb61: 2024-06-12T04:32:31.753+1400 [INFO] server-22894afa-22a3-fb46-6be6-70213ddabb61: entering candidate state: node="Node at 22894afa-22a3-fb46-6be6-70213ddabb61 [Candidate]" term=4 testing.go:149: server-22894afa-22a3-fb46-6be6-70213ddabb61: 2024-06-12T04:32:31.753+1400 [INFO] server-22894afa-22a3-fb46-6be6-70213ddabb61: entering candidate state: node="Node at 22894afa-22a3-fb46-6be6-70213ddabb61 [Candidate]" term=4 testLoggerAdapter verbose: server-35cc5789-32c0-509d-3472-243f4999d8a5: 2024-06-12T04:32:31.753+1400 [INFO] server-35cc5789-32c0-509d-3472-243f4999d8a5: entering follower state: follower="Node at 35cc5789-32c0-509d-3472-243f4999d8a5 [Follower]" leader= testing.go:149: server-35cc5789-32c0-509d-3472-243f4999d8a5: 2024-06-12T04:32:31.753+1400 [INFO] server-35cc5789-32c0-509d-3472-243f4999d8a5: entering follower state: follower="Node at 35cc5789-32c0-509d-3472-243f4999d8a5 [Follower]" leader= testLoggerAdapter verbose: server-22894afa-22a3-fb46-6be6-70213ddabb61: 2024-06-12T04:32:31.753+1400 [INFO] server-22894afa-22a3-fb46-6be6-70213ddabb61: election won: tally=2 testing.go:149: server-22894afa-22a3-fb46-6be6-70213ddabb61: 2024-06-12T04:32:31.753+1400 [INFO] server-22894afa-22a3-fb46-6be6-70213ddabb61: election won: tally=2 testLoggerAdapter verbose: server-22894afa-22a3-fb46-6be6-70213ddabb61: 2024-06-12T04:32:31.753+1400 [INFO] server-22894afa-22a3-fb46-6be6-70213ddabb61: entering leader state: leader="Node at 22894afa-22a3-fb46-6be6-70213ddabb61 [Leader]" testing.go:149: server-22894afa-22a3-fb46-6be6-70213ddabb61: 2024-06-12T04:32:31.753+1400 [INFO] server-22894afa-22a3-fb46-6be6-70213ddabb61: entering leader state: leader="Node at 22894afa-22a3-fb46-6be6-70213ddabb61 [Leader]" testLoggerAdapter verbose: server-22894afa-22a3-fb46-6be6-70213ddabb61: 2024-06-12T04:32:31.753+1400 [INFO] server-22894afa-22a3-fb46-6be6-70213ddabb61: added peer, starting replication: peer=server-35cc5789-32c0-509d-3472-243f4999d8a5 testing.go:149: server-22894afa-22a3-fb46-6be6-70213ddabb61: 2024-06-12T04:32:31.753+1400 [INFO] server-22894afa-22a3-fb46-6be6-70213ddabb61: added peer, starting replication: peer=server-35cc5789-32c0-509d-3472-243f4999d8a5 testLoggerAdapter verbose: server-22894afa-22a3-fb46-6be6-70213ddabb61: 2024-06-12T04:32:31.753+1400 [INFO] server-22894afa-22a3-fb46-6be6-70213ddabb61: pipelining replication: peer="{Voter server-35cc5789-32c0-509d-3472-243f4999d8a5 35cc5789-32c0-509d-3472-243f4999d8a5}" testing.go:149: server-22894afa-22a3-fb46-6be6-70213ddabb61: 2024-06-12T04:32:31.753+1400 [INFO] server-22894afa-22a3-fb46-6be6-70213ddabb61: pipelining replication: peer="{Voter server-35cc5789-32c0-509d-3472-243f4999d8a5 35cc5789-32c0-509d-3472-243f4999d8a5}" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:31.828+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-12T04:32:31.828+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-06-12T04:32:31.933+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:31.82869906 +1400 +14 m=+8.736217639 (1 nodes), 60.678µs from start of poll, 487.772208ms from cluster start. Timeout at 2024-06-12 04:32:31.933912771 +1400 +14 m=+8.841431390, 105.213751ms after stability testing.go:149: cluster: 2024-06-12T04:32:31.933+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:31.82869906 +1400 +14 m=+8.736217639 (1 nodes), 60.678µs from start of poll, 487.772208ms from cluster start. Timeout at 2024-06-12 04:32:31.933912771 +1400 +14 m=+8.841431390, 105.213751ms after stability testLoggerAdapter verbose: server-22894afa-22a3-fb46-6be6-70213ddabb61: 2024-06-12T04:32:31.934+1400 [INFO] server-22894afa-22a3-fb46-6be6-70213ddabb61: aborting pipeline replication: peer="{Voter server-35cc5789-32c0-509d-3472-243f4999d8a5 35cc5789-32c0-509d-3472-243f4999d8a5}" testing.go:149: server-22894afa-22a3-fb46-6be6-70213ddabb61: 2024-06-12T04:32:31.934+1400 [INFO] server-22894afa-22a3-fb46-6be6-70213ddabb61: aborting pipeline replication: peer="{Voter server-35cc5789-32c0-509d-3472-243f4999d8a5 35cc5789-32c0-509d-3472-243f4999d8a5}" --- PASS: TestRaft_RemoveLeader (0.60s) === RUN TestRaft_RemoveLeader_NoShutdown testLoggerAdapter verbose: server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: 2024-06-12T04:32:31.936+1400 [INFO] server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0 Address:3a835b66-4fb4-cfb0-2ed8-f7974de544d0} {Suffrage:Voter ID:server-ff61209f-da10-2114-5db9-ef83dfc32b83 Address:ff61209f-da10-2114-5db9-ef83dfc32b83} {Suffrage:Voter ID:server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0 Address:2b0eb6ae-173a-b016-69ac-0125a01f9cd0}]" testing.go:149: server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: 2024-06-12T04:32:31.936+1400 [INFO] server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0 Address:3a835b66-4fb4-cfb0-2ed8-f7974de544d0} {Suffrage:Voter ID:server-ff61209f-da10-2114-5db9-ef83dfc32b83 Address:ff61209f-da10-2114-5db9-ef83dfc32b83} {Suffrage:Voter ID:server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0 Address:2b0eb6ae-173a-b016-69ac-0125a01f9cd0}]" testLoggerAdapter verbose: server-ff61209f-da10-2114-5db9-ef83dfc32b83: 2024-06-12T04:32:31.936+1400 [INFO] server-ff61209f-da10-2114-5db9-ef83dfc32b83: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0 Address:3a835b66-4fb4-cfb0-2ed8-f7974de544d0} {Suffrage:Voter ID:server-ff61209f-da10-2114-5db9-ef83dfc32b83 Address:ff61209f-da10-2114-5db9-ef83dfc32b83} {Suffrage:Voter ID:server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0 Address:2b0eb6ae-173a-b016-69ac-0125a01f9cd0}]" testing.go:149: server-ff61209f-da10-2114-5db9-ef83dfc32b83: 2024-06-12T04:32:31.936+1400 [INFO] server-ff61209f-da10-2114-5db9-ef83dfc32b83: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0 Address:3a835b66-4fb4-cfb0-2ed8-f7974de544d0} {Suffrage:Voter ID:server-ff61209f-da10-2114-5db9-ef83dfc32b83 Address:ff61209f-da10-2114-5db9-ef83dfc32b83} {Suffrage:Voter ID:server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0 Address:2b0eb6ae-173a-b016-69ac-0125a01f9cd0}]" testLoggerAdapter verbose: server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: 2024-06-12T04:32:31.937+1400 [INFO] server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0 Address:3a835b66-4fb4-cfb0-2ed8-f7974de544d0} {Suffrage:Voter ID:server-ff61209f-da10-2114-5db9-ef83dfc32b83 Address:ff61209f-da10-2114-5db9-ef83dfc32b83} {Suffrage:Voter ID:server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0 Address:2b0eb6ae-173a-b016-69ac-0125a01f9cd0}]" testing.go:149: server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: 2024-06-12T04:32:31.937+1400 [INFO] server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0 Address:3a835b66-4fb4-cfb0-2ed8-f7974de544d0} {Suffrage:Voter ID:server-ff61209f-da10-2114-5db9-ef83dfc32b83 Address:ff61209f-da10-2114-5db9-ef83dfc32b83} {Suffrage:Voter ID:server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0 Address:2b0eb6ae-173a-b016-69ac-0125a01f9cd0}]" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:31.937+1400 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-06-12T04:32:31.937+1400 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: 2024-06-12T04:32:31.937+1400 [INFO] server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: entering follower state: follower="Node at 3a835b66-4fb4-cfb0-2ed8-f7974de544d0 [Follower]" leader= testing.go:149: server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: 2024-06-12T04:32:31.937+1400 [INFO] server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: entering follower state: follower="Node at 3a835b66-4fb4-cfb0-2ed8-f7974de544d0 [Follower]" leader= testLoggerAdapter verbose: server-ff61209f-da10-2114-5db9-ef83dfc32b83: 2024-06-12T04:32:31.937+1400 [INFO] server-ff61209f-da10-2114-5db9-ef83dfc32b83: entering follower state: follower="Node at ff61209f-da10-2114-5db9-ef83dfc32b83 [Follower]" leader= testing.go:149: server-ff61209f-da10-2114-5db9-ef83dfc32b83: 2024-06-12T04:32:31.937+1400 [INFO] server-ff61209f-da10-2114-5db9-ef83dfc32b83: entering follower state: follower="Node at ff61209f-da10-2114-5db9-ef83dfc32b83 [Follower]" leader= testLoggerAdapter verbose: server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: 2024-06-12T04:32:31.937+1400 [INFO] server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: entering follower state: follower="Node at 2b0eb6ae-173a-b016-69ac-0125a01f9cd0 [Follower]" leader= testing.go:149: server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: 2024-06-12T04:32:31.937+1400 [INFO] server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: entering follower state: follower="Node at 2b0eb6ae-173a-b016-69ac-0125a01f9cd0 [Follower]" leader= testLoggerAdapter verbose: server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: 2024-06-12T04:32:32.009+1400 [WARN] server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: 2024-06-12T04:32:32.009+1400 [WARN] server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: 2024-06-12T04:32:32.009+1400 [INFO] server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: entering candidate state: node="Node at 3a835b66-4fb4-cfb0-2ed8-f7974de544d0 [Candidate]" term=2 testing.go:149: server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: 2024-06-12T04:32:32.009+1400 [INFO] server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: entering candidate state: node="Node at 3a835b66-4fb4-cfb0-2ed8-f7974de544d0 [Candidate]" term=2 testLoggerAdapter verbose: server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: 2024-06-12T04:32:32.010+1400 [INFO] server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: election won: tally=2 testing.go:149: server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: 2024-06-12T04:32:32.010+1400 [INFO] server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: election won: tally=2 testLoggerAdapter verbose: server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: 2024-06-12T04:32:32.010+1400 [INFO] server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: entering leader state: leader="Node at 3a835b66-4fb4-cfb0-2ed8-f7974de544d0 [Leader]" testing.go:149: server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: 2024-06-12T04:32:32.010+1400 [INFO] server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: entering leader state: leader="Node at 3a835b66-4fb4-cfb0-2ed8-f7974de544d0 [Leader]" testLoggerAdapter verbose: server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: 2024-06-12T04:32:32.010+1400 [INFO] server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: added peer, starting replication: peer=server-ff61209f-da10-2114-5db9-ef83dfc32b83 testing.go:149: server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: 2024-06-12T04:32:32.010+1400 [INFO] server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: added peer, starting replication: peer=server-ff61209f-da10-2114-5db9-ef83dfc32b83 testLoggerAdapter verbose: server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: 2024-06-12T04:32:32.010+1400 [INFO] server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: added peer, starting replication: peer=server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0 testing.go:149: server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: 2024-06-12T04:32:32.010+1400 [INFO] server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: added peer, starting replication: peer=server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0 testLoggerAdapter verbose: server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: 2024-06-12T04:32:32.010+1400 [INFO] server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: pipelining replication: peer="{Voter server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0 2b0eb6ae-173a-b016-69ac-0125a01f9cd0}" testing.go:149: server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: 2024-06-12T04:32:32.010+1400 [INFO] server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: pipelining replication: peer="{Voter server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0 2b0eb6ae-173a-b016-69ac-0125a01f9cd0}" testLoggerAdapter verbose: server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: 2024-06-12T04:32:32.010+1400 [INFO] server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: pipelining replication: peer="{Voter server-ff61209f-da10-2114-5db9-ef83dfc32b83 ff61209f-da10-2114-5db9-ef83dfc32b83}" testing.go:149: server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: 2024-06-12T04:32:32.010+1400 [INFO] server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: pipelining replication: peer="{Voter server-ff61209f-da10-2114-5db9-ef83dfc32b83 ff61209f-da10-2114-5db9-ef83dfc32b83}" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:32.116+1400 [INFO] stable state for Follower reached at 2024-06-12 04:32:32.010570386 +1400 +14 m=+8.918088965 (2 nodes), 73.250497ms from start of poll, 74.071067ms from cluster start. Timeout at 2024-06-12 04:32:32.116673905 +1400 +14 m=+9.024192484, 106.103519ms after stability testing.go:149: cluster: 2024-06-12T04:32:32.116+1400 [INFO] stable state for Follower reached at 2024-06-12 04:32:32.010570386 +1400 +14 m=+8.918088965 (2 nodes), 73.250497ms from start of poll, 74.071067ms from cluster start. Timeout at 2024-06-12 04:32:32.116673905 +1400 +14 m=+9.024192484, 106.103519ms after stability testLoggerAdapter verbose: cluster: 2024-06-12T04:32:32.116+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-12T04:32:32.116+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2024-06-12T04:32:32.222+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:32.116895737 +1400 +14 m=+9.024414356 (1 nodes), 1.84µs from start of poll, 180.396458ms from cluster start. Timeout at 2024-06-12 04:32:32.222444916 +1400 +14 m=+9.129963495, 105.549139ms after stability testing.go:149: cluster: 2024-06-12T04:32:32.222+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:32.116895737 +1400 +14 m=+9.024414356 (1 nodes), 1.84µs from start of poll, 180.396458ms from cluster start. Timeout at 2024-06-12 04:32:32.222444916 +1400 +14 m=+9.129963495, 105.549139ms after stability testLoggerAdapter verbose: server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: 2024-06-12T04:32:32.226+1400 [INFO] server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: updating configuration: command=RemoveServer server-id=server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0 server-addr= servers="[{Suffrage:Voter ID:server-ff61209f-da10-2114-5db9-ef83dfc32b83 Address:ff61209f-da10-2114-5db9-ef83dfc32b83} {Suffrage:Voter ID:server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0 Address:2b0eb6ae-173a-b016-69ac-0125a01f9cd0}]" testing.go:149: server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: 2024-06-12T04:32:32.226+1400 [INFO] server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: updating configuration: command=RemoveServer server-id=server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0 server-addr= servers="[{Suffrage:Voter ID:server-ff61209f-da10-2114-5db9-ef83dfc32b83 Address:ff61209f-da10-2114-5db9-ef83dfc32b83} {Suffrage:Voter ID:server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0 Address:2b0eb6ae-173a-b016-69ac-0125a01f9cd0}]" testLoggerAdapter verbose: server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: 2024-06-12T04:32:32.226+1400 [INFO] server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: removed ourself, transitioning to follower testing.go:149: server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: 2024-06-12T04:32:32.226+1400 [INFO] server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: removed ourself, transitioning to follower testLoggerAdapter verbose: server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: 2024-06-12T04:32:32.227+1400 [INFO] server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: entering follower state: follower="Node at 3a835b66-4fb4-cfb0-2ed8-f7974de544d0 [Follower]" leader= testing.go:149: server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: 2024-06-12T04:32:32.227+1400 [INFO] server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: entering follower state: follower="Node at 3a835b66-4fb4-cfb0-2ed8-f7974de544d0 [Follower]" leader= testLoggerAdapter verbose: server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: 2024-06-12T04:32:32.227+1400 [INFO] server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: aborting pipeline replication: peer="{Voter server-ff61209f-da10-2114-5db9-ef83dfc32b83 ff61209f-da10-2114-5db9-ef83dfc32b83}" testing.go:149: server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: 2024-06-12T04:32:32.227+1400 [INFO] server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: aborting pipeline replication: peer="{Voter server-ff61209f-da10-2114-5db9-ef83dfc32b83 ff61209f-da10-2114-5db9-ef83dfc32b83}" testLoggerAdapter verbose: server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: 2024-06-12T04:32:32.227+1400 [INFO] server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: aborting pipeline replication: peer="{Voter server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0 2b0eb6ae-173a-b016-69ac-0125a01f9cd0}" testing.go:149: server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: 2024-06-12T04:32:32.227+1400 [INFO] server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: aborting pipeline replication: peer="{Voter server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0 2b0eb6ae-173a-b016-69ac-0125a01f9cd0}" testLoggerAdapter verbose: server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: 2024-06-12T04:32:32.306+1400 [WARN] server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: heartbeat timeout reached, starting election: last-leader=3a835b66-4fb4-cfb0-2ed8-f7974de544d0 testing.go:149: server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: 2024-06-12T04:32:32.306+1400 [WARN] server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: heartbeat timeout reached, starting election: last-leader=3a835b66-4fb4-cfb0-2ed8-f7974de544d0 testLoggerAdapter verbose: server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: 2024-06-12T04:32:32.306+1400 [INFO] server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: entering candidate state: node="Node at 2b0eb6ae-173a-b016-69ac-0125a01f9cd0 [Candidate]" term=3 testing.go:149: server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: 2024-06-12T04:32:32.306+1400 [INFO] server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: entering candidate state: node="Node at 2b0eb6ae-173a-b016-69ac-0125a01f9cd0 [Candidate]" term=3 testLoggerAdapter verbose: server-ff61209f-da10-2114-5db9-ef83dfc32b83: 2024-06-12T04:32:32.306+1400 [WARN] server-ff61209f-da10-2114-5db9-ef83dfc32b83: rejecting vote request since we have a leader: from=2b0eb6ae-173a-b016-69ac-0125a01f9cd0 leader=3a835b66-4fb4-cfb0-2ed8-f7974de544d0 testing.go:149: server-ff61209f-da10-2114-5db9-ef83dfc32b83: 2024-06-12T04:32:32.306+1400 [WARN] server-ff61209f-da10-2114-5db9-ef83dfc32b83: rejecting vote request since we have a leader: from=2b0eb6ae-173a-b016-69ac-0125a01f9cd0 leader=3a835b66-4fb4-cfb0-2ed8-f7974de544d0 testLoggerAdapter verbose: server-ff61209f-da10-2114-5db9-ef83dfc32b83: 2024-06-12T04:32:32.307+1400 [WARN] server-ff61209f-da10-2114-5db9-ef83dfc32b83: heartbeat timeout reached, starting election: last-leader=3a835b66-4fb4-cfb0-2ed8-f7974de544d0 testing.go:149: server-ff61209f-da10-2114-5db9-ef83dfc32b83: 2024-06-12T04:32:32.307+1400 [WARN] server-ff61209f-da10-2114-5db9-ef83dfc32b83: heartbeat timeout reached, starting election: last-leader=3a835b66-4fb4-cfb0-2ed8-f7974de544d0 testLoggerAdapter verbose: server-ff61209f-da10-2114-5db9-ef83dfc32b83: 2024-06-12T04:32:32.307+1400 [INFO] server-ff61209f-da10-2114-5db9-ef83dfc32b83: entering candidate state: node="Node at ff61209f-da10-2114-5db9-ef83dfc32b83 [Candidate]" term=3 testing.go:149: server-ff61209f-da10-2114-5db9-ef83dfc32b83: 2024-06-12T04:32:32.307+1400 [INFO] server-ff61209f-da10-2114-5db9-ef83dfc32b83: entering candidate state: node="Node at ff61209f-da10-2114-5db9-ef83dfc32b83 [Candidate]" term=3 testLoggerAdapter verbose: server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: 2024-06-12T04:32:32.307+1400 [INFO] server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: duplicate requestVote for same term: term=3 testing.go:149: server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: 2024-06-12T04:32:32.307+1400 [INFO] server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: duplicate requestVote for same term: term=3 testLoggerAdapter verbose: server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: 2024-06-12T04:32:32.318+1400 [WARN] server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: not part of stable configuration, aborting election testing.go:149: server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: 2024-06-12T04:32:32.318+1400 [WARN] server-3a835b66-4fb4-cfb0-2ed8-f7974de544d0: not part of stable configuration, aborting election testLoggerAdapter verbose: cluster: 2024-06-12T04:32:32.333+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-12T04:32:32.333+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: 2024-06-12T04:32:32.365+1400 [WARN] server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: Election timeout reached, restarting election testing.go:149: server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: 2024-06-12T04:32:32.365+1400 [WARN] server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: Election timeout reached, restarting election testLoggerAdapter verbose: server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: 2024-06-12T04:32:32.365+1400 [INFO] server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: entering candidate state: node="Node at 2b0eb6ae-173a-b016-69ac-0125a01f9cd0 [Candidate]" term=4 testing.go:149: server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: 2024-06-12T04:32:32.365+1400 [INFO] server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: entering candidate state: node="Node at 2b0eb6ae-173a-b016-69ac-0125a01f9cd0 [Candidate]" term=4 testLoggerAdapter verbose: server-ff61209f-da10-2114-5db9-ef83dfc32b83: 2024-06-12T04:32:32.365+1400 [INFO] server-ff61209f-da10-2114-5db9-ef83dfc32b83: entering follower state: follower="Node at ff61209f-da10-2114-5db9-ef83dfc32b83 [Follower]" leader= testing.go:149: server-ff61209f-da10-2114-5db9-ef83dfc32b83: 2024-06-12T04:32:32.365+1400 [INFO] server-ff61209f-da10-2114-5db9-ef83dfc32b83: entering follower state: follower="Node at ff61209f-da10-2114-5db9-ef83dfc32b83 [Follower]" leader= testLoggerAdapter verbose: server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: 2024-06-12T04:32:32.365+1400 [INFO] server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: election won: tally=2 testing.go:149: server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: 2024-06-12T04:32:32.365+1400 [INFO] server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: election won: tally=2 testLoggerAdapter verbose: server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: 2024-06-12T04:32:32.365+1400 [INFO] server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: entering leader state: leader="Node at 2b0eb6ae-173a-b016-69ac-0125a01f9cd0 [Leader]" testing.go:149: server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: 2024-06-12T04:32:32.365+1400 [INFO] server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: entering leader state: leader="Node at 2b0eb6ae-173a-b016-69ac-0125a01f9cd0 [Leader]" testLoggerAdapter verbose: server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: 2024-06-12T04:32:32.365+1400 [INFO] server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: added peer, starting replication: peer=server-ff61209f-da10-2114-5db9-ef83dfc32b83 testing.go:149: server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: 2024-06-12T04:32:32.365+1400 [INFO] server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: added peer, starting replication: peer=server-ff61209f-da10-2114-5db9-ef83dfc32b83 testLoggerAdapter verbose: server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: 2024-06-12T04:32:32.365+1400 [INFO] server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: pipelining replication: peer="{Voter server-ff61209f-da10-2114-5db9-ef83dfc32b83 ff61209f-da10-2114-5db9-ef83dfc32b83}" testing.go:149: server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: 2024-06-12T04:32:32.365+1400 [INFO] server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: pipelining replication: peer="{Voter server-ff61209f-da10-2114-5db9-ef83dfc32b83 ff61209f-da10-2114-5db9-ef83dfc32b83}" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:32.471+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:32.365769183 +1400 +14 m=+9.273287762 (1 nodes), 32.547275ms from start of poll, 429.269864ms from cluster start. Timeout at 2024-06-12 04:32:32.471807824 +1400 +14 m=+9.379326403, 106.038641ms after stability testing.go:149: cluster: 2024-06-12T04:32:32.471+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:32.365769183 +1400 +14 m=+9.273287762 (1 nodes), 32.547275ms from start of poll, 429.269864ms from cluster start. Timeout at 2024-06-12 04:32:32.471807824 +1400 +14 m=+9.379326403, 106.038641ms after stability testLoggerAdapter verbose: server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: 2024-06-12T04:32:32.577+1400 [INFO] server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: aborting pipeline replication: peer="{Voter server-ff61209f-da10-2114-5db9-ef83dfc32b83 ff61209f-da10-2114-5db9-ef83dfc32b83}" testing.go:149: server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: 2024-06-12T04:32:32.577+1400 [INFO] server-2b0eb6ae-173a-b016-69ac-0125a01f9cd0: aborting pipeline replication: peer="{Voter server-ff61209f-da10-2114-5db9-ef83dfc32b83 ff61209f-da10-2114-5db9-ef83dfc32b83}" --- PASS: TestRaft_RemoveLeader_NoShutdown (0.64s) === RUN TestRaft_RemoveFollower_SplitCluster testLoggerAdapter verbose: server-3752d97e-4476-231f-036e-426ffa2b8cc4: 2024-06-12T04:32:32.580+1400 [INFO] server-3752d97e-4476-231f-036e-426ffa2b8cc4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3752d97e-4476-231f-036e-426ffa2b8cc4 Address:3752d97e-4476-231f-036e-426ffa2b8cc4} {Suffrage:Voter ID:server-35b87fe5-0871-752e-2f20-244b7e1cc588 Address:35b87fe5-0871-752e-2f20-244b7e1cc588} {Suffrage:Voter ID:server-b715d6b2-772a-4683-e9c2-327ca0a06b02 Address:b715d6b2-772a-4683-e9c2-327ca0a06b02} {Suffrage:Voter ID:server-bc730afa-d518-0d9f-9227-85bdb50cda7a Address:bc730afa-d518-0d9f-9227-85bdb50cda7a}]" testing.go:149: server-3752d97e-4476-231f-036e-426ffa2b8cc4: 2024-06-12T04:32:32.580+1400 [INFO] server-3752d97e-4476-231f-036e-426ffa2b8cc4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3752d97e-4476-231f-036e-426ffa2b8cc4 Address:3752d97e-4476-231f-036e-426ffa2b8cc4} {Suffrage:Voter ID:server-35b87fe5-0871-752e-2f20-244b7e1cc588 Address:35b87fe5-0871-752e-2f20-244b7e1cc588} {Suffrage:Voter ID:server-b715d6b2-772a-4683-e9c2-327ca0a06b02 Address:b715d6b2-772a-4683-e9c2-327ca0a06b02} {Suffrage:Voter ID:server-bc730afa-d518-0d9f-9227-85bdb50cda7a Address:bc730afa-d518-0d9f-9227-85bdb50cda7a}]" testLoggerAdapter verbose: server-35b87fe5-0871-752e-2f20-244b7e1cc588: 2024-06-12T04:32:32.580+1400 [INFO] server-35b87fe5-0871-752e-2f20-244b7e1cc588: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3752d97e-4476-231f-036e-426ffa2b8cc4 Address:3752d97e-4476-231f-036e-426ffa2b8cc4} {Suffrage:Voter ID:server-35b87fe5-0871-752e-2f20-244b7e1cc588 Address:35b87fe5-0871-752e-2f20-244b7e1cc588} {Suffrage:Voter ID:server-b715d6b2-772a-4683-e9c2-327ca0a06b02 Address:b715d6b2-772a-4683-e9c2-327ca0a06b02} {Suffrage:Voter ID:server-bc730afa-d518-0d9f-9227-85bdb50cda7a Address:bc730afa-d518-0d9f-9227-85bdb50cda7a}]" testing.go:149: server-35b87fe5-0871-752e-2f20-244b7e1cc588: 2024-06-12T04:32:32.580+1400 [INFO] server-35b87fe5-0871-752e-2f20-244b7e1cc588: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3752d97e-4476-231f-036e-426ffa2b8cc4 Address:3752d97e-4476-231f-036e-426ffa2b8cc4} {Suffrage:Voter ID:server-35b87fe5-0871-752e-2f20-244b7e1cc588 Address:35b87fe5-0871-752e-2f20-244b7e1cc588} {Suffrage:Voter ID:server-b715d6b2-772a-4683-e9c2-327ca0a06b02 Address:b715d6b2-772a-4683-e9c2-327ca0a06b02} {Suffrage:Voter ID:server-bc730afa-d518-0d9f-9227-85bdb50cda7a Address:bc730afa-d518-0d9f-9227-85bdb50cda7a}]" testLoggerAdapter verbose: server-b715d6b2-772a-4683-e9c2-327ca0a06b02: 2024-06-12T04:32:32.580+1400 [INFO] server-b715d6b2-772a-4683-e9c2-327ca0a06b02: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3752d97e-4476-231f-036e-426ffa2b8cc4 Address:3752d97e-4476-231f-036e-426ffa2b8cc4} {Suffrage:Voter ID:server-35b87fe5-0871-752e-2f20-244b7e1cc588 Address:35b87fe5-0871-752e-2f20-244b7e1cc588} {Suffrage:Voter ID:server-b715d6b2-772a-4683-e9c2-327ca0a06b02 Address:b715d6b2-772a-4683-e9c2-327ca0a06b02} {Suffrage:Voter ID:server-bc730afa-d518-0d9f-9227-85bdb50cda7a Address:bc730afa-d518-0d9f-9227-85bdb50cda7a}]" testing.go:149: server-b715d6b2-772a-4683-e9c2-327ca0a06b02: 2024-06-12T04:32:32.580+1400 [INFO] server-b715d6b2-772a-4683-e9c2-327ca0a06b02: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3752d97e-4476-231f-036e-426ffa2b8cc4 Address:3752d97e-4476-231f-036e-426ffa2b8cc4} {Suffrage:Voter ID:server-35b87fe5-0871-752e-2f20-244b7e1cc588 Address:35b87fe5-0871-752e-2f20-244b7e1cc588} {Suffrage:Voter ID:server-b715d6b2-772a-4683-e9c2-327ca0a06b02 Address:b715d6b2-772a-4683-e9c2-327ca0a06b02} {Suffrage:Voter ID:server-bc730afa-d518-0d9f-9227-85bdb50cda7a Address:bc730afa-d518-0d9f-9227-85bdb50cda7a}]" testLoggerAdapter verbose: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.581+1400 [INFO] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3752d97e-4476-231f-036e-426ffa2b8cc4 Address:3752d97e-4476-231f-036e-426ffa2b8cc4} {Suffrage:Voter ID:server-35b87fe5-0871-752e-2f20-244b7e1cc588 Address:35b87fe5-0871-752e-2f20-244b7e1cc588} {Suffrage:Voter ID:server-b715d6b2-772a-4683-e9c2-327ca0a06b02 Address:b715d6b2-772a-4683-e9c2-327ca0a06b02} {Suffrage:Voter ID:server-bc730afa-d518-0d9f-9227-85bdb50cda7a Address:bc730afa-d518-0d9f-9227-85bdb50cda7a}]" testing.go:149: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.581+1400 [INFO] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3752d97e-4476-231f-036e-426ffa2b8cc4 Address:3752d97e-4476-231f-036e-426ffa2b8cc4} {Suffrage:Voter ID:server-35b87fe5-0871-752e-2f20-244b7e1cc588 Address:35b87fe5-0871-752e-2f20-244b7e1cc588} {Suffrage:Voter ID:server-b715d6b2-772a-4683-e9c2-327ca0a06b02 Address:b715d6b2-772a-4683-e9c2-327ca0a06b02} {Suffrage:Voter ID:server-bc730afa-d518-0d9f-9227-85bdb50cda7a Address:bc730afa-d518-0d9f-9227-85bdb50cda7a}]" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:32.581+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-12T04:32:32.581+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-3752d97e-4476-231f-036e-426ffa2b8cc4: 2024-06-12T04:32:32.581+1400 [INFO] server-3752d97e-4476-231f-036e-426ffa2b8cc4: entering follower state: follower="Node at 3752d97e-4476-231f-036e-426ffa2b8cc4 [Follower]" leader= testing.go:149: server-3752d97e-4476-231f-036e-426ffa2b8cc4: 2024-06-12T04:32:32.581+1400 [INFO] server-3752d97e-4476-231f-036e-426ffa2b8cc4: entering follower state: follower="Node at 3752d97e-4476-231f-036e-426ffa2b8cc4 [Follower]" leader= testLoggerAdapter verbose: server-35b87fe5-0871-752e-2f20-244b7e1cc588: 2024-06-12T04:32:32.581+1400 [INFO] server-35b87fe5-0871-752e-2f20-244b7e1cc588: entering follower state: follower="Node at 35b87fe5-0871-752e-2f20-244b7e1cc588 [Follower]" leader= testing.go:149: server-35b87fe5-0871-752e-2f20-244b7e1cc588: 2024-06-12T04:32:32.581+1400 [INFO] server-35b87fe5-0871-752e-2f20-244b7e1cc588: entering follower state: follower="Node at 35b87fe5-0871-752e-2f20-244b7e1cc588 [Follower]" leader= testLoggerAdapter verbose: server-b715d6b2-772a-4683-e9c2-327ca0a06b02: 2024-06-12T04:32:32.581+1400 [INFO] server-b715d6b2-772a-4683-e9c2-327ca0a06b02: entering follower state: follower="Node at b715d6b2-772a-4683-e9c2-327ca0a06b02 [Follower]" leader= testing.go:149: server-b715d6b2-772a-4683-e9c2-327ca0a06b02: 2024-06-12T04:32:32.581+1400 [INFO] server-b715d6b2-772a-4683-e9c2-327ca0a06b02: entering follower state: follower="Node at b715d6b2-772a-4683-e9c2-327ca0a06b02 [Follower]" leader= testLoggerAdapter verbose: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.581+1400 [INFO] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: entering follower state: follower="Node at bc730afa-d518-0d9f-9227-85bdb50cda7a [Follower]" leader= testing.go:149: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.581+1400 [INFO] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: entering follower state: follower="Node at bc730afa-d518-0d9f-9227-85bdb50cda7a [Follower]" leader= testLoggerAdapter verbose: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.636+1400 [WARN] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.636+1400 [WARN] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.636+1400 [INFO] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: entering candidate state: node="Node at bc730afa-d518-0d9f-9227-85bdb50cda7a [Candidate]" term=2 testing.go:149: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.636+1400 [INFO] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: entering candidate state: node="Node at bc730afa-d518-0d9f-9227-85bdb50cda7a [Candidate]" term=2 testLoggerAdapter verbose: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.636+1400 [INFO] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: election won: tally=3 testing.go:149: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.636+1400 [INFO] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: election won: tally=3 testLoggerAdapter verbose: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.636+1400 [INFO] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: entering leader state: leader="Node at bc730afa-d518-0d9f-9227-85bdb50cda7a [Leader]" testing.go:149: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.636+1400 [INFO] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: entering leader state: leader="Node at bc730afa-d518-0d9f-9227-85bdb50cda7a [Leader]" testLoggerAdapter verbose: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.636+1400 [INFO] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: added peer, starting replication: peer=server-3752d97e-4476-231f-036e-426ffa2b8cc4 testing.go:149: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.636+1400 [INFO] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: added peer, starting replication: peer=server-3752d97e-4476-231f-036e-426ffa2b8cc4 testLoggerAdapter verbose: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.636+1400 [INFO] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: added peer, starting replication: peer=server-35b87fe5-0871-752e-2f20-244b7e1cc588 testing.go:149: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.636+1400 [INFO] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: added peer, starting replication: peer=server-35b87fe5-0871-752e-2f20-244b7e1cc588 testLoggerAdapter verbose: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.636+1400 [INFO] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: added peer, starting replication: peer=server-b715d6b2-772a-4683-e9c2-327ca0a06b02 testing.go:149: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.636+1400 [INFO] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: added peer, starting replication: peer=server-b715d6b2-772a-4683-e9c2-327ca0a06b02 testLoggerAdapter verbose: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.636+1400 [INFO] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: pipelining replication: peer="{Voter server-b715d6b2-772a-4683-e9c2-327ca0a06b02 b715d6b2-772a-4683-e9c2-327ca0a06b02}" testing.go:149: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.636+1400 [INFO] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: pipelining replication: peer="{Voter server-b715d6b2-772a-4683-e9c2-327ca0a06b02 b715d6b2-772a-4683-e9c2-327ca0a06b02}" testLoggerAdapter verbose: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.636+1400 [INFO] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: pipelining replication: peer="{Voter server-3752d97e-4476-231f-036e-426ffa2b8cc4 3752d97e-4476-231f-036e-426ffa2b8cc4}" testing.go:149: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.636+1400 [INFO] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: pipelining replication: peer="{Voter server-3752d97e-4476-231f-036e-426ffa2b8cc4 3752d97e-4476-231f-036e-426ffa2b8cc4}" testLoggerAdapter verbose: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.637+1400 [INFO] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: pipelining replication: peer="{Voter server-35b87fe5-0871-752e-2f20-244b7e1cc588 35b87fe5-0871-752e-2f20-244b7e1cc588}" testing.go:149: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.637+1400 [INFO] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: pipelining replication: peer="{Voter server-35b87fe5-0871-752e-2f20-244b7e1cc588 35b87fe5-0871-752e-2f20-244b7e1cc588}" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:32.742+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:32.636950669 +1400 +14 m=+9.544469248 (1 nodes), 55.670046ms from start of poll, 56.596133ms from cluster start. Timeout at 2024-06-12 04:32:32.742258257 +1400 +14 m=+9.649776876, 105.307628ms after stability testing.go:149: cluster: 2024-06-12T04:32:32.742+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:32.636950669 +1400 +14 m=+9.544469248 (1 nodes), 55.670046ms from start of poll, 56.596133ms from cluster start. Timeout at 2024-06-12 04:32:32.742258257 +1400 +14 m=+9.649776876, 105.307628ms after stability testLoggerAdapter verbose: cluster: 2024-06-12T04:32:32.847+1400 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-06-12T04:32:32.847+1400 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2024-06-12T04:32:32.952+1400 [INFO] stable state for Follower reached at 2024-06-12 04:32:32.847783117 +1400 +14 m=+9.755301736 (3 nodes), 2.96µs from start of poll, 267.428621ms from cluster start. Timeout at 2024-06-12 04:32:32.952837514 +1400 +14 m=+9.860356133, 105.054397ms after stability testing.go:149: cluster: 2024-06-12T04:32:32.952+1400 [INFO] stable state for Follower reached at 2024-06-12 04:32:32.847783117 +1400 +14 m=+9.755301736 (3 nodes), 2.96µs from start of poll, 267.428621ms from cluster start. Timeout at 2024-06-12 04:32:32.952837514 +1400 +14 m=+9.860356133, 105.054397ms after stability testLoggerAdapter verbose: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.953+1400 [INFO] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: updating configuration: command=RemoveServer server-id=server-b715d6b2-772a-4683-e9c2-327ca0a06b02 server-addr= servers="[{Suffrage:Voter ID:server-3752d97e-4476-231f-036e-426ffa2b8cc4 Address:3752d97e-4476-231f-036e-426ffa2b8cc4} {Suffrage:Voter ID:server-35b87fe5-0871-752e-2f20-244b7e1cc588 Address:35b87fe5-0871-752e-2f20-244b7e1cc588} {Suffrage:Voter ID:server-bc730afa-d518-0d9f-9227-85bdb50cda7a Address:bc730afa-d518-0d9f-9227-85bdb50cda7a}]" testing.go:149: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.953+1400 [INFO] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: updating configuration: command=RemoveServer server-id=server-b715d6b2-772a-4683-e9c2-327ca0a06b02 server-addr= servers="[{Suffrage:Voter ID:server-3752d97e-4476-231f-036e-426ffa2b8cc4 Address:3752d97e-4476-231f-036e-426ffa2b8cc4} {Suffrage:Voter ID:server-35b87fe5-0871-752e-2f20-244b7e1cc588 Address:35b87fe5-0871-752e-2f20-244b7e1cc588} {Suffrage:Voter ID:server-bc730afa-d518-0d9f-9227-85bdb50cda7a Address:bc730afa-d518-0d9f-9227-85bdb50cda7a}]" testLoggerAdapter verbose: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.953+1400 [INFO] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: removed peer, stopping replication: peer=server-b715d6b2-772a-4683-e9c2-327ca0a06b02 last-index=3 testing.go:149: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.953+1400 [INFO] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: removed peer, stopping replication: peer=server-b715d6b2-772a-4683-e9c2-327ca0a06b02 last-index=3 testLoggerAdapter verbose: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.953+1400 [ERROR] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to pipeline appendEntries: peer="{Voter server-3752d97e-4476-231f-036e-426ffa2b8cc4 3752d97e-4476-231f-036e-426ffa2b8cc4}" error="append pipeline closed" testing.go:149: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.953+1400 [ERROR] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to pipeline appendEntries: peer="{Voter server-3752d97e-4476-231f-036e-426ffa2b8cc4 3752d97e-4476-231f-036e-426ffa2b8cc4}" error="append pipeline closed" testLoggerAdapter verbose: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.953+1400 [INFO] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: aborting pipeline replication: peer="{Voter server-3752d97e-4476-231f-036e-426ffa2b8cc4 3752d97e-4476-231f-036e-426ffa2b8cc4}" testing.go:149: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.953+1400 [INFO] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: aborting pipeline replication: peer="{Voter server-3752d97e-4476-231f-036e-426ffa2b8cc4 3752d97e-4476-231f-036e-426ffa2b8cc4}" testLoggerAdapter verbose: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.953+1400 [ERROR] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to pipeline appendEntries: peer="{Voter server-35b87fe5-0871-752e-2f20-244b7e1cc588 35b87fe5-0871-752e-2f20-244b7e1cc588}" error="append pipeline closed" testing.go:149: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.953+1400 [ERROR] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to pipeline appendEntries: peer="{Voter server-35b87fe5-0871-752e-2f20-244b7e1cc588 35b87fe5-0871-752e-2f20-244b7e1cc588}" error="append pipeline closed" testLoggerAdapter verbose: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.953+1400 [INFO] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: aborting pipeline replication: peer="{Voter server-35b87fe5-0871-752e-2f20-244b7e1cc588 35b87fe5-0871-752e-2f20-244b7e1cc588}" testing.go:149: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.953+1400 [INFO] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: aborting pipeline replication: peer="{Voter server-35b87fe5-0871-752e-2f20-244b7e1cc588 35b87fe5-0871-752e-2f20-244b7e1cc588}" testLoggerAdapter verbose: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.953+1400 [INFO] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: aborting pipeline replication: peer="{Voter server-b715d6b2-772a-4683-e9c2-327ca0a06b02 b715d6b2-772a-4683-e9c2-327ca0a06b02}" testing.go:149: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.953+1400 [INFO] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: aborting pipeline replication: peer="{Voter server-b715d6b2-772a-4683-e9c2-327ca0a06b02 b715d6b2-772a-4683-e9c2-327ca0a06b02}" testLoggerAdapter verbose: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.958+1400 [ERROR] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to heartbeat to: peer=35b87fe5-0871-752e-2f20-244b7e1cc588 error="failed to connect to peer: 35b87fe5-0871-752e-2f20-244b7e1cc588" testing.go:149: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.958+1400 [ERROR] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to heartbeat to: peer=35b87fe5-0871-752e-2f20-244b7e1cc588 error="failed to connect to peer: 35b87fe5-0871-752e-2f20-244b7e1cc588" testLoggerAdapter verbose: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.960+1400 [ERROR] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to appendEntries to: peer="{Voter server-3752d97e-4476-231f-036e-426ffa2b8cc4 3752d97e-4476-231f-036e-426ffa2b8cc4}" error="failed to connect to peer: 3752d97e-4476-231f-036e-426ffa2b8cc4" testing.go:149: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.960+1400 [ERROR] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to appendEntries to: peer="{Voter server-3752d97e-4476-231f-036e-426ffa2b8cc4 3752d97e-4476-231f-036e-426ffa2b8cc4}" error="failed to connect to peer: 3752d97e-4476-231f-036e-426ffa2b8cc4" testLoggerAdapter verbose: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.960+1400 [ERROR] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to heartbeat to: peer=3752d97e-4476-231f-036e-426ffa2b8cc4 error="failed to connect to peer: 3752d97e-4476-231f-036e-426ffa2b8cc4" testing.go:149: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.960+1400 [ERROR] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to heartbeat to: peer=3752d97e-4476-231f-036e-426ffa2b8cc4 error="failed to connect to peer: 3752d97e-4476-231f-036e-426ffa2b8cc4" testLoggerAdapter verbose: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.962+1400 [ERROR] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to appendEntries to: peer="{Voter server-35b87fe5-0871-752e-2f20-244b7e1cc588 35b87fe5-0871-752e-2f20-244b7e1cc588}" error="failed to connect to peer: 35b87fe5-0871-752e-2f20-244b7e1cc588" testing.go:149: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.962+1400 [ERROR] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to appendEntries to: peer="{Voter server-35b87fe5-0871-752e-2f20-244b7e1cc588 35b87fe5-0871-752e-2f20-244b7e1cc588}" error="failed to connect to peer: 35b87fe5-0871-752e-2f20-244b7e1cc588" testLoggerAdapter verbose: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.978+1400 [ERROR] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to heartbeat to: peer=35b87fe5-0871-752e-2f20-244b7e1cc588 error="failed to connect to peer: 35b87fe5-0871-752e-2f20-244b7e1cc588" testing.go:149: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.978+1400 [ERROR] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to heartbeat to: peer=35b87fe5-0871-752e-2f20-244b7e1cc588 error="failed to connect to peer: 35b87fe5-0871-752e-2f20-244b7e1cc588" testLoggerAdapter verbose: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.980+1400 [ERROR] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to heartbeat to: peer=3752d97e-4476-231f-036e-426ffa2b8cc4 error="failed to connect to peer: 3752d97e-4476-231f-036e-426ffa2b8cc4" testing.go:149: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.980+1400 [ERROR] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to heartbeat to: peer=3752d97e-4476-231f-036e-426ffa2b8cc4 error="failed to connect to peer: 3752d97e-4476-231f-036e-426ffa2b8cc4" testLoggerAdapter verbose: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.981+1400 [ERROR] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to appendEntries to: peer="{Voter server-3752d97e-4476-231f-036e-426ffa2b8cc4 3752d97e-4476-231f-036e-426ffa2b8cc4}" error="failed to connect to peer: 3752d97e-4476-231f-036e-426ffa2b8cc4" testing.go:149: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.981+1400 [ERROR] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to appendEntries to: peer="{Voter server-3752d97e-4476-231f-036e-426ffa2b8cc4 3752d97e-4476-231f-036e-426ffa2b8cc4}" error="failed to connect to peer: 3752d97e-4476-231f-036e-426ffa2b8cc4" testLoggerAdapter verbose: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.982+1400 [ERROR] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to appendEntries to: peer="{Voter server-35b87fe5-0871-752e-2f20-244b7e1cc588 35b87fe5-0871-752e-2f20-244b7e1cc588}" error="failed to connect to peer: 35b87fe5-0871-752e-2f20-244b7e1cc588" testing.go:149: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.982+1400 [ERROR] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to appendEntries to: peer="{Voter server-35b87fe5-0871-752e-2f20-244b7e1cc588 35b87fe5-0871-752e-2f20-244b7e1cc588}" error="failed to connect to peer: 35b87fe5-0871-752e-2f20-244b7e1cc588" testLoggerAdapter verbose: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.997+1400 [ERROR] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to heartbeat to: peer=3752d97e-4476-231f-036e-426ffa2b8cc4 error="failed to connect to peer: 3752d97e-4476-231f-036e-426ffa2b8cc4" testing.go:149: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.997+1400 [ERROR] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to heartbeat to: peer=3752d97e-4476-231f-036e-426ffa2b8cc4 error="failed to connect to peer: 3752d97e-4476-231f-036e-426ffa2b8cc4" testLoggerAdapter verbose: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.997+1400 [ERROR] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to heartbeat to: peer=35b87fe5-0871-752e-2f20-244b7e1cc588 error="failed to connect to peer: 35b87fe5-0871-752e-2f20-244b7e1cc588" testing.go:149: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.997+1400 [ERROR] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to heartbeat to: peer=35b87fe5-0871-752e-2f20-244b7e1cc588 error="failed to connect to peer: 35b87fe5-0871-752e-2f20-244b7e1cc588" testLoggerAdapter verbose: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.998+1400 [ERROR] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to appendEntries to: peer="{Voter server-3752d97e-4476-231f-036e-426ffa2b8cc4 3752d97e-4476-231f-036e-426ffa2b8cc4}" error="failed to connect to peer: 3752d97e-4476-231f-036e-426ffa2b8cc4" testing.go:149: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:32.998+1400 [ERROR] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to appendEntries to: peer="{Voter server-3752d97e-4476-231f-036e-426ffa2b8cc4 3752d97e-4476-231f-036e-426ffa2b8cc4}" error="failed to connect to peer: 3752d97e-4476-231f-036e-426ffa2b8cc4" testLoggerAdapter verbose: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:33.001+1400 [ERROR] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to appendEntries to: peer="{Voter server-35b87fe5-0871-752e-2f20-244b7e1cc588 35b87fe5-0871-752e-2f20-244b7e1cc588}" error="failed to connect to peer: 35b87fe5-0871-752e-2f20-244b7e1cc588" testing.go:149: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:33.001+1400 [ERROR] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to appendEntries to: peer="{Voter server-35b87fe5-0871-752e-2f20-244b7e1cc588 35b87fe5-0871-752e-2f20-244b7e1cc588}" error="failed to connect to peer: 35b87fe5-0871-752e-2f20-244b7e1cc588" testLoggerAdapter verbose: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:33.002+1400 [WARN] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to contact: server-id=server-35b87fe5-0871-752e-2f20-244b7e1cc588 time=50.724183ms testing.go:149: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:33.002+1400 [WARN] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to contact: server-id=server-35b87fe5-0871-752e-2f20-244b7e1cc588 time=50.724183ms testLoggerAdapter verbose: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:33.013+1400 [WARN] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to contact: server-id=server-3752d97e-4476-231f-036e-426ffa2b8cc4 time=60.430155ms testing.go:149: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:33.013+1400 [WARN] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to contact: server-id=server-3752d97e-4476-231f-036e-426ffa2b8cc4 time=60.430155ms testLoggerAdapter verbose: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:33.013+1400 [WARN] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to contact: server-id=server-35b87fe5-0871-752e-2f20-244b7e1cc588 time=61.624472ms testing.go:149: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:33.013+1400 [WARN] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to contact: server-id=server-35b87fe5-0871-752e-2f20-244b7e1cc588 time=61.624472ms testLoggerAdapter verbose: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:33.013+1400 [WARN] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to contact quorum of nodes, stepping down testing.go:149: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:33.013+1400 [WARN] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to contact quorum of nodes, stepping down testLoggerAdapter verbose: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:33.013+1400 [INFO] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: entering follower state: follower="Node at bc730afa-d518-0d9f-9227-85bdb50cda7a [Follower]" leader= testing.go:149: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:33.013+1400 [INFO] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: entering follower state: follower="Node at bc730afa-d518-0d9f-9227-85bdb50cda7a [Follower]" leader= testLoggerAdapter verbose: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:33.013+1400 [ERROR] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to appendEntries to: peer="{Voter server-35b87fe5-0871-752e-2f20-244b7e1cc588 35b87fe5-0871-752e-2f20-244b7e1cc588}" error="failed to connect to peer: 35b87fe5-0871-752e-2f20-244b7e1cc588" testing.go:149: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:33.013+1400 [ERROR] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to appendEntries to: peer="{Voter server-35b87fe5-0871-752e-2f20-244b7e1cc588 35b87fe5-0871-752e-2f20-244b7e1cc588}" error="failed to connect to peer: 35b87fe5-0871-752e-2f20-244b7e1cc588" testLoggerAdapter verbose: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:33.013+1400 [ERROR] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to appendEntries to: peer="{Voter server-3752d97e-4476-231f-036e-426ffa2b8cc4 3752d97e-4476-231f-036e-426ffa2b8cc4}" error="failed to connect to peer: 3752d97e-4476-231f-036e-426ffa2b8cc4" testing.go:149: server-bc730afa-d518-0d9f-9227-85bdb50cda7a: 2024-06-12T04:32:33.013+1400 [ERROR] server-bc730afa-d518-0d9f-9227-85bdb50cda7a: failed to appendEntries to: peer="{Voter server-3752d97e-4476-231f-036e-426ffa2b8cc4 3752d97e-4476-231f-036e-426ffa2b8cc4}" error="failed to connect to peer: 3752d97e-4476-231f-036e-426ffa2b8cc4" --- PASS: TestRaft_RemoveFollower_SplitCluster (0.44s) === RUN TestRaft_AddKnownPeer testLoggerAdapter verbose: server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: 2024-06-12T04:32:33.016+1400 [INFO] server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aeee72ce-d122-70b7-0c20-6d2de3753a9b Address:aeee72ce-d122-70b7-0c20-6d2de3753a9b} {Suffrage:Voter ID:server-fc7844a6-f4d7-529a-910b-35bc8543d450 Address:fc7844a6-f4d7-529a-910b-35bc8543d450} {Suffrage:Voter ID:server-00b2c93b-52c4-0149-9b88-876f27241473 Address:00b2c93b-52c4-0149-9b88-876f27241473}]" testing.go:149: server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: 2024-06-12T04:32:33.016+1400 [INFO] server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aeee72ce-d122-70b7-0c20-6d2de3753a9b Address:aeee72ce-d122-70b7-0c20-6d2de3753a9b} {Suffrage:Voter ID:server-fc7844a6-f4d7-529a-910b-35bc8543d450 Address:fc7844a6-f4d7-529a-910b-35bc8543d450} {Suffrage:Voter ID:server-00b2c93b-52c4-0149-9b88-876f27241473 Address:00b2c93b-52c4-0149-9b88-876f27241473}]" testLoggerAdapter verbose: server-fc7844a6-f4d7-529a-910b-35bc8543d450: 2024-06-12T04:32:33.017+1400 [INFO] server-fc7844a6-f4d7-529a-910b-35bc8543d450: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aeee72ce-d122-70b7-0c20-6d2de3753a9b Address:aeee72ce-d122-70b7-0c20-6d2de3753a9b} {Suffrage:Voter ID:server-fc7844a6-f4d7-529a-910b-35bc8543d450 Address:fc7844a6-f4d7-529a-910b-35bc8543d450} {Suffrage:Voter ID:server-00b2c93b-52c4-0149-9b88-876f27241473 Address:00b2c93b-52c4-0149-9b88-876f27241473}]" testing.go:149: server-fc7844a6-f4d7-529a-910b-35bc8543d450: 2024-06-12T04:32:33.017+1400 [INFO] server-fc7844a6-f4d7-529a-910b-35bc8543d450: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aeee72ce-d122-70b7-0c20-6d2de3753a9b Address:aeee72ce-d122-70b7-0c20-6d2de3753a9b} {Suffrage:Voter ID:server-fc7844a6-f4d7-529a-910b-35bc8543d450 Address:fc7844a6-f4d7-529a-910b-35bc8543d450} {Suffrage:Voter ID:server-00b2c93b-52c4-0149-9b88-876f27241473 Address:00b2c93b-52c4-0149-9b88-876f27241473}]" testLoggerAdapter verbose: server-00b2c93b-52c4-0149-9b88-876f27241473: 2024-06-12T04:32:33.017+1400 [INFO] server-00b2c93b-52c4-0149-9b88-876f27241473: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aeee72ce-d122-70b7-0c20-6d2de3753a9b Address:aeee72ce-d122-70b7-0c20-6d2de3753a9b} {Suffrage:Voter ID:server-fc7844a6-f4d7-529a-910b-35bc8543d450 Address:fc7844a6-f4d7-529a-910b-35bc8543d450} {Suffrage:Voter ID:server-00b2c93b-52c4-0149-9b88-876f27241473 Address:00b2c93b-52c4-0149-9b88-876f27241473}]" testing.go:149: server-00b2c93b-52c4-0149-9b88-876f27241473: 2024-06-12T04:32:33.017+1400 [INFO] server-00b2c93b-52c4-0149-9b88-876f27241473: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aeee72ce-d122-70b7-0c20-6d2de3753a9b Address:aeee72ce-d122-70b7-0c20-6d2de3753a9b} {Suffrage:Voter ID:server-fc7844a6-f4d7-529a-910b-35bc8543d450 Address:fc7844a6-f4d7-529a-910b-35bc8543d450} {Suffrage:Voter ID:server-00b2c93b-52c4-0149-9b88-876f27241473 Address:00b2c93b-52c4-0149-9b88-876f27241473}]" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:33.017+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-12T04:32:33.017+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: 2024-06-12T04:32:33.017+1400 [INFO] server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: entering follower state: follower="Node at aeee72ce-d122-70b7-0c20-6d2de3753a9b [Follower]" leader= testing.go:149: server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: 2024-06-12T04:32:33.017+1400 [INFO] server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: entering follower state: follower="Node at aeee72ce-d122-70b7-0c20-6d2de3753a9b [Follower]" leader= testLoggerAdapter verbose: server-fc7844a6-f4d7-529a-910b-35bc8543d450: 2024-06-12T04:32:33.017+1400 [INFO] server-fc7844a6-f4d7-529a-910b-35bc8543d450: entering follower state: follower="Node at fc7844a6-f4d7-529a-910b-35bc8543d450 [Follower]" leader= testing.go:149: server-fc7844a6-f4d7-529a-910b-35bc8543d450: 2024-06-12T04:32:33.017+1400 [INFO] server-fc7844a6-f4d7-529a-910b-35bc8543d450: entering follower state: follower="Node at fc7844a6-f4d7-529a-910b-35bc8543d450 [Follower]" leader= testLoggerAdapter verbose: server-00b2c93b-52c4-0149-9b88-876f27241473: 2024-06-12T04:32:33.017+1400 [INFO] server-00b2c93b-52c4-0149-9b88-876f27241473: entering follower state: follower="Node at 00b2c93b-52c4-0149-9b88-876f27241473 [Follower]" leader= testing.go:149: server-00b2c93b-52c4-0149-9b88-876f27241473: 2024-06-12T04:32:33.017+1400 [INFO] server-00b2c93b-52c4-0149-9b88-876f27241473: entering follower state: follower="Node at 00b2c93b-52c4-0149-9b88-876f27241473 [Follower]" leader= testLoggerAdapter verbose: server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: 2024-06-12T04:32:33.069+1400 [WARN] server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: 2024-06-12T04:32:33.069+1400 [WARN] server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: 2024-06-12T04:32:33.069+1400 [INFO] server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: entering candidate state: node="Node at aeee72ce-d122-70b7-0c20-6d2de3753a9b [Candidate]" term=2 testing.go:149: server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: 2024-06-12T04:32:33.069+1400 [INFO] server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: entering candidate state: node="Node at aeee72ce-d122-70b7-0c20-6d2de3753a9b [Candidate]" term=2 testLoggerAdapter verbose: server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: 2024-06-12T04:32:33.069+1400 [INFO] server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: election won: tally=2 testing.go:149: server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: 2024-06-12T04:32:33.069+1400 [INFO] server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: election won: tally=2 testLoggerAdapter verbose: server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: 2024-06-12T04:32:33.069+1400 [INFO] server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: entering leader state: leader="Node at aeee72ce-d122-70b7-0c20-6d2de3753a9b [Leader]" testing.go:149: server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: 2024-06-12T04:32:33.069+1400 [INFO] server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: entering leader state: leader="Node at aeee72ce-d122-70b7-0c20-6d2de3753a9b [Leader]" testLoggerAdapter verbose: server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: 2024-06-12T04:32:33.069+1400 [INFO] server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: added peer, starting replication: peer=server-fc7844a6-f4d7-529a-910b-35bc8543d450 testing.go:149: server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: 2024-06-12T04:32:33.069+1400 [INFO] server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: added peer, starting replication: peer=server-fc7844a6-f4d7-529a-910b-35bc8543d450 testLoggerAdapter verbose: server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: 2024-06-12T04:32:33.069+1400 [INFO] server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: added peer, starting replication: peer=server-00b2c93b-52c4-0149-9b88-876f27241473 testing.go:149: server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: 2024-06-12T04:32:33.069+1400 [INFO] server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: added peer, starting replication: peer=server-00b2c93b-52c4-0149-9b88-876f27241473 testLoggerAdapter verbose: server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: 2024-06-12T04:32:33.069+1400 [INFO] server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: pipelining replication: peer="{Voter server-00b2c93b-52c4-0149-9b88-876f27241473 00b2c93b-52c4-0149-9b88-876f27241473}" testing.go:149: server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: 2024-06-12T04:32:33.069+1400 [INFO] server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: pipelining replication: peer="{Voter server-00b2c93b-52c4-0149-9b88-876f27241473 00b2c93b-52c4-0149-9b88-876f27241473}" testLoggerAdapter verbose: server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: 2024-06-12T04:32:33.069+1400 [INFO] server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: pipelining replication: peer="{Voter server-fc7844a6-f4d7-529a-910b-35bc8543d450 fc7844a6-f4d7-529a-910b-35bc8543d450}" testing.go:149: server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: 2024-06-12T04:32:33.069+1400 [INFO] server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: pipelining replication: peer="{Voter server-fc7844a6-f4d7-529a-910b-35bc8543d450 fc7844a6-f4d7-529a-910b-35bc8543d450}" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:33.175+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:33.069825524 +1400 +14 m=+9.977344103 (1 nodes), 52.332686ms from start of poll, 53.090659ms from cluster start. Timeout at 2024-06-12 04:32:33.175791968 +1400 +14 m=+10.083310547, 105.966444ms after stability testing.go:149: cluster: 2024-06-12T04:32:33.175+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:33.069825524 +1400 +14 m=+9.977344103 (1 nodes), 52.332686ms from start of poll, 53.090659ms from cluster start. Timeout at 2024-06-12 04:32:33.175791968 +1400 +14 m=+10.083310547, 105.966444ms after stability testLoggerAdapter verbose: cluster: 2024-06-12T04:32:33.175+1400 [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2024-06-12T04:32:33.175+1400 [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2024-06-12T04:32:33.281+1400 [INFO] stable state for Follower reached at 2024-06-12 04:32:33.17601228 +1400 +14 m=+10.083530859 (2 nodes), 2.08µs from start of poll, 159.277415ms from cluster start. Timeout at 2024-06-12 04:32:33.281394106 +1400 +14 m=+10.188912685, 105.381826ms after stability testing.go:149: cluster: 2024-06-12T04:32:33.281+1400 [INFO] stable state for Follower reached at 2024-06-12 04:32:33.17601228 +1400 +14 m=+10.083530859 (2 nodes), 2.08µs from start of poll, 159.277415ms from cluster start. Timeout at 2024-06-12 04:32:33.281394106 +1400 +14 m=+10.188912685, 105.381826ms after stability testLoggerAdapter verbose: server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: 2024-06-12T04:32:33.281+1400 [INFO] server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: updating configuration: command=AddStaging server-id=server-fc7844a6-f4d7-529a-910b-35bc8543d450 server-addr=fc7844a6-f4d7-529a-910b-35bc8543d450 servers="[{Suffrage:Voter ID:server-aeee72ce-d122-70b7-0c20-6d2de3753a9b Address:aeee72ce-d122-70b7-0c20-6d2de3753a9b} {Suffrage:Voter ID:server-fc7844a6-f4d7-529a-910b-35bc8543d450 Address:fc7844a6-f4d7-529a-910b-35bc8543d450} {Suffrage:Voter ID:server-00b2c93b-52c4-0149-9b88-876f27241473 Address:00b2c93b-52c4-0149-9b88-876f27241473}]" testing.go:149: server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: 2024-06-12T04:32:33.281+1400 [INFO] server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: updating configuration: command=AddStaging server-id=server-fc7844a6-f4d7-529a-910b-35bc8543d450 server-addr=fc7844a6-f4d7-529a-910b-35bc8543d450 servers="[{Suffrage:Voter ID:server-aeee72ce-d122-70b7-0c20-6d2de3753a9b Address:aeee72ce-d122-70b7-0c20-6d2de3753a9b} {Suffrage:Voter ID:server-fc7844a6-f4d7-529a-910b-35bc8543d450 Address:fc7844a6-f4d7-529a-910b-35bc8543d450} {Suffrage:Voter ID:server-00b2c93b-52c4-0149-9b88-876f27241473 Address:00b2c93b-52c4-0149-9b88-876f27241473}]" testLoggerAdapter verbose: server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: 2024-06-12T04:32:33.281+1400 [INFO] server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: aborting pipeline replication: peer="{Voter server-00b2c93b-52c4-0149-9b88-876f27241473 00b2c93b-52c4-0149-9b88-876f27241473}" testing.go:149: server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: 2024-06-12T04:32:33.281+1400 [INFO] server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: aborting pipeline replication: peer="{Voter server-00b2c93b-52c4-0149-9b88-876f27241473 00b2c93b-52c4-0149-9b88-876f27241473}" testLoggerAdapter verbose: server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: 2024-06-12T04:32:33.282+1400 [INFO] server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: aborting pipeline replication: peer="{Voter server-fc7844a6-f4d7-529a-910b-35bc8543d450 fc7844a6-f4d7-529a-910b-35bc8543d450}" testing.go:149: server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: 2024-06-12T04:32:33.282+1400 [INFO] server-aeee72ce-d122-70b7-0c20-6d2de3753a9b: aborting pipeline replication: peer="{Voter server-fc7844a6-f4d7-529a-910b-35bc8543d450 fc7844a6-f4d7-529a-910b-35bc8543d450}" --- PASS: TestRaft_AddKnownPeer (0.27s) === RUN TestRaft_RemoveUnknownPeer testLoggerAdapter verbose: server-a26699da-244b-4810-d306-09f303bf1791: 2024-06-12T04:32:33.284+1400 [INFO] server-a26699da-244b-4810-d306-09f303bf1791: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a26699da-244b-4810-d306-09f303bf1791 Address:a26699da-244b-4810-d306-09f303bf1791} {Suffrage:Voter ID:server-104b0210-66ce-5126-f4a9-3e646128c308 Address:104b0210-66ce-5126-f4a9-3e646128c308} {Suffrage:Voter ID:server-1dcf86cc-12f0-c8c7-4730-9f107a299f98 Address:1dcf86cc-12f0-c8c7-4730-9f107a299f98}]" testing.go:149: server-a26699da-244b-4810-d306-09f303bf1791: 2024-06-12T04:32:33.284+1400 [INFO] server-a26699da-244b-4810-d306-09f303bf1791: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a26699da-244b-4810-d306-09f303bf1791 Address:a26699da-244b-4810-d306-09f303bf1791} {Suffrage:Voter ID:server-104b0210-66ce-5126-f4a9-3e646128c308 Address:104b0210-66ce-5126-f4a9-3e646128c308} {Suffrage:Voter ID:server-1dcf86cc-12f0-c8c7-4730-9f107a299f98 Address:1dcf86cc-12f0-c8c7-4730-9f107a299f98}]" testLoggerAdapter verbose: server-104b0210-66ce-5126-f4a9-3e646128c308: 2024-06-12T04:32:33.284+1400 [INFO] server-104b0210-66ce-5126-f4a9-3e646128c308: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a26699da-244b-4810-d306-09f303bf1791 Address:a26699da-244b-4810-d306-09f303bf1791} {Suffrage:Voter ID:server-104b0210-66ce-5126-f4a9-3e646128c308 Address:104b0210-66ce-5126-f4a9-3e646128c308} {Suffrage:Voter ID:server-1dcf86cc-12f0-c8c7-4730-9f107a299f98 Address:1dcf86cc-12f0-c8c7-4730-9f107a299f98}]" testing.go:149: server-104b0210-66ce-5126-f4a9-3e646128c308: 2024-06-12T04:32:33.284+1400 [INFO] server-104b0210-66ce-5126-f4a9-3e646128c308: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a26699da-244b-4810-d306-09f303bf1791 Address:a26699da-244b-4810-d306-09f303bf1791} {Suffrage:Voter ID:server-104b0210-66ce-5126-f4a9-3e646128c308 Address:104b0210-66ce-5126-f4a9-3e646128c308} {Suffrage:Voter ID:server-1dcf86cc-12f0-c8c7-4730-9f107a299f98 Address:1dcf86cc-12f0-c8c7-4730-9f107a299f98}]" testLoggerAdapter verbose: server-1dcf86cc-12f0-c8c7-4730-9f107a299f98: 2024-06-12T04:32:33.285+1400 [INFO] server-1dcf86cc-12f0-c8c7-4730-9f107a299f98: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a26699da-244b-4810-d306-09f303bf1791 Address:a26699da-244b-4810-d306-09f303bf1791} {Suffrage:Voter ID:server-104b0210-66ce-5126-f4a9-3e646128c308 Address:104b0210-66ce-5126-f4a9-3e646128c308} {Suffrage:Voter ID:server-1dcf86cc-12f0-c8c7-4730-9f107a299f98 Address:1dcf86cc-12f0-c8c7-4730-9f107a299f98}]" testing.go:149: server-1dcf86cc-12f0-c8c7-4730-9f107a299f98: 2024-06-12T04:32:33.285+1400 [INFO] server-1dcf86cc-12f0-c8c7-4730-9f107a299f98: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a26699da-244b-4810-d306-09f303bf1791 Address:a26699da-244b-4810-d306-09f303bf1791} {Suffrage:Voter ID:server-104b0210-66ce-5126-f4a9-3e646128c308 Address:104b0210-66ce-5126-f4a9-3e646128c308} {Suffrage:Voter ID:server-1dcf86cc-12f0-c8c7-4730-9f107a299f98 Address:1dcf86cc-12f0-c8c7-4730-9f107a299f98}]" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:33.285+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-12T04:32:33.285+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-a26699da-244b-4810-d306-09f303bf1791: 2024-06-12T04:32:33.285+1400 [INFO] server-a26699da-244b-4810-d306-09f303bf1791: entering follower state: follower="Node at a26699da-244b-4810-d306-09f303bf1791 [Follower]" leader= testing.go:149: server-a26699da-244b-4810-d306-09f303bf1791: 2024-06-12T04:32:33.285+1400 [INFO] server-a26699da-244b-4810-d306-09f303bf1791: entering follower state: follower="Node at a26699da-244b-4810-d306-09f303bf1791 [Follower]" leader= testLoggerAdapter verbose: server-104b0210-66ce-5126-f4a9-3e646128c308: 2024-06-12T04:32:33.285+1400 [INFO] server-104b0210-66ce-5126-f4a9-3e646128c308: entering follower state: follower="Node at 104b0210-66ce-5126-f4a9-3e646128c308 [Follower]" leader= testing.go:149: server-104b0210-66ce-5126-f4a9-3e646128c308: 2024-06-12T04:32:33.285+1400 [INFO] server-104b0210-66ce-5126-f4a9-3e646128c308: entering follower state: follower="Node at 104b0210-66ce-5126-f4a9-3e646128c308 [Follower]" leader= testLoggerAdapter verbose: server-1dcf86cc-12f0-c8c7-4730-9f107a299f98: 2024-06-12T04:32:33.285+1400 [INFO] server-1dcf86cc-12f0-c8c7-4730-9f107a299f98: entering follower state: follower="Node at 1dcf86cc-12f0-c8c7-4730-9f107a299f98 [Follower]" leader= testing.go:149: server-1dcf86cc-12f0-c8c7-4730-9f107a299f98: 2024-06-12T04:32:33.285+1400 [INFO] server-1dcf86cc-12f0-c8c7-4730-9f107a299f98: entering follower state: follower="Node at 1dcf86cc-12f0-c8c7-4730-9f107a299f98 [Follower]" leader= testLoggerAdapter verbose: server-104b0210-66ce-5126-f4a9-3e646128c308: 2024-06-12T04:32:33.343+1400 [WARN] server-104b0210-66ce-5126-f4a9-3e646128c308: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-104b0210-66ce-5126-f4a9-3e646128c308: 2024-06-12T04:32:33.343+1400 [WARN] server-104b0210-66ce-5126-f4a9-3e646128c308: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-104b0210-66ce-5126-f4a9-3e646128c308: 2024-06-12T04:32:33.343+1400 [INFO] server-104b0210-66ce-5126-f4a9-3e646128c308: entering candidate state: node="Node at 104b0210-66ce-5126-f4a9-3e646128c308 [Candidate]" term=2 testing.go:149: server-104b0210-66ce-5126-f4a9-3e646128c308: 2024-06-12T04:32:33.343+1400 [INFO] server-104b0210-66ce-5126-f4a9-3e646128c308: entering candidate state: node="Node at 104b0210-66ce-5126-f4a9-3e646128c308 [Candidate]" term=2 testLoggerAdapter verbose: server-104b0210-66ce-5126-f4a9-3e646128c308: 2024-06-12T04:32:33.343+1400 [INFO] server-104b0210-66ce-5126-f4a9-3e646128c308: election won: tally=2 testing.go:149: server-104b0210-66ce-5126-f4a9-3e646128c308: 2024-06-12T04:32:33.343+1400 [INFO] server-104b0210-66ce-5126-f4a9-3e646128c308: election won: tally=2 testLoggerAdapter verbose: server-104b0210-66ce-5126-f4a9-3e646128c308: 2024-06-12T04:32:33.343+1400 [INFO] server-104b0210-66ce-5126-f4a9-3e646128c308: entering leader state: leader="Node at 104b0210-66ce-5126-f4a9-3e646128c308 [Leader]" testing.go:149: server-104b0210-66ce-5126-f4a9-3e646128c308: 2024-06-12T04:32:33.343+1400 [INFO] server-104b0210-66ce-5126-f4a9-3e646128c308: entering leader state: leader="Node at 104b0210-66ce-5126-f4a9-3e646128c308 [Leader]" testLoggerAdapter verbose: server-104b0210-66ce-5126-f4a9-3e646128c308: 2024-06-12T04:32:33.343+1400 [INFO] server-104b0210-66ce-5126-f4a9-3e646128c308: added peer, starting replication: peer=server-a26699da-244b-4810-d306-09f303bf1791 testing.go:149: server-104b0210-66ce-5126-f4a9-3e646128c308: 2024-06-12T04:32:33.343+1400 [INFO] server-104b0210-66ce-5126-f4a9-3e646128c308: added peer, starting replication: peer=server-a26699da-244b-4810-d306-09f303bf1791 testLoggerAdapter verbose: server-104b0210-66ce-5126-f4a9-3e646128c308: 2024-06-12T04:32:33.343+1400 [INFO] server-104b0210-66ce-5126-f4a9-3e646128c308: added peer, starting replication: peer=server-1dcf86cc-12f0-c8c7-4730-9f107a299f98 testing.go:149: server-104b0210-66ce-5126-f4a9-3e646128c308: 2024-06-12T04:32:33.343+1400 [INFO] server-104b0210-66ce-5126-f4a9-3e646128c308: added peer, starting replication: peer=server-1dcf86cc-12f0-c8c7-4730-9f107a299f98 testLoggerAdapter verbose: server-104b0210-66ce-5126-f4a9-3e646128c308: 2024-06-12T04:32:33.344+1400 [INFO] server-104b0210-66ce-5126-f4a9-3e646128c308: pipelining replication: peer="{Voter server-a26699da-244b-4810-d306-09f303bf1791 a26699da-244b-4810-d306-09f303bf1791}" testing.go:149: server-104b0210-66ce-5126-f4a9-3e646128c308: 2024-06-12T04:32:33.344+1400 [INFO] server-104b0210-66ce-5126-f4a9-3e646128c308: pipelining replication: peer="{Voter server-a26699da-244b-4810-d306-09f303bf1791 a26699da-244b-4810-d306-09f303bf1791}" testLoggerAdapter verbose: server-104b0210-66ce-5126-f4a9-3e646128c308: 2024-06-12T04:32:33.344+1400 [INFO] server-104b0210-66ce-5126-f4a9-3e646128c308: pipelining replication: peer="{Voter server-1dcf86cc-12f0-c8c7-4730-9f107a299f98 1dcf86cc-12f0-c8c7-4730-9f107a299f98}" testing.go:149: server-104b0210-66ce-5126-f4a9-3e646128c308: 2024-06-12T04:32:33.344+1400 [INFO] server-104b0210-66ce-5126-f4a9-3e646128c308: pipelining replication: peer="{Voter server-1dcf86cc-12f0-c8c7-4730-9f107a299f98 1dcf86cc-12f0-c8c7-4730-9f107a299f98}" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:33.449+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:33.343860068 +1400 +14 m=+10.251378687 (1 nodes), 58.666539ms from start of poll, 59.437071ms from cluster start. Timeout at 2024-06-12 04:32:33.449532963 +1400 +14 m=+10.357051582, 105.672895ms after stability testing.go:149: cluster: 2024-06-12T04:32:33.449+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:33.343860068 +1400 +14 m=+10.251378687 (1 nodes), 58.666539ms from start of poll, 59.437071ms from cluster start. Timeout at 2024-06-12 04:32:33.449532963 +1400 +14 m=+10.357051582, 105.672895ms after stability testLoggerAdapter verbose: server-104b0210-66ce-5126-f4a9-3e646128c308: 2024-06-12T04:32:33.449+1400 [INFO] server-104b0210-66ce-5126-f4a9-3e646128c308: updating configuration: command=RemoveServer server-id=785bb06e-d5af-b38c-cd05-ae5106d67551 server-addr= servers="[{Suffrage:Voter ID:server-a26699da-244b-4810-d306-09f303bf1791 Address:a26699da-244b-4810-d306-09f303bf1791} {Suffrage:Voter ID:server-104b0210-66ce-5126-f4a9-3e646128c308 Address:104b0210-66ce-5126-f4a9-3e646128c308} {Suffrage:Voter ID:server-1dcf86cc-12f0-c8c7-4730-9f107a299f98 Address:1dcf86cc-12f0-c8c7-4730-9f107a299f98}]" testing.go:149: server-104b0210-66ce-5126-f4a9-3e646128c308: 2024-06-12T04:32:33.449+1400 [INFO] server-104b0210-66ce-5126-f4a9-3e646128c308: updating configuration: command=RemoveServer server-id=785bb06e-d5af-b38c-cd05-ae5106d67551 server-addr= servers="[{Suffrage:Voter ID:server-a26699da-244b-4810-d306-09f303bf1791 Address:a26699da-244b-4810-d306-09f303bf1791} {Suffrage:Voter ID:server-104b0210-66ce-5126-f4a9-3e646128c308 Address:104b0210-66ce-5126-f4a9-3e646128c308} {Suffrage:Voter ID:server-1dcf86cc-12f0-c8c7-4730-9f107a299f98 Address:1dcf86cc-12f0-c8c7-4730-9f107a299f98}]" testLoggerAdapter verbose: server-104b0210-66ce-5126-f4a9-3e646128c308: 2024-06-12T04:32:33.450+1400 [INFO] server-104b0210-66ce-5126-f4a9-3e646128c308: aborting pipeline replication: peer="{Voter server-a26699da-244b-4810-d306-09f303bf1791 a26699da-244b-4810-d306-09f303bf1791}" testing.go:149: server-104b0210-66ce-5126-f4a9-3e646128c308: 2024-06-12T04:32:33.450+1400 [INFO] server-104b0210-66ce-5126-f4a9-3e646128c308: aborting pipeline replication: peer="{Voter server-a26699da-244b-4810-d306-09f303bf1791 a26699da-244b-4810-d306-09f303bf1791}" testLoggerAdapter verbose: server-104b0210-66ce-5126-f4a9-3e646128c308: 2024-06-12T04:32:33.450+1400 [INFO] server-104b0210-66ce-5126-f4a9-3e646128c308: aborting pipeline replication: peer="{Voter server-1dcf86cc-12f0-c8c7-4730-9f107a299f98 1dcf86cc-12f0-c8c7-4730-9f107a299f98}" testing.go:149: server-104b0210-66ce-5126-f4a9-3e646128c308: 2024-06-12T04:32:33.450+1400 [INFO] server-104b0210-66ce-5126-f4a9-3e646128c308: aborting pipeline replication: peer="{Voter server-1dcf86cc-12f0-c8c7-4730-9f107a299f98 1dcf86cc-12f0-c8c7-4730-9f107a299f98}" --- PASS: TestRaft_RemoveUnknownPeer (0.17s) === RUN TestRaft_SnapshotRestore testLoggerAdapter verbose: server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: 2024-06-12T04:32:33.451+1400 [INFO] server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fa3b7c40-9756-1c66-e936-5c0087df4c6e Address:fa3b7c40-9756-1c66-e936-5c0087df4c6e}]" testing.go:149: server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: 2024-06-12T04:32:33.451+1400 [INFO] server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fa3b7c40-9756-1c66-e936-5c0087df4c6e Address:fa3b7c40-9756-1c66-e936-5c0087df4c6e}]" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:33.451+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-12T04:32:33.451+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: 2024-06-12T04:32:33.451+1400 [INFO] server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: entering follower state: follower="Node at fa3b7c40-9756-1c66-e936-5c0087df4c6e [Follower]" leader= testing.go:149: server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: 2024-06-12T04:32:33.451+1400 [INFO] server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: entering follower state: follower="Node at fa3b7c40-9756-1c66-e936-5c0087df4c6e [Follower]" leader= testLoggerAdapter verbose: server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: 2024-06-12T04:32:33.513+1400 [WARN] server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: 2024-06-12T04:32:33.513+1400 [WARN] server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: 2024-06-12T04:32:33.513+1400 [INFO] server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: entering candidate state: node="Node at fa3b7c40-9756-1c66-e936-5c0087df4c6e [Candidate]" term=2 testing.go:149: server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: 2024-06-12T04:32:33.513+1400 [INFO] server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: entering candidate state: node="Node at fa3b7c40-9756-1c66-e936-5c0087df4c6e [Candidate]" term=2 testLoggerAdapter verbose: server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: 2024-06-12T04:32:33.513+1400 [INFO] server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: election won: tally=1 testing.go:149: server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: 2024-06-12T04:32:33.513+1400 [INFO] server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: election won: tally=1 testLoggerAdapter verbose: server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: 2024-06-12T04:32:33.513+1400 [INFO] server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: entering leader state: leader="Node at fa3b7c40-9756-1c66-e936-5c0087df4c6e [Leader]" testing.go:149: server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: 2024-06-12T04:32:33.513+1400 [INFO] server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: entering leader state: leader="Node at fa3b7c40-9756-1c66-e936-5c0087df4c6e [Leader]" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:33.619+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:33.51383066 +1400 +14 m=+10.421349279 (1 nodes), 61.955661ms from start of poll, 62.25981ms from cluster start. Timeout at 2024-06-12 04:32:33.61991862 +1400 +14 m=+10.527437239, 106.08796ms after stability testing.go:149: cluster: 2024-06-12T04:32:33.619+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:33.51383066 +1400 +14 m=+10.421349279 (1 nodes), 61.955661ms from start of poll, 62.25981ms from cluster start. Timeout at 2024-06-12 04:32:33.61991862 +1400 +14 m=+10.527437239, 106.08796ms after stability testLoggerAdapter verbose: server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: 2024-06-12T04:32:33.623+1400 [INFO] server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: starting snapshot up to: index=102 testing.go:149: server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: 2024-06-12T04:32:33.623+1400 [INFO] server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: starting snapshot up to: index=102 testing.go:153: 2024-06-12T04:32:33.624+1400 [INFO] creating new snapshot: path=/tmp/raft321927674/snapshots/2-102-1718116353624.tmp testLoggerAdapter verbose: server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: 2024-06-12T04:32:33.640+1400 [INFO] server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: compacting logs: from=1 to=92 testing.go:149: server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: 2024-06-12T04:32:33.640+1400 [INFO] server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: 2024-06-12T04:32:33.640+1400 [INFO] server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: snapshot complete up to: index=102 testing.go:149: server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: 2024-06-12T04:32:33.640+1400 [INFO] server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: snapshot complete up to: index=102 testLoggerAdapter verbose: server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: 2024-06-12T04:32:33.641+1400 [INFO] server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: restored from snapshot: id=2-102-1718116353624 testing.go:149: server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: 2024-06-12T04:32:33.641+1400 [INFO] server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: restored from snapshot: id=2-102-1718116353624 testLoggerAdapter verbose: server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: 2024-06-12T04:32:33.641+1400 [INFO] server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fa3b7c40-9756-1c66-e936-5c0087df4c6e Address:fa3b7c40-9756-1c66-e936-5c0087df4c6e}]" testing.go:149: server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: 2024-06-12T04:32:33.641+1400 [INFO] server-fa3b7c40-9756-1c66-e936-5c0087df4c6e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fa3b7c40-9756-1c66-e936-5c0087df4c6e Address:fa3b7c40-9756-1c66-e936-5c0087df4c6e}]" --- PASS: TestRaft_SnapshotRestore (0.19s) === RUN TestRaft_NoRestoreOnStart testLoggerAdapter verbose: server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: 2024-06-12T04:32:33.642+1400 [INFO] server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b3285295-113b-81b1-255e-ae5ef1b3d8ea Address:b3285295-113b-81b1-255e-ae5ef1b3d8ea}]" testing.go:149: server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: 2024-06-12T04:32:33.642+1400 [INFO] server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b3285295-113b-81b1-255e-ae5ef1b3d8ea Address:b3285295-113b-81b1-255e-ae5ef1b3d8ea}]" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:33.642+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-12T04:32:33.642+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: 2024-06-12T04:32:33.642+1400 [INFO] server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: entering follower state: follower="Node at b3285295-113b-81b1-255e-ae5ef1b3d8ea [Follower]" leader= testing.go:149: server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: 2024-06-12T04:32:33.642+1400 [INFO] server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: entering follower state: follower="Node at b3285295-113b-81b1-255e-ae5ef1b3d8ea [Follower]" leader= testLoggerAdapter verbose: server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: 2024-06-12T04:32:33.726+1400 [WARN] server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: 2024-06-12T04:32:33.726+1400 [WARN] server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: 2024-06-12T04:32:33.726+1400 [INFO] server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: entering candidate state: node="Node at b3285295-113b-81b1-255e-ae5ef1b3d8ea [Candidate]" term=2 testing.go:149: server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: 2024-06-12T04:32:33.726+1400 [INFO] server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: entering candidate state: node="Node at b3285295-113b-81b1-255e-ae5ef1b3d8ea [Candidate]" term=2 testLoggerAdapter verbose: server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: 2024-06-12T04:32:33.726+1400 [INFO] server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: election won: tally=1 testing.go:149: server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: 2024-06-12T04:32:33.726+1400 [INFO] server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: election won: tally=1 testLoggerAdapter verbose: server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: 2024-06-12T04:32:33.726+1400 [INFO] server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: entering leader state: leader="Node at b3285295-113b-81b1-255e-ae5ef1b3d8ea [Leader]" testing.go:149: server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: 2024-06-12T04:32:33.726+1400 [INFO] server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: entering leader state: leader="Node at b3285295-113b-81b1-255e-ae5ef1b3d8ea [Leader]" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:33.832+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:33.726804951 +1400 +14 m=+10.634323530 (1 nodes), 84.196664ms from start of poll, 84.532292ms from cluster start. Timeout at 2024-06-12 04:32:33.832781635 +1400 +14 m=+10.740300214, 105.976684ms after stability testing.go:149: cluster: 2024-06-12T04:32:33.832+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:33.726804951 +1400 +14 m=+10.634323530 (1 nodes), 84.196664ms from start of poll, 84.532292ms from cluster start. Timeout at 2024-06-12 04:32:33.832781635 +1400 +14 m=+10.740300214, 105.976684ms after stability testLoggerAdapter verbose: server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: 2024-06-12T04:32:33.834+1400 [INFO] server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: starting snapshot up to: index=102 testing.go:149: server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: 2024-06-12T04:32:33.834+1400 [INFO] server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: starting snapshot up to: index=102 testing.go:153: 2024-06-12T04:32:33.834+1400 [INFO] creating new snapshot: path=/tmp/raft3375294114/snapshots/2-102-1718116353834.tmp testLoggerAdapter verbose: server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: 2024-06-12T04:32:33.848+1400 [INFO] server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: compacting logs: from=1 to=92 testing.go:149: server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: 2024-06-12T04:32:33.848+1400 [INFO] server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: 2024-06-12T04:32:33.848+1400 [INFO] server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: snapshot complete up to: index=102 testing.go:149: server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: 2024-06-12T04:32:33.848+1400 [INFO] server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: snapshot complete up to: index=102 testLoggerAdapter verbose: server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: 2024-06-12T04:32:33.848+1400 [INFO] server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b3285295-113b-81b1-255e-ae5ef1b3d8ea Address:b3285295-113b-81b1-255e-ae5ef1b3d8ea}]" testing.go:149: server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: 2024-06-12T04:32:33.848+1400 [INFO] server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b3285295-113b-81b1-255e-ae5ef1b3d8ea Address:b3285295-113b-81b1-255e-ae5ef1b3d8ea}]" --- PASS: TestRaft_NoRestoreOnStart (0.21s) === RUN TestRaft_SnapshotRestore_PeerChange raft_test.go:1026: Disabled unreliable test --- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s) === RUN TestRaft_AutoSnapshot raft_test.go:1132: Disabled unreliable test --- SKIP: TestRaft_AutoSnapshot (0.00s) === RUN TestRaft_UserSnapshot testLoggerAdapter verbose: server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: 2024-06-12T04:32:33.849+1400 [INFO] server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b3fd43eb-bd2c-988a-969a-f6131bc82b82 Address:b3fd43eb-bd2c-988a-969a-f6131bc82b82}]" testing.go:149: server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: 2024-06-12T04:32:33.849+1400 [INFO] server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b3fd43eb-bd2c-988a-969a-f6131bc82b82 Address:b3fd43eb-bd2c-988a-969a-f6131bc82b82}]" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:33.849+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-12T04:32:33.849+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: 2024-06-12T04:32:33.849+1400 [INFO] server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: entering follower state: follower="Node at b3285295-113b-81b1-255e-ae5ef1b3d8ea [Follower]" leader= testLoggerAdapter verbose: server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: 2024-06-12T04:32:33.849+1400 [INFO] server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: entering follower state: follower="Node at b3fd43eb-bd2c-988a-969a-f6131bc82b82 [Follower]" leader= testing.go:149: server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: 2024-06-12T04:32:33.849+1400 [INFO] server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: entering follower state: follower="Node at b3fd43eb-bd2c-988a-969a-f6131bc82b82 [Follower]" leader= testLoggerAdapter verbose: server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: 2024-06-12T04:32:33.910+1400 [WARN] server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: 2024-06-12T04:32:33.910+1400 [INFO] server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: entering candidate state: node="Node at b3285295-113b-81b1-255e-ae5ef1b3d8ea [Candidate]" term=3 testLoggerAdapter verbose: server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: 2024-06-12T04:32:33.910+1400 [INFO] server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: election won: tally=1 testLoggerAdapter verbose: server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: 2024-06-12T04:32:33.910+1400 [INFO] server-b3285295-113b-81b1-255e-ae5ef1b3d8ea: entering leader state: leader="Node at b3285295-113b-81b1-255e-ae5ef1b3d8ea [Leader]" testLoggerAdapter verbose: server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: 2024-06-12T04:32:33.918+1400 [WARN] server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: 2024-06-12T04:32:33.918+1400 [WARN] server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: 2024-06-12T04:32:33.918+1400 [INFO] server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: entering candidate state: node="Node at b3fd43eb-bd2c-988a-969a-f6131bc82b82 [Candidate]" term=2 testing.go:149: server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: 2024-06-12T04:32:33.918+1400 [INFO] server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: entering candidate state: node="Node at b3fd43eb-bd2c-988a-969a-f6131bc82b82 [Candidate]" term=2 testLoggerAdapter verbose: server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: 2024-06-12T04:32:33.919+1400 [INFO] server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: election won: tally=1 testing.go:149: server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: 2024-06-12T04:32:33.919+1400 [INFO] server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: election won: tally=1 testLoggerAdapter verbose: server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: 2024-06-12T04:32:33.919+1400 [INFO] server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: entering leader state: leader="Node at b3fd43eb-bd2c-988a-969a-f6131bc82b82 [Leader]" testing.go:149: server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: 2024-06-12T04:32:33.919+1400 [INFO] server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: entering leader state: leader="Node at b3fd43eb-bd2c-988a-969a-f6131bc82b82 [Leader]" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:34.025+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:33.919175741 +1400 +14 m=+10.826694360 (1 nodes), 69.286919ms from start of poll, 69.538549ms from cluster start. Timeout at 2024-06-12 04:32:34.02500335 +1400 +14 m=+10.932521969, 105.827609ms after stability testing.go:149: cluster: 2024-06-12T04:32:34.025+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:33.919175741 +1400 +14 m=+10.826694360 (1 nodes), 69.286919ms from start of poll, 69.538549ms from cluster start. Timeout at 2024-06-12 04:32:34.02500335 +1400 +14 m=+10.932521969, 105.827609ms after stability testLoggerAdapter verbose: server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: 2024-06-12T04:32:34.025+1400 [ERROR] server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: failed to take snapshot: error="nothing new to snapshot" testing.go:149: server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: 2024-06-12T04:32:34.025+1400 [ERROR] server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: failed to take snapshot: error="nothing new to snapshot" testLoggerAdapter verbose: server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: 2024-06-12T04:32:34.025+1400 [INFO] server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: starting snapshot up to: index=12 testing.go:149: server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: 2024-06-12T04:32:34.025+1400 [INFO] server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: starting snapshot up to: index=12 testing.go:153: 2024-06-12T04:32:34.025+1400 [INFO] creating new snapshot: path=/tmp/raft163758661/snapshots/2-12-1718116354025.tmp testLoggerAdapter verbose: server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: 2024-06-12T04:32:34.040+1400 [INFO] server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: compacting logs: from=1 to=2 testing.go:149: server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: 2024-06-12T04:32:34.040+1400 [INFO] server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: compacting logs: from=1 to=2 testLoggerAdapter verbose: server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: 2024-06-12T04:32:34.040+1400 [INFO] server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: snapshot complete up to: index=12 testing.go:149: server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: 2024-06-12T04:32:34.040+1400 [INFO] server-b3fd43eb-bd2c-988a-969a-f6131bc82b82: snapshot complete up to: index=12 --- PASS: TestRaft_UserSnapshot (0.19s) === RUN TestRaft_UserRestore testLoggerAdapter verbose: server-31dc35fc-435f-67cd-0a44-c91d963b34d6: 2024-06-12T04:32:34.042+1400 [INFO] server-31dc35fc-435f-67cd-0a44-c91d963b34d6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31dc35fc-435f-67cd-0a44-c91d963b34d6 Address:31dc35fc-435f-67cd-0a44-c91d963b34d6} {Suffrage:Voter ID:server-e3f27379-9cff-8617-88f3-80539ce8e42f Address:e3f27379-9cff-8617-88f3-80539ce8e42f} {Suffrage:Voter ID:server-b9be970b-d5d2-a979-7099-2aa5e66846e3 Address:b9be970b-d5d2-a979-7099-2aa5e66846e3}]" testing.go:149: server-31dc35fc-435f-67cd-0a44-c91d963b34d6: 2024-06-12T04:32:34.042+1400 [INFO] server-31dc35fc-435f-67cd-0a44-c91d963b34d6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31dc35fc-435f-67cd-0a44-c91d963b34d6 Address:31dc35fc-435f-67cd-0a44-c91d963b34d6} {Suffrage:Voter ID:server-e3f27379-9cff-8617-88f3-80539ce8e42f Address:e3f27379-9cff-8617-88f3-80539ce8e42f} {Suffrage:Voter ID:server-b9be970b-d5d2-a979-7099-2aa5e66846e3 Address:b9be970b-d5d2-a979-7099-2aa5e66846e3}]" testLoggerAdapter verbose: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.042+1400 [INFO] server-e3f27379-9cff-8617-88f3-80539ce8e42f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31dc35fc-435f-67cd-0a44-c91d963b34d6 Address:31dc35fc-435f-67cd-0a44-c91d963b34d6} {Suffrage:Voter ID:server-e3f27379-9cff-8617-88f3-80539ce8e42f Address:e3f27379-9cff-8617-88f3-80539ce8e42f} {Suffrage:Voter ID:server-b9be970b-d5d2-a979-7099-2aa5e66846e3 Address:b9be970b-d5d2-a979-7099-2aa5e66846e3}]" testing.go:149: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.042+1400 [INFO] server-e3f27379-9cff-8617-88f3-80539ce8e42f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31dc35fc-435f-67cd-0a44-c91d963b34d6 Address:31dc35fc-435f-67cd-0a44-c91d963b34d6} {Suffrage:Voter ID:server-e3f27379-9cff-8617-88f3-80539ce8e42f Address:e3f27379-9cff-8617-88f3-80539ce8e42f} {Suffrage:Voter ID:server-b9be970b-d5d2-a979-7099-2aa5e66846e3 Address:b9be970b-d5d2-a979-7099-2aa5e66846e3}]" testLoggerAdapter verbose: server-b9be970b-d5d2-a979-7099-2aa5e66846e3: 2024-06-12T04:32:34.042+1400 [INFO] server-b9be970b-d5d2-a979-7099-2aa5e66846e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31dc35fc-435f-67cd-0a44-c91d963b34d6 Address:31dc35fc-435f-67cd-0a44-c91d963b34d6} {Suffrage:Voter ID:server-e3f27379-9cff-8617-88f3-80539ce8e42f Address:e3f27379-9cff-8617-88f3-80539ce8e42f} {Suffrage:Voter ID:server-b9be970b-d5d2-a979-7099-2aa5e66846e3 Address:b9be970b-d5d2-a979-7099-2aa5e66846e3}]" testing.go:149: server-b9be970b-d5d2-a979-7099-2aa5e66846e3: 2024-06-12T04:32:34.042+1400 [INFO] server-b9be970b-d5d2-a979-7099-2aa5e66846e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31dc35fc-435f-67cd-0a44-c91d963b34d6 Address:31dc35fc-435f-67cd-0a44-c91d963b34d6} {Suffrage:Voter ID:server-e3f27379-9cff-8617-88f3-80539ce8e42f Address:e3f27379-9cff-8617-88f3-80539ce8e42f} {Suffrage:Voter ID:server-b9be970b-d5d2-a979-7099-2aa5e66846e3 Address:b9be970b-d5d2-a979-7099-2aa5e66846e3}]" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:34.043+1400 [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2024-06-12T04:32:34.043+1400 [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-31dc35fc-435f-67cd-0a44-c91d963b34d6: 2024-06-12T04:32:34.043+1400 [INFO] server-31dc35fc-435f-67cd-0a44-c91d963b34d6: entering follower state: follower="Node at 31dc35fc-435f-67cd-0a44-c91d963b34d6 [Follower]" leader= testing.go:149: server-31dc35fc-435f-67cd-0a44-c91d963b34d6: 2024-06-12T04:32:34.043+1400 [INFO] server-31dc35fc-435f-67cd-0a44-c91d963b34d6: entering follower state: follower="Node at 31dc35fc-435f-67cd-0a44-c91d963b34d6 [Follower]" leader= testLoggerAdapter verbose: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.043+1400 [INFO] server-e3f27379-9cff-8617-88f3-80539ce8e42f: entering follower state: follower="Node at e3f27379-9cff-8617-88f3-80539ce8e42f [Follower]" leader= testing.go:149: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.043+1400 [INFO] server-e3f27379-9cff-8617-88f3-80539ce8e42f: entering follower state: follower="Node at e3f27379-9cff-8617-88f3-80539ce8e42f [Follower]" leader= testLoggerAdapter verbose: server-b9be970b-d5d2-a979-7099-2aa5e66846e3: 2024-06-12T04:32:34.043+1400 [INFO] server-b9be970b-d5d2-a979-7099-2aa5e66846e3: entering follower state: follower="Node at b9be970b-d5d2-a979-7099-2aa5e66846e3 [Follower]" leader= testing.go:149: server-b9be970b-d5d2-a979-7099-2aa5e66846e3: 2024-06-12T04:32:34.043+1400 [INFO] server-b9be970b-d5d2-a979-7099-2aa5e66846e3: entering follower state: follower="Node at b9be970b-d5d2-a979-7099-2aa5e66846e3 [Follower]" leader= testLoggerAdapter verbose: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.096+1400 [WARN] server-e3f27379-9cff-8617-88f3-80539ce8e42f: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.096+1400 [WARN] server-e3f27379-9cff-8617-88f3-80539ce8e42f: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.096+1400 [INFO] server-e3f27379-9cff-8617-88f3-80539ce8e42f: entering candidate state: node="Node at e3f27379-9cff-8617-88f3-80539ce8e42f [Candidate]" term=2 testing.go:149: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.096+1400 [INFO] server-e3f27379-9cff-8617-88f3-80539ce8e42f: entering candidate state: node="Node at e3f27379-9cff-8617-88f3-80539ce8e42f [Candidate]" term=2 testLoggerAdapter verbose: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.096+1400 [INFO] server-e3f27379-9cff-8617-88f3-80539ce8e42f: election won: tally=2 testing.go:149: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.096+1400 [INFO] server-e3f27379-9cff-8617-88f3-80539ce8e42f: election won: tally=2 testLoggerAdapter verbose: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.096+1400 [INFO] server-e3f27379-9cff-8617-88f3-80539ce8e42f: entering leader state: leader="Node at e3f27379-9cff-8617-88f3-80539ce8e42f [Leader]" testing.go:149: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.096+1400 [INFO] server-e3f27379-9cff-8617-88f3-80539ce8e42f: entering leader state: leader="Node at e3f27379-9cff-8617-88f3-80539ce8e42f [Leader]" testLoggerAdapter verbose: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.096+1400 [INFO] server-e3f27379-9cff-8617-88f3-80539ce8e42f: added peer, starting replication: peer=server-31dc35fc-435f-67cd-0a44-c91d963b34d6 testing.go:149: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.096+1400 [INFO] server-e3f27379-9cff-8617-88f3-80539ce8e42f: added peer, starting replication: peer=server-31dc35fc-435f-67cd-0a44-c91d963b34d6 testLoggerAdapter verbose: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.096+1400 [INFO] server-e3f27379-9cff-8617-88f3-80539ce8e42f: added peer, starting replication: peer=server-b9be970b-d5d2-a979-7099-2aa5e66846e3 testing.go:149: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.096+1400 [INFO] server-e3f27379-9cff-8617-88f3-80539ce8e42f: added peer, starting replication: peer=server-b9be970b-d5d2-a979-7099-2aa5e66846e3 testLoggerAdapter verbose: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.096+1400 [INFO] server-e3f27379-9cff-8617-88f3-80539ce8e42f: pipelining replication: peer="{Voter server-b9be970b-d5d2-a979-7099-2aa5e66846e3 b9be970b-d5d2-a979-7099-2aa5e66846e3}" testing.go:149: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.096+1400 [INFO] server-e3f27379-9cff-8617-88f3-80539ce8e42f: pipelining replication: peer="{Voter server-b9be970b-d5d2-a979-7099-2aa5e66846e3 b9be970b-d5d2-a979-7099-2aa5e66846e3}" testLoggerAdapter verbose: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.096+1400 [INFO] server-e3f27379-9cff-8617-88f3-80539ce8e42f: pipelining replication: peer="{Voter server-31dc35fc-435f-67cd-0a44-c91d963b34d6 31dc35fc-435f-67cd-0a44-c91d963b34d6}" testing.go:149: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.096+1400 [INFO] server-e3f27379-9cff-8617-88f3-80539ce8e42f: pipelining replication: peer="{Voter server-31dc35fc-435f-67cd-0a44-c91d963b34d6 31dc35fc-435f-67cd-0a44-c91d963b34d6}" testLoggerAdapter verbose: cluster: 2024-06-12T04:32:34.202+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:34.096898455 +1400 +14 m=+11.004417034 (1 nodes), 53.827232ms from start of poll, 54.617204ms from cluster start. Timeout at 2024-06-12 04:32:34.202090967 +1400 +14 m=+11.109609546, 105.192512ms after stability testing.go:149: cluster: 2024-06-12T04:32:34.202+1400 [INFO] stable state for Leader reached at 2024-06-12 04:32:34.096898455 +1400 +14 m=+11.004417034 (1 nodes), 53.827232ms from start of poll, 54.617204ms from cluster start. Timeout at 2024-06-12 04:32:34.202090967 +1400 +14 m=+11.109609546, 105.192512ms after stability testLoggerAdapter verbose: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.202+1400 [INFO] server-e3f27379-9cff-8617-88f3-80539ce8e42f: starting snapshot up to: index=12 testing.go:149: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.202+1400 [INFO] server-e3f27379-9cff-8617-88f3-80539ce8e42f: starting snapshot up to: index=12 testing.go:153: 2024-06-12T04:32:34.202+1400 [INFO] creating new snapshot: path=/tmp/raft1004055718/snapshots/2-12-1718116354202.tmp testLoggerAdapter verbose: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.217+1400 [INFO] server-e3f27379-9cff-8617-88f3-80539ce8e42f: snapshot complete up to: index=12 testing.go:149: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.217+1400 [INFO] server-e3f27379-9cff-8617-88f3-80539ce8e42f: snapshot complete up to: index=12 testing.go:153: 2024-06-12T04:32:34.218+1400 [INFO] creating new snapshot: path=/tmp/raft1004055718/snapshots/2-23-1718116354218.tmp testLoggerAdapter verbose: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.232+1400 [INFO] server-e3f27379-9cff-8617-88f3-80539ce8e42f: copied to local snapshot: bytes=71 testing.go:149: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.232+1400 [INFO] server-e3f27379-9cff-8617-88f3-80539ce8e42f: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.233+1400 [INFO] server-e3f27379-9cff-8617-88f3-80539ce8e42f: restored user snapshot: index=1 testing.go:149: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.233+1400 [INFO] server-e3f27379-9cff-8617-88f3-80539ce8e42f: restored user snapshot: index=1 testLoggerAdapter verbose: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.233+1400 [ERROR] server-e3f27379-9cff-8617-88f3-80539ce8e42f: failed to get log: index=23 error="log not found" testing.go:149: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.233+1400 [ERROR] server-e3f27379-9cff-8617-88f3-80539ce8e42f: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.233+1400 [INFO] server-e3f27379-9cff-8617-88f3-80539ce8e42f: aborting pipeline replication: peer="{Voter server-b9be970b-d5d2-a979-7099-2aa5e66846e3 b9be970b-d5d2-a979-7099-2aa5e66846e3}" testing.go:149: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.233+1400 [INFO] server-e3f27379-9cff-8617-88f3-80539ce8e42f: aborting pipeline replication: peer="{Voter server-b9be970b-d5d2-a979-7099-2aa5e66846e3 b9be970b-d5d2-a979-7099-2aa5e66846e3}" testLoggerAdapter verbose: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.233+1400 [ERROR] server-e3f27379-9cff-8617-88f3-80539ce8e42f: failed to get log: index=23 error="log not found" testing.go:149: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.233+1400 [ERROR] server-e3f27379-9cff-8617-88f3-80539ce8e42f: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.233+1400 [INFO] server-e3f27379-9cff-8617-88f3-80539ce8e42f: aborting pipeline replication: peer="{Voter server-31dc35fc-435f-67cd-0a44-c91d963b34d6 31dc35fc-435f-67cd-0a44-c91d963b34d6}" testing.go:149: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.233+1400 [INFO] server-e3f27379-9cff-8617-88f3-80539ce8e42f: aborting pipeline replication: peer="{Voter server-31dc35fc-435f-67cd-0a44-c91d963b34d6 31dc35fc-435f-67cd-0a44-c91d963b34d6}" testLoggerAdapter verbose: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.243+1400 [ERROR] server-e3f27379-9cff-8617-88f3-80539ce8e42f: failed to get log: index=23 error="log not found" testing.go:149: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.243+1400 [ERROR] server-e3f27379-9cff-8617-88f3-80539ce8e42f: failed to get log: index=23 error="log not found" testing.go:153: 2024-06-12T04:32:34.244+1400 [INFO] creating new snapshot: path=/tmp/raft605465991/snapshots/2-23-1718116354244.tmp testLoggerAdapter verbose: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.244+1400 [ERROR] server-e3f27379-9cff-8617-88f3-80539ce8e42f: failed to get log: index=23 error="log not found" testing.go:149: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.244+1400 [ERROR] server-e3f27379-9cff-8617-88f3-80539ce8e42f: failed to get log: index=23 error="log not found" testing.go:153: 2024-06-12T04:32:34.245+1400 [INFO] creating new snapshot: path=/tmp/raft669631066/snapshots/2-23-1718116354245.tmp testLoggerAdapter verbose: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.293+1400 [WARN] server-e3f27379-9cff-8617-88f3-80539ce8e42f: failed to contact: server-id=server-31dc35fc-435f-67cd-0a44-c91d963b34d6 time=52.036296ms testing.go:149: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.293+1400 [WARN] server-e3f27379-9cff-8617-88f3-80539ce8e42f: failed to contact: server-id=server-31dc35fc-435f-67cd-0a44-c91d963b34d6 time=52.036296ms testLoggerAdapter verbose: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.293+1400 [WARN] server-e3f27379-9cff-8617-88f3-80539ce8e42f: failed to contact: server-id=server-b9be970b-d5d2-a979-7099-2aa5e66846e3 time=51.997138ms testing.go:149: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.293+1400 [WARN] server-e3f27379-9cff-8617-88f3-80539ce8e42f: failed to contact: server-id=server-b9be970b-d5d2-a979-7099-2aa5e66846e3 time=51.997138ms testLoggerAdapter verbose: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.293+1400 [WARN] server-e3f27379-9cff-8617-88f3-80539ce8e42f: failed to contact quorum of nodes, stepping down testing.go:149: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.293+1400 [WARN] server-e3f27379-9cff-8617-88f3-80539ce8e42f: failed to contact quorum of nodes, stepping down testLoggerAdapter verbose: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.293+1400 [INFO] server-e3f27379-9cff-8617-88f3-80539ce8e42f: entering follower state: follower="Node at e3f27379-9cff-8617-88f3-80539ce8e42f [Follower]" leader= testing.go:149: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.293+1400 [INFO] server-e3f27379-9cff-8617-88f3-80539ce8e42f: entering follower state: follower="Node at e3f27379-9cff-8617-88f3-80539ce8e42f [Follower]" leader= testLoggerAdapter verbose: cluster: 2024-06-12T04:32:34.293+1400 [ERROR] Restore failed: leadership lost while committing log testing.go:149: cluster: 2024-06-12T04:32:34.293+1400 [ERROR] Restore failed: leadership lost while committing log testLoggerAdapter verbose: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.298+1400 [ERROR] server-e3f27379-9cff-8617-88f3-80539ce8e42f: failed to heartbeat to: peer=31dc35fc-435f-67cd-0a44-c91d963b34d6 error="command timed out" testing.go:149: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.298+1400 [ERROR] server-e3f27379-9cff-8617-88f3-80539ce8e42f: failed to heartbeat to: peer=31dc35fc-435f-67cd-0a44-c91d963b34d6 error="command timed out" testLoggerAdapter verbose: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.298+1400 [ERROR] server-e3f27379-9cff-8617-88f3-80539ce8e42f: failed to heartbeat to: peer=b9be970b-d5d2-a979-7099-2aa5e66846e3 error="command timed out" testing.go:149: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.298+1400 [ERROR] server-e3f27379-9cff-8617-88f3-80539ce8e42f: failed to heartbeat to: peer=b9be970b-d5d2-a979-7099-2aa5e66846e3 error="command timed out" testLoggerAdapter verbose: server-b9be970b-d5d2-a979-7099-2aa5e66846e3: 2024-06-12T04:32:34.333+1400 [INFO] server-b9be970b-d5d2-a979-7099-2aa5e66846e3: copied to local snapshot: bytes=71 testing.go:149: server-b9be970b-d5d2-a979-7099-2aa5e66846e3: 2024-06-12T04:32:34.333+1400 [INFO] server-b9be970b-d5d2-a979-7099-2aa5e66846e3: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-31dc35fc-435f-67cd-0a44-c91d963b34d6: 2024-06-12T04:32:34.336+1400 [INFO] server-31dc35fc-435f-67cd-0a44-c91d963b34d6: copied to local snapshot: bytes=71 testing.go:149: server-31dc35fc-435f-67cd-0a44-c91d963b34d6: 2024-06-12T04:32:34.336+1400 [INFO] server-31dc35fc-435f-67cd-0a44-c91d963b34d6: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.336+1400 [WARN] server-e3f27379-9cff-8617-88f3-80539ce8e42f: installSnapshot rejected to: peer="{Voter server-31dc35fc-435f-67cd-0a44-c91d963b34d6 31dc35fc-435f-67cd-0a44-c91d963b34d6}" testing.go:149: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.336+1400 [WARN] server-e3f27379-9cff-8617-88f3-80539ce8e42f: installSnapshot rejected to: peer="{Voter server-31dc35fc-435f-67cd-0a44-c91d963b34d6 31dc35fc-435f-67cd-0a44-c91d963b34d6}" testLoggerAdapter verbose: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.363+1400 [ERROR] server-e3f27379-9cff-8617-88f3-80539ce8e42f: failed to heartbeat to: peer=31dc35fc-435f-67cd-0a44-c91d963b34d6 error="command timed out" testing.go:149: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.363+1400 [ERROR] server-e3f27379-9cff-8617-88f3-80539ce8e42f: failed to heartbeat to: peer=31dc35fc-435f-67cd-0a44-c91d963b34d6 error="command timed out" testLoggerAdapter verbose: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.369+1400 [ERROR] server-e3f27379-9cff-8617-88f3-80539ce8e42f: failed to heartbeat to: peer=b9be970b-d5d2-a979-7099-2aa5e66846e3 error="command timed out" testing.go:149: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.369+1400 [ERROR] server-e3f27379-9cff-8617-88f3-80539ce8e42f: failed to heartbeat to: peer=b9be970b-d5d2-a979-7099-2aa5e66846e3 error="command timed out" testLoggerAdapter verbose: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.439+1400 [ERROR] server-e3f27379-9cff-8617-88f3-80539ce8e42f: failed to heartbeat to: peer=b9be970b-d5d2-a979-7099-2aa5e66846e3 error="command timed out" testing.go:149: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.439+1400 [ERROR] server-e3f27379-9cff-8617-88f3-80539ce8e42f: failed to heartbeat to: peer=b9be970b-d5d2-a979-7099-2aa5e66846e3 error="command timed out" testLoggerAdapter verbose: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.518+1400 [ERROR] server-e3f27379-9cff-8617-88f3-80539ce8e42f: failed to heartbeat to: peer=b9be970b-d5d2-a979-7099-2aa5e66846e3 error="command timed out" testing.go:149: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.518+1400 [ERROR] server-e3f27379-9cff-8617-88f3-80539ce8e42f: failed to heartbeat to: peer=b9be970b-d5d2-a979-7099-2aa5e66846e3 error="command timed out" testLoggerAdapter verbose: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.618+1400 [ERROR] server-e3f27379-9cff-8617-88f3-80539ce8e42f: failed to heartbeat to: peer=b9be970b-d5d2-a979-7099-2aa5e66846e3 error="command timed out" testing.go:149: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.618+1400 [ERROR] server-e3f27379-9cff-8617-88f3-80539ce8e42f: failed to heartbeat to: peer=b9be970b-d5d2-a979-7099-2aa5e66846e3 error="command timed out" testLoggerAdapter verbose: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.745+1400 [ERROR] server-e3f27379-9cff-8617-88f3-80539ce8e42f: failed to install snapshot: id=2-23-1718116354218 error="command timed out" testing.go:149: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.745+1400 [ERROR] server-e3f27379-9cff-8617-88f3-80539ce8e42f: failed to install snapshot: id=2-23-1718116354218 error="command timed out" testLoggerAdapter verbose: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.745+1400 [ERROR] server-e3f27379-9cff-8617-88f3-80539ce8e42f: failed to send snapshot to: peer="{Voter server-b9be970b-d5d2-a979-7099-2aa5e66846e3 b9be970b-d5d2-a979-7099-2aa5e66846e3}" error="command timed out" testing.go:149: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.745+1400 [ERROR] server-e3f27379-9cff-8617-88f3-80539ce8e42f: failed to send snapshot to: peer="{Voter server-b9be970b-d5d2-a979-7099-2aa5e66846e3 b9be970b-d5d2-a979-7099-2aa5e66846e3}" error="command timed out" testLoggerAdapter verbose: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.757+1400 [ERROR] server-e3f27379-9cff-8617-88f3-80539ce8e42f: failed to heartbeat to: peer=b9be970b-d5d2-a979-7099-2aa5e66846e3 error="command timed out" testing.go:149: server-e3f27379-9cff-8617-88f3-80539ce8e42f: 2024-06-12T04:32:34.757+1400 [ERROR] server-e3f27379-9cff-8617-88f3-80539ce8e42f: failed to heartbeat to: peer=b9be970b-d5d2-a979-7099-2aa5e66846e3 error="command timed out" panic: timed out waiting for shutdown goroutine 1152 [running]: github.com/hashicorp/raft.(*cluster).Close.func1() /build/reproducible-path/golang-github-hashicorp-raft-1.1.2/obj-arm-linux-gnueabihf/src/github.com/hashicorp/raft/testing.go:260 +0x24 created by time.goFunc /usr/lib/go-1.19/src/time/sleep.go:176 +0x2c FAIL github.com/hashicorp/raft 16.212s ? github.com/hashicorp/raft/bench [no test files] FAIL dh_auto_test: error: cd obj-arm-linux-gnueabihf && go test -vet=off -v -p 4 github.com/hashicorp/raft github.com/hashicorp/raft/bench returned exit code 1 make: *** [debian/rules:9: build] Error 25 dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2 I: copying local configuration E: Failed autobuilding of package I: user script /srv/workspace/pbuilder/18637/tmp/hooks/C01_cleanup starting debug output: disk usage on i-capture-the-hostname at Tue Jun 11 14:32:39 UTC 2024 Filesystem Size Used Avail Use% Mounted on tmpfs 7.7G 0 7.7G 0% /dev/shm I: user script /srv/workspace/pbuilder/18637/tmp/hooks/C01_cleanup finished I: unmounting dev/ptmx filesystem I: unmounting dev/pts filesystem I: unmounting dev/shm filesystem I: unmounting proc filesystem I: unmounting sys filesystem I: cleaning the build env I: removing directory /srv/workspace/pbuilder/18637 and its subdirectories