Diff of the two buildlogs: -- --- b1/build.log 2025-01-14 01:12:58.818553684 +0000 +++ b2/build.log 2025-01-14 03:35:20.957743865 +0000 @@ -1,6 +1,6 @@ I: pbuilder: network access will be disabled during build -I: Current time: Sun Feb 15 18:57:41 -12 2026 -I: pbuilder-time-stamp: 1771225061 +I: Current time: Tue Jan 14 15:13:01 +14 2025 +I: pbuilder-time-stamp: 1736817181 I: Building the build Environment I: extracting base tarball [/var/cache/pbuilder/trixie-reproducible-base.tgz] I: copying local configuration @@ -44,52 +44,84 @@ dpkg-source: info: applying dask-matching-interpreter.patch I: Not using root during the build. I: Installing the build-deps -I: user script /srv/workspace/pbuilder/1505584/tmp/hooks/D02_print_environment starting +I: user script /srv/workspace/pbuilder/960870/tmp/hooks/D01_modify_environment starting +debug: Running on ionos1-amd64. +I: Changing host+domainname to test build reproducibility +I: Adding a custom variable just for the fun of it... +I: Changing /bin/sh to bash +'/bin/sh' -> '/bin/bash' +lrwxrwxrwx 1 root root 9 Jan 14 01:13 /bin/sh -> /bin/bash +I: Setting pbuilder2's login shell to /bin/bash +I: Setting pbuilder2's GECOS to second user,second room,second work-phone,second home-phone,second other +I: user script /srv/workspace/pbuilder/960870/tmp/hooks/D01_modify_environment finished +I: user script /srv/workspace/pbuilder/960870/tmp/hooks/D02_print_environment starting I: set - BUILDDIR='/build/reproducible-path' - BUILDUSERGECOS='first user,first room,first work-phone,first home-phone,first other' - BUILDUSERNAME='pbuilder1' - BUILD_ARCH='amd64' - DEBIAN_FRONTEND='noninteractive' - DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all parallel=42 ' - DISTRIBUTION='trixie' - HOME='/root' - HOST_ARCH='amd64' + BASH=/bin/sh + BASHOPTS=checkwinsize:cmdhist:complete_fullquote:extquote:force_fignore:globasciiranges:globskipdots:hostcomplete:interactive_comments:patsub_replacement:progcomp:promptvars:sourcepath + BASH_ALIASES=() + BASH_ARGC=() + BASH_ARGV=() + BASH_CMDS=() + BASH_LINENO=([0]="12" [1]="0") + BASH_LOADABLES_PATH=/usr/local/lib/bash:/usr/lib/bash:/opt/local/lib/bash:/usr/pkg/lib/bash:/opt/pkg/lib/bash:. + BASH_SOURCE=([0]="/tmp/hooks/D02_print_environment" [1]="/tmp/hooks/D02_print_environment") + BASH_VERSINFO=([0]="5" [1]="2" [2]="37" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu") + BASH_VERSION='5.2.37(1)-release' + BUILDDIR=/build/reproducible-path + BUILDUSERGECOS='second user,second room,second work-phone,second home-phone,second other' + BUILDUSERNAME=pbuilder2 + BUILD_ARCH=amd64 + DEBIAN_FRONTEND=noninteractive + DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all parallel=20 ' + DIRSTACK=() + DISTRIBUTION=trixie + EUID=0 + FUNCNAME=([0]="Echo" [1]="main") + GROUPS=() + HOME=/root + HOSTNAME=i-capture-the-hostname + HOSTTYPE=x86_64 + HOST_ARCH=amd64 IFS=' ' - INVOCATION_ID='f2d35ddf73734eac80df2d5788bd4560' - LANG='C' - LANGUAGE='en_US:en' - LC_ALL='C' - MAIL='/var/mail/root' - OPTIND='1' - PATH='/usr/sbin:/usr/bin:/sbin:/bin:/usr/games' - PBCURRENTCOMMANDLINEOPERATION='build' - PBUILDER_OPERATION='build' - PBUILDER_PKGDATADIR='/usr/share/pbuilder' - PBUILDER_PKGLIBDIR='/usr/lib/pbuilder' - PBUILDER_SYSCONFDIR='/etc' - PPID='1505584' - PS1='# ' - PS2='> ' + INVOCATION_ID=4e54523f5eb341e9aa492b3ab3d95925 + LANG=C + LANGUAGE=et_EE:et + LC_ALL=C + MACHTYPE=x86_64-pc-linux-gnu + MAIL=/var/mail/root + OPTERR=1 + OPTIND=1 + OSTYPE=linux-gnu + PATH=/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path + PBCURRENTCOMMANDLINEOPERATION=build + PBUILDER_OPERATION=build + PBUILDER_PKGDATADIR=/usr/share/pbuilder + PBUILDER_PKGLIBDIR=/usr/lib/pbuilder + PBUILDER_SYSCONFDIR=/etc + PIPESTATUS=([0]="0") + POSIXLY_CORRECT=y + PPID=960870 PS4='+ ' - PWD='/' - SHELL='/bin/bash' - SHLVL='2' - SUDO_COMMAND='/usr/bin/timeout -k 18.1h 18h /usr/bin/ionice -c 3 /usr/bin/nice /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.F59s879Z/pbuilderrc_67U1 --distribution trixie --hookdir /etc/pbuilder/first-build-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/trixie-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.F59s879Z/b1 --logfile b1/build.log dask.distributed_2024.12.1+ds-1.dsc' - SUDO_GID='111' - SUDO_UID='106' - SUDO_USER='jenkins' - TERM='unknown' - TZ='/usr/share/zoneinfo/Etc/GMT+12' - USER='root' - _='/usr/bin/systemd-run' - http_proxy='http://213.165.73.152:3128' + PWD=/ + SHELL=/bin/bash + SHELLOPTS=braceexpand:errexit:hashall:interactive-comments:posix + SHLVL=3 + SUDO_COMMAND='/usr/bin/timeout -k 24.1h 24h /usr/bin/ionice -c 3 /usr/bin/nice -n 11 /usr/bin/unshare --uts -- /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.F59s879Z/pbuilderrc_UUV2 --distribution trixie --hookdir /etc/pbuilder/rebuild-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/trixie-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.F59s879Z/b2 --logfile b2/build.log dask.distributed_2024.12.1+ds-1.dsc' + SUDO_GID=110 + SUDO_UID=105 + SUDO_USER=jenkins + TERM=unknown + TZ=/usr/share/zoneinfo/Etc/GMT-14 + UID=0 + USER=root + _='I: set' + http_proxy=http://46.16.76.132:3128 I: uname -a - Linux ionos15-amd64 6.11.10+bpo-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.11.10-1~bpo12+1 (2024-12-19) x86_64 GNU/Linux + Linux i-capture-the-hostname 6.1.0-29-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.123-1 (2025-01-02) x86_64 GNU/Linux I: ls -l /bin - lrwxrwxrwx 1 root root 7 Nov 22 2024 /bin -> usr/bin -I: user script /srv/workspace/pbuilder/1505584/tmp/hooks/D02_print_environment finished + lrwxrwxrwx 1 root root 7 Nov 22 14:40 /bin -> usr/bin +I: user script /srv/workspace/pbuilder/960870/tmp/hooks/D02_print_environment finished -> Attempting to satisfy build-dependencies -> Creating pbuilder-satisfydepends-dummy package Package: pbuilder-satisfydepends-dummy @@ -449,7 +481,7 @@ Get: 236 http://deb.debian.org/debian trixie/main amd64 python3-zict all 3.0.0-2 [29.7 kB] Get: 237 http://deb.debian.org/debian trixie/main amd64 tzdata-legacy all 2024b-4 [176 kB] Get: 238 http://deb.debian.org/debian trixie/main amd64 uglifyjs all 3.17.4-2 [12.1 kB] -Fetched 142 MB in 8s (18.8 MB/s) +Fetched 142 MB in 12s (11.7 MB/s) Preconfiguring packages ... Selecting previously unselected package fonts-lato. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 19853 files and directories currently installed.) @@ -1212,8 +1244,8 @@ Setting up tzdata (2024b-4) ... Current default time zone: 'Etc/UTC' -Local time is now: Mon Feb 16 06:58:27 UTC 2026. -Universal Time is now: Mon Feb 16 06:58:27 UTC 2026. +Local time is now: Tue Jan 14 01:18:52 UTC 2025. +Universal Time is now: Tue Jan 14 01:18:52 UTC 2025. Run 'dpkg-reconfigure tzdata' if you wish to change it. Setting up libpython3.13-minimal:amd64 (3.13.1-3) ... @@ -1443,7 +1475,11 @@ Building tag database... -> Finished parsing the build-deps I: Building the package -I: Running cd /build/reproducible-path/dask.distributed-2024.12.1+ds/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-genchanges -S > ../dask.distributed_2024.12.1+ds-1_source.changes +I: user script /srv/workspace/pbuilder/960870/tmp/hooks/A99_set_merged_usr starting +Not re-configuring usrmerge for trixie +I: user script /srv/workspace/pbuilder/960870/tmp/hooks/A99_set_merged_usr finished +hostname: Name or service not known +I: Running cd /build/reproducible-path/dask.distributed-2024.12.1+ds/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-genchanges -S > ../dask.distributed_2024.12.1+ds-1_source.changes dpkg-buildpackage: info: source package dask.distributed dpkg-buildpackage: info: source version 2024.12.1+ds-1 dpkg-buildpackage: info: source distribution unstable @@ -1499,198 +1535,198 @@ running build running build_py creating build/lib/distributed -copying distributed/security.py -> build/lib/distributed -copying distributed/pytest_resourceleaks.py -> build/lib/distributed -copying distributed/pubsub.py -> build/lib/distributed -copying distributed/config.py -> build/lib/distributed -copying distributed/_async_taskgroup.py -> build/lib/distributed -copying distributed/cfexecutor.py -> build/lib/distributed -copying distributed/_asyncio.py -> build/lib/distributed -copying distributed/nanny.py -> build/lib/distributed -copying distributed/client.py -> build/lib/distributed -copying distributed/recreate_tasks.py -> build/lib/distributed -copying distributed/system.py -> build/lib/distributed -copying distributed/profile.py -> build/lib/distributed -copying distributed/cluster_dump.py -> build/lib/distributed -copying distributed/spans.py -> build/lib/distributed -copying distributed/versions.py -> build/lib/distributed copying distributed/node.py -> build/lib/distributed -copying distributed/proctitle.py -> build/lib/distributed -copying distributed/utils.py -> build/lib/distributed -copying distributed/chaos.py -> build/lib/distributed -copying distributed/worker_client.py -> build/lib/distributed +copying distributed/nanny.py -> build/lib/distributed +copying distributed/metrics.py -> build/lib/distributed copying distributed/sizeof.py -> build/lib/distributed -copying distributed/_signals.py -> build/lib/distributed +copying distributed/semaphore.py -> build/lib/distributed +copying distributed/utils_comm.py -> build/lib/distributed copying distributed/_stories.py -> build/lib/distributed -copying distributed/publish.py -> build/lib/distributed -copying distributed/scheduler.py -> build/lib/distributed -copying distributed/itertools.py -> build/lib/distributed -copying distributed/core.py -> build/lib/distributed -copying distributed/batched.py -> build/lib/distributed +copying distributed/worker_state_machine.py -> build/lib/distributed +copying distributed/worker_client.py -> build/lib/distributed +copying distributed/spill.py -> build/lib/distributed copying distributed/event.py -> build/lib/distributed -copying distributed/process.py -> build/lib/distributed -copying distributed/worker.py -> build/lib/distributed copying distributed/lock.py -> build/lib/distributed -copying distributed/actor.py -> build/lib/distributed -copying distributed/preloading.py -> build/lib/distributed -copying distributed/_version.py -> build/lib/distributed -copying distributed/active_memory_manager.py -> build/lib/distributed -copying distributed/multi_lock.py -> build/lib/distributed -copying distributed/spill.py -> build/lib/distributed -copying distributed/bokeh.py -> build/lib/distributed copying distributed/threadpoolexecutor.py -> build/lib/distributed +copying distributed/pubsub.py -> build/lib/distributed +copying distributed/chaos.py -> build/lib/distributed +copying distributed/client.py -> build/lib/distributed +copying distributed/worker_memory.py -> build/lib/distributed +copying distributed/broker.py -> build/lib/distributed +copying distributed/counter.py -> build/lib/distributed copying distributed/stealing.py -> build/lib/distributed +copying distributed/worker.py -> build/lib/distributed +copying distributed/pytest_resourceleaks.py -> build/lib/distributed +copying distributed/utils_test.py -> build/lib/distributed +copying distributed/scheduler.py -> build/lib/distributed +copying distributed/multi_lock.py -> build/lib/distributed +copying distributed/core.py -> build/lib/distributed +copying distributed/bokeh.py -> build/lib/distributed +copying distributed/preloading.py -> build/lib/distributed +copying distributed/publish.py -> build/lib/distributed +copying distributed/exceptions.py -> build/lib/distributed copying distributed/__init__.py -> build/lib/distributed -copying distributed/queues.py -> build/lib/distributed -copying distributed/collections.py -> build/lib/distributed -copying distributed/gc.py -> build/lib/distributed -copying distributed/broker.py -> build/lib/distributed +copying distributed/profile.py -> build/lib/distributed +copying distributed/actor.py -> build/lib/distributed copying distributed/diskutils.py -> build/lib/distributed +copying distributed/gc.py -> build/lib/distributed +copying distributed/process.py -> build/lib/distributed +copying distributed/variable.py -> build/lib/distributed copying distributed/compatibility.py -> build/lib/distributed -copying distributed/worker_state_machine.py -> build/lib/distributed -copying distributed/metrics.py -> build/lib/distributed -copying distributed/exceptions.py -> build/lib/distributed -copying distributed/counter.py -> build/lib/distributed +copying distributed/config.py -> build/lib/distributed copying distributed/system_monitor.py -> build/lib/distributed -copying distributed/worker_memory.py -> build/lib/distributed -copying distributed/semaphore.py -> build/lib/distributed +copying distributed/batched.py -> build/lib/distributed +copying distributed/security.py -> build/lib/distributed +copying distributed/cfexecutor.py -> build/lib/distributed copying distributed/objects.py -> build/lib/distributed +copying distributed/itertools.py -> build/lib/distributed +copying distributed/proctitle.py -> build/lib/distributed +copying distributed/system.py -> build/lib/distributed +copying distributed/cluster_dump.py -> build/lib/distributed copying distributed/_concurrent_futures_thread.py -> build/lib/distributed -copying distributed/utils_test.py -> build/lib/distributed -copying distributed/variable.py -> build/lib/distributed -copying distributed/utils_comm.py -> build/lib/distributed +copying distributed/_signals.py -> build/lib/distributed +copying distributed/active_memory_manager.py -> build/lib/distributed +copying distributed/spans.py -> build/lib/distributed +copying distributed/utils.py -> build/lib/distributed +copying distributed/_asyncio.py -> build/lib/distributed +copying distributed/recreate_tasks.py -> build/lib/distributed +copying distributed/_version.py -> build/lib/distributed +copying distributed/_async_taskgroup.py -> build/lib/distributed +copying distributed/versions.py -> build/lib/distributed +copying distributed/collections.py -> build/lib/distributed +copying distributed/queues.py -> build/lib/distributed +creating build/lib/distributed/shuffle +copying distributed/shuffle/_pickle.py -> build/lib/distributed/shuffle +copying distributed/shuffle/_arrow.py -> build/lib/distributed/shuffle +copying distributed/shuffle/_merge.py -> build/lib/distributed/shuffle +copying distributed/shuffle/_disk.py -> build/lib/distributed/shuffle +copying distributed/shuffle/_shuffle.py -> build/lib/distributed/shuffle +copying distributed/shuffle/__init__.py -> build/lib/distributed/shuffle +copying distributed/shuffle/_memory.py -> build/lib/distributed/shuffle +copying distributed/shuffle/_core.py -> build/lib/distributed/shuffle +copying distributed/shuffle/_rechunk.py -> build/lib/distributed/shuffle +copying distributed/shuffle/_worker_plugin.py -> build/lib/distributed/shuffle +copying distributed/shuffle/_comms.py -> build/lib/distributed/shuffle +copying distributed/shuffle/_buffer.py -> build/lib/distributed/shuffle +copying distributed/shuffle/_exceptions.py -> build/lib/distributed/shuffle +copying distributed/shuffle/_limiter.py -> build/lib/distributed/shuffle +copying distributed/shuffle/_scheduler_plugin.py -> build/lib/distributed/shuffle +creating build/lib/distributed/http +copying distributed/http/proxy.py -> build/lib/distributed/http +copying distributed/http/prometheus.py -> build/lib/distributed/http +copying distributed/http/__init__.py -> build/lib/distributed/http +copying distributed/http/statics.py -> build/lib/distributed/http +copying distributed/http/health.py -> build/lib/distributed/http +copying distributed/http/utils.py -> build/lib/distributed/http +copying distributed/http/routing.py -> build/lib/distributed/http creating build/lib/distributed/deploy +copying distributed/deploy/local.py -> build/lib/distributed/deploy +copying distributed/deploy/adaptive.py -> build/lib/distributed/deploy +copying distributed/deploy/subprocess.py -> build/lib/distributed/deploy copying distributed/deploy/old_ssh.py -> build/lib/distributed/deploy +copying distributed/deploy/__init__.py -> build/lib/distributed/deploy copying distributed/deploy/adaptive_core.py -> build/lib/distributed/deploy +copying distributed/deploy/ssh.py -> build/lib/distributed/deploy copying distributed/deploy/utils.py -> build/lib/distributed/deploy -copying distributed/deploy/local.py -> build/lib/distributed/deploy copying distributed/deploy/cluster.py -> build/lib/distributed/deploy -copying distributed/deploy/__init__.py -> build/lib/distributed/deploy -copying distributed/deploy/adaptive.py -> build/lib/distributed/deploy copying distributed/deploy/spec.py -> build/lib/distributed/deploy -copying distributed/deploy/subprocess.py -> build/lib/distributed/deploy -copying distributed/deploy/ssh.py -> build/lib/distributed/deploy +creating build/lib/distributed/diagnostics +copying distributed/diagnostics/nvml.py -> build/lib/distributed/diagnostics +copying distributed/diagnostics/rmm.py -> build/lib/distributed/diagnostics +copying distributed/diagnostics/progressbar.py -> build/lib/distributed/diagnostics +copying distributed/diagnostics/task_stream.py -> build/lib/distributed/diagnostics +copying distributed/diagnostics/__init__.py -> build/lib/distributed/diagnostics +copying distributed/diagnostics/websocket.py -> build/lib/distributed/diagnostics +copying distributed/diagnostics/memray.py -> build/lib/distributed/diagnostics +copying distributed/diagnostics/plugin.py -> build/lib/distributed/diagnostics +copying distributed/diagnostics/graph_layout.py -> build/lib/distributed/diagnostics +copying distributed/diagnostics/cluster_dump.py -> build/lib/distributed/diagnostics +copying distributed/diagnostics/eventstream.py -> build/lib/distributed/diagnostics +copying distributed/diagnostics/progress.py -> build/lib/distributed/diagnostics +copying distributed/diagnostics/cudf.py -> build/lib/distributed/diagnostics +copying distributed/diagnostics/memory_sampler.py -> build/lib/distributed/diagnostics +copying distributed/diagnostics/progress_stream.py -> build/lib/distributed/diagnostics +creating build/lib/distributed/widgets +copying distributed/widgets/__init__.py -> build/lib/distributed/widgets creating build/lib/distributed/protocol -copying distributed/protocol/netcdf4.py -> build/lib/distributed/protocol -copying distributed/protocol/rmm.py -> build/lib/distributed/protocol -copying distributed/protocol/pickle.py -> build/lib/distributed/protocol copying distributed/protocol/sparse.py -> build/lib/distributed/protocol -copying distributed/protocol/numba.py -> build/lib/distributed/protocol -copying distributed/protocol/utils.py -> build/lib/distributed/protocol -copying distributed/protocol/serialize.py -> build/lib/distributed/protocol -copying distributed/protocol/compression.py -> build/lib/distributed/protocol +copying distributed/protocol/cuda.py -> build/lib/distributed/protocol copying distributed/protocol/h5py.py -> build/lib/distributed/protocol -copying distributed/protocol/torch.py -> build/lib/distributed/protocol -copying distributed/protocol/arrow.py -> build/lib/distributed/protocol +copying distributed/protocol/keras.py -> build/lib/distributed/protocol +copying distributed/protocol/serialize.py -> build/lib/distributed/protocol +copying distributed/protocol/rmm.py -> build/lib/distributed/protocol +copying distributed/protocol/utils_test.py -> build/lib/distributed/protocol copying distributed/protocol/core.py -> build/lib/distributed/protocol -copying distributed/protocol/cuda.py -> build/lib/distributed/protocol -copying distributed/protocol/scipy.py -> build/lib/distributed/protocol -copying distributed/protocol/cupy.py -> build/lib/distributed/protocol +copying distributed/protocol/torch.py -> build/lib/distributed/protocol +copying distributed/protocol/netcdf4.py -> build/lib/distributed/protocol copying distributed/protocol/__init__.py -> build/lib/distributed/protocol -copying distributed/protocol/keras.py -> build/lib/distributed/protocol +copying distributed/protocol/compression.py -> build/lib/distributed/protocol +copying distributed/protocol/scipy.py -> build/lib/distributed/protocol +copying distributed/protocol/arrow.py -> build/lib/distributed/protocol copying distributed/protocol/numpy.py -> build/lib/distributed/protocol -copying distributed/protocol/utils_test.py -> build/lib/distributed/protocol +copying distributed/protocol/utils.py -> build/lib/distributed/protocol +copying distributed/protocol/pickle.py -> build/lib/distributed/protocol +copying distributed/protocol/cupy.py -> build/lib/distributed/protocol +copying distributed/protocol/numba.py -> build/lib/distributed/protocol +creating build/lib/distributed/dashboard +copying distributed/dashboard/worker.py -> build/lib/distributed/dashboard +copying distributed/dashboard/scheduler.py -> build/lib/distributed/dashboard +copying distributed/dashboard/core.py -> build/lib/distributed/dashboard +copying distributed/dashboard/__init__.py -> build/lib/distributed/dashboard +copying distributed/dashboard/export_tool.py -> build/lib/distributed/dashboard +copying distributed/dashboard/utils.py -> build/lib/distributed/dashboard creating build/lib/distributed/comm -copying distributed/comm/inproc.py -> build/lib/distributed/comm -copying distributed/comm/utils.py -> build/lib/distributed/comm -copying distributed/comm/tcp.py -> build/lib/distributed/comm +copying distributed/comm/registry.py -> build/lib/distributed/comm copying distributed/comm/core.py -> build/lib/distributed/comm -copying distributed/comm/__init__.py -> build/lib/distributed/comm copying distributed/comm/addressing.py -> build/lib/distributed/comm -copying distributed/comm/registry.py -> build/lib/distributed/comm +copying distributed/comm/__init__.py -> build/lib/distributed/comm +copying distributed/comm/inproc.py -> build/lib/distributed/comm copying distributed/comm/ucx.py -> build/lib/distributed/comm +copying distributed/comm/tcp.py -> build/lib/distributed/comm +copying distributed/comm/utils.py -> build/lib/distributed/comm copying distributed/comm/ws.py -> build/lib/distributed/comm -creating build/lib/distributed/widgets -copying distributed/widgets/__init__.py -> build/lib/distributed/widgets creating build/lib/distributed/cli +copying distributed/cli/dask_worker.py -> build/lib/distributed/cli copying distributed/cli/dask_ssh.py -> build/lib/distributed/cli -copying distributed/cli/utils.py -> build/lib/distributed/cli -copying distributed/cli/dask_spec.py -> build/lib/distributed/cli copying distributed/cli/dask_scheduler.py -> build/lib/distributed/cli copying distributed/cli/__init__.py -> build/lib/distributed/cli -copying distributed/cli/dask_worker.py -> build/lib/distributed/cli -creating build/lib/distributed/diagnostics -copying distributed/diagnostics/rmm.py -> build/lib/distributed/diagnostics -copying distributed/diagnostics/cluster_dump.py -> build/lib/distributed/diagnostics -copying distributed/diagnostics/websocket.py -> build/lib/distributed/diagnostics -copying distributed/diagnostics/nvml.py -> build/lib/distributed/diagnostics -copying distributed/diagnostics/memory_sampler.py -> build/lib/distributed/diagnostics -copying distributed/diagnostics/progress_stream.py -> build/lib/distributed/diagnostics -copying distributed/diagnostics/memray.py -> build/lib/distributed/diagnostics -copying distributed/diagnostics/cudf.py -> build/lib/distributed/diagnostics -copying distributed/diagnostics/plugin.py -> build/lib/distributed/diagnostics -copying distributed/diagnostics/__init__.py -> build/lib/distributed/diagnostics -copying distributed/diagnostics/eventstream.py -> build/lib/distributed/diagnostics -copying distributed/diagnostics/task_stream.py -> build/lib/distributed/diagnostics -copying distributed/diagnostics/graph_layout.py -> build/lib/distributed/diagnostics -copying distributed/diagnostics/progressbar.py -> build/lib/distributed/diagnostics -copying distributed/diagnostics/progress.py -> build/lib/distributed/diagnostics -creating build/lib/distributed/shuffle -copying distributed/shuffle/_arrow.py -> build/lib/distributed/shuffle -copying distributed/shuffle/_exceptions.py -> build/lib/distributed/shuffle -copying distributed/shuffle/_rechunk.py -> build/lib/distributed/shuffle -copying distributed/shuffle/_limiter.py -> build/lib/distributed/shuffle -copying distributed/shuffle/_merge.py -> build/lib/distributed/shuffle -copying distributed/shuffle/_pickle.py -> build/lib/distributed/shuffle -copying distributed/shuffle/_worker_plugin.py -> build/lib/distributed/shuffle -copying distributed/shuffle/_shuffle.py -> build/lib/distributed/shuffle -copying distributed/shuffle/__init__.py -> build/lib/distributed/shuffle -copying distributed/shuffle/_buffer.py -> build/lib/distributed/shuffle -copying distributed/shuffle/_scheduler_plugin.py -> build/lib/distributed/shuffle -copying distributed/shuffle/_memory.py -> build/lib/distributed/shuffle -copying distributed/shuffle/_comms.py -> build/lib/distributed/shuffle -copying distributed/shuffle/_core.py -> build/lib/distributed/shuffle -copying distributed/shuffle/_disk.py -> build/lib/distributed/shuffle -creating build/lib/distributed/http -copying distributed/http/prometheus.py -> build/lib/distributed/http -copying distributed/http/utils.py -> build/lib/distributed/http -copying distributed/http/proxy.py -> build/lib/distributed/http -copying distributed/http/statics.py -> build/lib/distributed/http -copying distributed/http/__init__.py -> build/lib/distributed/http -copying distributed/http/routing.py -> build/lib/distributed/http -copying distributed/http/health.py -> build/lib/distributed/http -creating build/lib/distributed/dashboard -copying distributed/dashboard/utils.py -> build/lib/distributed/dashboard -copying distributed/dashboard/export_tool.py -> build/lib/distributed/dashboard -copying distributed/dashboard/scheduler.py -> build/lib/distributed/dashboard -copying distributed/dashboard/core.py -> build/lib/distributed/dashboard -copying distributed/dashboard/worker.py -> build/lib/distributed/dashboard -copying distributed/dashboard/__init__.py -> build/lib/distributed/dashboard -creating build/lib/distributed/widgets/templates -copying distributed/widgets/templates/__init__.py -> build/lib/distributed/widgets/templates -creating build/lib/distributed/http/static -copying distributed/http/static/__init__.py -> build/lib/distributed/http/static -creating build/lib/distributed/http/worker -copying distributed/http/worker/__init__.py -> build/lib/distributed/http/worker -creating build/lib/distributed/http/templates -copying distributed/http/templates/__init__.py -> build/lib/distributed/http/templates +copying distributed/cli/utils.py -> build/lib/distributed/cli +copying distributed/cli/dask_spec.py -> build/lib/distributed/cli creating build/lib/distributed/http/scheduler +copying distributed/http/scheduler/json.py -> build/lib/distributed/http/scheduler +copying distributed/http/scheduler/__init__.py -> build/lib/distributed/http/scheduler copying distributed/http/scheduler/api.py -> build/lib/distributed/http/scheduler copying distributed/http/scheduler/info.py -> build/lib/distributed/http/scheduler -copying distributed/http/scheduler/json.py -> build/lib/distributed/http/scheduler copying distributed/http/scheduler/missing_bokeh.py -> build/lib/distributed/http/scheduler -copying distributed/http/scheduler/__init__.py -> build/lib/distributed/http/scheduler +creating build/lib/distributed/http/static +copying distributed/http/static/__init__.py -> build/lib/distributed/http/static +creating build/lib/distributed/http/templates +copying distributed/http/templates/__init__.py -> build/lib/distributed/http/templates +creating build/lib/distributed/http/worker +copying distributed/http/worker/__init__.py -> build/lib/distributed/http/worker +creating build/lib/distributed/http/scheduler/prometheus +copying distributed/http/scheduler/prometheus/semaphore.py -> build/lib/distributed/http/scheduler/prometheus +copying distributed/http/scheduler/prometheus/stealing.py -> build/lib/distributed/http/scheduler/prometheus +copying distributed/http/scheduler/prometheus/core.py -> build/lib/distributed/http/scheduler/prometheus +copying distributed/http/scheduler/prometheus/__init__.py -> build/lib/distributed/http/scheduler/prometheus +creating build/lib/distributed/http/static/images +copying distributed/http/static/images/__init__.py -> build/lib/distributed/http/static/images creating build/lib/distributed/http/static/css copying distributed/http/static/css/__init__.py -> build/lib/distributed/http/static/css creating build/lib/distributed/http/static/js copying distributed/http/static/js/__init__.py -> build/lib/distributed/http/static/js -creating build/lib/distributed/http/static/images -copying distributed/http/static/images/__init__.py -> build/lib/distributed/http/static/images creating build/lib/distributed/http/worker/prometheus copying distributed/http/worker/prometheus/core.py -> build/lib/distributed/http/worker/prometheus copying distributed/http/worker/prometheus/__init__.py -> build/lib/distributed/http/worker/prometheus -creating build/lib/distributed/http/scheduler/prometheus -copying distributed/http/scheduler/prometheus/core.py -> build/lib/distributed/http/scheduler/prometheus -copying distributed/http/scheduler/prometheus/stealing.py -> build/lib/distributed/http/scheduler/prometheus -copying distributed/http/scheduler/prometheus/__init__.py -> build/lib/distributed/http/scheduler/prometheus -copying distributed/http/scheduler/prometheus/semaphore.py -> build/lib/distributed/http/scheduler/prometheus +creating build/lib/distributed/widgets/templates +copying distributed/widgets/templates/__init__.py -> build/lib/distributed/widgets/templates creating build/lib/distributed/dashboard/components -copying distributed/dashboard/components/rmm.py -> build/lib/distributed/dashboard/components copying distributed/dashboard/components/nvml.py -> build/lib/distributed/dashboard/components -copying distributed/dashboard/components/shared.py -> build/lib/distributed/dashboard/components -copying distributed/dashboard/components/scheduler.py -> build/lib/distributed/dashboard/components +copying distributed/dashboard/components/rmm.py -> build/lib/distributed/dashboard/components copying distributed/dashboard/components/worker.py -> build/lib/distributed/dashboard/components +copying distributed/dashboard/components/scheduler.py -> build/lib/distributed/dashboard/components copying distributed/dashboard/components/__init__.py -> build/lib/distributed/dashboard/components +copying distributed/dashboard/components/shared.py -> build/lib/distributed/dashboard/components creating build/lib/distributed/dashboard/templates copying distributed/dashboard/templates/__init__.py -> build/lib/distributed/dashboard/templates running egg_info @@ -2239,19 +2275,41 @@ copying distributed/tests/tls-key.pem -> build/lib/distributed/tests copying distributed/tests/tls-self-signed-cert.pem -> build/lib/distributed/tests copying distributed/tests/tls-self-signed-key.pem -> build/lib/distributed/tests -copying distributed/http/templates/base.html -> build/lib/distributed/http/templates +copying distributed/http/templates/call-stack.html -> build/lib/distributed/http/templates copying distributed/http/templates/status.html -> build/lib/distributed/http/templates -copying distributed/http/templates/worker-table.html -> build/lib/distributed/http/templates +copying distributed/http/templates/json-index.html -> build/lib/distributed/http/templates +copying distributed/http/templates/task.html -> build/lib/distributed/http/templates copying distributed/http/templates/simple.html -> build/lib/distributed/http/templates +copying distributed/http/templates/base.html -> build/lib/distributed/http/templates copying distributed/http/templates/main.html -> build/lib/distributed/http/templates +copying distributed/http/templates/worker-table.html -> build/lib/distributed/http/templates +copying distributed/http/templates/gpu.html -> build/lib/distributed/http/templates +copying distributed/http/templates/logs.html -> build/lib/distributed/http/templates copying distributed/http/templates/workers.html -> build/lib/distributed/http/templates -copying distributed/http/templates/task.html -> build/lib/distributed/http/templates copying distributed/http/templates/exceptions.html -> build/lib/distributed/http/templates -copying distributed/http/templates/gpu.html -> build/lib/distributed/http/templates -copying distributed/http/templates/json-index.html -> build/lib/distributed/http/templates copying distributed/http/templates/worker.html -> build/lib/distributed/http/templates -copying distributed/http/templates/logs.html -> build/lib/distributed/http/templates -copying distributed/http/templates/call-stack.html -> build/lib/distributed/http/templates +creating build/lib/distributed/shuffle/tests +copying distributed/shuffle/tests/__init__.py -> build/lib/distributed/shuffle/tests +copying distributed/shuffle/tests/test_buffer.py -> build/lib/distributed/shuffle/tests +copying distributed/shuffle/tests/test_comm_buffer.py -> build/lib/distributed/shuffle/tests +copying distributed/shuffle/tests/test_core.py -> build/lib/distributed/shuffle/tests +copying distributed/shuffle/tests/test_disk_buffer.py -> build/lib/distributed/shuffle/tests +copying distributed/shuffle/tests/test_graph.py -> build/lib/distributed/shuffle/tests +copying distributed/shuffle/tests/test_limiter.py -> build/lib/distributed/shuffle/tests +copying distributed/shuffle/tests/test_memory_buffer.py -> build/lib/distributed/shuffle/tests +copying distributed/shuffle/tests/test_merge.py -> build/lib/distributed/shuffle/tests +copying distributed/shuffle/tests/test_merge_column_and_index.py -> build/lib/distributed/shuffle/tests +copying distributed/shuffle/tests/test_metrics.py -> build/lib/distributed/shuffle/tests +copying distributed/shuffle/tests/test_pickle.py -> build/lib/distributed/shuffle/tests +copying distributed/shuffle/tests/test_read_write_lock.py -> build/lib/distributed/shuffle/tests +copying distributed/shuffle/tests/test_rechunk.py -> build/lib/distributed/shuffle/tests +copying distributed/shuffle/tests/test_shuffle.py -> build/lib/distributed/shuffle/tests +copying distributed/shuffle/tests/test_shuffle_plugins.py -> build/lib/distributed/shuffle/tests +copying distributed/shuffle/tests/utils.py -> build/lib/distributed/shuffle/tests +creating build/lib/distributed/http/tests +copying distributed/http/tests/__init__.py -> build/lib/distributed/http/tests +copying distributed/http/tests/test_core.py -> build/lib/distributed/http/tests +copying distributed/http/tests/test_routing.py -> build/lib/distributed/http/tests creating build/lib/distributed/deploy/tests copying distributed/deploy/tests/test_adaptive.py -> build/lib/distributed/deploy/tests copying distributed/deploy/tests/test_adaptive_core.py -> build/lib/distributed/deploy/tests @@ -2263,6 +2321,24 @@ copying distributed/deploy/tests/test_spec_cluster.py -> build/lib/distributed/deploy/tests copying distributed/deploy/tests/test_ssh.py -> build/lib/distributed/deploy/tests copying distributed/deploy/tests/test_subprocess.py -> build/lib/distributed/deploy/tests +creating build/lib/distributed/diagnostics/tests +copying distributed/diagnostics/tests/test_cluster_dump_plugin.py -> build/lib/distributed/diagnostics/tests +copying distributed/diagnostics/tests/test_cudf_diagnostics.py -> build/lib/distributed/diagnostics/tests +copying distributed/diagnostics/tests/test_eventstream.py -> build/lib/distributed/diagnostics/tests +copying distributed/diagnostics/tests/test_graph_layout.py -> build/lib/distributed/diagnostics/tests +copying distributed/diagnostics/tests/test_install_plugin.py -> build/lib/distributed/diagnostics/tests +copying distributed/diagnostics/tests/test_memory_sampler.py -> build/lib/distributed/diagnostics/tests +copying distributed/diagnostics/tests/test_memray.py -> build/lib/distributed/diagnostics/tests +copying distributed/diagnostics/tests/test_nanny_plugin.py -> build/lib/distributed/diagnostics/tests +copying distributed/diagnostics/tests/test_nvml.py -> build/lib/distributed/diagnostics/tests +copying distributed/diagnostics/tests/test_progress.py -> build/lib/distributed/diagnostics/tests +copying distributed/diagnostics/tests/test_progress_stream.py -> build/lib/distributed/diagnostics/tests +copying distributed/diagnostics/tests/test_progress_widgets.py -> build/lib/distributed/diagnostics/tests +copying distributed/diagnostics/tests/test_progressbar.py -> build/lib/distributed/diagnostics/tests +copying distributed/diagnostics/tests/test_rmm_diagnostics.py -> build/lib/distributed/diagnostics/tests +copying distributed/diagnostics/tests/test_scheduler_plugin.py -> build/lib/distributed/diagnostics/tests +copying distributed/diagnostics/tests/test_task_stream.py -> build/lib/distributed/diagnostics/tests +copying distributed/diagnostics/tests/test_worker_plugin.py -> build/lib/distributed/diagnostics/tests creating build/lib/distributed/protocol/tests copying distributed/protocol/tests/__init__.py -> build/lib/distributed/protocol/tests copying distributed/protocol/tests/test_arrow.py -> build/lib/distributed/protocol/tests @@ -2288,6 +2364,12 @@ copying distributed/protocol/tests/test_torch.py -> build/lib/distributed/protocol/tests copying distributed/protocol/tests/test_utils.py -> build/lib/distributed/protocol/tests copying distributed/protocol/tests/test_utils_test.py -> build/lib/distributed/protocol/tests +copying distributed/dashboard/theme.yaml -> build/lib/distributed/dashboard +creating build/lib/distributed/dashboard/tests +copying distributed/dashboard/tests/test_bokeh.py -> build/lib/distributed/dashboard/tests +copying distributed/dashboard/tests/test_components.py -> build/lib/distributed/dashboard/tests +copying distributed/dashboard/tests/test_scheduler_bokeh.py -> build/lib/distributed/dashboard/tests +copying distributed/dashboard/tests/test_worker_bokeh.py -> build/lib/distributed/dashboard/tests creating build/lib/distributed/comm/tests copying distributed/comm/tests/test_comms.py -> build/lib/distributed/comm/tests copying distributed/comm/tests/test_tcp.py -> build/lib/distributed/comm/tests @@ -2301,52 +2383,30 @@ copying distributed/cli/tests/test_dask_ssh.py -> build/lib/distributed/cli/tests copying distributed/cli/tests/test_dask_worker.py -> build/lib/distributed/cli/tests copying distributed/cli/tests/test_tls_cli.py -> build/lib/distributed/cli/tests -creating build/lib/distributed/diagnostics/tests -copying distributed/diagnostics/tests/test_cluster_dump_plugin.py -> build/lib/distributed/diagnostics/tests -copying distributed/diagnostics/tests/test_cudf_diagnostics.py -> build/lib/distributed/diagnostics/tests -copying distributed/diagnostics/tests/test_eventstream.py -> build/lib/distributed/diagnostics/tests -copying distributed/diagnostics/tests/test_graph_layout.py -> build/lib/distributed/diagnostics/tests -copying distributed/diagnostics/tests/test_install_plugin.py -> build/lib/distributed/diagnostics/tests -copying distributed/diagnostics/tests/test_memory_sampler.py -> build/lib/distributed/diagnostics/tests -copying distributed/diagnostics/tests/test_memray.py -> build/lib/distributed/diagnostics/tests -copying distributed/diagnostics/tests/test_nanny_plugin.py -> build/lib/distributed/diagnostics/tests -copying distributed/diagnostics/tests/test_nvml.py -> build/lib/distributed/diagnostics/tests -copying distributed/diagnostics/tests/test_progress.py -> build/lib/distributed/diagnostics/tests -copying distributed/diagnostics/tests/test_progress_stream.py -> build/lib/distributed/diagnostics/tests -copying distributed/diagnostics/tests/test_progress_widgets.py -> build/lib/distributed/diagnostics/tests -copying distributed/diagnostics/tests/test_progressbar.py -> build/lib/distributed/diagnostics/tests -copying distributed/diagnostics/tests/test_rmm_diagnostics.py -> build/lib/distributed/diagnostics/tests -copying distributed/diagnostics/tests/test_scheduler_plugin.py -> build/lib/distributed/diagnostics/tests -copying distributed/diagnostics/tests/test_task_stream.py -> build/lib/distributed/diagnostics/tests -copying distributed/diagnostics/tests/test_worker_plugin.py -> build/lib/distributed/diagnostics/tests -creating build/lib/distributed/shuffle/tests -copying distributed/shuffle/tests/__init__.py -> build/lib/distributed/shuffle/tests -copying distributed/shuffle/tests/test_buffer.py -> build/lib/distributed/shuffle/tests -copying distributed/shuffle/tests/test_comm_buffer.py -> build/lib/distributed/shuffle/tests -copying distributed/shuffle/tests/test_core.py -> build/lib/distributed/shuffle/tests -copying distributed/shuffle/tests/test_disk_buffer.py -> build/lib/distributed/shuffle/tests -copying distributed/shuffle/tests/test_graph.py -> build/lib/distributed/shuffle/tests -copying distributed/shuffle/tests/test_limiter.py -> build/lib/distributed/shuffle/tests -copying distributed/shuffle/tests/test_memory_buffer.py -> build/lib/distributed/shuffle/tests -copying distributed/shuffle/tests/test_merge.py -> build/lib/distributed/shuffle/tests -copying distributed/shuffle/tests/test_merge_column_and_index.py -> build/lib/distributed/shuffle/tests -copying distributed/shuffle/tests/test_metrics.py -> build/lib/distributed/shuffle/tests -copying distributed/shuffle/tests/test_pickle.py -> build/lib/distributed/shuffle/tests -copying distributed/shuffle/tests/test_read_write_lock.py -> build/lib/distributed/shuffle/tests -copying distributed/shuffle/tests/test_rechunk.py -> build/lib/distributed/shuffle/tests -copying distributed/shuffle/tests/test_shuffle.py -> build/lib/distributed/shuffle/tests -copying distributed/shuffle/tests/test_shuffle_plugins.py -> build/lib/distributed/shuffle/tests -copying distributed/shuffle/tests/utils.py -> build/lib/distributed/shuffle/tests -creating build/lib/distributed/http/tests -copying distributed/http/tests/__init__.py -> build/lib/distributed/http/tests -copying distributed/http/tests/test_core.py -> build/lib/distributed/http/tests -copying distributed/http/tests/test_routing.py -> build/lib/distributed/http/tests -copying distributed/dashboard/theme.yaml -> build/lib/distributed/dashboard -creating build/lib/distributed/dashboard/tests -copying distributed/dashboard/tests/test_bokeh.py -> build/lib/distributed/dashboard/tests -copying distributed/dashboard/tests/test_components.py -> build/lib/distributed/dashboard/tests -copying distributed/dashboard/tests/test_scheduler_bokeh.py -> build/lib/distributed/dashboard/tests -copying distributed/dashboard/tests/test_worker_bokeh.py -> build/lib/distributed/dashboard/tests +creating build/lib/distributed/http/scheduler/tests +copying distributed/http/scheduler/tests/test_missing_bokeh.py -> build/lib/distributed/http/scheduler/tests +copying distributed/http/scheduler/tests/test_scheduler_http.py -> build/lib/distributed/http/scheduler/tests +copying distributed/http/scheduler/tests/test_semaphore_http.py -> build/lib/distributed/http/scheduler/tests +copying distributed/http/scheduler/tests/test_stealing_http.py -> build/lib/distributed/http/scheduler/tests +copying distributed/http/static/individual-cluster-map.html -> build/lib/distributed/http/static +creating build/lib/distributed/http/worker/tests +copying distributed/http/worker/tests/test_worker_http.py -> build/lib/distributed/http/worker/tests +copying distributed/http/static/images/dask-logo.svg -> build/lib/distributed/http/static/images +copying distributed/http/static/images/fa-bars.svg -> build/lib/distributed/http/static/images +copying distributed/http/static/images/favicon.ico -> build/lib/distributed/http/static/images +copying distributed/http/static/images/jupyter.svg -> build/lib/distributed/http/static/images +copying distributed/http/static/images/numpy.png -> build/lib/distributed/http/static/images +copying distributed/http/static/images/pandas.png -> build/lib/distributed/http/static/images +copying distributed/http/static/images/python.png -> build/lib/distributed/http/static/images +copying distributed/http/static/css/base.css -> build/lib/distributed/http/static/css +copying distributed/http/static/css/gpu.css -> build/lib/distributed/http/static/css +copying distributed/http/static/css/individual-cluster-map.css -> build/lib/distributed/http/static/css +copying distributed/http/static/css/sortable.min.css -> build/lib/distributed/http/static/css +copying distributed/http/static/css/status.css -> build/lib/distributed/http/static/css +copying distributed/http/static/js/anime.min.js -> build/lib/distributed/http/static/js +copying distributed/http/static/js/individual-cluster-map.js -> build/lib/distributed/http/static/js +copying distributed/http/static/js/reconnecting-websocket.min.js -> build/lib/distributed/http/static/js +copying distributed/http/static/js/sortable.min.js -> build/lib/distributed/http/static/js copying distributed/widgets/templates/client.html.j2 -> build/lib/distributed/widgets/templates copying distributed/widgets/templates/cluster.html.j2 -> build/lib/distributed/widgets/templates copying distributed/widgets/templates/computation.html.j2 -> build/lib/distributed/widgets/templates @@ -2362,30 +2422,6 @@ copying distributed/widgets/templates/task_state.html.j2 -> build/lib/distributed/widgets/templates copying distributed/widgets/templates/who_has.html.j2 -> build/lib/distributed/widgets/templates copying distributed/widgets/templates/worker_state.html.j2 -> build/lib/distributed/widgets/templates -copying distributed/http/static/individual-cluster-map.html -> build/lib/distributed/http/static -creating build/lib/distributed/http/worker/tests -copying distributed/http/worker/tests/test_worker_http.py -> build/lib/distributed/http/worker/tests -creating build/lib/distributed/http/scheduler/tests -copying distributed/http/scheduler/tests/test_missing_bokeh.py -> build/lib/distributed/http/scheduler/tests -copying distributed/http/scheduler/tests/test_scheduler_http.py -> build/lib/distributed/http/scheduler/tests -copying distributed/http/scheduler/tests/test_semaphore_http.py -> build/lib/distributed/http/scheduler/tests -copying distributed/http/scheduler/tests/test_stealing_http.py -> build/lib/distributed/http/scheduler/tests -copying distributed/http/static/css/base.css -> build/lib/distributed/http/static/css -copying distributed/http/static/css/gpu.css -> build/lib/distributed/http/static/css -copying distributed/http/static/css/individual-cluster-map.css -> build/lib/distributed/http/static/css -copying distributed/http/static/css/sortable.min.css -> build/lib/distributed/http/static/css -copying distributed/http/static/css/status.css -> build/lib/distributed/http/static/css -copying distributed/http/static/js/anime.min.js -> build/lib/distributed/http/static/js -copying distributed/http/static/js/individual-cluster-map.js -> build/lib/distributed/http/static/js -copying distributed/http/static/js/reconnecting-websocket.min.js -> build/lib/distributed/http/static/js -copying distributed/http/static/js/sortable.min.js -> build/lib/distributed/http/static/js -copying distributed/http/static/images/dask-logo.svg -> build/lib/distributed/http/static/images -copying distributed/http/static/images/fa-bars.svg -> build/lib/distributed/http/static/images -copying distributed/http/static/images/favicon.ico -> build/lib/distributed/http/static/images -copying distributed/http/static/images/jupyter.svg -> build/lib/distributed/http/static/images -copying distributed/http/static/images/numpy.png -> build/lib/distributed/http/static/images -copying distributed/http/static/images/pandas.png -> build/lib/distributed/http/static/images -copying distributed/http/static/images/python.png -> build/lib/distributed/http/static/images copying distributed/dashboard/templates/performance_report.html -> build/lib/distributed/dashboard/templates UPDATING build/lib/distributed/_version.py set build/lib/distributed/_version.py to '2024.12.1' @@ -2394,433 +2430,433 @@ running install_lib creating build/bdist.linux-x86_64/wheel creating build/bdist.linux-x86_64/wheel/distributed -copying build/lib/distributed/security.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/pytest_resourceleaks.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/pubsub.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/config.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/_async_taskgroup.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/cfexecutor.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/distributed.yaml -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/_asyncio.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/nanny.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/client.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/recreate_tasks.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/system.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/profile.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/cluster_dump.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/spans.py -> build/bdist.linux-x86_64/wheel/./distributed -creating build/bdist.linux-x86_64/wheel/distributed/tests -copying build/lib/distributed/tests/test_active_memory_manager.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/tls-self-signed-key.pem -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_publish.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_worker_client.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/tls-cert.pem -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_reschedule.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/tls-ca-cert.pem -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_worker_metrics.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_spill.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_queues.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_parse_stdout.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_failed_workers.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_imports.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_semaphore.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_locks.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_stories.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_counter.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_threadpoolexecutor.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_asyncio.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_init.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_priorities.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_utils_comm.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_metrics.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_core.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_spec.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_security.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_variable.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_scheduler.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_client.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_computations.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/make_tls_certs.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_jupyter.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_tls_functional.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_diskutils.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_as_completed.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_asyncprocess.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_utils_test.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_system.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_chaos.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_sizeof.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_batched.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_nanny.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_compatibility.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_client_loop.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/tls-self-signed-cert.pem -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_spans.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/tls-key.pem -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_cluster_dump.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_worker_memory.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_versions.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/tls-key-cert.pem -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_collections.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_gc.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_profile.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_resources.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_utils.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_worker.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_event_logging.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_client_executor.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_steal.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_config.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_actor.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_worker_state_machine.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/tls-cert-chain.pem -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_preload.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_cancelled_state.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_events.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_pubsub.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_async_task_group.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_dask_collections.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/tls-ca-key.pem -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_multi_locks.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_deadline.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_stress.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_itertools.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_system_monitor.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/versions.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/node.py -> build/bdist.linux-x86_64/wheel/./distributed -creating build/bdist.linux-x86_64/wheel/distributed/deploy -copying build/lib/distributed/deploy/old_ssh.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy -copying build/lib/distributed/deploy/adaptive_core.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy -creating build/bdist.linux-x86_64/wheel/distributed/deploy/tests -copying build/lib/distributed/deploy/tests/test_adaptive_core.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests -copying build/lib/distributed/deploy/tests/test_spec_cluster.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests -copying build/lib/distributed/deploy/tests/test_subprocess.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests -copying build/lib/distributed/deploy/tests/test_deploy_utils.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests -copying build/lib/distributed/deploy/tests/test_local.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests -copying build/lib/distributed/deploy/tests/test_adaptive.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests -copying build/lib/distributed/deploy/tests/test_old_ssh.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests -copying build/lib/distributed/deploy/tests/test_cluster.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests -copying build/lib/distributed/deploy/tests/test_slow_adaptive.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests -copying build/lib/distributed/deploy/tests/test_ssh.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests -copying build/lib/distributed/deploy/utils.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy -copying build/lib/distributed/deploy/local.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy -copying build/lib/distributed/deploy/cluster.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy -copying build/lib/distributed/deploy/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy -copying build/lib/distributed/deploy/adaptive.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy -copying build/lib/distributed/deploy/spec.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy -copying build/lib/distributed/deploy/subprocess.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy -copying build/lib/distributed/deploy/ssh.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy -copying build/lib/distributed/proctitle.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/utils.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/chaos.py -> build/bdist.linux-x86_64/wheel/./distributed -creating build/bdist.linux-x86_64/wheel/distributed/protocol -copying build/lib/distributed/protocol/netcdf4.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol -copying build/lib/distributed/protocol/rmm.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol -copying build/lib/distributed/protocol/pickle.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol -copying build/lib/distributed/protocol/sparse.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol -copying build/lib/distributed/protocol/numba.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol -creating build/bdist.linux-x86_64/wheel/distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_compression.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_numpy.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_pickle.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_arrow.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_protocol_utils.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_keras.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_utils_test.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_scipy.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_cupy.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_sparse.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_rmm.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_utils.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_h5py.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_numba.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_serialize.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_protocol.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_pandas.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_highlevelgraph.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_torch.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_collection_cuda.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_to_pickle.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_collection.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_netcdf4.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/utils.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol -copying build/lib/distributed/protocol/serialize.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol -copying build/lib/distributed/protocol/compression.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol -copying build/lib/distributed/protocol/h5py.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol -copying build/lib/distributed/protocol/torch.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol -copying build/lib/distributed/protocol/arrow.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol -copying build/lib/distributed/protocol/core.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol -copying build/lib/distributed/protocol/cuda.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol -copying build/lib/distributed/protocol/scipy.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol -copying build/lib/distributed/protocol/cupy.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol -copying build/lib/distributed/protocol/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol -copying build/lib/distributed/protocol/keras.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol -copying build/lib/distributed/protocol/numpy.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol -copying build/lib/distributed/protocol/utils_test.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol -copying build/lib/distributed/worker_client.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/sizeof.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/_signals.py -> build/bdist.linux-x86_64/wheel/./distributed -creating build/bdist.linux-x86_64/wheel/distributed/comm -copying build/lib/distributed/comm/inproc.py -> build/bdist.linux-x86_64/wheel/./distributed/comm -creating build/bdist.linux-x86_64/wheel/distributed/comm/tests -copying build/lib/distributed/comm/tests/test_comms.py -> build/bdist.linux-x86_64/wheel/./distributed/comm/tests -copying build/lib/distributed/comm/tests/test_tcp.py -> build/bdist.linux-x86_64/wheel/./distributed/comm/tests -copying build/lib/distributed/comm/tests/test_ws.py -> build/bdist.linux-x86_64/wheel/./distributed/comm/tests -copying build/lib/distributed/comm/tests/test_ucx.py -> build/bdist.linux-x86_64/wheel/./distributed/comm/tests -copying build/lib/distributed/comm/tests/test_ucx_config.py -> build/bdist.linux-x86_64/wheel/./distributed/comm/tests -copying build/lib/distributed/comm/utils.py -> build/bdist.linux-x86_64/wheel/./distributed/comm -copying build/lib/distributed/comm/tcp.py -> build/bdist.linux-x86_64/wheel/./distributed/comm -copying build/lib/distributed/comm/core.py -> build/bdist.linux-x86_64/wheel/./distributed/comm -copying build/lib/distributed/comm/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/comm -copying build/lib/distributed/comm/addressing.py -> build/bdist.linux-x86_64/wheel/./distributed/comm -copying build/lib/distributed/comm/registry.py -> build/bdist.linux-x86_64/wheel/./distributed/comm -copying build/lib/distributed/comm/ucx.py -> build/bdist.linux-x86_64/wheel/./distributed/comm -copying build/lib/distributed/comm/ws.py -> build/bdist.linux-x86_64/wheel/./distributed/comm -creating build/bdist.linux-x86_64/wheel/distributed/widgets -copying build/lib/distributed/widgets/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/widgets -creating build/bdist.linux-x86_64/wheel/distributed/widgets/templates -copying build/lib/distributed/widgets/templates/security.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates -copying build/lib/distributed/widgets/templates/who_has.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates -copying build/lib/distributed/widgets/templates/local_cluster.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates -copying build/lib/distributed/widgets/templates/future.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates -copying build/lib/distributed/widgets/templates/scheduler_info.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates -copying build/lib/distributed/widgets/templates/computation.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates -copying build/lib/distributed/widgets/templates/process_interface.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates -copying build/lib/distributed/widgets/templates/log.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates -copying build/lib/distributed/widgets/templates/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates -copying build/lib/distributed/widgets/templates/logs.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates -copying build/lib/distributed/widgets/templates/cluster.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates -copying build/lib/distributed/widgets/templates/task_state.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates -copying build/lib/distributed/widgets/templates/has_what.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates -copying build/lib/distributed/widgets/templates/worker_state.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates -copying build/lib/distributed/widgets/templates/client.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates -copying build/lib/distributed/widgets/templates/scheduler.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates -copying build/lib/distributed/_stories.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/publish.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/scheduler.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/itertools.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/core.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/py.typed -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/batched.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/event.py -> build/bdist.linux-x86_64/wheel/./distributed -creating build/bdist.linux-x86_64/wheel/distributed/cli -creating build/bdist.linux-x86_64/wheel/distributed/cli/tests -copying build/lib/distributed/cli/tests/test_tls_cli.py -> build/bdist.linux-x86_64/wheel/./distributed/cli/tests -copying build/lib/distributed/cli/tests/test_dask_worker.py -> build/bdist.linux-x86_64/wheel/./distributed/cli/tests -copying build/lib/distributed/cli/tests/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/cli/tests -copying build/lib/distributed/cli/tests/test_dask_ssh.py -> build/bdist.linux-x86_64/wheel/./distributed/cli/tests -copying build/lib/distributed/cli/tests/test_dask_spec.py -> build/bdist.linux-x86_64/wheel/./distributed/cli/tests -copying build/lib/distributed/cli/tests/test_dask_scheduler.py -> build/bdist.linux-x86_64/wheel/./distributed/cli/tests -copying build/lib/distributed/cli/dask_ssh.py -> build/bdist.linux-x86_64/wheel/./distributed/cli -copying build/lib/distributed/cli/utils.py -> build/bdist.linux-x86_64/wheel/./distributed/cli -copying build/lib/distributed/cli/dask_spec.py -> build/bdist.linux-x86_64/wheel/./distributed/cli -copying build/lib/distributed/cli/dask_scheduler.py -> build/bdist.linux-x86_64/wheel/./distributed/cli -copying build/lib/distributed/cli/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/cli -copying build/lib/distributed/cli/dask_worker.py -> build/bdist.linux-x86_64/wheel/./distributed/cli -copying build/lib/distributed/process.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/worker.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/lock.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/actor.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/preloading.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/_version.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/active_memory_manager.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/multi_lock.py -> build/bdist.linux-x86_64/wheel/./distributed -creating build/bdist.linux-x86_64/wheel/distributed/diagnostics -copying build/lib/distributed/diagnostics/rmm.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics -copying build/lib/distributed/diagnostics/cluster_dump.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics -creating build/bdist.linux-x86_64/wheel/distributed/diagnostics/tests -copying build/lib/distributed/diagnostics/tests/test_cudf_diagnostics.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests -copying build/lib/distributed/diagnostics/tests/test_memray.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests -copying build/lib/distributed/diagnostics/tests/test_rmm_diagnostics.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests -copying build/lib/distributed/diagnostics/tests/test_nanny_plugin.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests -copying build/lib/distributed/diagnostics/tests/test_graph_layout.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests -copying build/lib/distributed/diagnostics/tests/test_cluster_dump_plugin.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests -copying build/lib/distributed/diagnostics/tests/test_scheduler_plugin.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests -copying build/lib/distributed/diagnostics/tests/test_install_plugin.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests -copying build/lib/distributed/diagnostics/tests/test_eventstream.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests -copying build/lib/distributed/diagnostics/tests/test_nvml.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests -copying build/lib/distributed/diagnostics/tests/test_progressbar.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests -copying build/lib/distributed/diagnostics/tests/test_memory_sampler.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests -copying build/lib/distributed/diagnostics/tests/test_worker_plugin.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests -copying build/lib/distributed/diagnostics/tests/test_progress_widgets.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests -copying build/lib/distributed/diagnostics/tests/test_progress_stream.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests -copying build/lib/distributed/diagnostics/tests/test_task_stream.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests -copying build/lib/distributed/diagnostics/tests/test_progress.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests -copying build/lib/distributed/diagnostics/websocket.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics -copying build/lib/distributed/diagnostics/nvml.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics -copying build/lib/distributed/diagnostics/memory_sampler.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics -copying build/lib/distributed/diagnostics/progress_stream.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics -copying build/lib/distributed/diagnostics/memray.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics -copying build/lib/distributed/diagnostics/cudf.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics -copying build/lib/distributed/diagnostics/plugin.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics -copying build/lib/distributed/diagnostics/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics -copying build/lib/distributed/diagnostics/eventstream.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics -copying build/lib/distributed/diagnostics/task_stream.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics -copying build/lib/distributed/diagnostics/graph_layout.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics -copying build/lib/distributed/diagnostics/progressbar.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics -copying build/lib/distributed/diagnostics/progress.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics -copying build/lib/distributed/spill.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/bokeh.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/threadpoolexecutor.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/stealing.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed creating build/bdist.linux-x86_64/wheel/distributed/shuffle +copying build/lib/distributed/shuffle/_pickle.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle copying build/lib/distributed/shuffle/_arrow.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle -copying build/lib/distributed/shuffle/_exceptions.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle +copying build/lib/distributed/shuffle/_merge.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle +copying build/lib/distributed/shuffle/_disk.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle +copying build/lib/distributed/shuffle/_shuffle.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle +copying build/lib/distributed/shuffle/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle +copying build/lib/distributed/shuffle/_memory.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle creating build/bdist.linux-x86_64/wheel/distributed/shuffle/tests -copying build/lib/distributed/shuffle/tests/test_graph.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests -copying build/lib/distributed/shuffle/tests/test_metrics.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests copying build/lib/distributed/shuffle/tests/test_core.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests copying build/lib/distributed/shuffle/tests/test_comm_buffer.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests -copying build/lib/distributed/shuffle/tests/utils.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests copying build/lib/distributed/shuffle/tests/test_limiter.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests copying build/lib/distributed/shuffle/tests/test_pickle.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests +copying build/lib/distributed/shuffle/tests/test_merge_column_and_index.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests +copying build/lib/distributed/shuffle/tests/test_disk_buffer.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests copying build/lib/distributed/shuffle/tests/test_buffer.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests +copying build/lib/distributed/shuffle/tests/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests +copying build/lib/distributed/shuffle/tests/test_read_write_lock.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests copying build/lib/distributed/shuffle/tests/test_shuffle.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests -copying build/lib/distributed/shuffle/tests/test_disk_buffer.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests copying build/lib/distributed/shuffle/tests/test_memory_buffer.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests +copying build/lib/distributed/shuffle/tests/test_metrics.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests +copying build/lib/distributed/shuffle/tests/utils.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests +copying build/lib/distributed/shuffle/tests/test_graph.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests copying build/lib/distributed/shuffle/tests/test_shuffle_plugins.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests -copying build/lib/distributed/shuffle/tests/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests -copying build/lib/distributed/shuffle/tests/test_merge_column_and_index.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests -copying build/lib/distributed/shuffle/tests/test_rechunk.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests -copying build/lib/distributed/shuffle/tests/test_read_write_lock.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests copying build/lib/distributed/shuffle/tests/test_merge.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests +copying build/lib/distributed/shuffle/tests/test_rechunk.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests +copying build/lib/distributed/shuffle/_core.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle copying build/lib/distributed/shuffle/_rechunk.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle -copying build/lib/distributed/shuffle/_limiter.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle -copying build/lib/distributed/shuffle/_merge.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle -copying build/lib/distributed/shuffle/_pickle.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle copying build/lib/distributed/shuffle/_worker_plugin.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle -copying build/lib/distributed/shuffle/_shuffle.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle -copying build/lib/distributed/shuffle/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle +copying build/lib/distributed/shuffle/_comms.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle copying build/lib/distributed/shuffle/_buffer.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle +copying build/lib/distributed/shuffle/_exceptions.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle +copying build/lib/distributed/shuffle/_limiter.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle copying build/lib/distributed/shuffle/_scheduler_plugin.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle -copying build/lib/distributed/shuffle/_memory.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle -copying build/lib/distributed/shuffle/_comms.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle -copying build/lib/distributed/shuffle/_core.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle -copying build/lib/distributed/shuffle/_disk.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle -copying build/lib/distributed/queues.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/collections.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/gc.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/broker.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/diskutils.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/compatibility.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/worker_state_machine.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/metrics.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/exceptions.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/node.py -> build/bdist.linux-x86_64/wheel/./distributed creating build/bdist.linux-x86_64/wheel/distributed/http +creating build/bdist.linux-x86_64/wheel/distributed/http/scheduler +copying build/lib/distributed/http/scheduler/json.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler +copying build/lib/distributed/http/scheduler/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler +creating build/bdist.linux-x86_64/wheel/distributed/http/scheduler/tests +copying build/lib/distributed/http/scheduler/tests/test_missing_bokeh.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler/tests +copying build/lib/distributed/http/scheduler/tests/test_stealing_http.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler/tests +copying build/lib/distributed/http/scheduler/tests/test_scheduler_http.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler/tests +copying build/lib/distributed/http/scheduler/tests/test_semaphore_http.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler/tests +copying build/lib/distributed/http/scheduler/api.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler +copying build/lib/distributed/http/scheduler/info.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler +copying build/lib/distributed/http/scheduler/missing_bokeh.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler +creating build/bdist.linux-x86_64/wheel/distributed/http/scheduler/prometheus +copying build/lib/distributed/http/scheduler/prometheus/semaphore.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler/prometheus +copying build/lib/distributed/http/scheduler/prometheus/stealing.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler/prometheus +copying build/lib/distributed/http/scheduler/prometheus/core.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler/prometheus +copying build/lib/distributed/http/scheduler/prometheus/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler/prometheus +copying build/lib/distributed/http/proxy.py -> build/bdist.linux-x86_64/wheel/./distributed/http +copying build/lib/distributed/http/prometheus.py -> build/bdist.linux-x86_64/wheel/./distributed/http +copying build/lib/distributed/http/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http +creating build/bdist.linux-x86_64/wheel/distributed/http/tests +copying build/lib/distributed/http/tests/test_core.py -> build/bdist.linux-x86_64/wheel/./distributed/http/tests +copying build/lib/distributed/http/tests/test_routing.py -> build/bdist.linux-x86_64/wheel/./distributed/http/tests +copying build/lib/distributed/http/tests/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http/tests +copying build/lib/distributed/http/statics.py -> build/bdist.linux-x86_64/wheel/./distributed/http +copying build/lib/distributed/http/health.py -> build/bdist.linux-x86_64/wheel/./distributed/http +copying build/lib/distributed/http/utils.py -> build/bdist.linux-x86_64/wheel/./distributed/http creating build/bdist.linux-x86_64/wheel/distributed/http/static +creating build/bdist.linux-x86_64/wheel/distributed/http/static/images +copying build/lib/distributed/http/static/images/jupyter.svg -> build/bdist.linux-x86_64/wheel/./distributed/http/static/images +copying build/lib/distributed/http/static/images/favicon.ico -> build/bdist.linux-x86_64/wheel/./distributed/http/static/images +copying build/lib/distributed/http/static/images/pandas.png -> build/bdist.linux-x86_64/wheel/./distributed/http/static/images +copying build/lib/distributed/http/static/images/numpy.png -> build/bdist.linux-x86_64/wheel/./distributed/http/static/images +copying build/lib/distributed/http/static/images/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http/static/images +copying build/lib/distributed/http/static/images/fa-bars.svg -> build/bdist.linux-x86_64/wheel/./distributed/http/static/images +copying build/lib/distributed/http/static/images/python.png -> build/bdist.linux-x86_64/wheel/./distributed/http/static/images +copying build/lib/distributed/http/static/images/dask-logo.svg -> build/bdist.linux-x86_64/wheel/./distributed/http/static/images creating build/bdist.linux-x86_64/wheel/distributed/http/static/css +copying build/lib/distributed/http/static/css/gpu.css -> build/bdist.linux-x86_64/wheel/./distributed/http/static/css copying build/lib/distributed/http/static/css/base.css -> build/bdist.linux-x86_64/wheel/./distributed/http/static/css copying build/lib/distributed/http/static/css/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http/static/css -copying build/lib/distributed/http/static/css/gpu.css -> build/bdist.linux-x86_64/wheel/./distributed/http/static/css copying build/lib/distributed/http/static/css/status.css -> build/bdist.linux-x86_64/wheel/./distributed/http/static/css -copying build/lib/distributed/http/static/css/individual-cluster-map.css -> build/bdist.linux-x86_64/wheel/./distributed/http/static/css copying build/lib/distributed/http/static/css/sortable.min.css -> build/bdist.linux-x86_64/wheel/./distributed/http/static/css +copying build/lib/distributed/http/static/css/individual-cluster-map.css -> build/bdist.linux-x86_64/wheel/./distributed/http/static/css +copying build/lib/distributed/http/static/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http/static +copying build/lib/distributed/http/static/individual-cluster-map.html -> build/bdist.linux-x86_64/wheel/./distributed/http/static creating build/bdist.linux-x86_64/wheel/distributed/http/static/js -copying build/lib/distributed/http/static/js/anime.min.js -> build/bdist.linux-x86_64/wheel/./distributed/http/static/js copying build/lib/distributed/http/static/js/individual-cluster-map.js -> build/bdist.linux-x86_64/wheel/./distributed/http/static/js -copying build/lib/distributed/http/static/js/reconnecting-websocket.min.js -> build/bdist.linux-x86_64/wheel/./distributed/http/static/js -copying build/lib/distributed/http/static/js/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http/static/js copying build/lib/distributed/http/static/js/sortable.min.js -> build/bdist.linux-x86_64/wheel/./distributed/http/static/js -copying build/lib/distributed/http/static/individual-cluster-map.html -> build/bdist.linux-x86_64/wheel/./distributed/http/static -copying build/lib/distributed/http/static/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http/static -creating build/bdist.linux-x86_64/wheel/distributed/http/static/images -copying build/lib/distributed/http/static/images/python.png -> build/bdist.linux-x86_64/wheel/./distributed/http/static/images -copying build/lib/distributed/http/static/images/dask-logo.svg -> build/bdist.linux-x86_64/wheel/./distributed/http/static/images -copying build/lib/distributed/http/static/images/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http/static/images -copying build/lib/distributed/http/static/images/fa-bars.svg -> build/bdist.linux-x86_64/wheel/./distributed/http/static/images -copying build/lib/distributed/http/static/images/jupyter.svg -> build/bdist.linux-x86_64/wheel/./distributed/http/static/images -copying build/lib/distributed/http/static/images/numpy.png -> build/bdist.linux-x86_64/wheel/./distributed/http/static/images -copying build/lib/distributed/http/static/images/pandas.png -> build/bdist.linux-x86_64/wheel/./distributed/http/static/images -copying build/lib/distributed/http/static/images/favicon.ico -> build/bdist.linux-x86_64/wheel/./distributed/http/static/images -creating build/bdist.linux-x86_64/wheel/distributed/http/tests -copying build/lib/distributed/http/tests/test_routing.py -> build/bdist.linux-x86_64/wheel/./distributed/http/tests -copying build/lib/distributed/http/tests/test_core.py -> build/bdist.linux-x86_64/wheel/./distributed/http/tests -copying build/lib/distributed/http/tests/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http/tests -copying build/lib/distributed/http/prometheus.py -> build/bdist.linux-x86_64/wheel/./distributed/http -copying build/lib/distributed/http/utils.py -> build/bdist.linux-x86_64/wheel/./distributed/http -copying build/lib/distributed/http/proxy.py -> build/bdist.linux-x86_64/wheel/./distributed/http -copying build/lib/distributed/http/statics.py -> build/bdist.linux-x86_64/wheel/./distributed/http -copying build/lib/distributed/http/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http -creating build/bdist.linux-x86_64/wheel/distributed/http/worker -creating build/bdist.linux-x86_64/wheel/distributed/http/worker/tests -copying build/lib/distributed/http/worker/tests/test_worker_http.py -> build/bdist.linux-x86_64/wheel/./distributed/http/worker/tests -copying build/lib/distributed/http/worker/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http/worker -creating build/bdist.linux-x86_64/wheel/distributed/http/worker/prometheus -copying build/lib/distributed/http/worker/prometheus/core.py -> build/bdist.linux-x86_64/wheel/./distributed/http/worker/prometheus -copying build/lib/distributed/http/worker/prometheus/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http/worker/prometheus +copying build/lib/distributed/http/static/js/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http/static/js +copying build/lib/distributed/http/static/js/reconnecting-websocket.min.js -> build/bdist.linux-x86_64/wheel/./distributed/http/static/js +copying build/lib/distributed/http/static/js/anime.min.js -> build/bdist.linux-x86_64/wheel/./distributed/http/static/js creating build/bdist.linux-x86_64/wheel/distributed/http/templates -copying build/lib/distributed/http/templates/base.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates +copying build/lib/distributed/http/templates/call-stack.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates copying build/lib/distributed/http/templates/status.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates -copying build/lib/distributed/http/templates/worker-table.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates +copying build/lib/distributed/http/templates/json-index.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates +copying build/lib/distributed/http/templates/task.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates copying build/lib/distributed/http/templates/simple.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates +copying build/lib/distributed/http/templates/base.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates copying build/lib/distributed/http/templates/main.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates +copying build/lib/distributed/http/templates/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http/templates +copying build/lib/distributed/http/templates/worker-table.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates +copying build/lib/distributed/http/templates/gpu.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates +copying build/lib/distributed/http/templates/logs.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates copying build/lib/distributed/http/templates/workers.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates -copying build/lib/distributed/http/templates/task.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates copying build/lib/distributed/http/templates/exceptions.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates -copying build/lib/distributed/http/templates/gpu.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates -copying build/lib/distributed/http/templates/json-index.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates -copying build/lib/distributed/http/templates/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http/templates copying build/lib/distributed/http/templates/worker.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates -copying build/lib/distributed/http/templates/logs.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates -copying build/lib/distributed/http/templates/call-stack.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates copying build/lib/distributed/http/routing.py -> build/bdist.linux-x86_64/wheel/./distributed/http -copying build/lib/distributed/http/health.py -> build/bdist.linux-x86_64/wheel/./distributed/http -creating build/bdist.linux-x86_64/wheel/distributed/http/scheduler -creating build/bdist.linux-x86_64/wheel/distributed/http/scheduler/tests -copying build/lib/distributed/http/scheduler/tests/test_semaphore_http.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler/tests -copying build/lib/distributed/http/scheduler/tests/test_missing_bokeh.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler/tests -copying build/lib/distributed/http/scheduler/tests/test_stealing_http.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler/tests -copying build/lib/distributed/http/scheduler/tests/test_scheduler_http.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler/tests -copying build/lib/distributed/http/scheduler/api.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler -copying build/lib/distributed/http/scheduler/info.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler -copying build/lib/distributed/http/scheduler/json.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler -copying build/lib/distributed/http/scheduler/missing_bokeh.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler -copying build/lib/distributed/http/scheduler/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler -creating build/bdist.linux-x86_64/wheel/distributed/http/scheduler/prometheus -copying build/lib/distributed/http/scheduler/prometheus/core.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler/prometheus -copying build/lib/distributed/http/scheduler/prometheus/stealing.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler/prometheus -copying build/lib/distributed/http/scheduler/prometheus/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler/prometheus -copying build/lib/distributed/http/scheduler/prometheus/semaphore.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler/prometheus +creating build/bdist.linux-x86_64/wheel/distributed/http/worker +copying build/lib/distributed/http/worker/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http/worker +creating build/bdist.linux-x86_64/wheel/distributed/http/worker/tests +copying build/lib/distributed/http/worker/tests/test_worker_http.py -> build/bdist.linux-x86_64/wheel/./distributed/http/worker/tests +creating build/bdist.linux-x86_64/wheel/distributed/http/worker/prometheus +copying build/lib/distributed/http/worker/prometheus/core.py -> build/bdist.linux-x86_64/wheel/./distributed/http/worker/prometheus +copying build/lib/distributed/http/worker/prometheus/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http/worker/prometheus +copying build/lib/distributed/nanny.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/metrics.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/sizeof.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/semaphore.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/utils_comm.py -> build/bdist.linux-x86_64/wheel/./distributed +creating build/bdist.linux-x86_64/wheel/distributed/deploy +copying build/lib/distributed/deploy/local.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy +copying build/lib/distributed/deploy/adaptive.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy +copying build/lib/distributed/deploy/subprocess.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy +copying build/lib/distributed/deploy/old_ssh.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy +copying build/lib/distributed/deploy/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy +copying build/lib/distributed/deploy/adaptive_core.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy +creating build/bdist.linux-x86_64/wheel/distributed/deploy/tests +copying build/lib/distributed/deploy/tests/test_slow_adaptive.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests +copying build/lib/distributed/deploy/tests/test_adaptive_core.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests +copying build/lib/distributed/deploy/tests/test_deploy_utils.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests +copying build/lib/distributed/deploy/tests/test_adaptive.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests +copying build/lib/distributed/deploy/tests/test_ssh.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests +copying build/lib/distributed/deploy/tests/test_spec_cluster.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests +copying build/lib/distributed/deploy/tests/test_local.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests +copying build/lib/distributed/deploy/tests/test_cluster.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests +copying build/lib/distributed/deploy/tests/test_subprocess.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests +copying build/lib/distributed/deploy/tests/test_old_ssh.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests +copying build/lib/distributed/deploy/ssh.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy +copying build/lib/distributed/deploy/utils.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy +copying build/lib/distributed/deploy/cluster.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy +copying build/lib/distributed/deploy/spec.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy +copying build/lib/distributed/_stories.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/worker_state_machine.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/worker_client.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/spill.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/event.py -> build/bdist.linux-x86_64/wheel/./distributed +creating build/bdist.linux-x86_64/wheel/distributed/diagnostics +copying build/lib/distributed/diagnostics/nvml.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics +copying build/lib/distributed/diagnostics/rmm.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics +copying build/lib/distributed/diagnostics/progressbar.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics +copying build/lib/distributed/diagnostics/task_stream.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics +copying build/lib/distributed/diagnostics/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics +copying build/lib/distributed/diagnostics/websocket.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics +copying build/lib/distributed/diagnostics/memray.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics +creating build/bdist.linux-x86_64/wheel/distributed/diagnostics/tests +copying build/lib/distributed/diagnostics/tests/test_install_plugin.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests +copying build/lib/distributed/diagnostics/tests/test_worker_plugin.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests +copying build/lib/distributed/diagnostics/tests/test_graph_layout.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests +copying build/lib/distributed/diagnostics/tests/test_task_stream.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests +copying build/lib/distributed/diagnostics/tests/test_nvml.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests +copying build/lib/distributed/diagnostics/tests/test_cluster_dump_plugin.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests +copying build/lib/distributed/diagnostics/tests/test_memory_sampler.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests +copying build/lib/distributed/diagnostics/tests/test_nanny_plugin.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests +copying build/lib/distributed/diagnostics/tests/test_progress.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests +copying build/lib/distributed/diagnostics/tests/test_memray.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests +copying build/lib/distributed/diagnostics/tests/test_scheduler_plugin.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests +copying build/lib/distributed/diagnostics/tests/test_progressbar.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests +copying build/lib/distributed/diagnostics/tests/test_progress_widgets.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests +copying build/lib/distributed/diagnostics/tests/test_progress_stream.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests +copying build/lib/distributed/diagnostics/tests/test_cudf_diagnostics.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests +copying build/lib/distributed/diagnostics/tests/test_eventstream.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests +copying build/lib/distributed/diagnostics/tests/test_rmm_diagnostics.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests +copying build/lib/distributed/diagnostics/plugin.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics +copying build/lib/distributed/diagnostics/graph_layout.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics +copying build/lib/distributed/diagnostics/cluster_dump.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics +copying build/lib/distributed/diagnostics/eventstream.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics +copying build/lib/distributed/diagnostics/progress.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics +copying build/lib/distributed/diagnostics/cudf.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics +copying build/lib/distributed/diagnostics/memory_sampler.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics +copying build/lib/distributed/diagnostics/progress_stream.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics +copying build/lib/distributed/lock.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/threadpoolexecutor.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/pubsub.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/chaos.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/client.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/worker_memory.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/broker.py -> build/bdist.linux-x86_64/wheel/./distributed copying build/lib/distributed/counter.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/stealing.py -> build/bdist.linux-x86_64/wheel/./distributed +creating build/bdist.linux-x86_64/wheel/distributed/widgets +copying build/lib/distributed/widgets/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/widgets +creating build/bdist.linux-x86_64/wheel/distributed/widgets/templates +copying build/lib/distributed/widgets/templates/log.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates +copying build/lib/distributed/widgets/templates/who_has.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates +copying build/lib/distributed/widgets/templates/local_cluster.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates +copying build/lib/distributed/widgets/templates/security.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates +copying build/lib/distributed/widgets/templates/client.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates +copying build/lib/distributed/widgets/templates/computation.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates +copying build/lib/distributed/widgets/templates/task_state.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates +copying build/lib/distributed/widgets/templates/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates +copying build/lib/distributed/widgets/templates/has_what.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates +copying build/lib/distributed/widgets/templates/future.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates +copying build/lib/distributed/widgets/templates/cluster.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates +copying build/lib/distributed/widgets/templates/logs.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates +copying build/lib/distributed/widgets/templates/scheduler.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates +copying build/lib/distributed/widgets/templates/worker_state.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates +copying build/lib/distributed/widgets/templates/scheduler_info.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates +copying build/lib/distributed/widgets/templates/process_interface.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates +copying build/lib/distributed/worker.py -> build/bdist.linux-x86_64/wheel/./distributed +creating build/bdist.linux-x86_64/wheel/distributed/protocol +copying build/lib/distributed/protocol/sparse.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol +copying build/lib/distributed/protocol/cuda.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol +copying build/lib/distributed/protocol/h5py.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol +copying build/lib/distributed/protocol/keras.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol +copying build/lib/distributed/protocol/serialize.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol +copying build/lib/distributed/protocol/rmm.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol +copying build/lib/distributed/protocol/utils_test.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol +copying build/lib/distributed/protocol/core.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol +copying build/lib/distributed/protocol/torch.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol +copying build/lib/distributed/protocol/netcdf4.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol +copying build/lib/distributed/protocol/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol +copying build/lib/distributed/protocol/compression.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol +creating build/bdist.linux-x86_64/wheel/distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_serialize.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_scipy.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_sparse.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_protocol.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_collection.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_arrow.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_pickle.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_numba.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_highlevelgraph.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_utils_test.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_cupy.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_numpy.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_pandas.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_netcdf4.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_compression.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_protocol_utils.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_rmm.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_to_pickle.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_keras.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_utils.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_torch.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_collection_cuda.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_h5py.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/scipy.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol +copying build/lib/distributed/protocol/arrow.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol +copying build/lib/distributed/protocol/numpy.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol +copying build/lib/distributed/protocol/utils.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol +copying build/lib/distributed/protocol/pickle.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol +copying build/lib/distributed/protocol/cupy.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol +copying build/lib/distributed/protocol/numba.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol +copying build/lib/distributed/pytest_resourceleaks.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/utils_test.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/scheduler.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/multi_lock.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/core.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/bokeh.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/preloading.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/publish.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/exceptions.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/profile.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/actor.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/diskutils.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/py.typed -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/gc.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/process.py -> build/bdist.linux-x86_64/wheel/./distributed +creating build/bdist.linux-x86_64/wheel/distributed/tests +copying build/lib/distributed/tests/test_events.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_asyncio.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_counter.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/tls-self-signed-key.pem -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_system_monitor.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_gc.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_steal.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_threadpoolexecutor.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_security.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_semaphore.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_jupyter.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_worker_metrics.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_stories.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_queues.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_failed_workers.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_worker_client.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_core.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_stress.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_nanny.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_cancelled_state.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_system.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_utils_test.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_dask_collections.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_imports.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_profile.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_collections.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/tls-self-signed-cert.pem -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_worker_state_machine.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_active_memory_manager.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_parse_stdout.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_batched.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/tls-cert-chain.pem -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_deadline.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_cluster_dump.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_diskutils.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_spans.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_pubsub.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_publish.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_sizeof.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_as_completed.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_scheduler.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/tls-ca-key.pem -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_spill.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/tls-key.pem -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_client_loop.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_variable.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/tls-ca-cert.pem -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_versions.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_computations.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_utils_comm.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_locks.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_chaos.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_worker_memory.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_spec.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_client_executor.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/tls-key-cert.pem -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_actor.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_reschedule.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_client.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_utils.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_init.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/tls-cert.pem -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_worker.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_itertools.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_tls_functional.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_asyncprocess.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_config.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_resources.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_metrics.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_preload.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_event_logging.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_compatibility.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_async_task_group.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/make_tls_certs.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_priorities.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_multi_locks.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/variable.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/compatibility.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/distributed.yaml -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/config.py -> build/bdist.linux-x86_64/wheel/./distributed copying build/lib/distributed/system_monitor.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/worker_memory.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/semaphore.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/batched.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/security.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/cfexecutor.py -> build/bdist.linux-x86_64/wheel/./distributed copying build/lib/distributed/objects.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/itertools.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/proctitle.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/system.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/cluster_dump.py -> build/bdist.linux-x86_64/wheel/./distributed copying build/lib/distributed/_concurrent_futures_thread.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/_signals.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/active_memory_manager.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/spans.py -> build/bdist.linux-x86_64/wheel/./distributed copying build/lib/distributed/distributed-schema.yaml -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/utils_test.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/variable.py -> build/bdist.linux-x86_64/wheel/./distributed creating build/bdist.linux-x86_64/wheel/distributed/dashboard -creating build/bdist.linux-x86_64/wheel/distributed/dashboard/tests -copying build/lib/distributed/dashboard/tests/test_scheduler_bokeh.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard/tests -copying build/lib/distributed/dashboard/tests/test_components.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard/tests -copying build/lib/distributed/dashboard/tests/test_worker_bokeh.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard/tests -copying build/lib/distributed/dashboard/tests/test_bokeh.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard/tests -copying build/lib/distributed/dashboard/theme.yaml -> build/bdist.linux-x86_64/wheel/./distributed/dashboard -copying build/lib/distributed/dashboard/utils.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard -copying build/lib/distributed/dashboard/export_tool.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard -copying build/lib/distributed/dashboard/scheduler.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard -copying build/lib/distributed/dashboard/core.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard creating build/bdist.linux-x86_64/wheel/distributed/dashboard/components -copying build/lib/distributed/dashboard/components/rmm.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard/components copying build/lib/distributed/dashboard/components/nvml.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard/components -copying build/lib/distributed/dashboard/components/shared.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard/components -copying build/lib/distributed/dashboard/components/scheduler.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard/components +copying build/lib/distributed/dashboard/components/rmm.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard/components copying build/lib/distributed/dashboard/components/worker.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard/components +copying build/lib/distributed/dashboard/components/scheduler.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard/components copying build/lib/distributed/dashboard/components/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard/components +copying build/lib/distributed/dashboard/components/shared.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard/components copying build/lib/distributed/dashboard/worker.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard +copying build/lib/distributed/dashboard/scheduler.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard +copying build/lib/distributed/dashboard/core.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard +copying build/lib/distributed/dashboard/theme.yaml -> build/bdist.linux-x86_64/wheel/./distributed/dashboard copying build/lib/distributed/dashboard/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard +creating build/bdist.linux-x86_64/wheel/distributed/dashboard/tests +copying build/lib/distributed/dashboard/tests/test_worker_bokeh.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard/tests +copying build/lib/distributed/dashboard/tests/test_scheduler_bokeh.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard/tests +copying build/lib/distributed/dashboard/tests/test_bokeh.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard/tests +copying build/lib/distributed/dashboard/tests/test_components.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard/tests +copying build/lib/distributed/dashboard/export_tool.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard +copying build/lib/distributed/dashboard/utils.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard creating build/bdist.linux-x86_64/wheel/distributed/dashboard/templates copying build/lib/distributed/dashboard/templates/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard/templates copying build/lib/distributed/dashboard/templates/performance_report.html -> build/bdist.linux-x86_64/wheel/./distributed/dashboard/templates -copying build/lib/distributed/utils_comm.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/utils.py -> build/bdist.linux-x86_64/wheel/./distributed +creating build/bdist.linux-x86_64/wheel/distributed/comm +copying build/lib/distributed/comm/registry.py -> build/bdist.linux-x86_64/wheel/./distributed/comm +copying build/lib/distributed/comm/core.py -> build/bdist.linux-x86_64/wheel/./distributed/comm +copying build/lib/distributed/comm/addressing.py -> build/bdist.linux-x86_64/wheel/./distributed/comm +copying build/lib/distributed/comm/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/comm +copying build/lib/distributed/comm/inproc.py -> build/bdist.linux-x86_64/wheel/./distributed/comm +creating build/bdist.linux-x86_64/wheel/distributed/comm/tests +copying build/lib/distributed/comm/tests/test_ucx_config.py -> build/bdist.linux-x86_64/wheel/./distributed/comm/tests +copying build/lib/distributed/comm/tests/test_tcp.py -> build/bdist.linux-x86_64/wheel/./distributed/comm/tests +copying build/lib/distributed/comm/tests/test_ws.py -> build/bdist.linux-x86_64/wheel/./distributed/comm/tests +copying build/lib/distributed/comm/tests/test_ucx.py -> build/bdist.linux-x86_64/wheel/./distributed/comm/tests +copying build/lib/distributed/comm/tests/test_comms.py -> build/bdist.linux-x86_64/wheel/./distributed/comm/tests +copying build/lib/distributed/comm/ucx.py -> build/bdist.linux-x86_64/wheel/./distributed/comm +copying build/lib/distributed/comm/tcp.py -> build/bdist.linux-x86_64/wheel/./distributed/comm +copying build/lib/distributed/comm/utils.py -> build/bdist.linux-x86_64/wheel/./distributed/comm +copying build/lib/distributed/comm/ws.py -> build/bdist.linux-x86_64/wheel/./distributed/comm +copying build/lib/distributed/_asyncio.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/recreate_tasks.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/_version.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/_async_taskgroup.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/versions.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/collections.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/queues.py -> build/bdist.linux-x86_64/wheel/./distributed +creating build/bdist.linux-x86_64/wheel/distributed/cli +copying build/lib/distributed/cli/dask_worker.py -> build/bdist.linux-x86_64/wheel/./distributed/cli +copying build/lib/distributed/cli/dask_ssh.py -> build/bdist.linux-x86_64/wheel/./distributed/cli +copying build/lib/distributed/cli/dask_scheduler.py -> build/bdist.linux-x86_64/wheel/./distributed/cli +copying build/lib/distributed/cli/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/cli +creating build/bdist.linux-x86_64/wheel/distributed/cli/tests +copying build/lib/distributed/cli/tests/test_dask_worker.py -> build/bdist.linux-x86_64/wheel/./distributed/cli/tests +copying build/lib/distributed/cli/tests/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/cli/tests +copying build/lib/distributed/cli/tests/test_dask_scheduler.py -> build/bdist.linux-x86_64/wheel/./distributed/cli/tests +copying build/lib/distributed/cli/tests/test_dask_ssh.py -> build/bdist.linux-x86_64/wheel/./distributed/cli/tests +copying build/lib/distributed/cli/tests/test_tls_cli.py -> build/bdist.linux-x86_64/wheel/./distributed/cli/tests +copying build/lib/distributed/cli/tests/test_dask_spec.py -> build/bdist.linux-x86_64/wheel/./distributed/cli/tests +copying build/lib/distributed/cli/utils.py -> build/bdist.linux-x86_64/wheel/./distributed/cli +copying build/lib/distributed/cli/dask_spec.py -> build/bdist.linux-x86_64/wheel/./distributed/cli running install_egg_info Copying distributed.egg-info to build/bdist.linux-x86_64/wheel/./distributed-2024.12.1.egg-info running install_scripts creating build/bdist.linux-x86_64/wheel/distributed-2024.12.1.dist-info/WHEEL -creating '/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.13_distributed/.tmp-w20s47ho/distributed-2024.12.1-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it +creating '/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.13_distributed/.tmp-a3f4rbfp/distributed-2024.12.1-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it adding 'distributed/__init__.py' adding 'distributed/_async_taskgroup.py' adding 'distributed/_asyncio.py' @@ -3705,433 +3741,433 @@ running install_lib creating build/bdist.linux-x86_64/wheel creating build/bdist.linux-x86_64/wheel/distributed -copying build/lib/distributed/security.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/pytest_resourceleaks.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/pubsub.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/config.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/_async_taskgroup.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/cfexecutor.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/distributed.yaml -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/_asyncio.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/nanny.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/client.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/recreate_tasks.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/system.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/profile.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/cluster_dump.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/spans.py -> build/bdist.linux-x86_64/wheel/./distributed -creating build/bdist.linux-x86_64/wheel/distributed/tests -copying build/lib/distributed/tests/test_active_memory_manager.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/tls-self-signed-key.pem -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_publish.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_worker_client.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/tls-cert.pem -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_reschedule.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/tls-ca-cert.pem -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_worker_metrics.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_spill.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_queues.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_parse_stdout.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_failed_workers.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_imports.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_semaphore.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_locks.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_stories.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_counter.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_threadpoolexecutor.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_asyncio.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_init.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_priorities.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_utils_comm.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_metrics.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_core.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_spec.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_security.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_variable.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_scheduler.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_client.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_computations.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/make_tls_certs.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_jupyter.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_tls_functional.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_diskutils.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_as_completed.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_asyncprocess.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_utils_test.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_system.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_chaos.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_sizeof.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_batched.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_nanny.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_compatibility.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_client_loop.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/tls-self-signed-cert.pem -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_spans.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/tls-key.pem -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_cluster_dump.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_worker_memory.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_versions.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/tls-key-cert.pem -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_collections.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_gc.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_profile.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_resources.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_utils.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_worker.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_event_logging.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_client_executor.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_steal.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_config.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_actor.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_worker_state_machine.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/tls-cert-chain.pem -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_preload.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_cancelled_state.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_events.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_pubsub.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_async_task_group.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_dask_collections.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/tls-ca-key.pem -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_multi_locks.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_deadline.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_stress.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_itertools.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/tests/test_system_monitor.py -> build/bdist.linux-x86_64/wheel/./distributed/tests -copying build/lib/distributed/versions.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/node.py -> build/bdist.linux-x86_64/wheel/./distributed -creating build/bdist.linux-x86_64/wheel/distributed/deploy -copying build/lib/distributed/deploy/old_ssh.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy -copying build/lib/distributed/deploy/adaptive_core.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy -creating build/bdist.linux-x86_64/wheel/distributed/deploy/tests -copying build/lib/distributed/deploy/tests/test_adaptive_core.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests -copying build/lib/distributed/deploy/tests/test_spec_cluster.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests -copying build/lib/distributed/deploy/tests/test_subprocess.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests -copying build/lib/distributed/deploy/tests/test_deploy_utils.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests -copying build/lib/distributed/deploy/tests/test_local.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests -copying build/lib/distributed/deploy/tests/test_adaptive.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests -copying build/lib/distributed/deploy/tests/test_old_ssh.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests -copying build/lib/distributed/deploy/tests/test_cluster.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests -copying build/lib/distributed/deploy/tests/test_slow_adaptive.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests -copying build/lib/distributed/deploy/tests/test_ssh.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests -copying build/lib/distributed/deploy/utils.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy -copying build/lib/distributed/deploy/local.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy -copying build/lib/distributed/deploy/cluster.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy -copying build/lib/distributed/deploy/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy -copying build/lib/distributed/deploy/adaptive.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy -copying build/lib/distributed/deploy/spec.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy -copying build/lib/distributed/deploy/subprocess.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy -copying build/lib/distributed/deploy/ssh.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy -copying build/lib/distributed/proctitle.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/utils.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/chaos.py -> build/bdist.linux-x86_64/wheel/./distributed -creating build/bdist.linux-x86_64/wheel/distributed/protocol -copying build/lib/distributed/protocol/netcdf4.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol -copying build/lib/distributed/protocol/rmm.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol -copying build/lib/distributed/protocol/pickle.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol -copying build/lib/distributed/protocol/sparse.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol -copying build/lib/distributed/protocol/numba.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol -creating build/bdist.linux-x86_64/wheel/distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_compression.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_numpy.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_pickle.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_arrow.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_protocol_utils.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_keras.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_utils_test.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_scipy.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_cupy.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_sparse.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_rmm.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_utils.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_h5py.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_numba.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_serialize.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_protocol.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_pandas.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_highlevelgraph.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_torch.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_collection_cuda.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_to_pickle.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_collection.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/tests/test_netcdf4.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests -copying build/lib/distributed/protocol/utils.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol -copying build/lib/distributed/protocol/serialize.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol -copying build/lib/distributed/protocol/compression.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol -copying build/lib/distributed/protocol/h5py.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol -copying build/lib/distributed/protocol/torch.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol -copying build/lib/distributed/protocol/arrow.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol -copying build/lib/distributed/protocol/core.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol -copying build/lib/distributed/protocol/cuda.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol -copying build/lib/distributed/protocol/scipy.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol -copying build/lib/distributed/protocol/cupy.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol -copying build/lib/distributed/protocol/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol -copying build/lib/distributed/protocol/keras.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol -copying build/lib/distributed/protocol/numpy.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol -copying build/lib/distributed/protocol/utils_test.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol -copying build/lib/distributed/worker_client.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/sizeof.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/_signals.py -> build/bdist.linux-x86_64/wheel/./distributed -creating build/bdist.linux-x86_64/wheel/distributed/comm -copying build/lib/distributed/comm/inproc.py -> build/bdist.linux-x86_64/wheel/./distributed/comm -creating build/bdist.linux-x86_64/wheel/distributed/comm/tests -copying build/lib/distributed/comm/tests/test_comms.py -> build/bdist.linux-x86_64/wheel/./distributed/comm/tests -copying build/lib/distributed/comm/tests/test_tcp.py -> build/bdist.linux-x86_64/wheel/./distributed/comm/tests -copying build/lib/distributed/comm/tests/test_ws.py -> build/bdist.linux-x86_64/wheel/./distributed/comm/tests -copying build/lib/distributed/comm/tests/test_ucx.py -> build/bdist.linux-x86_64/wheel/./distributed/comm/tests -copying build/lib/distributed/comm/tests/test_ucx_config.py -> build/bdist.linux-x86_64/wheel/./distributed/comm/tests -copying build/lib/distributed/comm/utils.py -> build/bdist.linux-x86_64/wheel/./distributed/comm -copying build/lib/distributed/comm/tcp.py -> build/bdist.linux-x86_64/wheel/./distributed/comm -copying build/lib/distributed/comm/core.py -> build/bdist.linux-x86_64/wheel/./distributed/comm -copying build/lib/distributed/comm/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/comm -copying build/lib/distributed/comm/addressing.py -> build/bdist.linux-x86_64/wheel/./distributed/comm -copying build/lib/distributed/comm/registry.py -> build/bdist.linux-x86_64/wheel/./distributed/comm -copying build/lib/distributed/comm/ucx.py -> build/bdist.linux-x86_64/wheel/./distributed/comm -copying build/lib/distributed/comm/ws.py -> build/bdist.linux-x86_64/wheel/./distributed/comm -creating build/bdist.linux-x86_64/wheel/distributed/widgets -copying build/lib/distributed/widgets/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/widgets -creating build/bdist.linux-x86_64/wheel/distributed/widgets/templates -copying build/lib/distributed/widgets/templates/security.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates -copying build/lib/distributed/widgets/templates/who_has.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates -copying build/lib/distributed/widgets/templates/local_cluster.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates -copying build/lib/distributed/widgets/templates/future.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates -copying build/lib/distributed/widgets/templates/scheduler_info.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates -copying build/lib/distributed/widgets/templates/computation.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates -copying build/lib/distributed/widgets/templates/process_interface.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates -copying build/lib/distributed/widgets/templates/log.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates -copying build/lib/distributed/widgets/templates/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates -copying build/lib/distributed/widgets/templates/logs.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates -copying build/lib/distributed/widgets/templates/cluster.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates -copying build/lib/distributed/widgets/templates/task_state.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates -copying build/lib/distributed/widgets/templates/has_what.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates -copying build/lib/distributed/widgets/templates/worker_state.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates -copying build/lib/distributed/widgets/templates/client.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates -copying build/lib/distributed/widgets/templates/scheduler.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates -copying build/lib/distributed/_stories.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/publish.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/scheduler.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/itertools.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/core.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/py.typed -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/batched.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/event.py -> build/bdist.linux-x86_64/wheel/./distributed -creating build/bdist.linux-x86_64/wheel/distributed/cli -creating build/bdist.linux-x86_64/wheel/distributed/cli/tests -copying build/lib/distributed/cli/tests/test_tls_cli.py -> build/bdist.linux-x86_64/wheel/./distributed/cli/tests -copying build/lib/distributed/cli/tests/test_dask_worker.py -> build/bdist.linux-x86_64/wheel/./distributed/cli/tests -copying build/lib/distributed/cli/tests/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/cli/tests -copying build/lib/distributed/cli/tests/test_dask_ssh.py -> build/bdist.linux-x86_64/wheel/./distributed/cli/tests -copying build/lib/distributed/cli/tests/test_dask_spec.py -> build/bdist.linux-x86_64/wheel/./distributed/cli/tests -copying build/lib/distributed/cli/tests/test_dask_scheduler.py -> build/bdist.linux-x86_64/wheel/./distributed/cli/tests -copying build/lib/distributed/cli/dask_ssh.py -> build/bdist.linux-x86_64/wheel/./distributed/cli -copying build/lib/distributed/cli/utils.py -> build/bdist.linux-x86_64/wheel/./distributed/cli -copying build/lib/distributed/cli/dask_spec.py -> build/bdist.linux-x86_64/wheel/./distributed/cli -copying build/lib/distributed/cli/dask_scheduler.py -> build/bdist.linux-x86_64/wheel/./distributed/cli -copying build/lib/distributed/cli/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/cli -copying build/lib/distributed/cli/dask_worker.py -> build/bdist.linux-x86_64/wheel/./distributed/cli -copying build/lib/distributed/process.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/worker.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/lock.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/actor.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/preloading.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/_version.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/active_memory_manager.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/multi_lock.py -> build/bdist.linux-x86_64/wheel/./distributed -creating build/bdist.linux-x86_64/wheel/distributed/diagnostics -copying build/lib/distributed/diagnostics/rmm.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics -copying build/lib/distributed/diagnostics/cluster_dump.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics -creating build/bdist.linux-x86_64/wheel/distributed/diagnostics/tests -copying build/lib/distributed/diagnostics/tests/test_cudf_diagnostics.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests -copying build/lib/distributed/diagnostics/tests/test_memray.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests -copying build/lib/distributed/diagnostics/tests/test_rmm_diagnostics.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests -copying build/lib/distributed/diagnostics/tests/test_nanny_plugin.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests -copying build/lib/distributed/diagnostics/tests/test_graph_layout.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests -copying build/lib/distributed/diagnostics/tests/test_cluster_dump_plugin.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests -copying build/lib/distributed/diagnostics/tests/test_scheduler_plugin.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests -copying build/lib/distributed/diagnostics/tests/test_install_plugin.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests -copying build/lib/distributed/diagnostics/tests/test_eventstream.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests -copying build/lib/distributed/diagnostics/tests/test_nvml.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests -copying build/lib/distributed/diagnostics/tests/test_progressbar.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests -copying build/lib/distributed/diagnostics/tests/test_memory_sampler.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests -copying build/lib/distributed/diagnostics/tests/test_worker_plugin.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests -copying build/lib/distributed/diagnostics/tests/test_progress_widgets.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests -copying build/lib/distributed/diagnostics/tests/test_progress_stream.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests -copying build/lib/distributed/diagnostics/tests/test_task_stream.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests -copying build/lib/distributed/diagnostics/tests/test_progress.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests -copying build/lib/distributed/diagnostics/websocket.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics -copying build/lib/distributed/diagnostics/nvml.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics -copying build/lib/distributed/diagnostics/memory_sampler.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics -copying build/lib/distributed/diagnostics/progress_stream.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics -copying build/lib/distributed/diagnostics/memray.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics -copying build/lib/distributed/diagnostics/cudf.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics -copying build/lib/distributed/diagnostics/plugin.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics -copying build/lib/distributed/diagnostics/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics -copying build/lib/distributed/diagnostics/eventstream.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics -copying build/lib/distributed/diagnostics/task_stream.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics -copying build/lib/distributed/diagnostics/graph_layout.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics -copying build/lib/distributed/diagnostics/progressbar.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics -copying build/lib/distributed/diagnostics/progress.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics -copying build/lib/distributed/spill.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/bokeh.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/threadpoolexecutor.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/stealing.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed creating build/bdist.linux-x86_64/wheel/distributed/shuffle +copying build/lib/distributed/shuffle/_pickle.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle copying build/lib/distributed/shuffle/_arrow.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle -copying build/lib/distributed/shuffle/_exceptions.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle +copying build/lib/distributed/shuffle/_merge.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle +copying build/lib/distributed/shuffle/_disk.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle +copying build/lib/distributed/shuffle/_shuffle.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle +copying build/lib/distributed/shuffle/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle +copying build/lib/distributed/shuffle/_memory.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle creating build/bdist.linux-x86_64/wheel/distributed/shuffle/tests -copying build/lib/distributed/shuffle/tests/test_graph.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests -copying build/lib/distributed/shuffle/tests/test_metrics.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests copying build/lib/distributed/shuffle/tests/test_core.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests copying build/lib/distributed/shuffle/tests/test_comm_buffer.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests -copying build/lib/distributed/shuffle/tests/utils.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests copying build/lib/distributed/shuffle/tests/test_limiter.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests copying build/lib/distributed/shuffle/tests/test_pickle.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests +copying build/lib/distributed/shuffle/tests/test_merge_column_and_index.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests +copying build/lib/distributed/shuffle/tests/test_disk_buffer.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests copying build/lib/distributed/shuffle/tests/test_buffer.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests +copying build/lib/distributed/shuffle/tests/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests +copying build/lib/distributed/shuffle/tests/test_read_write_lock.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests copying build/lib/distributed/shuffle/tests/test_shuffle.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests -copying build/lib/distributed/shuffle/tests/test_disk_buffer.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests copying build/lib/distributed/shuffle/tests/test_memory_buffer.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests +copying build/lib/distributed/shuffle/tests/test_metrics.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests +copying build/lib/distributed/shuffle/tests/utils.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests +copying build/lib/distributed/shuffle/tests/test_graph.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests copying build/lib/distributed/shuffle/tests/test_shuffle_plugins.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests -copying build/lib/distributed/shuffle/tests/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests -copying build/lib/distributed/shuffle/tests/test_merge_column_and_index.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests -copying build/lib/distributed/shuffle/tests/test_rechunk.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests -copying build/lib/distributed/shuffle/tests/test_read_write_lock.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests copying build/lib/distributed/shuffle/tests/test_merge.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests +copying build/lib/distributed/shuffle/tests/test_rechunk.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle/tests +copying build/lib/distributed/shuffle/_core.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle copying build/lib/distributed/shuffle/_rechunk.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle -copying build/lib/distributed/shuffle/_limiter.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle -copying build/lib/distributed/shuffle/_merge.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle -copying build/lib/distributed/shuffle/_pickle.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle copying build/lib/distributed/shuffle/_worker_plugin.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle -copying build/lib/distributed/shuffle/_shuffle.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle -copying build/lib/distributed/shuffle/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle +copying build/lib/distributed/shuffle/_comms.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle copying build/lib/distributed/shuffle/_buffer.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle +copying build/lib/distributed/shuffle/_exceptions.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle +copying build/lib/distributed/shuffle/_limiter.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle copying build/lib/distributed/shuffle/_scheduler_plugin.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle -copying build/lib/distributed/shuffle/_memory.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle -copying build/lib/distributed/shuffle/_comms.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle -copying build/lib/distributed/shuffle/_core.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle -copying build/lib/distributed/shuffle/_disk.py -> build/bdist.linux-x86_64/wheel/./distributed/shuffle -copying build/lib/distributed/queues.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/collections.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/gc.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/broker.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/diskutils.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/compatibility.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/worker_state_machine.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/metrics.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/exceptions.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/node.py -> build/bdist.linux-x86_64/wheel/./distributed creating build/bdist.linux-x86_64/wheel/distributed/http +creating build/bdist.linux-x86_64/wheel/distributed/http/scheduler +copying build/lib/distributed/http/scheduler/json.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler +copying build/lib/distributed/http/scheduler/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler +creating build/bdist.linux-x86_64/wheel/distributed/http/scheduler/tests +copying build/lib/distributed/http/scheduler/tests/test_missing_bokeh.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler/tests +copying build/lib/distributed/http/scheduler/tests/test_stealing_http.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler/tests +copying build/lib/distributed/http/scheduler/tests/test_scheduler_http.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler/tests +copying build/lib/distributed/http/scheduler/tests/test_semaphore_http.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler/tests +copying build/lib/distributed/http/scheduler/api.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler +copying build/lib/distributed/http/scheduler/info.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler +copying build/lib/distributed/http/scheduler/missing_bokeh.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler +creating build/bdist.linux-x86_64/wheel/distributed/http/scheduler/prometheus +copying build/lib/distributed/http/scheduler/prometheus/semaphore.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler/prometheus +copying build/lib/distributed/http/scheduler/prometheus/stealing.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler/prometheus +copying build/lib/distributed/http/scheduler/prometheus/core.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler/prometheus +copying build/lib/distributed/http/scheduler/prometheus/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler/prometheus +copying build/lib/distributed/http/proxy.py -> build/bdist.linux-x86_64/wheel/./distributed/http +copying build/lib/distributed/http/prometheus.py -> build/bdist.linux-x86_64/wheel/./distributed/http +copying build/lib/distributed/http/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http +creating build/bdist.linux-x86_64/wheel/distributed/http/tests +copying build/lib/distributed/http/tests/test_core.py -> build/bdist.linux-x86_64/wheel/./distributed/http/tests +copying build/lib/distributed/http/tests/test_routing.py -> build/bdist.linux-x86_64/wheel/./distributed/http/tests +copying build/lib/distributed/http/tests/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http/tests +copying build/lib/distributed/http/statics.py -> build/bdist.linux-x86_64/wheel/./distributed/http +copying build/lib/distributed/http/health.py -> build/bdist.linux-x86_64/wheel/./distributed/http +copying build/lib/distributed/http/utils.py -> build/bdist.linux-x86_64/wheel/./distributed/http creating build/bdist.linux-x86_64/wheel/distributed/http/static +creating build/bdist.linux-x86_64/wheel/distributed/http/static/images +copying build/lib/distributed/http/static/images/jupyter.svg -> build/bdist.linux-x86_64/wheel/./distributed/http/static/images +copying build/lib/distributed/http/static/images/favicon.ico -> build/bdist.linux-x86_64/wheel/./distributed/http/static/images +copying build/lib/distributed/http/static/images/pandas.png -> build/bdist.linux-x86_64/wheel/./distributed/http/static/images +copying build/lib/distributed/http/static/images/numpy.png -> build/bdist.linux-x86_64/wheel/./distributed/http/static/images +copying build/lib/distributed/http/static/images/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http/static/images +copying build/lib/distributed/http/static/images/fa-bars.svg -> build/bdist.linux-x86_64/wheel/./distributed/http/static/images +copying build/lib/distributed/http/static/images/python.png -> build/bdist.linux-x86_64/wheel/./distributed/http/static/images +copying build/lib/distributed/http/static/images/dask-logo.svg -> build/bdist.linux-x86_64/wheel/./distributed/http/static/images creating build/bdist.linux-x86_64/wheel/distributed/http/static/css +copying build/lib/distributed/http/static/css/gpu.css -> build/bdist.linux-x86_64/wheel/./distributed/http/static/css copying build/lib/distributed/http/static/css/base.css -> build/bdist.linux-x86_64/wheel/./distributed/http/static/css copying build/lib/distributed/http/static/css/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http/static/css -copying build/lib/distributed/http/static/css/gpu.css -> build/bdist.linux-x86_64/wheel/./distributed/http/static/css copying build/lib/distributed/http/static/css/status.css -> build/bdist.linux-x86_64/wheel/./distributed/http/static/css -copying build/lib/distributed/http/static/css/individual-cluster-map.css -> build/bdist.linux-x86_64/wheel/./distributed/http/static/css copying build/lib/distributed/http/static/css/sortable.min.css -> build/bdist.linux-x86_64/wheel/./distributed/http/static/css +copying build/lib/distributed/http/static/css/individual-cluster-map.css -> build/bdist.linux-x86_64/wheel/./distributed/http/static/css +copying build/lib/distributed/http/static/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http/static +copying build/lib/distributed/http/static/individual-cluster-map.html -> build/bdist.linux-x86_64/wheel/./distributed/http/static creating build/bdist.linux-x86_64/wheel/distributed/http/static/js -copying build/lib/distributed/http/static/js/anime.min.js -> build/bdist.linux-x86_64/wheel/./distributed/http/static/js copying build/lib/distributed/http/static/js/individual-cluster-map.js -> build/bdist.linux-x86_64/wheel/./distributed/http/static/js -copying build/lib/distributed/http/static/js/reconnecting-websocket.min.js -> build/bdist.linux-x86_64/wheel/./distributed/http/static/js -copying build/lib/distributed/http/static/js/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http/static/js copying build/lib/distributed/http/static/js/sortable.min.js -> build/bdist.linux-x86_64/wheel/./distributed/http/static/js -copying build/lib/distributed/http/static/individual-cluster-map.html -> build/bdist.linux-x86_64/wheel/./distributed/http/static -copying build/lib/distributed/http/static/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http/static -creating build/bdist.linux-x86_64/wheel/distributed/http/static/images -copying build/lib/distributed/http/static/images/python.png -> build/bdist.linux-x86_64/wheel/./distributed/http/static/images -copying build/lib/distributed/http/static/images/dask-logo.svg -> build/bdist.linux-x86_64/wheel/./distributed/http/static/images -copying build/lib/distributed/http/static/images/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http/static/images -copying build/lib/distributed/http/static/images/fa-bars.svg -> build/bdist.linux-x86_64/wheel/./distributed/http/static/images -copying build/lib/distributed/http/static/images/jupyter.svg -> build/bdist.linux-x86_64/wheel/./distributed/http/static/images -copying build/lib/distributed/http/static/images/numpy.png -> build/bdist.linux-x86_64/wheel/./distributed/http/static/images -copying build/lib/distributed/http/static/images/pandas.png -> build/bdist.linux-x86_64/wheel/./distributed/http/static/images -copying build/lib/distributed/http/static/images/favicon.ico -> build/bdist.linux-x86_64/wheel/./distributed/http/static/images -creating build/bdist.linux-x86_64/wheel/distributed/http/tests -copying build/lib/distributed/http/tests/test_routing.py -> build/bdist.linux-x86_64/wheel/./distributed/http/tests -copying build/lib/distributed/http/tests/test_core.py -> build/bdist.linux-x86_64/wheel/./distributed/http/tests -copying build/lib/distributed/http/tests/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http/tests -copying build/lib/distributed/http/prometheus.py -> build/bdist.linux-x86_64/wheel/./distributed/http -copying build/lib/distributed/http/utils.py -> build/bdist.linux-x86_64/wheel/./distributed/http -copying build/lib/distributed/http/proxy.py -> build/bdist.linux-x86_64/wheel/./distributed/http -copying build/lib/distributed/http/statics.py -> build/bdist.linux-x86_64/wheel/./distributed/http -copying build/lib/distributed/http/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http -creating build/bdist.linux-x86_64/wheel/distributed/http/worker -creating build/bdist.linux-x86_64/wheel/distributed/http/worker/tests -copying build/lib/distributed/http/worker/tests/test_worker_http.py -> build/bdist.linux-x86_64/wheel/./distributed/http/worker/tests -copying build/lib/distributed/http/worker/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http/worker -creating build/bdist.linux-x86_64/wheel/distributed/http/worker/prometheus -copying build/lib/distributed/http/worker/prometheus/core.py -> build/bdist.linux-x86_64/wheel/./distributed/http/worker/prometheus -copying build/lib/distributed/http/worker/prometheus/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http/worker/prometheus +copying build/lib/distributed/http/static/js/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http/static/js +copying build/lib/distributed/http/static/js/reconnecting-websocket.min.js -> build/bdist.linux-x86_64/wheel/./distributed/http/static/js +copying build/lib/distributed/http/static/js/anime.min.js -> build/bdist.linux-x86_64/wheel/./distributed/http/static/js creating build/bdist.linux-x86_64/wheel/distributed/http/templates -copying build/lib/distributed/http/templates/base.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates +copying build/lib/distributed/http/templates/call-stack.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates copying build/lib/distributed/http/templates/status.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates -copying build/lib/distributed/http/templates/worker-table.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates +copying build/lib/distributed/http/templates/json-index.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates +copying build/lib/distributed/http/templates/task.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates copying build/lib/distributed/http/templates/simple.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates +copying build/lib/distributed/http/templates/base.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates copying build/lib/distributed/http/templates/main.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates +copying build/lib/distributed/http/templates/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http/templates +copying build/lib/distributed/http/templates/worker-table.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates +copying build/lib/distributed/http/templates/gpu.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates +copying build/lib/distributed/http/templates/logs.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates copying build/lib/distributed/http/templates/workers.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates -copying build/lib/distributed/http/templates/task.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates copying build/lib/distributed/http/templates/exceptions.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates -copying build/lib/distributed/http/templates/gpu.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates -copying build/lib/distributed/http/templates/json-index.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates -copying build/lib/distributed/http/templates/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http/templates copying build/lib/distributed/http/templates/worker.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates -copying build/lib/distributed/http/templates/logs.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates -copying build/lib/distributed/http/templates/call-stack.html -> build/bdist.linux-x86_64/wheel/./distributed/http/templates copying build/lib/distributed/http/routing.py -> build/bdist.linux-x86_64/wheel/./distributed/http -copying build/lib/distributed/http/health.py -> build/bdist.linux-x86_64/wheel/./distributed/http -creating build/bdist.linux-x86_64/wheel/distributed/http/scheduler -creating build/bdist.linux-x86_64/wheel/distributed/http/scheduler/tests -copying build/lib/distributed/http/scheduler/tests/test_semaphore_http.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler/tests -copying build/lib/distributed/http/scheduler/tests/test_missing_bokeh.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler/tests -copying build/lib/distributed/http/scheduler/tests/test_stealing_http.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler/tests -copying build/lib/distributed/http/scheduler/tests/test_scheduler_http.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler/tests -copying build/lib/distributed/http/scheduler/api.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler -copying build/lib/distributed/http/scheduler/info.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler -copying build/lib/distributed/http/scheduler/json.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler -copying build/lib/distributed/http/scheduler/missing_bokeh.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler -copying build/lib/distributed/http/scheduler/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler -creating build/bdist.linux-x86_64/wheel/distributed/http/scheduler/prometheus -copying build/lib/distributed/http/scheduler/prometheus/core.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler/prometheus -copying build/lib/distributed/http/scheduler/prometheus/stealing.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler/prometheus -copying build/lib/distributed/http/scheduler/prometheus/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler/prometheus -copying build/lib/distributed/http/scheduler/prometheus/semaphore.py -> build/bdist.linux-x86_64/wheel/./distributed/http/scheduler/prometheus +creating build/bdist.linux-x86_64/wheel/distributed/http/worker +copying build/lib/distributed/http/worker/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http/worker +creating build/bdist.linux-x86_64/wheel/distributed/http/worker/tests +copying build/lib/distributed/http/worker/tests/test_worker_http.py -> build/bdist.linux-x86_64/wheel/./distributed/http/worker/tests +creating build/bdist.linux-x86_64/wheel/distributed/http/worker/prometheus +copying build/lib/distributed/http/worker/prometheus/core.py -> build/bdist.linux-x86_64/wheel/./distributed/http/worker/prometheus +copying build/lib/distributed/http/worker/prometheus/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/http/worker/prometheus +copying build/lib/distributed/nanny.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/metrics.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/sizeof.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/semaphore.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/utils_comm.py -> build/bdist.linux-x86_64/wheel/./distributed +creating build/bdist.linux-x86_64/wheel/distributed/deploy +copying build/lib/distributed/deploy/local.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy +copying build/lib/distributed/deploy/adaptive.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy +copying build/lib/distributed/deploy/subprocess.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy +copying build/lib/distributed/deploy/old_ssh.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy +copying build/lib/distributed/deploy/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy +copying build/lib/distributed/deploy/adaptive_core.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy +creating build/bdist.linux-x86_64/wheel/distributed/deploy/tests +copying build/lib/distributed/deploy/tests/test_slow_adaptive.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests +copying build/lib/distributed/deploy/tests/test_adaptive_core.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests +copying build/lib/distributed/deploy/tests/test_deploy_utils.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests +copying build/lib/distributed/deploy/tests/test_adaptive.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests +copying build/lib/distributed/deploy/tests/test_ssh.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests +copying build/lib/distributed/deploy/tests/test_spec_cluster.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests +copying build/lib/distributed/deploy/tests/test_local.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests +copying build/lib/distributed/deploy/tests/test_cluster.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests +copying build/lib/distributed/deploy/tests/test_subprocess.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests +copying build/lib/distributed/deploy/tests/test_old_ssh.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy/tests +copying build/lib/distributed/deploy/ssh.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy +copying build/lib/distributed/deploy/utils.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy +copying build/lib/distributed/deploy/cluster.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy +copying build/lib/distributed/deploy/spec.py -> build/bdist.linux-x86_64/wheel/./distributed/deploy +copying build/lib/distributed/_stories.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/worker_state_machine.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/worker_client.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/spill.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/event.py -> build/bdist.linux-x86_64/wheel/./distributed +creating build/bdist.linux-x86_64/wheel/distributed/diagnostics +copying build/lib/distributed/diagnostics/nvml.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics +copying build/lib/distributed/diagnostics/rmm.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics +copying build/lib/distributed/diagnostics/progressbar.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics +copying build/lib/distributed/diagnostics/task_stream.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics +copying build/lib/distributed/diagnostics/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics +copying build/lib/distributed/diagnostics/websocket.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics +copying build/lib/distributed/diagnostics/memray.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics +creating build/bdist.linux-x86_64/wheel/distributed/diagnostics/tests +copying build/lib/distributed/diagnostics/tests/test_install_plugin.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests +copying build/lib/distributed/diagnostics/tests/test_worker_plugin.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests +copying build/lib/distributed/diagnostics/tests/test_graph_layout.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests +copying build/lib/distributed/diagnostics/tests/test_task_stream.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests +copying build/lib/distributed/diagnostics/tests/test_nvml.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests +copying build/lib/distributed/diagnostics/tests/test_cluster_dump_plugin.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests +copying build/lib/distributed/diagnostics/tests/test_memory_sampler.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests +copying build/lib/distributed/diagnostics/tests/test_nanny_plugin.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests +copying build/lib/distributed/diagnostics/tests/test_progress.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests +copying build/lib/distributed/diagnostics/tests/test_memray.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests +copying build/lib/distributed/diagnostics/tests/test_scheduler_plugin.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests +copying build/lib/distributed/diagnostics/tests/test_progressbar.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests +copying build/lib/distributed/diagnostics/tests/test_progress_widgets.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests +copying build/lib/distributed/diagnostics/tests/test_progress_stream.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests +copying build/lib/distributed/diagnostics/tests/test_cudf_diagnostics.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests +copying build/lib/distributed/diagnostics/tests/test_eventstream.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests +copying build/lib/distributed/diagnostics/tests/test_rmm_diagnostics.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics/tests +copying build/lib/distributed/diagnostics/plugin.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics +copying build/lib/distributed/diagnostics/graph_layout.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics +copying build/lib/distributed/diagnostics/cluster_dump.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics +copying build/lib/distributed/diagnostics/eventstream.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics +copying build/lib/distributed/diagnostics/progress.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics +copying build/lib/distributed/diagnostics/cudf.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics +copying build/lib/distributed/diagnostics/memory_sampler.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics +copying build/lib/distributed/diagnostics/progress_stream.py -> build/bdist.linux-x86_64/wheel/./distributed/diagnostics +copying build/lib/distributed/lock.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/threadpoolexecutor.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/pubsub.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/chaos.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/client.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/worker_memory.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/broker.py -> build/bdist.linux-x86_64/wheel/./distributed copying build/lib/distributed/counter.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/stealing.py -> build/bdist.linux-x86_64/wheel/./distributed +creating build/bdist.linux-x86_64/wheel/distributed/widgets +copying build/lib/distributed/widgets/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/widgets +creating build/bdist.linux-x86_64/wheel/distributed/widgets/templates +copying build/lib/distributed/widgets/templates/log.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates +copying build/lib/distributed/widgets/templates/who_has.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates +copying build/lib/distributed/widgets/templates/local_cluster.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates +copying build/lib/distributed/widgets/templates/security.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates +copying build/lib/distributed/widgets/templates/client.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates +copying build/lib/distributed/widgets/templates/computation.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates +copying build/lib/distributed/widgets/templates/task_state.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates +copying build/lib/distributed/widgets/templates/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates +copying build/lib/distributed/widgets/templates/has_what.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates +copying build/lib/distributed/widgets/templates/future.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates +copying build/lib/distributed/widgets/templates/cluster.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates +copying build/lib/distributed/widgets/templates/logs.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates +copying build/lib/distributed/widgets/templates/scheduler.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates +copying build/lib/distributed/widgets/templates/worker_state.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates +copying build/lib/distributed/widgets/templates/scheduler_info.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates +copying build/lib/distributed/widgets/templates/process_interface.html.j2 -> build/bdist.linux-x86_64/wheel/./distributed/widgets/templates +copying build/lib/distributed/worker.py -> build/bdist.linux-x86_64/wheel/./distributed +creating build/bdist.linux-x86_64/wheel/distributed/protocol +copying build/lib/distributed/protocol/sparse.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol +copying build/lib/distributed/protocol/cuda.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol +copying build/lib/distributed/protocol/h5py.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol +copying build/lib/distributed/protocol/keras.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol +copying build/lib/distributed/protocol/serialize.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol +copying build/lib/distributed/protocol/rmm.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol +copying build/lib/distributed/protocol/utils_test.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol +copying build/lib/distributed/protocol/core.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol +copying build/lib/distributed/protocol/torch.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol +copying build/lib/distributed/protocol/netcdf4.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol +copying build/lib/distributed/protocol/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol +copying build/lib/distributed/protocol/compression.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol +creating build/bdist.linux-x86_64/wheel/distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_serialize.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_scipy.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_sparse.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_protocol.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_collection.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_arrow.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_pickle.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_numba.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_highlevelgraph.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_utils_test.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_cupy.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_numpy.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_pandas.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_netcdf4.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_compression.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_protocol_utils.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_rmm.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_to_pickle.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_keras.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_utils.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_torch.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_collection_cuda.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/tests/test_h5py.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol/tests +copying build/lib/distributed/protocol/scipy.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol +copying build/lib/distributed/protocol/arrow.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol +copying build/lib/distributed/protocol/numpy.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol +copying build/lib/distributed/protocol/utils.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol +copying build/lib/distributed/protocol/pickle.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol +copying build/lib/distributed/protocol/cupy.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol +copying build/lib/distributed/protocol/numba.py -> build/bdist.linux-x86_64/wheel/./distributed/protocol +copying build/lib/distributed/pytest_resourceleaks.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/utils_test.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/scheduler.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/multi_lock.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/core.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/bokeh.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/preloading.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/publish.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/exceptions.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/profile.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/actor.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/diskutils.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/py.typed -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/gc.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/process.py -> build/bdist.linux-x86_64/wheel/./distributed +creating build/bdist.linux-x86_64/wheel/distributed/tests +copying build/lib/distributed/tests/test_events.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_asyncio.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_counter.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/tls-self-signed-key.pem -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_system_monitor.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_gc.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_steal.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_threadpoolexecutor.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_security.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_semaphore.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_jupyter.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_worker_metrics.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_stories.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_queues.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_failed_workers.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_worker_client.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_core.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_stress.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_nanny.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_cancelled_state.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_system.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_utils_test.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_dask_collections.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_imports.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_profile.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_collections.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/tls-self-signed-cert.pem -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_worker_state_machine.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_active_memory_manager.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_parse_stdout.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_batched.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/tls-cert-chain.pem -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_deadline.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_cluster_dump.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_diskutils.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_spans.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_pubsub.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_publish.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_sizeof.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_as_completed.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_scheduler.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/tls-ca-key.pem -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_spill.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/tls-key.pem -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_client_loop.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_variable.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/tls-ca-cert.pem -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_versions.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_computations.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_utils_comm.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_locks.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_chaos.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_worker_memory.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_spec.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_client_executor.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/tls-key-cert.pem -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_actor.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_reschedule.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_client.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_utils.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_init.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/tls-cert.pem -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_worker.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_itertools.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_tls_functional.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_asyncprocess.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_config.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_resources.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_metrics.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_preload.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_event_logging.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_compatibility.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_async_task_group.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/make_tls_certs.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_priorities.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/tests/test_multi_locks.py -> build/bdist.linux-x86_64/wheel/./distributed/tests +copying build/lib/distributed/variable.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/compatibility.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/distributed.yaml -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/config.py -> build/bdist.linux-x86_64/wheel/./distributed copying build/lib/distributed/system_monitor.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/worker_memory.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/semaphore.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/batched.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/security.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/cfexecutor.py -> build/bdist.linux-x86_64/wheel/./distributed copying build/lib/distributed/objects.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/itertools.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/proctitle.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/system.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/cluster_dump.py -> build/bdist.linux-x86_64/wheel/./distributed copying build/lib/distributed/_concurrent_futures_thread.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/_signals.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/active_memory_manager.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/spans.py -> build/bdist.linux-x86_64/wheel/./distributed copying build/lib/distributed/distributed-schema.yaml -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/utils_test.py -> build/bdist.linux-x86_64/wheel/./distributed -copying build/lib/distributed/variable.py -> build/bdist.linux-x86_64/wheel/./distributed creating build/bdist.linux-x86_64/wheel/distributed/dashboard -creating build/bdist.linux-x86_64/wheel/distributed/dashboard/tests -copying build/lib/distributed/dashboard/tests/test_scheduler_bokeh.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard/tests -copying build/lib/distributed/dashboard/tests/test_components.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard/tests -copying build/lib/distributed/dashboard/tests/test_worker_bokeh.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard/tests -copying build/lib/distributed/dashboard/tests/test_bokeh.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard/tests -copying build/lib/distributed/dashboard/theme.yaml -> build/bdist.linux-x86_64/wheel/./distributed/dashboard -copying build/lib/distributed/dashboard/utils.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard -copying build/lib/distributed/dashboard/export_tool.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard -copying build/lib/distributed/dashboard/scheduler.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard -copying build/lib/distributed/dashboard/core.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard creating build/bdist.linux-x86_64/wheel/distributed/dashboard/components -copying build/lib/distributed/dashboard/components/rmm.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard/components copying build/lib/distributed/dashboard/components/nvml.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard/components -copying build/lib/distributed/dashboard/components/shared.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard/components -copying build/lib/distributed/dashboard/components/scheduler.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard/components +copying build/lib/distributed/dashboard/components/rmm.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard/components copying build/lib/distributed/dashboard/components/worker.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard/components +copying build/lib/distributed/dashboard/components/scheduler.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard/components copying build/lib/distributed/dashboard/components/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard/components +copying build/lib/distributed/dashboard/components/shared.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard/components copying build/lib/distributed/dashboard/worker.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard +copying build/lib/distributed/dashboard/scheduler.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard +copying build/lib/distributed/dashboard/core.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard +copying build/lib/distributed/dashboard/theme.yaml -> build/bdist.linux-x86_64/wheel/./distributed/dashboard copying build/lib/distributed/dashboard/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard +creating build/bdist.linux-x86_64/wheel/distributed/dashboard/tests +copying build/lib/distributed/dashboard/tests/test_worker_bokeh.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard/tests +copying build/lib/distributed/dashboard/tests/test_scheduler_bokeh.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard/tests +copying build/lib/distributed/dashboard/tests/test_bokeh.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard/tests +copying build/lib/distributed/dashboard/tests/test_components.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard/tests +copying build/lib/distributed/dashboard/export_tool.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard +copying build/lib/distributed/dashboard/utils.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard creating build/bdist.linux-x86_64/wheel/distributed/dashboard/templates copying build/lib/distributed/dashboard/templates/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/dashboard/templates copying build/lib/distributed/dashboard/templates/performance_report.html -> build/bdist.linux-x86_64/wheel/./distributed/dashboard/templates -copying build/lib/distributed/utils_comm.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/utils.py -> build/bdist.linux-x86_64/wheel/./distributed +creating build/bdist.linux-x86_64/wheel/distributed/comm +copying build/lib/distributed/comm/registry.py -> build/bdist.linux-x86_64/wheel/./distributed/comm +copying build/lib/distributed/comm/core.py -> build/bdist.linux-x86_64/wheel/./distributed/comm +copying build/lib/distributed/comm/addressing.py -> build/bdist.linux-x86_64/wheel/./distributed/comm +copying build/lib/distributed/comm/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/comm +copying build/lib/distributed/comm/inproc.py -> build/bdist.linux-x86_64/wheel/./distributed/comm +creating build/bdist.linux-x86_64/wheel/distributed/comm/tests +copying build/lib/distributed/comm/tests/test_ucx_config.py -> build/bdist.linux-x86_64/wheel/./distributed/comm/tests +copying build/lib/distributed/comm/tests/test_tcp.py -> build/bdist.linux-x86_64/wheel/./distributed/comm/tests +copying build/lib/distributed/comm/tests/test_ws.py -> build/bdist.linux-x86_64/wheel/./distributed/comm/tests +copying build/lib/distributed/comm/tests/test_ucx.py -> build/bdist.linux-x86_64/wheel/./distributed/comm/tests +copying build/lib/distributed/comm/tests/test_comms.py -> build/bdist.linux-x86_64/wheel/./distributed/comm/tests +copying build/lib/distributed/comm/ucx.py -> build/bdist.linux-x86_64/wheel/./distributed/comm +copying build/lib/distributed/comm/tcp.py -> build/bdist.linux-x86_64/wheel/./distributed/comm +copying build/lib/distributed/comm/utils.py -> build/bdist.linux-x86_64/wheel/./distributed/comm +copying build/lib/distributed/comm/ws.py -> build/bdist.linux-x86_64/wheel/./distributed/comm +copying build/lib/distributed/_asyncio.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/recreate_tasks.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/_version.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/_async_taskgroup.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/versions.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/collections.py -> build/bdist.linux-x86_64/wheel/./distributed +copying build/lib/distributed/queues.py -> build/bdist.linux-x86_64/wheel/./distributed +creating build/bdist.linux-x86_64/wheel/distributed/cli +copying build/lib/distributed/cli/dask_worker.py -> build/bdist.linux-x86_64/wheel/./distributed/cli +copying build/lib/distributed/cli/dask_ssh.py -> build/bdist.linux-x86_64/wheel/./distributed/cli +copying build/lib/distributed/cli/dask_scheduler.py -> build/bdist.linux-x86_64/wheel/./distributed/cli +copying build/lib/distributed/cli/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/cli +creating build/bdist.linux-x86_64/wheel/distributed/cli/tests +copying build/lib/distributed/cli/tests/test_dask_worker.py -> build/bdist.linux-x86_64/wheel/./distributed/cli/tests +copying build/lib/distributed/cli/tests/__init__.py -> build/bdist.linux-x86_64/wheel/./distributed/cli/tests +copying build/lib/distributed/cli/tests/test_dask_scheduler.py -> build/bdist.linux-x86_64/wheel/./distributed/cli/tests +copying build/lib/distributed/cli/tests/test_dask_ssh.py -> build/bdist.linux-x86_64/wheel/./distributed/cli/tests +copying build/lib/distributed/cli/tests/test_tls_cli.py -> build/bdist.linux-x86_64/wheel/./distributed/cli/tests +copying build/lib/distributed/cli/tests/test_dask_spec.py -> build/bdist.linux-x86_64/wheel/./distributed/cli/tests +copying build/lib/distributed/cli/utils.py -> build/bdist.linux-x86_64/wheel/./distributed/cli +copying build/lib/distributed/cli/dask_spec.py -> build/bdist.linux-x86_64/wheel/./distributed/cli running install_egg_info Copying distributed.egg-info to build/bdist.linux-x86_64/wheel/./distributed-2024.12.1.egg-info running install_scripts creating build/bdist.linux-x86_64/wheel/distributed-2024.12.1.dist-info/WHEEL -creating '/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/.tmp-vhxkhx6r/distributed-2024.12.1-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it +creating '/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/.tmp-12jfu3bs/distributed-2024.12.1-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it adding 'distributed/__init__.py' adding 'distributed/_async_taskgroup.py' adding 'distributed/_asyncio.py' @@ -4606,9 +4642,9 @@ preparing documents... done copying assets... copying static files... -Writing evaluated template result to /build/reproducible-path/dask.distributed-2024.12.1+ds/build/html/_static/documentation_options.js Writing evaluated template result to /build/reproducible-path/dask.distributed-2024.12.1+ds/build/html/_static/language_data.js Writing evaluated template result to /build/reproducible-path/dask.distributed-2024.12.1+ds/build/html/_static/basic.css +Writing evaluated template result to /build/reproducible-path/dask.distributed-2024.12.1+ds/build/html/_static/documentation_options.js Writing evaluated template result to /build/reproducible-path/dask.distributed-2024.12.1+ds/build/html/_static/alabaster.css copying static files: done copying extra files... @@ -4733,7 +4769,7 @@ cachedir: .pytest_cache rootdir: /build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.13_distributed/build configfile: pyproject.toml -plugins: typeguard-4.4.1, timeout-2.3.1, rerunfailures-14.0, anyio-4.7.0 +plugins: rerunfailures-14.0, anyio-4.7.0, timeout-2.3.1, typeguard-4.4.1 timeout: 300.0s timeout method: signal timeout func_only: False @@ -5394,7 +5430,7 @@ distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_partially_unknown_dimension[x2-chunks2] PASSED [ 20%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_partially_unknown_dimension[x3-chunks3] PASSED [ 20%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_partially_unknown_dimension[x4-chunks4] PASSED [ 20%] -distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_partially_unknown_dimension[x5-chunks5] PASSED [ 20%] +distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_partially_unknown_dimension[x5-chunks5] FAILED [ 20%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_partially_unknown_dimension[x6-chunks6] PASSED [ 20%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_partially_unknown_dimension[x7-chunks7] PASSED [ 20%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_partially_unknown_dimension[x8-chunks8] PASSED [ 20%] @@ -5459,7 +5495,7 @@ distributed/shuffle/tests/test_rechunk.py::test_calculate_prechunking_splitting[old0-new0-expected0] PASSED [ 22%] distributed/shuffle/tests/test_rechunk.py::test_calculate_prechunking_splitting[old1-new1-expected1] PASSED [ 22%] distributed/shuffle/tests/test_rechunk.py::test_calculate_prechunking_splitting[old2-new2-expected2] PASSED [ 22%] -distributed/shuffle/tests/test_rechunk.py::test_homogeneously_schedule_unpack PASSED [ 22%] +distributed/shuffle/tests/test_rechunk.py::test_homogeneously_schedule_unpack FAILED [ 22%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_datetime[tasks] PASSED [ 22%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_datetime[p2p] PASSED [ 22%] distributed/shuffle/tests/test_shuffle_plugins.py::test_installation_on_worker PASSED [ 22%] @@ -6215,7 +6251,7 @@ distributed/tests/test_compatibility.py::test_randbytes PASSED [ 45%] distributed/tests/test_compatibility.py::test_randbytes_seed PASSED [ 45%] distributed/tests/test_computations.py::test_computations PASSED [ 45%] -distributed/tests/test_computations.py::test_computations_futures PASSED [ 46%] +distributed/tests/test_computations.py::test_computations_futures FAILED [ 46%] distributed/tests/test_computations.py::test_computations_no_workers PASSED [ 46%] distributed/tests/test_computations.py::test_computations_no_resources PASSED [ 46%] distributed/tests/test_computations.py::test_computations_long_running PASSED [ 46%] @@ -6381,7 +6417,7 @@ distributed/tests/test_multi_locks.py::test_multiple_locks PASSED [ 50%] distributed/tests/test_multi_locks.py::test_num_locks PASSED [ 50%] distributed/tests/test_nanny.py::test_str PASSED [ 50%] -distributed/tests/test_nanny.py::test_nanny_process_failure PASSED [ 51%] +distributed/tests/test_nanny.py::test_nanny_process_failure FAILED [ 51%] distributed/tests/test_nanny.py::test_run PASSED [ 51%] distributed/tests/test_nanny.py::test_no_hang_when_scheduler_closes SKIPPED [ 51%] distributed/tests/test_nanny.py::test_close_on_disconnect SKIPPED (need @@ -7119,7 +7155,7 @@ distributed/tests/test_scheduler.py::test_deadlock_dependency_of_queued_released_when_worker_removed[True] PASSED [ 73%] distributed/tests/test_scheduler.py::test_deadlock_dependency_of_queued_released_when_worker_removed[False] PASSED [ 73%] distributed/tests/test_scheduler.py::test_submit_dependency_of_erred_task PASSED [ 73%] -distributed/tests/test_scheduler.py::test_tell_workers_when_peers_have_left PASSED [ 73%] +distributed/tests/test_scheduler.py::test_tell_workers_when_peers_have_left FAILED [ 73%] distributed/tests/test_scheduler.py::test_scatter_creates_ts PASSED [ 73%] distributed/tests/test_scheduler.py::test_refuse_to_schedule_huge_task[False] PASSED [ 73%] distributed/tests/test_scheduler.py::test_refuse_to_schedule_huge_task[True] PASSED [ 73%] @@ -7721,7 +7757,7 @@ distributed/tests/test_worker.py::test_missing_released_zombie_tasks PASSED [ 91%] distributed/tests/test_worker.py::test_missing_released_zombie_tasks_2 PASSED [ 92%] distributed/tests/test_worker.py::test_worker_status_sync PASSED [ 92%] -distributed/tests/test_worker.py::test_log_remove_worker PASSED [ 92%] +distributed/tests/test_worker.py::test_log_remove_worker FAILED [ 92%] distributed/tests/test_worker.py::test_task_flight_compute_oserror PASSED [ 92%] distributed/tests/test_worker.py::test_gather_dep_cancelled_rescheduled PASSED [ 92%] distributed/tests/test_worker.py::test_gather_dep_do_not_handle_response_of_not_requested_tasks PASSED [ 92%] @@ -7778,7 +7814,7 @@ distributed/tests/test_worker_memory.py::test_workerstate_fail_to_pickle_execute_1[long-running] PASSED [ 93%] distributed/tests/test_worker_memory.py::test_workerstate_fail_to_pickle_flight PASSED [ 93%] distributed/tests/test_worker_memory.py::test_fail_to_pickle_execute_2 PASSED [ 93%] -distributed/tests/test_worker_memory.py::test_fail_to_pickle_spill PASSED [ 93%] +distributed/tests/test_worker_memory.py::test_fail_to_pickle_spill FAILED [ 93%] distributed/tests/test_worker_memory.py::test_spill_target_threshold PASSED [ 93%] distributed/tests/test_worker_memory.py::test_spill_constrained PASSED [ 93%] distributed/tests/test_worker_memory.py::test_spill_spill_threshold PASSED [ 93%] @@ -7976,27 +8012,1448 @@ distributed/tests/test_worker_state_machine.py::test_remove_worker_while_in_fetch PASSED [ 99%] distributed/tests/test_worker_state_machine.py::test_remove_worker_unknown PASSED [100%] +=================================== FAILURES =================================== +__________ test_rechunk_with_partially_unknown_dimension[x5-chunks5] ___________ + +args = () +kwds = {'chunks': (None, (5, 5)), 'x': dask.array} + + @wraps(func) + def inner(*args, **kwds): + with self._recreate_cm(): +> return func(*args, **kwds) + +/usr/lib/python3.13/contextlib.py:85: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +/usr/lib/python3.13/contextlib.py:85: in inner + return func(*args, **kwds) +distributed/utils_test.py:1090: in test_func + return _run_and_close_tornado(async_fn_outer) +distributed/utils_test.py:380: in _run_and_close_tornado + return asyncio_run(inner_fn(), loop_factory=get_loop_factory()) +/usr/lib/python3.13/asyncio/runners.py:195: in run + return runner.run(main) +/usr/lib/python3.13/asyncio/runners.py:118: in run + return self._loop.run_until_complete(task) +/usr/lib/python3.13/asyncio/base_events.py:720: in run_until_complete + return future.result() +distributed/utils_test.py:377: in inner_fn + return await async_fn(*args, **kwargs) +distributed/utils_test.py:1087: in async_fn_outer + return await utils_wait_for(async_fn(), timeout=timeout * 2) +distributed/utils.py:1914: in wait_for + return await fut +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + + async def async_fn(): + result = None + with dask.config.set(config): + async with ( + _cluster_factory() as (s, workers), + _client_factory(s) as c, + ): + args = [s] + workers + if c is not None: + args = [c] + args + try: + coro = func(*args, *outer_args, **kwargs) + task = asyncio.create_task(coro) + coro2 = utils_wait_for( + asyncio.shield(task), timeout=deadline.remaining + ) + result = await coro2 + validate_state(s, *workers) + + except asyncio.TimeoutError: + assert task + elapsed = deadline.elapsed + buffer = io.StringIO() + # This stack indicates where the coro/test is suspended + task.print_stack(file=buffer) + + task.cancel() + while not task.cancelled(): + await asyncio.sleep(0.01) + + # Hopefully, the hang has been caused by inconsistent + # state, which should be much more meaningful than the + # timeout + validate_state(s, *workers) + + # Remove as much of the traceback as possible; it's + # uninteresting boilerplate from utils_test and asyncio + # and not from the code being tested. +> raise asyncio.TimeoutError( + f"Test timeout ({timeout}) hit after {elapsed}s.\n" + "========== Test stack trace starts here ==========\n" + f"{buffer.getvalue()}" + ) from None +E TimeoutError: Test timeout (30) hit after 30.031416482990608s. +E ========== Test stack trace starts here ========== +E Stack for wait_for=> (most recent call last): +E File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.13_distributed/build/distributed/shuffle/tests/test_rechunk.py", line 801, in test_rechunk_with_partially_unknown_dimension +E assert_eq(await c.compute(result), await c.compute(expected)) + +distributed/utils_test.py:1041: TimeoutError +----------------------------- Captured stderr call ----------------------------- +2025-01-14 15:36:20,495 - distributed.scheduler - INFO - State start +2025-01-14 15:36:20,521 - distributed.scheduler - INFO - Scheduler at: tcp://127.0.0.1:37119 +2025-01-14 15:36:20,522 - distributed.scheduler - INFO - dashboard at: http://127.0.0.1:34521/status +2025-01-14 15:36:20,523 - distributed.scheduler - INFO - Registering Worker plugin shuffle +2025-01-14 15:36:20,563 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:41453 +2025-01-14 15:36:20,564 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:41453 +2025-01-14 15:36:20,565 - distributed.worker - INFO - Worker name: 0 +2025-01-14 15:36:20,565 - distributed.worker - INFO - dashboard at: 127.0.0.1:42919 +2025-01-14 15:36:20,565 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:37119 +2025-01-14 15:36:20,566 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 15:36:20,566 - distributed.worker - INFO - Threads: 1 +2025-01-14 15:36:20,566 - distributed.worker - INFO - Memory: 78.53 GiB +2025-01-14 15:36:20,566 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-us4ofuae +2025-01-14 15:36:20,567 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 15:36:20,568 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:46311 +2025-01-14 15:36:20,569 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:46311 +2025-01-14 15:36:20,569 - distributed.worker - INFO - Worker name: 1 +2025-01-14 15:36:20,577 - distributed.worker - INFO - dashboard at: 127.0.0.1:41851 +2025-01-14 15:36:20,578 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:37119 +2025-01-14 15:36:20,579 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 15:36:20,580 - distributed.worker - INFO - Threads: 2 +2025-01-14 15:36:20,580 - distributed.worker - INFO - Memory: 78.53 GiB +2025-01-14 15:36:20,581 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-zle1ezp5 +2025-01-14 15:36:20,581 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 15:36:20,635 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:41453 name: 0 +2025-01-14 15:36:20,658 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:41453 +2025-01-14 15:36:20,659 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:47024 +2025-01-14 15:36:20,660 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:46311 name: 1 +2025-01-14 15:36:20,683 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:46311 +2025-01-14 15:36:20,685 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:47028 +2025-01-14 15:36:20,686 - distributed.worker - INFO - Starting Worker plugin shuffle +2025-01-14 15:36:20,697 - distributed.worker - INFO - Starting Worker plugin shuffle +2025-01-14 15:36:20,699 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:37119 +2025-01-14 15:36:20,700 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 15:36:20,701 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:37119 +2025-01-14 15:36:20,702 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 15:36:20,704 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:37119 +2025-01-14 15:36:20,704 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:37119 +2025-01-14 15:36:20,738 - distributed.scheduler - INFO - Receive client connection: Client-f4ed5e95-d217-11ef-a851-8d61c06d7977 +2025-01-14 15:36:20,757 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:47040 +2025-01-14 15:36:21,217 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 26c3317288d95f2ce53956244b9b3f3e initialized by task ('concatenate-rechunk-transfer-580c4862fda5190b82a6426149915f57', 8, 0, 8, 0) executed on worker tcp://127.0.0.1:41453 +2025-01-14 15:36:21,218 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle ce8b0e6800f58b68869b1e966c8288f5 initialized by task ('concatenate-rechunk-transfer-580c4862fda5190b82a6426149915f57', 9, 0, 9, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:21,244 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle da54cdb5b864510d583bfcc87b5e037e initialized by task ('concatenate-rechunk-transfer-580c4862fda5190b82a6426149915f57', 7, 0, 7, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:21,383 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 99094ac41f7b4dfc15456ba0eff3ef53 initialized by task ('concatenate-rechunk-transfer-580c4862fda5190b82a6426149915f57', 6, 0, 6, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:21,470 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle c7dd9af280764437a34f376f22267c0d initialized by task ('concatenate-rechunk-transfer-580c4862fda5190b82a6426149915f57', 5, 0, 5, 0) executed on worker tcp://127.0.0.1:41453 +2025-01-14 15:36:21,532 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 9756430cac596814b7da9a6d3c7f62a5 initialized by task ('concatenate-from-dask-array-values-rechunk-transfer-580c4862fda5190b82a6426149915f57', 29, 0, 29, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:25,531 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle d744ba7c15c691cd059c6a18a145920a initialized by task ('concatenate-from-dask-array-values-rechunk-transfer-580c4862fda5190b82a6426149915f57', 28, 0, 28, 0) executed on worker tcp://127.0.0.1:41453 +2025-01-14 15:36:25,534 - distributed.core - INFO - Event loop was unresponsive in Scheduler for 3.95s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability. +2025-01-14 15:36:25,547 - distributed.core - INFO - Event loop was unresponsive in Worker for 3.82s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability. +2025-01-14 15:36:25,565 - distributed.core - INFO - Event loop was unresponsive in Worker for 3.84s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability. +2025-01-14 15:36:25,985 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle ce8b0e6800f58b68869b1e966c8288f5 deactivated due to stimulus 'task-finished-1736818585.9565525' +2025-01-14 15:36:26,106 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 4306751e65c5b865c431006e1e96235b initialized by task ('concatenate-from-dask-array-values-rechunk-transfer-580c4862fda5190b82a6426149915f57', 27, 0, 27, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:26,162 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle a69a03691c3b2410fb0a9e8094608111 initialized by task ('concatenate-from-dask-array-values-rechunk-transfer-580c4862fda5190b82a6426149915f57', 26, 0, 26, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:26,534 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 26c3317288d95f2ce53956244b9b3f3e deactivated due to stimulus 'task-finished-1736818586.3047705' +2025-01-14 15:36:26,803 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle da54cdb5b864510d583bfcc87b5e037e deactivated due to stimulus 'task-finished-1736818586.7783346' +2025-01-14 15:36:27,138 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 99094ac41f7b4dfc15456ba0eff3ef53 deactivated due to stimulus 'task-finished-1736818587.113826' +2025-01-14 15:36:27,692 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 9756430cac596814b7da9a6d3c7f62a5 deactivated due to stimulus 'task-finished-1736818587.6387305' +2025-01-14 15:36:28,039 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 4306751e65c5b865c431006e1e96235b deactivated due to stimulus 'task-finished-1736818588.0113106' +2025-01-14 15:36:28,125 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 16bd48bf4cc743e932110cdd8320c5b8 initialized by task ('concatenate-rechunk-transfer-580c4862fda5190b82a6426149915f57', 4, 0, 4, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:28,348 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 544be6a5fcfae52cf0ddf4e2634b7e18 initialized by task ('concatenate-from-dask-array-values-rechunk-transfer-580c4862fda5190b82a6426149915f57', 24, 0, 24, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:28,379 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle c7dd9af280764437a34f376f22267c0d deactivated due to stimulus 'task-finished-1736818588.3770857' +2025-01-14 15:36:28,839 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle fa78bbee4fb0174c7a34c2cf0fca1400 initialized by task ('concatenate-from-dask-array-values-rechunk-transfer-580c4862fda5190b82a6426149915f57', 39, 0, 39, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:28,897 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle a69a03691c3b2410fb0a9e8094608111 deactivated due to stimulus 'task-finished-1736818588.8729057' +2025-01-14 15:36:28,931 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 4c363b72e8803e63dd274d181c3a16ec initialized by task ('concatenate-from-dask-array-values-rechunk-transfer-580c4862fda5190b82a6426149915f57', 25, 0, 25, 0) executed on worker tcp://127.0.0.1:41453 +2025-01-14 15:36:29,109 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 915324de98120b957e60c8cb0cf554a2 initialized by task ('concatenate-from-dask-array-values-rechunk-transfer-580c4862fda5190b82a6426149915f57', 38, 0, 38, 0) executed on worker tcp://127.0.0.1:41453 +2025-01-14 15:36:29,369 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle d744ba7c15c691cd059c6a18a145920a deactivated due to stimulus 'task-finished-1736818589.3566086' +2025-01-14 15:36:29,834 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 0c6b2c7f7de77093d2c21a6259d013af initialized by task ('concatenate-rechunk-transfer-580c4862fda5190b82a6426149915f57', 19, 0, 19, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:29,894 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 16bd48bf4cc743e932110cdd8320c5b8 deactivated due to stimulus 'task-finished-1736818589.8586757' +2025-01-14 15:36:29,967 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 065dbba6d9744fdb29654b34a0508950 initialized by task ('concatenate-from-dask-array-values-rechunk-transfer-580c4862fda5190b82a6426149915f57', 37, 0, 37, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:29,995 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle b3f2ebe05f68ee67700b7f6981831182 initialized by task ('concatenate-rechunk-transfer-580c4862fda5190b82a6426149915f57', 17, 0, 17, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:30,122 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle fa78bbee4fb0174c7a34c2cf0fca1400 deactivated due to stimulus 'task-finished-1736818590.0987155' +2025-01-14 15:36:30,154 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 9aef8fc1118f52c398d37b5a1903bf33 initialized by task ('concatenate-from-dask-array-values-rechunk-transfer-580c4862fda5190b82a6426149915f57', 36, 0, 36, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:30,280 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 0a420d592e276eaebf2381f98f71e14b initialized by task ('concatenate-rechunk-transfer-580c4862fda5190b82a6426149915f57', 16, 0, 16, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:30,448 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 915324de98120b957e60c8cb0cf554a2 deactivated due to stimulus 'task-finished-1736818590.4435313' +2025-01-14 15:36:30,773 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 4c363b72e8803e63dd274d181c3a16ec deactivated due to stimulus 'task-finished-1736818590.7687576' +2025-01-14 15:36:30,812 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 065dbba6d9744fdb29654b34a0508950 deactivated due to stimulus 'task-finished-1736818590.8087606' +2025-01-14 15:36:30,954 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 544be6a5fcfae52cf0ddf4e2634b7e18 deactivated due to stimulus 'task-finished-1736818590.9299517' +2025-01-14 15:36:31,030 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 9aef8fc1118f52c398d37b5a1903bf33 deactivated due to stimulus 'task-finished-1736818591.0255628' +2025-01-14 15:36:31,166 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 0c6b2c7f7de77093d2c21a6259d013af deactivated due to stimulus 'task-finished-1736818591.1624498' +2025-01-14 15:36:31,199 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 304aca75b19ebbcdbd1c5599d79cb9a7 initialized by task ('concatenate-rechunk-transfer-580c4862fda5190b82a6426149915f57', 18, 0, 18, 0) executed on worker tcp://127.0.0.1:41453 +2025-01-14 15:36:31,286 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 5b3f1b9e0965c1f17998911e7feb5d59 initialized by task ('concatenate-from-dask-array-values-rechunk-transfer-580c4862fda5190b82a6426149915f57', 35, 0, 35, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:31,458 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle f8ac76d7f05e61c09dc76f29589861bc initialized by task ('concatenate-from-dask-array-values-rechunk-transfer-580c4862fda5190b82a6426149915f57', 34, 0, 34, 0) executed on worker tcp://127.0.0.1:41453 +2025-01-14 15:36:31,552 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 4ada31f5d44c3132d3e7174e69912392 initialized by task ('concatenate-rechunk-transfer-580c4862fda5190b82a6426149915f57', 15, 0, 15, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:31,614 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle d2d37bf71e9b6f630a670e3ec30a49f4 initialized by task ('concatenate-from-dask-array-values-rechunk-transfer-580c4862fda5190b82a6426149915f57', 33, 0, 33, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:31,652 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle f58152e5d04e648d488ad479f9e5184e initialized by task ('concatenate-rechunk-transfer-580c4862fda5190b82a6426149915f57', 13, 0, 13, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:31,663 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 5a9b60710dd85108cdc0846a76ddba4a initialized by task ('concatenate-from-dask-array-values-rechunk-transfer-580c4862fda5190b82a6426149915f57', 32, 0, 32, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:31,818 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 5b3f1b9e0965c1f17998911e7feb5d59 deactivated due to stimulus 'task-finished-1736818591.7980819' +2025-01-14 15:36:32,301 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 076383f284df1561d679cfed9d6130ad initialized by task ('concatenate-rechunk-transfer-580c4862fda5190b82a6426149915f57', 14, 0, 14, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:32,419 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle d2d37bf71e9b6f630a670e3ec30a49f4 deactivated due to stimulus 'task-finished-1736818592.401391' +2025-01-14 15:36:32,446 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle f8ac76d7f05e61c09dc76f29589861bc deactivated due to stimulus 'task-finished-1736818592.4169137' +2025-01-14 15:36:32,629 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle b3f2ebe05f68ee67700b7f6981831182 deactivated due to stimulus 'task-finished-1736818592.6195343' +2025-01-14 15:36:32,725 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 5a9b60710dd85108cdc0846a76ddba4a deactivated due to stimulus 'task-finished-1736818592.6925228' +2025-01-14 15:36:32,726 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 304aca75b19ebbcdbd1c5599d79cb9a7 deactivated due to stimulus 'task-finished-1736818592.6926525' +2025-01-14 15:36:32,870 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 0a420d592e276eaebf2381f98f71e14b deactivated due to stimulus 'task-finished-1736818592.8543324' +2025-01-14 15:36:32,899 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 5cf58e8c0ec89d188451f465a5458fae initialized by task ('concatenate-rechunk-transfer-580c4862fda5190b82a6426149915f57', 12, 0, 12, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:32,994 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 4ada31f5d44c3132d3e7174e69912392 deactivated due to stimulus 'task-finished-1736818592.9664207' +2025-01-14 15:36:33,135 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 5d3e7411c31b3cbe306359565e89da70 initialized by task ('concatenate-from-dask-array-values-rechunk-transfer-580c4862fda5190b82a6426149915f57', 31, 0, 31, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:33,148 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 53821c1f6d345fc603e8d81df66881f8 initialized by task ('concatenate-rechunk-transfer-580c4862fda5190b82a6426149915f57', 11, 0, 11, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:33,151 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle f58152e5d04e648d488ad479f9e5184e deactivated due to stimulus 'task-finished-1736818593.1187427' +2025-01-14 15:36:33,355 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 49abf960ec09769d21d20215794a71d2 initialized by task ('concatenate-from-dask-array-values-rechunk-transfer-580c4862fda5190b82a6426149915f57', 30, 0, 30, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:33,417 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 076383f284df1561d679cfed9d6130ad deactivated due to stimulus 'task-finished-1736818593.3672788' +2025-01-14 15:36:33,577 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 0a88f6ac425a30b97898968bca631647 initialized by task ('concatenate-rechunk-transfer-580c4862fda5190b82a6426149915f57', 3, 0, 3, 0) executed on worker tcp://127.0.0.1:41453 +2025-01-14 15:36:33,623 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 9715ae9bb7b9d9ba68df63704ee48bd3 initialized by task ('concatenate-rechunk-transfer-580c4862fda5190b82a6426149915f57', 10, 0, 10, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:33,813 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle cdd2dbf50131c1bb4ddb590b756e9eb7 initialized by task ('concatenate-from-dask-array-values-rechunk-transfer-580c4862fda5190b82a6426149915f57', 23, 0, 23, 0) executed on worker tcp://127.0.0.1:41453 +2025-01-14 15:36:33,980 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 8be1cdff735f46a9c954489b629735ff initialized by task ('concatenate-from-dask-array-values-rechunk-transfer-580c4862fda5190b82a6426149915f57', 22, 0, 22, 0) executed on worker tcp://127.0.0.1:41453 +2025-01-14 15:36:34,378 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 5d3e7411c31b3cbe306359565e89da70 deactivated due to stimulus 'task-finished-1736818594.3592427' +2025-01-14 15:36:34,477 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 49abf960ec09769d21d20215794a71d2 deactivated due to stimulus 'task-finished-1736818594.4707417' +2025-01-14 15:36:34,525 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 69643ae65652aa25eed4615e62422b67 initialized by task ('concatenate-from-dask-array-values-rechunk-transfer-580c4862fda5190b82a6426149915f57', 21, 0, 21, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:34,786 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 0a88f6ac425a30b97898968bca631647 deactivated due to stimulus 'task-finished-1736818594.7404852' +2025-01-14 15:36:34,861 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 02e76c9d8dc420a3149bf42562de67b8 initialized by task ('concatenate-rechunk-transfer-580c4862fda5190b82a6426149915f57', 1, 0, 1, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:34,975 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 160efda853b705f2f3b0c50ec98593f0 initialized by task ('concatenate-rechunk-transfer-580c4862fda5190b82a6426149915f57', 2, 0, 2, 0) executed on worker tcp://127.0.0.1:41453 +2025-01-14 15:36:35,149 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 08a07088bc1808c6052b3f0141a5ad3b initialized by task ('concatenate-rechunk-transfer-580c4862fda5190b82a6426149915f57', 0, 0, 0, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:35,151 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 86767d536aa63f35d12166fe92e01e63 initialized by task ('concatenate-from-dask-array-values-rechunk-transfer-580c4862fda5190b82a6426149915f57', 20, 0, 20, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:35,517 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 5cf58e8c0ec89d188451f465a5458fae deactivated due to stimulus 'task-finished-1736818595.508702' +2025-01-14 15:36:35,646 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle cdd2dbf50131c1bb4ddb590b756e9eb7 deactivated due to stimulus 'task-finished-1736818595.6328175' +2025-01-14 15:36:35,953 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 69643ae65652aa25eed4615e62422b67 deactivated due to stimulus 'task-finished-1736818595.9408424' +2025-01-14 15:36:36,208 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 8be1cdff735f46a9c954489b629735ff deactivated due to stimulus 'task-finished-1736818596.1964605' +2025-01-14 15:36:36,397 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 86767d536aa63f35d12166fe92e01e63 deactivated due to stimulus 'task-finished-1736818596.3516548' +2025-01-14 15:36:36,456 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 53821c1f6d345fc603e8d81df66881f8 deactivated due to stimulus 'task-finished-1736818596.4485533' +2025-01-14 15:36:36,601 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 9715ae9bb7b9d9ba68df63704ee48bd3 deactivated due to stimulus 'task-finished-1736818596.5955808' +2025-01-14 15:36:36,670 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 160efda853b705f2f3b0c50ec98593f0 deactivated due to stimulus 'task-finished-1736818596.6630027' +2025-01-14 15:36:36,697 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 02e76c9d8dc420a3149bf42562de67b8 deactivated due to stimulus 'task-finished-1736818596.6941442' +2025-01-14 15:36:36,737 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 08a07088bc1808c6052b3f0141a5ad3b deactivated due to stimulus 'task-finished-1736818596.7347238' +2025-01-14 15:36:37,030 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 8df9fe463269d893a469618291c3cd80 initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 9, 0, 9, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:37,057 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 46629bb68b3978db98af7e8175adbe79 initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 7, 0, 7, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:37,075 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 6b3ef5e62626971016a1aa030e24a157 initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 6, 0, 6, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:37,147 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle a58b083325154e90890be177e668c4fa initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 29, 0, 29, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:37,178 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 0346d2432940895b2cf6975f7b82297a initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 8, 0, 8, 0) executed on worker tcp://127.0.0.1:41453 +2025-01-14 15:36:37,322 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 72c0f33da8da1176a8bf5446eca58f8a initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 27, 0, 27, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:37,398 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 4ccb0081d7e79796b62c9f4824d4c690 initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 5, 0, 5, 0) executed on worker tcp://127.0.0.1:41453 +2025-01-14 15:36:37,616 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 54eeb25b994d4878df3ae2060f73d515 initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 28, 0, 28, 0) executed on worker tcp://127.0.0.1:41453 +2025-01-14 15:36:37,781 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 8df9fe463269d893a469618291c3cd80 deactivated due to stimulus 'task-finished-1736818597.7782857' +2025-01-14 15:36:37,834 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle c9130564459d894ea34be58d4a4748bd initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 26, 0, 26, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:37,923 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 0346d2432940895b2cf6975f7b82297a deactivated due to stimulus 'task-finished-1736818597.9155927' +2025-01-14 15:36:37,978 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 46629bb68b3978db98af7e8175adbe79 deactivated due to stimulus 'task-finished-1736818597.9706895' +2025-01-14 15:36:38,005 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle e973c77fd175dfde94f86ec5d533967d initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 4, 0, 4, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:38,073 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 027cfccc12e4642709c047ee1c65bae2 initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 25, 0, 25, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:38,075 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 6b3ef5e62626971016a1aa030e24a157 deactivated due to stimulus 'task-finished-1736818598.0398183' +2025-01-14 15:36:38,161 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle f9b9d4659dd17913d2bb4de3d308cb64 initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 24, 0, 24, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:38,344 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle a58b083325154e90890be177e668c4fa deactivated due to stimulus 'task-finished-1736818598.3202963' +2025-01-14 15:36:38,582 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 72c0f33da8da1176a8bf5446eca58f8a deactivated due to stimulus 'task-finished-1736818598.5779514' +2025-01-14 15:36:38,924 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 8b720bb185d0c79311afc7af61095e8c initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 39, 0, 39, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:38,926 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 4ccb0081d7e79796b62c9f4824d4c690 deactivated due to stimulus 'task-finished-1736818598.9207375' +2025-01-14 15:36:39,050 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle e973c77fd175dfde94f86ec5d533967d deactivated due to stimulus 'task-finished-1736818599.0462852' +2025-01-14 15:36:39,118 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle c9130564459d894ea34be58d4a4748bd deactivated due to stimulus 'task-finished-1736818599.1086664' +2025-01-14 15:36:39,142 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 85a2af1777bb06b8652ee322a4673a4f initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 19, 0, 19, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:39,273 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 54eeb25b994d4878df3ae2060f73d515 deactivated due to stimulus 'task-finished-1736818599.204908' +2025-01-14 15:36:39,355 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 027cfccc12e4642709c047ee1c65bae2 deactivated due to stimulus 'task-finished-1736818599.3411813' +2025-01-14 15:36:39,483 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle f9b9d4659dd17913d2bb4de3d308cb64 deactivated due to stimulus 'task-finished-1736818599.4654677' +2025-01-14 15:36:39,569 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle fb76b4a5787523e0edbca18de9652ab8 initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 38, 0, 38, 0) executed on worker tcp://127.0.0.1:41453 +2025-01-14 15:36:39,881 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 8e7206bc8a4005b7afe7a40a9a685287 initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 18, 0, 18, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:39,932 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 8b720bb185d0c79311afc7af61095e8c deactivated due to stimulus 'task-finished-1736818599.9059265' +2025-01-14 15:36:39,983 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle fa85133c112254014ee1c995553972f5 initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 37, 0, 37, 0) executed on worker tcp://127.0.0.1:41453 +2025-01-14 15:36:40,047 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 2df307e81810f1927e3a6b6e4b147edb initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 36, 0, 36, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:40,102 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 122d4ecbf0503661c32f995be24d667f initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 35, 0, 35, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:40,271 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 326d2eba2048d7d309145583a1652b98 initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 17, 0, 17, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:40,563 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 85a2af1777bb06b8652ee322a4673a4f deactivated due to stimulus 'task-finished-1736818600.531532' +2025-01-14 15:36:40,846 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle fb76b4a5787523e0edbca18de9652ab8 deactivated due to stimulus 'task-finished-1736818600.8229766' +2025-01-14 15:36:40,986 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 059ebb5c119ab9512600f8d94e48a962 initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 16, 0, 16, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:41,078 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle cd09afd6610b87f8fb2104d33a62bd4c initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 15, 0, 15, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:41,165 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 2df307e81810f1927e3a6b6e4b147edb deactivated due to stimulus 'task-finished-1736818601.1405566' +2025-01-14 15:36:41,454 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle fa85133c112254014ee1c995553972f5 deactivated due to stimulus 'task-finished-1736818601.4411173' +2025-01-14 15:36:41,590 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 122d4ecbf0503661c32f995be24d667f deactivated due to stimulus 'task-finished-1736818601.5645988' +2025-01-14 15:36:41,688 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 8e7206bc8a4005b7afe7a40a9a685287 deactivated due to stimulus 'task-finished-1736818601.6694102' +2025-01-14 15:36:41,803 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 6c2457af56a2541cdc250be63fee9827 initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 34, 0, 34, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:41,967 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 326d2eba2048d7d309145583a1652b98 deactivated due to stimulus 'task-finished-1736818601.88239' +2025-01-14 15:36:42,311 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 0130de6d22665437754c363d876a9778 initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 14, 0, 14, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:42,465 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 059ebb5c119ab9512600f8d94e48a962 deactivated due to stimulus 'task-finished-1736818602.3824348' +2025-01-14 15:36:42,467 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 29df81c2fab3ff135a91e9d08b60880f initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 33, 0, 33, 0) executed on worker tcp://127.0.0.1:41453 +2025-01-14 15:36:43,031 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 3d2b3fe8ac203e3fce0c58b8d31db873 initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 13, 0, 13, 0) executed on worker tcp://127.0.0.1:41453 +2025-01-14 15:36:43,151 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle e27dbd338acc5cd1c7db495977494ac7 initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 32, 0, 32, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:43,389 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle b28d8072c00d162533ca97d9d2c0eaf9 initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 12, 0, 12, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:44,488 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle cd09afd6610b87f8fb2104d33a62bd4c deactivated due to stimulus 'task-finished-1736818604.468911' +2025-01-14 15:36:44,901 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 6c2457af56a2541cdc250be63fee9827 deactivated due to stimulus 'task-finished-1736818604.8834252' +2025-01-14 15:36:44,994 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 5654c16538fd7f846dedb83a375cf190 initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 31, 0, 31, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:45,786 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 29df81c2fab3ff135a91e9d08b60880f deactivated due to stimulus 'task-finished-1736818605.7091815' +2025-01-14 15:36:45,823 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle e83f4ad4f1631ced4a53dfe4d2c01eaf initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 11, 0, 11, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:46,331 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle e27dbd338acc5cd1c7db495977494ac7 deactivated due to stimulus 'task-finished-1736818606.3046186' +2025-01-14 15:36:46,591 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 0207bb699c0052b2461ae8f18a152d26 initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 30, 0, 30, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:46,950 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 9bdf1e1c97e51e3e0fc97b5e6432f970 initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 10, 0, 10, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:47,187 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 5654c16538fd7f846dedb83a375cf190 deactivated due to stimulus 'task-finished-1736818607.1620233' +2025-01-14 15:36:47,334 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 0130de6d22665437754c363d876a9778 deactivated due to stimulus 'task-finished-1736818607.3316066' +2025-01-14 15:36:47,581 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle e4e6a7b664f65d6edfb259b45651ed27 initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 3, 0, 3, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:47,604 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle b28d8072c00d162533ca97d9d2c0eaf9 deactivated due to stimulus 'task-finished-1736818607.599291' +2025-01-14 15:36:47,635 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle b2dfa77b3e2745f343c2221a272a322a initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 23, 0, 23, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:47,773 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 0207bb699c0052b2461ae8f18a152d26 deactivated due to stimulus 'task-finished-1736818607.7606225' +2025-01-14 15:36:48,012 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle e83f4ad4f1631ced4a53dfe4d2c01eaf deactivated due to stimulus 'task-finished-1736818608.0004861' +2025-01-14 15:36:48,139 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 3d2b3fe8ac203e3fce0c58b8d31db873 deactivated due to stimulus 'task-finished-1736818608.1170926' +2025-01-14 15:36:48,329 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle d8daf9ada0991b2f3e792dc48eccbe5d initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 22, 0, 22, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:48,349 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 9bdf1e1c97e51e3e0fc97b5e6432f970 deactivated due to stimulus 'task-finished-1736818608.3402982' +2025-01-14 15:36:48,393 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 6fc0fe775099cc7d020c8d0d7fce14db initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 21, 0, 21, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:48,657 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle e4e6a7b664f65d6edfb259b45651ed27 deactivated due to stimulus 'task-finished-1736818608.6034257' +2025-01-14 15:36:48,800 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle e8e430d096f576da265f40b25acd59eb initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 1, 0, 1, 0) executed on worker tcp://127.0.0.1:41453 +2025-01-14 15:36:48,912 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle ab4e7ccf9aa7da1ba5dbbe932f424b7e initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 2, 0, 2, 0) executed on worker tcp://127.0.0.1:46311 +2025-01-14 15:36:49,417 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle b2dfa77b3e2745f343c2221a272a322a deactivated due to stimulus 'task-finished-1736818609.3023732' +2025-01-14 15:36:49,437 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle e7ab79b69bf5f494a3f6a7a98f40ad9b initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 20, 0, 20, 0) executed on worker tcp://127.0.0.1:41453 +2025-01-14 15:36:50,033 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 8d90864cd03ac15b1170f31c0af57011 initialized by task ('concatenate-rechunk-transfer-6a7de7f9d0d3e6c8d5d8514bf2ae355d', 0, 0, 0, 0) executed on worker tcp://127.0.0.1:41453 +2025-01-14 15:36:50,657 - distributed.scheduler - INFO - Remove client Client-f4ed5e95-d217-11ef-a851-8d61c06d7977 +2025-01-14 15:36:50,676 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 6fc0fe775099cc7d020c8d0d7fce14db deactivated due to stimulus 'remove-client-1736818610.6575077' +2025-01-14 15:36:50,677 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle d8daf9ada0991b2f3e792dc48eccbe5d deactivated due to stimulus 'remove-client-1736818610.6575077' +2025-01-14 15:36:50,679 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle ab4e7ccf9aa7da1ba5dbbe932f424b7e deactivated due to stimulus 'remove-client-1736818610.6575077' +2025-01-14 15:36:50,685 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle e7ab79b69bf5f494a3f6a7a98f40ad9b deactivated due to stimulus 'remove-client-1736818610.6575077' +2025-01-14 15:36:50,687 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle e8e430d096f576da265f40b25acd59eb deactivated due to stimulus 'remove-client-1736818610.6575077' +2025-01-14 15:36:50,702 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 8d90864cd03ac15b1170f31c0af57011 deactivated due to stimulus 'remove-client-1736818610.6575077' +2025-01-14 15:36:50,712 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:47040; closing. +2025-01-14 15:36:50,713 - distributed.scheduler - INFO - Remove client Client-f4ed5e95-d217-11ef-a851-8d61c06d7977 +2025-01-14 15:36:50,791 - distributed.scheduler - INFO - Close client connection: Client-f4ed5e95-d217-11ef-a851-8d61c06d7977 +2025-01-14 15:36:50,874 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:41453. Reason: worker-close +2025-01-14 15:36:50,876 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:46311. Reason: worker-close +2025-01-14 15:36:50,895 - distributed.worker.state_machine - WARNING - Async instruction for > ended with CancelledError +2025-01-14 15:36:50,897 - distributed.worker.state_machine - WARNING - Async instruction for > ended with CancelledError +2025-01-14 15:36:50,925 - distributed.worker - INFO - Removing Worker plugin shuffle +2025-01-14 15:36:50,936 - distributed.worker - INFO - Removing Worker plugin shuffle +2025-01-14 15:36:50,972 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:47024; closing. +2025-01-14 15:36:50,972 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:47028; closing. +2025-01-14 15:36:50,973 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:41453 name: 0 (stimulus_id='handle-worker-cleanup-1736818610.9735432') +2025-01-14 15:36:50,975 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:46311 name: 1 (stimulus_id='handle-worker-cleanup-1736818610.9752793') +2025-01-14 15:36:50,988 - distributed.scheduler - INFO - Lost all workers +2025-01-14 15:36:50,990 - distributed.core - INFO - Connection to tcp://127.0.0.1:37119 has been closed. +2025-01-14 15:36:51,000 - distributed.core - INFO - Connection to tcp://127.0.0.1:37119 has been closed. +2025-01-14 15:36:51,056 - distributed.scheduler - INFO - Closing scheduler. Reason: unknown +2025-01-14 15:36:51,062 - distributed.scheduler - INFO - Scheduler closing all comms +______________________ test_homogeneously_schedule_unpack ______________________ + +args = (), kwds = {} + + @wraps(func) + def inner(*args, **kwds): + with self._recreate_cm(): +> return func(*args, **kwds) + +/usr/lib/python3.13/contextlib.py:85: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +/usr/lib/python3.13/contextlib.py:85: in inner + return func(*args, **kwds) +distributed/utils_test.py:1090: in test_func + return _run_and_close_tornado(async_fn_outer) +distributed/utils_test.py:380: in _run_and_close_tornado + return asyncio_run(inner_fn(), loop_factory=get_loop_factory()) +/usr/lib/python3.13/asyncio/runners.py:195: in run + return runner.run(main) +/usr/lib/python3.13/asyncio/runners.py:118: in run + return self._loop.run_until_complete(task) +/usr/lib/python3.13/asyncio/base_events.py:720: in run_until_complete + return future.result() +distributed/utils_test.py:377: in inner_fn + return await async_fn(*args, **kwargs) +distributed/utils_test.py:1087: in async_fn_outer + return await utils_wait_for(async_fn(), timeout=timeout * 2) +distributed/utils.py:1914: in wait_for + return await fut +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + + async def async_fn(): + result = None + with dask.config.set(config): + async with ( + _cluster_factory() as (s, workers), + _client_factory(s) as c, + ): + args = [s] + workers + if c is not None: + args = [c] + args + try: + coro = func(*args, *outer_args, **kwargs) + task = asyncio.create_task(coro) + coro2 = utils_wait_for( + asyncio.shield(task), timeout=deadline.remaining + ) + result = await coro2 + validate_state(s, *workers) + + except asyncio.TimeoutError: + assert task + elapsed = deadline.elapsed + buffer = io.StringIO() + # This stack indicates where the coro/test is suspended + task.print_stack(file=buffer) + + task.cancel() + while not task.cancelled(): + await asyncio.sleep(0.01) + + # Hopefully, the hang has been caused by inconsistent + # state, which should be much more meaningful than the + # timeout + validate_state(s, *workers) + + # Remove as much of the traceback as possible; it's + # uninteresting boilerplate from utils_test and asyncio + # and not from the code being tested. +> raise asyncio.TimeoutError( + f"Test timeout ({timeout}) hit after {elapsed}s.\n" + "========== Test stack trace starts here ==========\n" + f"{buffer.getvalue()}" + ) from None +E TimeoutError: Test timeout (30) hit after 30.414505616994575s. +E ========== Test stack trace starts here ========== +E Stack for wait_for=> (most recent call last): +E File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.13_distributed/build/distributed/shuffle/tests/test_rechunk.py", line 1522, in test_homogeneously_schedule_unpack +E await c.compute(res) + +distributed/utils_test.py:1041: TimeoutError +----------------------------- Captured stderr call ----------------------------- +2025-01-14 15:38:00,479 - distributed.scheduler - INFO - State start +2025-01-14 15:38:00,527 - distributed.scheduler - INFO - Scheduler at: tcp://127.0.0.1:41089 +2025-01-14 15:38:00,529 - distributed.scheduler - INFO - dashboard at: http://127.0.0.1:34895/status +2025-01-14 15:38:00,545 - distributed.scheduler - INFO - Registering Worker plugin shuffle +2025-01-14 15:38:00,713 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:39317 +2025-01-14 15:38:00,715 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:39317 +2025-01-14 15:38:00,724 - distributed.worker - INFO - Worker name: 0 +2025-01-14 15:38:00,725 - distributed.worker - INFO - dashboard at: 127.0.0.1:33101 +2025-01-14 15:38:00,726 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:41089 +2025-01-14 15:38:00,726 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 15:38:00,727 - distributed.worker - INFO - Threads: 1 +2025-01-14 15:38:00,727 - distributed.worker - INFO - Memory: 78.53 GiB +2025-01-14 15:38:00,740 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-vhzbddxy +2025-01-14 15:38:00,742 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 15:38:00,758 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:38551 +2025-01-14 15:38:00,768 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:38551 +2025-01-14 15:38:00,770 - distributed.worker - INFO - Worker name: 1 +2025-01-14 15:38:00,770 - distributed.worker - INFO - dashboard at: 127.0.0.1:38519 +2025-01-14 15:38:00,771 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:41089 +2025-01-14 15:38:00,771 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 15:38:00,788 - distributed.worker - INFO - Threads: 1 +2025-01-14 15:38:00,789 - distributed.worker - INFO - Memory: 78.53 GiB +2025-01-14 15:38:00,790 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-e0s1uzq7 +2025-01-14 15:38:00,790 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 15:38:00,810 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:34561 +2025-01-14 15:38:00,811 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:34561 +2025-01-14 15:38:00,811 - distributed.worker - INFO - Worker name: 2 +2025-01-14 15:38:00,829 - distributed.worker - INFO - dashboard at: 127.0.0.1:43205 +2025-01-14 15:38:00,830 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:41089 +2025-01-14 15:38:00,830 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 15:38:00,831 - distributed.worker - INFO - Threads: 1 +2025-01-14 15:38:00,831 - distributed.worker - INFO - Memory: 78.53 GiB +2025-01-14 15:38:00,848 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-g6o44xyi +2025-01-14 15:38:00,850 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 15:38:00,873 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:44689 +2025-01-14 15:38:00,988 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:44689 +2025-01-14 15:38:00,989 - distributed.worker - INFO - Worker name: 3 +2025-01-14 15:38:00,990 - distributed.worker - INFO - dashboard at: 127.0.0.1:41687 +2025-01-14 15:38:00,990 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:41089 +2025-01-14 15:38:00,991 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 15:38:00,991 - distributed.worker - INFO - Threads: 1 +2025-01-14 15:38:01,012 - distributed.worker - INFO - Memory: 78.53 GiB +2025-01-14 15:38:01,013 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-l_sw1iq2 +2025-01-14 15:38:01,014 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 15:38:01,107 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:39317 name: 0 +2025-01-14 15:38:01,254 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:39317 +2025-01-14 15:38:01,484 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:48614 +2025-01-14 15:38:01,486 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:38551 name: 1 +2025-01-14 15:38:01,653 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:38551 +2025-01-14 15:38:01,852 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:48620 +2025-01-14 15:38:01,854 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:34561 name: 2 +2025-01-14 15:38:01,894 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:34561 +2025-01-14 15:38:02,156 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:48624 +2025-01-14 15:38:02,157 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:44689 name: 3 +2025-01-14 15:38:02,236 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:44689 +2025-01-14 15:38:02,397 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:48634 +2025-01-14 15:38:02,402 - distributed.worker - INFO - Starting Worker plugin shuffle +2025-01-14 15:38:02,429 - distributed.worker - INFO - Starting Worker plugin shuffle +2025-01-14 15:38:02,430 - distributed.worker - INFO - Starting Worker plugin shuffle +2025-01-14 15:38:02,454 - distributed.worker - INFO - Starting Worker plugin shuffle +2025-01-14 15:38:02,471 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:41089 +2025-01-14 15:38:02,472 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 15:38:02,497 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:41089 +2025-01-14 15:38:02,498 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 15:38:02,499 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:41089 +2025-01-14 15:38:02,499 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 15:38:02,504 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:41089 +2025-01-14 15:38:02,506 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 15:38:02,525 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:41089 +2025-01-14 15:38:02,525 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:41089 +2025-01-14 15:38:02,525 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:41089 +2025-01-14 15:38:02,525 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:41089 +2025-01-14 15:38:02,576 - distributed.scheduler - INFO - Receive client connection: Client-319e07d9-d218-11ef-a851-8d61c06d7977 +2025-01-14 15:38:02,592 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:58148 +2025-01-14 15:38:02,786 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 5c832b79a665de169eb155d58c90ec3e initialized by task ('random_sample-rechunk-transfer-3123949ca75adf213058c457d6f4b8fa', 0, 0, 99, 0) executed on worker tcp://127.0.0.1:44689 +2025-01-14 15:38:31,302 - distributed.scheduler - INFO - Remove client Client-319e07d9-d218-11ef-a851-8d61c06d7977 +2025-01-14 15:38:31,343 - distributed.shuffle._scheduler_plugin - WARNING - Shuffle 5c832b79a665de169eb155d58c90ec3e deactivated due to stimulus 'remove-client-1736818711.302187' +2025-01-14 15:38:31,394 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:58148; closing. +2025-01-14 15:38:31,418 - distributed.scheduler - INFO - Remove client Client-319e07d9-d218-11ef-a851-8d61c06d7977 +2025-01-14 15:38:31,819 - distributed.scheduler - INFO - Close client connection: Client-319e07d9-d218-11ef-a851-8d61c06d7977 +2025-01-14 15:38:31,845 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:39317. Reason: worker-close +2025-01-14 15:38:31,847 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:38551. Reason: worker-close +2025-01-14 15:38:31,856 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:34561. Reason: worker-close +2025-01-14 15:38:31,872 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:44689. Reason: worker-close +2025-01-14 15:38:32,069 - distributed.worker - INFO - Removing Worker plugin shuffle +2025-01-14 15:38:32,226 - distributed.worker.state_machine - WARNING - Async instruction for > ended with CancelledError +2025-01-14 15:38:32,236 - distributed.worker.state_machine - WARNING - Async instruction for > ended with CancelledError +2025-01-14 15:38:32,238 - distributed.worker.state_machine - WARNING - Async instruction for > ended with CancelledError +2025-01-14 15:38:32,326 - distributed.worker - INFO - Removing Worker plugin shuffle +2025-01-14 15:38:32,327 - distributed.worker - INFO - Removing Worker plugin shuffle +2025-01-14 15:38:32,336 - distributed.worker - INFO - Removing Worker plugin shuffle +2025-01-14 15:38:32,778 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:48634; closing. +2025-01-14 15:38:32,806 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:44689 name: 3 (stimulus_id='handle-worker-cleanup-1736818712.8064811') +2025-01-14 15:38:32,818 - distributed.core - INFO - Connection to tcp://127.0.0.1:41089 has been closed. +2025-01-14 15:38:32,862 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:48620; closing. +2025-01-14 15:38:32,877 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:38551 name: 1 (stimulus_id='handle-worker-cleanup-1736818712.8773332') +2025-01-14 15:38:32,900 - distributed.core - INFO - Connection to tcp://127.0.0.1:41089 has been closed. +2025-01-14 15:38:32,947 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:48624; closing. +2025-01-14 15:38:32,949 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:34561 name: 2 (stimulus_id='handle-worker-cleanup-1736818712.9495075') +2025-01-14 15:38:32,951 - distributed.core - INFO - Connection to tcp://127.0.0.1:41089 has been closed. +2025-01-14 15:38:32,990 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:48614; closing. +2025-01-14 15:38:32,997 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:39317 name: 0 (stimulus_id='handle-worker-cleanup-1736818712.997064') +2025-01-14 15:38:32,998 - distributed.scheduler - INFO - Lost all workers +2025-01-14 15:38:32,999 - distributed.core - INFO - Connection to tcp://127.0.0.1:41089 has been closed. +2025-01-14 15:38:33,025 - distributed.scheduler - INFO - Closing scheduler. Reason: unknown +2025-01-14 15:38:33,027 - distributed.scheduler - INFO - Scheduler closing all comms +__________________________ test_computations_futures ___________________________ + +c = +s = +a = +b = + + @gen_cluster(client=True) + async def test_computations_futures(c, s, a, b): + futures = [c.submit(inc, i) for i in range(10)] + total = c.submit(sum, futures) + await total + +> [computation] = s.computations +E ValueError: too many values to unpack (expected 1) + +distributed/tests/test_computations.py:41: ValueError +----------------------------- Captured stderr call ----------------------------- +2025-01-14 15:56:44,066 - distributed.scheduler - INFO - State start +2025-01-14 15:56:44,074 - distributed.scheduler - INFO - Scheduler at: tcp://127.0.0.1:42323 +2025-01-14 15:56:44,077 - distributed.scheduler - INFO - dashboard at: http://127.0.0.1:42499/status +2025-01-14 15:56:44,079 - distributed.scheduler - INFO - Registering Worker plugin shuffle +2025-01-14 15:56:44,091 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:34551 +2025-01-14 15:56:44,093 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:34551 +2025-01-14 15:56:44,095 - distributed.worker - INFO - Worker name: 0 +2025-01-14 15:56:44,097 - distributed.worker - INFO - dashboard at: 127.0.0.1:43277 +2025-01-14 15:56:44,098 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:42323 +2025-01-14 15:56:44,100 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 15:56:44,102 - distributed.worker - INFO - Threads: 1 +2025-01-14 15:56:44,103 - distributed.worker - INFO - Memory: 78.53 GiB +2025-01-14 15:56:44,105 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-x7k3wvdz +2025-01-14 15:56:44,107 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 15:56:44,110 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:46809 +2025-01-14 15:56:44,112 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:46809 +2025-01-14 15:56:44,114 - distributed.worker - INFO - Worker name: 1 +2025-01-14 15:56:44,117 - distributed.worker - INFO - dashboard at: 127.0.0.1:36109 +2025-01-14 15:56:44,119 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:42323 +2025-01-14 15:56:44,121 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 15:56:44,123 - distributed.worker - INFO - Threads: 2 +2025-01-14 15:56:44,125 - distributed.worker - INFO - Memory: 78.53 GiB +2025-01-14 15:56:44,126 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-co2qh9kl +2025-01-14 15:56:44,128 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 15:56:44,193 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:34551 name: 0 +2025-01-14 15:56:44,226 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:34551 +2025-01-14 15:56:44,229 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:57546 +2025-01-14 15:56:44,231 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:46809 name: 1 +2025-01-14 15:56:44,264 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:46809 +2025-01-14 15:56:44,267 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:57562 +2025-01-14 15:56:44,269 - distributed.worker - INFO - Starting Worker plugin shuffle +2025-01-14 15:56:44,274 - distributed.worker - INFO - Starting Worker plugin shuffle +2025-01-14 15:56:44,279 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:42323 +2025-01-14 15:56:44,283 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 15:56:44,287 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:42323 +2025-01-14 15:56:44,289 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 15:56:44,294 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:42323 +2025-01-14 15:56:44,294 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:42323 +2025-01-14 15:56:44,337 - distributed.scheduler - INFO - Receive client connection: Client-ce3e4394-d21a-11ef-a851-8d61c06d7977 +2025-01-14 15:56:44,375 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:57574 +2025-01-14 15:56:44,458 - distributed.scheduler - INFO - Remove client Client-ce3e4394-d21a-11ef-a851-8d61c06d7977 +2025-01-14 15:56:44,462 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:57574; closing. +2025-01-14 15:56:44,462 - distributed.scheduler - INFO - Remove client Client-ce3e4394-d21a-11ef-a851-8d61c06d7977 +2025-01-14 15:56:44,464 - distributed.scheduler - INFO - Close client connection: Client-ce3e4394-d21a-11ef-a851-8d61c06d7977 +2025-01-14 15:56:44,469 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:34551. Reason: worker-close +2025-01-14 15:56:44,472 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:46809. Reason: worker-close +2025-01-14 15:56:44,475 - distributed.worker - INFO - Removing Worker plugin shuffle +2025-01-14 15:56:44,477 - distributed.worker - INFO - Removing Worker plugin shuffle +2025-01-14 15:56:44,481 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:57546; closing. +2025-01-14 15:56:44,481 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:57562; closing. +2025-01-14 15:56:44,482 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:34551 name: 0 (stimulus_id='handle-worker-cleanup-1736819804.4821873') +2025-01-14 15:56:44,485 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:46809 name: 1 (stimulus_id='handle-worker-cleanup-1736819804.485028') +2025-01-14 15:56:44,486 - distributed.scheduler - INFO - Lost all workers +2025-01-14 15:56:44,487 - distributed.core - INFO - Connection to tcp://127.0.0.1:42323 has been closed. +2025-01-14 15:56:44,488 - distributed.core - INFO - Connection to tcp://127.0.0.1:42323 has been closed. +2025-01-14 15:56:44,493 - distributed.scheduler - INFO - Closing scheduler. Reason: unknown +2025-01-14 15:56:44,496 - distributed.scheduler - INFO - Scheduler closing all comms +__________________________ test_nanny_process_failure __________________________ + +c = +s = + + @gen_cluster(nthreads=[], client=True) + async def test_nanny_process_failure(c, s): + async with Nanny(s.address, nthreads=2) as n: + first_dir = n.worker_dir + + assert os.path.exists(first_dir) + + pid = n.pid + assert pid is not None + with suppress(CommClosedError): + await c.run(os._exit, 0, workers=[n.worker_address]) + + # Wait while process dies + await async_poll_for(lambda: n.pid != pid, timeout=5) + # Wait while process comes back + await async_poll_for(lambda: n.is_alive(), timeout=5) + + # assert n.worker_address != original_address # most likely + +> await async_poll_for( + lambda: n.worker_address in s.workers and n.worker_dir is not None, + timeout=5, + ) + +distributed/tests/test_nanny.py:70: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +predicate = . at 0x7f9625ad7880> +timeout = 5, fail_func = None, period = 0.05 + + async def async_poll_for(predicate, timeout, fail_func=None, period=0.05): + deadline = time() + timeout + while not predicate(): + await asyncio.sleep(period) + if time() > deadline: + if fail_func is not None: + fail_func() +> pytest.fail(f"condition not reached until {timeout} seconds") +E Failed: condition not reached until 5 seconds + +distributed/utils_test.py:1232: Failed +----------------------------- Captured stderr call ----------------------------- +2025-01-14 16:02:07,345 - distributed.scheduler - INFO - State start +2025-01-14 16:02:07,370 - distributed.scheduler - INFO - Scheduler at: tcp://127.0.0.1:44287 +2025-01-14 16:02:07,382 - distributed.scheduler - INFO - dashboard at: http://127.0.0.1:34747/status +2025-01-14 16:02:07,388 - distributed.scheduler - INFO - Registering Worker plugin shuffle +2025-01-14 16:02:07,461 - distributed.scheduler - INFO - Receive client connection: Client-8ed378b4-d21b-11ef-a851-8d61c06d7977 +2025-01-14 16:02:07,525 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:56064 +2025-01-14 16:02:07,556 - distributed.nanny - INFO - Start Nanny at: 'tcp://127.0.0.1:41045' +2025-01-14 16:02:11,218 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:37273 +2025-01-14 16:02:11,218 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:37273 +2025-01-14 16:02:11,218 - distributed.worker - INFO - dashboard at: 127.0.0.1:46161 +2025-01-14 16:02:11,218 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:44287 +2025-01-14 16:02:11,218 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 16:02:11,218 - distributed.worker - INFO - Threads: 2 +2025-01-14 16:02:11,218 - distributed.worker - INFO - Memory: 7.85 GiB +2025-01-14 16:02:11,218 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-itsp2szm +2025-01-14 16:02:11,218 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 16:02:13,801 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:37273 name: tcp://127.0.0.1:37273 +2025-01-14 16:02:13,941 - distributed.worker - INFO - Starting Worker plugin shuffle +2025-01-14 16:02:13,948 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:44287 +2025-01-14 16:02:13,948 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 16:02:13,938 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:37273 +2025-01-14 16:02:13,953 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:56088 +2025-01-14 16:02:13,961 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:44287 +2025-01-14 16:02:14,038 - distributed.worker - INFO - Run out-of-band function '_exit' +2025-01-14 16:02:14,132 - distributed.scheduler - ERROR - broadcast to tcp://127.0.0.1:37273 failed: CommClosedError: in : Stream is closed +2025-01-14 16:02:14,136 - distributed.core - INFO - Connection to tcp://127.0.0.1:56088 has been closed. +2025-01-14 16:02:14,136 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:37273 name: tcp://127.0.0.1:37273 (stimulus_id='handle-worker-cleanup-1736820134.136388') +2025-01-14 16:02:14,138 - distributed.scheduler - INFO - Lost all workers +2025-01-14 16:02:14,155 - distributed.nanny - WARNING - Restarting worker +2025-01-14 16:02:18,127 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:39209 +2025-01-14 16:02:18,196 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:39209 +2025-01-14 16:02:18,196 - distributed.worker - INFO - dashboard at: 127.0.0.1:44039 +2025-01-14 16:02:18,196 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:44287 +2025-01-14 16:02:18,196 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 16:02:18,196 - distributed.worker - INFO - Threads: 2 +2025-01-14 16:02:18,197 - distributed.worker - INFO - Memory: 7.85 GiB +2025-01-14 16:02:18,197 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-dc4yvmaa +2025-01-14 16:02:18,197 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 16:02:19,476 - distributed.nanny - INFO - Closing Nanny at 'tcp://127.0.0.1:41045'. Reason: nanny-close +2025-01-14 16:02:19,841 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:39209 name: tcp://127.0.0.1:39209 +2025-01-14 16:02:19,930 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:39209 +2025-01-14 16:02:19,941 - distributed.worker - INFO - Starting Worker plugin shuffle +2025-01-14 16:02:19,941 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:58012 +2025-01-14 16:02:19,942 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:44287 +2025-01-14 16:02:19,943 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 16:02:19,969 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:44287 +2025-01-14 16:02:19,996 - distributed.nanny - INFO - Nanny asking worker to close. Reason: nanny-close +2025-01-14 16:02:20,081 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:39209. Reason: nanny-close +2025-01-14 16:02:20,081 - distributed.worker - INFO - Removing Worker plugin shuffle +2025-01-14 16:02:20,084 - distributed.core - INFO - Connection to tcp://127.0.0.1:44287 has been closed. +2025-01-14 16:02:20,084 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:58012; closing. +2025-01-14 16:02:20,085 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:39209 name: tcp://127.0.0.1:39209 (stimulus_id='handle-worker-cleanup-1736820140.0854642') +2025-01-14 16:02:20,093 - distributed.scheduler - INFO - Lost all workers +2025-01-14 16:02:20,105 - distributed.nanny - INFO - Worker closed +2025-01-14 16:02:20,773 - distributed.nanny - INFO - Nanny at 'tcp://127.0.0.1:41045' closed. +2025-01-14 16:02:20,781 - distributed.scheduler - INFO - Remove client Client-8ed378b4-d21b-11ef-a851-8d61c06d7977 +2025-01-14 16:02:20,794 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:56064; closing. +2025-01-14 16:02:20,795 - distributed.scheduler - INFO - Remove client Client-8ed378b4-d21b-11ef-a851-8d61c06d7977 +2025-01-14 16:02:20,814 - distributed.scheduler - INFO - Close client connection: Client-8ed378b4-d21b-11ef-a851-8d61c06d7977 +2025-01-14 16:02:20,824 - distributed.scheduler - INFO - Closing scheduler. Reason: unknown +2025-01-14 16:02:20,833 - distributed.scheduler - INFO - Scheduler closing all comms +____________________ test_tell_workers_when_peers_have_left ____________________ + +c = +s = +a = +b = + + @pytest.mark.skipif( + sys.version_info < (3, 11), + reason="asyncio.wait_for is unreliable on 3.10 and below", + ) + @gen_cluster( + client=True, + config={ + # In this test we want to make sure that the connections are severed + # before the timeout hits. Therefore, the connection timeout should be + # higher than the test timeout. + # At the time of writing, the test timeout was 30s + "distributed.comm.timeouts.connect": "120s" + }, + ) + async def test_tell_workers_when_peers_have_left(c, s, a, b): + f = (await c.scatter({"f": 1}, workers=[a.address, b.address], broadcast=True))["f"] + + workers = {a.address: a, b.address: b} + connect_timeout = parse_timedelta( + dask.config.get("distributed.comm.timeouts.connect"), default="seconds" + ) + + class BrokenGatherDep(Worker): + async def gather_dep(self, worker, *args, **kwargs): + w = workers.pop(worker, None) + if w is not None and workers: + w.listener.stop() + s.stream_comms[worker].abort() + + return await super().gather_dep(worker, *args, **kwargs) + + async with BrokenGatherDep(s.address, nthreads=1) as w3: + start = time() +> g = await c.submit(inc, f, key="g", workers=[w3.address]) + +distributed/tests/test_scheduler.py:4870: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = , raiseit = True + + async def _result(self, raiseit=True): + await self._state.wait() + if self.status == "error": + exc = clean_exception(self._state.exception, self._state.traceback) + if raiseit: + typ, exc, tb = exc + raise exc.with_traceback(tb) + else: + return exc + elif self.cancelled(): + assert self._state + exception = self._state.exception + assert isinstance(exception, CancelledError) + if raiseit: +> raise exception +E distributed.client.FutureCancelledError: g cancelled for reason: unknown. + +distributed/client.py:418: FutureCancelledError +----------------------------- Captured stderr call ----------------------------- +2025-01-14 16:11:10,412 - distributed.scheduler - INFO - State start +2025-01-14 16:11:10,424 - distributed.scheduler - INFO - Scheduler at: tcp://127.0.0.1:46261 +2025-01-14 16:11:10,431 - distributed.scheduler - INFO - dashboard at: http://127.0.0.1:40311/status +2025-01-14 16:11:10,436 - distributed.scheduler - INFO - Registering Worker plugin shuffle +2025-01-14 16:11:10,457 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:39331 +2025-01-14 16:11:10,463 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:39331 +2025-01-14 16:11:10,467 - distributed.worker - INFO - Worker name: 0 +2025-01-14 16:11:10,473 - distributed.worker - INFO - dashboard at: 127.0.0.1:41277 +2025-01-14 16:11:10,477 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:46261 +2025-01-14 16:11:10,481 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 16:11:10,485 - distributed.worker - INFO - Threads: 1 +2025-01-14 16:11:10,490 - distributed.worker - INFO - Memory: 78.53 GiB +2025-01-14 16:11:10,495 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-7zj26133 +2025-01-14 16:11:10,499 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 16:11:10,506 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:42207 +2025-01-14 16:11:10,510 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:42207 +2025-01-14 16:11:10,515 - distributed.worker - INFO - Worker name: 1 +2025-01-14 16:11:10,519 - distributed.worker - INFO - dashboard at: 127.0.0.1:42031 +2025-01-14 16:11:10,524 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:46261 +2025-01-14 16:11:10,528 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 16:11:10,533 - distributed.worker - INFO - Threads: 2 +2025-01-14 16:11:10,536 - distributed.worker - INFO - Memory: 78.53 GiB +2025-01-14 16:11:10,539 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-tydmj67j +2025-01-14 16:11:10,542 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 16:11:10,915 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:39331 name: 0 +2025-01-14 16:11:11,497 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:39331 +2025-01-14 16:11:11,501 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:45232 +2025-01-14 16:11:11,502 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:42207 name: 1 +2025-01-14 16:11:11,562 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:42207 +2025-01-14 16:11:11,566 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:45248 +2025-01-14 16:11:11,570 - distributed.worker - INFO - Starting Worker plugin shuffle +2025-01-14 16:11:11,575 - distributed.worker - INFO - Starting Worker plugin shuffle +2025-01-14 16:11:11,580 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:46261 +2025-01-14 16:11:11,586 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 16:11:11,592 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:46261 +2025-01-14 16:11:11,596 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 16:11:11,601 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:46261 +2025-01-14 16:11:11,601 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:46261 +2025-01-14 16:11:11,662 - distributed.scheduler - INFO - Receive client connection: Client-d332e619-d21c-11ef-a851-8d61c06d7977 +2025-01-14 16:11:11,941 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:45258 +2025-01-14 16:11:11,982 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:46411 +2025-01-14 16:11:11,998 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:46411 +2025-01-14 16:11:12,002 - distributed.worker - INFO - dashboard at: 127.0.0.1:34675 +2025-01-14 16:11:12,005 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:46261 +2025-01-14 16:11:12,009 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 16:11:12,014 - distributed.worker - INFO - Threads: 1 +2025-01-14 16:11:12,018 - distributed.worker - INFO - Memory: 3.93 GiB +2025-01-14 16:11:12,031 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/brokengatherdep-wlfy4bea +2025-01-14 16:11:12,037 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 16:11:12,099 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:46411 name: tcp://127.0.0.1:46411 +2025-01-14 16:11:12,415 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:46411 +2025-01-14 16:11:12,620 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:48028 +2025-01-14 16:11:12,639 - distributed.worker - INFO - Starting Worker plugin shuffle +2025-01-14 16:11:12,658 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:46261 +2025-01-14 16:11:12,670 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 16:11:12,693 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:46261 +2025-01-14 16:11:12,717 - distributed.core - INFO - Connection to tcp://127.0.0.1:45248 has been closed. +2025-01-14 16:11:12,796 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:42207 name: 1 (stimulus_id='handle-worker-cleanup-1736820672.7962487') +2025-01-14 16:11:12,800 - distributed.scheduler - ERROR - Removing worker 'tcp://127.0.0.1:42207' caused the cluster to lose scattered data, which can't be recovered: {'f'} (stimulus_id='handle-worker-cleanup-1736820672.7962487') +2025-01-14 16:11:12,812 - distributed.core - INFO - Connection to tcp://127.0.0.1:46261 has been closed. +2025-01-14 16:11:12,928 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:42207. Reason: worker-handle-scheduler-connection-broken +2025-01-14 16:11:12,942 - distributed.worker - INFO - Removing Worker plugin shuffle +2025-01-14 16:11:12,957 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:46411. Reason: worker-close +2025-01-14 16:11:12,970 - distributed.worker.state_machine - WARNING - Async instruction for .BrokenGatherDep.gather_dep() done, defined at /build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.13_distributed/build/distributed/worker_state_machine.py:3606>> ended with CancelledError +2025-01-14 16:11:12,977 - distributed.worker - INFO - Removing Worker plugin shuffle +2025-01-14 16:11:12,987 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:48028; closing. +2025-01-14 16:11:12,987 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:46411 name: tcp://127.0.0.1:46411 (stimulus_id='handle-worker-cleanup-1736820672.9875455') +2025-01-14 16:11:12,993 - distributed.core - INFO - Connection to tcp://127.0.0.1:46261 has been closed. +2025-01-14 16:11:12,999 - distributed.scheduler - INFO - Remove client Client-d332e619-d21c-11ef-a851-8d61c06d7977 +2025-01-14 16:11:13,010 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:45258; closing. +2025-01-14 16:11:13,011 - distributed.scheduler - INFO - Remove client Client-d332e619-d21c-11ef-a851-8d61c06d7977 +2025-01-14 16:11:13,019 - distributed.scheduler - INFO - Close client connection: Client-d332e619-d21c-11ef-a851-8d61c06d7977 +2025-01-14 16:11:13,026 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:39331. Reason: worker-close +2025-01-14 16:11:13,043 - distributed.worker - INFO - Removing Worker plugin shuffle +2025-01-14 16:11:13,063 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:45232; closing. +2025-01-14 16:11:13,064 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:39331 name: 0 (stimulus_id='handle-worker-cleanup-1736820673.0642374') +2025-01-14 16:11:13,077 - distributed.scheduler - INFO - Lost all workers +2025-01-14 16:11:13,090 - distributed.core - INFO - Connection to tcp://127.0.0.1:46261 has been closed. +2025-01-14 16:11:13,105 - distributed.scheduler - INFO - Closing scheduler. Reason: unknown +2025-01-14 16:11:13,110 - distributed.scheduler - INFO - Scheduler closing all comms +____________________________ test_log_remove_worker ____________________________ + +c = +s = +a = +b = + + @gen_cluster(client=True) + async def test_log_remove_worker(c, s, a, b): + # Computed task + x = c.submit(inc, 1, key="x", workers=a.address) + await x + ev = Event() + # Processing task + y = c.submit( + lambda ev: ev.wait(), ev, key="y", workers=a.address, allow_other_workers=True + ) + await wait_for_state("y", "processing", s) + # Scattered task + z = await c.scatter({"z": 3}, workers=a.address) + + s._broker.truncate() + + with captured_logger("distributed.scheduler", level=logging.INFO) as log: + # Successful graceful shutdown + await s.retire_workers([a.address], stimulus_id="graceful") + # Refuse to retire gracefully as there's nowhere to put x and z + await s.retire_workers([b.address], stimulus_id="graceful_abort") + await asyncio.sleep(0.2) + # Ungraceful shutdown + await s.remove_worker(b.address, stimulus_id="ungraceful") + await asyncio.sleep(0.2) + await ev.set() + + assert log.getvalue().splitlines() == [ + # Successful graceful + f"Retire worker addresses (stimulus_id='graceful') ['{a.address}']", + f"Remove worker addr: {a.address} name: {a.name} (stimulus_id='graceful')", + f"Retired worker '{a.address}' (stimulus_id='graceful')", + # Aborted graceful + f"Retire worker addresses (stimulus_id='graceful_abort') ['{b.address}']", + f"Could not retire worker '{b.address}': unique data could not be " + "moved to any other worker (stimulus_id='graceful_abort')", + # Ungraceful + f"Remove worker addr: {b.address} name: {b.name} (stimulus_id='ungraceful')", + f"Removing worker '{b.address}' caused the cluster to lose already " + "computed task(s), which will be recomputed elsewhere: {'x'} " + "(stimulus_id='ungraceful')", + f"Removing worker '{b.address}' caused the cluster to lose scattered " + "data, which can't be recovered: {'z'} (stimulus_id='ungraceful')", + "Lost all workers", + ] + + events = {topic: [ev for _, ev in evs] for topic, evs in s.get_events().items()} + for evs in events.values(): + for ev in evs: + if ev.get("action", None) == "retire-workers": + for k in ("retired", "could-not-retire"): + ev[k] = {addr: "snip" for addr in ev[k]} + if "stimulus_id" in ev: # Strip timestamp + ev["stimulus_id"] = ev["stimulus_id"].rsplit("-", 1)[0] + +> assert events == { + a.address: [ + { + "action": "worker-status-change", + "prev-status": "running", + "status": "closing_gracefully", + "stimulus_id": "graceful", + }, + { + "action": "remove-worker", + "lost-computed-tasks": set(), + "lost-scattered-tasks": set(), + "processing-tasks": {"y"}, + "expected": True, + "stimulus_id": "graceful", + }, + {"action": "retired", "stimulus_id": "graceful"}, + ], + b.address: [ + { + "action": "worker-status-change", + "prev-status": "running", + "status": "closing_gracefully", + "stimulus_id": "graceful_abort", + }, + {"action": "could-not-retire", "stimulus_id": "graceful_abort"}, + { + "action": "worker-status-change", + "prev-status": "closing_gracefully", + "status": "running", + "stimulus_id": "worker-status-change", + }, + { + "action": "remove-worker", + "lost-computed-tasks": {"x"}, + "lost-scattered-tasks": {"z"}, + "processing-tasks": {"y"}, + "expected": False, + "stimulus_id": "ungraceful", + }, + {"action": "closing-worker", "reason": "scheduler-remove-worker"}, + ], + "all": [ + { + "action": "remove-worker", + "lost-computed-tasks": set(), + "lost-scattered-tasks": set(), + "processing-tasks": {"y"}, + "expected": True, + "stimulus_id": "graceful", + "worker": a.address, + }, + { + "action": "retire-workers", + "stimulus_id": "graceful", + "retired": {a.address: "snip"}, + "could-not-retire": {}, + }, + { + "action": "retire-workers", + "stimulus_id": "graceful_abort", + "retired": {}, + "could-not-retire": {b.address: "snip"}, + }, + { + "action": "remove-worker", + "lost-computed-tasks": {"x"}, + "lost-scattered-tasks": {"z"}, + "processing-tasks": {"y"}, + "expected": False, + "stimulus_id": "ungraceful", + "worker": b.address, + }, + ], + "worker-get-client": [{"client": c.id, "timeout": 5, "worker": b.address}], + } +E AssertionError: assert {'tcp://127.0.0.1:45689': [{'action': 'worker-status-change', 'prev-status': 'running', 'status': 'closing_gracefully', 'stimulus_id': 'graceful'}, {'action': 'remove-worker', 'processing-tasks': {'y'}, 'lost-computed-tasks': set(), 'lost-scattered-tasks': set(), 'stimulus_id': 'graceful', 'expected': True}, {'action': 'retired', 'stimulus_id': 'graceful'}], 'all': [{'action': 'remove-worker', 'processing-tasks': {'y'}, 'lost-computed-tasks': set(), 'lost-scattered-tasks': set(), 'stimulus_id': 'graceful', 'expected': True, 'worker': 'tcp://127.0.0.1:45689'}, {'action': 'retire-workers', 'retired': {'tcp://127.0.0.1:45689': 'snip'}, 'could-not-retire': {}, 'stimulus_id': 'graceful'}, {'action': 'retire-workers', 'retired': {}, 'could-not-retire': {'tcp://127.0.0.1:39417': 'snip'}, 'stimulus_id': 'graceful_abort'}, {'action': 'remove-worker', 'processing-tasks': {'y'}, 'lost-computed-tasks': {'x'}, 'lost-scattered-tasks': {'z'}, 'stimulus_id': 'ungraceful', 'expected': False, 'worker': 'tcp://127.0.0.1:39417'}], 'tcp://127.0.0.1:39417': [{'action': 'worker-status-change', 'prev-status': 'running', 'status': 'closing_gracefully', 'stimulus_id': 'graceful_abort'}, {'action': 'could-not-retire', 'stimulus_id': 'graceful_abort'}, {'action': 'worker-status-change', 'prev-status': 'closing_gracefully', 'status': 'running', 'stimulus_id': 'worker-status-change'}, {'action': 'remove-worker', 'processing-tasks': {'y'}, 'lost-computed-tasks': {'x'}, 'lost-scattered-tasks': {'z'}, 'stimulus_id': 'ungraceful', 'expected': False}, {'action': 'closing-worker', 'reason': 'scheduler-remove-worker'}], 'worker-get-client': [{'client': 'Client-22871129-d21e-11ef-a851-8d61c06d7977', 'timeout': 5, 'worker': 'tcp://127.0.0.1:45689'}, {'client': 'Client-22871129-d21e-11ef-a851-8d61c06d7977', 'timeout': 5, 'worker': 'tcp://127.0.0.1:39417'}]} == {'tcp://127.0.0.1:45689': [{'action': 'worker-status-change', 'prev-status': 'running', 'status': 'closing_gracefully', 'stimulus_id': 'graceful'}, {'action': 'remove-worker', 'lost-computed-tasks': set(), 'lost-scattered-tasks': set(), 'processing-tasks': {'y'}, 'expected': True, 'stimulus_id': 'graceful'}, {'action': 'retired', 'stimulus_id': 'graceful'}], 'tcp://127.0.0.1:39417': [{'action': 'worker-status-change', 'prev-status': 'running', 'status': 'closing_gracefully', 'stimulus_id': 'graceful_abort'}, {'action': 'could-not-retire', 'stimulus_id': 'graceful_abort'}, {'action': 'worker-status-change', 'prev-status': 'closing_gracefully', 'status': 'running', 'stimulus_id': 'worker-status-change'}, {'action': 'remove-worker', 'lost-computed-tasks': {'x'}, 'lost-scattered-tasks': {'z'}, 'processing-tasks': {'y'}, 'expected': False, 'stimulus_id': 'ungraceful'}, {'action': 'closing-worker', 'reason': 'scheduler-remove-worker'}], 'all': [{'action': 'remove-worker', 'lost-computed-tasks': set(), 'lost-scattered-tasks': set(), 'processing-tasks': {'y'}, 'expected': True, 'stimulus_id': 'graceful', 'worker': 'tcp://127.0.0.1:45689'}, {'action': 'retire-workers', 'stimulus_id': 'graceful', 'retired': {'tcp://127.0.0.1:45689': 'snip'}, 'could-not-retire': {}}, {'action': 'retire-workers', 'stimulus_id': 'graceful_abort', 'retired': {}, 'could-not-retire': {'tcp://127.0.0.1:39417': 'snip'}}, {'action': 'remove-worker', 'lost-computed-tasks': {'x'}, 'lost-scattered-tasks': {'z'}, 'processing-tasks': {'y'}, 'expected': False, 'stimulus_id': 'ungraceful', 'worker': 'tcp://127.0.0.1:39417'}], 'worker-get-client': [{'client': 'Client-22871129-d21e-11ef-a851-8d61c06d7977', 'timeout': 5, 'worker': 'tcp://127.0.0.1:39417'}]} +E +E Common items: +E {'all': [{'action': 'remove-worker', +E 'expected': True, +E 'lost-computed-tasks': set(), +E 'lost-scattered-tasks': set(), +E 'processing-tasks': {'y'}, +E 'stimulus_id': 'graceful', +E 'worker': 'tcp://127.0.0.1:45689'}, +E {'action': 'retire-workers', +E 'could-not-retire': {}, +E 'retired': {'tcp://127.0.0.1:45689': 'snip'}, +E 'stimulus_id': 'graceful'}, +E {'action': 'retire-workers', +E 'could-not-retire': {'tcp://127.0.0.1:39417': 'snip'}, +E 'retired': {}, +E 'stimulus_id': 'graceful_abort'}, +E {'action': 'remove-worker', +E 'expected': False, +E 'lost-computed-tasks': {'x'}, +E 'lost-scattered-tasks': {'z'}, +E 'processing-tasks': {'y'}, +E 'stimulus_id': 'ungraceful', +E 'worker': 'tcp://127.0.0.1:39417'}], +E 'tcp://127.0.0.1:39417': [{'action': 'worker-status-change', +E 'prev-status': 'running', +E 'status': 'closing_gracefully', +E 'stimulus_id': 'graceful_abort'}, +E {'action': 'could-not-retire', +E 'stimulus_id': 'graceful_abort'}, +E {'action': 'worker-status-change', +E 'prev-status': 'closing_gracefully', +E 'status': 'running', +E 'stimulus_id': 'worker-status-change'}, +E {'action': 'remove-worker', +E 'expected': False, +E 'lost-computed-tasks': {'x'}, +E 'lost-scattered-tasks': {'z'}, +E 'processing-tasks': {'y'}, +E 'stimulus_id': 'ungraceful'}, +E {'action': 'closing-worker', +E 'reason': 'scheduler-remove-worker'}], +E 'tcp://127.0.0.1:45689': [{'action': 'worker-status-change', +E 'prev-status': 'running', +E 'status': 'closing_gracefully', +E 'stimulus_id': 'graceful'}, +E {'action': 'remove-worker', +E 'expected': True, +E 'lost-computed-tasks': set(), +E 'lost-scattered-tasks': set(), +E 'processing-tasks': {'y'}, +E 'stimulus_id': 'graceful'}, +E {'action': 'retired', 'stimulus_id': 'graceful'}]} +E Differing items: +E {'worker-get-client': [{'client': 'Client-22871129-d21e-11ef-a851-8d61c06d7977', 'timeout': 5, 'worker': 'tcp://127.0.0.1:45689'}, {'client': 'Client-22871129-d21e-11ef-a851-8d61c06d7977', 'timeout': 5, 'worker': 'tcp://127.0.0.1:39417'}]} != {'worker-get-client': [{'client': 'Client-22871129-d21e-11ef-a851-8d61c06d7977', 'timeout': 5, 'worker': 'tcp://127.0.0.1:39417'}]} +E +E Full diff: +E { +E 'all': [ +E { +E 'action': 'remove-worker', +E 'expected': True, +E 'lost-computed-tasks': set(), +E 'lost-scattered-tasks': set(), +E 'processing-tasks': { +E 'y', +E }, +E 'stimulus_id': 'graceful', +E 'worker': 'tcp://127.0.0.1:45689', +E }, +E { +E 'action': 'retire-workers', +E 'could-not-retire': {}, +E 'retired': { +E 'tcp://127.0.0.1:45689': 'snip', +E }, +E 'stimulus_id': 'graceful', +E }, +E { +E 'action': 'retire-workers', +E 'could-not-retire': { +E 'tcp://127.0.0.1:39417': 'snip', +E }, +E 'retired': {}, +E 'stimulus_id': 'graceful_abort', +E }, +E { +E 'action': 'remove-worker', +E 'expected': False, +E 'lost-computed-tasks': { +E 'x', +E }, +E 'lost-scattered-tasks': { +E 'z', +E }, +E 'processing-tasks': { +E 'y', +E }, +E 'stimulus_id': 'ungraceful', +E 'worker': 'tcp://127.0.0.1:39417', +E }, +E ], +E 'tcp://127.0.0.1:39417': [ +E { +E 'action': 'worker-status-change', +E 'prev-status': 'running', +E 'status': 'closing_gracefully', +E 'stimulus_id': 'graceful_abort', +E }, +E { +E 'action': 'could-not-retire', +E 'stimulus_id': 'graceful_abort', +E }, +E { +E 'action': 'worker-status-change', +E 'prev-status': 'closing_gracefully', +E 'status': 'running', +E 'stimulus_id': 'worker-status-change', +E }, +E { +E 'action': 'remove-worker', +E 'expected': False, +E 'lost-computed-tasks': { +E 'x', +E }, +E 'lost-scattered-tasks': { +E 'z', +E }, +E 'processing-tasks': { +E 'y', +E }, +E 'stimulus_id': 'ungraceful', +E }, +E { +E 'action': 'closing-worker', +E 'reason': 'scheduler-remove-worker', +E }, +E ], +E 'tcp://127.0.0.1:45689': [ +E { +E 'action': 'worker-status-change', +E 'prev-status': 'running', +E 'status': 'closing_gracefully', +E 'stimulus_id': 'graceful', +E }, +E { +E 'action': 'remove-worker', +E 'expected': True, +E 'lost-computed-tasks': set(), +E 'lost-scattered-tasks': set(), +E 'processing-tasks': { +E 'y', +E }, +E 'stimulus_id': 'graceful', +E }, +E { +E 'action': 'retired', +E 'stimulus_id': 'graceful', +E }, +E ], +E 'worker-get-client': [ +E { +E 'client': 'Client-22871129-d21e-11ef-a851-8d61c06d7977', +E 'timeout': 5, +E + 'worker': 'tcp://127.0.0.1:45689', +E + }, +E + { +E + 'client': 'Client-22871129-d21e-11ef-a851-8d61c06d7977', +E + 'timeout': 5, +E 'worker': 'tcp://127.0.0.1:39417', +E }, +E ], +E } + +distributed/tests/test_worker.py:3016: AssertionError +----------------------------- Captured stderr call ----------------------------- +2025-01-14 16:20:33,135 - distributed.scheduler - INFO - State start +2025-01-14 16:20:33,179 - distributed.scheduler - INFO - Scheduler at: tcp://127.0.0.1:37373 +2025-01-14 16:20:33,197 - distributed.scheduler - INFO - dashboard at: http://127.0.0.1:36711/status +2025-01-14 16:20:33,214 - distributed.scheduler - INFO - Registering Worker plugin shuffle +2025-01-14 16:20:33,292 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:45689 +2025-01-14 16:20:33,310 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:45689 +2025-01-14 16:20:33,327 - distributed.worker - INFO - Worker name: 0 +2025-01-14 16:20:33,354 - distributed.worker - INFO - dashboard at: 127.0.0.1:42841 +2025-01-14 16:20:33,373 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:37373 +2025-01-14 16:20:33,391 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 16:20:33,418 - distributed.worker - INFO - Threads: 1 +2025-01-14 16:20:33,436 - distributed.worker - INFO - Memory: 78.53 GiB +2025-01-14 16:20:33,454 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-vza_25w3 +2025-01-14 16:20:33,480 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 16:20:33,498 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:39417 +2025-01-14 16:20:33,515 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:39417 +2025-01-14 16:20:33,531 - distributed.worker - INFO - Worker name: 1 +2025-01-14 16:20:33,543 - distributed.worker - INFO - dashboard at: 127.0.0.1:45849 +2025-01-14 16:20:33,562 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:37373 +2025-01-14 16:20:33,586 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 16:20:33,608 - distributed.worker - INFO - Threads: 2 +2025-01-14 16:20:33,627 - distributed.worker - INFO - Memory: 78.53 GiB +2025-01-14 16:20:33,650 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-xvzv6lg7 +2025-01-14 16:20:33,677 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 16:20:33,961 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:45689 name: 0 +2025-01-14 16:20:34,045 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:45689 +2025-01-14 16:20:34,054 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:43756 +2025-01-14 16:20:34,059 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:39417 name: 1 +2025-01-14 16:20:34,131 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:39417 +2025-01-14 16:20:34,138 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:43764 +2025-01-14 16:20:34,140 - distributed.worker - INFO - Starting Worker plugin shuffle +2025-01-14 16:20:34,148 - distributed.worker - INFO - Starting Worker plugin shuffle +2025-01-14 16:20:34,156 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:37373 +2025-01-14 16:20:34,164 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 16:20:34,172 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:37373 +2025-01-14 16:20:34,179 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 16:20:34,189 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:37373 +2025-01-14 16:20:34,190 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:37373 +2025-01-14 16:20:34,288 - distributed.scheduler - INFO - Receive client connection: Client-22871129-d21e-11ef-a851-8d61c06d7977 +2025-01-14 16:20:34,469 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:43772 +2025-01-14 16:20:34,570 - distributed.scheduler - INFO - Retire worker addresses (stimulus_id='graceful') ['tcp://127.0.0.1:45689'] +2025-01-14 16:20:34,571 - distributed.active_memory_manager - INFO - Retiring worker tcp://127.0.0.1:45689; 2 keys are being moved away. +2025-01-14 16:20:34,624 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:45689 name: 0 (stimulus_id='graceful') +2025-01-14 16:20:34,626 - distributed.scheduler - INFO - Retired worker 'tcp://127.0.0.1:45689' (stimulus_id='graceful') +2025-01-14 16:20:34,632 - distributed.scheduler - INFO - Retire worker addresses (stimulus_id='graceful_abort') ['tcp://127.0.0.1:39417'] +2025-01-14 16:20:34,633 - distributed.active_memory_manager - WARNING - Tried retiring worker tcp://127.0.0.1:39417, but 2 tasks could not be moved as there are no suitable workers to receive them. The worker will not be retired. +2025-01-14 16:20:34,634 - distributed.scheduler - WARNING - Could not retire worker 'tcp://127.0.0.1:39417': unique data could not be moved to any other worker (stimulus_id='graceful_abort') +2025-01-14 16:20:34,844 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:39417 name: 1 (stimulus_id='ungraceful') +2025-01-14 16:20:34,845 - distributed.scheduler - WARNING - Removing worker 'tcp://127.0.0.1:39417' caused the cluster to lose already computed task(s), which will be recomputed elsewhere: {'x'} (stimulus_id='ungraceful') +2025-01-14 16:20:34,845 - distributed.scheduler - ERROR - Removing worker 'tcp://127.0.0.1:39417' caused the cluster to lose scattered data, which can't be recovered: {'z'} (stimulus_id='ungraceful') +2025-01-14 16:20:34,846 - distributed.scheduler - INFO - Lost all workers +2025-01-14 16:20:34,857 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:39417. Reason: scheduler-remove-worker +2025-01-14 16:20:34,879 - distributed.worker.state_machine - WARNING - Async instruction for > ended with CancelledError +2025-01-14 16:20:34,895 - distributed.worker - INFO - Removing Worker plugin shuffle +2025-01-14 16:20:34,905 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:43764; closing. +2025-01-14 16:20:35,127 - distributed.scheduler - INFO - Remove client Client-22871129-d21e-11ef-a851-8d61c06d7977 +2025-01-14 16:20:35,174 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:43772; closing. +2025-01-14 16:20:35,175 - distributed.scheduler - INFO - Remove client Client-22871129-d21e-11ef-a851-8d61c06d7977 +2025-01-14 16:20:35,250 - distributed.core - INFO - Connection to tcp://127.0.0.1:37373 has been closed. +2025-01-14 16:20:35,252 - distributed.scheduler - INFO - Close client connection: Client-22871129-d21e-11ef-a851-8d61c06d7977 +2025-01-14 16:20:35,292 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:45689. Reason: worker-close +2025-01-14 16:20:35,316 - distributed.worker - INFO - Removing Worker plugin shuffle +2025-01-14 16:20:35,349 - distributed.scheduler - WARNING - Received heartbeat from unregistered worker 'tcp://127.0.0.1:45689'. +2025-01-14 16:20:35,376 - distributed.worker - ERROR - Failed to communicate with scheduler during heartbeat. +Traceback (most recent call last): + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.13_distributed/build/distributed/comm/tcp.py", line 225, in read + frames_nosplit_nbytes_bin = await stream.read_bytes(fmt_size) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +tornado.iostream.StreamClosedError: Stream is closed + +The above exception was the direct cause of the following exception: + +Traceback (most recent call last): + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.13_distributed/build/distributed/worker.py", line 1269, in heartbeat + response = await retry_operation( + ^^^^^^^^^^^^^^^^^^^^^^ + ...<14 lines>... + ) + ^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.13_distributed/build/distributed/utils_comm.py", line 441, in retry_operation + return await retry( + ^^^^^^^^^^^^ + ...<5 lines>... + ) + ^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.13_distributed/build/distributed/utils_comm.py", line 420, in retry + return await coro() + ^^^^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.13_distributed/build/distributed/core.py", line 1259, in send_recv_from_rpc + return await send_recv(comm=comm, op=key, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.13_distributed/build/distributed/core.py", line 1018, in send_recv + response = await comm.read(deserializers=deserializers) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.13_distributed/build/distributed/comm/tcp.py", line 236, in read + convert_stream_closed_error(self, e) + ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.13_distributed/build/distributed/comm/tcp.py", line 142, in convert_stream_closed_error + raise CommClosedError(f"in {obj}: {exc}") from exc +distributed.comm.core.CommClosedError: in : Stream is closed +2025-01-14 16:20:35,410 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:43756; closing. +2025-01-14 16:20:35,411 - distributed.core - INFO - Connection to tcp://127.0.0.1:37373 has been closed. +2025-01-14 16:20:35,434 - distributed.scheduler - INFO - Closing scheduler. Reason: unknown +2025-01-14 16:20:35,452 - distributed.scheduler - INFO - Scheduler closing all comms +__________________________ test_fail_to_pickle_spill ___________________________ + +fut = , timeout = 5 + + async def wait_for(fut: Awaitable[T], timeout: float) -> T: + async with asyncio.timeout(timeout): +> return await fut + +distributed/utils.py:1914: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = Scheduler local=tcp://127.0.0.1:33642 remote=tcp://127.0.0.1:46261> +deserializers = None + + async def read(self, deserializers=None): + stream = self.stream + if stream is None: + raise CommClosedError() + + fmt = "Q" + fmt_size = struct.calcsize(fmt) + + try: + # Don't store multiple numpy or parquet buffers into the same buffer, or + # none will be released until all are released. +> frames_nosplit_nbytes_bin = await stream.read_bytes(fmt_size) +E asyncio.exceptions.CancelledError + +distributed/comm/tcp.py:225: CancelledError + +The above exception was the direct cause of the following exception: + +args = (), kwds = {} + + @wraps(func) + def inner(*args, **kwds): + with self._recreate_cm(): +> return func(*args, **kwds) + +/usr/lib/python3.13/contextlib.py:85: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +/usr/lib/python3.13/contextlib.py:85: in inner + return func(*args, **kwds) +distributed/utils_test.py:1090: in test_func + return _run_and_close_tornado(async_fn_outer) +distributed/utils_test.py:380: in _run_and_close_tornado + return asyncio_run(inner_fn(), loop_factory=get_loop_factory()) +/usr/lib/python3.13/asyncio/runners.py:195: in run + return runner.run(main) +/usr/lib/python3.13/asyncio/runners.py:118: in run + return self._loop.run_until_complete(task) +/usr/lib/python3.13/asyncio/base_events.py:720: in run_until_complete + return future.result() +distributed/utils_test.py:377: in inner_fn + return await async_fn(*args, **kwargs) +distributed/utils_test.py:1087: in async_fn_outer + return await utils_wait_for(async_fn(), timeout=timeout * 2) +distributed/utils.py:1914: in wait_for + return await fut +distributed/utils_test.py:1008: in async_fn + _client_factory(s) as c, +/usr/lib/python3.13/contextlib.py:214: in __aenter__ + return await anext(self.gen) +distributed/utils_test.py:957: in _client_factory + async with Client( +distributed/client.py:1700: in __aenter__ + await self +distributed/client.py:1512: in _start + await self._ensure_connected(timeout=timeout) +distributed/client.py:1604: in _ensure_connected + msg = await wait_for(comm.read(), timeout) +distributed/utils.py:1913: in wait_for + async with asyncio.timeout(timeout): +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = +exc_type = +exc_val = CancelledError(), exc_tb = + + async def __aexit__( + self, + exc_type: Optional[Type[BaseException]], + exc_val: Optional[BaseException], + exc_tb: Optional[TracebackType], + ) -> Optional[bool]: + assert self._state in (_State.ENTERED, _State.EXPIRING) + + if self._timeout_handler is not None: + self._timeout_handler.cancel() + self._timeout_handler = None + + if self._state is _State.EXPIRING: + self._state = _State.EXPIRED + + if self._task.uncancel() <= self._cancelling and exc_type is not None: + # Since there are no new cancel requests, we're + # handling this. + if issubclass(exc_type, exceptions.CancelledError): +> raise TimeoutError from exc_val +E TimeoutError + +/usr/lib/python3.13/asyncio/timeouts.py:116: TimeoutError +----------------------------- Captured stderr call ----------------------------- +2025-01-14 16:21:40,264 - distributed.scheduler - INFO - State start +2025-01-14 16:21:40,278 - distributed.scheduler - INFO - Scheduler at: tcp://127.0.0.1:46261 +2025-01-14 16:21:40,284 - distributed.scheduler - INFO - dashboard at: http://127.0.0.1:33317/status +2025-01-14 16:21:40,291 - distributed.scheduler - INFO - Registering Worker plugin shuffle +2025-01-14 16:21:40,306 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:37981 +2025-01-14 16:21:40,313 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:37981 +2025-01-14 16:21:40,321 - distributed.worker - INFO - Worker name: 0 +2025-01-14 16:21:40,327 - distributed.worker - INFO - dashboard at: 127.0.0.1:39389 +2025-01-14 16:21:40,334 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:46261 +2025-01-14 16:21:40,342 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 16:21:40,349 - distributed.worker - INFO - Threads: 1 +2025-01-14 16:21:40,355 - distributed.worker - INFO - Memory: 0.98 kiB +2025-01-14 16:21:40,362 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-rnnwaked +2025-01-14 16:21:40,369 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 16:21:40,449 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:37981 name: 0 +2025-01-14 16:21:40,519 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:37981 +2025-01-14 16:21:40,524 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:33626 +2025-01-14 16:21:40,526 - distributed.worker - INFO - Starting Worker plugin shuffle +2025-01-14 16:21:40,533 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:46261 +2025-01-14 16:21:40,540 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 16:21:40,547 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:46261 +2025-01-14 16:21:40,631 - distributed.scheduler - INFO - Receive client connection: Client-4a145ded-d21e-11ef-a851-8d61c06d7977 +2025-01-14 16:21:40,703 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:33642 +2025-01-14 16:21:46,377 - distributed.worker.memory - WARNING - gc.collect() took 4.293s. This is usually a sign that some tasks handle too many Python objects at the same time. Rechunking the work into smaller tasks might help. +2025-01-14 16:21:46,383 - distributed.worker.memory - WARNING - Worker is at 96584908% memory usage. Pausing worker. Process memory: 921.11 MiB -- Worker memory limit: 0.98 kiB +2025-01-14 16:21:46,387 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 921.11 MiB -- Worker memory limit: 0.98 kiB +2025-01-14 16:21:46,394 - distributed.core - INFO - Event loop was unresponsive in Scheduler for 6.10s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability. +2025-01-14 16:21:46,395 - distributed.core - INFO - Event loop was unresponsive in Worker for 5.85s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability. +2025-01-14 16:21:46,397 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:37981. Reason: worker-close +2025-01-14 16:21:46,404 - distributed.core - INFO - Connection to tcp://127.0.0.1:33642 has been closed. +2025-01-14 16:21:46,404 - distributed.scheduler - INFO - Remove client Client-4a145ded-d21e-11ef-a851-8d61c06d7977 +2025-01-14 16:21:46,410 - distributed.worker - INFO - Removing Worker plugin shuffle +2025-01-14 16:21:46,421 - distributed.scheduler - INFO - Close client connection: Client-4a145ded-d21e-11ef-a851-8d61c06d7977 +2025-01-14 16:21:46,426 - distributed.worker - ERROR - Failed to communicate with scheduler during heartbeat. +Traceback (most recent call last): + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.13_distributed/build/distributed/comm/tcp.py", line 225, in read + frames_nosplit_nbytes_bin = await stream.read_bytes(fmt_size) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +tornado.iostream.StreamClosedError: Stream is closed + +The above exception was the direct cause of the following exception: + +Traceback (most recent call last): + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.13_distributed/build/distributed/worker.py", line 1269, in heartbeat + response = await retry_operation( + ^^^^^^^^^^^^^^^^^^^^^^ + ...<14 lines>... + ) + ^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.13_distributed/build/distributed/utils_comm.py", line 441, in retry_operation + return await retry( + ^^^^^^^^^^^^ + ...<5 lines>... + ) + ^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.13_distributed/build/distributed/utils_comm.py", line 420, in retry + return await coro() + ^^^^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.13_distributed/build/distributed/core.py", line 1259, in send_recv_from_rpc + return await send_recv(comm=comm, op=key, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.13_distributed/build/distributed/core.py", line 1018, in send_recv + response = await comm.read(deserializers=deserializers) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.13_distributed/build/distributed/comm/tcp.py", line 236, in read + convert_stream_closed_error(self, e) + ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.13_distributed/build/distributed/comm/tcp.py", line 142, in convert_stream_closed_error + raise CommClosedError(f"in {obj}: {exc}") from exc +distributed.comm.core.CommClosedError: in : Stream is closed +2025-01-14 16:21:46,436 - distributed.core - INFO - Connection to tcp://127.0.0.1:46261 has been closed. +2025-01-14 16:21:46,437 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:33626; closing. +2025-01-14 16:21:46,437 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:37981 name: 0 (stimulus_id='handle-worker-cleanup-1736821306.4379246') +2025-01-14 16:21:46,443 - distributed.scheduler - INFO - Lost all workers +2025-01-14 16:21:46,448 - distributed.scheduler - INFO - Closing scheduler. Reason: unknown +2025-01-14 16:21:46,453 - distributed.scheduler - INFO - Scheduler closing all comms ============================= slowest 20 durations ============================= -11.69s call distributed/tests/test_gc.py::test_gc_diagnosis_cpu_time -10.02s call distributed/tests/test_utils_test.py::test_popen_timeout -6.15s call distributed/tests/test_failed_workers.py::test_worker_doesnt_await_task_completion -6.00s call distributed/tests/test_stress.py::test_cancel_stress -5.83s call distributed/tests/test_nanny.py::test_num_fds -5.18s call distributed/tests/test_steal.py::test_balance_with_longer_task -4.86s call distributed/tests/test_chaos.py::test_KillWorker[sys.exit] -4.59s call distributed/tests/test_stress.py::test_cancel_stress_sync -4.22s call distributed/diagnostics/tests/test_progress.py::test_group_timing -4.16s call distributed/shuffle/tests/test_rechunk.py::test_homogeneously_schedule_unpack -4.04s call distributed/tests/test_worker.py::test_upload_file -4.00s call distributed/shuffle/tests/test_rechunk.py::test_rechunk_method -3.55s call distributed/tests/test_failed_workers.py::test_restart_sync -3.54s call distributed/tests/test_client.py::test_logs -3.46s call distributed/tests/test_tls_functional.py::test_retire_workers -3.41s call distributed/tests/test_nanny.py::test_environ_plugin -3.35s call distributed/tests/test_worker.py::test_avoid_oversubscription -3.33s call distributed/diagnostics/tests/test_progress.py::test_AllProgress -3.31s setup distributed/tests/test_client.py::test_recreate_task_sync -3.24s call distributed/tests/test_client.py::test_default_get +41.37s call distributed/tests/test_gc.py::test_gc_diagnosis_cpu_time +32.59s call distributed/shuffle/tests/test_rechunk.py::test_homogeneously_schedule_unpack +30.62s call distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_partially_unknown_dimension[x5-chunks5] +23.84s call distributed/tests/test_nanny.py::test_num_fds +23.81s call distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_partially_unknown_dimension[x3-chunks3] +20.12s call distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_partially_unknown_dimension[x4-chunks4] +18.77s call distributed/shuffle/tests/test_rechunk.py::test_rechunk_method +17.02s call distributed/tests/test_failed_workers.py::test_restart_sync +16.26s call distributed/tests/test_stress.py::test_cancel_stress +13.99s call distributed/diagnostics/tests/test_progress.py::test_AllProgress +13.87s call distributed/tests/test_gc.py::test_gc_diagnosis_rss_win +13.52s call distributed/tests/test_nanny.py::test_nanny_process_failure +12.88s call distributed/tests/test_stress.py::test_cancel_stress_sync +12.49s call distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_fully_unknown_dimension[x3-chunks3] +11.79s call distributed/tests/test_failed_workers.py::test_multiple_clients_restart +11.44s call distributed/diagnostics/tests/test_progress.py::test_group_timing +11.43s call distributed/tests/test_nanny.py::test_nanny_restart_timeout_stress +11.29s call distributed/tests/test_failed_workers.py::test_restart +11.13s call distributed/tests/test_failed_workers.py::test_restart_cleared +11.04s call distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_fully_unknown_dimension[x5-chunks5] =========================== short test summary info ============================ SKIPPED [1] distributed/cli/tests/test_dask_ssh.py:9: could not import 'paramiko': No module named 'paramiko' SKIPPED [1] distributed/comm/tests/test_ucx.py:15: could not import 'ucp': No module named 'ucp' @@ -8212,8 +9669,314 @@ SKIPPED [1] distributed/tests/test_worker_memory.py:871: need --runslow option to run SKIPPED [2] distributed/tests/test_worker_memory.py:883: need --runslow option to run SKIPPED [1] distributed/tests/test_worker_memory.py:997: need --runslow option to run -= 2912 passed, 265 skipped, 222 deselected, 15 xfailed, 8 xpassed in 1107.59s (0:18:27) = -*** END OF RUN 1: ALL TESTS RUN HAVE NOW PASSED/XFAILED *** +FAILED distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_partially_unknown_dimension[x5-chunks5] - TimeoutError: Test timeout (30) hit after 30.031416482990608s. +========== Test stack trace starts here ========== +Stack for wait_for=> (most recent call last): + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.13_distributed/build/distributed/shuffle/tests/test_rechunk.py", line 801, in test_rechunk_with_partially_unknown_dimension + assert_eq(await c.compute(result), await c.compute(expected)) +FAILED distributed/shuffle/tests/test_rechunk.py::test_homogeneously_schedule_unpack - TimeoutError: Test timeout (30) hit after 30.414505616994575s. +========== Test stack trace starts here ========== +Stack for wait_for=> (most recent call last): + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.13_distributed/build/distributed/shuffle/tests/test_rechunk.py", line 1522, in test_homogeneously_schedule_unpack + await c.compute(res) +FAILED distributed/tests/test_computations.py::test_computations_futures - ValueError: too many values to unpack (expected 1) +FAILED distributed/tests/test_nanny.py::test_nanny_process_failure - Failed: condition not reached until 5 seconds +FAILED distributed/tests/test_scheduler.py::test_tell_workers_when_peers_have_left - distributed.client.FutureCancelledError: g cancelled for reason: unknown. +FAILED distributed/tests/test_worker.py::test_log_remove_worker - AssertionError: assert {'tcp://127.0.0.1:45689': [{'action': 'worker-status-change', 'prev-status': 'running', 'status': 'closing_gracefully', 'stimulus_id': 'graceful'}, {'action': 'remove-worker', 'processing-tasks': {'y'}, 'lost-computed-tasks': set(), 'lost-scattered-tasks': set(), 'stimulus_id': 'graceful', 'expected': True}, {'action': 'retired', 'stimulus_id': 'graceful'}], 'all': [{'action': 'remove-worker', 'processing-tasks': {'y'}, 'lost-computed-tasks': set(), 'lost-scattered-tasks': set(), 'stimulus_id': 'graceful', 'expected': True, 'worker': 'tcp://127.0.0.1:45689'}, {'action': 'retire-workers', 'retired': {'tcp://127.0.0.1:45689': 'snip'}, 'could-not-retire': {}, 'stimulus_id': 'graceful'}, {'action': 'retire-workers', 'retired': {}, 'could-not-retire': {'tcp://127.0.0.1:39417': 'snip'}, 'stimulus_id': 'graceful_abort'}, {'action': 'remove-worker', 'processing-tasks': {'y'}, 'lost-computed-tasks': {'x'}, 'lost-scattered-tasks': {'z'}, 'stimulus_id': 'ungraceful', 'expected': False, 'worker': 'tcp://127.0.0.1:39417'}], 'tcp://127.0.0.1:39417': [{'action': 'worker-status-change', 'prev-status': 'running', 'status': 'closing_gracefully', 'stimulus_id': 'graceful_abort'}, {'action': 'could-not-retire', 'stimulus_id': 'graceful_abort'}, {'action': 'worker-status-change', 'prev-status': 'closing_gracefully', 'status': 'running', 'stimulus_id': 'worker-status-change'}, {'action': 'remove-worker', 'processing-tasks': {'y'}, 'lost-computed-tasks': {'x'}, 'lost-scattered-tasks': {'z'}, 'stimulus_id': 'ungraceful', 'expected': False}, {'action': 'closing-worker', 'reason': 'scheduler-remove-worker'}], 'worker-get-client': [{'client': 'Client-22871129-d21e-11ef-a851-8d61c06d7977', 'timeout': 5, 'worker': 'tcp://127.0.0.1:45689'}, {'client': 'Client-22871129-d21e-11ef-a851-8d61c06d7977', 'timeout': 5, 'worker': 'tcp://127.0.0.1:39417'}]} == {'tcp://127.0.0.1:45689': [{'action': 'worker-status-change', 'prev-status': 'running', 'status': 'closing_gracefully', 'stimulus_id': 'graceful'}, {'action': 'remove-worker', 'lost-computed-tasks': set(), 'lost-scattered-tasks': set(), 'processing-tasks': {'y'}, 'expected': True, 'stimulus_id': 'graceful'}, {'action': 'retired', 'stimulus_id': 'graceful'}], 'tcp://127.0.0.1:39417': [{'action': 'worker-status-change', 'prev-status': 'running', 'status': 'closing_gracefully', 'stimulus_id': 'graceful_abort'}, {'action': 'could-not-retire', 'stimulus_id': 'graceful_abort'}, {'action': 'worker-status-change', 'prev-status': 'closing_gracefully', 'status': 'running', 'stimulus_id': 'worker-status-change'}, {'action': 'remove-worker', 'lost-computed-tasks': {'x'}, 'lost-scattered-tasks': {'z'}, 'processing-tasks': {'y'}, 'expected': False, 'stimulus_id': 'ungraceful'}, {'action': 'closing-worker', 'reason': 'scheduler-remove-worker'}], 'all': [{'action': 'remove-worker', 'lost-computed-tasks': set(), 'lost-scattered-tasks': set(), 'processing-tasks': {'y'}, 'expected': True, 'stimulus_id': 'graceful', 'worker': 'tcp://127.0.0.1:45689'}, {'action': 'retire-workers', 'stimulus_id': 'graceful', 'retired': {'tcp://127.0.0.1:45689': 'snip'}, 'could-not-retire': {}}, {'action': 'retire-workers', 'stimulus_id': 'graceful_abort', 'retired': {}, 'could-not-retire': {'tcp://127.0.0.1:39417': 'snip'}}, {'action': 'remove-worker', 'lost-computed-tasks': {'x'}, 'lost-scattered-tasks': {'z'}, 'processing-tasks': {'y'}, 'expected': False, 'stimulus_id': 'ungraceful', 'worker': 'tcp://127.0.0.1:39417'}], 'worker-get-client': [{'client': 'Client-22871129-d21e-11ef-a851-8d61c06d7977', 'timeout': 5, 'worker': 'tcp://127.0.0.1:39417'}]} + + Common items: + {'all': [{'action': 'remove-worker', + 'expected': True, + 'lost-computed-tasks': set(), + 'lost-scattered-tasks': set(), + 'processing-tasks': {'y'}, + 'stimulus_id': 'graceful', + 'worker': 'tcp://127.0.0.1:45689'}, + {'action': 'retire-workers', + 'could-not-retire': {}, + 'retired': {'tcp://127.0.0.1:45689': 'snip'}, + 'stimulus_id': 'graceful'}, + {'action': 'retire-workers', + 'could-not-retire': {'tcp://127.0.0.1:39417': 'snip'}, + 'retired': {}, + 'stimulus_id': 'graceful_abort'}, + {'action': 'remove-worker', + 'expected': False, + 'lost-computed-tasks': {'x'}, + 'lost-scattered-tasks': {'z'}, + 'processing-tasks': {'y'}, + 'stimulus_id': 'ungraceful', + 'worker': 'tcp://127.0.0.1:39417'}], + 'tcp://127.0.0.1:39417': [{'action': 'worker-status-change', + 'prev-status': 'running', + 'status': 'closing_gracefully', + 'stimulus_id': 'graceful_abort'}, + {'action': 'could-not-retire', + 'stimulus_id': 'graceful_abort'}, + {'action': 'worker-status-change', + 'prev-status': 'closing_gracefully', + 'status': 'running', + 'stimulus_id': 'worker-status-change'}, + {'action': 'remove-worker', + 'expected': False, + 'lost-computed-tasks': {'x'}, + 'lost-scattered-tasks': {'z'}, + 'processing-tasks': {'y'}, + 'stimulus_id': 'ungraceful'}, + {'action': 'closing-worker', + 'reason': 'scheduler-remove-worker'}], + 'tcp://127.0.0.1:45689': [{'action': 'worker-status-change', + 'prev-status': 'running', + 'status': 'closing_gracefully', + 'stimulus_id': 'graceful'}, + {'action': 'remove-worker', + 'expected': True, + 'lost-computed-tasks': set(), + 'lost-scattered-tasks': set(), + 'processing-tasks': {'y'}, + 'stimulus_id': 'graceful'}, + {'action': 'retired', 'stimulus_id': 'graceful'}]} + Differing items: + {'worker-get-client': [{'client': 'Client-22871129-d21e-11ef-a851-8d61c06d7977', 'timeout': 5, 'worker': 'tcp://127.0.0.1:45689'}, {'client': 'Client-22871129-d21e-11ef-a851-8d61c06d7977', 'timeout': 5, 'worker': 'tcp://127.0.0.1:39417'}]} != {'worker-get-client': [{'client': 'Client-22871129-d21e-11ef-a851-8d61c06d7977', 'timeout': 5, 'worker': 'tcp://127.0.0.1:39417'}]} + + Full diff: + { + 'all': [ + { + 'action': 'remove-worker', + 'expected': True, + 'lost-computed-tasks': set(), + 'lost-scattered-tasks': set(), + 'processing-tasks': { + 'y', + }, + 'stimulus_id': 'graceful', + 'worker': 'tcp://127.0.0.1:45689', + }, + { + 'action': 'retire-workers', + 'could-not-retire': {}, + 'retired': { + 'tcp://127.0.0.1:45689': 'snip', + }, + 'stimulus_id': 'graceful', + }, + { + 'action': 'retire-workers', + 'could-not-retire': { + 'tcp://127.0.0.1:39417': 'snip', + }, + 'retired': {}, + 'stimulus_id': 'graceful_abort', + }, + { + 'action': 'remove-worker', + 'expected': False, + 'lost-computed-tasks': { + 'x', + }, + 'lost-scattered-tasks': { + 'z', + }, + 'processing-tasks': { + 'y', + }, + 'stimulus_id': 'ungraceful', + 'worker': 'tcp://127.0.0.1:39417', + }, + ], + 'tcp://127.0.0.1:39417': [ + { + 'action': 'worker-status-change', + 'prev-status': 'running', + 'status': 'closing_gracefully', + 'stimulus_id': 'graceful_abort', + }, + { + 'action': 'could-not-retire', + 'stimulus_id': 'graceful_abort', + }, + { + 'action': 'worker-status-change', + 'prev-status': 'closing_gracefully', + 'status': 'running', + 'stimulus_id': 'worker-status-change', + }, + { + 'action': 'remove-worker', + 'expected': False, + 'lost-computed-tasks': { + 'x', + }, + 'lost-scattered-tasks': { + 'z', + }, + 'processing-tasks': { + 'y', + }, + 'stimulus_id': 'ungraceful', + }, + { + 'action': 'closing-worker', + 'reason': 'scheduler-remove-worker', + }, + ], + 'tcp://127.0.0.1:45689': [ + { + 'action': 'worker-status-change', + 'prev-status': 'running', + 'status': 'closing_gracefully', + 'stimulus_id': 'graceful', + }, + { + 'action': 'remove-worker', + 'expected': True, + 'lost-computed-tasks': set(), + 'lost-scattered-tasks': set(), + 'processing-tasks': { + 'y', + }, + 'stimulus_id': 'graceful', + }, + { + 'action': 'retired', + 'stimulus_id': 'graceful', + }, + ], + 'worker-get-client': [ + { + 'client': 'Client-22871129-d21e-11ef-a851-8d61c06d7977', + 'timeout': 5, + + 'worker': 'tcp://127.0.0.1:45689', + + }, + + { + + 'client': 'Client-22871129-d21e-11ef-a851-8d61c06d7977', + + 'timeout': 5, + 'worker': 'tcp://127.0.0.1:39417', + }, + ], + } +FAILED distributed/tests/test_worker_memory.py::test_fail_to_pickle_spill - TimeoutError += 7 failed, 2905 passed, 265 skipped, 222 deselected, 15 xfailed, 8 xpassed in 3340.34s (0:55:40) = +*** END OF RUN 1: NOT ALL TESTS HAVE YET PASSED/XFAILED *** +*** STARTING RUN 2: python3.13 -m pytest --pyargs distributed --verbose --color=no --timeout-method=signal --timeout=300 -m not avoid_ci -rfE --last-failed --last-failed-no-failures none --ignore=distributed/comm/tests/test_comms.py --ignore=distributed/comm/tests/test_ws.py --ignore=distributed/deploy/tests/test_adaptive.py --ignore=distributed/deploy/tests/test_local.py --ignore=distributed/deploy/tests/test_slow_adaptive.py --ignore=distributed/deploy/tests/test_spec_cluster.py --deselect=distributed/cli/tests/test_dask_scheduler.py::test_no_dashboard --deselect=distributed/deploy/tests/test_local.py::test_localcluster_get_client --deselect=distributed/deploy/tests/test_old_ssh.py::test_cluster --deselect=distributed/deploy/tests/test_old_ssh.py::test_old_ssh_nprocs_renamed_to_n_workers --deselect=distributed/deploy/tests/test_old_ssh.py::test_nprocs_attribute_is_deprecated --deselect=distributed/deploy/tests/test_ssh.py::test_nprocs_attribute_is_deprecated --deselect=distributed/http/tests/test_core.py::test_prometheus_api_doc --deselect=distributed/tests/test_init.py::test_git_revision --deselect=distributed/tests/test_jupyter.py::test_jupyter_idle_timeout --deselect=distributed/tests/test_jupyter.py::test_jupyter_idle_timeout_returned --deselect=distributed/tests/test_jupyter.py::test_jupyter_server --deselect=distributed/tests/test_jupyter.py::test_shutsdown_cleanly --deselect=distributed/tests/test_profile.py::test_stack_overflow --deselect=distributed/tests/test_pubsub.py::test_client_worker --deselect=distributed/tests/test_queues.py::test_queue_in_task --deselect=distributed/tests/test_spill.py::test_spillbuffer_oserror --deselect=distributed/tests/test_steal.py::test_steal_twice --deselect=distributed/tests/test_utils_test.py::test_cluster --deselect=distributed/tests/test_variable.py::test_variable_in_task --deselect=distributed/tests/test_worker.py::test_process_executor_kills_process --deselect=distributed/tests/test_worker_memory.py::test_fail_to_pickle_execute_1 --deselect=distributed/tests/test_worker_state_machine.py::test_task_state_instance_are_garbage_collected --deselect=distributed/protocol/tests/test_protocol.py::test_deeply_nested_structures --deselect=distributed/protocol/tests/test_serialize.py::test_deeply_nested_structures --deselect=distributed/cli/tests/test_dask_scheduler.py::test_defaults --deselect=distributed/cli/tests/test_dask_scheduler.py::test_hostport --deselect=distributed/cli/tests/test_dask_spec.py::test_errors --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_file --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_file --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_file --deselect=distributed/dashboard/tests/test_scheduler_bokeh.py::test_counters --deselect=distributed/dashboard/tests/test_worker_bokeh.py::test_counters --deselect=distributed/deploy/tests/test_local.py::test_adapt_then_manual --deselect=distributed/deploy/tests/test_local.py::test_async_with --deselect=distributed/deploy/tests/test_local.py::test_close_twice --deselect=distributed/deploy/tests/test_local.py::test_cluster_info_sync --deselect=distributed/deploy/tests/test_local.py::test_local_tls --deselect=distributed/deploy/tests/test_local.py::test_no_dangling_asyncio_tasks --deselect=distributed/deploy/tests/test_local.py::test_only_local_access --deselect=distributed/deploy/tests/test_local.py::test_remote_access --deselect=distributed/diagnostics/tests/test_progress_widgets.py::test_serializers --deselect=distributed/diagnostics/tests/test_scheduler_plugin.py::test_lifecycle --deselect=distributed/http/scheduler/tests/test_missing_bokeh.py::test_missing_bokeh --deselect=distributed/http/scheduler/tests/test_scheduler_http.py::test_metrics_when_prometheus_client_not_installed --deselect=distributed/protocol/tests/test_serialize.py::test_errors --deselect=distributed/tests/test_batched.py::test_BatchedSend --deselect=distributed/tests/test_batched.py::test_close_closed --deselect=distributed/tests/test_batched.py::test_close_twice --deselect=distributed/tests/test_batched.py::test_send_after_stream_start --deselect=distributed/tests/test_batched.py::test_send_before_close --deselect=distributed/tests/test_batched.py::test_send_before_start --deselect=distributed/tests/test_batched.py::test_sending_traffic_jam --deselect=distributed/tests/test_batched.py::test_serializers --deselect=distributed/tests/test_client.py::TestClientSecurityLoader::test_security_loader --deselect=distributed/tests/test_client.py::TestClientSecurityLoader::test_security_loader_ignored_if_explicit_security_provided --deselect=distributed/tests/test_client.py::TestClientSecurityLoader::test_security_loader_ignored_if_returns_none --deselect=distributed/tests/test_client.py::test_async_with --deselect=distributed/tests/test_client.py::test_client_is_quiet_cluster_close --deselect=distributed/tests/test_client.py::test_dashboard_link_cluster --deselect=distributed/tests/test_client.py::test_dashboard_link_inproc --deselect=distributed/tests/test_client.py::test_file_descriptors_dont_leak --deselect=distributed/tests/test_client.py::test_mixing_clients_different_scheduler --deselect=distributed/tests/test_client.py::test_quiet_client_close --deselect=distributed/tests/test_client.py::test_rebalance_sync --deselect=distributed/tests/test_client.py::test_repr_localcluster --deselect=distributed/tests/test_client.py::test_security_loader --deselect=distributed/tests/test_client.py::test_security_loader_ignored_if_explicit_security_provided --deselect=distributed/tests/test_client.py::test_security_loader_ignored_if_returns_none --deselect=distributed/tests/test_client.py::test_shutdown --deselect=distributed/tests/test_client.py::test_shutdown_is_quiet_with_cluster --deselect=distributed/tests/test_client.py::test_shutdown_localcluster --deselect=distributed/tests/test_client.py::test_shutdown_stops_callbacks --deselect=distributed/tests/test_client_loop.py::test_close_loop_sync_start_new_loop --deselect=distributed/tests/test_client_loop.py::test_close_loop_sync_use_running_loop --deselect=distributed/tests/test_core.py::test_close_fast_without_active_handlers --deselect=distributed/tests/test_core.py::test_close_grace_period_for_handlers --deselect=distributed/tests/test_core.py::test_close_properly --deselect=distributed/tests/test_core.py::test_compression --deselect=distributed/tests/test_core.py::test_connection_pool --deselect=distributed/tests/test_core.py::test_connection_pool_close_while_connecting --deselect=distributed/tests/test_core.py::test_connection_pool_detects_remote_close --deselect=distributed/tests/test_core.py::test_connection_pool_outside_cancellation --deselect=distributed/tests/test_core.py::test_connection_pool_remove --deselect=distributed/tests/test_core.py::test_connection_pool_respects_limit --deselect=distributed/tests/test_core.py::test_connection_pool_tls --deselect=distributed/tests/test_core.py::test_counters --deselect=distributed/tests/test_core.py::test_deserialize_error --deselect=distributed/tests/test_core.py::test_errors --deselect=distributed/tests/test_core.py::test_identity_inproc --deselect=distributed/tests/test_core.py::test_identity_tcp --deselect=distributed/tests/test_core.py::test_large_packets_inproc --deselect=distributed/tests/test_core.py::test_messages_are_ordered_bsend --deselect=distributed/tests/test_core.py::test_messages_are_ordered_raw --deselect=distributed/tests/test_core.py::test_ports --deselect=distributed/tests/test_core.py::test_rpc_default --deselect=distributed/tests/test_core.py::test_rpc_inproc --deselect=distributed/tests/test_core.py::test_rpc_message_lifetime_default --deselect=distributed/tests/test_core.py::test_rpc_message_lifetime_inproc --deselect=distributed/tests/test_core.py::test_rpc_message_lifetime_tcp --deselect=distributed/tests/test_core.py::test_rpc_serialization --deselect=distributed/tests/test_core.py::test_rpc_tcp --deselect=distributed/tests/test_core.py::test_rpc_tls --deselect=distributed/tests/test_core.py::test_rpc_with_many_connections_inproc --deselect=distributed/tests/test_core.py::test_rpc_with_many_connections_tcp --deselect=distributed/tests/test_core.py::test_send_recv_args --deselect=distributed/tests/test_core.py::test_send_recv_cancelled --deselect=distributed/tests/test_core.py::test_server --deselect=distributed/tests/test_core.py::test_server_comms_mark_active_handlers --deselect=distributed/tests/test_core.py::test_server_raises_on_blocked_handlers --deselect=distributed/tests/test_jupyter.py::test_jupyter_idle_timeout --deselect=distributed/tests/test_jupyter.py::test_jupyter_server --deselect=distributed/tests/test_locks.py::test_errors --deselect=distributed/tests/test_nanny.py::test_scheduler_file --deselect=distributed/tests/test_nanny.py::test_scheduler_file --deselect=distributed/tests/test_nanny.py::test_scheduler_file --deselect=distributed/tests/test_nanny.py::test_worker_uses_same_host_as_nanny --deselect=distributed/tests/test_preload.py::test_failure_doesnt_crash_scheduler --deselect=distributed/tests/test_preload.py::test_preload_import_time --deselect=distributed/tests/test_preload.py::test_preload_manager_sequence --deselect=distributed/tests/test_preload.py::test_worker_preload_text --deselect=distributed/tests/test_scheduler.py::test_allowed_failures_config --deselect=distributed/tests/test_scheduler.py::test_async_context_manager --deselect=distributed/tests/test_scheduler.py::test_dashboard_host --deselect=distributed/tests/test_scheduler.py::test_file_descriptors_dont_leak --deselect=distributed/tests/test_scheduler.py::test_finished --deselect=distributed/tests/test_scheduler.py::test_multiple_listeners --deselect=distributed/tests/test_scheduler.py::test_no_dangling_asyncio_tasks --deselect=distributed/tests/test_scheduler.py::test_scheduler_file --deselect=distributed/tests/test_scheduler.py::test_scheduler_file --deselect=distributed/tests/test_scheduler.py::test_scheduler_file --deselect=distributed/tests/test_security.py::test_require_encryption --deselect=distributed/tests/test_security.py::test_tls_listen_connect --deselect=distributed/tests/test_security.py::test_tls_temporary_credentials_functional --deselect=distributed/tests/test_semaphore.py::test_threadpoolworkers_pick_correct_ioloop --deselect=distributed/tests/test_tls_functional.py::test_security_dict_input_no_security --deselect=distributed/tests/test_utils_test.py::test_ensure_no_new_clients --deselect=distributed/tests/test_utils_test.py::test_freeze_batched_send --deselect=distributed/tests/test_utils_test.py::test_locked_comm_drop_in_replacement --deselect=distributed/tests/test_utils_test.py::test_locked_comm_intercept_read --deselect=distributed/tests/test_utils_test.py::test_locked_comm_intercept_write --deselect=distributed/tests/test_worker.py::test_host_uses_scheduler_protocol --deselect=distributed/tests/test_worker.py::test_plugin_exception --deselect=distributed/tests/test_worker.py::test_plugin_internal_exception --deselect=distributed/tests/test_worker.py::test_plugin_multiple_exceptions --deselect=distributed/tests/test_worker.py::test_scheduler_file --deselect=distributed/tests/test_worker.py::test_scheduler_file --deselect=distributed/tests/test_worker.py::test_scheduler_file --deselect=distributed/tests/test_worker_client.py::test_dont_override_default_get --deselect=distributed/cli/tests/test_dask_scheduler.py::test_dashboard --deselect=distributed/cli/tests/test_dask_scheduler.py::test_dashboard_allowlist --deselect=distributed/cli/tests/test_dask_scheduler.py::test_dashboard_non_standard_ports --deselect=distributed/cli/tests/test_dask_scheduler.py::test_dashboard_port_zero --deselect=distributed/cli/tests/test_dask_scheduler.py::test_defaults --deselect=distributed/cli/tests/test_dask_scheduler.py::test_hostport --deselect=distributed/cli/tests/test_dask_scheduler.py::test_multiple_protocols --deselect=distributed/cli/tests/test_dask_scheduler.py::test_multiple_workers --deselect=distributed/cli/tests/test_dask_scheduler.py::test_multiple_workers_2 --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_command --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_command_default --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_config --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_file --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_module --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_remote_module --deselect=distributed/cli/tests/test_dask_scheduler.py::test_scheduler_port_zero --deselect=distributed/cli/tests/test_dask_scheduler.py::test_single_executable_deprecated --deselect=distributed/cli/tests/test_dask_worker.py::test_contact_listen_address --deselect=distributed/cli/tests/test_dask_worker.py::test_dashboard_non_standard_ports --deselect=distributed/cli/tests/test_dask_worker.py::test_error_during_startup --deselect=distributed/cli/tests/test_dask_worker.py::test_integer_names --deselect=distributed/cli/tests/test_dask_worker.py::test_listen_address_ipv6 --deselect=distributed/cli/tests/test_dask_worker.py::test_local_directory --deselect=distributed/cli/tests/test_dask_worker.py::test_memory_limit --deselect=distributed/cli/tests/test_dask_worker.py::test_nanny_worker_port_range --deselect=distributed/cli/tests/test_dask_worker.py::test_nanny_worker_port_range_too_many_workers_raises --deselect=distributed/cli/tests/test_dask_worker.py::test_nanny_worker_ports --deselect=distributed/cli/tests/test_dask_worker.py::test_no_nanny --deselect=distributed/cli/tests/test_dask_worker.py::test_nworkers_auto --deselect=distributed/cli/tests/test_dask_worker.py::test_nworkers_expands_name --deselect=distributed/cli/tests/test_dask_worker.py::test_nworkers_negative --deselect=distributed/cli/tests/test_dask_worker.py::test_nworkers_requires_nanny --deselect=distributed/cli/tests/test_dask_worker.py::test_preload_config --deselect=distributed/cli/tests/test_dask_worker.py::test_resources --deselect=distributed/cli/tests/test_dask_worker.py::test_respect_host_listen_address --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_address_env --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_file --deselect=distributed/cli/tests/test_dask_worker.py::test_set_lifetime_restart_via_env_var --deselect=distributed/cli/tests/test_dask_worker.py::test_set_lifetime_stagger_via_env_var --deselect=distributed/cli/tests/test_dask_worker.py::test_signal_handling --deselect=distributed/cli/tests/test_dask_worker.py::test_single_executable_deprecated --deselect=distributed/cli/tests/test_dask_worker.py::test_single_executable_works --deselect=distributed/cli/tests/test_dask_worker.py::test_timeout --deselect=distributed/cli/tests/test_dask_worker.py::test_worker_class --deselect=distributed/tests/test_config.py::test_logging_extended --deselect=distributed/tests/test_config.py::test_logging_file_config --deselect=distributed/tests/test_config.py::test_logging_mutual_exclusive --deselect=distributed/tests/test_config.py::test_logging_simple --deselect=distributed/tests/test_config.py::test_logging_simple_under_distributed --deselect=distributed/tests/test_queues.py::test_queue_in_task --deselect=distributed/tests/test_variable.py::test_variable_in_task +============================= test session starts ============================== +platform linux -- Python 3.13.1, pytest-8.3.4, pluggy-1.5.0 -- /usr/bin/python3.13 +cachedir: .pytest_cache +rootdir: /build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.13_distributed/build +configfile: pyproject.toml +plugins: rerunfailures-14.0, anyio-4.7.0, timeout-2.3.1, typeguard-4.4.1 +timeout: 300.0s +timeout method: signal +timeout func_only: False +collecting ... collected 46 items / 1 error / 41 deselected / 5 skipped / 5 selected +run-last-failure: rerun previous 5 failures (skipped 129 files) + +==================================== ERRORS ==================================== +__________ ERROR collecting distributed/shuffle/tests/test_rechunk.py __________ +distributed/shuffle/tests/test_rechunk.py:39: in + from distributed.shuffle.tests.utils import AbstractShuffleTestPool +distributed/shuffle/tests/utils.py:11: in + import dask.dataframe as dd +/usr/lib/python3/dist-packages/dask/dataframe/__init__.py:115: in + if _dask_expr_enabled(): +/usr/lib/python3/dist-packages/dask/dataframe/__init__.py:31: in _dask_expr_enabled + warnings.warn( +E FutureWarning: The legacy Dask DataFrame implementation is deprecated and will be removed in a future version. Set the configuration option `dataframe.query-planning` to `True` or None to enable the new Dask Dataframe implementation and silence this warning. +=========================== short test summary info ============================ +ERROR distributed/shuffle/tests/test_rechunk.py - FutureWarning: The legacy Dask DataFrame implementation is deprecated and will be removed in a future version. Set the configuration option `dataframe.query-planning` to `True` or None to enable the new Dask Dataframe implementation and silence this warning. +!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!! +================== 5 skipped, 41 deselected, 1 error in 2.47s ================== +*** END OF RUN 2: NOT ALL TESTS HAVE YET PASSED/XFAILED *** +*** STARTING RUN 3: python3.13 -m pytest --pyargs distributed --verbose --color=no --timeout-method=signal --timeout=300 -m not avoid_ci -rfE --last-failed --last-failed-no-failures none --ignore=distributed/comm/tests/test_comms.py --ignore=distributed/comm/tests/test_ws.py --ignore=distributed/deploy/tests/test_adaptive.py --ignore=distributed/deploy/tests/test_local.py --ignore=distributed/deploy/tests/test_slow_adaptive.py --ignore=distributed/deploy/tests/test_spec_cluster.py --deselect=distributed/cli/tests/test_dask_scheduler.py::test_no_dashboard --deselect=distributed/deploy/tests/test_local.py::test_localcluster_get_client --deselect=distributed/deploy/tests/test_old_ssh.py::test_cluster --deselect=distributed/deploy/tests/test_old_ssh.py::test_old_ssh_nprocs_renamed_to_n_workers --deselect=distributed/deploy/tests/test_old_ssh.py::test_nprocs_attribute_is_deprecated --deselect=distributed/deploy/tests/test_ssh.py::test_nprocs_attribute_is_deprecated --deselect=distributed/http/tests/test_core.py::test_prometheus_api_doc --deselect=distributed/tests/test_init.py::test_git_revision --deselect=distributed/tests/test_jupyter.py::test_jupyter_idle_timeout --deselect=distributed/tests/test_jupyter.py::test_jupyter_idle_timeout_returned --deselect=distributed/tests/test_jupyter.py::test_jupyter_server --deselect=distributed/tests/test_jupyter.py::test_shutsdown_cleanly --deselect=distributed/tests/test_profile.py::test_stack_overflow --deselect=distributed/tests/test_pubsub.py::test_client_worker --deselect=distributed/tests/test_queues.py::test_queue_in_task --deselect=distributed/tests/test_spill.py::test_spillbuffer_oserror --deselect=distributed/tests/test_steal.py::test_steal_twice --deselect=distributed/tests/test_utils_test.py::test_cluster --deselect=distributed/tests/test_variable.py::test_variable_in_task --deselect=distributed/tests/test_worker.py::test_process_executor_kills_process --deselect=distributed/tests/test_worker_memory.py::test_fail_to_pickle_execute_1 --deselect=distributed/tests/test_worker_state_machine.py::test_task_state_instance_are_garbage_collected --deselect=distributed/protocol/tests/test_protocol.py::test_deeply_nested_structures --deselect=distributed/protocol/tests/test_serialize.py::test_deeply_nested_structures --deselect=distributed/cli/tests/test_dask_scheduler.py::test_defaults --deselect=distributed/cli/tests/test_dask_scheduler.py::test_hostport --deselect=distributed/cli/tests/test_dask_spec.py::test_errors --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_file --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_file --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_file --deselect=distributed/dashboard/tests/test_scheduler_bokeh.py::test_counters --deselect=distributed/dashboard/tests/test_worker_bokeh.py::test_counters --deselect=distributed/deploy/tests/test_local.py::test_adapt_then_manual --deselect=distributed/deploy/tests/test_local.py::test_async_with --deselect=distributed/deploy/tests/test_local.py::test_close_twice --deselect=distributed/deploy/tests/test_local.py::test_cluster_info_sync --deselect=distributed/deploy/tests/test_local.py::test_local_tls --deselect=distributed/deploy/tests/test_local.py::test_no_dangling_asyncio_tasks --deselect=distributed/deploy/tests/test_local.py::test_only_local_access --deselect=distributed/deploy/tests/test_local.py::test_remote_access --deselect=distributed/diagnostics/tests/test_progress_widgets.py::test_serializers --deselect=distributed/diagnostics/tests/test_scheduler_plugin.py::test_lifecycle --deselect=distributed/http/scheduler/tests/test_missing_bokeh.py::test_missing_bokeh --deselect=distributed/http/scheduler/tests/test_scheduler_http.py::test_metrics_when_prometheus_client_not_installed --deselect=distributed/protocol/tests/test_serialize.py::test_errors --deselect=distributed/tests/test_batched.py::test_BatchedSend --deselect=distributed/tests/test_batched.py::test_close_closed --deselect=distributed/tests/test_batched.py::test_close_twice --deselect=distributed/tests/test_batched.py::test_send_after_stream_start --deselect=distributed/tests/test_batched.py::test_send_before_close --deselect=distributed/tests/test_batched.py::test_send_before_start --deselect=distributed/tests/test_batched.py::test_sending_traffic_jam --deselect=distributed/tests/test_batched.py::test_serializers --deselect=distributed/tests/test_client.py::TestClientSecurityLoader::test_security_loader --deselect=distributed/tests/test_client.py::TestClientSecurityLoader::test_security_loader_ignored_if_explicit_security_provided --deselect=distributed/tests/test_client.py::TestClientSecurityLoader::test_security_loader_ignored_if_returns_none --deselect=distributed/tests/test_client.py::test_async_with --deselect=distributed/tests/test_client.py::test_client_is_quiet_cluster_close --deselect=distributed/tests/test_client.py::test_dashboard_link_cluster --deselect=distributed/tests/test_client.py::test_dashboard_link_inproc --deselect=distributed/tests/test_client.py::test_file_descriptors_dont_leak --deselect=distributed/tests/test_client.py::test_mixing_clients_different_scheduler --deselect=distributed/tests/test_client.py::test_quiet_client_close --deselect=distributed/tests/test_client.py::test_rebalance_sync --deselect=distributed/tests/test_client.py::test_repr_localcluster --deselect=distributed/tests/test_client.py::test_security_loader --deselect=distributed/tests/test_client.py::test_security_loader_ignored_if_explicit_security_provided --deselect=distributed/tests/test_client.py::test_security_loader_ignored_if_returns_none --deselect=distributed/tests/test_client.py::test_shutdown --deselect=distributed/tests/test_client.py::test_shutdown_is_quiet_with_cluster --deselect=distributed/tests/test_client.py::test_shutdown_localcluster --deselect=distributed/tests/test_client.py::test_shutdown_stops_callbacks --deselect=distributed/tests/test_client_loop.py::test_close_loop_sync_start_new_loop --deselect=distributed/tests/test_client_loop.py::test_close_loop_sync_use_running_loop --deselect=distributed/tests/test_core.py::test_close_fast_without_active_handlers --deselect=distributed/tests/test_core.py::test_close_grace_period_for_handlers --deselect=distributed/tests/test_core.py::test_close_properly --deselect=distributed/tests/test_core.py::test_compression --deselect=distributed/tests/test_core.py::test_connection_pool --deselect=distributed/tests/test_core.py::test_connection_pool_close_while_connecting --deselect=distributed/tests/test_core.py::test_connection_pool_detects_remote_close --deselect=distributed/tests/test_core.py::test_connection_pool_outside_cancellation --deselect=distributed/tests/test_core.py::test_connection_pool_remove --deselect=distributed/tests/test_core.py::test_connection_pool_respects_limit --deselect=distributed/tests/test_core.py::test_connection_pool_tls --deselect=distributed/tests/test_core.py::test_counters --deselect=distributed/tests/test_core.py::test_deserialize_error --deselect=distributed/tests/test_core.py::test_errors --deselect=distributed/tests/test_core.py::test_identity_inproc --deselect=distributed/tests/test_core.py::test_identity_tcp --deselect=distributed/tests/test_core.py::test_large_packets_inproc --deselect=distributed/tests/test_core.py::test_messages_are_ordered_bsend --deselect=distributed/tests/test_core.py::test_messages_are_ordered_raw --deselect=distributed/tests/test_core.py::test_ports --deselect=distributed/tests/test_core.py::test_rpc_default --deselect=distributed/tests/test_core.py::test_rpc_inproc --deselect=distributed/tests/test_core.py::test_rpc_message_lifetime_default --deselect=distributed/tests/test_core.py::test_rpc_message_lifetime_inproc --deselect=distributed/tests/test_core.py::test_rpc_message_lifetime_tcp --deselect=distributed/tests/test_core.py::test_rpc_serialization --deselect=distributed/tests/test_core.py::test_rpc_tcp --deselect=distributed/tests/test_core.py::test_rpc_tls --deselect=distributed/tests/test_core.py::test_rpc_with_many_connections_inproc --deselect=distributed/tests/test_core.py::test_rpc_with_many_connections_tcp --deselect=distributed/tests/test_core.py::test_send_recv_args --deselect=distributed/tests/test_core.py::test_send_recv_cancelled --deselect=distributed/tests/test_core.py::test_server --deselect=distributed/tests/test_core.py::test_server_comms_mark_active_handlers --deselect=distributed/tests/test_core.py::test_server_raises_on_blocked_handlers --deselect=distributed/tests/test_jupyter.py::test_jupyter_idle_timeout --deselect=distributed/tests/test_jupyter.py::test_jupyter_server --deselect=distributed/tests/test_locks.py::test_errors --deselect=distributed/tests/test_nanny.py::test_scheduler_file --deselect=distributed/tests/test_nanny.py::test_scheduler_file --deselect=distributed/tests/test_nanny.py::test_scheduler_file --deselect=distributed/tests/test_nanny.py::test_worker_uses_same_host_as_nanny --deselect=distributed/tests/test_preload.py::test_failure_doesnt_crash_scheduler --deselect=distributed/tests/test_preload.py::test_preload_import_time --deselect=distributed/tests/test_preload.py::test_preload_manager_sequence --deselect=distributed/tests/test_preload.py::test_worker_preload_text --deselect=distributed/tests/test_scheduler.py::test_allowed_failures_config --deselect=distributed/tests/test_scheduler.py::test_async_context_manager --deselect=distributed/tests/test_scheduler.py::test_dashboard_host --deselect=distributed/tests/test_scheduler.py::test_file_descriptors_dont_leak --deselect=distributed/tests/test_scheduler.py::test_finished --deselect=distributed/tests/test_scheduler.py::test_multiple_listeners --deselect=distributed/tests/test_scheduler.py::test_no_dangling_asyncio_tasks --deselect=distributed/tests/test_scheduler.py::test_scheduler_file --deselect=distributed/tests/test_scheduler.py::test_scheduler_file --deselect=distributed/tests/test_scheduler.py::test_scheduler_file --deselect=distributed/tests/test_security.py::test_require_encryption --deselect=distributed/tests/test_security.py::test_tls_listen_connect --deselect=distributed/tests/test_security.py::test_tls_temporary_credentials_functional --deselect=distributed/tests/test_semaphore.py::test_threadpoolworkers_pick_correct_ioloop --deselect=distributed/tests/test_tls_functional.py::test_security_dict_input_no_security --deselect=distributed/tests/test_utils_test.py::test_ensure_no_new_clients --deselect=distributed/tests/test_utils_test.py::test_freeze_batched_send --deselect=distributed/tests/test_utils_test.py::test_locked_comm_drop_in_replacement --deselect=distributed/tests/test_utils_test.py::test_locked_comm_intercept_read --deselect=distributed/tests/test_utils_test.py::test_locked_comm_intercept_write --deselect=distributed/tests/test_worker.py::test_host_uses_scheduler_protocol --deselect=distributed/tests/test_worker.py::test_plugin_exception --deselect=distributed/tests/test_worker.py::test_plugin_internal_exception --deselect=distributed/tests/test_worker.py::test_plugin_multiple_exceptions --deselect=distributed/tests/test_worker.py::test_scheduler_file --deselect=distributed/tests/test_worker.py::test_scheduler_file --deselect=distributed/tests/test_worker.py::test_scheduler_file --deselect=distributed/tests/test_worker_client.py::test_dont_override_default_get --deselect=distributed/cli/tests/test_dask_scheduler.py::test_dashboard --deselect=distributed/cli/tests/test_dask_scheduler.py::test_dashboard_allowlist --deselect=distributed/cli/tests/test_dask_scheduler.py::test_dashboard_non_standard_ports --deselect=distributed/cli/tests/test_dask_scheduler.py::test_dashboard_port_zero --deselect=distributed/cli/tests/test_dask_scheduler.py::test_defaults --deselect=distributed/cli/tests/test_dask_scheduler.py::test_hostport --deselect=distributed/cli/tests/test_dask_scheduler.py::test_multiple_protocols --deselect=distributed/cli/tests/test_dask_scheduler.py::test_multiple_workers --deselect=distributed/cli/tests/test_dask_scheduler.py::test_multiple_workers_2 --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_command --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_command_default --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_config --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_file --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_module --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_remote_module --deselect=distributed/cli/tests/test_dask_scheduler.py::test_scheduler_port_zero --deselect=distributed/cli/tests/test_dask_scheduler.py::test_single_executable_deprecated --deselect=distributed/cli/tests/test_dask_worker.py::test_contact_listen_address --deselect=distributed/cli/tests/test_dask_worker.py::test_dashboard_non_standard_ports --deselect=distributed/cli/tests/test_dask_worker.py::test_error_during_startup --deselect=distributed/cli/tests/test_dask_worker.py::test_integer_names --deselect=distributed/cli/tests/test_dask_worker.py::test_listen_address_ipv6 --deselect=distributed/cli/tests/test_dask_worker.py::test_local_directory --deselect=distributed/cli/tests/test_dask_worker.py::test_memory_limit --deselect=distributed/cli/tests/test_dask_worker.py::test_nanny_worker_port_range --deselect=distributed/cli/tests/test_dask_worker.py::test_nanny_worker_port_range_too_many_workers_raises --deselect=distributed/cli/tests/test_dask_worker.py::test_nanny_worker_ports --deselect=distributed/cli/tests/test_dask_worker.py::test_no_nanny --deselect=distributed/cli/tests/test_dask_worker.py::test_nworkers_auto --deselect=distributed/cli/tests/test_dask_worker.py::test_nworkers_expands_name --deselect=distributed/cli/tests/test_dask_worker.py::test_nworkers_negative --deselect=distributed/cli/tests/test_dask_worker.py::test_nworkers_requires_nanny --deselect=distributed/cli/tests/test_dask_worker.py::test_preload_config --deselect=distributed/cli/tests/test_dask_worker.py::test_resources --deselect=distributed/cli/tests/test_dask_worker.py::test_respect_host_listen_address --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_address_env --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_file --deselect=distributed/cli/tests/test_dask_worker.py::test_set_lifetime_restart_via_env_var --deselect=distributed/cli/tests/test_dask_worker.py::test_set_lifetime_stagger_via_env_var --deselect=distributed/cli/tests/test_dask_worker.py::test_signal_handling --deselect=distributed/cli/tests/test_dask_worker.py::test_single_executable_deprecated --deselect=distributed/cli/tests/test_dask_worker.py::test_single_executable_works --deselect=distributed/cli/tests/test_dask_worker.py::test_timeout --deselect=distributed/cli/tests/test_dask_worker.py::test_worker_class --deselect=distributed/tests/test_config.py::test_logging_extended --deselect=distributed/tests/test_config.py::test_logging_file_config --deselect=distributed/tests/test_config.py::test_logging_mutual_exclusive --deselect=distributed/tests/test_config.py::test_logging_simple --deselect=distributed/tests/test_config.py::test_logging_simple_under_distributed --deselect=distributed/tests/test_queues.py::test_queue_in_task --deselect=distributed/tests/test_variable.py::test_variable_in_task +============================= test session starts ============================== +platform linux -- Python 3.13.1, pytest-8.3.4, pluggy-1.5.0 -- /usr/bin/python3.13 +cachedir: .pytest_cache +rootdir: /build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.13_distributed/build +configfile: pyproject.toml +plugins: rerunfailures-14.0, anyio-4.7.0, timeout-2.3.1, typeguard-4.4.1 +timeout: 300.0s +timeout method: signal +timeout func_only: False +collecting ... collected 46 items / 1 error / 41 deselected / 5 skipped / 5 selected +run-last-failure: rerun previous 5 failures (skipped 129 files) + +==================================== ERRORS ==================================== +__________ ERROR collecting distributed/shuffle/tests/test_rechunk.py __________ +distributed/shuffle/tests/test_rechunk.py:39: in + from distributed.shuffle.tests.utils import AbstractShuffleTestPool +distributed/shuffle/tests/utils.py:11: in + import dask.dataframe as dd +/usr/lib/python3/dist-packages/dask/dataframe/__init__.py:115: in + if _dask_expr_enabled(): +/usr/lib/python3/dist-packages/dask/dataframe/__init__.py:31: in _dask_expr_enabled + warnings.warn( +E FutureWarning: The legacy Dask DataFrame implementation is deprecated and will be removed in a future version. Set the configuration option `dataframe.query-planning` to `True` or None to enable the new Dask Dataframe implementation and silence this warning. +=========================== short test summary info ============================ +ERROR distributed/shuffle/tests/test_rechunk.py - FutureWarning: The legacy Dask DataFrame implementation is deprecated and will be removed in a future version. Set the configuration option `dataframe.query-planning` to `True` or None to enable the new Dask Dataframe implementation and silence this warning. +!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!! +================== 5 skipped, 41 deselected, 1 error in 2.01s ================== +*** END OF RUN 3: NOT ALL TESTS HAVE YET PASSED/XFAILED *** +*** STARTING RUN 4: python3.13 -m pytest --pyargs distributed --verbose --color=no --timeout-method=signal --timeout=300 -m not avoid_ci -rfE --last-failed --last-failed-no-failures none --ignore=distributed/comm/tests/test_comms.py --ignore=distributed/comm/tests/test_ws.py --ignore=distributed/deploy/tests/test_adaptive.py --ignore=distributed/deploy/tests/test_local.py --ignore=distributed/deploy/tests/test_slow_adaptive.py --ignore=distributed/deploy/tests/test_spec_cluster.py --deselect=distributed/cli/tests/test_dask_scheduler.py::test_no_dashboard --deselect=distributed/deploy/tests/test_local.py::test_localcluster_get_client --deselect=distributed/deploy/tests/test_old_ssh.py::test_cluster --deselect=distributed/deploy/tests/test_old_ssh.py::test_old_ssh_nprocs_renamed_to_n_workers --deselect=distributed/deploy/tests/test_old_ssh.py::test_nprocs_attribute_is_deprecated --deselect=distributed/deploy/tests/test_ssh.py::test_nprocs_attribute_is_deprecated --deselect=distributed/http/tests/test_core.py::test_prometheus_api_doc --deselect=distributed/tests/test_init.py::test_git_revision --deselect=distributed/tests/test_jupyter.py::test_jupyter_idle_timeout --deselect=distributed/tests/test_jupyter.py::test_jupyter_idle_timeout_returned --deselect=distributed/tests/test_jupyter.py::test_jupyter_server --deselect=distributed/tests/test_jupyter.py::test_shutsdown_cleanly --deselect=distributed/tests/test_profile.py::test_stack_overflow --deselect=distributed/tests/test_pubsub.py::test_client_worker --deselect=distributed/tests/test_queues.py::test_queue_in_task --deselect=distributed/tests/test_spill.py::test_spillbuffer_oserror --deselect=distributed/tests/test_steal.py::test_steal_twice --deselect=distributed/tests/test_utils_test.py::test_cluster --deselect=distributed/tests/test_variable.py::test_variable_in_task --deselect=distributed/tests/test_worker.py::test_process_executor_kills_process --deselect=distributed/tests/test_worker_memory.py::test_fail_to_pickle_execute_1 --deselect=distributed/tests/test_worker_state_machine.py::test_task_state_instance_are_garbage_collected --deselect=distributed/protocol/tests/test_protocol.py::test_deeply_nested_structures --deselect=distributed/protocol/tests/test_serialize.py::test_deeply_nested_structures --deselect=distributed/cli/tests/test_dask_scheduler.py::test_defaults --deselect=distributed/cli/tests/test_dask_scheduler.py::test_hostport --deselect=distributed/cli/tests/test_dask_spec.py::test_errors --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_file --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_file --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_file --deselect=distributed/dashboard/tests/test_scheduler_bokeh.py::test_counters --deselect=distributed/dashboard/tests/test_worker_bokeh.py::test_counters --deselect=distributed/deploy/tests/test_local.py::test_adapt_then_manual --deselect=distributed/deploy/tests/test_local.py::test_async_with --deselect=distributed/deploy/tests/test_local.py::test_close_twice --deselect=distributed/deploy/tests/test_local.py::test_cluster_info_sync --deselect=distributed/deploy/tests/test_local.py::test_local_tls --deselect=distributed/deploy/tests/test_local.py::test_no_dangling_asyncio_tasks --deselect=distributed/deploy/tests/test_local.py::test_only_local_access --deselect=distributed/deploy/tests/test_local.py::test_remote_access --deselect=distributed/diagnostics/tests/test_progress_widgets.py::test_serializers --deselect=distributed/diagnostics/tests/test_scheduler_plugin.py::test_lifecycle --deselect=distributed/http/scheduler/tests/test_missing_bokeh.py::test_missing_bokeh --deselect=distributed/http/scheduler/tests/test_scheduler_http.py::test_metrics_when_prometheus_client_not_installed --deselect=distributed/protocol/tests/test_serialize.py::test_errors --deselect=distributed/tests/test_batched.py::test_BatchedSend --deselect=distributed/tests/test_batched.py::test_close_closed --deselect=distributed/tests/test_batched.py::test_close_twice --deselect=distributed/tests/test_batched.py::test_send_after_stream_start --deselect=distributed/tests/test_batched.py::test_send_before_close --deselect=distributed/tests/test_batched.py::test_send_before_start --deselect=distributed/tests/test_batched.py::test_sending_traffic_jam --deselect=distributed/tests/test_batched.py::test_serializers --deselect=distributed/tests/test_client.py::TestClientSecurityLoader::test_security_loader --deselect=distributed/tests/test_client.py::TestClientSecurityLoader::test_security_loader_ignored_if_explicit_security_provided --deselect=distributed/tests/test_client.py::TestClientSecurityLoader::test_security_loader_ignored_if_returns_none --deselect=distributed/tests/test_client.py::test_async_with --deselect=distributed/tests/test_client.py::test_client_is_quiet_cluster_close --deselect=distributed/tests/test_client.py::test_dashboard_link_cluster --deselect=distributed/tests/test_client.py::test_dashboard_link_inproc --deselect=distributed/tests/test_client.py::test_file_descriptors_dont_leak --deselect=distributed/tests/test_client.py::test_mixing_clients_different_scheduler --deselect=distributed/tests/test_client.py::test_quiet_client_close --deselect=distributed/tests/test_client.py::test_rebalance_sync --deselect=distributed/tests/test_client.py::test_repr_localcluster --deselect=distributed/tests/test_client.py::test_security_loader --deselect=distributed/tests/test_client.py::test_security_loader_ignored_if_explicit_security_provided --deselect=distributed/tests/test_client.py::test_security_loader_ignored_if_returns_none --deselect=distributed/tests/test_client.py::test_shutdown --deselect=distributed/tests/test_client.py::test_shutdown_is_quiet_with_cluster --deselect=distributed/tests/test_client.py::test_shutdown_localcluster --deselect=distributed/tests/test_client.py::test_shutdown_stops_callbacks --deselect=distributed/tests/test_client_loop.py::test_close_loop_sync_start_new_loop --deselect=distributed/tests/test_client_loop.py::test_close_loop_sync_use_running_loop --deselect=distributed/tests/test_core.py::test_close_fast_without_active_handlers --deselect=distributed/tests/test_core.py::test_close_grace_period_for_handlers --deselect=distributed/tests/test_core.py::test_close_properly --deselect=distributed/tests/test_core.py::test_compression --deselect=distributed/tests/test_core.py::test_connection_pool --deselect=distributed/tests/test_core.py::test_connection_pool_close_while_connecting --deselect=distributed/tests/test_core.py::test_connection_pool_detects_remote_close --deselect=distributed/tests/test_core.py::test_connection_pool_outside_cancellation --deselect=distributed/tests/test_core.py::test_connection_pool_remove --deselect=distributed/tests/test_core.py::test_connection_pool_respects_limit --deselect=distributed/tests/test_core.py::test_connection_pool_tls --deselect=distributed/tests/test_core.py::test_counters --deselect=distributed/tests/test_core.py::test_deserialize_error --deselect=distributed/tests/test_core.py::test_errors --deselect=distributed/tests/test_core.py::test_identity_inproc --deselect=distributed/tests/test_core.py::test_identity_tcp --deselect=distributed/tests/test_core.py::test_large_packets_inproc --deselect=distributed/tests/test_core.py::test_messages_are_ordered_bsend --deselect=distributed/tests/test_core.py::test_messages_are_ordered_raw --deselect=distributed/tests/test_core.py::test_ports --deselect=distributed/tests/test_core.py::test_rpc_default --deselect=distributed/tests/test_core.py::test_rpc_inproc --deselect=distributed/tests/test_core.py::test_rpc_message_lifetime_default --deselect=distributed/tests/test_core.py::test_rpc_message_lifetime_inproc --deselect=distributed/tests/test_core.py::test_rpc_message_lifetime_tcp --deselect=distributed/tests/test_core.py::test_rpc_serialization --deselect=distributed/tests/test_core.py::test_rpc_tcp --deselect=distributed/tests/test_core.py::test_rpc_tls --deselect=distributed/tests/test_core.py::test_rpc_with_many_connections_inproc --deselect=distributed/tests/test_core.py::test_rpc_with_many_connections_tcp --deselect=distributed/tests/test_core.py::test_send_recv_args --deselect=distributed/tests/test_core.py::test_send_recv_cancelled --deselect=distributed/tests/test_core.py::test_server --deselect=distributed/tests/test_core.py::test_server_comms_mark_active_handlers --deselect=distributed/tests/test_core.py::test_server_raises_on_blocked_handlers --deselect=distributed/tests/test_jupyter.py::test_jupyter_idle_timeout --deselect=distributed/tests/test_jupyter.py::test_jupyter_server --deselect=distributed/tests/test_locks.py::test_errors --deselect=distributed/tests/test_nanny.py::test_scheduler_file --deselect=distributed/tests/test_nanny.py::test_scheduler_file --deselect=distributed/tests/test_nanny.py::test_scheduler_file --deselect=distributed/tests/test_nanny.py::test_worker_uses_same_host_as_nanny --deselect=distributed/tests/test_preload.py::test_failure_doesnt_crash_scheduler --deselect=distributed/tests/test_preload.py::test_preload_import_time --deselect=distributed/tests/test_preload.py::test_preload_manager_sequence --deselect=distributed/tests/test_preload.py::test_worker_preload_text --deselect=distributed/tests/test_scheduler.py::test_allowed_failures_config --deselect=distributed/tests/test_scheduler.py::test_async_context_manager --deselect=distributed/tests/test_scheduler.py::test_dashboard_host --deselect=distributed/tests/test_scheduler.py::test_file_descriptors_dont_leak --deselect=distributed/tests/test_scheduler.py::test_finished --deselect=distributed/tests/test_scheduler.py::test_multiple_listeners --deselect=distributed/tests/test_scheduler.py::test_no_dangling_asyncio_tasks --deselect=distributed/tests/test_scheduler.py::test_scheduler_file --deselect=distributed/tests/test_scheduler.py::test_scheduler_file --deselect=distributed/tests/test_scheduler.py::test_scheduler_file --deselect=distributed/tests/test_security.py::test_require_encryption --deselect=distributed/tests/test_security.py::test_tls_listen_connect --deselect=distributed/tests/test_security.py::test_tls_temporary_credentials_functional --deselect=distributed/tests/test_semaphore.py::test_threadpoolworkers_pick_correct_ioloop --deselect=distributed/tests/test_tls_functional.py::test_security_dict_input_no_security --deselect=distributed/tests/test_utils_test.py::test_ensure_no_new_clients --deselect=distributed/tests/test_utils_test.py::test_freeze_batched_send --deselect=distributed/tests/test_utils_test.py::test_locked_comm_drop_in_replacement --deselect=distributed/tests/test_utils_test.py::test_locked_comm_intercept_read --deselect=distributed/tests/test_utils_test.py::test_locked_comm_intercept_write --deselect=distributed/tests/test_worker.py::test_host_uses_scheduler_protocol --deselect=distributed/tests/test_worker.py::test_plugin_exception --deselect=distributed/tests/test_worker.py::test_plugin_internal_exception --deselect=distributed/tests/test_worker.py::test_plugin_multiple_exceptions --deselect=distributed/tests/test_worker.py::test_scheduler_file --deselect=distributed/tests/test_worker.py::test_scheduler_file --deselect=distributed/tests/test_worker.py::test_scheduler_file --deselect=distributed/tests/test_worker_client.py::test_dont_override_default_get --deselect=distributed/cli/tests/test_dask_scheduler.py::test_dashboard --deselect=distributed/cli/tests/test_dask_scheduler.py::test_dashboard_allowlist --deselect=distributed/cli/tests/test_dask_scheduler.py::test_dashboard_non_standard_ports --deselect=distributed/cli/tests/test_dask_scheduler.py::test_dashboard_port_zero --deselect=distributed/cli/tests/test_dask_scheduler.py::test_defaults --deselect=distributed/cli/tests/test_dask_scheduler.py::test_hostport --deselect=distributed/cli/tests/test_dask_scheduler.py::test_multiple_protocols --deselect=distributed/cli/tests/test_dask_scheduler.py::test_multiple_workers --deselect=distributed/cli/tests/test_dask_scheduler.py::test_multiple_workers_2 --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_command --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_command_default --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_config --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_file --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_module --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_remote_module --deselect=distributed/cli/tests/test_dask_scheduler.py::test_scheduler_port_zero --deselect=distributed/cli/tests/test_dask_scheduler.py::test_single_executable_deprecated --deselect=distributed/cli/tests/test_dask_worker.py::test_contact_listen_address --deselect=distributed/cli/tests/test_dask_worker.py::test_dashboard_non_standard_ports --deselect=distributed/cli/tests/test_dask_worker.py::test_error_during_startup --deselect=distributed/cli/tests/test_dask_worker.py::test_integer_names --deselect=distributed/cli/tests/test_dask_worker.py::test_listen_address_ipv6 --deselect=distributed/cli/tests/test_dask_worker.py::test_local_directory --deselect=distributed/cli/tests/test_dask_worker.py::test_memory_limit --deselect=distributed/cli/tests/test_dask_worker.py::test_nanny_worker_port_range --deselect=distributed/cli/tests/test_dask_worker.py::test_nanny_worker_port_range_too_many_workers_raises --deselect=distributed/cli/tests/test_dask_worker.py::test_nanny_worker_ports --deselect=distributed/cli/tests/test_dask_worker.py::test_no_nanny --deselect=distributed/cli/tests/test_dask_worker.py::test_nworkers_auto --deselect=distributed/cli/tests/test_dask_worker.py::test_nworkers_expands_name --deselect=distributed/cli/tests/test_dask_worker.py::test_nworkers_negative --deselect=distributed/cli/tests/test_dask_worker.py::test_nworkers_requires_nanny --deselect=distributed/cli/tests/test_dask_worker.py::test_preload_config --deselect=distributed/cli/tests/test_dask_worker.py::test_resources --deselect=distributed/cli/tests/test_dask_worker.py::test_respect_host_listen_address --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_address_env --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_file --deselect=distributed/cli/tests/test_dask_worker.py::test_set_lifetime_restart_via_env_var --deselect=distributed/cli/tests/test_dask_worker.py::test_set_lifetime_stagger_via_env_var --deselect=distributed/cli/tests/test_dask_worker.py::test_signal_handling --deselect=distributed/cli/tests/test_dask_worker.py::test_single_executable_deprecated --deselect=distributed/cli/tests/test_dask_worker.py::test_single_executable_works --deselect=distributed/cli/tests/test_dask_worker.py::test_timeout --deselect=distributed/cli/tests/test_dask_worker.py::test_worker_class --deselect=distributed/tests/test_config.py::test_logging_extended --deselect=distributed/tests/test_config.py::test_logging_file_config --deselect=distributed/tests/test_config.py::test_logging_mutual_exclusive --deselect=distributed/tests/test_config.py::test_logging_simple --deselect=distributed/tests/test_config.py::test_logging_simple_under_distributed --deselect=distributed/tests/test_queues.py::test_queue_in_task --deselect=distributed/tests/test_variable.py::test_variable_in_task +============================= test session starts ============================== +platform linux -- Python 3.13.1, pytest-8.3.4, pluggy-1.5.0 -- /usr/bin/python3.13 +cachedir: .pytest_cache +rootdir: /build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.13_distributed/build +configfile: pyproject.toml +plugins: rerunfailures-14.0, anyio-4.7.0, timeout-2.3.1, typeguard-4.4.1 +timeout: 300.0s +timeout method: signal +timeout func_only: False +collecting ... collected 46 items / 1 error / 41 deselected / 5 skipped / 5 selected +run-last-failure: rerun previous 5 failures (skipped 129 files) + +==================================== ERRORS ==================================== +__________ ERROR collecting distributed/shuffle/tests/test_rechunk.py __________ +distributed/shuffle/tests/test_rechunk.py:39: in + from distributed.shuffle.tests.utils import AbstractShuffleTestPool +distributed/shuffle/tests/utils.py:11: in + import dask.dataframe as dd +/usr/lib/python3/dist-packages/dask/dataframe/__init__.py:115: in + if _dask_expr_enabled(): +/usr/lib/python3/dist-packages/dask/dataframe/__init__.py:31: in _dask_expr_enabled + warnings.warn( +E FutureWarning: The legacy Dask DataFrame implementation is deprecated and will be removed in a future version. Set the configuration option `dataframe.query-planning` to `True` or None to enable the new Dask Dataframe implementation and silence this warning. +=========================== short test summary info ============================ +ERROR distributed/shuffle/tests/test_rechunk.py - FutureWarning: The legacy Dask DataFrame implementation is deprecated and will be removed in a future version. Set the configuration option `dataframe.query-planning` to `True` or None to enable the new Dask Dataframe implementation and silence this warning. +!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!! +================== 5 skipped, 41 deselected, 1 error in 2.29s ================== +*** END OF RUN 4: NOT ALL TESTS HAVE YET PASSED/XFAILED *** +*** STARTING RUN 5: python3.13 -m pytest --pyargs distributed --verbose --color=no --timeout-method=signal --timeout=300 -m not avoid_ci -rfE --last-failed --last-failed-no-failures none --ignore=distributed/comm/tests/test_comms.py --ignore=distributed/comm/tests/test_ws.py --ignore=distributed/deploy/tests/test_adaptive.py --ignore=distributed/deploy/tests/test_local.py --ignore=distributed/deploy/tests/test_slow_adaptive.py --ignore=distributed/deploy/tests/test_spec_cluster.py --deselect=distributed/cli/tests/test_dask_scheduler.py::test_no_dashboard --deselect=distributed/deploy/tests/test_local.py::test_localcluster_get_client --deselect=distributed/deploy/tests/test_old_ssh.py::test_cluster --deselect=distributed/deploy/tests/test_old_ssh.py::test_old_ssh_nprocs_renamed_to_n_workers --deselect=distributed/deploy/tests/test_old_ssh.py::test_nprocs_attribute_is_deprecated --deselect=distributed/deploy/tests/test_ssh.py::test_nprocs_attribute_is_deprecated --deselect=distributed/http/tests/test_core.py::test_prometheus_api_doc --deselect=distributed/tests/test_init.py::test_git_revision --deselect=distributed/tests/test_jupyter.py::test_jupyter_idle_timeout --deselect=distributed/tests/test_jupyter.py::test_jupyter_idle_timeout_returned --deselect=distributed/tests/test_jupyter.py::test_jupyter_server --deselect=distributed/tests/test_jupyter.py::test_shutsdown_cleanly --deselect=distributed/tests/test_profile.py::test_stack_overflow --deselect=distributed/tests/test_pubsub.py::test_client_worker --deselect=distributed/tests/test_queues.py::test_queue_in_task --deselect=distributed/tests/test_spill.py::test_spillbuffer_oserror --deselect=distributed/tests/test_steal.py::test_steal_twice --deselect=distributed/tests/test_utils_test.py::test_cluster --deselect=distributed/tests/test_variable.py::test_variable_in_task --deselect=distributed/tests/test_worker.py::test_process_executor_kills_process --deselect=distributed/tests/test_worker_memory.py::test_fail_to_pickle_execute_1 --deselect=distributed/tests/test_worker_state_machine.py::test_task_state_instance_are_garbage_collected --deselect=distributed/protocol/tests/test_protocol.py::test_deeply_nested_structures --deselect=distributed/protocol/tests/test_serialize.py::test_deeply_nested_structures --deselect=distributed/cli/tests/test_dask_scheduler.py::test_defaults --deselect=distributed/cli/tests/test_dask_scheduler.py::test_hostport --deselect=distributed/cli/tests/test_dask_spec.py::test_errors --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_file --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_file --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_file --deselect=distributed/dashboard/tests/test_scheduler_bokeh.py::test_counters --deselect=distributed/dashboard/tests/test_worker_bokeh.py::test_counters --deselect=distributed/deploy/tests/test_local.py::test_adapt_then_manual --deselect=distributed/deploy/tests/test_local.py::test_async_with --deselect=distributed/deploy/tests/test_local.py::test_close_twice --deselect=distributed/deploy/tests/test_local.py::test_cluster_info_sync --deselect=distributed/deploy/tests/test_local.py::test_local_tls --deselect=distributed/deploy/tests/test_local.py::test_no_dangling_asyncio_tasks --deselect=distributed/deploy/tests/test_local.py::test_only_local_access --deselect=distributed/deploy/tests/test_local.py::test_remote_access --deselect=distributed/diagnostics/tests/test_progress_widgets.py::test_serializers --deselect=distributed/diagnostics/tests/test_scheduler_plugin.py::test_lifecycle --deselect=distributed/http/scheduler/tests/test_missing_bokeh.py::test_missing_bokeh --deselect=distributed/http/scheduler/tests/test_scheduler_http.py::test_metrics_when_prometheus_client_not_installed --deselect=distributed/protocol/tests/test_serialize.py::test_errors --deselect=distributed/tests/test_batched.py::test_BatchedSend --deselect=distributed/tests/test_batched.py::test_close_closed --deselect=distributed/tests/test_batched.py::test_close_twice --deselect=distributed/tests/test_batched.py::test_send_after_stream_start --deselect=distributed/tests/test_batched.py::test_send_before_close --deselect=distributed/tests/test_batched.py::test_send_before_start --deselect=distributed/tests/test_batched.py::test_sending_traffic_jam --deselect=distributed/tests/test_batched.py::test_serializers --deselect=distributed/tests/test_client.py::TestClientSecurityLoader::test_security_loader --deselect=distributed/tests/test_client.py::TestClientSecurityLoader::test_security_loader_ignored_if_explicit_security_provided --deselect=distributed/tests/test_client.py::TestClientSecurityLoader::test_security_loader_ignored_if_returns_none --deselect=distributed/tests/test_client.py::test_async_with --deselect=distributed/tests/test_client.py::test_client_is_quiet_cluster_close --deselect=distributed/tests/test_client.py::test_dashboard_link_cluster --deselect=distributed/tests/test_client.py::test_dashboard_link_inproc --deselect=distributed/tests/test_client.py::test_file_descriptors_dont_leak --deselect=distributed/tests/test_client.py::test_mixing_clients_different_scheduler --deselect=distributed/tests/test_client.py::test_quiet_client_close --deselect=distributed/tests/test_client.py::test_rebalance_sync --deselect=distributed/tests/test_client.py::test_repr_localcluster --deselect=distributed/tests/test_client.py::test_security_loader --deselect=distributed/tests/test_client.py::test_security_loader_ignored_if_explicit_security_provided --deselect=distributed/tests/test_client.py::test_security_loader_ignored_if_returns_none --deselect=distributed/tests/test_client.py::test_shutdown --deselect=distributed/tests/test_client.py::test_shutdown_is_quiet_with_cluster --deselect=distributed/tests/test_client.py::test_shutdown_localcluster --deselect=distributed/tests/test_client.py::test_shutdown_stops_callbacks --deselect=distributed/tests/test_client_loop.py::test_close_loop_sync_start_new_loop --deselect=distributed/tests/test_client_loop.py::test_close_loop_sync_use_running_loop --deselect=distributed/tests/test_core.py::test_close_fast_without_active_handlers --deselect=distributed/tests/test_core.py::test_close_grace_period_for_handlers --deselect=distributed/tests/test_core.py::test_close_properly --deselect=distributed/tests/test_core.py::test_compression --deselect=distributed/tests/test_core.py::test_connection_pool --deselect=distributed/tests/test_core.py::test_connection_pool_close_while_connecting --deselect=distributed/tests/test_core.py::test_connection_pool_detects_remote_close --deselect=distributed/tests/test_core.py::test_connection_pool_outside_cancellation --deselect=distributed/tests/test_core.py::test_connection_pool_remove --deselect=distributed/tests/test_core.py::test_connection_pool_respects_limit --deselect=distributed/tests/test_core.py::test_connection_pool_tls --deselect=distributed/tests/test_core.py::test_counters --deselect=distributed/tests/test_core.py::test_deserialize_error --deselect=distributed/tests/test_core.py::test_errors --deselect=distributed/tests/test_core.py::test_identity_inproc --deselect=distributed/tests/test_core.py::test_identity_tcp --deselect=distributed/tests/test_core.py::test_large_packets_inproc --deselect=distributed/tests/test_core.py::test_messages_are_ordered_bsend --deselect=distributed/tests/test_core.py::test_messages_are_ordered_raw --deselect=distributed/tests/test_core.py::test_ports --deselect=distributed/tests/test_core.py::test_rpc_default --deselect=distributed/tests/test_core.py::test_rpc_inproc --deselect=distributed/tests/test_core.py::test_rpc_message_lifetime_default --deselect=distributed/tests/test_core.py::test_rpc_message_lifetime_inproc --deselect=distributed/tests/test_core.py::test_rpc_message_lifetime_tcp --deselect=distributed/tests/test_core.py::test_rpc_serialization --deselect=distributed/tests/test_core.py::test_rpc_tcp --deselect=distributed/tests/test_core.py::test_rpc_tls --deselect=distributed/tests/test_core.py::test_rpc_with_many_connections_inproc --deselect=distributed/tests/test_core.py::test_rpc_with_many_connections_tcp --deselect=distributed/tests/test_core.py::test_send_recv_args --deselect=distributed/tests/test_core.py::test_send_recv_cancelled --deselect=distributed/tests/test_core.py::test_server --deselect=distributed/tests/test_core.py::test_server_comms_mark_active_handlers --deselect=distributed/tests/test_core.py::test_server_raises_on_blocked_handlers --deselect=distributed/tests/test_jupyter.py::test_jupyter_idle_timeout --deselect=distributed/tests/test_jupyter.py::test_jupyter_server --deselect=distributed/tests/test_locks.py::test_errors --deselect=distributed/tests/test_nanny.py::test_scheduler_file --deselect=distributed/tests/test_nanny.py::test_scheduler_file --deselect=distributed/tests/test_nanny.py::test_scheduler_file --deselect=distributed/tests/test_nanny.py::test_worker_uses_same_host_as_nanny --deselect=distributed/tests/test_preload.py::test_failure_doesnt_crash_scheduler --deselect=distributed/tests/test_preload.py::test_preload_import_time --deselect=distributed/tests/test_preload.py::test_preload_manager_sequence --deselect=distributed/tests/test_preload.py::test_worker_preload_text --deselect=distributed/tests/test_scheduler.py::test_allowed_failures_config --deselect=distributed/tests/test_scheduler.py::test_async_context_manager --deselect=distributed/tests/test_scheduler.py::test_dashboard_host --deselect=distributed/tests/test_scheduler.py::test_file_descriptors_dont_leak --deselect=distributed/tests/test_scheduler.py::test_finished --deselect=distributed/tests/test_scheduler.py::test_multiple_listeners --deselect=distributed/tests/test_scheduler.py::test_no_dangling_asyncio_tasks --deselect=distributed/tests/test_scheduler.py::test_scheduler_file --deselect=distributed/tests/test_scheduler.py::test_scheduler_file --deselect=distributed/tests/test_scheduler.py::test_scheduler_file --deselect=distributed/tests/test_security.py::test_require_encryption --deselect=distributed/tests/test_security.py::test_tls_listen_connect --deselect=distributed/tests/test_security.py::test_tls_temporary_credentials_functional --deselect=distributed/tests/test_semaphore.py::test_threadpoolworkers_pick_correct_ioloop --deselect=distributed/tests/test_tls_functional.py::test_security_dict_input_no_security --deselect=distributed/tests/test_utils_test.py::test_ensure_no_new_clients --deselect=distributed/tests/test_utils_test.py::test_freeze_batched_send --deselect=distributed/tests/test_utils_test.py::test_locked_comm_drop_in_replacement --deselect=distributed/tests/test_utils_test.py::test_locked_comm_intercept_read --deselect=distributed/tests/test_utils_test.py::test_locked_comm_intercept_write --deselect=distributed/tests/test_worker.py::test_host_uses_scheduler_protocol --deselect=distributed/tests/test_worker.py::test_plugin_exception --deselect=distributed/tests/test_worker.py::test_plugin_internal_exception --deselect=distributed/tests/test_worker.py::test_plugin_multiple_exceptions --deselect=distributed/tests/test_worker.py::test_scheduler_file --deselect=distributed/tests/test_worker.py::test_scheduler_file --deselect=distributed/tests/test_worker.py::test_scheduler_file --deselect=distributed/tests/test_worker_client.py::test_dont_override_default_get --deselect=distributed/cli/tests/test_dask_scheduler.py::test_dashboard --deselect=distributed/cli/tests/test_dask_scheduler.py::test_dashboard_allowlist --deselect=distributed/cli/tests/test_dask_scheduler.py::test_dashboard_non_standard_ports --deselect=distributed/cli/tests/test_dask_scheduler.py::test_dashboard_port_zero --deselect=distributed/cli/tests/test_dask_scheduler.py::test_defaults --deselect=distributed/cli/tests/test_dask_scheduler.py::test_hostport --deselect=distributed/cli/tests/test_dask_scheduler.py::test_multiple_protocols --deselect=distributed/cli/tests/test_dask_scheduler.py::test_multiple_workers --deselect=distributed/cli/tests/test_dask_scheduler.py::test_multiple_workers_2 --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_command --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_command_default --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_config --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_file --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_module --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_remote_module --deselect=distributed/cli/tests/test_dask_scheduler.py::test_scheduler_port_zero --deselect=distributed/cli/tests/test_dask_scheduler.py::test_single_executable_deprecated --deselect=distributed/cli/tests/test_dask_worker.py::test_contact_listen_address --deselect=distributed/cli/tests/test_dask_worker.py::test_dashboard_non_standard_ports --deselect=distributed/cli/tests/test_dask_worker.py::test_error_during_startup --deselect=distributed/cli/tests/test_dask_worker.py::test_integer_names --deselect=distributed/cli/tests/test_dask_worker.py::test_listen_address_ipv6 --deselect=distributed/cli/tests/test_dask_worker.py::test_local_directory --deselect=distributed/cli/tests/test_dask_worker.py::test_memory_limit --deselect=distributed/cli/tests/test_dask_worker.py::test_nanny_worker_port_range --deselect=distributed/cli/tests/test_dask_worker.py::test_nanny_worker_port_range_too_many_workers_raises --deselect=distributed/cli/tests/test_dask_worker.py::test_nanny_worker_ports --deselect=distributed/cli/tests/test_dask_worker.py::test_no_nanny --deselect=distributed/cli/tests/test_dask_worker.py::test_nworkers_auto --deselect=distributed/cli/tests/test_dask_worker.py::test_nworkers_expands_name --deselect=distributed/cli/tests/test_dask_worker.py::test_nworkers_negative --deselect=distributed/cli/tests/test_dask_worker.py::test_nworkers_requires_nanny --deselect=distributed/cli/tests/test_dask_worker.py::test_preload_config --deselect=distributed/cli/tests/test_dask_worker.py::test_resources --deselect=distributed/cli/tests/test_dask_worker.py::test_respect_host_listen_address --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_address_env --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_file --deselect=distributed/cli/tests/test_dask_worker.py::test_set_lifetime_restart_via_env_var --deselect=distributed/cli/tests/test_dask_worker.py::test_set_lifetime_stagger_via_env_var --deselect=distributed/cli/tests/test_dask_worker.py::test_signal_handling --deselect=distributed/cli/tests/test_dask_worker.py::test_single_executable_deprecated --deselect=distributed/cli/tests/test_dask_worker.py::test_single_executable_works --deselect=distributed/cli/tests/test_dask_worker.py::test_timeout --deselect=distributed/cli/tests/test_dask_worker.py::test_worker_class --deselect=distributed/tests/test_config.py::test_logging_extended --deselect=distributed/tests/test_config.py::test_logging_file_config --deselect=distributed/tests/test_config.py::test_logging_mutual_exclusive --deselect=distributed/tests/test_config.py::test_logging_simple --deselect=distributed/tests/test_config.py::test_logging_simple_under_distributed --deselect=distributed/tests/test_queues.py::test_queue_in_task --deselect=distributed/tests/test_variable.py::test_variable_in_task +============================= test session starts ============================== +platform linux -- Python 3.13.1, pytest-8.3.4, pluggy-1.5.0 -- /usr/bin/python3.13 +cachedir: .pytest_cache +rootdir: /build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.13_distributed/build +configfile: pyproject.toml +plugins: rerunfailures-14.0, anyio-4.7.0, timeout-2.3.1, typeguard-4.4.1 +timeout: 300.0s +timeout method: signal +timeout func_only: False +collecting ... collected 46 items / 1 error / 41 deselected / 5 skipped / 5 selected +run-last-failure: rerun previous 5 failures (skipped 129 files) + +==================================== ERRORS ==================================== +__________ ERROR collecting distributed/shuffle/tests/test_rechunk.py __________ +distributed/shuffle/tests/test_rechunk.py:39: in + from distributed.shuffle.tests.utils import AbstractShuffleTestPool +distributed/shuffle/tests/utils.py:11: in + import dask.dataframe as dd +/usr/lib/python3/dist-packages/dask/dataframe/__init__.py:115: in + if _dask_expr_enabled(): +/usr/lib/python3/dist-packages/dask/dataframe/__init__.py:31: in _dask_expr_enabled + warnings.warn( +E FutureWarning: The legacy Dask DataFrame implementation is deprecated and will be removed in a future version. Set the configuration option `dataframe.query-planning` to `True` or None to enable the new Dask Dataframe implementation and silence this warning. +=========================== short test summary info ============================ +ERROR distributed/shuffle/tests/test_rechunk.py - FutureWarning: The legacy Dask DataFrame implementation is deprecated and will be removed in a future version. Set the configuration option `dataframe.query-planning` to `True` or None to enable the new Dask Dataframe implementation and silence this warning. +!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!! +================== 5 skipped, 41 deselected, 1 error in 1.92s ================== +*** END OF RUN 5: NOT ALL TESTS HAVE YET PASSED/XFAILED *** +*** SOME TESTS FAILED/ERRORED EVERY RUN, ABORTING *** +E: pybuild pybuild:389: test: plugin pyproject failed with: exit code=1: cd /build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.13_distributed/build; cd {build_dir}; /build/reproducible-path/dask.distributed-2024.12.1+ds/debian/run-tests /build/reproducible-path/dask.distributed-2024.12.1+ds/debian/get-test-exclusions rules {interpreter} -m pytest --pyargs distributed --verbose --color=no --timeout-method=signal --timeout=300 -m "not avoid_ci" I: pybuild pybuild:308: mkdir -pv /build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/.config && printf '\173 "dataframe": \173 "query-planning": False \175 \175\n' > /build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/.config/dask mkdir: created directory '/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/.config' I: pybuild base:311: cd /build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build; cd /build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build; /build/reproducible-path/dask.distributed-2024.12.1+ds/debian/run-tests /build/reproducible-path/dask.distributed-2024.12.1+ds/debian/get-test-exclusions rules python3.12 -m pytest --pyargs distributed --verbose --color=no --timeout-method=signal --timeout=300 -m "not avoid_ci" @@ -8223,7 +9986,7 @@ cachedir: .pytest_cache rootdir: /build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build configfile: pyproject.toml -plugins: typeguard-4.4.1, timeout-2.3.1, rerunfailures-14.0, anyio-4.7.0 +plugins: rerunfailures-14.0, anyio-4.7.0, timeout-2.3.1, typeguard-4.4.1 timeout: 300.0s timeout method: signal timeout func_only: False @@ -8844,7 +10607,7 @@ distributed/shuffle/tests/test_rechunk.py::test_rechunk_4d[False] PASSED [ 19%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_expand PASSED [ 19%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_expand2 PASSED [ 19%] -distributed/shuffle/tests/test_rechunk.py::test_rechunk_method PASSED [ 19%] +distributed/shuffle/tests/test_rechunk.py::test_rechunk_method FAILED [ 19%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_blockshape PASSED [ 19%] distributed/shuffle/tests/test_rechunk.py::test_dtype PASSED [ 19%] distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_dict PASSED [ 19%] @@ -9960,7 +11723,7 @@ distributed/tests/test_priorities.py::test_annotate_persist[queue on scheduler] PASSED [ 53%] distributed/tests/test_priorities.py::test_repeated_persists_same_priority[queue on worker] PASSED [ 53%] distributed/tests/test_priorities.py::test_repeated_persists_same_priority[queue on scheduler] PASSED [ 53%] -distributed/tests/test_priorities.py::test_last_in_first_out[queue on worker] PASSED [ 53%] +distributed/tests/test_priorities.py::test_last_in_first_out[queue on worker] FAILED [ 53%] distributed/tests/test_priorities.py::test_last_in_first_out[queue on scheduler] PASSED [ 53%] distributed/tests/test_profile.py::test_basic PASSED [ 53%] distributed/tests/test_profile.py::test_basic_low_level SKIPPED (could @@ -10804,7 +12567,7 @@ distributed/tests/test_steal.py::test_balance_multiple_to_replica PASSED [ 79%] distributed/tests/test_steal.py::test_balance_to_larger_dependency PASSED [ 79%] distributed/tests/test_steal.py::test_balance_prefers_busier_with_dependency PASSED [ 79%] -distributed/tests/test_steal.py::test_trivial_workload_should_not_cause_work_stealing PASSED [ 79%] +distributed/tests/test_steal.py::test_trivial_workload_should_not_cause_work_stealing FAILED [ 79%] distributed/tests/test_stories.py::test_scheduler_story_stimulus_success PASSED [ 79%] distributed/tests/test_stories.py::test_scheduler_story_stimulus_retry PASSED [ 79%] distributed/tests/test_stories.py::test_client_story PASSED [ 79%] @@ -11209,7 +12972,7 @@ distributed/tests/test_worker.py::test_missing_released_zombie_tasks PASSED [ 91%] distributed/tests/test_worker.py::test_missing_released_zombie_tasks_2 PASSED [ 92%] distributed/tests/test_worker.py::test_worker_status_sync PASSED [ 92%] -distributed/tests/test_worker.py::test_log_remove_worker PASSED [ 92%] +distributed/tests/test_worker.py::test_log_remove_worker FAILED [ 92%] distributed/tests/test_worker.py::test_task_flight_compute_oserror PASSED [ 92%] distributed/tests/test_worker.py::test_gather_dep_cancelled_rescheduled PASSED [ 92%] distributed/tests/test_worker.py::test_gather_dep_do_not_handle_response_of_not_requested_tasks PASSED [ 92%] @@ -11266,7 +13029,7 @@ distributed/tests/test_worker_memory.py::test_workerstate_fail_to_pickle_execute_1[long-running] PASSED [ 93%] distributed/tests/test_worker_memory.py::test_workerstate_fail_to_pickle_flight PASSED [ 93%] distributed/tests/test_worker_memory.py::test_fail_to_pickle_execute_2 PASSED [ 93%] -distributed/tests/test_worker_memory.py::test_fail_to_pickle_spill PASSED [ 93%] +distributed/tests/test_worker_memory.py::test_fail_to_pickle_spill FAILED [ 93%] distributed/tests/test_worker_memory.py::test_spill_target_threshold PASSED [ 93%] distributed/tests/test_worker_memory.py::test_spill_constrained PASSED [ 93%] distributed/tests/test_worker_memory.py::test_spill_spill_threshold PASSED [ 93%] @@ -11285,7 +13048,7 @@ distributed/tests/test_worker_memory.py::test_nanny_terminate SKIPPED [ 94%] distributed/tests/test_worker_memory.py::test_disk_cleanup_on_terminate[False] SKIPPED [ 94%] distributed/tests/test_worker_memory.py::test_disk_cleanup_on_terminate[True] SKIPPED [ 94%] -distributed/tests/test_worker_memory.py::test_pause_while_spilling PASSED [ 94%] +distributed/tests/test_worker_memory.py::test_pause_while_spilling FAILED [ 94%] distributed/tests/test_worker_memory.py::test_release_evloop_while_spilling SKIPPED [ 94%] distributed/tests/test_worker_memory.py::test_deprecated_attributes[Worker-memory_limit-123000000000.0] PASSED [ 94%] distributed/tests/test_worker_memory.py::test_deprecated_attributes[Worker-memory_target_fraction-0.789] PASSED [ 94%] @@ -11298,8 +13061,8 @@ distributed/tests/test_worker_memory.py::test_deprecated_params[memory_target_fraction] PASSED [ 94%] distributed/tests/test_worker_memory.py::test_deprecated_params[memory_spill_fraction] PASSED [ 94%] distributed/tests/test_worker_memory.py::test_deprecated_params[memory_pause_fraction] PASSED [ 94%] -distributed/tests/test_worker_memory.py::test_pause_while_idle PASSED [ 94%] -distributed/tests/test_worker_memory.py::test_pause_while_saturated PASSED [ 94%] +distributed/tests/test_worker_memory.py::test_pause_while_idle FAILED [ 94%] +distributed/tests/test_worker_memory.py::test_pause_while_saturated FAILED [ 94%] distributed/tests/test_worker_memory.py::test_worker_log_memory_limit_too_high PASSED [ 94%] distributed/tests/test_worker_memory.py::test_high_unmanaged_memory_warning PASSED [ 94%] distributed/tests/test_worker_memory.py::test_delete_spilled_keys PASSED [ 94%] @@ -11464,27 +13227,1227 @@ distributed/tests/test_worker_state_machine.py::test_remove_worker_while_in_fetch PASSED [ 99%] distributed/tests/test_worker_state_machine.py::test_remove_worker_unknown PASSED [100%] +=================================== FAILURES =================================== +_____________________________ test_rechunk_method ______________________________ + +args = (), kwds = {} + + @wraps(func) + def inner(*args, **kwds): + with self._recreate_cm(): +> return func(*args, **kwds) + +/usr/lib/python3.12/contextlib.py:81: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +/usr/lib/python3.12/contextlib.py:81: in inner + return func(*args, **kwds) +distributed/utils_test.py:1090: in test_func + return _run_and_close_tornado(async_fn_outer) +distributed/utils_test.py:380: in _run_and_close_tornado + return asyncio_run(inner_fn(), loop_factory=get_loop_factory()) +/usr/lib/python3.12/asyncio/runners.py:194: in run + return runner.run(main) +/usr/lib/python3.12/asyncio/runners.py:118: in run + return self._loop.run_until_complete(task) +/usr/lib/python3.12/asyncio/base_events.py:686: in run_until_complete + return future.result() +distributed/utils_test.py:377: in inner_fn + return await async_fn(*args, **kwargs) +distributed/utils_test.py:1087: in async_fn_outer + return await utils_wait_for(async_fn(), timeout=timeout * 2) +distributed/utils.py:1914: in wait_for + return await fut +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + + async def async_fn(): + result = None + with dask.config.set(config): + async with ( + _cluster_factory() as (s, workers), + _client_factory(s) as c, + ): + args = [s] + workers + if c is not None: + args = [c] + args + try: + coro = func(*args, *outer_args, **kwargs) + task = asyncio.create_task(coro) + coro2 = utils_wait_for( + asyncio.shield(task), timeout=deadline.remaining + ) + result = await coro2 + validate_state(s, *workers) + + except asyncio.TimeoutError: + assert task + elapsed = deadline.elapsed + buffer = io.StringIO() + # This stack indicates where the coro/test is suspended + task.print_stack(file=buffer) + + task.cancel() + while not task.cancelled(): + await asyncio.sleep(0.01) + + # Hopefully, the hang has been caused by inconsistent + # state, which should be much more meaningful than the + # timeout + validate_state(s, *workers) + + # Remove as much of the traceback as possible; it's + # uninteresting boilerplate from utils_test and asyncio + # and not from the code being tested. +> raise asyncio.TimeoutError( + f"Test timeout ({timeout}) hit after {elapsed}s.\n" + "========== Test stack trace starts here ==========\n" + f"{buffer.getvalue()}" + ) from None +E TimeoutError: Test timeout (30) hit after 30.050075880018994s. +E ========== Test stack trace starts here ========== +E Stack for wait_for=> (most recent call last): +E File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/shuffle/tests/test_rechunk.py", line 414, in test_rechunk_method +E assert np.all(await c.compute(x2) == a) + +distributed/utils_test.py:1041: TimeoutError +----------------------------- Captured stderr call ----------------------------- +2025-01-14 16:32:25,478 - distributed.scheduler - INFO - State start +2025-01-14 16:32:25,525 - distributed.scheduler - INFO - Scheduler at: tcp://127.0.0.1:35341 +2025-01-14 16:32:25,526 - distributed.scheduler - INFO - dashboard at: http://127.0.0.1:41769/status +2025-01-14 16:32:25,527 - distributed.scheduler - INFO - Registering Worker plugin shuffle +2025-01-14 16:32:25,605 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:33683 +2025-01-14 16:32:25,606 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:33683 +2025-01-14 16:32:25,606 - distributed.worker - INFO - Worker name: 0 +2025-01-14 16:32:25,606 - distributed.worker - INFO - dashboard at: 127.0.0.1:39913 +2025-01-14 16:32:25,607 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:35341 +2025-01-14 16:32:25,607 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 16:32:25,624 - distributed.worker - INFO - Threads: 1 +2025-01-14 16:32:25,625 - distributed.worker - INFO - Memory: 78.53 GiB +2025-01-14 16:32:25,625 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-fyc79aji +2025-01-14 16:32:25,625 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 16:32:25,641 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:45773 +2025-01-14 16:32:25,642 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:45773 +2025-01-14 16:32:25,642 - distributed.worker - INFO - Worker name: 1 +2025-01-14 16:32:25,643 - distributed.worker - INFO - dashboard at: 127.0.0.1:40489 +2025-01-14 16:32:25,643 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:35341 +2025-01-14 16:32:25,643 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 16:32:25,657 - distributed.worker - INFO - Threads: 2 +2025-01-14 16:32:25,657 - distributed.worker - INFO - Memory: 78.53 GiB +2025-01-14 16:32:25,657 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-djoaeku9 +2025-01-14 16:32:25,658 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 16:32:25,765 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:33683 name: 0 +2025-01-14 16:32:25,798 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:33683 +2025-01-14 16:32:25,812 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:53538 +2025-01-14 16:32:25,821 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:45773 name: 1 +2025-01-14 16:32:25,854 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:45773 +2025-01-14 16:32:25,855 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:53554 +2025-01-14 16:32:25,869 - distributed.worker - INFO - Starting Worker plugin shuffle +2025-01-14 16:32:25,870 - distributed.worker - INFO - Starting Worker plugin shuffle +2025-01-14 16:32:25,884 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:35341 +2025-01-14 16:32:25,885 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 16:32:25,886 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:35341 +2025-01-14 16:32:25,886 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 16:32:25,932 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:35341 +2025-01-14 16:32:25,933 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:35341 +2025-01-14 16:32:26,013 - distributed.scheduler - INFO - Receive client connection: Client-cac27ccf-d21f-11ef-8215-e761ae0addd2 +2025-01-14 16:32:26,047 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:53562 +2025-01-14 16:32:33,196 - distributed.core - INFO - Event loop was unresponsive in Scheduler for 5.23s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability. +2025-01-14 16:32:33,200 - distributed.core - INFO - Event loop was unresponsive in Worker for 5.24s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability. +2025-01-14 16:32:33,202 - distributed.core - INFO - Event loop was unresponsive in Worker for 5.24s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability. +2025-01-14 16:32:55,957 - distributed.scheduler - INFO - Remove client Client-cac27ccf-d21f-11ef-8215-e761ae0addd2 +2025-01-14 16:32:56,563 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:53562; closing. +2025-01-14 16:32:56,604 - distributed.scheduler - INFO - Remove client Client-cac27ccf-d21f-11ef-8215-e761ae0addd2 +2025-01-14 16:32:57,158 - distributed.scheduler - INFO - Close client connection: Client-cac27ccf-d21f-11ef-8215-e761ae0addd2 +2025-01-14 16:32:57,359 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:33683. Reason: worker-close +2025-01-14 16:32:57,369 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:45773. Reason: worker-close +2025-01-14 16:32:57,390 - distributed.worker - INFO - Removing Worker plugin shuffle +2025-01-14 16:32:57,458 - distributed.worker - INFO - Removing Worker plugin shuffle +2025-01-14 16:32:57,501 - distributed.worker - ERROR - Failed to communicate with scheduler during heartbeat. +Traceback (most recent call last): + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/comm/tcp.py", line 225, in read + frames_nosplit_nbytes_bin = await stream.read_bytes(fmt_size) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +tornado.iostream.StreamClosedError: Stream is closed + +The above exception was the direct cause of the following exception: + +Traceback (most recent call last): + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/worker.py", line 1269, in heartbeat + response = await retry_operation( + ^^^^^^^^^^^^^^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/utils_comm.py", line 441, in retry_operation + return await retry( + ^^^^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/utils_comm.py", line 420, in retry + return await coro() + ^^^^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/core.py", line 1259, in send_recv_from_rpc + return await send_recv(comm=comm, op=key, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/core.py", line 1018, in send_recv + response = await comm.read(deserializers=deserializers) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/comm/tcp.py", line 236, in read + convert_stream_closed_error(self, e) + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/comm/tcp.py", line 142, in convert_stream_closed_error + raise CommClosedError(f"in {obj}: {exc}") from exc +distributed.comm.core.CommClosedError: in : Stream is closed +2025-01-14 16:32:57,541 - distributed.worker - ERROR - Failed to communicate with scheduler during heartbeat. +Traceback (most recent call last): + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/comm/tcp.py", line 225, in read + frames_nosplit_nbytes_bin = await stream.read_bytes(fmt_size) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +tornado.iostream.StreamClosedError: Stream is closed + +The above exception was the direct cause of the following exception: + +Traceback (most recent call last): + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/worker.py", line 1269, in heartbeat + response = await retry_operation( + ^^^^^^^^^^^^^^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/utils_comm.py", line 441, in retry_operation + return await retry( + ^^^^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/utils_comm.py", line 420, in retry + return await coro() + ^^^^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/core.py", line 1259, in send_recv_from_rpc + return await send_recv(comm=comm, op=key, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/core.py", line 1018, in send_recv + response = await comm.read(deserializers=deserializers) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/comm/tcp.py", line 236, in read + convert_stream_closed_error(self, e) + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/comm/tcp.py", line 142, in convert_stream_closed_error + raise CommClosedError(f"in {obj}: {exc}") from exc +distributed.comm.core.CommClosedError: in : Stream is closed +2025-01-14 16:32:57,559 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:53554; closing. +2025-01-14 16:32:57,582 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:45773 name: 1 (stimulus_id='handle-worker-cleanup-1736821977.581589') +2025-01-14 16:32:57,597 - distributed.core - INFO - Connection to tcp://127.0.0.1:35341 has been closed. +2025-01-14 16:32:57,617 - distributed.core - INFO - Connection to tcp://127.0.0.1:35341 has been closed. +2025-01-14 16:32:57,618 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:53538; closing. +2025-01-14 16:32:57,638 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:33683 name: 0 (stimulus_id='handle-worker-cleanup-1736821977.6383178') +2025-01-14 16:32:57,649 - distributed.scheduler - INFO - Lost all workers +2025-01-14 16:32:57,682 - distributed.scheduler - INFO - Closing scheduler. Reason: unknown +2025-01-14 16:32:57,692 - distributed.scheduler - INFO - Scheduler closing all comms +___________________ test_last_in_first_out[queue on worker] ____________________ + +c = +s = +a = +pause = False + + @gen_blockable_cluster + async def test_last_in_first_out(c, s, a, pause): + async with block_worker(c, s, a, pause, 15, 5): + xs = [c.submit(slowinc, i, delay=0.05, key=f"x{i}") for i in range(5)] + ys = [c.submit(slowinc, xs[i], delay=0.05, key=f"y{i}") for i in range(5)] + zs = [c.submit(slowinc, ys[i], delay=0.05, key=f"z{i}") for i in range(5)] + + while not any(s.tasks[z.key].state == "memory" for z in zs): + await asyncio.sleep(0.01) +> assert not all(s.tasks[x.key].state == "memory" for x in xs) +E assert not True +E + where True = all(. at 0x7f68853d6f60>) + +distributed/tests/test_priorities.py:234: AssertionError +----------------------------- Captured stderr call ----------------------------- +2025-01-14 17:00:47,795 - distributed.scheduler - INFO - State start +2025-01-14 17:00:47,809 - distributed.scheduler - INFO - Scheduler at: tcp://127.0.0.1:45047 +2025-01-14 17:00:47,815 - distributed.scheduler - INFO - dashboard at: http://127.0.0.1:42287/status +2025-01-14 17:00:47,821 - distributed.scheduler - INFO - Registering Worker plugin shuffle +2025-01-14 17:00:47,841 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:42937 +2025-01-14 17:00:47,847 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:42937 +2025-01-14 17:00:47,854 - distributed.worker - INFO - Worker name: 0 +2025-01-14 17:00:47,859 - distributed.worker - INFO - dashboard at: 127.0.0.1:37471 +2025-01-14 17:00:47,866 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:45047 +2025-01-14 17:00:47,870 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 17:00:47,878 - distributed.worker - INFO - Threads: 1 +2025-01-14 17:00:47,884 - distributed.worker - INFO - Memory: 78.53 GiB +2025-01-14 17:00:47,890 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-23w2mryq +2025-01-14 17:00:47,896 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 17:00:47,952 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:42937 name: 0 +2025-01-14 17:00:47,998 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:42937 +2025-01-14 17:00:48,003 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:41902 +2025-01-14 17:00:48,005 - distributed.worker - INFO - Starting Worker plugin shuffle +2025-01-14 17:00:48,011 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:45047 +2025-01-14 17:00:48,016 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 17:00:48,023 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:45047 +2025-01-14 17:00:48,095 - distributed.scheduler - INFO - Receive client connection: Client-c148dbba-d223-11ef-8215-e761ae0addd2 +2025-01-14 17:00:48,194 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:41914 +2025-01-14 17:00:48,966 - distributed.scheduler - INFO - Remove client Client-c148dbba-d223-11ef-8215-e761ae0addd2 +2025-01-14 17:00:48,981 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:41914; closing. +2025-01-14 17:00:48,982 - distributed.scheduler - INFO - Remove client Client-c148dbba-d223-11ef-8215-e761ae0addd2 +2025-01-14 17:00:48,997 - distributed.scheduler - INFO - Close client connection: Client-c148dbba-d223-11ef-8215-e761ae0addd2 +2025-01-14 17:00:49,019 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:42937. Reason: worker-close +2025-01-14 17:00:49,040 - distributed.worker.state_machine - WARNING - Async instruction for > ended with CancelledError +2025-01-14 17:00:49,051 - distributed.worker - INFO - Removing Worker plugin shuffle +2025-01-14 17:00:49,067 - distributed.worker - ERROR - Failed to communicate with scheduler during heartbeat. +Traceback (most recent call last): + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/comm/tcp.py", line 225, in read + frames_nosplit_nbytes_bin = await stream.read_bytes(fmt_size) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +tornado.iostream.StreamClosedError: Stream is closed + +The above exception was the direct cause of the following exception: + +Traceback (most recent call last): + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/worker.py", line 1269, in heartbeat + response = await retry_operation( + ^^^^^^^^^^^^^^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/utils_comm.py", line 441, in retry_operation + return await retry( + ^^^^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/utils_comm.py", line 420, in retry + return await coro() + ^^^^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/core.py", line 1259, in send_recv_from_rpc + return await send_recv(comm=comm, op=key, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/core.py", line 1018, in send_recv + response = await comm.read(deserializers=deserializers) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/comm/tcp.py", line 236, in read + convert_stream_closed_error(self, e) + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/comm/tcp.py", line 142, in convert_stream_closed_error + raise CommClosedError(f"in {obj}: {exc}") from exc +distributed.comm.core.CommClosedError: in : Stream is closed +2025-01-14 17:00:49,090 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:41902; closing. +2025-01-14 17:00:49,090 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:42937 name: 0 (stimulus_id='handle-worker-cleanup-1736823649.0906692') +2025-01-14 17:00:49,113 - distributed.scheduler - INFO - Lost all workers +2025-01-14 17:00:49,128 - distributed.core - INFO - Connection to tcp://127.0.0.1:45047 has been closed. +2025-01-14 17:00:49,161 - distributed.scheduler - INFO - Closing scheduler. Reason: unknown +2025-01-14 17:00:49,176 - distributed.scheduler - INFO - Scheduler closing all comms +_____________ test_trivial_workload_should_not_cause_work_stealing _____________ + +args = (), kwds = {} + + @wraps(func) + def inner(*args, **kwds): + with self._recreate_cm(): +> return func(*args, **kwds) + +/usr/lib/python3.12/contextlib.py:81: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +/usr/lib/python3.12/contextlib.py:81: in inner + return func(*args, **kwds) +distributed/utils_test.py:1090: in test_func + return _run_and_close_tornado(async_fn_outer) +distributed/utils_test.py:380: in _run_and_close_tornado + return asyncio_run(inner_fn(), loop_factory=get_loop_factory()) +/usr/lib/python3.12/asyncio/runners.py:194: in run + return runner.run(main) +/usr/lib/python3.12/asyncio/runners.py:118: in run + return self._loop.run_until_complete(task) +/usr/lib/python3.12/asyncio/base_events.py:686: in run_until_complete + return future.result() +distributed/utils_test.py:377: in inner_fn + return await async_fn(*args, **kwargs) +distributed/utils_test.py:1087: in async_fn_outer + return await utils_wait_for(async_fn(), timeout=timeout * 2) +distributed/utils.py:1914: in wait_for + return await fut +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + + async def async_fn(): + result = None + with dask.config.set(config): + async with ( + _cluster_factory() as (s, workers), + _client_factory(s) as c, + ): + args = [s] + workers + if c is not None: + args = [c] + args + try: + coro = func(*args, *outer_args, **kwargs) + task = asyncio.create_task(coro) + coro2 = utils_wait_for( + asyncio.shield(task), timeout=deadline.remaining + ) + result = await coro2 + validate_state(s, *workers) + + except asyncio.TimeoutError: + assert task + elapsed = deadline.elapsed + buffer = io.StringIO() + # This stack indicates where the coro/test is suspended + task.print_stack(file=buffer) + + task.cancel() + while not task.cancelled(): + await asyncio.sleep(0.01) + + # Hopefully, the hang has been caused by inconsistent + # state, which should be much more meaningful than the + # timeout + validate_state(s, *workers) + + # Remove as much of the traceback as possible; it's + # uninteresting boilerplate from utils_test and asyncio + # and not from the code being tested. +> raise asyncio.TimeoutError( + f"Test timeout ({timeout}) hit after {elapsed}s.\n" + "========== Test stack trace starts here ==========\n" + f"{buffer.getvalue()}" + ) from None +E TimeoutError: Test timeout (30) hit after 31.710481404006714s. +E ========== Test stack trace starts here ========== +E Stack for wait_for=> (most recent call last): +E File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/tests/test_steal.py", line 1878, in test_trivial_workload_should_not_cause_work_stealing +E await c.gather(futs) + +distributed/utils_test.py:1041: TimeoutError +----------------------------- Captured stderr call ----------------------------- +2025-01-14 17:11:25,368 - distributed.scheduler - INFO - State start +2025-01-14 17:11:25,436 - distributed.scheduler - INFO - Scheduler at: tcp://127.0.0.1:33671 +2025-01-14 17:11:25,443 - distributed.scheduler - INFO - dashboard at: http://127.0.0.1:36679/status +2025-01-14 17:11:25,463 - distributed.scheduler - INFO - Registering Worker plugin shuffle +2025-01-14 17:11:25,584 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:44039 +2025-01-14 17:11:25,600 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:44039 +2025-01-14 17:11:25,621 - distributed.worker - INFO - Worker name: 0 +2025-01-14 17:11:25,632 - distributed.worker - INFO - dashboard at: 127.0.0.1:44041 +2025-01-14 17:11:25,656 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:33671 +2025-01-14 17:11:25,673 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 17:11:25,698 - distributed.worker - INFO - Threads: 1 +2025-01-14 17:11:25,718 - distributed.worker - INFO - Memory: 78.53 GiB +2025-01-14 17:11:25,731 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-aswfc_5y +2025-01-14 17:11:25,764 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 17:11:25,795 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:33739 +2025-01-14 17:11:25,818 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:33739 +2025-01-14 17:11:25,830 - distributed.worker - INFO - Worker name: 1 +2025-01-14 17:11:25,846 - distributed.worker - INFO - dashboard at: 127.0.0.1:39753 +2025-01-14 17:11:25,867 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:33671 +2025-01-14 17:11:25,881 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 17:11:25,902 - distributed.worker - INFO - Threads: 1 +2025-01-14 17:11:25,925 - distributed.worker - INFO - Memory: 78.53 GiB +2025-01-14 17:11:25,941 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-bq2ikf96 +2025-01-14 17:11:25,963 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 17:11:25,981 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:39043 +2025-01-14 17:11:26,006 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:39043 +2025-01-14 17:11:26,027 - distributed.worker - INFO - Worker name: 2 +2025-01-14 17:11:26,049 - distributed.worker - INFO - dashboard at: 127.0.0.1:42359 +2025-01-14 17:11:26,062 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:33671 +2025-01-14 17:11:26,082 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 17:11:26,098 - distributed.worker - INFO - Threads: 1 +2025-01-14 17:11:26,115 - distributed.worker - INFO - Memory: 78.53 GiB +2025-01-14 17:11:26,130 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-abjwxp55 +2025-01-14 17:11:26,149 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 17:11:26,178 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:45439 +2025-01-14 17:11:26,195 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:45439 +2025-01-14 17:11:26,212 - distributed.worker - INFO - Worker name: 3 +2025-01-14 17:11:26,243 - distributed.worker - INFO - dashboard at: 127.0.0.1:44379 +2025-01-14 17:11:26,259 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:33671 +2025-01-14 17:11:26,281 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 17:11:26,301 - distributed.worker - INFO - Threads: 1 +2025-01-14 17:11:26,328 - distributed.worker - INFO - Memory: 78.53 GiB +2025-01-14 17:11:26,342 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-710qwrgp +2025-01-14 17:11:26,359 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 17:11:26,933 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:44039 name: 0 +2025-01-14 17:11:27,061 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:44039 +2025-01-14 17:11:27,067 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:52046 +2025-01-14 17:11:27,077 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:33739 name: 1 +2025-01-14 17:11:27,250 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:33739 +2025-01-14 17:11:27,280 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:52060 +2025-01-14 17:11:27,281 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:39043 name: 2 +2025-01-14 17:11:27,474 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:39043 +2025-01-14 17:11:27,500 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:52074 +2025-01-14 17:11:27,502 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:45439 name: 3 +2025-01-14 17:11:27,655 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:45439 +2025-01-14 17:11:27,671 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:52088 +2025-01-14 17:11:27,685 - distributed.worker - INFO - Starting Worker plugin shuffle +2025-01-14 17:11:27,699 - distributed.worker - INFO - Starting Worker plugin shuffle +2025-01-14 17:11:27,726 - distributed.worker - INFO - Starting Worker plugin shuffle +2025-01-14 17:11:27,741 - distributed.worker - INFO - Starting Worker plugin shuffle +2025-01-14 17:11:27,782 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:33671 +2025-01-14 17:11:27,802 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 17:11:27,826 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:33671 +2025-01-14 17:11:27,868 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 17:11:27,901 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:33671 +2025-01-14 17:11:27,984 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 17:11:28,010 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:33671 +2025-01-14 17:11:28,035 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 17:11:28,071 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:33671 +2025-01-14 17:11:28,071 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:33671 +2025-01-14 17:11:28,071 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:33671 +2025-01-14 17:11:28,071 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:33671 +2025-01-14 17:11:28,305 - distributed.scheduler - INFO - Receive client connection: Client-3eca5166-d225-11ef-8215-e761ae0addd2 +2025-01-14 17:11:28,497 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:52096 +2025-01-14 17:11:57,017 - distributed.core - INFO - Event loop was unresponsive in Scheduler for 13.01s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability. +2025-01-14 17:11:57,035 - distributed.core - INFO - Event loop was unresponsive in Worker for 12.98s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability. +2025-01-14 17:11:57,056 - distributed.core - INFO - Event loop was unresponsive in Worker for 12.95s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability. +2025-01-14 17:11:57,066 - distributed.core - INFO - Event loop was unresponsive in Worker for 12.68s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability. +2025-01-14 17:11:57,070 - distributed.core - INFO - Event loop was unresponsive in Worker for 12.65s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability. +2025-01-14 17:11:57,272 - distributed.scheduler - INFO - Remove client Client-3eca5166-d225-11ef-8215-e761ae0addd2 +2025-01-14 17:11:57,509 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:52096; closing. +2025-01-14 17:11:57,514 - distributed.scheduler - INFO - Remove client Client-3eca5166-d225-11ef-8215-e761ae0addd2 +2025-01-14 17:11:57,551 - distributed.batched - INFO - Batched Comm Closed Client local=tcp://127.0.0.1:33671 remote=tcp://127.0.0.1:52096> +Traceback (most recent call last): + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/batched.py", line 115, in _background_send + nbytes = yield coro + ^^^^^^^^^^ + File "/usr/lib/python3/dist-packages/tornado/gen.py", line 766, in run + value = future.result() + ^^^^^^^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/comm/tcp.py", line 262, in write + raise CommClosedError() +distributed.comm.core.CommClosedError +2025-01-14 17:11:57,723 - distributed.scheduler - INFO - Close client connection: Client-3eca5166-d225-11ef-8215-e761ae0addd2 +2025-01-14 17:11:58,415 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:44039. Reason: worker-close +2025-01-14 17:11:58,446 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:33739. Reason: worker-close +2025-01-14 17:11:58,484 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:39043. Reason: worker-close +2025-01-14 17:11:58,518 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:45439. Reason: worker-close +2025-01-14 17:11:58,673 - distributed.worker - INFO - Removing Worker plugin shuffle +2025-01-14 17:11:58,711 - distributed.worker - INFO - Removing Worker plugin shuffle +2025-01-14 17:11:58,826 - distributed.worker - INFO - Removing Worker plugin shuffle +2025-01-14 17:11:58,865 - distributed.worker - INFO - Removing Worker plugin shuffle +2025-01-14 17:11:58,947 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:52046; closing. +2025-01-14 17:11:58,957 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:52074; closing. +2025-01-14 17:11:58,958 - distributed.core - INFO - Connection to tcp://127.0.0.1:33671 has been closed. +2025-01-14 17:11:58,958 - distributed.core - INFO - Connection to tcp://127.0.0.1:33671 has been closed. +2025-01-14 17:11:58,959 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:44039 name: 0 (stimulus_id='handle-worker-cleanup-1736824318.9592624') +2025-01-14 17:11:58,975 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:39043 name: 2 (stimulus_id='handle-worker-cleanup-1736824318.9752097') +2025-01-14 17:11:59,006 - distributed.core - INFO - Connection to tcp://127.0.0.1:33671 has been closed. +2025-01-14 17:11:59,020 - distributed.core - INFO - Connection to tcp://127.0.0.1:33671 has been closed. +2025-01-14 17:11:59,022 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:52088; closing. +2025-01-14 17:11:59,028 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:52060; closing. +2025-01-14 17:11:59,045 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:45439 name: 3 (stimulus_id='handle-worker-cleanup-1736824319.044918') +2025-01-14 17:11:59,070 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:33739 name: 1 (stimulus_id='handle-worker-cleanup-1736824319.0703394') +2025-01-14 17:11:59,101 - distributed.scheduler - INFO - Lost all workers +2025-01-14 17:11:59,158 - distributed.scheduler - INFO - Closing scheduler. Reason: unknown +2025-01-14 17:11:59,178 - distributed.scheduler - INFO - Scheduler closing all comms +____________________________ test_log_remove_worker ____________________________ + +c = +s = +a = +b = + + @gen_cluster(client=True) + async def test_log_remove_worker(c, s, a, b): + # Computed task + x = c.submit(inc, 1, key="x", workers=a.address) + await x + ev = Event() + # Processing task + y = c.submit( + lambda ev: ev.wait(), ev, key="y", workers=a.address, allow_other_workers=True + ) + await wait_for_state("y", "processing", s) + # Scattered task + z = await c.scatter({"z": 3}, workers=a.address) + + s._broker.truncate() + + with captured_logger("distributed.scheduler", level=logging.INFO) as log: + # Successful graceful shutdown + await s.retire_workers([a.address], stimulus_id="graceful") + # Refuse to retire gracefully as there's nowhere to put x and z + await s.retire_workers([b.address], stimulus_id="graceful_abort") + await asyncio.sleep(0.2) + # Ungraceful shutdown + await s.remove_worker(b.address, stimulus_id="ungraceful") + await asyncio.sleep(0.2) + await ev.set() + +> assert log.getvalue().splitlines() == [ + # Successful graceful + f"Retire worker addresses (stimulus_id='graceful') ['{a.address}']", + f"Remove worker addr: {a.address} name: {a.name} (stimulus_id='graceful')", + f"Retired worker '{a.address}' (stimulus_id='graceful')", + # Aborted graceful + f"Retire worker addresses (stimulus_id='graceful_abort') ['{b.address}']", + f"Could not retire worker '{b.address}': unique data could not be " + "moved to any other worker (stimulus_id='graceful_abort')", + # Ungraceful + f"Remove worker addr: {b.address} name: {b.name} (stimulus_id='ungraceful')", + f"Removing worker '{b.address}' caused the cluster to lose already " + "computed task(s), which will be recomputed elsewhere: {'x'} " + "(stimulus_id='ungraceful')", + f"Removing worker '{b.address}' caused the cluster to lose scattered " + "data, which can't be recovered: {'z'} (stimulus_id='ungraceful')", + "Lost all workers", + ] +E assert ["Retire worker addresses (stimulus_id='graceful') ['tcp://127.0.0.1:40277']", "Remove worker addr: tcp://127.0.0.1:40277 name: 0 (stimulus_id='graceful')", "Retired worker 'tcp://127.0.0.1:40277' (stimulus_id='graceful')", "Retire worker addresses (stimulus_id='graceful_abort') ['tcp://127.0.0.1:33441']", "Could not retire worker 'tcp://127.0.0.1:33441': unique data could not be moved to any other worker (stimulus_id='graceful_abort')", "Received heartbeat from unregistered worker 'tcp://127.0.0.1:40277'.", "Remove worker addr: tcp://127.0.0.1:33441 name: 1 (stimulus_id='ungraceful')", "Removing worker 'tcp://127.0.0.1:33441' caused the cluster to lose already computed task(s), which will be recomputed elsewhere: {'x'} (stimulus_id='ungraceful')", "Removing worker 'tcp://127.0.0.1:33441' caused the cluster to lose scattered data, which can't be recovered: {'z'} (stimulus_id='ungraceful')", 'Lost all workers', "Received heartbeat from unregistered worker 'tcp://127.0.0.1:33441'."] == ["Retire worker addresses (stimulus_id='graceful') ['tcp://127.0.0.1:40277']", "Remove worker addr: tcp://127.0.0.1:40277 name: 0 (stimulus_id='graceful')", "Retired worker 'tcp://127.0.0.1:40277' (stimulus_id='graceful')", "Retire worker addresses (stimulus_id='graceful_abort') ['tcp://127.0.0.1:33441']", "Could not retire worker 'tcp://127.0.0.1:33441': unique data could not be moved to any other worker (stimulus_id='graceful_abort')", "Remove worker addr: tcp://127.0.0.1:33441 name: 1 (stimulus_id='ungraceful')", "Removing worker 'tcp://127.0.0.1:33441' caused the cluster to lose already computed task(s), which will be recomputed elsewhere: {'x'} (stimulus_id='ungraceful')", "Removing worker 'tcp://127.0.0.1:33441' caused the cluster to lose scattered data, which can't be recovered: {'z'} (stimulus_id='ungraceful')", 'Lost all workers'] +E +E At index 5 diff: "Received heartbeat from unregistered worker 'tcp://127.0.0.1:40277'." != "Remove worker addr: tcp://127.0.0.1:33441 name: 1 (stimulus_id='ungraceful')" +E Left contains 2 more items, first extra item: 'Lost all workers' +E +E Full diff: +E [ +E "Retire worker addresses (stimulus_id='graceful') " +E "['tcp://127.0.0.1:40277']", +E 'Remove worker addr: tcp://127.0.0.1:40277 name: 0 ' +E "(stimulus_id='graceful')", +E "Retired worker 'tcp://127.0.0.1:40277' (stimulus_id='graceful')", +E "Retire worker addresses (stimulus_id='graceful_abort') " +E "['tcp://127.0.0.1:33441']", +E "Could not retire worker 'tcp://127.0.0.1:33441': unique data could not be " +E "moved to any other worker (stimulus_id='graceful_abort')", +E + "Received heartbeat from unregistered worker 'tcp://127.0.0.1:40277'.", +E 'Remove worker addr: tcp://127.0.0.1:33441 name: 1 ' +E "(stimulus_id='ungraceful')", +E "Removing worker 'tcp://127.0.0.1:33441' caused the cluster to lose " +E "already computed task(s), which will be recomputed elsewhere: {'x'} " +E "(stimulus_id='ungraceful')", +E "Removing worker 'tcp://127.0.0.1:33441' caused the cluster to lose " +E "scattered data, which can't be recovered: {'z'} " +E "(stimulus_id='ungraceful')", +E 'Lost all workers', +E + "Received heartbeat from unregistered worker 'tcp://127.0.0.1:33441'.", +E ] + +distributed/tests/test_worker.py:2988: AssertionError +----------------------------- Captured stderr call ----------------------------- +2025-01-14 17:24:58,043 - distributed.scheduler - INFO - State start +2025-01-14 17:24:58,134 - distributed.scheduler - INFO - Scheduler at: tcp://127.0.0.1:43245 +2025-01-14 17:24:58,161 - distributed.scheduler - INFO - dashboard at: http://127.0.0.1:38021/status +2025-01-14 17:24:58,185 - distributed.scheduler - INFO - Registering Worker plugin shuffle +2025-01-14 17:24:58,276 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:40277 +2025-01-14 17:24:58,291 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:40277 +2025-01-14 17:24:58,318 - distributed.worker - INFO - Worker name: 0 +2025-01-14 17:24:58,348 - distributed.worker - INFO - dashboard at: 127.0.0.1:45429 +2025-01-14 17:24:58,381 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:43245 +2025-01-14 17:24:58,400 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 17:24:58,406 - distributed.worker - INFO - Threads: 1 +2025-01-14 17:24:58,424 - distributed.worker - INFO - Memory: 78.53 GiB +2025-01-14 17:24:58,435 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-xmcdmm94 +2025-01-14 17:24:58,450 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 17:24:58,474 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:33441 +2025-01-14 17:24:58,484 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:33441 +2025-01-14 17:24:58,501 - distributed.worker - INFO - Worker name: 1 +2025-01-14 17:24:58,511 - distributed.worker - INFO - dashboard at: 127.0.0.1:33979 +2025-01-14 17:24:58,526 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:43245 +2025-01-14 17:24:58,541 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 17:24:58,552 - distributed.worker - INFO - Threads: 2 +2025-01-14 17:24:58,571 - distributed.worker - INFO - Memory: 78.53 GiB +2025-01-14 17:24:58,579 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-i45swd15 +2025-01-14 17:24:58,607 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 17:24:58,915 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:40277 name: 0 +2025-01-14 17:24:59,074 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:40277 +2025-01-14 17:24:59,088 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:54556 +2025-01-14 17:24:59,089 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:33441 name: 1 +2025-01-14 17:24:59,252 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:33441 +2025-01-14 17:24:59,258 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:54568 +2025-01-14 17:24:59,262 - distributed.worker - INFO - Starting Worker plugin shuffle +2025-01-14 17:24:59,287 - distributed.worker - INFO - Starting Worker plugin shuffle +2025-01-14 17:24:59,332 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:43245 +2025-01-14 17:24:59,350 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 17:24:59,381 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:43245 +2025-01-14 17:24:59,399 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 17:24:59,431 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:43245 +2025-01-14 17:24:59,431 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:43245 +2025-01-14 17:24:59,714 - distributed.scheduler - INFO - Receive client connection: Client-2265f9ee-d227-11ef-8215-e761ae0addd2 +2025-01-14 17:25:00,005 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:54570 +2025-01-14 17:25:00,171 - distributed.scheduler - INFO - Retire worker addresses (stimulus_id='graceful') ['tcp://127.0.0.1:40277'] +2025-01-14 17:25:00,184 - distributed.active_memory_manager - INFO - Retiring worker tcp://127.0.0.1:40277; 2 keys are being moved away. +2025-01-14 17:25:00,249 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:40277 name: 0 (stimulus_id='graceful') +2025-01-14 17:25:00,250 - distributed.scheduler - INFO - Retired worker 'tcp://127.0.0.1:40277' (stimulus_id='graceful') +2025-01-14 17:25:00,262 - distributed.scheduler - INFO - Retire worker addresses (stimulus_id='graceful_abort') ['tcp://127.0.0.1:33441'] +2025-01-14 17:25:00,263 - distributed.active_memory_manager - WARNING - Tried retiring worker tcp://127.0.0.1:33441, but 2 tasks could not be moved as there are no suitable workers to receive them. The worker will not be retired. +2025-01-14 17:25:00,268 - distributed.scheduler - WARNING - Could not retire worker 'tcp://127.0.0.1:33441': unique data could not be moved to any other worker (stimulus_id='graceful_abort') +2025-01-14 17:25:00,403 - distributed.scheduler - WARNING - Received heartbeat from unregistered worker 'tcp://127.0.0.1:40277'. +2025-01-14 17:25:00,416 - distributed.worker - WARNING - Scheduler was unaware of this worker; shutting down. +2025-01-14 17:25:00,448 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:40277. Reason: worker-heartbeat-missing +2025-01-14 17:25:00,467 - distributed.worker.state_machine - WARNING - Async instruction for > ended with CancelledError +2025-01-14 17:25:00,513 - distributed.worker - INFO - Removing Worker plugin shuffle +2025-01-14 17:25:00,545 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:33441 name: 1 (stimulus_id='ungraceful') +2025-01-14 17:25:00,545 - distributed.scheduler - WARNING - Removing worker 'tcp://127.0.0.1:33441' caused the cluster to lose already computed task(s), which will be recomputed elsewhere: {'x'} (stimulus_id='ungraceful') +2025-01-14 17:25:00,545 - distributed.scheduler - ERROR - Removing worker 'tcp://127.0.0.1:33441' caused the cluster to lose scattered data, which can't be recovered: {'z'} (stimulus_id='ungraceful') +2025-01-14 17:25:00,546 - distributed.scheduler - INFO - Lost all workers +2025-01-14 17:25:00,577 - distributed.scheduler - WARNING - Received heartbeat from unregistered worker 'tcp://127.0.0.1:33441'. +2025-01-14 17:25:00,578 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:33441. Reason: scheduler-remove-worker +2025-01-14 17:25:00,611 - distributed.worker.state_machine - WARNING - Async instruction for > ended with CancelledError +2025-01-14 17:25:00,642 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:54556; closing. +2025-01-14 17:25:00,642 - distributed.worker - WARNING - Scheduler was unaware of this worker; shutting down. +2025-01-14 17:25:00,678 - distributed.core - INFO - Connection to tcp://127.0.0.1:43245 has been closed. +2025-01-14 17:25:00,688 - distributed.worker - INFO - Removing Worker plugin shuffle +2025-01-14 17:25:00,736 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:54568; closing. +2025-01-14 17:25:00,786 - distributed.scheduler - INFO - Remove client Client-2265f9ee-d227-11ef-8215-e761ae0addd2 +2025-01-14 17:25:00,817 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:54570; closing. +2025-01-14 17:25:00,817 - distributed.scheduler - INFO - Remove client Client-2265f9ee-d227-11ef-8215-e761ae0addd2 +2025-01-14 17:25:00,861 - distributed.core - INFO - Connection to tcp://127.0.0.1:43245 has been closed. +2025-01-14 17:25:00,861 - distributed.scheduler - INFO - Close client connection: Client-2265f9ee-d227-11ef-8215-e761ae0addd2 +2025-01-14 17:25:00,886 - distributed.scheduler - INFO - Closing scheduler. Reason: unknown +2025-01-14 17:25:00,921 - distributed.scheduler - INFO - Scheduler closing all comms +__________________________ test_fail_to_pickle_spill ___________________________ + +fut = , timeout = 5 + + async def wait_for(fut: Awaitable[T], timeout: float) -> T: + async with asyncio.timeout(timeout): +> return await fut + +distributed/utils.py:1914: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = Scheduler local=tcp://127.0.0.1:38676 remote=tcp://127.0.0.1:36495> +deserializers = None + + async def read(self, deserializers=None): + stream = self.stream + if stream is None: + raise CommClosedError() + + fmt = "Q" + fmt_size = struct.calcsize(fmt) + + try: + # Don't store multiple numpy or parquet buffers into the same buffer, or + # none will be released until all are released. +> frames_nosplit_nbytes_bin = await stream.read_bytes(fmt_size) +E asyncio.exceptions.CancelledError + +distributed/comm/tcp.py:225: CancelledError + +The above exception was the direct cause of the following exception: + +args = (), kwds = {} + + @wraps(func) + def inner(*args, **kwds): + with self._recreate_cm(): +> return func(*args, **kwds) + +/usr/lib/python3.12/contextlib.py:81: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +/usr/lib/python3.12/contextlib.py:81: in inner + return func(*args, **kwds) +distributed/utils_test.py:1090: in test_func + return _run_and_close_tornado(async_fn_outer) +distributed/utils_test.py:380: in _run_and_close_tornado + return asyncio_run(inner_fn(), loop_factory=get_loop_factory()) +/usr/lib/python3.12/asyncio/runners.py:194: in run + return runner.run(main) +/usr/lib/python3.12/asyncio/runners.py:118: in run + return self._loop.run_until_complete(task) +/usr/lib/python3.12/asyncio/base_events.py:686: in run_until_complete + return future.result() +distributed/utils_test.py:377: in inner_fn + return await async_fn(*args, **kwargs) +distributed/utils_test.py:1087: in async_fn_outer + return await utils_wait_for(async_fn(), timeout=timeout * 2) +distributed/utils.py:1914: in wait_for + return await fut +distributed/utils_test.py:1008: in async_fn + _client_factory(s) as c, +/usr/lib/python3.12/contextlib.py:210: in __aenter__ + return await anext(self.gen) +distributed/utils_test.py:957: in _client_factory + async with Client( +distributed/client.py:1700: in __aenter__ + await self +distributed/client.py:1512: in _start + await self._ensure_connected(timeout=timeout) +distributed/client.py:1604: in _ensure_connected + msg = await wait_for(comm.read(), timeout) +distributed/utils.py:1913: in wait_for + async with asyncio.timeout(timeout): +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = +exc_type = +exc_val = CancelledError(), exc_tb = + + async def __aexit__( + self, + exc_type: Optional[Type[BaseException]], + exc_val: Optional[BaseException], + exc_tb: Optional[TracebackType], + ) -> Optional[bool]: + assert self._state in (_State.ENTERED, _State.EXPIRING) + + if self._timeout_handler is not None: + self._timeout_handler.cancel() + self._timeout_handler = None + + if self._state is _State.EXPIRING: + self._state = _State.EXPIRED + + if self._task.uncancel() <= self._cancelling and exc_type is exceptions.CancelledError: + # Since there are no new cancel requests, we're + # handling this. +> raise TimeoutError from exc_val +E TimeoutError + +/usr/lib/python3.12/asyncio/timeouts.py:115: TimeoutError +----------------------------- Captured stderr call ----------------------------- +2025-01-14 17:27:43,739 - distributed.scheduler - INFO - State start +2025-01-14 17:27:43,773 - distributed.scheduler - INFO - Scheduler at: tcp://127.0.0.1:36495 +2025-01-14 17:27:43,792 - distributed.scheduler - INFO - dashboard at: http://127.0.0.1:34933/status +2025-01-14 17:27:43,809 - distributed.scheduler - INFO - Registering Worker plugin shuffle +2025-01-14 17:27:43,839 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:40539 +2025-01-14 17:27:43,854 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:40539 +2025-01-14 17:27:43,867 - distributed.worker - INFO - Worker name: 0 +2025-01-14 17:27:43,871 - distributed.worker - INFO - dashboard at: 127.0.0.1:39415 +2025-01-14 17:27:43,892 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:36495 +2025-01-14 17:27:43,898 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 17:27:43,920 - distributed.worker - INFO - Threads: 1 +2025-01-14 17:27:43,925 - distributed.worker - INFO - Memory: 0.98 kiB +2025-01-14 17:27:43,937 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-mnmc_3gu +2025-01-14 17:27:43,956 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 17:27:44,105 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:40539 name: 0 +2025-01-14 17:27:44,192 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:40539 +2025-01-14 17:27:44,197 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:38660 +2025-01-14 17:27:44,198 - distributed.worker - INFO - Starting Worker plugin shuffle +2025-01-14 17:27:44,205 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:36495 +2025-01-14 17:27:44,212 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 17:27:44,230 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:36495 +2025-01-14 17:27:44,365 - distributed.scheduler - INFO - Receive client connection: Client-849e60ae-d227-11ef-8215-e761ae0addd2 +2025-01-14 17:27:44,509 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:38676 +2025-01-14 17:27:55,242 - distributed.worker.memory - WARNING - gc.collect() took 3.803s. This is usually a sign that some tasks handle too many Python objects at the same time. Rechunking the work into smaller tasks might help. +2025-01-14 17:27:55,248 - distributed.worker.memory - WARNING - Worker is at 96335052% memory usage. Pausing worker. Process memory: 918.72 MiB -- Worker memory limit: 0.98 kiB +2025-01-14 17:27:55,252 - distributed.worker.memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 918.72 MiB -- Worker memory limit: 0.98 kiB +2025-01-14 17:27:55,258 - distributed.core - INFO - Event loop was unresponsive in Worker for 11.03s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability. +2025-01-14 17:27:55,259 - distributed.core - INFO - Event loop was unresponsive in Scheduler for 10.75s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability. +2025-01-14 17:27:55,263 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:40539. Reason: worker-close +2025-01-14 17:27:55,269 - distributed.core - INFO - Connection to tcp://127.0.0.1:38676 has been closed. +2025-01-14 17:27:55,270 - distributed.scheduler - INFO - Remove client Client-849e60ae-d227-11ef-8215-e761ae0addd2 +2025-01-14 17:27:55,275 - distributed.worker - INFO - Removing Worker plugin shuffle +2025-01-14 17:27:55,283 - distributed.scheduler - INFO - Close client connection: Client-849e60ae-d227-11ef-8215-e761ae0addd2 +2025-01-14 17:27:55,289 - distributed.worker - ERROR - Failed to communicate with scheduler during heartbeat. +Traceback (most recent call last): + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/comm/tcp.py", line 225, in read + frames_nosplit_nbytes_bin = await stream.read_bytes(fmt_size) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +tornado.iostream.StreamClosedError: Stream is closed + +The above exception was the direct cause of the following exception: + +Traceback (most recent call last): + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/worker.py", line 1269, in heartbeat + response = await retry_operation( + ^^^^^^^^^^^^^^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/utils_comm.py", line 441, in retry_operation + return await retry( + ^^^^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/utils_comm.py", line 420, in retry + return await coro() + ^^^^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/core.py", line 1259, in send_recv_from_rpc + return await send_recv(comm=comm, op=key, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/core.py", line 1018, in send_recv + response = await comm.read(deserializers=deserializers) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/comm/tcp.py", line 236, in read + convert_stream_closed_error(self, e) + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/comm/tcp.py", line 142, in convert_stream_closed_error + raise CommClosedError(f"in {obj}: {exc}") from exc +distributed.comm.core.CommClosedError: in : Stream is closed +2025-01-14 17:27:55,298 - distributed.core - INFO - Connection to tcp://127.0.0.1:36495 has been closed. +2025-01-14 17:27:55,316 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:38660; closing. +2025-01-14 17:27:55,317 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:40539 name: 0 (stimulus_id='handle-worker-cleanup-1736825275.3171036') +2025-01-14 17:27:55,322 - distributed.scheduler - INFO - Lost all workers +2025-01-14 17:27:55,486 - distributed.scheduler - INFO - Closing scheduler. Reason: unknown +2025-01-14 17:27:55,497 - distributed.scheduler - INFO - Scheduler closing all comms +__________________________ test_pause_while_spilling ___________________________ + +c = +s = +a = + + @gen_cluster( + nthreads=[("", 1)], + client=True, + worker_kwargs={"memory_limit": "2 GiB"}, + # ^ must be smaller than system memory limit, otherwise that will take precedence + config={ + "distributed.worker.memory.target": False, + "distributed.worker.memory.spill": 0.5, + "distributed.worker.memory.pause": 0.8, + "distributed.worker.memory.monitor-interval": "10ms", + }, + ) + async def test_pause_while_spilling(c, s, a): + N_PAUSE = 3 + N_TOTAL = 5 + + if a.memory_manager.memory_limit < parse_bytes("2 GiB"): + pytest.fail( + f"Set 2 GiB memory limit, got {format_bytes(a.memory_manager.memory_limit)}." + ) + + def get_process_memory(): + if len(a.data) < N_PAUSE: + # Don't trigger spilling until after some tasks have completed + return 0 + elif a.data.fast and not a.data.slow: + # Trigger spilling + return parse_bytes("1.6 GiB") + else: + # Trigger pause, but only after we started spilling + return parse_bytes("1.9 GiB") + + a.monitor.get_process_memory = get_process_memory + + class SlowSpill: + def __init__(self): + # We need to record the worker while we are inside a task; can't do it in + # __reduce__ or it will pick up an arbitrary one among all running workers + self.worker = distributed.get_worker() + while len(self.worker.data.fast) >= N_PAUSE: + sleep(0.01) + + def __reduce__(self): + paused = self.worker.status == Status.paused + if not paused: + sleep(0.1) + return bool, (paused,) + + futs = [c.submit(SlowSpill, pure=False) for _ in range(N_TOTAL)] + +> await async_poll_for(lambda: len(a.data.slow) >= N_PAUSE, timeout=5, period=0) + +distributed/tests/test_worker_memory.py:982: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +predicate = . at 0x7f685765e480> +timeout = 5, fail_func = None, period = 0 + + async def async_poll_for(predicate, timeout, fail_func=None, period=0.05): + deadline = time() + timeout + while not predicate(): + await asyncio.sleep(period) + if time() > deadline: + if fail_func is not None: + fail_func() +> pytest.fail(f"condition not reached until {timeout} seconds") +E Failed: condition not reached until 5 seconds + +distributed/utils_test.py:1232: Failed +----------------------------- Captured stderr call ----------------------------- +2025-01-14 17:29:12,319 - distributed.scheduler - INFO - State start +2025-01-14 17:29:12,356 - distributed.scheduler - INFO - Scheduler at: tcp://127.0.0.1:39543 +2025-01-14 17:29:12,363 - distributed.scheduler - INFO - dashboard at: http://127.0.0.1:38129/status +2025-01-14 17:29:12,378 - distributed.scheduler - INFO - Registering Worker plugin shuffle +2025-01-14 17:29:12,422 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:33735 +2025-01-14 17:29:12,437 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:33735 +2025-01-14 17:29:12,447 - distributed.worker - INFO - Worker name: 0 +2025-01-14 17:29:12,461 - distributed.worker - INFO - dashboard at: 127.0.0.1:33181 +2025-01-14 17:29:12,471 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:39543 +2025-01-14 17:29:12,485 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 17:29:12,494 - distributed.worker - INFO - Threads: 1 +2025-01-14 17:29:12,508 - distributed.worker - INFO - Memory: 2.00 GiB +2025-01-14 17:29:12,518 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-m8551gy2 +2025-01-14 17:29:12,533 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 17:29:12,758 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:33735 name: 0 +2025-01-14 17:29:12,917 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:33735 +2025-01-14 17:29:12,931 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:41750 +2025-01-14 17:29:12,943 - distributed.worker - INFO - Starting Worker plugin shuffle +2025-01-14 17:29:12,967 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:39543 +2025-01-14 17:29:12,989 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 17:29:13,013 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:39543 +2025-01-14 17:29:13,232 - distributed.scheduler - INFO - Receive client connection: Client-b989b1d8-d227-11ef-8215-e761ae0addd2 +2025-01-14 17:29:13,409 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:41754 +2025-01-14 17:29:23,279 - distributed.worker.memory - WARNING - gc.collect() took 3.373s. This is usually a sign that some tasks handle too many Python objects at the same time. Rechunking the work into smaller tasks might help. +2025-01-14 17:29:23,293 - distributed.worker.memory - WARNING - Worker is at 94% memory usage. Pausing worker. Process memory: 1.90 GiB -- Worker memory limit: 2.00 GiB +2025-01-14 17:29:23,322 - distributed.core - INFO - Event loop was unresponsive in Scheduler for 9.91s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability. +2025-01-14 17:29:23,331 - distributed.core - INFO - Event loop was unresponsive in Worker for 9.81s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability. +2025-01-14 17:29:23,349 - distributed.scheduler - INFO - Remove client Client-b989b1d8-d227-11ef-8215-e761ae0addd2 +2025-01-14 17:29:23,359 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:41754; closing. +2025-01-14 17:29:23,359 - distributed.scheduler - INFO - Remove client Client-b989b1d8-d227-11ef-8215-e761ae0addd2 +2025-01-14 17:29:23,372 - distributed.scheduler - INFO - Close client connection: Client-b989b1d8-d227-11ef-8215-e761ae0addd2 +2025-01-14 17:29:23,381 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:33735. Reason: worker-close +2025-01-14 17:29:23,391 - distributed.worker - INFO - Removing Worker plugin shuffle +2025-01-14 17:29:23,402 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:41750; closing. +2025-01-14 17:29:23,402 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:33735 name: 0 (stimulus_id='handle-worker-cleanup-1736825363.402662') +2025-01-14 17:29:23,411 - distributed.scheduler - INFO - Lost all workers +2025-01-14 17:29:23,430 - distributed.core - INFO - Connection to tcp://127.0.0.1:39543 has been closed. +2025-01-14 17:29:23,468 - distributed.scheduler - INFO - Closing scheduler. Reason: unknown +2025-01-14 17:29:23,488 - distributed.scheduler - INFO - Scheduler closing all comms +____________________________ test_pause_while_idle _____________________________ + +s = +a = +b = + + @gen_cluster(config={"distributed.worker.memory.monitor-interval": "10ms"}) + async def test_pause_while_idle(s, a, b): + sa = s.workers[a.address] + assert a.address in s.idle + assert sa in s.running + + a.monitor.get_process_memory = lambda: 2**40 +> await async_poll_for(lambda: sa.status == Status.paused, timeout=5) + +distributed/tests/test_worker_memory.py:1105: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +predicate = . at 0x7f685765c720> +timeout = 5, fail_func = None, period = 0.05 + + async def async_poll_for(predicate, timeout, fail_func=None, period=0.05): + deadline = time() + timeout + while not predicate(): + await asyncio.sleep(period) + if time() > deadline: + if fail_func is not None: + fail_func() +> pytest.fail(f"condition not reached until {timeout} seconds") +E Failed: condition not reached until 5 seconds + +distributed/utils_test.py:1232: Failed +----------------------------- Captured stderr call ----------------------------- +2025-01-14 17:30:07,038 - distributed.scheduler - INFO - State start +2025-01-14 17:30:07,099 - distributed.scheduler - INFO - Scheduler at: tcp://127.0.0.1:42533 +2025-01-14 17:30:07,131 - distributed.scheduler - INFO - dashboard at: http://127.0.0.1:42665/status +2025-01-14 17:30:07,165 - distributed.scheduler - INFO - Registering Worker plugin shuffle +2025-01-14 17:30:07,250 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:33649 +2025-01-14 17:30:07,281 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:33649 +2025-01-14 17:30:07,298 - distributed.worker - INFO - Worker name: 0 +2025-01-14 17:30:07,333 - distributed.worker - INFO - dashboard at: 127.0.0.1:35169 +2025-01-14 17:30:07,350 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:42533 +2025-01-14 17:30:07,383 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 17:30:07,402 - distributed.worker - INFO - Threads: 1 +2025-01-14 17:30:07,437 - distributed.worker - INFO - Memory: 78.53 GiB +2025-01-14 17:30:07,455 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-m7xuf3yd +2025-01-14 17:30:07,489 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 17:30:07,619 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:42179 +2025-01-14 17:30:07,662 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:42179 +2025-01-14 17:30:07,701 - distributed.worker - INFO - Worker name: 1 +2025-01-14 17:30:07,745 - distributed.worker - INFO - dashboard at: 127.0.0.1:45797 +2025-01-14 17:30:07,767 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:42533 +2025-01-14 17:30:07,807 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 17:30:07,850 - distributed.worker - INFO - Threads: 2 +2025-01-14 17:30:07,890 - distributed.worker - INFO - Memory: 78.53 GiB +2025-01-14 17:30:07,934 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-01xnmheo +2025-01-14 17:30:07,974 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 17:30:08,629 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:33649 name: 0 +2025-01-14 17:30:08,921 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:33649 +2025-01-14 17:30:08,964 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:42768 +2025-01-14 17:30:08,965 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:42179 name: 1 +2025-01-14 17:30:09,286 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:42179 +2025-01-14 17:30:09,328 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:42782 +2025-01-14 17:30:09,349 - distributed.worker - INFO - Starting Worker plugin shuffle +2025-01-14 17:30:09,393 - distributed.worker - INFO - Starting Worker plugin shuffle +2025-01-14 17:30:09,433 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:42533 +2025-01-14 17:30:09,459 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 17:30:09,499 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:42533 +2025-01-14 17:30:09,542 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 17:30:09,583 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:42533 +2025-01-14 17:30:09,583 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:42533 +2025-01-14 17:30:26,899 - distributed.worker.memory - WARNING - gc.collect() took 3.899s. This is usually a sign that some tasks handle too many Python objects at the same time. Rechunking the work into smaller tasks might help. +2025-01-14 17:30:26,919 - distributed.worker.memory - WARNING - Worker is at 1303% memory usage. Pausing worker. Process memory: 1.00 TiB -- Worker memory limit: 78.53 GiB +2025-01-14 17:30:26,944 - distributed.core - INFO - Event loop was unresponsive in Scheduler for 17.60s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability. +2025-01-14 17:30:26,948 - distributed.core - INFO - Event loop was unresponsive in Worker for 17.45s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability. +2025-01-14 17:30:26,949 - distributed.core - INFO - Event loop was unresponsive in Worker for 17.37s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability. +2025-01-14 17:30:26,968 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:33649. Reason: worker-close +2025-01-14 17:30:26,982 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:42179. Reason: worker-close +2025-01-14 17:30:27,015 - distributed.worker - INFO - Removing Worker plugin shuffle +2025-01-14 17:30:27,051 - distributed.worker - INFO - Removing Worker plugin shuffle +2025-01-14 17:30:27,074 - distributed.worker - ERROR - Failed to communicate with scheduler during heartbeat. +Traceback (most recent call last): + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/comm/tcp.py", line 225, in read + frames_nosplit_nbytes_bin = await stream.read_bytes(fmt_size) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +tornado.iostream.StreamClosedError: Stream is closed + +The above exception was the direct cause of the following exception: + +Traceback (most recent call last): + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/worker.py", line 1269, in heartbeat + response = await retry_operation( + ^^^^^^^^^^^^^^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/utils_comm.py", line 441, in retry_operation + return await retry( + ^^^^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/utils_comm.py", line 420, in retry + return await coro() + ^^^^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/core.py", line 1259, in send_recv_from_rpc + return await send_recv(comm=comm, op=key, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/core.py", line 1018, in send_recv + response = await comm.read(deserializers=deserializers) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/comm/tcp.py", line 236, in read + convert_stream_closed_error(self, e) + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/comm/tcp.py", line 142, in convert_stream_closed_error + raise CommClosedError(f"in {obj}: {exc}") from exc +distributed.comm.core.CommClosedError: in : Stream is closed +2025-01-14 17:30:27,087 - distributed.worker - ERROR - Failed to communicate with scheduler during heartbeat. +Traceback (most recent call last): + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/comm/tcp.py", line 225, in read + frames_nosplit_nbytes_bin = await stream.read_bytes(fmt_size) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +tornado.iostream.StreamClosedError: Stream is closed + +The above exception was the direct cause of the following exception: + +Traceback (most recent call last): + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/worker.py", line 1269, in heartbeat + response = await retry_operation( + ^^^^^^^^^^^^^^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/utils_comm.py", line 441, in retry_operation + return await retry( + ^^^^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/utils_comm.py", line 420, in retry + return await coro() + ^^^^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/core.py", line 1259, in send_recv_from_rpc + return await send_recv(comm=comm, op=key, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/core.py", line 1018, in send_recv + response = await comm.read(deserializers=deserializers) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/comm/tcp.py", line 236, in read + convert_stream_closed_error(self, e) + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/comm/tcp.py", line 142, in convert_stream_closed_error + raise CommClosedError(f"in {obj}: {exc}") from exc +distributed.comm.core.CommClosedError: in : Stream is closed +2025-01-14 17:30:27,100 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:42768; closing. +2025-01-14 17:30:27,101 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:33649 name: 0 (stimulus_id='handle-worker-cleanup-1736825427.101748') +2025-01-14 17:30:27,109 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:42782; closing. +2025-01-14 17:30:27,110 - distributed.core - INFO - Connection to tcp://127.0.0.1:42533 has been closed. +2025-01-14 17:30:27,133 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:42179 name: 1 (stimulus_id='handle-worker-cleanup-1736825427.1335752') +2025-01-14 17:30:27,154 - distributed.scheduler - INFO - Lost all workers +2025-01-14 17:30:27,182 - distributed.core - INFO - Connection to tcp://127.0.0.1:42533 has been closed. +2025-01-14 17:30:27,182 - distributed.batched - INFO - Batched Comm Closed +Traceback (most recent call last): + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/batched.py", line 115, in _background_send + nbytes = yield coro + ^^^^^^^^^^ + File "/usr/lib/python3/dist-packages/tornado/gen.py", line 766, in run + value = future.result() + ^^^^^^^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/comm/tcp.py", line 262, in write + raise CommClosedError() +distributed.comm.core.CommClosedError +2025-01-14 17:30:27,195 - distributed.scheduler - INFO - Closing scheduler. Reason: unknown +2025-01-14 17:30:27,203 - distributed.scheduler - INFO - Scheduler closing all comms +__________________________ test_pause_while_saturated __________________________ + +c = +s = +a = +b = + + @gen_cluster(client=True, config={"distributed.worker.memory.monitor-interval": "10ms"}) + async def test_pause_while_saturated(c, s, a, b): + sa = s.workers[a.address] + ev = Event() + futs = c.map(lambda i, ev: ev.wait(), range(3), ev=ev, workers=[a.address]) + await async_poll_for(lambda: len(a.state.tasks) == 3, timeout=5) + assert sa in s.saturated + assert sa in s.running + + a.monitor.get_process_memory = lambda: 2**40 +> await async_poll_for(lambda: sa.status == Status.paused, timeout=5) + +distributed/tests/test_worker_memory.py:1125: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +predicate = . at 0x7f6856998860> +timeout = 5, fail_func = None, period = 0.05 + + async def async_poll_for(predicate, timeout, fail_func=None, period=0.05): + deadline = time() + timeout + while not predicate(): + await asyncio.sleep(period) + if time() > deadline: + if fail_func is not None: + fail_func() +> pytest.fail(f"condition not reached until {timeout} seconds") +E Failed: condition not reached until 5 seconds + +distributed/utils_test.py:1232: Failed +----------------------------- Captured stderr call ----------------------------- +2025-01-14 17:30:27,403 - distributed.scheduler - INFO - State start +2025-01-14 17:30:27,418 - distributed.scheduler - INFO - Scheduler at: tcp://127.0.0.1:32855 +2025-01-14 17:30:27,449 - distributed.scheduler - INFO - dashboard at: http://127.0.0.1:44167/status +2025-01-14 17:30:27,467 - distributed.scheduler - INFO - Registering Worker plugin shuffle +2025-01-14 17:30:27,602 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:44335 +2025-01-14 17:30:27,637 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:44335 +2025-01-14 17:30:27,677 - distributed.worker - INFO - Worker name: 0 +2025-01-14 17:30:27,713 - distributed.worker - INFO - dashboard at: 127.0.0.1:40353 +2025-01-14 17:30:27,756 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:32855 +2025-01-14 17:30:27,796 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 17:30:27,832 - distributed.worker - INFO - Threads: 1 +2025-01-14 17:30:27,866 - distributed.worker - INFO - Memory: 78.53 GiB +2025-01-14 17:30:27,911 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-z5tn4mve +2025-01-14 17:30:27,951 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 17:30:28,025 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:38497 +2025-01-14 17:30:28,062 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:38497 +2025-01-14 17:30:28,103 - distributed.worker - INFO - Worker name: 1 +2025-01-14 17:30:28,152 - distributed.worker - INFO - dashboard at: 127.0.0.1:40519 +2025-01-14 17:30:28,193 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:32855 +2025-01-14 17:30:28,230 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 17:30:28,274 - distributed.worker - INFO - Threads: 2 +2025-01-14 17:30:28,311 - distributed.worker - INFO - Memory: 78.53 GiB +2025-01-14 17:30:28,351 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-t7tz1ygl +2025-01-14 17:30:28,408 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 17:30:29,158 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:44335 name: 0 +2025-01-14 17:30:29,746 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:44335 +2025-01-14 17:30:29,810 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:45108 +2025-01-14 17:30:29,836 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:38497 name: 1 +2025-01-14 17:30:30,687 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:38497 +2025-01-14 17:30:30,749 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:45124 +2025-01-14 17:30:30,777 - distributed.worker - INFO - Starting Worker plugin shuffle +2025-01-14 17:30:30,900 - distributed.worker - INFO - Starting Worker plugin shuffle +2025-01-14 17:30:31,097 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:32855 +2025-01-14 17:30:31,148 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 17:30:31,183 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:32855 +2025-01-14 17:30:31,284 - distributed.worker - INFO - ------------------------------------------------- +2025-01-14 17:30:31,552 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:32855 +2025-01-14 17:30:31,552 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:32855 +2025-01-14 17:30:32,101 - distributed.scheduler - INFO - Receive client connection: Client-e859cd5e-d227-11ef-8215-e761ae0addd2 +2025-01-14 17:30:32,562 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:45136 +2025-01-14 17:30:53,124 - distributed.worker.memory - WARNING - gc.collect() took 4.272s. This is usually a sign that some tasks handle too many Python objects at the same time. Rechunking the work into smaller tasks might help. +2025-01-14 17:30:53,147 - distributed.worker.memory - WARNING - Worker is at 1303% memory usage. Pausing worker. Process memory: 1.00 TiB -- Worker memory limit: 78.53 GiB +2025-01-14 17:30:53,191 - distributed.core - INFO - Event loop was unresponsive in Scheduler for 20.60s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability. +2025-01-14 17:30:53,207 - distributed.core - INFO - Event loop was unresponsive in Worker for 20.62s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability. +2025-01-14 17:30:53,219 - distributed.core - INFO - Event loop was unresponsive in Worker for 20.50s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability. +2025-01-14 17:30:53,278 - distributed.scheduler - INFO - Remove client Client-e859cd5e-d227-11ef-8215-e761ae0addd2 +2025-01-14 17:30:53,301 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:45136; closing. +2025-01-14 17:30:53,302 - distributed.scheduler - INFO - Remove client Client-e859cd5e-d227-11ef-8215-e761ae0addd2 +2025-01-14 17:30:53,335 - distributed.scheduler - INFO - Close client connection: Client-e859cd5e-d227-11ef-8215-e761ae0addd2 +2025-01-14 17:30:53,359 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:44335. Reason: worker-close +2025-01-14 17:30:53,390 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:38497. Reason: worker-close +2025-01-14 17:30:53,456 - distributed.worker - INFO - Removing Worker plugin shuffle +2025-01-14 17:30:53,489 - distributed.worker.state_machine - WARNING - Async instruction for > ended with CancelledError +2025-01-14 17:30:53,515 - distributed.worker - INFO - Removing Worker plugin shuffle +2025-01-14 17:30:53,571 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:45124; closing. +2025-01-14 17:30:53,580 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:38497 name: 1 (stimulus_id='handle-worker-cleanup-1736825453.5807102') +2025-01-14 17:30:53,599 - distributed.core - INFO - Connection to tcp://127.0.0.1:32855 has been closed. +2025-01-14 17:30:53,625 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:45108; closing. +2025-01-14 17:30:53,689 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:44335 name: 0 (stimulus_id='handle-worker-cleanup-1736825453.6891234') +2025-01-14 17:30:53,716 - distributed.scheduler - INFO - Lost all workers +2025-01-14 17:30:53,731 - distributed.core - INFO - Connection to tcp://127.0.0.1:32855 has been closed. +2025-01-14 17:30:53,773 - distributed.scheduler - INFO - Closing scheduler. Reason: unknown +2025-01-14 17:30:53,796 - distributed.scheduler - INFO - Scheduler closing all comms +2025-01-14 17:30:53,820 - distributed.event - ERROR - +Traceback (most recent call last): + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/utils.py", line 809, in wrapper + return await func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/event.py", line 77, in event_wait + await future + File "/usr/lib/python3.12/asyncio/locks.py", line 212, in wait + await fut +asyncio.exceptions.CancelledError ============================= slowest 20 durations ============================= -12.08s call distributed/tests/test_gc.py::test_gc_diagnosis_cpu_time -10.02s call distributed/tests/test_utils_test.py::test_popen_timeout -6.29s call distributed/tests/test_failed_workers.py::test_worker_doesnt_await_task_completion -5.82s call distributed/tests/test_stress.py::test_cancel_stress_sync -5.57s call distributed/tests/test_nanny.py::test_num_fds -5.19s call distributed/tests/test_steal.py::test_balance_with_longer_task -4.91s call distributed/tests/test_chaos.py::test_KillWorker[sys.exit] -4.75s call distributed/tests/test_stress.py::test_cancel_stress -4.57s call distributed/tests/test_worker.py::test_acquire_replicas_many -4.51s call distributed/diagnostics/tests/test_progress.py::test_group_timing -4.38s call distributed/shuffle/tests/test_rechunk.py::test_rechunk_method -4.37s call distributed/tests/test_worker.py::test_log_remove_worker -4.09s call distributed/tests/test_failed_workers.py::test_restart_sync -3.68s call distributed/shuffle/tests/test_rechunk.py::test_homogeneously_schedule_unpack -3.62s call distributed/diagnostics/tests/test_progress.py::test_AllProgress -3.54s call distributed/tests/test_nanny.py::test_environ_plugin -3.53s call distributed/tests/test_tls_functional.py::test_retire_workers -3.23s call distributed/tests/test_failed_workers.py::test_restart_cleared -3.20s call distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_partially_unknown_dimension[x5-chunks5] -3.19s call distributed/tests/test_nanny.py::test_nanny_closes_cleanly_if_worker_is_terminated +38.88s call distributed/tests/test_stress.py::test_cancel_stress +33.90s call distributed/tests/test_steal.py::test_trivial_workload_should_not_cause_work_stealing +32.28s call distributed/shuffle/tests/test_rechunk.py::test_rechunk_method +31.88s call distributed/tests/test_gc.py::test_gc_diagnosis_cpu_time +26.47s call distributed/tests/test_worker_memory.py::test_pause_while_saturated +23.99s call distributed/tests/test_steal.py::test_balance_multiple_to_replica +21.80s call distributed/protocol/tests/test_numpy.py::test_dumps_large +21.43s call distributed/tests/test_steal.py::test_balance_to_larger_dependency +21.01s call distributed/tests/test_steal.py::test_balance_prefers_busier_with_dependency +20.20s call distributed/tests/test_worker_memory.py::test_pause_while_idle +19.68s call distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_fully_unknown_dimension[x4-chunks4] +19.29s call distributed/tests/test_nanny.py::test_environ_plugin +19.13s call distributed/tests/test_cancelled_state.py::test_deadlock_cancelled_after_inflight_before_gather_from_worker[True-cancelled] +19.12s call distributed/tests/test_steal.py::test_balance_to_replica +18.74s call distributed/tests/test_worker_memory.py::test_pause_executor_with_memory_monitor +18.39s call distributed/tests/test_worker.py::test_avoid_oversubscription +17.45s call distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_fully_unknown_dimension[x3-chunks3] +16.55s call distributed/protocol/tests/test_serialize.py::test_large_pickled_object +16.45s call distributed/tests/test_stress.py::test_cancel_stress_sync +15.79s call distributed/shuffle/tests/test_rechunk.py::test_rechunk_with_partially_unknown_dimension[x3-chunks3] =========================== short test summary info ============================ SKIPPED [1] distributed/cli/tests/test_dask_ssh.py:9: could not import 'paramiko': No module named 'paramiko' SKIPPED [1] distributed/comm/tests/test_ucx.py:15: could not import 'ucp': No module named 'ucp' @@ -11699,65 +14662,185 @@ SKIPPED [1] distributed/tests/test_worker_memory.py:871: need --runslow option to run SKIPPED [2] distributed/tests/test_worker_memory.py:883: need --runslow option to run SKIPPED [1] distributed/tests/test_worker_memory.py:997: need --runslow option to run -= 2913 passed, 264 skipped, 222 deselected, 15 xfailed, 8 xpassed in 1027.60s (0:17:07) = -*** END OF RUN 1: ALL TESTS RUN HAVE NOW PASSED/XFAILED *** - create-stamp debian/debhelper-build-stamp - dh_testroot -O--buildsystem=pybuild - dh_prep -O--buildsystem=pybuild - debian/rules override_dh_auto_install -make[1]: Entering directory '/build/reproducible-path/dask.distributed-2024.12.1+ds' -dh_auto_install -I: pybuild plugin_pyproject:178: Copying package built for python3.13 to destdir -I: pybuild plugin_pyproject:178: Copying package built for python3.12 to destdir -dh_link -find debian/ -name *.ico -exec chmod a-x {} \; -rm -rf debian/python3-distributed/usr/lib/python3*/dist-packages/dask-worker-space debian/python3-distributed/usr/lib/python3*/dist-packages/core debian/python3-distributed/usr/lib/python3*/dist-packages/test_cluster_dump -make[1]: Leaving directory '/build/reproducible-path/dask.distributed-2024.12.1+ds' - debian/rules override_dh_install -make[1]: Entering directory '/build/reproducible-path/dask.distributed-2024.12.1+ds' -dh_install -dh_installchangelogs docs/source/changelog.rst -make[1]: Leaving directory '/build/reproducible-path/dask.distributed-2024.12.1+ds' - debian/rules override_dh_installdocs -make[1]: Entering directory '/build/reproducible-path/dask.distributed-2024.12.1+ds' -dh_installdocs -ppython-distributed-doc --doc-main-package=python3-distributed -dh_installdocs --remaining-packages -rm -vf debian/python-distributed-doc/usr/share/doc/python3-distributed/html/_static/vendor/fontawesome/*/LICENSE.txt -make[1]: Leaving directory '/build/reproducible-path/dask.distributed-2024.12.1+ds' - dh_sphinxdoc -O--buildsystem=pybuild -dh_sphinxdoc: warning: ignoring unknown JavaScript code: debian/python-distributed-doc/usr/share/doc/python3-distributed/html/_static/design-tabs.js - dh_installchangelogs -O--buildsystem=pybuild - dh_python3 -O--buildsystem=pybuild - dh_installsystemduser -O--buildsystem=pybuild - dh_perl -O--buildsystem=pybuild - dh_link -O--buildsystem=pybuild - dh_strip_nondeterminism -O--buildsystem=pybuild - debian/rules override_dh_compress -make[1]: Entering directory '/build/reproducible-path/dask.distributed-2024.12.1+ds' -dh_compress -Xchangelog.html -make[1]: Leaving directory '/build/reproducible-path/dask.distributed-2024.12.1+ds' - dh_fixperms -O--buildsystem=pybuild - dh_missing -O--buildsystem=pybuild - dh_installdeb -O--buildsystem=pybuild - dh_gencontrol -O--buildsystem=pybuild - dh_md5sums -O--buildsystem=pybuild - dh_builddeb -O--buildsystem=pybuild -dpkg-deb: building package 'python-distributed-doc' in '../python-distributed-doc_2024.12.1+ds-1_all.deb'. -dpkg-deb: building package 'python3-distributed' in '../python3-distributed_2024.12.1+ds-1_all.deb'. - dpkg-genbuildinfo --build=binary -O../dask.distributed_2024.12.1+ds-1_amd64.buildinfo - dpkg-genchanges --build=binary -O../dask.distributed_2024.12.1+ds-1_amd64.changes -dpkg-genchanges: info: binary-only upload (no source code included) - dpkg-source --after-build . -dpkg-source: info: using options from dask.distributed-2024.12.1+ds/debian/source/options: --extend-diff-ignore=^[^/]+\.egg-info/ -dpkg-buildpackage: info: binary-only upload (no source included) -dpkg-genchanges: info: including full source code in upload +FAILED distributed/shuffle/tests/test_rechunk.py::test_rechunk_method - TimeoutError: Test timeout (30) hit after 30.050075880018994s. +========== Test stack trace starts here ========== +Stack for wait_for=> (most recent call last): + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/shuffle/tests/test_rechunk.py", line 414, in test_rechunk_method + assert np.all(await c.compute(x2) == a) +FAILED distributed/tests/test_priorities.py::test_last_in_first_out[queue on worker] - assert not True + + where True = all(. at 0x7f68853d6f60>) +FAILED distributed/tests/test_steal.py::test_trivial_workload_should_not_cause_work_stealing - TimeoutError: Test timeout (30) hit after 31.710481404006714s. +========== Test stack trace starts here ========== +Stack for wait_for=> (most recent call last): + File "/build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build/distributed/tests/test_steal.py", line 1878, in test_trivial_workload_should_not_cause_work_stealing + await c.gather(futs) +FAILED distributed/tests/test_worker.py::test_log_remove_worker - assert ["Retire worker addresses (stimulus_id='graceful') ['tcp://127.0.0.1:40277']", "Remove worker addr: tcp://127.0.0.1:40277 name: 0 (stimulus_id='graceful')", "Retired worker 'tcp://127.0.0.1:40277' (stimulus_id='graceful')", "Retire worker addresses (stimulus_id='graceful_abort') ['tcp://127.0.0.1:33441']", "Could not retire worker 'tcp://127.0.0.1:33441': unique data could not be moved to any other worker (stimulus_id='graceful_abort')", "Received heartbeat from unregistered worker 'tcp://127.0.0.1:40277'.", "Remove worker addr: tcp://127.0.0.1:33441 name: 1 (stimulus_id='ungraceful')", "Removing worker 'tcp://127.0.0.1:33441' caused the cluster to lose already computed task(s), which will be recomputed elsewhere: {'x'} (stimulus_id='ungraceful')", "Removing worker 'tcp://127.0.0.1:33441' caused the cluster to lose scattered data, which can't be recovered: {'z'} (stimulus_id='ungraceful')", 'Lost all workers', "Received heartbeat from unregistered worker 'tcp://127.0.0.1:33441'."] == ["Retire worker addresses (stimulus_id='graceful') ['tcp://127.0.0.1:40277']", "Remove worker addr: tcp://127.0.0.1:40277 name: 0 (stimulus_id='graceful')", "Retired worker 'tcp://127.0.0.1:40277' (stimulus_id='graceful')", "Retire worker addresses (stimulus_id='graceful_abort') ['tcp://127.0.0.1:33441']", "Could not retire worker 'tcp://127.0.0.1:33441': unique data could not be moved to any other worker (stimulus_id='graceful_abort')", "Remove worker addr: tcp://127.0.0.1:33441 name: 1 (stimulus_id='ungraceful')", "Removing worker 'tcp://127.0.0.1:33441' caused the cluster to lose already computed task(s), which will be recomputed elsewhere: {'x'} (stimulus_id='ungraceful')", "Removing worker 'tcp://127.0.0.1:33441' caused the cluster to lose scattered data, which can't be recovered: {'z'} (stimulus_id='ungraceful')", 'Lost all workers'] + + At index 5 diff: "Received heartbeat from unregistered worker 'tcp://127.0.0.1:40277'." != "Remove worker addr: tcp://127.0.0.1:33441 name: 1 (stimulus_id='ungraceful')" + Left contains 2 more items, first extra item: 'Lost all workers' + + Full diff: + [ + "Retire worker addresses (stimulus_id='graceful') " + "['tcp://127.0.0.1:40277']", + 'Remove worker addr: tcp://127.0.0.1:40277 name: 0 ' + "(stimulus_id='graceful')", + "Retired worker 'tcp://127.0.0.1:40277' (stimulus_id='graceful')", + "Retire worker addresses (stimulus_id='graceful_abort') " + "['tcp://127.0.0.1:33441']", + "Could not retire worker 'tcp://127.0.0.1:33441': unique data could not be " + "moved to any other worker (stimulus_id='graceful_abort')", + + "Received heartbeat from unregistered worker 'tcp://127.0.0.1:40277'.", + 'Remove worker addr: tcp://127.0.0.1:33441 name: 1 ' + "(stimulus_id='ungraceful')", + "Removing worker 'tcp://127.0.0.1:33441' caused the cluster to lose " + "already computed task(s), which will be recomputed elsewhere: {'x'} " + "(stimulus_id='ungraceful')", + "Removing worker 'tcp://127.0.0.1:33441' caused the cluster to lose " + "scattered data, which can't be recovered: {'z'} " + "(stimulus_id='ungraceful')", + 'Lost all workers', + + "Received heartbeat from unregistered worker 'tcp://127.0.0.1:33441'.", + ] +FAILED distributed/tests/test_worker_memory.py::test_fail_to_pickle_spill - TimeoutError +FAILED distributed/tests/test_worker_memory.py::test_pause_while_spilling - Failed: condition not reached until 5 seconds +FAILED distributed/tests/test_worker_memory.py::test_pause_while_idle - Failed: condition not reached until 5 seconds +FAILED distributed/tests/test_worker_memory.py::test_pause_while_saturated - Failed: condition not reached until 5 seconds += 8 failed, 2905 passed, 264 skipped, 222 deselected, 15 xfailed, 8 xpassed in 4164.96s (1:09:24) = +*** END OF RUN 1: NOT ALL TESTS HAVE YET PASSED/XFAILED *** +*** STARTING RUN 2: python3.12 -m pytest --pyargs distributed --verbose --color=no --timeout-method=signal --timeout=300 -m not avoid_ci -rfE --last-failed --last-failed-no-failures none --ignore=distributed/comm/tests/test_comms.py --ignore=distributed/comm/tests/test_ws.py --ignore=distributed/deploy/tests/test_adaptive.py --ignore=distributed/deploy/tests/test_local.py --ignore=distributed/deploy/tests/test_slow_adaptive.py --ignore=distributed/deploy/tests/test_spec_cluster.py --deselect=distributed/cli/tests/test_dask_scheduler.py::test_no_dashboard --deselect=distributed/deploy/tests/test_local.py::test_localcluster_get_client --deselect=distributed/deploy/tests/test_old_ssh.py::test_cluster --deselect=distributed/deploy/tests/test_old_ssh.py::test_old_ssh_nprocs_renamed_to_n_workers --deselect=distributed/deploy/tests/test_old_ssh.py::test_nprocs_attribute_is_deprecated --deselect=distributed/deploy/tests/test_ssh.py::test_nprocs_attribute_is_deprecated --deselect=distributed/http/tests/test_core.py::test_prometheus_api_doc --deselect=distributed/tests/test_init.py::test_git_revision --deselect=distributed/tests/test_jupyter.py::test_jupyter_idle_timeout --deselect=distributed/tests/test_jupyter.py::test_jupyter_idle_timeout_returned --deselect=distributed/tests/test_jupyter.py::test_jupyter_server --deselect=distributed/tests/test_jupyter.py::test_shutsdown_cleanly --deselect=distributed/tests/test_profile.py::test_stack_overflow --deselect=distributed/tests/test_pubsub.py::test_client_worker --deselect=distributed/tests/test_queues.py::test_queue_in_task --deselect=distributed/tests/test_spill.py::test_spillbuffer_oserror --deselect=distributed/tests/test_steal.py::test_steal_twice --deselect=distributed/tests/test_utils_test.py::test_cluster --deselect=distributed/tests/test_variable.py::test_variable_in_task --deselect=distributed/tests/test_worker.py::test_process_executor_kills_process --deselect=distributed/tests/test_worker_memory.py::test_fail_to_pickle_execute_1 --deselect=distributed/tests/test_worker_state_machine.py::test_task_state_instance_are_garbage_collected --deselect=distributed/protocol/tests/test_protocol.py::test_deeply_nested_structures --deselect=distributed/protocol/tests/test_serialize.py::test_deeply_nested_structures --deselect=distributed/cli/tests/test_dask_scheduler.py::test_defaults --deselect=distributed/cli/tests/test_dask_scheduler.py::test_hostport --deselect=distributed/cli/tests/test_dask_spec.py::test_errors --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_file --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_file --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_file --deselect=distributed/dashboard/tests/test_scheduler_bokeh.py::test_counters --deselect=distributed/dashboard/tests/test_worker_bokeh.py::test_counters --deselect=distributed/deploy/tests/test_local.py::test_adapt_then_manual --deselect=distributed/deploy/tests/test_local.py::test_async_with --deselect=distributed/deploy/tests/test_local.py::test_close_twice --deselect=distributed/deploy/tests/test_local.py::test_cluster_info_sync --deselect=distributed/deploy/tests/test_local.py::test_local_tls --deselect=distributed/deploy/tests/test_local.py::test_no_dangling_asyncio_tasks --deselect=distributed/deploy/tests/test_local.py::test_only_local_access --deselect=distributed/deploy/tests/test_local.py::test_remote_access --deselect=distributed/diagnostics/tests/test_progress_widgets.py::test_serializers --deselect=distributed/diagnostics/tests/test_scheduler_plugin.py::test_lifecycle --deselect=distributed/http/scheduler/tests/test_missing_bokeh.py::test_missing_bokeh --deselect=distributed/http/scheduler/tests/test_scheduler_http.py::test_metrics_when_prometheus_client_not_installed --deselect=distributed/protocol/tests/test_serialize.py::test_errors --deselect=distributed/tests/test_batched.py::test_BatchedSend --deselect=distributed/tests/test_batched.py::test_close_closed --deselect=distributed/tests/test_batched.py::test_close_twice --deselect=distributed/tests/test_batched.py::test_send_after_stream_start --deselect=distributed/tests/test_batched.py::test_send_before_close --deselect=distributed/tests/test_batched.py::test_send_before_start --deselect=distributed/tests/test_batched.py::test_sending_traffic_jam --deselect=distributed/tests/test_batched.py::test_serializers --deselect=distributed/tests/test_client.py::TestClientSecurityLoader::test_security_loader --deselect=distributed/tests/test_client.py::TestClientSecurityLoader::test_security_loader_ignored_if_explicit_security_provided --deselect=distributed/tests/test_client.py::TestClientSecurityLoader::test_security_loader_ignored_if_returns_none --deselect=distributed/tests/test_client.py::test_async_with --deselect=distributed/tests/test_client.py::test_client_is_quiet_cluster_close --deselect=distributed/tests/test_client.py::test_dashboard_link_cluster --deselect=distributed/tests/test_client.py::test_dashboard_link_inproc --deselect=distributed/tests/test_client.py::test_file_descriptors_dont_leak --deselect=distributed/tests/test_client.py::test_mixing_clients_different_scheduler --deselect=distributed/tests/test_client.py::test_quiet_client_close --deselect=distributed/tests/test_client.py::test_rebalance_sync --deselect=distributed/tests/test_client.py::test_repr_localcluster --deselect=distributed/tests/test_client.py::test_security_loader --deselect=distributed/tests/test_client.py::test_security_loader_ignored_if_explicit_security_provided --deselect=distributed/tests/test_client.py::test_security_loader_ignored_if_returns_none --deselect=distributed/tests/test_client.py::test_shutdown --deselect=distributed/tests/test_client.py::test_shutdown_is_quiet_with_cluster --deselect=distributed/tests/test_client.py::test_shutdown_localcluster --deselect=distributed/tests/test_client.py::test_shutdown_stops_callbacks --deselect=distributed/tests/test_client_loop.py::test_close_loop_sync_start_new_loop --deselect=distributed/tests/test_client_loop.py::test_close_loop_sync_use_running_loop --deselect=distributed/tests/test_core.py::test_close_fast_without_active_handlers --deselect=distributed/tests/test_core.py::test_close_grace_period_for_handlers --deselect=distributed/tests/test_core.py::test_close_properly --deselect=distributed/tests/test_core.py::test_compression --deselect=distributed/tests/test_core.py::test_connection_pool --deselect=distributed/tests/test_core.py::test_connection_pool_close_while_connecting --deselect=distributed/tests/test_core.py::test_connection_pool_detects_remote_close --deselect=distributed/tests/test_core.py::test_connection_pool_outside_cancellation --deselect=distributed/tests/test_core.py::test_connection_pool_remove --deselect=distributed/tests/test_core.py::test_connection_pool_respects_limit --deselect=distributed/tests/test_core.py::test_connection_pool_tls --deselect=distributed/tests/test_core.py::test_counters --deselect=distributed/tests/test_core.py::test_deserialize_error --deselect=distributed/tests/test_core.py::test_errors --deselect=distributed/tests/test_core.py::test_identity_inproc --deselect=distributed/tests/test_core.py::test_identity_tcp --deselect=distributed/tests/test_core.py::test_large_packets_inproc --deselect=distributed/tests/test_core.py::test_messages_are_ordered_bsend --deselect=distributed/tests/test_core.py::test_messages_are_ordered_raw --deselect=distributed/tests/test_core.py::test_ports --deselect=distributed/tests/test_core.py::test_rpc_default --deselect=distributed/tests/test_core.py::test_rpc_inproc --deselect=distributed/tests/test_core.py::test_rpc_message_lifetime_default --deselect=distributed/tests/test_core.py::test_rpc_message_lifetime_inproc --deselect=distributed/tests/test_core.py::test_rpc_message_lifetime_tcp --deselect=distributed/tests/test_core.py::test_rpc_serialization --deselect=distributed/tests/test_core.py::test_rpc_tcp --deselect=distributed/tests/test_core.py::test_rpc_tls --deselect=distributed/tests/test_core.py::test_rpc_with_many_connections_inproc --deselect=distributed/tests/test_core.py::test_rpc_with_many_connections_tcp --deselect=distributed/tests/test_core.py::test_send_recv_args --deselect=distributed/tests/test_core.py::test_send_recv_cancelled --deselect=distributed/tests/test_core.py::test_server --deselect=distributed/tests/test_core.py::test_server_comms_mark_active_handlers --deselect=distributed/tests/test_core.py::test_server_raises_on_blocked_handlers --deselect=distributed/tests/test_jupyter.py::test_jupyter_idle_timeout --deselect=distributed/tests/test_jupyter.py::test_jupyter_server --deselect=distributed/tests/test_locks.py::test_errors --deselect=distributed/tests/test_nanny.py::test_scheduler_file --deselect=distributed/tests/test_nanny.py::test_scheduler_file --deselect=distributed/tests/test_nanny.py::test_scheduler_file --deselect=distributed/tests/test_nanny.py::test_worker_uses_same_host_as_nanny --deselect=distributed/tests/test_preload.py::test_failure_doesnt_crash_scheduler --deselect=distributed/tests/test_preload.py::test_preload_import_time --deselect=distributed/tests/test_preload.py::test_preload_manager_sequence --deselect=distributed/tests/test_preload.py::test_worker_preload_text --deselect=distributed/tests/test_scheduler.py::test_allowed_failures_config --deselect=distributed/tests/test_scheduler.py::test_async_context_manager --deselect=distributed/tests/test_scheduler.py::test_dashboard_host --deselect=distributed/tests/test_scheduler.py::test_file_descriptors_dont_leak --deselect=distributed/tests/test_scheduler.py::test_finished --deselect=distributed/tests/test_scheduler.py::test_multiple_listeners --deselect=distributed/tests/test_scheduler.py::test_no_dangling_asyncio_tasks --deselect=distributed/tests/test_scheduler.py::test_scheduler_file --deselect=distributed/tests/test_scheduler.py::test_scheduler_file --deselect=distributed/tests/test_scheduler.py::test_scheduler_file --deselect=distributed/tests/test_security.py::test_require_encryption --deselect=distributed/tests/test_security.py::test_tls_listen_connect --deselect=distributed/tests/test_security.py::test_tls_temporary_credentials_functional --deselect=distributed/tests/test_semaphore.py::test_threadpoolworkers_pick_correct_ioloop --deselect=distributed/tests/test_tls_functional.py::test_security_dict_input_no_security --deselect=distributed/tests/test_utils_test.py::test_ensure_no_new_clients --deselect=distributed/tests/test_utils_test.py::test_freeze_batched_send --deselect=distributed/tests/test_utils_test.py::test_locked_comm_drop_in_replacement --deselect=distributed/tests/test_utils_test.py::test_locked_comm_intercept_read --deselect=distributed/tests/test_utils_test.py::test_locked_comm_intercept_write --deselect=distributed/tests/test_worker.py::test_host_uses_scheduler_protocol --deselect=distributed/tests/test_worker.py::test_plugin_exception --deselect=distributed/tests/test_worker.py::test_plugin_internal_exception --deselect=distributed/tests/test_worker.py::test_plugin_multiple_exceptions --deselect=distributed/tests/test_worker.py::test_scheduler_file --deselect=distributed/tests/test_worker.py::test_scheduler_file --deselect=distributed/tests/test_worker.py::test_scheduler_file --deselect=distributed/tests/test_worker_client.py::test_dont_override_default_get --deselect=distributed/cli/tests/test_dask_scheduler.py::test_dashboard --deselect=distributed/cli/tests/test_dask_scheduler.py::test_dashboard_allowlist --deselect=distributed/cli/tests/test_dask_scheduler.py::test_dashboard_non_standard_ports --deselect=distributed/cli/tests/test_dask_scheduler.py::test_dashboard_port_zero --deselect=distributed/cli/tests/test_dask_scheduler.py::test_defaults --deselect=distributed/cli/tests/test_dask_scheduler.py::test_hostport --deselect=distributed/cli/tests/test_dask_scheduler.py::test_multiple_protocols --deselect=distributed/cli/tests/test_dask_scheduler.py::test_multiple_workers --deselect=distributed/cli/tests/test_dask_scheduler.py::test_multiple_workers_2 --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_command --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_command_default --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_config --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_file --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_module --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_remote_module --deselect=distributed/cli/tests/test_dask_scheduler.py::test_scheduler_port_zero --deselect=distributed/cli/tests/test_dask_scheduler.py::test_single_executable_deprecated --deselect=distributed/cli/tests/test_dask_worker.py::test_contact_listen_address --deselect=distributed/cli/tests/test_dask_worker.py::test_dashboard_non_standard_ports --deselect=distributed/cli/tests/test_dask_worker.py::test_error_during_startup --deselect=distributed/cli/tests/test_dask_worker.py::test_integer_names --deselect=distributed/cli/tests/test_dask_worker.py::test_listen_address_ipv6 --deselect=distributed/cli/tests/test_dask_worker.py::test_local_directory --deselect=distributed/cli/tests/test_dask_worker.py::test_memory_limit --deselect=distributed/cli/tests/test_dask_worker.py::test_nanny_worker_port_range --deselect=distributed/cli/tests/test_dask_worker.py::test_nanny_worker_port_range_too_many_workers_raises --deselect=distributed/cli/tests/test_dask_worker.py::test_nanny_worker_ports --deselect=distributed/cli/tests/test_dask_worker.py::test_no_nanny --deselect=distributed/cli/tests/test_dask_worker.py::test_nworkers_auto --deselect=distributed/cli/tests/test_dask_worker.py::test_nworkers_expands_name --deselect=distributed/cli/tests/test_dask_worker.py::test_nworkers_negative --deselect=distributed/cli/tests/test_dask_worker.py::test_nworkers_requires_nanny --deselect=distributed/cli/tests/test_dask_worker.py::test_preload_config --deselect=distributed/cli/tests/test_dask_worker.py::test_resources --deselect=distributed/cli/tests/test_dask_worker.py::test_respect_host_listen_address --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_address_env --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_file --deselect=distributed/cli/tests/test_dask_worker.py::test_set_lifetime_restart_via_env_var --deselect=distributed/cli/tests/test_dask_worker.py::test_set_lifetime_stagger_via_env_var --deselect=distributed/cli/tests/test_dask_worker.py::test_signal_handling --deselect=distributed/cli/tests/test_dask_worker.py::test_single_executable_deprecated --deselect=distributed/cli/tests/test_dask_worker.py::test_single_executable_works --deselect=distributed/cli/tests/test_dask_worker.py::test_timeout --deselect=distributed/cli/tests/test_dask_worker.py::test_worker_class --deselect=distributed/tests/test_config.py::test_logging_extended --deselect=distributed/tests/test_config.py::test_logging_file_config --deselect=distributed/tests/test_config.py::test_logging_mutual_exclusive --deselect=distributed/tests/test_config.py::test_logging_simple --deselect=distributed/tests/test_config.py::test_logging_simple_under_distributed --deselect=distributed/tests/test_queues.py::test_queue_in_task --deselect=distributed/tests/test_variable.py::test_variable_in_task +============================= test session starts ============================== +platform linux -- Python 3.12.8, pytest-8.3.4, pluggy-1.5.0 -- /usr/bin/python3.12 +cachedir: .pytest_cache +rootdir: /build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build +configfile: pyproject.toml +plugins: rerunfailures-14.0, anyio-4.7.0, timeout-2.3.1, typeguard-4.4.1 +timeout: 300.0s +timeout method: signal +timeout func_only: False +collecting ... collected 48 items / 1 error / 41 deselected / 5 skipped / 7 selected +run-last-failure: rerun previous 7 failures (skipped 130 files) + +==================================== ERRORS ==================================== +__________ ERROR collecting distributed/shuffle/tests/test_rechunk.py __________ +distributed/shuffle/tests/test_rechunk.py:39: in + from distributed.shuffle.tests.utils import AbstractShuffleTestPool +distributed/shuffle/tests/utils.py:11: in + import dask.dataframe as dd +/usr/lib/python3/dist-packages/dask/dataframe/__init__.py:115: in + if _dask_expr_enabled(): +/usr/lib/python3/dist-packages/dask/dataframe/__init__.py:31: in _dask_expr_enabled + warnings.warn( +E FutureWarning: The legacy Dask DataFrame implementation is deprecated and will be removed in a future version. Set the configuration option `dataframe.query-planning` to `True` or None to enable the new Dask Dataframe implementation and silence this warning. +=========================== short test summary info ============================ +ERROR distributed/shuffle/tests/test_rechunk.py - FutureWarning: The legacy Dask DataFrame implementation is deprecated and will be removed in a future version. Set the configuration option `dataframe.query-planning` to `True` or None to enable the new Dask Dataframe implementation and silence this warning. +!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!! +================== 5 skipped, 41 deselected, 1 error in 4.21s ================== +*** END OF RUN 2: NOT ALL TESTS HAVE YET PASSED/XFAILED *** +*** STARTING RUN 3: python3.12 -m pytest --pyargs distributed --verbose --color=no --timeout-method=signal --timeout=300 -m not avoid_ci -rfE --last-failed --last-failed-no-failures none --ignore=distributed/comm/tests/test_comms.py --ignore=distributed/comm/tests/test_ws.py --ignore=distributed/deploy/tests/test_adaptive.py --ignore=distributed/deploy/tests/test_local.py --ignore=distributed/deploy/tests/test_slow_adaptive.py --ignore=distributed/deploy/tests/test_spec_cluster.py --deselect=distributed/cli/tests/test_dask_scheduler.py::test_no_dashboard --deselect=distributed/deploy/tests/test_local.py::test_localcluster_get_client --deselect=distributed/deploy/tests/test_old_ssh.py::test_cluster --deselect=distributed/deploy/tests/test_old_ssh.py::test_old_ssh_nprocs_renamed_to_n_workers --deselect=distributed/deploy/tests/test_old_ssh.py::test_nprocs_attribute_is_deprecated --deselect=distributed/deploy/tests/test_ssh.py::test_nprocs_attribute_is_deprecated --deselect=distributed/http/tests/test_core.py::test_prometheus_api_doc --deselect=distributed/tests/test_init.py::test_git_revision --deselect=distributed/tests/test_jupyter.py::test_jupyter_idle_timeout --deselect=distributed/tests/test_jupyter.py::test_jupyter_idle_timeout_returned --deselect=distributed/tests/test_jupyter.py::test_jupyter_server --deselect=distributed/tests/test_jupyter.py::test_shutsdown_cleanly --deselect=distributed/tests/test_profile.py::test_stack_overflow --deselect=distributed/tests/test_pubsub.py::test_client_worker --deselect=distributed/tests/test_queues.py::test_queue_in_task --deselect=distributed/tests/test_spill.py::test_spillbuffer_oserror --deselect=distributed/tests/test_steal.py::test_steal_twice --deselect=distributed/tests/test_utils_test.py::test_cluster --deselect=distributed/tests/test_variable.py::test_variable_in_task --deselect=distributed/tests/test_worker.py::test_process_executor_kills_process --deselect=distributed/tests/test_worker_memory.py::test_fail_to_pickle_execute_1 --deselect=distributed/tests/test_worker_state_machine.py::test_task_state_instance_are_garbage_collected --deselect=distributed/protocol/tests/test_protocol.py::test_deeply_nested_structures --deselect=distributed/protocol/tests/test_serialize.py::test_deeply_nested_structures --deselect=distributed/cli/tests/test_dask_scheduler.py::test_defaults --deselect=distributed/cli/tests/test_dask_scheduler.py::test_hostport --deselect=distributed/cli/tests/test_dask_spec.py::test_errors --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_file --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_file --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_file --deselect=distributed/dashboard/tests/test_scheduler_bokeh.py::test_counters --deselect=distributed/dashboard/tests/test_worker_bokeh.py::test_counters --deselect=distributed/deploy/tests/test_local.py::test_adapt_then_manual --deselect=distributed/deploy/tests/test_local.py::test_async_with --deselect=distributed/deploy/tests/test_local.py::test_close_twice --deselect=distributed/deploy/tests/test_local.py::test_cluster_info_sync --deselect=distributed/deploy/tests/test_local.py::test_local_tls --deselect=distributed/deploy/tests/test_local.py::test_no_dangling_asyncio_tasks --deselect=distributed/deploy/tests/test_local.py::test_only_local_access --deselect=distributed/deploy/tests/test_local.py::test_remote_access --deselect=distributed/diagnostics/tests/test_progress_widgets.py::test_serializers --deselect=distributed/diagnostics/tests/test_scheduler_plugin.py::test_lifecycle --deselect=distributed/http/scheduler/tests/test_missing_bokeh.py::test_missing_bokeh --deselect=distributed/http/scheduler/tests/test_scheduler_http.py::test_metrics_when_prometheus_client_not_installed --deselect=distributed/protocol/tests/test_serialize.py::test_errors --deselect=distributed/tests/test_batched.py::test_BatchedSend --deselect=distributed/tests/test_batched.py::test_close_closed --deselect=distributed/tests/test_batched.py::test_close_twice --deselect=distributed/tests/test_batched.py::test_send_after_stream_start --deselect=distributed/tests/test_batched.py::test_send_before_close --deselect=distributed/tests/test_batched.py::test_send_before_start --deselect=distributed/tests/test_batched.py::test_sending_traffic_jam --deselect=distributed/tests/test_batched.py::test_serializers --deselect=distributed/tests/test_client.py::TestClientSecurityLoader::test_security_loader --deselect=distributed/tests/test_client.py::TestClientSecurityLoader::test_security_loader_ignored_if_explicit_security_provided --deselect=distributed/tests/test_client.py::TestClientSecurityLoader::test_security_loader_ignored_if_returns_none --deselect=distributed/tests/test_client.py::test_async_with --deselect=distributed/tests/test_client.py::test_client_is_quiet_cluster_close --deselect=distributed/tests/test_client.py::test_dashboard_link_cluster --deselect=distributed/tests/test_client.py::test_dashboard_link_inproc --deselect=distributed/tests/test_client.py::test_file_descriptors_dont_leak --deselect=distributed/tests/test_client.py::test_mixing_clients_different_scheduler --deselect=distributed/tests/test_client.py::test_quiet_client_close --deselect=distributed/tests/test_client.py::test_rebalance_sync --deselect=distributed/tests/test_client.py::test_repr_localcluster --deselect=distributed/tests/test_client.py::test_security_loader --deselect=distributed/tests/test_client.py::test_security_loader_ignored_if_explicit_security_provided --deselect=distributed/tests/test_client.py::test_security_loader_ignored_if_returns_none --deselect=distributed/tests/test_client.py::test_shutdown --deselect=distributed/tests/test_client.py::test_shutdown_is_quiet_with_cluster --deselect=distributed/tests/test_client.py::test_shutdown_localcluster --deselect=distributed/tests/test_client.py::test_shutdown_stops_callbacks --deselect=distributed/tests/test_client_loop.py::test_close_loop_sync_start_new_loop --deselect=distributed/tests/test_client_loop.py::test_close_loop_sync_use_running_loop --deselect=distributed/tests/test_core.py::test_close_fast_without_active_handlers --deselect=distributed/tests/test_core.py::test_close_grace_period_for_handlers --deselect=distributed/tests/test_core.py::test_close_properly --deselect=distributed/tests/test_core.py::test_compression --deselect=distributed/tests/test_core.py::test_connection_pool --deselect=distributed/tests/test_core.py::test_connection_pool_close_while_connecting --deselect=distributed/tests/test_core.py::test_connection_pool_detects_remote_close --deselect=distributed/tests/test_core.py::test_connection_pool_outside_cancellation --deselect=distributed/tests/test_core.py::test_connection_pool_remove --deselect=distributed/tests/test_core.py::test_connection_pool_respects_limit --deselect=distributed/tests/test_core.py::test_connection_pool_tls --deselect=distributed/tests/test_core.py::test_counters --deselect=distributed/tests/test_core.py::test_deserialize_error --deselect=distributed/tests/test_core.py::test_errors --deselect=distributed/tests/test_core.py::test_identity_inproc --deselect=distributed/tests/test_core.py::test_identity_tcp --deselect=distributed/tests/test_core.py::test_large_packets_inproc --deselect=distributed/tests/test_core.py::test_messages_are_ordered_bsend --deselect=distributed/tests/test_core.py::test_messages_are_ordered_raw --deselect=distributed/tests/test_core.py::test_ports --deselect=distributed/tests/test_core.py::test_rpc_default --deselect=distributed/tests/test_core.py::test_rpc_inproc --deselect=distributed/tests/test_core.py::test_rpc_message_lifetime_default --deselect=distributed/tests/test_core.py::test_rpc_message_lifetime_inproc --deselect=distributed/tests/test_core.py::test_rpc_message_lifetime_tcp --deselect=distributed/tests/test_core.py::test_rpc_serialization --deselect=distributed/tests/test_core.py::test_rpc_tcp --deselect=distributed/tests/test_core.py::test_rpc_tls --deselect=distributed/tests/test_core.py::test_rpc_with_many_connections_inproc --deselect=distributed/tests/test_core.py::test_rpc_with_many_connections_tcp --deselect=distributed/tests/test_core.py::test_send_recv_args --deselect=distributed/tests/test_core.py::test_send_recv_cancelled --deselect=distributed/tests/test_core.py::test_server --deselect=distributed/tests/test_core.py::test_server_comms_mark_active_handlers --deselect=distributed/tests/test_core.py::test_server_raises_on_blocked_handlers --deselect=distributed/tests/test_jupyter.py::test_jupyter_idle_timeout --deselect=distributed/tests/test_jupyter.py::test_jupyter_server --deselect=distributed/tests/test_locks.py::test_errors --deselect=distributed/tests/test_nanny.py::test_scheduler_file --deselect=distributed/tests/test_nanny.py::test_scheduler_file --deselect=distributed/tests/test_nanny.py::test_scheduler_file --deselect=distributed/tests/test_nanny.py::test_worker_uses_same_host_as_nanny --deselect=distributed/tests/test_preload.py::test_failure_doesnt_crash_scheduler --deselect=distributed/tests/test_preload.py::test_preload_import_time --deselect=distributed/tests/test_preload.py::test_preload_manager_sequence --deselect=distributed/tests/test_preload.py::test_worker_preload_text --deselect=distributed/tests/test_scheduler.py::test_allowed_failures_config --deselect=distributed/tests/test_scheduler.py::test_async_context_manager --deselect=distributed/tests/test_scheduler.py::test_dashboard_host --deselect=distributed/tests/test_scheduler.py::test_file_descriptors_dont_leak --deselect=distributed/tests/test_scheduler.py::test_finished --deselect=distributed/tests/test_scheduler.py::test_multiple_listeners --deselect=distributed/tests/test_scheduler.py::test_no_dangling_asyncio_tasks --deselect=distributed/tests/test_scheduler.py::test_scheduler_file --deselect=distributed/tests/test_scheduler.py::test_scheduler_file --deselect=distributed/tests/test_scheduler.py::test_scheduler_file --deselect=distributed/tests/test_security.py::test_require_encryption --deselect=distributed/tests/test_security.py::test_tls_listen_connect --deselect=distributed/tests/test_security.py::test_tls_temporary_credentials_functional --deselect=distributed/tests/test_semaphore.py::test_threadpoolworkers_pick_correct_ioloop --deselect=distributed/tests/test_tls_functional.py::test_security_dict_input_no_security --deselect=distributed/tests/test_utils_test.py::test_ensure_no_new_clients --deselect=distributed/tests/test_utils_test.py::test_freeze_batched_send --deselect=distributed/tests/test_utils_test.py::test_locked_comm_drop_in_replacement --deselect=distributed/tests/test_utils_test.py::test_locked_comm_intercept_read --deselect=distributed/tests/test_utils_test.py::test_locked_comm_intercept_write --deselect=distributed/tests/test_worker.py::test_host_uses_scheduler_protocol --deselect=distributed/tests/test_worker.py::test_plugin_exception --deselect=distributed/tests/test_worker.py::test_plugin_internal_exception --deselect=distributed/tests/test_worker.py::test_plugin_multiple_exceptions --deselect=distributed/tests/test_worker.py::test_scheduler_file --deselect=distributed/tests/test_worker.py::test_scheduler_file --deselect=distributed/tests/test_worker.py::test_scheduler_file --deselect=distributed/tests/test_worker_client.py::test_dont_override_default_get --deselect=distributed/cli/tests/test_dask_scheduler.py::test_dashboard --deselect=distributed/cli/tests/test_dask_scheduler.py::test_dashboard_allowlist --deselect=distributed/cli/tests/test_dask_scheduler.py::test_dashboard_non_standard_ports --deselect=distributed/cli/tests/test_dask_scheduler.py::test_dashboard_port_zero --deselect=distributed/cli/tests/test_dask_scheduler.py::test_defaults --deselect=distributed/cli/tests/test_dask_scheduler.py::test_hostport --deselect=distributed/cli/tests/test_dask_scheduler.py::test_multiple_protocols --deselect=distributed/cli/tests/test_dask_scheduler.py::test_multiple_workers --deselect=distributed/cli/tests/test_dask_scheduler.py::test_multiple_workers_2 --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_command --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_command_default --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_config --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_file --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_module --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_remote_module --deselect=distributed/cli/tests/test_dask_scheduler.py::test_scheduler_port_zero --deselect=distributed/cli/tests/test_dask_scheduler.py::test_single_executable_deprecated --deselect=distributed/cli/tests/test_dask_worker.py::test_contact_listen_address --deselect=distributed/cli/tests/test_dask_worker.py::test_dashboard_non_standard_ports --deselect=distributed/cli/tests/test_dask_worker.py::test_error_during_startup --deselect=distributed/cli/tests/test_dask_worker.py::test_integer_names --deselect=distributed/cli/tests/test_dask_worker.py::test_listen_address_ipv6 --deselect=distributed/cli/tests/test_dask_worker.py::test_local_directory --deselect=distributed/cli/tests/test_dask_worker.py::test_memory_limit --deselect=distributed/cli/tests/test_dask_worker.py::test_nanny_worker_port_range --deselect=distributed/cli/tests/test_dask_worker.py::test_nanny_worker_port_range_too_many_workers_raises --deselect=distributed/cli/tests/test_dask_worker.py::test_nanny_worker_ports --deselect=distributed/cli/tests/test_dask_worker.py::test_no_nanny --deselect=distributed/cli/tests/test_dask_worker.py::test_nworkers_auto --deselect=distributed/cli/tests/test_dask_worker.py::test_nworkers_expands_name --deselect=distributed/cli/tests/test_dask_worker.py::test_nworkers_negative --deselect=distributed/cli/tests/test_dask_worker.py::test_nworkers_requires_nanny --deselect=distributed/cli/tests/test_dask_worker.py::test_preload_config --deselect=distributed/cli/tests/test_dask_worker.py::test_resources --deselect=distributed/cli/tests/test_dask_worker.py::test_respect_host_listen_address --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_address_env --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_file --deselect=distributed/cli/tests/test_dask_worker.py::test_set_lifetime_restart_via_env_var --deselect=distributed/cli/tests/test_dask_worker.py::test_set_lifetime_stagger_via_env_var --deselect=distributed/cli/tests/test_dask_worker.py::test_signal_handling --deselect=distributed/cli/tests/test_dask_worker.py::test_single_executable_deprecated --deselect=distributed/cli/tests/test_dask_worker.py::test_single_executable_works --deselect=distributed/cli/tests/test_dask_worker.py::test_timeout --deselect=distributed/cli/tests/test_dask_worker.py::test_worker_class --deselect=distributed/tests/test_config.py::test_logging_extended --deselect=distributed/tests/test_config.py::test_logging_file_config --deselect=distributed/tests/test_config.py::test_logging_mutual_exclusive --deselect=distributed/tests/test_config.py::test_logging_simple --deselect=distributed/tests/test_config.py::test_logging_simple_under_distributed --deselect=distributed/tests/test_queues.py::test_queue_in_task --deselect=distributed/tests/test_variable.py::test_variable_in_task +============================= test session starts ============================== +platform linux -- Python 3.12.8, pytest-8.3.4, pluggy-1.5.0 -- /usr/bin/python3.12 +cachedir: .pytest_cache +rootdir: /build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build +configfile: pyproject.toml +plugins: rerunfailures-14.0, anyio-4.7.0, timeout-2.3.1, typeguard-4.4.1 +timeout: 300.0s +timeout method: signal +timeout func_only: False +collecting ... collected 48 items / 1 error / 41 deselected / 5 skipped / 7 selected +run-last-failure: rerun previous 7 failures (skipped 130 files) + +==================================== ERRORS ==================================== +__________ ERROR collecting distributed/shuffle/tests/test_rechunk.py __________ +distributed/shuffle/tests/test_rechunk.py:39: in + from distributed.shuffle.tests.utils import AbstractShuffleTestPool +distributed/shuffle/tests/utils.py:11: in + import dask.dataframe as dd +/usr/lib/python3/dist-packages/dask/dataframe/__init__.py:115: in + if _dask_expr_enabled(): +/usr/lib/python3/dist-packages/dask/dataframe/__init__.py:31: in _dask_expr_enabled + warnings.warn( +E FutureWarning: The legacy Dask DataFrame implementation is deprecated and will be removed in a future version. Set the configuration option `dataframe.query-planning` to `True` or None to enable the new Dask Dataframe implementation and silence this warning. +=========================== short test summary info ============================ +ERROR distributed/shuffle/tests/test_rechunk.py - FutureWarning: The legacy Dask DataFrame implementation is deprecated and will be removed in a future version. Set the configuration option `dataframe.query-planning` to `True` or None to enable the new Dask Dataframe implementation and silence this warning. +!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!! +================== 5 skipped, 41 deselected, 1 error in 5.01s ================== +*** END OF RUN 3: NOT ALL TESTS HAVE YET PASSED/XFAILED *** +*** STARTING RUN 4: python3.12 -m pytest --pyargs distributed --verbose --color=no --timeout-method=signal --timeout=300 -m not avoid_ci -rfE --last-failed --last-failed-no-failures none --ignore=distributed/comm/tests/test_comms.py --ignore=distributed/comm/tests/test_ws.py --ignore=distributed/deploy/tests/test_adaptive.py --ignore=distributed/deploy/tests/test_local.py --ignore=distributed/deploy/tests/test_slow_adaptive.py --ignore=distributed/deploy/tests/test_spec_cluster.py --deselect=distributed/cli/tests/test_dask_scheduler.py::test_no_dashboard --deselect=distributed/deploy/tests/test_local.py::test_localcluster_get_client --deselect=distributed/deploy/tests/test_old_ssh.py::test_cluster --deselect=distributed/deploy/tests/test_old_ssh.py::test_old_ssh_nprocs_renamed_to_n_workers --deselect=distributed/deploy/tests/test_old_ssh.py::test_nprocs_attribute_is_deprecated --deselect=distributed/deploy/tests/test_ssh.py::test_nprocs_attribute_is_deprecated --deselect=distributed/http/tests/test_core.py::test_prometheus_api_doc --deselect=distributed/tests/test_init.py::test_git_revision --deselect=distributed/tests/test_jupyter.py::test_jupyter_idle_timeout --deselect=distributed/tests/test_jupyter.py::test_jupyter_idle_timeout_returned --deselect=distributed/tests/test_jupyter.py::test_jupyter_server --deselect=distributed/tests/test_jupyter.py::test_shutsdown_cleanly --deselect=distributed/tests/test_profile.py::test_stack_overflow --deselect=distributed/tests/test_pubsub.py::test_client_worker --deselect=distributed/tests/test_queues.py::test_queue_in_task --deselect=distributed/tests/test_spill.py::test_spillbuffer_oserror --deselect=distributed/tests/test_steal.py::test_steal_twice --deselect=distributed/tests/test_utils_test.py::test_cluster --deselect=distributed/tests/test_variable.py::test_variable_in_task --deselect=distributed/tests/test_worker.py::test_process_executor_kills_process --deselect=distributed/tests/test_worker_memory.py::test_fail_to_pickle_execute_1 --deselect=distributed/tests/test_worker_state_machine.py::test_task_state_instance_are_garbage_collected --deselect=distributed/protocol/tests/test_protocol.py::test_deeply_nested_structures --deselect=distributed/protocol/tests/test_serialize.py::test_deeply_nested_structures --deselect=distributed/cli/tests/test_dask_scheduler.py::test_defaults --deselect=distributed/cli/tests/test_dask_scheduler.py::test_hostport --deselect=distributed/cli/tests/test_dask_spec.py::test_errors --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_file --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_file --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_file --deselect=distributed/dashboard/tests/test_scheduler_bokeh.py::test_counters --deselect=distributed/dashboard/tests/test_worker_bokeh.py::test_counters --deselect=distributed/deploy/tests/test_local.py::test_adapt_then_manual --deselect=distributed/deploy/tests/test_local.py::test_async_with --deselect=distributed/deploy/tests/test_local.py::test_close_twice --deselect=distributed/deploy/tests/test_local.py::test_cluster_info_sync --deselect=distributed/deploy/tests/test_local.py::test_local_tls --deselect=distributed/deploy/tests/test_local.py::test_no_dangling_asyncio_tasks --deselect=distributed/deploy/tests/test_local.py::test_only_local_access --deselect=distributed/deploy/tests/test_local.py::test_remote_access --deselect=distributed/diagnostics/tests/test_progress_widgets.py::test_serializers --deselect=distributed/diagnostics/tests/test_scheduler_plugin.py::test_lifecycle --deselect=distributed/http/scheduler/tests/test_missing_bokeh.py::test_missing_bokeh --deselect=distributed/http/scheduler/tests/test_scheduler_http.py::test_metrics_when_prometheus_client_not_installed --deselect=distributed/protocol/tests/test_serialize.py::test_errors --deselect=distributed/tests/test_batched.py::test_BatchedSend --deselect=distributed/tests/test_batched.py::test_close_closed --deselect=distributed/tests/test_batched.py::test_close_twice --deselect=distributed/tests/test_batched.py::test_send_after_stream_start --deselect=distributed/tests/test_batched.py::test_send_before_close --deselect=distributed/tests/test_batched.py::test_send_before_start --deselect=distributed/tests/test_batched.py::test_sending_traffic_jam --deselect=distributed/tests/test_batched.py::test_serializers --deselect=distributed/tests/test_client.py::TestClientSecurityLoader::test_security_loader --deselect=distributed/tests/test_client.py::TestClientSecurityLoader::test_security_loader_ignored_if_explicit_security_provided --deselect=distributed/tests/test_client.py::TestClientSecurityLoader::test_security_loader_ignored_if_returns_none --deselect=distributed/tests/test_client.py::test_async_with --deselect=distributed/tests/test_client.py::test_client_is_quiet_cluster_close --deselect=distributed/tests/test_client.py::test_dashboard_link_cluster --deselect=distributed/tests/test_client.py::test_dashboard_link_inproc --deselect=distributed/tests/test_client.py::test_file_descriptors_dont_leak --deselect=distributed/tests/test_client.py::test_mixing_clients_different_scheduler --deselect=distributed/tests/test_client.py::test_quiet_client_close --deselect=distributed/tests/test_client.py::test_rebalance_sync --deselect=distributed/tests/test_client.py::test_repr_localcluster --deselect=distributed/tests/test_client.py::test_security_loader --deselect=distributed/tests/test_client.py::test_security_loader_ignored_if_explicit_security_provided --deselect=distributed/tests/test_client.py::test_security_loader_ignored_if_returns_none --deselect=distributed/tests/test_client.py::test_shutdown --deselect=distributed/tests/test_client.py::test_shutdown_is_quiet_with_cluster --deselect=distributed/tests/test_client.py::test_shutdown_localcluster --deselect=distributed/tests/test_client.py::test_shutdown_stops_callbacks --deselect=distributed/tests/test_client_loop.py::test_close_loop_sync_start_new_loop --deselect=distributed/tests/test_client_loop.py::test_close_loop_sync_use_running_loop --deselect=distributed/tests/test_core.py::test_close_fast_without_active_handlers --deselect=distributed/tests/test_core.py::test_close_grace_period_for_handlers --deselect=distributed/tests/test_core.py::test_close_properly --deselect=distributed/tests/test_core.py::test_compression --deselect=distributed/tests/test_core.py::test_connection_pool --deselect=distributed/tests/test_core.py::test_connection_pool_close_while_connecting --deselect=distributed/tests/test_core.py::test_connection_pool_detects_remote_close --deselect=distributed/tests/test_core.py::test_connection_pool_outside_cancellation --deselect=distributed/tests/test_core.py::test_connection_pool_remove --deselect=distributed/tests/test_core.py::test_connection_pool_respects_limit --deselect=distributed/tests/test_core.py::test_connection_pool_tls --deselect=distributed/tests/test_core.py::test_counters --deselect=distributed/tests/test_core.py::test_deserialize_error --deselect=distributed/tests/test_core.py::test_errors --deselect=distributed/tests/test_core.py::test_identity_inproc --deselect=distributed/tests/test_core.py::test_identity_tcp --deselect=distributed/tests/test_core.py::test_large_packets_inproc --deselect=distributed/tests/test_core.py::test_messages_are_ordered_bsend --deselect=distributed/tests/test_core.py::test_messages_are_ordered_raw --deselect=distributed/tests/test_core.py::test_ports --deselect=distributed/tests/test_core.py::test_rpc_default --deselect=distributed/tests/test_core.py::test_rpc_inproc --deselect=distributed/tests/test_core.py::test_rpc_message_lifetime_default --deselect=distributed/tests/test_core.py::test_rpc_message_lifetime_inproc --deselect=distributed/tests/test_core.py::test_rpc_message_lifetime_tcp --deselect=distributed/tests/test_core.py::test_rpc_serialization --deselect=distributed/tests/test_core.py::test_rpc_tcp --deselect=distributed/tests/test_core.py::test_rpc_tls --deselect=distributed/tests/test_core.py::test_rpc_with_many_connections_inproc --deselect=distributed/tests/test_core.py::test_rpc_with_many_connections_tcp --deselect=distributed/tests/test_core.py::test_send_recv_args --deselect=distributed/tests/test_core.py::test_send_recv_cancelled --deselect=distributed/tests/test_core.py::test_server --deselect=distributed/tests/test_core.py::test_server_comms_mark_active_handlers --deselect=distributed/tests/test_core.py::test_server_raises_on_blocked_handlers --deselect=distributed/tests/test_jupyter.py::test_jupyter_idle_timeout --deselect=distributed/tests/test_jupyter.py::test_jupyter_server --deselect=distributed/tests/test_locks.py::test_errors --deselect=distributed/tests/test_nanny.py::test_scheduler_file --deselect=distributed/tests/test_nanny.py::test_scheduler_file --deselect=distributed/tests/test_nanny.py::test_scheduler_file --deselect=distributed/tests/test_nanny.py::test_worker_uses_same_host_as_nanny --deselect=distributed/tests/test_preload.py::test_failure_doesnt_crash_scheduler --deselect=distributed/tests/test_preload.py::test_preload_import_time --deselect=distributed/tests/test_preload.py::test_preload_manager_sequence --deselect=distributed/tests/test_preload.py::test_worker_preload_text --deselect=distributed/tests/test_scheduler.py::test_allowed_failures_config --deselect=distributed/tests/test_scheduler.py::test_async_context_manager --deselect=distributed/tests/test_scheduler.py::test_dashboard_host --deselect=distributed/tests/test_scheduler.py::test_file_descriptors_dont_leak --deselect=distributed/tests/test_scheduler.py::test_finished --deselect=distributed/tests/test_scheduler.py::test_multiple_listeners --deselect=distributed/tests/test_scheduler.py::test_no_dangling_asyncio_tasks --deselect=distributed/tests/test_scheduler.py::test_scheduler_file --deselect=distributed/tests/test_scheduler.py::test_scheduler_file --deselect=distributed/tests/test_scheduler.py::test_scheduler_file --deselect=distributed/tests/test_security.py::test_require_encryption --deselect=distributed/tests/test_security.py::test_tls_listen_connect --deselect=distributed/tests/test_security.py::test_tls_temporary_credentials_functional --deselect=distributed/tests/test_semaphore.py::test_threadpoolworkers_pick_correct_ioloop --deselect=distributed/tests/test_tls_functional.py::test_security_dict_input_no_security --deselect=distributed/tests/test_utils_test.py::test_ensure_no_new_clients --deselect=distributed/tests/test_utils_test.py::test_freeze_batched_send --deselect=distributed/tests/test_utils_test.py::test_locked_comm_drop_in_replacement --deselect=distributed/tests/test_utils_test.py::test_locked_comm_intercept_read --deselect=distributed/tests/test_utils_test.py::test_locked_comm_intercept_write --deselect=distributed/tests/test_worker.py::test_host_uses_scheduler_protocol --deselect=distributed/tests/test_worker.py::test_plugin_exception --deselect=distributed/tests/test_worker.py::test_plugin_internal_exception --deselect=distributed/tests/test_worker.py::test_plugin_multiple_exceptions --deselect=distributed/tests/test_worker.py::test_scheduler_file --deselect=distributed/tests/test_worker.py::test_scheduler_file --deselect=distributed/tests/test_worker.py::test_scheduler_file --deselect=distributed/tests/test_worker_client.py::test_dont_override_default_get --deselect=distributed/cli/tests/test_dask_scheduler.py::test_dashboard --deselect=distributed/cli/tests/test_dask_scheduler.py::test_dashboard_allowlist --deselect=distributed/cli/tests/test_dask_scheduler.py::test_dashboard_non_standard_ports --deselect=distributed/cli/tests/test_dask_scheduler.py::test_dashboard_port_zero --deselect=distributed/cli/tests/test_dask_scheduler.py::test_defaults --deselect=distributed/cli/tests/test_dask_scheduler.py::test_hostport --deselect=distributed/cli/tests/test_dask_scheduler.py::test_multiple_protocols --deselect=distributed/cli/tests/test_dask_scheduler.py::test_multiple_workers --deselect=distributed/cli/tests/test_dask_scheduler.py::test_multiple_workers_2 --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_command --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_command_default --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_config --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_file --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_module --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_remote_module --deselect=distributed/cli/tests/test_dask_scheduler.py::test_scheduler_port_zero --deselect=distributed/cli/tests/test_dask_scheduler.py::test_single_executable_deprecated --deselect=distributed/cli/tests/test_dask_worker.py::test_contact_listen_address --deselect=distributed/cli/tests/test_dask_worker.py::test_dashboard_non_standard_ports --deselect=distributed/cli/tests/test_dask_worker.py::test_error_during_startup --deselect=distributed/cli/tests/test_dask_worker.py::test_integer_names --deselect=distributed/cli/tests/test_dask_worker.py::test_listen_address_ipv6 --deselect=distributed/cli/tests/test_dask_worker.py::test_local_directory --deselect=distributed/cli/tests/test_dask_worker.py::test_memory_limit --deselect=distributed/cli/tests/test_dask_worker.py::test_nanny_worker_port_range --deselect=distributed/cli/tests/test_dask_worker.py::test_nanny_worker_port_range_too_many_workers_raises --deselect=distributed/cli/tests/test_dask_worker.py::test_nanny_worker_ports --deselect=distributed/cli/tests/test_dask_worker.py::test_no_nanny --deselect=distributed/cli/tests/test_dask_worker.py::test_nworkers_auto --deselect=distributed/cli/tests/test_dask_worker.py::test_nworkers_expands_name --deselect=distributed/cli/tests/test_dask_worker.py::test_nworkers_negative --deselect=distributed/cli/tests/test_dask_worker.py::test_nworkers_requires_nanny --deselect=distributed/cli/tests/test_dask_worker.py::test_preload_config --deselect=distributed/cli/tests/test_dask_worker.py::test_resources --deselect=distributed/cli/tests/test_dask_worker.py::test_respect_host_listen_address --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_address_env --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_file --deselect=distributed/cli/tests/test_dask_worker.py::test_set_lifetime_restart_via_env_var --deselect=distributed/cli/tests/test_dask_worker.py::test_set_lifetime_stagger_via_env_var --deselect=distributed/cli/tests/test_dask_worker.py::test_signal_handling --deselect=distributed/cli/tests/test_dask_worker.py::test_single_executable_deprecated --deselect=distributed/cli/tests/test_dask_worker.py::test_single_executable_works --deselect=distributed/cli/tests/test_dask_worker.py::test_timeout --deselect=distributed/cli/tests/test_dask_worker.py::test_worker_class --deselect=distributed/tests/test_config.py::test_logging_extended --deselect=distributed/tests/test_config.py::test_logging_file_config --deselect=distributed/tests/test_config.py::test_logging_mutual_exclusive --deselect=distributed/tests/test_config.py::test_logging_simple --deselect=distributed/tests/test_config.py::test_logging_simple_under_distributed --deselect=distributed/tests/test_queues.py::test_queue_in_task --deselect=distributed/tests/test_variable.py::test_variable_in_task +============================= test session starts ============================== +platform linux -- Python 3.12.8, pytest-8.3.4, pluggy-1.5.0 -- /usr/bin/python3.12 +cachedir: .pytest_cache +rootdir: /build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build +configfile: pyproject.toml +plugins: rerunfailures-14.0, anyio-4.7.0, timeout-2.3.1, typeguard-4.4.1 +timeout: 300.0s +timeout method: signal +timeout func_only: False +collecting ... collected 48 items / 1 error / 41 deselected / 5 skipped / 7 selected +run-last-failure: rerun previous 7 failures (skipped 130 files) + +==================================== ERRORS ==================================== +__________ ERROR collecting distributed/shuffle/tests/test_rechunk.py __________ +distributed/shuffle/tests/test_rechunk.py:39: in + from distributed.shuffle.tests.utils import AbstractShuffleTestPool +distributed/shuffle/tests/utils.py:11: in + import dask.dataframe as dd +/usr/lib/python3/dist-packages/dask/dataframe/__init__.py:115: in + if _dask_expr_enabled(): +/usr/lib/python3/dist-packages/dask/dataframe/__init__.py:31: in _dask_expr_enabled + warnings.warn( +E FutureWarning: The legacy Dask DataFrame implementation is deprecated and will be removed in a future version. Set the configuration option `dataframe.query-planning` to `True` or None to enable the new Dask Dataframe implementation and silence this warning. +=========================== short test summary info ============================ +ERROR distributed/shuffle/tests/test_rechunk.py - FutureWarning: The legacy Dask DataFrame implementation is deprecated and will be removed in a future version. Set the configuration option `dataframe.query-planning` to `True` or None to enable the new Dask Dataframe implementation and silence this warning. +!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!! +================== 5 skipped, 41 deselected, 1 error in 5.62s ================== +*** END OF RUN 4: NOT ALL TESTS HAVE YET PASSED/XFAILED *** +*** STARTING RUN 5: python3.12 -m pytest --pyargs distributed --verbose --color=no --timeout-method=signal --timeout=300 -m not avoid_ci -rfE --last-failed --last-failed-no-failures none --ignore=distributed/comm/tests/test_comms.py --ignore=distributed/comm/tests/test_ws.py --ignore=distributed/deploy/tests/test_adaptive.py --ignore=distributed/deploy/tests/test_local.py --ignore=distributed/deploy/tests/test_slow_adaptive.py --ignore=distributed/deploy/tests/test_spec_cluster.py --deselect=distributed/cli/tests/test_dask_scheduler.py::test_no_dashboard --deselect=distributed/deploy/tests/test_local.py::test_localcluster_get_client --deselect=distributed/deploy/tests/test_old_ssh.py::test_cluster --deselect=distributed/deploy/tests/test_old_ssh.py::test_old_ssh_nprocs_renamed_to_n_workers --deselect=distributed/deploy/tests/test_old_ssh.py::test_nprocs_attribute_is_deprecated --deselect=distributed/deploy/tests/test_ssh.py::test_nprocs_attribute_is_deprecated --deselect=distributed/http/tests/test_core.py::test_prometheus_api_doc --deselect=distributed/tests/test_init.py::test_git_revision --deselect=distributed/tests/test_jupyter.py::test_jupyter_idle_timeout --deselect=distributed/tests/test_jupyter.py::test_jupyter_idle_timeout_returned --deselect=distributed/tests/test_jupyter.py::test_jupyter_server --deselect=distributed/tests/test_jupyter.py::test_shutsdown_cleanly --deselect=distributed/tests/test_profile.py::test_stack_overflow --deselect=distributed/tests/test_pubsub.py::test_client_worker --deselect=distributed/tests/test_queues.py::test_queue_in_task --deselect=distributed/tests/test_spill.py::test_spillbuffer_oserror --deselect=distributed/tests/test_steal.py::test_steal_twice --deselect=distributed/tests/test_utils_test.py::test_cluster --deselect=distributed/tests/test_variable.py::test_variable_in_task --deselect=distributed/tests/test_worker.py::test_process_executor_kills_process --deselect=distributed/tests/test_worker_memory.py::test_fail_to_pickle_execute_1 --deselect=distributed/tests/test_worker_state_machine.py::test_task_state_instance_are_garbage_collected --deselect=distributed/protocol/tests/test_protocol.py::test_deeply_nested_structures --deselect=distributed/protocol/tests/test_serialize.py::test_deeply_nested_structures --deselect=distributed/cli/tests/test_dask_scheduler.py::test_defaults --deselect=distributed/cli/tests/test_dask_scheduler.py::test_hostport --deselect=distributed/cli/tests/test_dask_spec.py::test_errors --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_file --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_file --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_file --deselect=distributed/dashboard/tests/test_scheduler_bokeh.py::test_counters --deselect=distributed/dashboard/tests/test_worker_bokeh.py::test_counters --deselect=distributed/deploy/tests/test_local.py::test_adapt_then_manual --deselect=distributed/deploy/tests/test_local.py::test_async_with --deselect=distributed/deploy/tests/test_local.py::test_close_twice --deselect=distributed/deploy/tests/test_local.py::test_cluster_info_sync --deselect=distributed/deploy/tests/test_local.py::test_local_tls --deselect=distributed/deploy/tests/test_local.py::test_no_dangling_asyncio_tasks --deselect=distributed/deploy/tests/test_local.py::test_only_local_access --deselect=distributed/deploy/tests/test_local.py::test_remote_access --deselect=distributed/diagnostics/tests/test_progress_widgets.py::test_serializers --deselect=distributed/diagnostics/tests/test_scheduler_plugin.py::test_lifecycle --deselect=distributed/http/scheduler/tests/test_missing_bokeh.py::test_missing_bokeh --deselect=distributed/http/scheduler/tests/test_scheduler_http.py::test_metrics_when_prometheus_client_not_installed --deselect=distributed/protocol/tests/test_serialize.py::test_errors --deselect=distributed/tests/test_batched.py::test_BatchedSend --deselect=distributed/tests/test_batched.py::test_close_closed --deselect=distributed/tests/test_batched.py::test_close_twice --deselect=distributed/tests/test_batched.py::test_send_after_stream_start --deselect=distributed/tests/test_batched.py::test_send_before_close --deselect=distributed/tests/test_batched.py::test_send_before_start --deselect=distributed/tests/test_batched.py::test_sending_traffic_jam --deselect=distributed/tests/test_batched.py::test_serializers --deselect=distributed/tests/test_client.py::TestClientSecurityLoader::test_security_loader --deselect=distributed/tests/test_client.py::TestClientSecurityLoader::test_security_loader_ignored_if_explicit_security_provided --deselect=distributed/tests/test_client.py::TestClientSecurityLoader::test_security_loader_ignored_if_returns_none --deselect=distributed/tests/test_client.py::test_async_with --deselect=distributed/tests/test_client.py::test_client_is_quiet_cluster_close --deselect=distributed/tests/test_client.py::test_dashboard_link_cluster --deselect=distributed/tests/test_client.py::test_dashboard_link_inproc --deselect=distributed/tests/test_client.py::test_file_descriptors_dont_leak --deselect=distributed/tests/test_client.py::test_mixing_clients_different_scheduler --deselect=distributed/tests/test_client.py::test_quiet_client_close --deselect=distributed/tests/test_client.py::test_rebalance_sync --deselect=distributed/tests/test_client.py::test_repr_localcluster --deselect=distributed/tests/test_client.py::test_security_loader --deselect=distributed/tests/test_client.py::test_security_loader_ignored_if_explicit_security_provided --deselect=distributed/tests/test_client.py::test_security_loader_ignored_if_returns_none --deselect=distributed/tests/test_client.py::test_shutdown --deselect=distributed/tests/test_client.py::test_shutdown_is_quiet_with_cluster --deselect=distributed/tests/test_client.py::test_shutdown_localcluster --deselect=distributed/tests/test_client.py::test_shutdown_stops_callbacks --deselect=distributed/tests/test_client_loop.py::test_close_loop_sync_start_new_loop --deselect=distributed/tests/test_client_loop.py::test_close_loop_sync_use_running_loop --deselect=distributed/tests/test_core.py::test_close_fast_without_active_handlers --deselect=distributed/tests/test_core.py::test_close_grace_period_for_handlers --deselect=distributed/tests/test_core.py::test_close_properly --deselect=distributed/tests/test_core.py::test_compression --deselect=distributed/tests/test_core.py::test_connection_pool --deselect=distributed/tests/test_core.py::test_connection_pool_close_while_connecting --deselect=distributed/tests/test_core.py::test_connection_pool_detects_remote_close --deselect=distributed/tests/test_core.py::test_connection_pool_outside_cancellation --deselect=distributed/tests/test_core.py::test_connection_pool_remove --deselect=distributed/tests/test_core.py::test_connection_pool_respects_limit --deselect=distributed/tests/test_core.py::test_connection_pool_tls --deselect=distributed/tests/test_core.py::test_counters --deselect=distributed/tests/test_core.py::test_deserialize_error --deselect=distributed/tests/test_core.py::test_errors --deselect=distributed/tests/test_core.py::test_identity_inproc --deselect=distributed/tests/test_core.py::test_identity_tcp --deselect=distributed/tests/test_core.py::test_large_packets_inproc --deselect=distributed/tests/test_core.py::test_messages_are_ordered_bsend --deselect=distributed/tests/test_core.py::test_messages_are_ordered_raw --deselect=distributed/tests/test_core.py::test_ports --deselect=distributed/tests/test_core.py::test_rpc_default --deselect=distributed/tests/test_core.py::test_rpc_inproc --deselect=distributed/tests/test_core.py::test_rpc_message_lifetime_default --deselect=distributed/tests/test_core.py::test_rpc_message_lifetime_inproc --deselect=distributed/tests/test_core.py::test_rpc_message_lifetime_tcp --deselect=distributed/tests/test_core.py::test_rpc_serialization --deselect=distributed/tests/test_core.py::test_rpc_tcp --deselect=distributed/tests/test_core.py::test_rpc_tls --deselect=distributed/tests/test_core.py::test_rpc_with_many_connections_inproc --deselect=distributed/tests/test_core.py::test_rpc_with_many_connections_tcp --deselect=distributed/tests/test_core.py::test_send_recv_args --deselect=distributed/tests/test_core.py::test_send_recv_cancelled --deselect=distributed/tests/test_core.py::test_server --deselect=distributed/tests/test_core.py::test_server_comms_mark_active_handlers --deselect=distributed/tests/test_core.py::test_server_raises_on_blocked_handlers --deselect=distributed/tests/test_jupyter.py::test_jupyter_idle_timeout --deselect=distributed/tests/test_jupyter.py::test_jupyter_server --deselect=distributed/tests/test_locks.py::test_errors --deselect=distributed/tests/test_nanny.py::test_scheduler_file --deselect=distributed/tests/test_nanny.py::test_scheduler_file --deselect=distributed/tests/test_nanny.py::test_scheduler_file --deselect=distributed/tests/test_nanny.py::test_worker_uses_same_host_as_nanny --deselect=distributed/tests/test_preload.py::test_failure_doesnt_crash_scheduler --deselect=distributed/tests/test_preload.py::test_preload_import_time --deselect=distributed/tests/test_preload.py::test_preload_manager_sequence --deselect=distributed/tests/test_preload.py::test_worker_preload_text --deselect=distributed/tests/test_scheduler.py::test_allowed_failures_config --deselect=distributed/tests/test_scheduler.py::test_async_context_manager --deselect=distributed/tests/test_scheduler.py::test_dashboard_host --deselect=distributed/tests/test_scheduler.py::test_file_descriptors_dont_leak --deselect=distributed/tests/test_scheduler.py::test_finished --deselect=distributed/tests/test_scheduler.py::test_multiple_listeners --deselect=distributed/tests/test_scheduler.py::test_no_dangling_asyncio_tasks --deselect=distributed/tests/test_scheduler.py::test_scheduler_file --deselect=distributed/tests/test_scheduler.py::test_scheduler_file --deselect=distributed/tests/test_scheduler.py::test_scheduler_file --deselect=distributed/tests/test_security.py::test_require_encryption --deselect=distributed/tests/test_security.py::test_tls_listen_connect --deselect=distributed/tests/test_security.py::test_tls_temporary_credentials_functional --deselect=distributed/tests/test_semaphore.py::test_threadpoolworkers_pick_correct_ioloop --deselect=distributed/tests/test_tls_functional.py::test_security_dict_input_no_security --deselect=distributed/tests/test_utils_test.py::test_ensure_no_new_clients --deselect=distributed/tests/test_utils_test.py::test_freeze_batched_send --deselect=distributed/tests/test_utils_test.py::test_locked_comm_drop_in_replacement --deselect=distributed/tests/test_utils_test.py::test_locked_comm_intercept_read --deselect=distributed/tests/test_utils_test.py::test_locked_comm_intercept_write --deselect=distributed/tests/test_worker.py::test_host_uses_scheduler_protocol --deselect=distributed/tests/test_worker.py::test_plugin_exception --deselect=distributed/tests/test_worker.py::test_plugin_internal_exception --deselect=distributed/tests/test_worker.py::test_plugin_multiple_exceptions --deselect=distributed/tests/test_worker.py::test_scheduler_file --deselect=distributed/tests/test_worker.py::test_scheduler_file --deselect=distributed/tests/test_worker.py::test_scheduler_file --deselect=distributed/tests/test_worker_client.py::test_dont_override_default_get --deselect=distributed/cli/tests/test_dask_scheduler.py::test_dashboard --deselect=distributed/cli/tests/test_dask_scheduler.py::test_dashboard_allowlist --deselect=distributed/cli/tests/test_dask_scheduler.py::test_dashboard_non_standard_ports --deselect=distributed/cli/tests/test_dask_scheduler.py::test_dashboard_port_zero --deselect=distributed/cli/tests/test_dask_scheduler.py::test_defaults --deselect=distributed/cli/tests/test_dask_scheduler.py::test_hostport --deselect=distributed/cli/tests/test_dask_scheduler.py::test_multiple_protocols --deselect=distributed/cli/tests/test_dask_scheduler.py::test_multiple_workers --deselect=distributed/cli/tests/test_dask_scheduler.py::test_multiple_workers_2 --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_command --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_command_default --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_config --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_file --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_module --deselect=distributed/cli/tests/test_dask_scheduler.py::test_preload_remote_module --deselect=distributed/cli/tests/test_dask_scheduler.py::test_scheduler_port_zero --deselect=distributed/cli/tests/test_dask_scheduler.py::test_single_executable_deprecated --deselect=distributed/cli/tests/test_dask_worker.py::test_contact_listen_address --deselect=distributed/cli/tests/test_dask_worker.py::test_dashboard_non_standard_ports --deselect=distributed/cli/tests/test_dask_worker.py::test_error_during_startup --deselect=distributed/cli/tests/test_dask_worker.py::test_integer_names --deselect=distributed/cli/tests/test_dask_worker.py::test_listen_address_ipv6 --deselect=distributed/cli/tests/test_dask_worker.py::test_local_directory --deselect=distributed/cli/tests/test_dask_worker.py::test_memory_limit --deselect=distributed/cli/tests/test_dask_worker.py::test_nanny_worker_port_range --deselect=distributed/cli/tests/test_dask_worker.py::test_nanny_worker_port_range_too_many_workers_raises --deselect=distributed/cli/tests/test_dask_worker.py::test_nanny_worker_ports --deselect=distributed/cli/tests/test_dask_worker.py::test_no_nanny --deselect=distributed/cli/tests/test_dask_worker.py::test_nworkers_auto --deselect=distributed/cli/tests/test_dask_worker.py::test_nworkers_expands_name --deselect=distributed/cli/tests/test_dask_worker.py::test_nworkers_negative --deselect=distributed/cli/tests/test_dask_worker.py::test_nworkers_requires_nanny --deselect=distributed/cli/tests/test_dask_worker.py::test_preload_config --deselect=distributed/cli/tests/test_dask_worker.py::test_resources --deselect=distributed/cli/tests/test_dask_worker.py::test_respect_host_listen_address --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_address_env --deselect=distributed/cli/tests/test_dask_worker.py::test_scheduler_file --deselect=distributed/cli/tests/test_dask_worker.py::test_set_lifetime_restart_via_env_var --deselect=distributed/cli/tests/test_dask_worker.py::test_set_lifetime_stagger_via_env_var --deselect=distributed/cli/tests/test_dask_worker.py::test_signal_handling --deselect=distributed/cli/tests/test_dask_worker.py::test_single_executable_deprecated --deselect=distributed/cli/tests/test_dask_worker.py::test_single_executable_works --deselect=distributed/cli/tests/test_dask_worker.py::test_timeout --deselect=distributed/cli/tests/test_dask_worker.py::test_worker_class --deselect=distributed/tests/test_config.py::test_logging_extended --deselect=distributed/tests/test_config.py::test_logging_file_config --deselect=distributed/tests/test_config.py::test_logging_mutual_exclusive --deselect=distributed/tests/test_config.py::test_logging_simple --deselect=distributed/tests/test_config.py::test_logging_simple_under_distributed --deselect=distributed/tests/test_queues.py::test_queue_in_task --deselect=distributed/tests/test_variable.py::test_variable_in_task +============================= test session starts ============================== +platform linux -- Python 3.12.8, pytest-8.3.4, pluggy-1.5.0 -- /usr/bin/python3.12 +cachedir: .pytest_cache +rootdir: /build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build +configfile: pyproject.toml +plugins: rerunfailures-14.0, anyio-4.7.0, timeout-2.3.1, typeguard-4.4.1 +timeout: 300.0s +timeout method: signal +timeout func_only: False +collecting ... collected 48 items / 1 error / 41 deselected / 5 skipped / 7 selected +run-last-failure: rerun previous 7 failures (skipped 130 files) + +==================================== ERRORS ==================================== +__________ ERROR collecting distributed/shuffle/tests/test_rechunk.py __________ +distributed/shuffle/tests/test_rechunk.py:39: in + from distributed.shuffle.tests.utils import AbstractShuffleTestPool +distributed/shuffle/tests/utils.py:11: in + import dask.dataframe as dd +/usr/lib/python3/dist-packages/dask/dataframe/__init__.py:115: in + if _dask_expr_enabled(): +/usr/lib/python3/dist-packages/dask/dataframe/__init__.py:31: in _dask_expr_enabled + warnings.warn( +E FutureWarning: The legacy Dask DataFrame implementation is deprecated and will be removed in a future version. Set the configuration option `dataframe.query-planning` to `True` or None to enable the new Dask Dataframe implementation and silence this warning. +=========================== short test summary info ============================ +ERROR distributed/shuffle/tests/test_rechunk.py - FutureWarning: The legacy Dask DataFrame implementation is deprecated and will be removed in a future version. Set the configuration option `dataframe.query-planning` to `True` or None to enable the new Dask Dataframe implementation and silence this warning. +!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!! +================== 5 skipped, 41 deselected, 1 error in 4.78s ================== +*** END OF RUN 5: NOT ALL TESTS HAVE YET PASSED/XFAILED *** +*** SOME TESTS FAILED/ERRORED EVERY RUN, ABORTING *** +E: pybuild pybuild:389: test: plugin pyproject failed with: exit code=1: cd /build/reproducible-path/dask.distributed-2024.12.1+ds/.pybuild/cpython3_3.12_distributed/build; cd {build_dir}; /build/reproducible-path/dask.distributed-2024.12.1+ds/debian/run-tests /build/reproducible-path/dask.distributed-2024.12.1+ds/debian/get-test-exclusions rules {interpreter} -m pytest --pyargs distributed --verbose --color=no --timeout-method=signal --timeout=300 -m "not avoid_ci" +dh_auto_test: error: pybuild --test -i python{version} -p "3.13 3.12" returned exit code 13 +make: *** [debian/rules:35: binary] Error 25 +dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2 I: copying local configuration +E: Failed autobuilding of package +I: user script /srv/workspace/pbuilder/960870/tmp/hooks/C01_cleanup starting +debug output: disk usage on i-capture-the-hostname at Tue Jan 14 03:35:12 UTC 2025 +Filesystem Size Used Avail Use% Mounted on +tmpfs 40G 432K 40G 1% /dev/shm + +I: user script /srv/workspace/pbuilder/960870/tmp/hooks/C01_cleanup finished I: unmounting dev/ptmx filesystem I: unmounting dev/pts filesystem I: unmounting dev/shm filesystem I: unmounting proc filesystem I: unmounting sys filesystem I: cleaning the build env -I: removing directory /srv/workspace/pbuilder/1505584 and its subdirectories -I: Current time: Sun Feb 15 19:35:56 -12 2026 -I: pbuilder-time-stamp: 1771227356 +I: removing directory /srv/workspace/pbuilder/960870 and its subdirectories