So this takes ~90 microseconds per task. About 100ms of this is from overhead:
\nIn [5]: x = da.ones(1000, chunks=(1000,)).sum()\n \n In [6]: %timeit x.compute()\n-6.42 ms +- 2.85 ms per loop (mean +- std. dev. of 7 runs, 100 loops each)\n+2.76 ms +- 27.4 us per loop (mean +- std. dev. of 7 runs, 100 loops each)\n There is some overhead from spinning up a ThreadPoolExecutor each time.\n This may be mediated by using a global or contextual pool:
\n>>> from concurrent.futures import ThreadPoolExecutor\n >>> pool = ThreadPoolExecutor()\n >>> dask.config.set(pool=pool) # set global ThreadPoolExecutor\n", "details": [{"source1": "html2text {}", "source2": "html2text {}", "unified_diff": "@@ -170,20 +170,20 @@\n \n In [2]: x = da.ones(1000, chunks=(2,)).sum()\n \n In [3]: len(x.dask)\n Out[3]: 1168\n \n In [4]: %timeit x.compute()\n-602 ms +- 178 ms per loop (mean +- std. dev. of 7 runs, 1 loop each)\n+296 ms +- 4.78 ms per loop (mean +- std. dev. of 7 runs, 1 loop each)\n So this takes ~90 microseconds per task. About 100ms of this is from overhead:\n In [5]: x = da.ones(1000, chunks=(1000,)).sum()\n \n In [6]: %timeit x.compute()\n-6.42 ms +- 2.85 ms per loop (mean +- std. dev. of 7 runs, 100 loops each)\n+2.76 ms +- 27.4 us per loop (mean +- std. dev. of 7 runs, 100 loops each)\n There is some overhead from spinning up a ThreadPoolExecutor each time. This\n may be mediated by using a global or contextual pool:\n >>> from concurrent.futures import ThreadPoolExecutor\n >>> pool = ThreadPoolExecutor()\n >>> dask.config.set(pool=pool) # set global ThreadPoolExecutor\n \n or\n"}]}]}]}]}]}