So this takes ~90 microseconds per task. About 100ms of this is from overhead:
\nIn [5]: x = da.ones(1000, chunks=(1000,)).sum()\n \n In [6]: %timeit x.compute()\n-2.5 ms +- 713 us per loop (mean +- std. dev. of 7 runs, 100 loops each)\n+1.01 ms +- 37.9 us per loop (mean +- std. dev. of 7 runs, 1,000 loops each)\n There is some overhead from spinning up a ThreadPoolExecutor each time.\n This may be mediated by using a global or contextual pool:
\n>>> from concurrent.futures import ThreadPoolExecutor\n >>> pool = ThreadPoolExecutor()\n >>> dask.config.set(pool=pool) # set global ThreadPoolExecutor\n", "details": [{"source1": "html2text {}", "source2": "html2text {}", "unified_diff": "@@ -170,20 +170,20 @@\n \n In [2]: x = da.ones(1000, chunks=(2,)).sum()\n \n In [3]: len(x.dask)\n Out[3]: 1168\n \n In [4]: %timeit x.compute()\n-261 ms +- 19.6 ms per loop (mean +- std. dev. of 7 runs, 1 loop each)\n+74.3 ms +- 988 us per loop (mean +- std. dev. of 7 runs, 10 loops each)\n So this takes ~90 microseconds per task. About 100ms of this is from overhead:\n In [5]: x = da.ones(1000, chunks=(1000,)).sum()\n \n In [6]: %timeit x.compute()\n-2.5 ms +- 713 us per loop (mean +- std. dev. of 7 runs, 100 loops each)\n+1.01 ms +- 37.9 us per loop (mean +- std. dev. of 7 runs, 1,000 loops each)\n There is some overhead from spinning up a ThreadPoolExecutor each time. This\n may be mediated by using a global or contextual pool:\n >>> from concurrent.futures import ThreadPoolExecutor\n >>> pool = ThreadPoolExecutor()\n >>> dask.config.set(pool=pool) # set global ThreadPoolExecutor\n \n or\n"}]}]}]}]}]}