The function took 5.12 s to compute.\n+The function took 5.10 s to compute.\n \n The transformed data are:\n [ 0.49671415 -0.1382643 0.64768854 1.52302986 -0.23415337 -0.23413696\n 1.57921282 0.76743473 -0.46947439 0.54256004]\n
\n \n At the first call, the results will be cached. Therefore, the computation\n@@ -125,15 +125,15 @@\n data_trans = costly_compute_cached(data)\n end = time.time()\n \n print('\\nThe function took {:.2f} s to compute.'.format(end - start))\n print('\\nThe transformed data are:\\n {}'.format(data_trans))\n
The function took 0.04 s to compute.\n+The function took 0.03 s to compute.\n \n The transformed data are:\n [ 0.49671415 -0.1382643 0.64768854 1.52302986 -0.23415337 -0.23413696\n 1.57921282 0.76743473 -0.46947439 0.54256004]\n
\n \n At the second call, the computation time is largely reduced since the results\n@@ -168,45 +168,45 @@\n data_trans = transformer.transform(data)\n end = time.time()\n \n print('\\nThe function took {:.2f} s to compute.'.format(end - start))\n print('\\nThe transformed data are:\\n {}'.format(data_trans))\n
The function took 5.11 s to compute.\n+The function took 5.10 s to compute.\n \n The transformed data are:\n [ 0.49671415 -0.1382643 0.64768854 1.52302986 -0.23415337 -0.23413696\n 1.57921282 0.76743473 -0.46947439 0.54256004]\n
\n \n start = time.time()\n data_trans = transformer.transform(data)\n end = time.time()\n \n print('\\nThe function took {:.2f} s to compute.'.format(end - start))\n print('\\nThe transformed data are:\\n {}'.format(data_trans))\n
\n \n-The function took 0.03 s to compute.\n+The function took 0.05 s to compute.\n \n The transformed data are:\n [ 0.49671415 -0.1382643 0.64768854 1.52302986 -0.23415337 -0.23413696\n 1.57921282 0.76743473 -0.46947439 0.54256004]\n
\n \n As expected, the second call to the transform method load the results\n which have been cached.
\n \n \n Clean up cache directory\u00b6
\n memory.clear(warn=False)\n
\n \n-Total running time of the script: (0 minutes 15.381 seconds)
\n+Total running time of the script: (0 minutes 15.367 seconds)
\n \n
\n First round - caching the data\n-Elapsed time for the entire processing: 4.86 s\n+Elapsed time for the entire processing: 4.66 s\n
\n \n By using 2 workers, the parallel processing gives a x2 speed-up compared to\n the sequential case. By executing again the same process, the intermediate\n results obtained by calling costly_compute_cached will be loaded from the\n cache instead of executing the function.
\n \n \n Second round - reloading from the cache\n-Elapsed time for the entire processing: 0.05 s\n+Elapsed time for the entire processing: 0.03 s\n
\n \n \n \n Reuse intermediate checkpoints\u00b6
\n \n \n
\n Reusing intermediate checkpoints\n-Elapsed time for the entire processing: 0.03 s\n+Elapsed time for the entire processing: 0.04 s\n
\n \n The processing time only corresponds to the execution of the max\n function. The internal call to costly_compute_cached is reloading the\n results from the cache.
\n \n \n Clean-up the cache folder\u00b6
\n memory.clear(warn=False)\n
\n \n-Total running time of the script: (0 minutes 12.966 seconds)
\n+Total running time of the script: (0 minutes 12.769 seconds)
\n \n Running tasks with return_as='list'...\n Accumulate results:......................................................................................................................................................\n All tasks completed and reduced successfully.\n-Peak memory usage: 2.44GB\n+Peak memory usage: 2.47GB\n If we use return_as="generator", res is simply a generator on the\n results that are ready. Here we consume the results as soon as they arrive\n with the accumulator_sum and once they have been used, they are collected\n by the gc. The memory footprint is thus reduced, typically around 300MB.
monitor_gen = MemoryMonitor()\n@@ -169,15 +169,15 @@\n peak = max(monitor_gen.memory_buffer) / 1e6\n print(f"Peak memory usage: {peak:.2f}MB")\n Create result generator with return_as='generator'...\n Accumulate results:......................................................................................................................................................\n All tasks completed and reduced successfully.\n-Peak memory usage: 219.11MB\n+Peak memory usage: 210.11MB\n We can then report the memory usage accross time of the two runs using the\n MemoryMonitor.
\nIn the first case, as the results accumulate in res, the memory grows\n linearly and it is freed once the accumulator_sum function finishes.
In the second case, the results are processed by the accumulator as soon as\n@@ -200,15 +200,15 @@\n plt.show()\n
![\"parallel](\"../_images/sphx_glr_parallel_generator_001.png\")
It is important to note that with return_as="generator", the results are\n still accumulated in RAM after computation. But as we asynchronously process\n them, they can be freed sooner. However, if the generator is not consumed\n the memory still grows linearly.
Total running time of the script: (0 minutes 28.004 seconds)
\n+Total running time of the script: (0 minutes 33.228 seconds)
\n \n \n-Elapsed time computing the average of couple of slices 0.85 s\n+Elapsed time computing the average of couple of slices 0.88 s\n
\n \n Parallel processing is already faster than the sequential processing. It is\n also possible to remove a bit of overhead by dumping the data array to a\n memmap and pass the memmap to joblib.Parallel.
\n \n \n-Elapsed time computing the average of couple of slices 0.60 s\n+Elapsed time computing the average of couple of slices 0.62 s\n
\n \n Therefore, dumping large data array ahead of calling\n joblib.Parallel can speed up the processing by removing some\n overhead.
\n \n \n \n Clean-up the memmap\u00b6
\n \n Remove the different memmap that we created. It might fail in Windows due\n@@ -225,15 +225,15 @@\n \n try:\n shutil.rmtree(folder)\n except: # noqa\n print('Could not clean-up automatically.')\n
\n
\n-Total running time of the script: (0 minutes 3.262 seconds)
\n+Total running time of the script: (0 minutes 3.471 seconds)
\n \n \n The different generated vectors in a sequential manner are:\n- [[3 4 5 4 1]\n- [7 7 8 6 7]\n- [2 1 5 3 3]\n- [5 6 9 8 7]\n- [0 5 0 8 0]]\n+ [[3 9 3 3 6]\n+ [0 3 8 0 5]\n+ [6 2 0 8 4]\n+ [3 3 4 7 9]\n+ [5 8 2 9 5]]\n
\n \n \n \n Parallel behavior\u00b6
\n \n Joblib provides three different backends: loky (default), threading, and\n@@ -98,33 +98,33 @@\n
backend = 'loky'\n random_vector = Parallel(n_jobs=2, backend=backend)(delayed(\n stochastic_function)(10) for _ in range(n_vectors))\n print_vector(random_vector, backend)\n
\n \n The different generated vectors using the loky backend are:\n- [[1 4 9 7 7]\n- [7 7 2 0 4]\n- [2 5 5 8 4]\n- [9 0 3 3 7]\n- [4 5 8 4 0]]\n+ [[5 1 1 7 4]\n+ [0 9 0 5 7]\n+ [7 7 6 1 3]\n+ [6 4 1 7 8]\n+ [7 2 3 6 2]]\n
\n \n backend = 'threading'\n random_vector = Parallel(n_jobs=2, backend=backend)(delayed(\n stochastic_function)(10) for _ in range(n_vectors))\n print_vector(random_vector, backend)\n
\n \n The different generated vectors using the threading backend are:\n- [[4 3 6 4 0]\n- [9 3 2 2 6]\n- [6 6 9 3 1]\n- [6 8 0 1 5]\n- [3 3 7 2 9]]\n+ [[3 5 1 1 5]\n+ [9 6 1 3 2]\n+ [1 6 9 5 4]\n+ [6 0 6 5 1]\n+ [3 2 5 9 3]]\n
\n \n Loky and the threading backends behave exactly as in the sequential case and\n do not require more care. However, this is not the case regarding the\n multiprocessing backend with the \u201cfork\u201d or \u201cforkserver\u201d start method because\n the state of the global numpy random stated will be exactly duplicated\n in all the workers
\n@@ -137,19 +137,19 @@\n backend = 'multiprocessing'\n random_vector = Parallel(n_jobs=2, backend=backend)(delayed(\n stochastic_function)(10) for _ in range(n_vectors))\n print_vector(random_vector, backend)\n \n
The different generated vectors using the multiprocessing backend are:\n- [[4 6 5 8 5]\n- [3 7 3 5 3]\n- [9 7 9 6 9]\n- [1 0 7 2 8]\n- [2 0 7 3 4]]\n+ [[2 0 1 5 8]\n+ [6 8 1 3 4]\n+ [2 5 5 1 0]\n+ [4 6 6 4 2]\n+ [0 1 1 6 7]]\n Some of the generated vectors are exactly the same, which can be a\n problem for the application.
\nTechnically, the reason is that all forked Python processes share the\n same exact random seed. As a result, we obtain twice the same randomly\n generated vectors because we are using n_jobs=2. A solution is to\n@@ -166,19 +166,19 @@\n
if mp.get_start_method() != "spawn":\n random_vector = Parallel(n_jobs=2, backend=backend)(delayed(\n stochastic_function_seeded)(10, None) for _ in range(n_vectors))\n print_vector(random_vector, backend)\n The different generated vectors using the multiprocessing backend are:\n- [[2 4 8 7 5]\n- [2 8 6 9 6]\n- [5 6 7 2 7]\n- [1 0 2 4 6]\n- [3 0 1 6 8]]\n+ [[9 7 1 3 1]\n+ [7 1 5 6 3]\n+ [4 7 0 3 2]\n+ [1 2 2 1 1]\n+ [4 9 4 9 9]]\n \n\nThe pattern of
stochastic_function_seededhas another advantage: it\n@@ -195,29 +195,29 @@\n \n random_vector = Parallel(n_jobs=2, backend=backend)(delayed(\n stochastic_function_seeded)(10, rng) for rng in random_state)\n print_vector(random_vector, backend)\n
The different generated vectors using the multiprocessing backend are:\n- [[1 4 0 3 1]\n- [4 0 8 2 6]\n- [6 8 9 7 0]\n- [1 5 0 6 8]\n- [5 0 2 3 2]]\n+ [[7 0 1 2 1]\n+ [7 8 9 8 6]\n+ [9 6 1 6 6]\n+ [6 0 5 2 8]\n+ [7 6 2 0 4]]\n \n The different generated vectors using the multiprocessing backend are:\n- [[1 4 0 3 1]\n- [4 0 8 2 6]\n- [6 8 9 7 0]\n- [1 5 0 6 8]\n- [5 0 2 3 2]]\n+ [[7 0 1 2 1]\n+ [7 8 9 8 6]\n+ [9 6 1 6 6]\n+ [6 0 5 2 8]\n+ [7 6 2 0 4]]\n Total running time of the script: (0 minutes 1.472 seconds)
\n+Total running time of the script: (0 minutes 1.871 seconds)
\n \n \n-With loky backend and cloudpickle serialization: 0.158s\n+With loky backend and cloudpickle serialization: 0.257s\n
\n \n If you are on a UNIX system, it is possible to fallback to the old\n multiprocessing backend, which can pickle interactively defined functions\n with the default pickle module, which is faster for such large objects.
\n import multiprocessing as mp\n if mp.get_start_method() != "spawn":\n@@ -105,15 +105,15 @@\n t_start = time.time()\n Parallel(n_jobs=2)(\n delayed(func_async)(21, large_list) for _ in range(1))\n print("With multiprocessing backend and pickle serialization: {:.3f}s"\n .format(time.time() - t_start))\n
\n \n-With multiprocessing backend and pickle serialization: 0.505s\n+With multiprocessing backend and pickle serialization: 0.619s\n
\n \n However, using fork to start new processes can cause violation of the\n POSIX specification and can have bad interaction with compiled extensions\n that use openmp. Also, it is not possible to start processes with\n fork on windows where only spawn is available. The loky backend\n has been developed to mitigate these issues.
\n@@ -130,15 +130,15 @@\n \n set_loky_pickler('pickle')\n t_start = time.time()\n Parallel(n_jobs=2)(delayed(id)(large_list) for _ in range(1))\n print("With pickle serialization: {:.3f}s".format(time.time() - t_start))\n
\n \n-With pickle serialization: 0.169s\n+With pickle serialization: 0.251s\n
\n \n However, the function and objects defined in __main__ are not\n serializable anymore using pickle and it is not possible to call\n func_async using this pickler.
\n \n \n-With pickle from stdlib and wrapper: 0.619s\n+With pickle from stdlib and wrapper: 0.842s\n
\n \n The same wrapper can also be used for non-picklable classes. Note that the\n side effects of wrap_non_picklable_objects on objects can break magic\n methods such as __add__ and can mess up the isinstance and\n issubclass functions. Some improvements will be considered if use-cases\n are reported.
\n # Reset the loky_pickler to avoid border effects with other examples in\n # sphinx-gallery.\n set_loky_pickler()\n
\n \n-Total running time of the script: (0 minutes 1.648 seconds)
\n+Total running time of the script: (0 minutes 2.218 seconds)
\n