Yann Büchau :python: on Nostr: Insight of the week: If you want to parallelize data processing with #Python with ...
Insight of the week:
If you want to parallelize data processing with #Python with
#scons, call separate scripts. scons runs Python-only builders as threads, which is amazing to keep the code simple and concise and even shared progress bars are possible, but Python's GIL (I guess) makes it super ineffective. Running the builders in a multiprocessing.Process is unreliable and prone to freezing or is downright impossible when your data structures aren't pickleable.
If you want to parallelize data processing with #Python with
#scons, call separate scripts. scons runs Python-only builders as threads, which is amazing to keep the code simple and concise and even shared progress bars are possible, but Python's GIL (I guess) makes it super ineffective. Running the builders in a multiprocessing.Process is unreliable and prone to freezing or is downright impossible when your data structures aren't pickleable.