Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Deploy Apache Ray

Please refer to Deploy Apache Ray for details.

Tutorials and Examples

How to scale Python multiprocessing to a cluster with one line of code demonstrates how you can scale a Python multiprocess application to run on an Apache Ray cluster with only one line change of your code (change import multiprocessing.Pool to ray.util.multiprocessing.Pool). Of course, you have to export the environment variable RAY_ADDRESS=auto before running the Python code or you have to make an another line of change which is to change Pool() to Pool(ray_address="auto"). For more details, please refer to Distributed multiprocessing.Pool .

https://github.com/ray-project/tutorial

References

Fault Tolerance

Development Tips

Distributed Iterators

Distributed Scikit-learn / Joblib

https://ray.readthedocs.io/en/latest/multiprocessing.html

https://github.com/ParallelSSH/parallel-ssh

https://ray.readthedocs.io/en/latest/walkthrough.html#remote-functions-tasks

https://arrow.apache.org/docs/python/plasma.html#the-plasma-in-memory-object-store

https://readthedocs.org/projects/ray/downloads/pdf/latest/

https://ray.readthedocs.io/en/latest/package-ref.html

https://towardsdatascience.com/benchmarking-python-distributed-ai-backends-with-wordbatch-9872457b785c

https://ray.readthedocs.io/en/latest/tune.html

https://ray.readthedocs.io/en/latest/rllib.html