Tips on Apache Ray

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Deploy Apache Ray¶

Please refer to Deploy Apache Ray for details.

Tutorials and Examples¶

How to scale Python multiprocessing to a cluster with one line of code demonstrates how you can scale a Python multiprocess application to run on an Apache Ray cluster with only one line change of your code (change import multiprocessing.Pool to ray.util.multiprocessing.Pool). Of course, you have to export the environment variable RAY_ADDRESS=auto before running the Python code or you have to make an another line of change which is to change Pool() to Pool(ray_address="auto"). For more details, please refer to Distributed multiprocessing.Pool .

https://github.com/ray-project/tutorial

References¶

Fault Tolerance

Development Tips

Distributed Iterators

Distributed Scikit-learn / Joblib

[https://ray.readthedocs.io/en/latest/multiprocessing.html](Distributed multiprocessing.Pool)

https://github.com/ParallelSSH/parallel-ssh

https://ray.readthedocs.io/en/latest/walkthrough.html#remote-functions-tasks

https://arrow.apache.org/docs/python/plasma.html#the-plasma-in-memory-object-store

https://readthedocs.org/projects/ray/downloads/pdf/latest/

https://ray.readthedocs.io/en/latest/package-ref.html

https://towardsdatascience.com/benchmarking-python-distributed-ai-backends-with-wordbatch-9872457b785c

https://ray.readthedocs.io/en/latest/tune.html

https://ray.readthedocs.io/en/latest/rllib.html