Distributed Training of Models on Spark

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

XGBoost¶

Use XGBoost With Spark

LightGBM¶

Use LightGBM With Spark

BigDL¶

MMLSpark¶

Apache Ray¶

You can run Apache Ray on top of Spark via analytics-zoo, which enables you to run any Python machine lerning library in distributed fashion. But I’m not sure whether this is a good idea.

yahoo/TensorFlowOnSpark¶

PyTorch¶

H2O¶

https://github.com/h2oai/sparkling-water http://docs.h2o.ai/sparkling-water/2.2/latest-stable/doc/pysparkling.html http://h2o-release.s3.amazonaws.com/h2o/master/4273/docs-website/h2o-docs/faq/sparkling-water.html https://docs.databricks.com/_static/notebooks/h2o-sparkling-water-python.html

SystemML¶

elephas¶

Distributed training with Keras and Spark.

References¶

https://towardsdatascience.com/deep-learning-with-apache-spark-part-1-6d397c16abd