Ben Chuanlong Du's Blog

It is never too late to learn.

Spark Issue: java.lang.OutOfMemoryError

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symptom

OutOfMemoryError

Cause

java.lang.OutOfMemoryError is thrown when there is not enough heap memory (for JVM to allocating new objects).

Solution

Increase executor memory.

--executor-memory=20G

Reference:

http://stackoverflow.com/questions/27462061/why-does-spark-fail-with-java-lang-outofmemoryerror-gc-overhead-limit-exceeded

Comments