Ben Chuanlong Du's Blog

It is never too late to learn.

Spark Issue: InsertOperationConflictException: Failed to Hold Insert Operation Lock

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Symptom

org.apache.spark.InsertOperationConflictException: Failed to hold insert operation lock ...

Cause

Multiple Spark applications attempts to write to the same directory at the same time.

Solution

Resubmit your Spark application with a different output path or make sure that it is the only application writing to the output path.

Comments