Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Show Error Messages Only¶
When you run Spark or PySpark in a Jupyter/Lab notebook, it is recommended that you show ERROR messages only. Otherwise, there might be too much logging information polluting your notebook. You can set the log level of Spark to ERROR using the following line of code.
In [ ]:
spark.sparkContext.setLogLevel("ERROR")