PySpark Issue: Java Gateway Process Exited Before Sending the Driver Its Port Number

Oct 10, 2021

I countered the issue when using PySpark locally (the issue can happen to a cluster as well). It turned out to be caused by a misconfiguration of the environment variable JAVA_HOME in Docker.

References

PySpark: Exception: Java gateway process exited before sending the driver its port number

Serialize and Deserialize Object Using Pickle in Python

Oct 03, 2021

Tips and Traps¶

Make sure to use the mode rb/wb when read/write pickle files.

Date Functions in Spark

Apr 27, 2021

Tips and Traps¶

HDFS table might contain invalid data (I'm not clear about the reasons at this time) with respct to the column types (e.g., Date and Timestamp). This will cause issues when Spark tries to load the data. For more discussions, please refer to Unrecognized column type:TIMESTAMP_TYP.

datetime.datetime or datetime.date

Count Number of Fields in Each Line

Jun 13, 2016

Sometimes, a structured text file might be malformatted. A simple way to verify it is to count the number of fields in each line.

Using awk

You can count the number of fields in each line using the following awk command. Unfortunately, awk does not take escaped characters into consideration …

Quickly Create a Scala Project Using Gradle in Intellij IDEA

Jan 26, 2019

Easy Way

Create a directory (e.g., demo_proj) for your project.
Run gradle init --type scala-library in terminal in the above directory.
Import the directory as a Gradle project in IntelliJ IDEA. Alternatively, you can add apply plugin: 'idea' into build.gradle and then run the command ./gradlew openIdea to …