The simplest way to check whether a Linux system is running systemd, upstart or SysV is by running the following command.
ps -p1 | grep "init\|upstart\|systemd"
References
How to determine which system manager is running on Linux System
PySpark Issue: Java Gateway Process Exited Before Sending the Driver Its Port Number
I countered the issue when using PySpark locally
(the issue can happen to a cluster as well).
It turned out to be caused by a misconfiguration of the environment variable JAVA_HOME in Docker.
References
PySpark: Exception: Java gateway process exited before sending the driver its port number
Serialize and Deserialize Object Using Pickle in Python
Tips and Traps¶
- Make sure to use the mode
rb/wbwhen read/write pickle files.
Date Functions in Spark
Tips and Traps¶
- HDFS table might contain invalid data (I'm not clear about the reasons at this time) with respct to the column types (e.g., Date and Timestamp). This will cause issues when Spark tries to load the data. For more discussions, please refer to Unrecognized column type:TIMESTAMP_TYP.
datetime.datetimeordatetime.date
Count Number of Fields in Each Line
Sometimes, a structured text file might be malformatted. A simple way to verify it is to count the number of fields in each line.
Using awk
You can count the number of fields in each line using the following awk command. Unfortunately, awk does not take escaped characters into consideration …
Quickly Create a Scala Project Using Gradle in Intellij IDEA
Easy Way
-
Create a directory (e.g.,
demo_proj) for your project. -
Run
gradle init --type scala-libraryin terminal in the above directory. -
Import the directory as a Gradle project in IntelliJ IDEA. Alternatively, you can add
apply plugin: 'idea'intobuild.gradleand then run the command./gradlew openIdeato …