The simplest way to check whether a Linux system is running systemd, upstart or SysV is by running the following command.
ps -p1 | grep "init\|upstart\|systemd"
References
How to determine which system manager is running on Linux System
PySpark Issue: Java Gateway Process Exited Before Sending the Driver Its Port Number
I countered the issue when using PySpark locally
(the issue can happen to a cluster as well).
It turned out to be caused by a misconfiguration of the environment variable JAVA_HOME in Docker.
References
PySpark: Exception: Java gateway process exited before sending the driver its port number
Serialize and Deserialize Object Using Pickle in Python
Tips and Traps¶
- Make sure to use the mode
rb/wbwhen read/write pickle files.
Visualize Nvidia GPU Usage
You can use the tool nvtop (Linux only)
to visualize the usage of Nvidia GPUs.
However,
it is only available on Linux
and is not suitable for tracking and visualize the GPU usage in a long time period.
Another simple approach to track and visualize the GPU usage is
to dump GPU usage statistics into a CSV file
using the following command
Build Docker Images Using Kaniko
-
Kaniko works differently from Docker. It runs inside a Docker container and detect and extract new layers to build Docker images. Since Kaniko manipulates the filesystem (layers) inside the Docker container, it can have unexpected side effect if not used carefully. For this reason, the developer team suggests users to …
WSL2 Filesystem
Performance Issue
The performance of IO is extremely bad if you access the the Windows filesystem from WSL 2. This is also true if you use Docker in WSL 2 (since Docker on WSL 2 share the same kernel as WSL 2). It is suggested that you stick to WSL …