The simplest way to check whether a Linux system is running systemd, upstart or SysV is by running the following command.

ps -p1 | grep "init\|upstart\|systemd"

References

How to determine which system manager is running on Linux System

How to know if I am using systemd on Linux?

I countered the issue when using PySpark locally (the issue can happen to a cluster as well). It turned out to be caused by a misconfiguration of the environment variable JAVA_HOME in Docker.

References

PySpark: Exception: Java gateway process exited before sending the driver its port number

Sometimes, a structured text file might be malformatted. A simple way to verify it is to count the number of fields in each line.

Using awk

You can count the number of fields in each line using the following awk command. Unfortunately, awk does not take escaped characters into consideration …

Easy Way

1. Create a directory (e.g., demo_proj) for your project.

2. Run gradle init --type scala-library in terminal in the above directory.

3. Import the directory as a Gradle project in IntelliJ IDEA. Alternatively, you can add apply plugin: 'idea' into build.gradle and then run the command ./gradlew openIdea to …

Extensions

Please refer to Useful Visual Studio Code Extensions .

Set Python Environment for Visual Studio Code Server

1. File -> Preference -> Settings

2. Click on Workspace.

3. Search for Python Path.

4. Change Python Path to the one you want to use.

Debug a Python Project

Visual Studio Live Share

Visual Studio Live Share

What …

It is recommended that you use pip to install Python packages.

1. If you don't already know the proxy in use (in your company), read the post Find out Proxy in Use to figure it out.

2. Set proxy environment variables.

   set http_proxy=http://user:password@proxy_ip:port
   set https_proxy=https://user …