I countered the issue when using PySpark locally (the issue can happen to a cluster as well). It turned out to be caused by a misconfiguration of the environment variable JAVA_HOME in Docker.

References

PySpark: Exception: Java gateway process exited before sending the driver its port number

1. Kaniko works differently from Docker. It runs inside a Docker container and detect and extract new layers to build Docker images. Since Kaniko manipulates the filesystem (layers) inside the Docker container, it can have unexpected side effect if not used carefully. For this reason, the developer team suggests users to …

Performance Issue

The performance of IO is extremely bad if you access the the Windows filesystem from WSL 2. This is also true if you use Docker in WSL 2 (since Docker on WSL 2 share the same kernel as WSL 2). It is suggested that you stick to WSL …

You can run Docker containers inside a Docker container. To allow this, you have to pass the docker socks into the container using the option -v /var/run/docker.sock:/var/run/docker.sock. For more discussions, please refer to How To Run Docker in Docker Container [3 Easy Methods …

You can build a portable Python environment following steps below.

1. Install python-build-standalone.

2. Install Python packages using pip of the installed python-build-standalone distribution.

3. Pack the whole python-build-standalone directory into a compressed file, e.g., env.tar.gz.

The GitHub repo dclong/python-portable has good examples of building portable Python environments leveraging …

You can use Maven to download dependencies of Java packages without creating a Java project. For example, if you want to download all dependencies of arrow-jvm and arrow-memory and build everything into a single fat jar (for easy use in other places), you can first crate a file pom.xml …