1. PySpark 2.4 and older does not support Python 3.8. You have to use Python 3.7 with PySpark 2.4 or older.

2. It can be extremely helpful to run a PySpark application locally to detect possible issues before submitting it to the Spark cluster.

   #!/usr/bin/env bash …

Sometime when you install softwares in Linux, you get an error message saying that you do not have permission while you used sudo or the root account. This probably means that you do not have full access to some installation files. An easy way to solve this problem is to …

Before Installation

Debian Specific

1. You'd better not install back ported Debian images, as it might cause issues with other software (e.g., VirtualBox). It is suggested that you use Debian test.

Other Debian-based Linux Distributions

1. Download the right ISO image of the Linux distribution that you want to install.

2. Create …

Use Ventoy

Ventoy is the best graphical tool for making bootable flash drives currently.

Use the Command dd or cat

You can use

dd if=path_to_linux_image of=path_to_device bs=4M; sync

or

cat path_to_linux_image > path_to_device

to write a hybird Linux image into a flash drive. Note that you must …

1. BuildKit is a good tool for building Docker images on a Kubernetes cluster where you have root access.

2. Kaniko is another usable tool but it is not as intuitive as buildkit-cli-for-kubectl to use. As a matter of fact, tricky issues might arise when building Docker images using Kaniko.

3. buildah is …

buildkit-cli-for-kubectl is a plugin for kubectl which provides a similar experience building Docker images on Kubernetes as building Docker images locally using docker build. buildkit-cli-for-kubectl works perfectly in a personal/development Kubernetes cluster (e.g., minikube running locally), however, it doesn't work in an enterprise production environment due to permission …