BuildKit is a good tool for building Docker images on a Kubernetes cluster where you have root access.
Kaniko is another usable tool but it is not as intuitive as buildkit-cli-for-kubectl to use. As a matter of fact, tricky issues might arise when building Docker images using Kaniko.
buildah is …
buildkit-cli-for-kubectl
is a plugin for kubectl
which provides a similar experience building Docker images on Kubernetes
as building Docker images locally using docker build.
buildkit-cli-for-kubectl
works perfectly in a personal/development Kubernetes cluster (e.g., minikube running locally),
however,
it doesn't work in an enterprise production environment
due to permission …
You can get the version of CentOS using the following command.
rpm -q centos-release
This trick can be used to get the version of the CentOS distribution on a Spark cluster. Basically, you run this command in the driver or workers to print the versions and then parse the log …
DataFrame.repartition repartitions the DataFrame by hash code of each row.
If you specify a (multiple) column(s) (instead of number of partitions)
to the method DataFrame.repartition,
then hash code of the column(s) are calculated for repartition.
In some situations,
there are lots of hash conflictions
even if the total number of rows is small (e.g., a few thousand),
which means that
partitions generated might be skewed
/bin/sh: hdfs: command not found
The command hdfs is not on the search path.
PATH before you use the command.