buildkit-cli-for-kubectl
is a plugin for kubectl
which provides a similar experience building Docker images on Kubernetes
as building Docker images locally using docker build.
buildkit-cli-for-kubectl
works perfectly in a personal/development Kubernetes cluster (e.g., minikube running locally),
however,
it doesn't work in an enterprise production environment
due to permission …
Get CentOS Version
You can get the version of CentOS using the following command.
rpm -q centos-release
This trick can be used to get the version of the CentOS distribution on a Spark cluster. Basically, you run this command in the driver or workers to print the versions and then parse the log …
Control Number of Partitions of a DataFrame in Spark
Tips and Traps¶
DataFrame.repartitionrepartitions the DataFrame by hash code of each row. If you specify a (multiple) column(s) (instead of number of partitions) to the methodDataFrame.repartition, then hash code of the column(s) are calculated for repartition. In some situations, there are lots of hash conflictions even if the total number of rows is small (e.g., a few thousand), which means that partitions generated might be skewed
Spark Issue: Shell Related
Symptom 1
/bin/sh: hdfs: command not found
Possible Causes of Symptom 1
The command hdfs is not on the search path.
Possible Solutions to Symptom 1
- Use the full path to the command.
- Configure the environment variable
PATHbefore you use the command. - Find other alternatives to the command …
Spark Issue: Namespace Quota Is Exceeded
Symptom
Caused by: org.apache.hadoop.hdfs.protocol.NSQuotaExceededException: The NameSpace quota (directories and files) of directory /user/user_name is exceeded: quota=163840 file count=163841
Cause
The namespace quota of the directory /user/user_name is execeeded.
Solutions
-
Remove non-needed files from the directory
/user/user_nameto release some namespace …
Spark Issue: Rust Panic
If you use Rust with Spark/PySpark and there are issues in the Rust code, you might get Rust panic error messages.
Symptom
Error: b"thread 'main' panicked at 'index out of bounds: the len is 15 but the index is 15', src/game.rs:131:39\nnote: run with …