You can get the version of CentOS using the following command.
rpm -q centos-release
This trick can be used to get the version of the CentOS distribution on a Spark cluster. Basically, you run this command in the driver or workers to print the versions and then parse the log …
DataFrame.repartition repartitions the DataFrame by hash code of each row.
If you specify a (multiple) column(s) (instead of number of partitions)
to the method DataFrame.repartition,
then hash code of the column(s) are calculated for repartition.
In some situations,
there are lots of hash conflictions
even if the total number of rows is small (e.g., a few thousand),
which means that
partitions generated might be skewed
/bin/sh: hdfs: command not found
The command hdfs is not on the search path.
PATH before you use the command.Caused by: org.apache.hadoop.hdfs.protocol.NSQuotaExceededException: The NameSpace quota (directories and files) of directory /user/user_name is exceeded: quota=163840 file count=163841
The namespace quota of the directory /user/user_name is execeeded.
Remove non-needed files from the directory /user/user_name to release some namespace …
If you use Rust with Spark/PySpark and there are issues in the Rust code, you might get Rust panic error messages.
Error: b"thread 'main' panicked at 'index out of bounds: the len is 15 but the index is 15', src/game.rs:131:39\nnote: run with …
java.lang.RuntimeException: Unsupported literal type class java.util.ArrayList [1]
This happens in PySpark
when a Python list is provide where a scalar is required.
Assuming id0 is an integer column in the DataFrame df,
the following code throws the above error.
v = [1, 2, 3 …