java.lang.RuntimeException: Unsupported literal type class java.util.ArrayList [1]
This happens in PySpark
when a Python list is provide where a scalar is required.
Assuming id0 is an integer column in the DataFrame df,
the following code throws the above error.
v = [1, 2, 3 …I countered the issue when using PySpark locally
(the issue can happen to a cluster as well).
It turned out to be caused by a misconfiguration of the environment variable JAVA_HOME in Docker.
PySpark: Exception: Java gateway process exited before sending the driver its port number
datetime.datetime or datetime.date
python-build-standalone is a better alternative to conda-pack on managing Python dependencies for PySpark. Please refer to Packaging Python Dependencies for PySpark Using python-build-standalone for tutorials on how to use python-build-standalone to manage Python dependencies for PySpark.
Please refer to the GitHub repo dclong/conda_environ …