Symptom
java.lang.RuntimeException: Unsupported literal type class java.util.ArrayList [1]
Possible Causes
This happens in PySpark
when a Python list is provide where a scalar is required.
Assuming id0
is an integer column in the DataFrame df
,
the following code throws the above error.
v = [1, 2, 3]
df.filter(col("id0") == v)
Possible Solutions
- Use a scalar value for
v
in the above code example. - Use
isin
to check whether the value ofid0
is in the listv
.v = [1, 2, 3] df.filter(col("id0").isin(v))