It is suggested that you use python-build-standlone instead of conda-pack to build portable Python environments. Please refer to Packaging Python Dependencies for PySpark Using Python-Build-Standalone for more details.
All packages in a virtual environment must be managed by conda (rather than pip) so that it can be packe using conda-pack …
Start a Python Docker image with the right version of Python interpreter installed. For example,
docker run -it -v $(pwd):/workdir python:3.5-buster /bin/bash
Install pex.
pip3 install pex
Build a pex environment file.
pex --python=python3 -v pyspark findspark -o …Spark Internals - 6-CacheAndCheckpoint.md has a good explanation of persist vs checkpoint.
Persist/Cache in Spark is lazy and doesn't truncate the lineage while checkpoint is eager (by default) and truncates the lineage.
Generally speaking,
DataFrame.persist has a better performance than DataFrame.checkpoint.
However,
DataFrame.checkpoint is more robust and is preferred in any of the following situations.
It is suggested that you always pass a list of columns to the parameter on even if there's only one column for joining.
None in a pandas DataFrame is converted to NaN instead of null!
Spark allows using following join types:
inner (default)crossouterfull, fullouter, full_outerleft, leftouter, left_outerright, rightouter, right_outersemi, leftsemi, left_semianti, leftanti, left_antiStarting from Python 3.7,
dict preserves insertion order
(i.e., dict is ordered).
There is no need to use OrderedDict any more in Python 3.7+.
However,
set in Python is implemented as an unordered hashset
and thus is neither ordered nor sorted.
A trick to dedup an iterable values