Start a Python Docker image with the right version of Python interpreter installed. For example,
docker run -it -v $(pwd):/workdir python:3.5-buster /bin/bash
Install pex.
pip3 install pex
Build a pex environment file.
pex --python=python3 -v pyspark findspark -o …Spark Internals - 6-CacheAndCheckpoint.md has a good explanation of persist vs checkpoint.
Persist/Cache in Spark is lazy and doesn't truncate the lineage while checkpoint is eager (by default) and truncates the lineage.
Generally speaking,
DataFrame.persist has a better performance than DataFrame.checkpoint.
However,
DataFrame.checkpoint is more robust and is preferred in any of the following situations.
Starting from Python 3.7,
dict preserves insertion order
(i.e., dict is ordered).
There is no need to use OrderedDict any more in Python 3.7+.
However,
set in Python is implemented as an unordered hashset
and thus is neither ordered nor sorted.
A trick to dedup an iterable values
Online Regular Expression Tester
The Python module re automatically compiles a plain/text pattern
using re.compile and caches it,
so there's not much benefit to compile plain/text patterns by yourself.
Some regular expression patterns are defined using a single leading backslash,
e.g., \s, \b, etc.
However,
since special characters (e.g., \) need to be escaped in strings in most programming languages,
you will need the string "\\s"