The set Collection in Python

Apr 30, 2021

General Tips and Traps¶

The set class is implemented based on hash table which means that its elements must be hashable (has methods __hash__ and __eq__). The set class implements the mathematical concepts of set which means that its elements are unordered and does not perserve insertion order of elements. Notice that this is different from the dict class which is also implemented based on hash table but keeps insertion order of elements! The article Why don't Python sets preserve insertion order?

Apr 30, 2021

Visual Studio Code Marketplace and Open VSX Registry are 2 places to find VSCode compatible extensions.

https://stackoverflow.com/questions/34286515/how-to-install-visual-studio-code-extensions-from-command-line/34339780#34339780

If you install extension in Dockerfile using root, the extensions are installed …

Apr 30, 2021

It is suggested that you use python-build-standlone instead of conda-pack to build portable Python environments. Please refer to Packaging Python Dependencies for PySpark Using Python-Build-Standalone for more details.

All packages in a virtual environment must be managed by conda (rather than pip) so that it can be packe using conda-pack …

Apr 26, 2021

Start a Python Docker image with the right version of Python interpreter installed. For example,
```
docker run -it -v $(pwd):/workdir python:3.5-buster /bin/bash
```
Install pex.
```
pip3 install pex
```

Build a pex environment file.

pex --python=python3 -v pyspark findspark -o …

Apr 26, 2021

Jan 24, 2021

Spark Internals - 6-CacheAndCheckpoint.md has a good explanation of persist vs checkpoint.

Persist/Cache in Spark is lazy and doesn't truncate the lineage while checkpoint is eager (by default) and truncates the lineage.
Generally speaking, DataFrame.persist has a better performance than DataFrame.checkpoint. However, DataFrame.checkpoint is more robust and is preferred in any of the following situations.