Visual Studio Code Marketplace and Open VSX Registry are 2 places to find VSCode compatible extensions.
https://stackoverflow.com/questions/34286515/how-to-install-visual-studio-code-extensions-from-command-line/34339780#34339780
If you install extension in Dockerfile using root,
the extensions are installed …
It is suggested that you use python-build-standlone instead of conda-pack to build portable Python environments. Please refer to Packaging Python Dependencies for PySpark Using Python-Build-Standalone for more details.
All packages in a virtual environment must be managed by conda (rather than pip) so that it can be packe using conda-pack …
Start a Python Docker image with the right version of Python interpreter installed. For example,
docker run -it -v $(pwd):/workdir python:3.5-buster /bin/bash
Install pex.
pip3 install pex
Build a pex environment file.
pex --python=python3 -v pyspark findspark -o …Spark Internals - 6-CacheAndCheckpoint.md has a good explanation of persist vs checkpoint.
Persist/Cache in Spark is lazy and doesn't truncate the lineage while checkpoint is eager (by default) and truncates the lineage.
Generally speaking,
DataFrame.persist has a better performance than DataFrame.checkpoint.
However,
DataFrame.checkpoint is more robust and is preferred in any of the following situations.
It is suggested that you always pass a list of columns to the parameter on even if there's only one column for joining.
None in a pandas DataFrame is converted to NaN instead of null!
Spark allows using following join types:
inner (default)crossouterfull, fullouter, full_outerleft, leftouter, left_outerright, rightouter, right_outersemi, leftsemi, left_semianti, leftanti, left_anti