Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Things under legendu.net/outdated are outdated technologies that the author does not plan to update any more. Please look for better alternatives. It is suggested that you use python-build-standlone instead of conda-pack to build portable Python environments. Please refer to Packaging Python Dependencies for PySpark Using Python-Build-Standalone for more details.

Things under legendu.net/outdated are outdated technologies that the author does not plan to update any more. Please look for better alternatives.

  1. All packages in a virtual environment must be managed by conda (rather than pip) so that it can be packe using conda-pack.

  2. When using a conda-pack virtual environment with PySpark, the Python package pyyspark comes with Spark is automatically injected into PYTHONPATH so that users do not have to install pyspark into the virtual environemnt by themselves. As a matter of fact, the pyspark comes with Spark is always used even if you have a local copy installed when you submit a PySpark application with a conda-pack virtual environment. For more discussions, please refer to this isue.

References

Pack a Conda Virtual Environment

https://conda.github.io/conda-pack/

https://conda.github.io/conda-pack/cli.html

Apr
The BeakerX JupyterLab Kernel
Apr
Use Spark With the BeakerX Scala Kernel