python-build-standalone is a better alternative to conda-pack on managing Python dependencies for PySpark. Please refer to Packaging Python Dependencies for PySpark Using python-build-standalone for tutorials on how to use python-build-standalone to manage Python dependencies for PySpark.

Build Portable Python Environments Using conda-pack

Please refer to the GitHub repo dclong/conda_environ …

It is suggested that you use python-build-standlone instead of conda-pack to build portable Python environments. Please refer to Packaging Python Dependencies for PySpark Using Python-Build-Standalone for more details.

1. All packages in a virtual environment must be managed by conda (rather than pip) so that it can be packe using conda-pack …

Steps to Build a pex Environment File

1. Start a Python Docker image with the right version of Python interpreter installed. For example,

   docker run -it -v $(pwd):/workdir python:3.5-buster /bin/bash
   

2. Install pex.

   pip3 install pex
   

3. Build a pex environment file.

   pex --python=python3 -v pyspark findspark -o …