Ben Chuanlong Du's Blog

And let it direct your passion with reason.

Install Python Packages Behind Firewall

It is recommended that you use pip to install Python packages.

  1. If you don't already know the proxy in use (in your company), read the post Find out Proxy in Use to figure it out.

  2. Set proxy environment variables.

    set http_proxy=http://user:password@proxy_ip:port
    set https_proxy=https://user …

Visualize Nvidia GPU Usage

You can use the tool nvtop (Linux only) to visualize the usage of Nvidia GPUs. However, it is only available on Linux and is not suitable for tracking and visualize the GPU usage in a long time period. Another simple approach to track and visualize the GPU usage is to dump GPU usage statistics into a CSV file using the following command

Build Docker Images Using Kaniko

  1. Kaniko works differently from Docker. It runs inside a Docker container and detect and extract new layers to build Docker images. Since Kaniko manipulates the filesystem (layers) inside the Docker container, it can have unexpected side effect if not used carefully. For this reason, the developer team suggests users to …

Hands on the Python module subprocess

General Tips

  1. The method subprocess.run is preferred over older high-level APIs (subprocess.call, subprocess.check_call and subprocess.check_output). The method subprocess.Popen (which powers the high-level APIs) can be used if you need advanced controls. When running a shell command using subprocess.run,

    1. Avoid using system shell (i.e., avoid using shell=True) for 2 reasons. First, avoid shell injection attack. Second, there is no need for you to manually escape special characters in the command.