Ben Chuanlong Du's Blog

And let it direct your passion with reason.

Visualize Nvidia GPU Usage

You can use the tool nvtop (Linux only) to visualize the usage of Nvidia GPUs. However, it is only available on Linux and is not suitable for tracking and visualize the GPU usage in a long time period. Another simple approach to track and visualize the GPU usage is to dump GPU usage statistics into a CSV file using the following command

nvidia-smi --query-gpu=timestamp,name,pci.bus_id,driver_version,pstate,\
    pcie.link.gen.max,pcie.link.gen.current,temperature.gpu,utilization.gpu,\
    utilization.memory,memory.total,memory.free,memory.used --format=csv -l 5

and then visualize it using Python.

In [1]:
import pandas as pd
In [30]:
import hvplot
import hvplot.pandas
In [40]:
!curl -sSL www.legendu.net/media/ai/gpu_usage.csv -o gpu_usage.csv
In [38]:
df = pd.read_csv("gpu_usage.csv")
df.columns = [
    col.replace("[MiB]", "").replace("[%]", "").replace(".", "_").strip()
    for col in df.columns
]
df["timestamp"] = pd.to_datetime(df.timestamp)
df["memory_used"] = df["memory_used"].str.replace("MiB", "").astype(float)
df
Out[38]:
timestamp name pci_bus_id driver_version pstate pcie_link_gen_max pcie_link_gen_current temperature_gpu utilization_gpu utilization_memory memory_total memory_free memory_used
0 2021-09-17 10:37:12.676 NVIDIA GeForce RTX 3070 00000000:01:00.0 470.57.02 P8 3 1 40 0 % 0 % 7948 MiB 6887 MiB 1061.0
1 2021-09-17 10:37:17.678 NVIDIA GeForce RTX 3070 00000000:01:00.0 470.57.02 P8 3 1 39 0 % 0 % 7948 MiB 6887 MiB 1061.0
2 2021-09-17 10:37:22.679 NVIDIA GeForce RTX 3070 00000000:01:00.0 470.57.02 P8 3 1 40 0 % 0 % 7948 MiB 6887 MiB 1061.0
3 2021-09-17 10:37:27.680 NVIDIA GeForce RTX 3070 00000000:01:00.0 470.57.02 P8 3 1 39 0 % 0 % 7948 MiB 6887 MiB 1061.0
4 2021-09-17 10:37:32.682 NVIDIA GeForce RTX 3070 00000000:01:00.0 470.57.02 P8 3 1 40 0 % 0 % 7948 MiB 6887 MiB 1061.0
... ... ... ... ... ... ... ... ... ... ... ... ... ...
8675 2021-09-17 22:40:18.576 NVIDIA GeForce RTX 3070 00000000:01:00.0 470.57.02 P8 3 1 41 15 % 16 % 7948 MiB 7076 MiB 872.0
8676 2021-09-17 22:40:23.578 NVIDIA GeForce RTX 3070 00000000:01:00.0 470.57.02 P8 3 1 42 0 % 9 % 7948 MiB 6989 MiB 959.0
8677 2021-09-17 22:40:28.579 NVIDIA GeForce RTX 3070 00000000:01:00.0 470.57.02 P8 3 1 41 0 % 9 % 7948 MiB 6989 MiB 959.0
8678 2021-09-17 22:40:33.580 NVIDIA GeForce RTX 3070 00000000:01:00.0 470.57.02 P8 3 1 41 23 % 20 % 7948 MiB 7012 MiB 936.0
8679 2021-09-17 22:40:38.582 NVIDIA GeForce RTX 3070 00000000:01:00.0 470.57.02 P8 3 1 42 14 % 16 % 7948 MiB 7027 MiB 921.0

8680 rows × 13 columns

In [39]:
df.hvplot.scatter("timestamp", "memory_used", size=0.5)
Out[39]:

Comments