Ben Chuanlong Du's Blog

And let it direct your passion with reason.

Get CentOS Version

You can get the version of CentOS using the following command.

rpm -q centos-release

This trick can be used to get the version of the CentOS distribution on a Spark cluster. Basically, you run this command in the driver or workers to print the versions and then parse the log of the Spark application.

#!/usr/bin/env python3
import subprocess as sp
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("CentOS_Version") \
    .enableHiveSupport().getOrCreate()
sp.run("rpm -q centos-release", shell=True, check=True)

Comments