Ben Chuanlong Du's Blog

And let it direct your passion with reason.

Hands on the Python Module argparse

  1. argparse is the best library to use to parse command-line arguments in Python. It is included in Python standard libaries (which menas that you can use it out of the box).

  2. It is suggestedd that you always log a parsed Namespace object so that you can check whether it is as expected.

Special Characters to Avoid in Strings

This articles talks about special characters to avoid in various places. You might not encounter issues most of the time when violating rules stated below, however, you never know when things will break. It is more for a good-practice suggestion.

String for Shell

  1. When you pass parameters from one program …

Sample Rows from a Spark DataFrame

Tips and Traps

  1. TABLESAMPLE must be immedidately after a table name.

  2. The WHERE clause in the following SQL query runs after TABLESAMPLE.

     SELECT 
     *
 FROM 
     table_name 
 TABLESAMPLE (10 PERCENT) 
 WHERE 
     id = 1

    If you want to run a WHERE

Union RDDs in Spark

No deduplication is done (to be efficient) when unioning RDDs/DataFrames in Spark 2.1.0+.

  1. Union 2 RDDs.

    df1.union(df2)
// or for old-fashioned RDD
rdd1.union(rdd_2)

  2. Union multiple RDDs.

    df = spark.union([df1, df2, df3]) // spark is a SparkSession object
// or for old-fashioned RDD
rdd …

Union DataFrames in Spark

Comment

  1. union relies on column order rather than column names. This is the same as in SQL. For columns that the type don't match, the super type is used. However, this is really dangerous if you are careful. It is suggested that you define a function call unionByName to hanle this.