Ben Chuanlong Du's Blog

And let it direct your passion with reason.

Sample Rows from a Spark DataFrame

Tips and Traps

  1. TABLESAMPLE must be immedidately after a table name.

  2. The WHERE clause in the following SQL query runs after TABLESAMPLE.

     SELECT 
         *
     FROM 
         table_name 
     TABLESAMPLE (10 PERCENT) 
     WHERE 
         id = 1
    
    

    If you want to run a WHERE

Union DataFrames in Spark

Comment

  1. union relies on column order rather than column names. This is the same as in SQL. For columns that the type don't match, the super type is used. However, this is really dangerous if you are careful. It is suggested that you define a function call unionByName to hanle this.