Ben Chuanlong Du's Blog

It is never too late to learn.

UDF in Spark

Comments

Use the higher-level standard Column-based functions with Dataset operators whenever possible before reverting to using your own custom UDF functions since UDFs are a blackbox for Spark and so it does not even try to optimize them.

In [2]:
interp.load.ivy("org.apache.spark" %% "spark-core" % "3.0.0")
interp.load.ivy("org.apache.spark" %% "spark-sql" % "3.0.0")
In [5]:
import org.apache.spark.sql.SparkSession

val spark = SparkSession
    .builder()
    .master("local[2]")
    .appName("Spark UDF Examples")
    .getOrCreate()
import spark.implicits._

20/09/07 11:51:53 WARN SparkSession$Builder: Using an existing SparkSession; some spark core configurations may not take effect.
Out[5]:
import org.apache.spark.sql.SparkSession


spark: SparkSession = org.apache.spark.sql.SparkSession@72b56dce
import spark.implicits._
In [4]:
val df = Seq(
    (0, "hello"), 
    (1, "world")
).toDF("id", "text")
df.show

20/09/07 11:43:35 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/workdir/archives/blog/misc/content/spark-warehouse/').
20/09/07 11:43:35 INFO SharedState: Warehouse path is 'file:/workdir/archives/blog/misc/content/spark-warehouse/'.
20/09/07 11:43:37 INFO CodeGenerator: Code generated in 405.037 ms
20/09/07 11:43:38 INFO CodeGenerator: Code generated in 17.1985 ms
20/09/07 11:43:38 INFO CodeGenerator: Code generated in 23.8635 ms

+---+-----+
| id| text|
+---+-----+
|  0|hello|
|  1|world|
+---+-----+
Out[4]:
df: org.apache.spark.sql.package.DataFrame = [id: int, text: string]
In [4]:
import org.apache.spark.sql.functions.udf

val upper: String => String = _.toUpperCase
val upperUDF = udf(upper)
Out[4]:
UserDefinedFunction(<function1>,StringType,Some(List(StringType)))
In [5]:
df.withColumn("upper", upperUDF($"text")).show

+---+-----+-----+
| id| text|upper|
+---+-----+-----+
|  0|hello|HELLO|
|  1|world|WORLD|
+---+-----+-----+
In [6]:
val someUDF = udf((arg1: Long, arg2: Long) => {
    arg1 + arg2
})
Out[6]:
UserDefinedFunction(<function2>,LongType,Some(List(LongType, LongType)))

Map vs UDF

https://stackoverflow.com/questions/38860808/performance-impact-of-rdd-api-vs-udfs-mixed-with-dataframe-api

https://stackoverflow.com/questions/39039081/difference-between-a-map-and-udf

https://stackoverflow.com/questions/43411234/spark-sql-whether-to-use-row-transformation-or-udf

References

https://spark.apache.org/docs/latest/api/java/index.html?org/apache/spark/sql/Dataset.html

https://spark.apache.org/docs/latest/api/java/index.html?org/apache/spark/sql/functions.html

https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/Row.html

https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-sql-udfs.html

https://blog.cloudera.com/blog/2017/02/working-with-udfs-in-apache-spark/

In [ ]:

Comments