Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
%%classpath add mvn
org.apache.spark spark-core_2.11 2.3.1
org.apache.spark spark-sql_2.11 2.3.1Loading...
Loading...
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._
import org.apache.spark.sql.Row
val spark = SparkSession.builder()
.master("local[2]")
.appName("Spark Example")
.config("spark.some.config.option", "some-value")
.getOrCreate()
import spark.implicits._org.apache.spark.sql.SparkSession$implicits$@78b8022eimport org.apache.spark.sql.functions._
val df = Seq(
("Ben", "Du", 1),
("Ben", "Du", 2),
("Ben", "Tu", 3),
("Ben", "Tu", 4),
("Ken", "Xu", 1),
("Ken", "Xu", 9)
).toDF("fname", "lname", "score")
df.show+-----+-----+-----+
|fname|lname|score|
+-----+-----+-----+
| Ben| Du| 1|
| Ben| Du| 2|
| Ben| Tu| 3|
| Ben| Tu| 4|
| Ken| Xu| 1|
| Ken| Xu| 9|
+-----+-----+-----+
nullGet the First Row¶
df.first[Ben,Du,1]Get the first element of the first row as String.
df.first.getString(0)BenConstruct a Row¶
Row(1, "how", 'i', 2.0)[1,how,i,2.0]Row.fromSeq(Seq(1, "how", true, 2.0))[1,how,true,2.0]Get Elements of a Row¶
val row = Row(1, "how", 'i', 2.0)
row[1,how,i,2.0]row.getInt(0)1row.getAs(0)java.lang.ClassCastException: java.lang.Integer cannot be cast to scala.runtime.Nothing$
... 52 elidedrow.getString(1)howrow.getAs(1)java.lang.ClassCastException: java.lang.String cannot be cast to scala.runtime.Nothing$
... 52 elided