Ben Chuanlong Du's Blog

It is never too late to learn.

Use Spark with the Almond Scala Kernel in JupyterLab

This notebook present a minimum example on how to use Spark with the Almond Scala kernel in Jupyterlab. Notice that Spark 2.4.2 is used since it is the only stable Spark version that supports Scala 2.12 as of now. Please refer to almond-sh/examples for more examples.

In [6]:
scala.util.Properties.versionString
Out[6]:
res5: String = "version 2.12.10"
In [2]:
interp.load.ivy("org.apache.spark" % "spark-core_2.12" % "2.4.2")
interp.load.ivy("org.apache.spark" % "spark-sql_2.12" % "2.4.2")
Downloading https://repo1.maven.org/maven2/org/apache/spark/spark-sql_2.12/2.4.2/spark-sql_2.12-2.4.2.pom
Downloaded https://repo1.maven.org/maven2/org/apache/spark/spark-sql_2.12/2.4.2/spark-sql_2.12-2.4.2.pom
Downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-column/1.10.1/parquet-column-1.10.1.pom
Downloading https://repo1.maven.org/maven2/org/apache/orc/orc-mapreduce/1.5.5/orc-mapreduce-1.5.5.pom
Downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-hadoop/1.10.1/parquet-hadoop-1.10.1.pom
Downloading https://repo1.maven.org/maven2/org/apache/spark/spark-sketch_2.12/2.4.2/spark-sketch_2.12-2.4.2.pom
Downloading https://repo1.maven.org/maven2/org/apache/orc/orc-core/1.5.5/orc-core-1.5.5.pom
Downloading https://repo1.maven.org/maven2/org/apache/arrow/arrow-vector/0.10.0/arrow-vector-0.10.0.pom
Downloaded https://repo1.maven.org/maven2/org/apache/parquet/parquet-column/1.10.1/parquet-column-1.10.1.pom
Downloading https://repo1.maven.org/maven2/com/univocity/univocity-parsers/2.7.3/univocity-parsers-2.7.3.pom
Downloaded https://repo1.maven.org/maven2/com/univocity/univocity-parsers/2.7.3/univocity-parsers-2.7.3.pom
Downloading https://repo1.maven.org/maven2/org/apache/spark/spark-catalyst_2.12/2.4.2/spark-catalyst_2.12-2.4.2.pom
Downloaded https://repo1.maven.org/maven2/org/apache/spark/spark-catalyst_2.12/2.4.2/spark-catalyst_2.12-2.4.2.pom
Downloaded https://repo1.maven.org/maven2/org/apache/arrow/arrow-vector/0.10.0/arrow-vector-0.10.0.pom
Downloaded https://repo1.maven.org/maven2/org/apache/orc/orc-core/1.5.5/orc-core-1.5.5.pom
Downloaded https://repo1.maven.org/maven2/org/apache/spark/spark-sketch_2.12/2.4.2/spark-sketch_2.12-2.4.2.pom
Downloaded https://repo1.maven.org/maven2/org/apache/parquet/parquet-hadoop/1.10.1/parquet-hadoop-1.10.1.pom
Downloaded https://repo1.maven.org/maven2/org/apache/orc/orc-mapreduce/1.5.5/orc-mapreduce-1.5.5.pom
Downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet/1.10.1/parquet-1.10.1.pom
Downloading https://repo1.maven.org/maven2/org/apache/orc/orc/1.5.5/orc-1.5.5.pom
Downloading https://repo1.maven.org/maven2/org/apache/arrow/arrow-java-root/0.10.0/arrow-java-root-0.10.0.pom
Downloaded https://repo1.maven.org/maven2/org/apache/arrow/arrow-java-root/0.10.0/arrow-java-root-0.10.0.pom
Downloaded https://repo1.maven.org/maven2/org/apache/parquet/parquet/1.10.1/parquet-1.10.1.pom
Downloaded https://repo1.maven.org/maven2/org/apache/orc/orc/1.5.5/orc-1.5.5.pom
Downloading https://repo1.maven.org/maven2/joda-time/joda-time/2.9.9/joda-time-2.9.9.pom
Downloading https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.7.9/jackson-core-2.7.9.pom
Downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-format/2.4.0/parquet-format-2.4.0.pom
Downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-common/1.10.1/parquet-common-1.10.1.pom
Downloading https://repo1.maven.org/maven2/com/google/code/findbugs/jsr305/3.0.2/jsr305-3.0.2.pom
Downloading https://repo1.maven.org/maven2/com/vlkan/flatbuffers/1.2.0-3f79e055/flatbuffers-1.2.0-3f79e055.pom
Downloaded https://repo1.maven.org/maven2/org/apache/parquet/parquet-format/2.4.0/parquet-format-2.4.0.pom
Downloaded https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.7.9/jackson-core-2.7.9.pom
Downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-encoding/1.10.1/parquet-encoding-1.10.1.pom
Downloading https://repo1.maven.org/maven2/com/carrotsearch/hppc/0.7.2/hppc-0.7.2.pom
Downloaded https://repo1.maven.org/maven2/joda-time/joda-time/2.9.9/joda-time-2.9.9.pom
Downloading https://repo1.maven.org/maven2/org/antlr/antlr4-runtime/4.7/antlr4-runtime-4.7.pom
Downloaded https://repo1.maven.org/maven2/org/apache/parquet/parquet-encoding/1.10.1/parquet-encoding-1.10.1.pom
Downloading https://repo1.maven.org/maven2/commons-codec/commons-codec/1.10/commons-codec-1.10.pom
Downloaded https://repo1.maven.org/maven2/com/carrotsearch/hppc/0.7.2/hppc-0.7.2.pom
Downloading https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.6.0/jackson-annotations-2.6.0.pom
Downloaded https://repo1.maven.org/maven2/com/vlkan/flatbuffers/1.2.0-3f79e055/flatbuffers-1.2.0-3f79e055.pom
Downloading https://repo1.maven.org/maven2/org/apache/arrow/arrow-format/0.10.0/arrow-format-0.10.0.pom
Downloaded https://repo1.maven.org/maven2/com/google/code/findbugs/jsr305/3.0.2/jsr305-3.0.2.pom
Downloading https://repo1.maven.org/maven2/io/airlift/aircompressor/0.10/aircompressor-0.10.pom
Downloaded https://repo1.maven.org/maven2/commons-codec/commons-codec/1.10/commons-codec-1.10.pom
Downloading https://repo1.maven.org/maven2/org/apache/orc/orc-shims/1.5.5/orc-shims-1.5.5.pom
Downloaded https://repo1.maven.org/maven2/org/apache/arrow/arrow-format/0.10.0/arrow-format-0.10.0.pom
Downloaded https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.6.0/jackson-annotations-2.6.0.pom
Downloading https://repo1.maven.org/maven2/org/scala-lang/modules/scala-parser-combinators_2.12/1.1.0/scala-parser-combinators_2.12-1.1.0.pom
Downloading https://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.7.25/slf4j-api-1.7.25.pom
Downloaded https://repo1.maven.org/maven2/org/antlr/antlr4-runtime/4.7/antlr4-runtime-4.7.pom
Downloading https://repo1.maven.org/maven2/org/codehaus/janino/commons-compiler/3.0.9/commons-compiler-3.0.9.pom
Downloaded https://repo1.maven.org/maven2/io/airlift/aircompressor/0.10/aircompressor-0.10.pom
Downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-jackson/1.10.1/parquet-jackson-1.10.1.pom
Downloaded https://repo1.maven.org/maven2/org/apache/orc/orc-shims/1.5.5/orc-shims-1.5.5.pom
Downloaded https://repo1.maven.org/maven2/org/apache/parquet/parquet-jackson/1.10.1/parquet-jackson-1.10.1.pom
Downloaded https://repo1.maven.org/maven2/org/codehaus/janino/commons-compiler/3.0.9/commons-compiler-3.0.9.pom
Downloaded https://repo1.maven.org/maven2/org/scala-lang/modules/scala-parser-combinators_2.12/1.1.0/scala-parser-combinators_2.12-1.1.0.pom
Downloading https://repo1.maven.org/maven2/org/apache/arrow/arrow-memory/0.10.0/arrow-memory-0.10.0.pom
Downloading https://repo1.maven.org/maven2/org/scala-lang/scala-reflect/2.11.12/scala-reflect-2.11.12.pom
Downloaded https://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.7.25/slf4j-api-1.7.25.pom
Downloading https://repo1.maven.org/maven2/org/codehaus/janino/janino/3.0.9/janino-3.0.9.pom
Downloaded https://repo1.maven.org/maven2/org/apache/parquet/parquet-common/1.10.1/parquet-common-1.10.1.pom
Downloaded https://repo1.maven.org/maven2/org/scala-lang/scala-reflect/2.11.12/scala-reflect-2.11.12.pom
Downloaded https://repo1.maven.org/maven2/org/apache/arrow/arrow-memory/0.10.0/arrow-memory-0.10.0.pom
Downloaded https://repo1.maven.org/maven2/org/codehaus/janino/janino/3.0.9/janino-3.0.9.pom
Downloading https://repo1.maven.org/maven2/com/carrotsearch/hppc-parent/0.7.2/hppc-parent-0.7.2.pom
Downloading https://repo1.maven.org/maven2/org/codehaus/janino/janino-parent/3.0.9/janino-parent-3.0.9.pom
Downloading https://repo1.maven.org/maven2/org/antlr/antlr4-master/4.7/antlr4-master-4.7.pom
Downloading https://repo1.maven.org/maven2/io/airlift/airbase/78/airbase-78.pom
Downloading https://repo1.maven.org/maven2/org/slf4j/slf4j-parent/1.7.25/slf4j-parent-1.7.25.pom
Downloaded https://repo1.maven.org/maven2/org/codehaus/janino/janino-parent/3.0.9/janino-parent-3.0.9.pom
Downloaded https://repo1.maven.org/maven2/org/slf4j/slf4j-parent/1.7.25/slf4j-parent-1.7.25.pom
Downloaded https://repo1.maven.org/maven2/com/carrotsearch/hppc-parent/0.7.2/hppc-parent-0.7.2.pom
Downloaded https://repo1.maven.org/maven2/io/airlift/airbase/78/airbase-78.pom
Downloaded https://repo1.maven.org/maven2/org/antlr/antlr4-master/4.7/antlr4-master-4.7.pom
Downloading https://repo1.maven.org/maven2/org/apache/commons/commons-parent/35/commons-parent-35.pom
Downloaded https://repo1.maven.org/maven2/org/apache/commons/commons-parent/35/commons-parent-35.pom
Downloading https://repo1.maven.org/maven2/org/apache/apache/15/apache-15.pom
Downloaded https://repo1.maven.org/maven2/org/apache/apache/15/apache-15.pom
Downloading https://repo1.maven.org/maven2/org/apache/spark/spark-sketch_2.12/2.4.2/spark-sketch_2.12-2.4.2.jar
Downloading https://repo1.maven.org/maven2/com/univocity/univocity-parsers/2.7.3/univocity-parsers-2.7.3.jar
Downloading https://repo1.maven.org/maven2/commons-codec/commons-codec/1.10/commons-codec-1.10.jar
Downloading https://repo1.maven.org/maven2/com/google/code/findbugs/jsr305/3.0.2/jsr305-3.0.2.jar
Downloading https://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.7.25/slf4j-api-1.7.25.jar
Downloading https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.7.9/jackson-core-2.7.9.jar
Downloaded https://repo1.maven.org/maven2/com/google/code/findbugs/jsr305/3.0.2/jsr305-3.0.2.jar
Downloading https://repo1.maven.org/maven2/org/scala-lang/modules/scala-parser-combinators_2.12/1.1.0/scala-parser-combinators_2.12-1.1.0.jar
Downloaded https://repo1.maven.org/maven2/org/apache/spark/spark-sketch_2.12/2.4.2/spark-sketch_2.12-2.4.2.jar
Downloading https://repo1.maven.org/maven2/org/codehaus/janino/commons-compiler/3.0.9/commons-compiler-3.0.9.jar
Downloaded https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.7.9/jackson-core-2.7.9.jar
Downloading https://repo1.maven.org/maven2/org/antlr/antlr4-runtime/4.7/antlr4-runtime-4.7.jar
Downloaded https://repo1.maven.org/maven2/org/codehaus/janino/commons-compiler/3.0.9/commons-compiler-3.0.9.jar
Downloading https://repo1.maven.org/maven2/org/apache/spark/spark-catalyst_2.12/2.4.2/spark-catalyst_2.12-2.4.2.jar
Downloaded https://repo1.maven.org/maven2/com/univocity/univocity-parsers/2.7.3/univocity-parsers-2.7.3.jar
Downloading https://repo1.maven.org/maven2/org/codehaus/janino/janino/3.0.9/janino-3.0.9.jar
Downloaded https://repo1.maven.org/maven2/org/scala-lang/modules/scala-parser-combinators_2.12/1.1.0/scala-parser-combinators_2.12-1.1.0.jar
Downloading https://repo1.maven.org/maven2/org/apache/orc/orc-core/1.5.5/orc-core-1.5.5-nohive.jar
Downloaded https://repo1.maven.org/maven2/commons-codec/commons-codec/1.10/commons-codec-1.10.jar
Downloading https://repo1.maven.org/maven2/org/apache/orc/orc-mapreduce/1.5.5/orc-mapreduce-1.5.5-nohive.jar
Downloaded https://repo1.maven.org/maven2/org/antlr/antlr4-runtime/4.7/antlr4-runtime-4.7.jar
Downloading https://repo1.maven.org/maven2/org/apache/orc/orc-shims/1.5.5/orc-shims-1.5.5.jar
Downloaded https://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.7.25/slf4j-api-1.7.25.jar
Downloading https://repo1.maven.org/maven2/io/airlift/aircompressor/0.10/aircompressor-0.10.jar
Downloaded https://repo1.maven.org/maven2/org/codehaus/janino/janino/3.0.9/janino-3.0.9.jar
Downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-jackson/1.10.1/parquet-jackson-1.10.1.jar
Downloaded https://repo1.maven.org/maven2/org/apache/orc/orc-shims/1.5.5/orc-shims-1.5.5.jar
Downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-format/2.4.0/parquet-format-2.4.0.jar
Downloaded https://repo1.maven.org/maven2/io/airlift/aircompressor/0.10/aircompressor-0.10.jar
Downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-hadoop/1.10.1/parquet-hadoop-1.10.1.jar
Downloaded https://repo1.maven.org/maven2/org/apache/parquet/parquet-format/2.4.0/parquet-format-2.4.0.jar
Downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-column/1.10.1/parquet-column-1.10.1.jar
Downloaded https://repo1.maven.org/maven2/org/apache/parquet/parquet-jackson/1.10.1/parquet-jackson-1.10.1.jar
Downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-encoding/1.10.1/parquet-encoding-1.10.1.jar
Downloaded https://repo1.maven.org/maven2/org/apache/parquet/parquet-hadoop/1.10.1/parquet-hadoop-1.10.1.jar
Downloading https://repo1.maven.org/maven2/org/apache/arrow/arrow-format/0.10.0/arrow-format-0.10.0.jar
Downloaded https://repo1.maven.org/maven2/org/apache/arrow/arrow-format/0.10.0/arrow-format-0.10.0.jar
Downloading https://repo1.maven.org/maven2/joda-time/joda-time/2.9.9/joda-time-2.9.9.jar
Downloaded https://repo1.maven.org/maven2/org/apache/orc/orc-mapreduce/1.5.5/orc-mapreduce-1.5.5-nohive.jar
Downloading https://repo1.maven.org/maven2/com/vlkan/flatbuffers/1.2.0-3f79e055/flatbuffers-1.2.0-3f79e055.jar
Downloaded https://repo1.maven.org/maven2/org/apache/orc/orc-core/1.5.5/orc-core-1.5.5-nohive.jar
Downloading https://repo1.maven.org/maven2/org/apache/arrow/arrow-memory/0.10.0/arrow-memory-0.10.0.jar
Downloaded https://repo1.maven.org/maven2/org/apache/parquet/parquet-column/1.10.1/parquet-column-1.10.1.jar
Downloading https://repo1.maven.org/maven2/org/apache/arrow/arrow-vector/0.10.0/arrow-vector-0.10.0.jar
Downloaded https://repo1.maven.org/maven2/com/vlkan/flatbuffers/1.2.0-3f79e055/flatbuffers-1.2.0-3f79e055.jar
Downloading https://repo1.maven.org/maven2/org/apache/spark/spark-sql_2.12/2.4.2/spark-sql_2.12-2.4.2.jar
Downloaded https://repo1.maven.org/maven2/org/apache/arrow/arrow-memory/0.10.0/arrow-memory-0.10.0.jar
Downloading https://repo1.maven.org/maven2/com/carrotsearch/hppc/0.7.2/hppc-0.7.2.jar
Downloaded https://repo1.maven.org/maven2/org/apache/arrow/arrow-vector/0.10.0/arrow-vector-0.10.0.jar
Downloading https://repo1.maven.org/maven2/org/apache/spark/spark-sketch_2.12/2.4.2/spark-sketch_2.12-2.4.2-sources.jar
Downloaded https://repo1.maven.org/maven2/joda-time/joda-time/2.9.9/joda-time-2.9.9.jar
Downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-common/1.10.1/parquet-common-1.10.1.jar
Downloaded https://repo1.maven.org/maven2/org/apache/spark/spark-sketch_2.12/2.4.2/spark-sketch_2.12-2.4.2-sources.jar
Downloading https://repo1.maven.org/maven2/com/univocity/univocity-parsers/2.7.3/univocity-parsers-2.7.3-sources.jar
Downloaded https://repo1.maven.org/maven2/org/apache/parquet/parquet-common/1.10.1/parquet-common-1.10.1.jar
Downloading https://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.7.25/slf4j-api-1.7.25-sources.jar
Downloaded https://repo1.maven.org/maven2/com/carrotsearch/hppc/0.7.2/hppc-0.7.2.jar
Downloading https://repo1.maven.org/maven2/commons-codec/commons-codec/1.10/commons-codec-1.10-sources.jar
Downloaded https://repo1.maven.org/maven2/com/univocity/univocity-parsers/2.7.3/univocity-parsers-2.7.3-sources.jar
Downloading https://repo1.maven.org/maven2/com/google/code/findbugs/jsr305/3.0.2/jsr305-3.0.2-sources.jar
Downloaded https://repo1.maven.org/maven2/commons-codec/commons-codec/1.10/commons-codec-1.10-sources.jar
Downloading https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.7.9/jackson-core-2.7.9-sources.jar
Downloaded https://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.7.25/slf4j-api-1.7.25-sources.jar
Downloading https://repo1.maven.org/maven2/org/antlr/antlr4-runtime/4.7/antlr4-runtime-4.7-sources.jar
Downloaded https://repo1.maven.org/maven2/com/google/code/findbugs/jsr305/3.0.2/jsr305-3.0.2-sources.jar
Downloading https://repo1.maven.org/maven2/org/apache/spark/spark-catalyst_2.12/2.4.2/spark-catalyst_2.12-2.4.2-sources.jar
Downloaded https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.7.9/jackson-core-2.7.9-sources.jar
Downloading https://repo1.maven.org/maven2/org/scala-lang/modules/scala-parser-combinators_2.12/1.1.0/scala-parser-combinators_2.12-1.1.0-sources.jar
Downloaded https://repo1.maven.org/maven2/org/apache/parquet/parquet-encoding/1.10.1/parquet-encoding-1.10.1.jar
Downloading https://repo1.maven.org/maven2/org/apache/orc/orc-shims/1.5.5/orc-shims-1.5.5-sources.jar
Downloaded https://repo1.maven.org/maven2/org/apache/orc/orc-shims/1.5.5/orc-shims-1.5.5-sources.jar
Downloading https://repo1.maven.org/maven2/org/codehaus/janino/janino/3.0.9/janino-3.0.9-sources.jar
Downloaded https://repo1.maven.org/maven2/org/scala-lang/modules/scala-parser-combinators_2.12/1.1.0/scala-parser-combinators_2.12-1.1.0-sources.jar
Downloading https://repo1.maven.org/maven2/org/apache/orc/orc-mapreduce/1.5.5/orc-mapreduce-1.5.5-sources.jar
Downloaded https://repo1.maven.org/maven2/org/apache/orc/orc-mapreduce/1.5.5/orc-mapreduce-1.5.5-sources.jar
Downloading https://repo1.maven.org/maven2/org/apache/orc/orc-core/1.5.5/orc-core-1.5.5-sources.jar
Downloaded https://repo1.maven.org/maven2/org/antlr/antlr4-runtime/4.7/antlr4-runtime-4.7-sources.jar
Downloading https://repo1.maven.org/maven2/io/airlift/aircompressor/0.10/aircompressor-0.10-sources.jar
Downloaded https://repo1.maven.org/maven2/org/codehaus/janino/janino/3.0.9/janino-3.0.9-sources.jar
Downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-encoding/1.10.1/parquet-encoding-1.10.1-sources.jar
Downloaded https://repo1.maven.org/maven2/org/apache/spark/spark-catalyst_2.12/2.4.2/spark-catalyst_2.12-2.4.2-sources.jar
Downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-jackson/1.10.1/parquet-jackson-1.10.1-sources.jar
Downloaded https://repo1.maven.org/maven2/io/airlift/aircompressor/0.10/aircompressor-0.10-sources.jar
Downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-hadoop/1.10.1/parquet-hadoop-1.10.1-sources.jar
Downloaded https://repo1.maven.org/maven2/org/apache/orc/orc-core/1.5.5/orc-core-1.5.5-sources.jar
Downloading https://repo1.maven.org/maven2/org/apache/arrow/arrow-format/0.10.0/arrow-format-0.10.0-sources.jar
Downloaded https://repo1.maven.org/maven2/org/apache/parquet/parquet-jackson/1.10.1/parquet-jackson-1.10.1-sources.jar
Downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-common/1.10.1/parquet-common-1.10.1-sources.jar
Downloaded https://repo1.maven.org/maven2/org/apache/arrow/arrow-format/0.10.0/arrow-format-0.10.0-sources.jar
Downloading https://repo1.maven.org/maven2/joda-time/joda-time/2.9.9/joda-time-2.9.9-sources.jar
Downloaded https://repo1.maven.org/maven2/org/apache/parquet/parquet-common/1.10.1/parquet-common-1.10.1-sources.jar
Downloading https://repo1.maven.org/maven2/com/vlkan/flatbuffers/1.2.0-3f79e055/flatbuffers-1.2.0-3f79e055-sources.jar
Downloaded https://repo1.maven.org/maven2/org/apache/parquet/parquet-hadoop/1.10.1/parquet-hadoop-1.10.1-sources.jar
Downloading https://repo1.maven.org/maven2/com/carrotsearch/hppc/0.7.2/hppc-0.7.2-sources.jar
Downloaded https://repo1.maven.org/maven2/org/apache/parquet/parquet-encoding/1.10.1/parquet-encoding-1.10.1-sources.jar
Downloading https://repo1.maven.org/maven2/org/apache/spark/spark-sql_2.12/2.4.2/spark-sql_2.12-2.4.2-sources.jar
Downloaded https://repo1.maven.org/maven2/com/vlkan/flatbuffers/1.2.0-3f79e055/flatbuffers-1.2.0-3f79e055-sources.jar
Downloading https://repo1.maven.org/maven2/org/codehaus/janino/commons-compiler/3.0.9/commons-compiler-3.0.9-sources.jar
Downloaded https://repo1.maven.org/maven2/org/codehaus/janino/commons-compiler/3.0.9/commons-compiler-3.0.9-sources.jar
Downloading https://repo1.maven.org/maven2/org/apache/arrow/arrow-vector/0.10.0/arrow-vector-0.10.0-sources.jar
Downloaded https://repo1.maven.org/maven2/joda-time/joda-time/2.9.9/joda-time-2.9.9-sources.jar
Downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-column/1.10.1/parquet-column-1.10.1-sources.jar
Downloaded https://repo1.maven.org/maven2/org/apache/parquet/parquet-column/1.10.1/parquet-column-1.10.1-sources.jar
Downloading https://repo1.maven.org/maven2/org/apache/arrow/arrow-memory/0.10.0/arrow-memory-0.10.0-sources.jar
Downloaded https://repo1.maven.org/maven2/org/apache/arrow/arrow-vector/0.10.0/arrow-vector-0.10.0-sources.jar
Downloading https://repo1.maven.org/maven2/org/apache/parquet/parquet-format/2.4.0/parquet-format-2.4.0-sources.jar
Downloaded https://repo1.maven.org/maven2/org/apache/arrow/arrow-memory/0.10.0/arrow-memory-0.10.0-sources.jar
Downloaded https://repo1.maven.org/maven2/org/apache/parquet/parquet-format/2.4.0/parquet-format-2.4.0-sources.jar
Downloaded https://repo1.maven.org/maven2/com/carrotsearch/hppc/0.7.2/hppc-0.7.2-sources.jar
Downloaded https://repo1.maven.org/maven2/org/apache/spark/spark-sql_2.12/2.4.2/spark-sql_2.12-2.4.2-sources.jar
Downloaded https://repo1.maven.org/maven2/org/apache/spark/spark-catalyst_2.12/2.4.2/spark-catalyst_2.12-2.4.2.jar
Downloaded https://repo1.maven.org/maven2/org/apache/spark/spark-sql_2.12/2.4.2/spark-sql_2.12-2.4.2.jar
In [3]:
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._

val spark = SparkSession.builder()
    .master("local[2]")
    .appName("Spark Example")
    .getOrCreate()

import spark.implicits._
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
20/03/23 17:30:21 INFO SparkContext: Running Spark version 2.4.2
20/03/23 17:30:22 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/03/23 17:30:22 INFO SparkContext: Submitted application: Spark Example
20/03/23 17:30:22 INFO SecurityManager: Changing view acls to: dclong
20/03/23 17:30:22 INFO SecurityManager: Changing modify acls to: dclong
20/03/23 17:30:22 INFO SecurityManager: Changing view acls groups to: 
20/03/23 17:30:22 INFO SecurityManager: Changing modify acls groups to: 
20/03/23 17:30:22 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(dclong); groups with view permissions: Set(); users  with modify permissions: Set(dclong); groups with modify permissions: Set()
20/03/23 17:30:23 INFO Utils: Successfully started service 'sparkDriver' on port 46185.
20/03/23 17:30:23 INFO SparkEnv: Registering MapOutputTracker
20/03/23 17:30:23 INFO SparkEnv: Registering BlockManagerMaster
20/03/23 17:30:23 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
20/03/23 17:30:23 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
20/03/23 17:30:23 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-8657440f-2a62-4cb0-8134-acf22f99a429
20/03/23 17:30:23 INFO MemoryStore: MemoryStore started with capacity 1315.2 MB
20/03/23 17:30:23 INFO SparkEnv: Registering OutputCommitCoordinator
20/03/23 17:30:23 INFO Utils: Successfully started service 'SparkUI' on port 4040.
20/03/23 17:30:23 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://jupyterhub-ds:4040
20/03/23 17:30:23 INFO Executor: Starting executor ID driver on host localhost
20/03/23 17:30:23 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 33483.
20/03/23 17:30:23 INFO NettyBlockTransferService: Server created on jupyterhub-ds:33483
20/03/23 17:30:23 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
20/03/23 17:30:23 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, jupyterhub-ds, 33483, None)
20/03/23 17:30:23 INFO BlockManagerMasterEndpoint: Registering block manager jupyterhub-ds:33483 with 1315.2 MB RAM, BlockManagerId(driver, jupyterhub-ds, 33483, None)
20/03/23 17:30:23 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, jupyterhub-ds, 33483, None)
20/03/23 17:30:23 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, jupyterhub-ds, 33483, None)
Out[3]:
import org.apache.spark.sql.SparkSession

import org.apache.spark.sql.functions._


spark: SparkSession = org.apache.spark.sql.SparkSession@6721d392
import spark.implicits._
In [5]:
val df = Seq(
    (0, "hello"), 
    (1, "world")
).toDF("id", "text")
df.show
20/03/23 17:31:19 INFO ContextCleaner: Cleaned accumulator 0
20/03/23 17:31:19 INFO CodeGenerator: Code generated in 33.2682 ms
20/03/23 17:31:20 INFO CodeGenerator: Code generated in 17.3675 ms
20/03/23 17:31:20 INFO CodeGenerator: Code generated in 15.8421 ms
+---+-----+
| id| text|
+---+-----+
|  0|hello|
|  1|world|
+---+-----+

Out[5]:
df: org.apache.spark.sql.package.DataFrame = [id: int, text: string]

Comments