IntelliJ IDEA 開發環境

建立視覺化的 IDE 與編譯環境

環境設定

OS: ubuntu 16.04

預先安裝: JAVA、Python、Scala、SBT，請參考: https://spark-nctu.gitbook.io/spark/~/edit/drafts/-LVOFMS4dLpEWdkNY_y_/spark-de-jing-jian-li

設定目標: 建立 IDEA 開發環境，可以以 Scala 寫程式，並以 SBT 編譯執行

安裝 IntelliJ IDEA

下載 IntelliJ IDEA，選擇 Community: https://www.jetbrains.com/idea/download/#section=linux

解壓縮，並執行 idea.run

jxke@jxke-Spark:~/Downloads$ cd idea-IC-183.4886.37/
jxke@jxke-Spark:~/Downloads/idea-IC-183.4886.37$ cd bin/
jxke@jxke-Spark:~/Downloads/idea-IC-183.4886.37/bin$ ./idea.sh

執行後，會出現圖形化介面 ，記得要安裝 Scala 的 plug-in:

select Scala plug-in

建立 Scala 專案和 SBT 編譯環境

選擇建立新的專案 (project)，選擇 Scala -> sbt (如下圖所示)

create project - 1

create project -2

接著設定 SBT 的環境參數:

name := "test"

version := "0.1"

scalaVersion := "2.11.6"

libraryDependencies += "org.apache.spark" %% "spark-core" % "2.1.2"

libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.1.2"

resolvers += "Akka Repository" at "http://repo.akka.io/releases/"

注意: 原本設定中，應該是: scalaVersion := "2.12.8"，但是 Scala 2.12 似乎和 Spark 有相容性問題，因此要改回 2.11 的版本。請參考: https://stackoverflow.com/questions/41531852/spark-word-count-error-in-intellij-due-to-scala-2-12-1

更改完設定後，會出現是否要更新環境變數的視窗 (如下圖)，選擇 "Enable Auto-Import"，若是第一次執行會耗費很久的時間。

sbt update message

選擇 New -> Scala class，貼上範例程式 (SparkPi)，如下:

package org.apache.spark.examples

import scala.math.random
import org.apache.spark.sql.SparkSession
import org.apache.log4j.{Level, Logger}

/** Computes an approximation to pi */
object SparkPi {
  def main(args: Array[String]) {

    Logger.getLogger("org.apache.spark").setLevel(Level.WARN)
    Logger.getLogger("org.eclipse.jetty.server").setLevel(Level.OFF)

    val spark = SparkSession
      .builder
      .appName("sparkpi")
      .config("spark.master", "local")
      .getOrCreate()
    val slices = if (args.length > 0) args(0).toInt else 2
    val n = math.min(100000L * slices, Int.MaxValue).toInt // avoid overflow
    val count = spark.sparkContext.parallelize(1 until n, slices).map { i =>
      val x = random * 2 - 1
      val y = random * 2 - 1
      if (x * x + y * y <= 1) 1 else 0
    }.reduce(_ + _)
    
    println(s"Pi is roughly ${4.0 * count / (n - 1)}")
    spark.stop()
  }
}

// scalastyle:on println

編譯後執行，可以看到以下結果:

/usr/lib/jvm/java-8-oracle/bin/java -javaagent:/home/jxke/Downloads/idea-IC-183.4886.37/lib/idea_rt.jar=45331:/home/jxke/Downloads/idea-IC-183.4886.37/bin -Dfile.encoding=UTF-8 -classpath /usr/lib/jvm/java-8-oracle/jre/lib/charsets.jar:/usr/lib/jvm/java-8-oracle/jre/lib/deploy.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/cldrdata.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/dnsns.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/jaccess.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/jfxrt.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/localedata.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/nashorn.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/sunec.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/sunjce_provider.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/sunpkcs11.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/zipfs.jar:/usr/lib/jvm/java-8-oracle/jre/lib/javaws.jar:/usr/lib/jvm/java-8-oracle/jre/lib/jce.jar:/usr/lib/jvm/java-8-oracle/jre/lib/jfr.jar:/usr/lib/jvm/java-8-oracle/jre/lib/jfxswt.jar:/usr/lib/jvm/java-8-oracle/jre/lib/jsse.jar:/usr/lib/jvm/java-8-oracle/jre/lib/management-agent.jar:/usr/lib/jvm/java-8-oracle/jre/lib/plugin.jar:/usr/lib/jvm/java-8-oracle/jre/lib/resources.jar:/usr/lib/jvm/java-8-oracle/jre/lib/rt.jar:/home/jxke/IdeaProjects/test/target/scala-2.11/classes:/home/jxke/.ivy2/cache/aopalliance/aopalliance/jars/aopalliance-1.0.jar:/home/jxke/.ivy2/cache/xmlenc/xmlenc/jars/xmlenc-0.52.jar:/home/jxke/.ivy2/cache/oro/oro/jars/oro-2.0.8.jar:/home/jxke/.ivy2/cache/org.xerial.snappy/snappy-java/bundles/snappy-java-1.1.2.6.jar:/home/jxke/.ivy2/cache/org.tukaani/xz/jars/xz-1.0.jar:/home/jxke/.ivy2/cache/org.spark-project.spark/unused/jars/unused-1.0.0.jar:/home/jxke/.ivy2/cache/org.sonatype.sisu.inject/cglib/jars/cglib-2.2.1-v20090111.jar:/home/jxke/.ivy2/cache/org.slf4j/slf4j-log4j12/jars/slf4j-log4j12-1.7.16.jar:/home/jxke/.ivy2/cache/org.slf4j/slf4j-api/jars/slf4j-api-1.7.16.jar:/home/jxke/.ivy2/cache/org.slf4j/jul-to-slf4j/jars/jul-to-slf4j-1.7.16.jar:/home/jxke/.ivy2/cache/org.slf4j/jcl-over-slf4j/jars/jcl-over-slf4j-1.7.16.jar:/home/jxke/.ivy2/cache/org.scala-lang.modules/scala-xml_2.11/bundles/scala-xml_2.11-1.0.3.jar:/home/jxke/.ivy2/cache/org.scala-lang.modules/scala-parser-combinators_2.11/bundles/scala-parser-combinators_2.11-1.0.3.jar:/home/jxke/.ivy2/cache/org.scala-lang/scalap/jars/scalap-2.11.6.jar:/home/jxke/.ivy2/cache/org.scala-lang/scala-reflect/jars/scala-reflect-2.11.6.jar:/home/jxke/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.11.6.jar:/home/jxke/.ivy2/cache/org.scala-lang/scala-compiler/jars/scala-compiler-2.11.6.jar:/home/jxke/.ivy2/cache/org.roaringbitmap/RoaringBitmap/bundles/RoaringBitmap-0.5.11.jar:/home/jxke/.ivy2/cache/org.objenesis/objenesis/jars/objenesis-2.1.jar:/home/jxke/.ivy2/cache/org.mortbay.jetty/jetty-util/jars/jetty-util-6.1.26.jar:/home/jxke/.ivy2/cache/org.json4s/json4s-jackson_2.11/jars/json4s-jackson_2.11-3.2.11.jar:/home/jxke/.ivy2/cache/org.json4s/json4s-core_2.11/jars/json4s-core_2.11-3.2.11.jar:/home/jxke/.ivy2/cache/org.json4s/json4s-ast_2.11/jars/json4s-ast_2.11-3.2.11.jar:/home/jxke/.ivy2/cache/org.javassist/javassist/bundles/javassist-3.18.1-GA.jar:/home/jxke/.ivy2/cache/org.glassfish.jersey.media/jersey-media-jaxb/jars/jersey-media-jaxb-2.22.2.jar:/home/jxke/.ivy2/cache/org.glassfish.jersey.core/jersey-server/jars/jersey-server-2.22.2.jar:/home/jxke/.ivy2/cache/org.glassfish.jersey.core/jersey-common/jars/jersey-common-2.22.2.jar:/home/jxke/.ivy2/cache/org.glassfish.jersey.core/jersey-client/jars/jersey-client-2.22.2.jar:/home/jxke/.ivy2/cache/org.glassfish.jersey.containers/jersey-container-servlet-core/jars/jersey-container-servlet-core-2.22.2.jar:/home/jxke/.ivy2/cache/org.glassfish.jersey.containers/jersey-container-servlet/jars/jersey-container-servlet-2.22.2.jar:/home/jxke/.ivy2/cache/org.glassfish.jersey.bundles.repackaged/jersey-guava/bundles/jersey-guava-2.22.2.jar:/home/jxke/.ivy2/cache/org.glassfish.hk2.external/javax.inject/jars/javax.inject-2.4.0-b34.jar:/home/jxke/.ivy2/cache/org.glassfish.hk2.external/aopalliance-repackaged/jars/aopalliance-repackaged-2.4.0-b34.jar:/home/jxke/.ivy2/cache/org.glassfish.hk2/osgi-resource-locator/jars/osgi-resource-locator-1.0.1.jar:/home/jxke/.ivy2/cache/org.glassfish.hk2/hk2-utils/jars/hk2-utils-2.4.0-b34.jar:/home/jxke/.ivy2/cache/org.glassfish.hk2/hk2-locator/jars/hk2-locator-2.4.0-b34.jar:/home/jxke/.ivy2/cache/org.glassfish.hk2/hk2-api/jars/hk2-api-2.4.0-b34.jar:/home/jxke/.ivy2/cache/org.fusesource.leveldbjni/leveldbjni-all/bundles/leveldbjni-all-1.8.jar:/home/jxke/.ivy2/cache/org.codehaus.jackson/jackson-mapper-asl/jars/jackson-mapper-asl-1.9.13.jar:/home/jxke/.ivy2/cache/org.codehaus.jackson/jackson-core-asl/jars/jackson-core-asl-1.9.13.jar:/home/jxke/.ivy2/cache/org.apache.zookeeper/zookeeper/jars/zookeeper-3.4.5.jar:/home/jxke/.ivy2/cache/org.apache.xbean/xbean-asm5-shaded/bundles/xbean-asm5-shaded-4.4.jar:/home/jxke/.ivy2/cache/org.apache.spark/spark-unsafe_2.11/jars/spark-unsafe_2.11-2.1.2.jar:/home/jxke/.ivy2/cache/org.apache.spark/spark-tags_2.11/jars/spark-tags_2.11-2.1.2.jar:/home/jxke/.ivy2/cache/org.apache.spark/spark-network-shuffle_2.11/jars/spark-network-shuffle_2.11-2.1.2.jar:/home/jxke/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-2.1.2.jar:/home/jxke/.ivy2/cache/org.apache.spark/spark-launcher_2.11/jars/spark-launcher_2.11-2.1.2.jar:/home/jxke/.ivy2/cache/org.apache.spark/spark-core_2.11/jars/spark-core_2.11-2.1.2.jar:/home/jxke/.ivy2/cache/org.apache.ivy/ivy/jars/ivy-2.4.0.jar:/home/jxke/.ivy2/cache/org.apache.hadoop/hadoop-yarn-server-common/jars/hadoop-yarn-server-common-2.2.0.jar:/home/jxke/.ivy2/cache/org.apache.hadoop/hadoop-yarn-common/jars/hadoop-yarn-common-2.2.0.jar:/home/jxke/.ivy2/cache/org.apache.hadoop/hadoop-yarn-client/jars/hadoop-yarn-client-2.2.0.jar:/home/jxke/.ivy2/cache/org.apache.hadoop/hadoop-yarn-api/jars/hadoop-yarn-api-2.2.0.jar:/home/jxke/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-shuffle/jars/hadoop-mapreduce-client-shuffle-2.2.0.jar:/home/jxke/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-jobclient/jars/hadoop-mapreduce-client-jobclient-2.2.0.jar:/home/jxke/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-core/jars/hadoop-mapreduce-client-core-2.2.0.jar:/home/jxke/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-common/jars/hadoop-mapreduce-client-common-2.2.0.jar:/home/jxke/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/jars/hadoop-mapreduce-client-app-2.2.0.jar:/home/jxke/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/jars/hadoop-hdfs-2.2.0.jar:/home/jxke/.ivy2/cache/org.apache.hadoop/hadoop-common/jars/hadoop-common-2.2.0.jar:/home/jxke/.ivy2/cache/org.apache.hadoop/hadoop-client/jars/hadoop-client-2.2.0.jar:/home/jxke/.ivy2/cache/org.apache.hadoop/hadoop-auth/jars/hadoop-auth-2.2.0.jar:/home/jxke/.ivy2/cache/org.apache.hadoop/hadoop-annotations/jars/hadoop-annotations-2.2.0.jar:/home/jxke/.ivy2/cache/org.apache.curator/curator-recipes/bundles/curator-recipes-2.4.0.jar:/home/jxke/.ivy2/cache/org.apache.curator/curator-framework/bundles/curator-framework-2.4.0.jar:/home/jxke/.ivy2/cache/org.apache.curator/curator-client/bundles/curator-client-2.4.0.jar:/home/jxke/.ivy2/cache/org.apache.commons/commons-math3/jars/commons-math3-3.4.1.jar:/home/jxke/.ivy2/cache/org.apache.commons/commons-math/jars/commons-math-2.1.jar:/home/jxke/.ivy2/cache/org.apache.commons/commons-lang3/jars/commons-lang3-3.5.jar:/home/jxke/.ivy2/cache/org.apache.commons/commons-crypto/jars/commons-crypto-1.0.0.jar:/home/jxke/.ivy2/cache/org.apache.commons/commons-compress/jars/commons-compress-1.4.1.jar:/home/jxke/.ivy2/cache/org.apache.avro/avro-mapred/jars/avro-mapred-1.7.7-hadoop2.jar:/home/jxke/.ivy2/cache/org.apache.avro/avro-ipc/jars/avro-ipc-1.7.7-tests.jar:/home/jxke/.ivy2/cache/org.apache.avro/avro-ipc/jars/avro-ipc-1.7.7.jar:/home/jxke/.ivy2/cache/org.apache.avro/avro/jars/avro-1.7.7.jar:/home/jxke/.ivy2/cache/net.sf.py4j/py4j/jars/py4j-0.10.4.jar:/home/jxke/.ivy2/cache/net.razorvine/pyrolite/jars/pyrolite-4.13.jar:/home/jxke/.ivy2/cache/net.jpountz.lz4/lz4/jars/lz4-1.3.0.jar:/home/jxke/.ivy2/cache/net.java.dev.jets3t/jets3t/jars/jets3t-0.7.1.jar:/home/jxke/.ivy2/cache/log4j/log4j/bundles/log4j-1.2.17.jar:/home/jxke/.ivy2/cache/javax.ws.rs/javax.ws.rs-api/jars/javax.ws.rs-api-2.0.1.jar:/home/jxke/.ivy2/cache/javax.validation/validation-api/jars/validation-api-1.1.0.Final.jar:/home/jxke/.ivy2/cache/javax.servlet/javax.servlet-api/jars/javax.servlet-api-3.1.0.jar:/home/jxke/.ivy2/cache/javax.inject/javax.inject/jars/javax.inject-1.jar:/home/jxke/.ivy2/cache/javax.annotation/javax.annotation-api/jars/javax.annotation-api-1.2.jar:/home/jxke/.ivy2/cache/io.netty/netty-all/jars/netty-all-4.0.43.Final.jar:/home/jxke/.ivy2/cache/io.netty/netty/bundles/netty-3.8.0.Final.jar:/home/jxke/.ivy2/cache/io.dropwizard.metrics/metrics-jvm/bundles/metrics-jvm-3.1.2.jar:/home/jxke/.ivy2/cache/io.dropwizard.metrics/metrics-json/bundles/metrics-json-3.1.2.jar:/home/jxke/.ivy2/cache/io.dropwizard.metrics/metrics-graphite/bundles/metrics-graphite-3.1.2.jar:/home/jxke/.ivy2/cache/io.dropwizard.metrics/metrics-core/bundles/metrics-core-3.1.2.jar:/home/jxke/.ivy2/cache/commons-net/commons-net/jars/commons-net-2.2.jar:/home/jxke/.ivy2/cache/commons-lang/commons-lang/jars/commons-lang-2.5.jar:/home/jxke/.ivy2/cache/commons-io/commons-io/jars/commons-io-2.1.jar:/home/jxke/.ivy2/cache/commons-httpclient/commons-httpclient/jars/commons-httpclient-3.1.jar:/home/jxke/.ivy2/cache/commons-digester/commons-digester/jars/commons-digester-1.8.jar:/home/jxke/.ivy2/cache/commons-configuration/commons-configuration/jars/commons-configuration-1.6.jar:/home/jxke/.ivy2/cache/commons-collections/commons-collections/jars/commons-collections-3.2.1.jar:/home/jxke/.ivy2/cache/commons-cli/commons-cli/jars/commons-cli-1.2.jar:/home/jxke/.ivy2/cache/commons-beanutils/commons-beanutils-core/jars/commons-beanutils-core-1.8.0.jar:/home/jxke/.ivy2/cache/commons-beanutils/commons-beanutils/jars/commons-beanutils-1.7.0.jar:/home/jxke/.ivy2/cache/com.twitter/chill_2.11/jars/chill_2.11-0.8.0.jar:/home/jxke/.ivy2/cache/com.twitter/chill-java/jars/chill-java-0.8.0.jar:/home/jxke/.ivy2/cache/com.thoughtworks.paranamer/paranamer/jars/paranamer-2.6.jar:/home/jxke/.ivy2/cache/com.ning/compress-lzf/bundles/compress-lzf-1.0.3.jar:/home/jxke/.ivy2/cache/com.google.protobuf/protobuf-java/bundles/protobuf-java-2.5.0.jar:/home/jxke/.ivy2/cache/com.google.inject/guice/jars/guice-3.0.jar:/home/jxke/.ivy2/cache/com.google.guava/guava/bundles/guava-14.0.1.jar:/home/jxke/.ivy2/cache/com.google.code.findbugs/jsr305/jars/jsr305-1.3.9.jar:/home/jxke/.ivy2/cache/com.fasterxml.jackson.module/jackson-module-scala_2.11/bundles/jackson-module-scala_2.11-2.6.5.jar:/home/jxke/.ivy2/cache/com.fasterxml.jackson.module/jackson-module-paranamer/bundles/jackson-module-paranamer-2.6.5.jar:/home/jxke/.ivy2/cache/com.fasterxml.jackson.core/jackson-databind/bundles/jackson-databind-2.6.5.jar:/home/jxke/.ivy2/cache/com.fasterxml.jackson.core/jackson-core/bundles/jackson-core-2.6.5.jar:/home/jxke/.ivy2/cache/com.fasterxml.jackson.core/jackson-annotations/bundles/jackson-annotations-2.6.5.jar:/home/jxke/.ivy2/cache/com.esotericsoftware/minlog/bundles/minlog-1.3.0.jar:/home/jxke/.ivy2/cache/com.esotericsoftware/kryo-shaded/bundles/kryo-shaded-3.0.3.jar:/home/jxke/.ivy2/cache/com.clearspring.analytics/stream/jars/stream-2.7.0.jar:/home/jxke/.ivy2/cache/com.univocity/univocity-parsers/jars/univocity-parsers-2.2.1.jar:/home/jxke/.ivy2/cache/commons-codec/commons-codec/jars/commons-codec-1.10.jar:/home/jxke/.ivy2/cache/org.antlr/antlr4-runtime/jars/antlr4-runtime-4.5.3.jar:/home/jxke/.ivy2/cache/org.apache.parquet/parquet-column/jars/parquet-column-1.8.1.jar:/home/jxke/.ivy2/cache/org.apache.parquet/parquet-common/jars/parquet-common-1.8.1.jar:/home/jxke/.ivy2/cache/org.apache.parquet/parquet-encoding/jars/parquet-encoding-1.8.1.jar:/home/jxke/.ivy2/cache/org.apache.parquet/parquet-format/jars/parquet-format-2.3.0-incubating.jar:/home/jxke/.ivy2/cache/org.apache.parquet/parquet-hadoop/jars/parquet-hadoop-1.8.1.jar:/home/jxke/.ivy2/cache/org.apache.parquet/parquet-jackson/jars/parquet-jackson-1.8.1.jar:/home/jxke/.ivy2/cache/org.apache.spark/spark-catalyst_2.11/jars/spark-catalyst_2.11-2.1.2.jar:/home/jxke/.ivy2/cache/org.apache.spark/spark-sketch_2.11/jars/spark-sketch_2.11-2.1.2.jar:/home/jxke/.ivy2/cache/org.apache.spark/spark-sql_2.11/jars/spark-sql_2.11-2.1.2.jar:/home/jxke/.ivy2/cache/org.codehaus.janino/commons-compiler/jars/commons-compiler-3.0.0.jar:/home/jxke/.ivy2/cache/org.codehaus.janino/janino/jars/janino-3.0.0.jar org.apache.spark.examples.SparkPi
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/01/04 23:43:39 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/01/04 23:43:40 WARN Utils: Your hostname, ubuntu resolves to a loopback address: 127.0.1.1; using 10.0.2.15 instead (on interface enp0s3)
19/01/04 23:43:40 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
  Pi is roughly 3.139035695178476

Process finished with exit code 0

執行結果