Showing posts from May, 2017

A simple and easy Spark standalone program for beginners.

A simple and easy Spark standalone program for beginners....

Courtsey: Hariprasad Bhaskaran

Aim :
To demonstrate writing a simple spark scala program
use sbt to build it
create dependencies for spark-core and spark-sql in build.sbt
create a final jar that can be submitted using spark-submit...

Program code below:


import org.apache.spark.SparkContext
import org.apache.spark.SparkConf
import org.apache.spark.sql._
import org.apache.spark.sql.hive.HiveContext
import org.apache.spark.sql.hive._
import org.apache.spark.sql.SQLContext
import org.apache.spark.sql.SQLContext._

object MyAdd2

def main(args: Array[String] )

val conf = new SparkConf().setAppName("t1").setMaster("local")
val sc2 = new SparkContext(conf)
val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc2)

import sqlContext.implicits._

val mylist = sc2.parallelize(List( 1 ,2,3,4,5,6,7,8,9,100) )
val mylistDF = x => (x,1) ).toDF("vals","…