Posts

Showing posts from May, 2017

A simple and easy Spark standalone program for beginners.

A simple and easy Spark standalone program for beginners....

Courtsey: Hariprasad Bhaskaran


Aim :
To demonstrate writing a simple spark scala program
use sbt to build it
create dependencies for spark-core and spark-sql in build.sbt
create a final jar that can be submitted using spark-submit...


Program code below:


MyAdd2.scala
=================

import org.apache.spark.SparkContext
import org.apache.spark.SparkConf
import org.apache.spark.sql._
import org.apache.spark.sql.hive.HiveContext
import org.apache.spark.sql.hive._
import org.apache.spark.sql.SQLContext
import org.apache.spark.sql.SQLContext._


object MyAdd2
{

def main(args: Array[String] )
{

val conf = new SparkConf().setAppName("t1").setMaster("local")
val sc2 = new SparkContext(conf)
val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc2)

import sqlContext.implicits._

val mylist = sc2.parallelize(List( 1 ,2,3,4,5,6,7,8,9,100) )
val mylistDF = mylist.map( x => (x,1) ).toDF("vals","…