스파크 관련 핵심 개념과 활용사례 참조

<스파크 관련 핵심 개념과 활용사례 참조>

http://www.comworld.co.kr/news/articleView.html?idxno=47869
http://www.slideshare.net/rxin/stanford-cs347-guest-lecture-apache-spark
https://databricks.com/blog/2014/11/05/spark-officially-sets-a-new-record-in-large-scale-sorting.html
http://spark.apache.org



https://spark-summit.org/east-2017/events/what-to-expect-for-big-data-and-apache-spark-in-2017
https://www.youtube.com/watch?v=kmrWkU0PCCs
https://spark-summit.org/east-2017/events/what-to-expect-for-big-data-and-apache-spark-in-2017

<스파크 아키텍처>

http://spark.apache.org/docs/latest/cluster-overview.html


http://www-bcf.usc.edu/~minlanyu/teach/csci599-fall12/papers/nsdi_spark.pdf
https://databricks.com/blog/2015/06/22/understanding-your-spark-application-through-visualization.html
courses.csail.mit.edu/18.337/2015/docs/6338.pptx

<스파크 서밋 서베이>
http://go.databricks.com/hubfs/DataBricks_Surveys_-_Content/Spark-Survey-2015-Infographic.pdf
http://cdn2.hubspot.net/hubfs/438089/DataBricks_Surveys_-_Content/2016_Spark_Survey/2016_Spark_Infographic.pdf

<활용사례>
http://cdn2.hubspot.net/hubfs/438089/DataBricks_Surveys_-_Content/2016_Spark_Survey/2016_Spark_Infographic.pdf
ING : https://conferences.oreilly.com/strata/strata-ny-2016/public/schedule/detail/51013
Goldman Sachs: http://www.slideshare.net/SparkSummit/how-spark-is-making-an-impact-at-goldman-sachs-by-vincent-saulys
baidu: http://www.slideshare.net/SparkSummit/how-spark-fits-into-baidus-scale-james-peng
Toyota: http://www.slideshare.net/SparkSummit/brian-kursar
ING: https://conferences.oreilly.com/strata/strata-ny-2016/public/schedule/detail/51013
Netflix : https://cdn.oreillystatic.com/en/assets/1/event/132/Netflix_%20Integrating%20Spark%20at%20petabyte%20scale%20Presentation.pdf
telefonica: https://spark-summit.org/2014/spark-use-case-at-telefonica-cbs/

Example Source: Advanced Analytics with Spark

source: https://github.com/sryza/aas

Advanced Analytics with Spark
목차:
1. 오디오스크로블로 데이터셋으로 음악 추천하기
2. 의사 결정 나무로 산림 식샐 분포 예측하기
3. K-평균 군집화로 네트워크 이상 감지하기
4. 숨은 의미 분석으로 위키 백과 이해하기
5. 그래프엑스로 동시 발생 네트워크 분석하기
6. 뉴욕 택시 운행 데이터로 위치 및 시간 데이터 분석하기
7. 몬테카를로 시뮬레이션으로 금융 리스크 추정하기
8. BDG 프로젝트와 유전체학 데이터 분석하기
9. PySpark와 Thunder로 신경 영상 데이터 분석하기.

Code to accompany Advanced Analytics with Spark from O'Reilly Media

•  291 commits
•  8 branches
•  2 releases
•  7 contributors

1.  Scala 91.8%
2.  Python 4.9%
3.  R 2.8%
4.  Shell 0.5%

ScalaPythonRShell

Find file

Latest commit e8754e0 2 days ago sryza committed on GitHub Fix LSA issues and harmonize with the text (#104)

	ch02-intro	Remove ref to Juliet as an author for 2nd edition	2 months ago
	ch03-recommender	Tiny change to match text changes from review	26 days ago
	ch04-rdf	Use getEstimatorParamMaps per reviewer suggestion	2 months ago
	ch05-kmeans	Remove ref to Juliet as an author for 2nd edition	2 months ago
	ch06-lsa	Fix LSA issues and harmonize with the text (#104)	2 days ago
	ch07-graph	Remove ref to Juliet as an author for 2nd edition	2 months ago
	ch08-geotime	Remove ref to Juliet as an author for 2nd edition	2 months ago
	ch09-risk	Remove kernel density implementation (#102)	3 days ago
	ch10-genomics	Remove ref to Juliet as an author for 2nd edition	2 months ago
	ch11-neuro	Remove ref to Juliet as an author for 2nd edition	2 months ago
	simplesparkproject	Use Spark 2.1 in example project	12 days ago
	.gitignore	Define intro module	2 years ago
	.travis.yml	Update deps, plugins (#91)	3 months ago
	LICENSE	Remove ref to Juliet as an author for 2nd edition	2 months ago
	README.md	Also revert author order to match book	a month ago
	pom.xml	Fix last-second typo	a month ago

 README.md

Advanced Analytics with Spark Source Code

Code to accompany Advanced Analytics with Spark, by Sandy Ryza, Uri Laserson, Sean Owen, and Josh Wills.

1st Edition (current)

The source to accompany the 1st edition may be found in the 1st-edition branch.

2nd Edition (coming H1 2017)

The source to accompany the 2nd edition is found in this, the default master branch.

Build

Apache Maven 3.2.5+ and Java 8+ are required to build. From the root level of the project, run mvn package to compile artifacts into target/ subdirectories beneath each chapter's directory.

Data Sets

• Chapter 2: https://archive.ics.uci.edu/ml/machine-learning-databases/00210/
• Chapter 3: http://www-etud.iro.umontreal.ca/~bergstrj/audioscrobbler_data.html
• Chapter 4: https://archive.ics.uci.edu/ml/machine-learning-databases/covtype/
• Chapter 5: https://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html (do not use http://www.sigkdd.org/kdd-cup-1999-computer-network-intrusion-detection as the copy has a corrupted line)
• Chapter 6: https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles-multistream.xml.bz2
• Chapter 7: ftp://ftp.nlm.nih.gov/nlmdata/sample/medline/ (*.gz)
• Chapter 8: http://www.andresmh.com/nyctaxitrips/
• Chapter 9: (see ch09-risk/data/download-all-symbols.sh script)
• Chapter 10: ftp://ftp.ncbi.nih.gov/1000genomes/ftp/phase3/data/HG00103/alignment/HG00103.mapped.ILLUMINA.bwa.GBR.low_coverage.20120522.bam
• Chapter 11: https://github.com/thunder-project/thunder/tree/v0.4.1/python/thunder/utils/data/fish/tif-stack

[SPARK] Programmatically Specifying the Schema

source: http://spark.apache.org/docs/latest/sql-programming-guide.html#creating-dataframes

Programmatically Specifying the Schema

1. Create an RDD of Rows from the original RDD;
2. Create the schema represented by a StructType matching the structure of Rows in the RDD created in Step 1.
3. Apply the schema to the RDD of Rows via createDataFrame method provided by SparkSession.

import org.apache.spark.sql.types._

// Create an RDD
val peopleRDD = spark.sparkContext.textFile("examples/src/main/resources/people.txt")

// The schema is encoded in a string
val schemaString = "name age"

// Generate the schema based on the string of schema
val fields = schemaString.split(" ")
  .map(fieldName => StructField(fieldName, StringType, nullable = true))
val schema = StructType(fields)

// Convert records of the RDD (people) to Rows
val rowRDD = peopleRDD
  .map(_.split(","))
  .map(attributes => Row(attributes(0), attributes(1).trim))

// Apply the schema to the RDD
val peopleDF = spark.createDataFrame(rowRDD, schema)

// Creates a temporary view using the DataFrame
peopleDF.createOrReplaceTempView("people")

// SQL can be run over a temporary view created using DataFrames
val results = spark.sql("SELECT name FROM people")

// The results of SQL queries are DataFrames and support all the normal RDD operations
// The columns of a row in the result can be accessed by field index or by field name
results.map(attributes => "Name: " + attributes(0)).show()
// +-------------+
// |        value|
// +-------------+
// |Name: Michael|
// |   Name: Andy|
// | Name: Justin|
// +-------------+

Spark 2.0 Technical Preview

source: http://tomining.tistory.com/124

Spark 2.0 Technical Preview

BigData/Spark

Databricks 에서 게제한 Spark 2.0 Technical Preview 글을 요약해 보았습니다.

spark 1.0 이 공개된 뒤 2년 만에 2.0 release 를 앞두고 있습니다.

Databricks 에서 공개한 Technical Preview 에서는 Spark 2.0의 3가지의 주요 특징을 소개하고 있습니다.

Easier, Faster, Smarter

• Easier
  • 표준 SQL 지원
    서브쿼리도 지원하는 새로운 Ansi-SQL 파서 적용
  • DataFrame/Dataset API 통합
    • Java/Scala 에서 DataFrame/Dataset 통합
    • SparkSession
      SQLConext 나 HiveContext 를 대체할 DataFrame API 를 위한 진입점
    • 좀 더 간단하고 성능 좋은 Accumlator API
    • 머신러닝 기반의 DataFrame
    • R 을 위한 분산 알고리즘
• Faster
  • 물리적 실행 영역을 다시 설계
    • CPU 낭비시간 해소
      • 가상함수 호출 시간
      • CPU cache 나 memory 에 데이터를 쓰고 읽는 시간
  • 10억 건을 집계/Join 한 결과
  • 
    
    
    
  • Parquet Scan 성능도 3배 이상 개선
• Smarter
  • Streaming engine 이상의 역할
    외부 저장 시스템(예, RDBMS) 과의 연계, 비즈니스 로직을 잘 처리하는 능력 등
    End-to End “Continuous application” (전체적인 흐름을 아우르는 Application)
  • Structured Streaming API + DataFrame/Dataset API
    실시간 데이터 분석을 가능

'BigData/Spark'에 해당되는 글 22건

1. 2016.05.15Spark Streaming Resiliency(자동복구) (2)
2. 2016.05.15Spark 2.0 Technical Preview
3. 2015.08.20Learning Spark Chapter. 10 Spark Streaming
4. 2015.07.31Learning Spark Chapter. 9 Spark SQL
5. 2015.07.23Learning Spark Chapter. 8 Tuning & Debugging
6. 2015.07.23Learning Spark Chapter. 7 Cluster 환경에서 수행하기
7. 2015.07.23Learning Spark Chapter. 6 Spark 프로그래밍 고급편
8. 2015.07.18Spark on YARN : Where have all the memory gone?
9. 2015.07.17RDD persist() or cache() 시 주의사항
10. 2015.07.17spark-submit 을 이용하여 Custom Argument 전달하기

아직은 Spark 2.0 이 preview package 이나 몇 주 내로 release 된다고 하니 기대가 됩니다.

Spark Streaming 과 DataFrame/Dataset API 를 잘 활용하면 실시간 분석을 쉽고 간단하게 할 수 있을 것 같습니다.