Example Source: Advanced Analytics with Spark

source: https://github.com/sryza/aas

Advanced Analytics with Spark
목차:
1. 오디오스크로블로 데이터셋으로 음악 추천하기
2. 의사 결정 나무로 산림 식샐 분포 예측하기
3. K-평균 군집화로 네트워크 이상 감지하기
4. 숨은 의미 분석으로 위키 백과 이해하기
5. 그래프엑스로 동시 발생 네트워크 분석하기
6. 뉴욕 택시 운행 데이터로 위치 및 시간 데이터 분석하기
7. 몬테카를로 시뮬레이션으로 금융 리스크 추정하기
8. BDG 프로젝트와 유전체학 데이터 분석하기
9. PySpark와 Thunder로 신경 영상 데이터 분석하기.

Code to accompany Advanced Analytics with Spark from O'Reilly Media

•  291 commits
•  8 branches
•  2 releases
•  7 contributors

1.  Scala 91.8%
2.  Python 4.9%
3.  R 2.8%
4.  Shell 0.5%

ScalaPythonRShell

Find file

Latest commit e8754e0 2 days ago sryza committed on GitHub Fix LSA issues and harmonize with the text (#104)

	ch02-intro	Remove ref to Juliet as an author for 2nd edition	2 months ago
	ch03-recommender	Tiny change to match text changes from review	26 days ago
	ch04-rdf	Use getEstimatorParamMaps per reviewer suggestion	2 months ago
	ch05-kmeans	Remove ref to Juliet as an author for 2nd edition	2 months ago
	ch06-lsa	Fix LSA issues and harmonize with the text (#104)	2 days ago
	ch07-graph	Remove ref to Juliet as an author for 2nd edition	2 months ago
	ch08-geotime	Remove ref to Juliet as an author for 2nd edition	2 months ago
	ch09-risk	Remove kernel density implementation (#102)	3 days ago
	ch10-genomics	Remove ref to Juliet as an author for 2nd edition	2 months ago
	ch11-neuro	Remove ref to Juliet as an author for 2nd edition	2 months ago
	simplesparkproject	Use Spark 2.1 in example project	12 days ago
	.gitignore	Define intro module	2 years ago
	.travis.yml	Update deps, plugins (#91)	3 months ago
	LICENSE	Remove ref to Juliet as an author for 2nd edition	2 months ago
	README.md	Also revert author order to match book	a month ago
	pom.xml	Fix last-second typo	a month ago

 README.md

Advanced Analytics with Spark Source Code

Code to accompany Advanced Analytics with Spark, by Sandy Ryza, Uri Laserson, Sean Owen, and Josh Wills.

1st Edition (current)

The source to accompany the 1st edition may be found in the 1st-edition branch.

2nd Edition (coming H1 2017)

The source to accompany the 2nd edition is found in this, the default master branch.

Build

Apache Maven 3.2.5+ and Java 8+ are required to build. From the root level of the project, run mvn package to compile artifacts into target/ subdirectories beneath each chapter's directory.

Data Sets

• Chapter 2: https://archive.ics.uci.edu/ml/machine-learning-databases/00210/
• Chapter 3: http://www-etud.iro.umontreal.ca/~bergstrj/audioscrobbler_data.html
• Chapter 4: https://archive.ics.uci.edu/ml/machine-learning-databases/covtype/
• Chapter 5: https://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html (do not use http://www.sigkdd.org/kdd-cup-1999-computer-network-intrusion-detection as the copy has a corrupted line)
• Chapter 6: https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles-multistream.xml.bz2
• Chapter 7: ftp://ftp.nlm.nih.gov/nlmdata/sample/medline/ (*.gz)
• Chapter 8: http://www.andresmh.com/nyctaxitrips/
• Chapter 9: (see ch09-risk/data/download-all-symbols.sh script)
• Chapter 10: ftp://ftp.ncbi.nih.gov/1000genomes/ftp/phase3/data/HG00103/alignment/HG00103.mapped.ILLUMINA.bwa.GBR.low_coverage.20120522.bam
• Chapter 11: https://github.com/thunder-project/thunder/tree/v0.4.1/python/thunder/utils/data/fish/tif-stack

Applied Data Mining and Statistical Learning-Analysis of German Credit Data

source: https://onlinecourses.science.psu.edu/stat857/node/215

Analysis of German Credit Data

Printer-friendly versionData mining is a critical step in knowledge discovery involving theories, methodologies and tools for revealing patterns in data. It is important to understand the rationale behind the methods so that tools and methods have appropriate fit with the data and the objective of pattern recognition. There may be several options for tools available for a data set.
When a bank receives a loan application, based on the applicant’s profile the bank has to make a decision regarding whether to go ahead with the loan approval or not. Two types of risks are associated with the bank’s decision –

• If the applicant is a good credit risk, i.e. is likely to repay the loan, then not approving the loan to the person results in a loss of business to the bank
• If the applicant is a bad credit risk, i.e. is not likely to repay the loan, then approving the loan to the person results in a financial loss to the bank

Objective of Analysis:

Minimization of risk and maximization of profit on behalf of the bank.

To minimize loss from the bank’s perspective, the bank needs a decision rule regarding who to give approval of the loan and who not to. An applicant’s demographic and socio-economic profiles are considered by loan managers before a decision is taken regarding his/her loan application.
The German Credit Data contains data on 20 variables and the classification whether an applicant is considered a Good or a Bad credit risk for 1000 loan applicants. Here is a link to the German Credit data (right-click and "save as" ).  A predictive model developed on this data is expected to provide a bank manager guidance for making a decision whether to approve a loan to a prospective applicant based on his/her profiles.
Data Files for this case (right-click and "save as" ) :

• German Credit data - german_credit.csv
• Training dataset - Training50.csv
• Test dataset - Test.csv

The following analytical approaches are taken:

• Logistic regression: The response is binary (Good credit risk or Bad) and several predictors are available.
• Discriminant Analysis:
• Tree-based method and Random Forest

Sample R code for Reading a .csv file

• GCD.1 - Exploratory Data Analysis (EDA) and Data Pre-processing
• GCD.2 - Towards Building a Logistic Regression Model
• GCD.3 - Applying Discriminant Analysis
• GCD.4 - Applying Tree-Based Methods
• GCD.5 - Random Forest
• GCD.6 - Cost-Profit Consideration
• GCD - Appendix - Description of Dataset

///////////////////////////////////////////////////////////////////////////////////////////////////////

Lessons

• Lesson 1(a): Introduction to Data Mining
  • 1(a) .1 - What is Data Mining?
  • 1(a) .2 - Examples of Data Mining Applications
  • 1(a) .3 - The Nature of The Problem
  • 1(a) .4 - Terminology
  • 1(a) .5 - Classification Problems in Real Life
  • 1(a) .6 - Outline of this Course - What Topics Will Follow?
• Lesson 1 (b): Exploratory Data Analysis (EDA)
• Lesson 2: Statistical Learning and Model Selection
• Lesson 3 : Linear Regression
• Lesson 4: Variable Selection
• Lesson 5: Regression Shrinkage Methods
• Lesson 6: Principal Components Analysis
• Lesson 7: Dimension Reduction Methods
• Lesson 8: Modeling Non-linear Relationships
• Lesson 9: Classification
• Lesson 10: Support Vector Machines
• Lesson 11: Tree-based Methods
• Lesson 12: Cluster Analysis

Resources

• Analysis of German Credit Data
• Analysis of Wine Quality Data
• Analysis of Classification Data
• Final Project - Sample Work