Spark高级数据分析（影印版 英文版）

在《Spark高级数据分析（影印版 英文版）》中，4位Cloude阳公司的数据科学家讲解了一系列自包含模式，用于在Spark中进行大规模数据分析。《Spark高级数据分析（影印版 英文版）》作者们把Spark、统计原理和现实世界中的数据集合放到一起，通过实例教你如何解决数据分析问题。
你将从Spark及其生态系统的介绍开始，然后深入运用标准技巧的模式——归类、聚合过滤及异常检测等，这些技巧被用于生物基因、安全和金融等行业。如果你对机器学习和统计学有初步了解，使用Java、Pytton或者Scala编程，就会发现这些模式对于你的数据分析应用程序会非常有用。
模式包括： 音乐推荐和Audioscrobbler数据集合 用决策树分析森林覆盖 用K均值聚合检测网络流量中的异常 用潜在语义分析理解维基百科 用GraphX分析共生网络 用地理空间和瞬态数据分析纽约市出租车路线的数据 用蒙地卡罗模拟来估计金融风险 分析基因数据和BDG项目 通过PySpark和Thunder分析神经造影数据

Foreword
Preface
1． Analyzing Big Data
The Challenges of Data Science
Introducing Apache Spark
About This Book

2． Introduction to Data Analysis with Scala and Spark
Scala for Data Scientists
The Spark Programming Model
Record Linkage
Getting Started： The Spark Shell and SparkContext
Bringing Data from the Cluster to the Client
Shipping Code from the Client to the Cluster
Structuring Data with Tuples and Case Classes
Aggregations
Creating Histograms
Summary Statistics for Continuous Variables
Creating Reusable Code for Computing Summary Statistics
Simple Variable Selection and Scoring
Where to Go from Here

3． Recommending Music and the Audioscrobbler Data Set
Data Set
The Alternating Least Squares Recommender Algorithm
Preparing the Data
Building a First Model
Spot Checking Recommendations
Evaluating Recommendation Quality
Computing AUC
Hyperparameter Selection
Making Recommendations
Where to Go from Here

4． Predicting Forest Cover with Decision Trees
Fast Forward to Regression
Vectors and Features
Training Examples
Decision Trees and Forests
Covtype Data Set
Preparing the Data
A First Decision Tree
Decision Tree Hyperparameters
Tuning Decision Trees
Categorical Features Revisited
Random Decision Forests
Making Predictions
Where to Go from Here

5． Anomaly Detection in Network Traffic with K-means Clustering
Anomaly Detection
K-means Clustering
Network Intrusion
KDD Cup 1999 Data Set
A First Take on Clustering
Choosing k
Visualization in R
Feature Normalization
Categorical Variables
Using Labels with Entropy
Clustering in Action
Where to Go from Here

6． Understanding Wikipedia with Latent Semantic Analysis
The Term-Document Matrix
Getting the Data
Parsing and Preparing the Data
Lemmatization
Computing the TF-IDFs
Singular Value Decomposition
Finding Important Concepts
Querying and Scoring with the Low-Dimensional Representation
Term-Term Relevance
Document-Document Relevance
Term-Document Relevance
Multiple-Term Queries
Where to Go from Here

7． Analyzing Co-occurrence Networks with GraphX
The MEDLINE Citation Index： A Network Analysis
Getting the Data
Parsing XML Documents with Scala's XML Library
Analyzing the MeSH Major Topics and Their Co-occurrences
Constructing a Co-occurrence Network with GraphX
Understanding the Structure of Networks
Connected Components
Degree Distribution
Filtering Out Noisy Edges
Processing EdgeTriplets
Analyzing the Filtered Graph
Small-World Networks
Cliques and Clustering Coefficients
Computing Average Path Length with Pregel
Where to Go from Here

8． 6eospatial and Temporal Data Analysis on the New York City Taxi Trip Data
Getting the Data
Working with Temporal and Geospatial Data in Spark
Temporal Data with JodaTime and NScalaTime
Geospatial Data with the Esri Geometry API and Spray
Exploring the Esri Geometry API
Intro to GeoJSON
Preparing the New York City Taxi Trip Data
Handling Invalid Records at Scale
Geospatial Analysis
Sessionization in Spark
Building Sessions： Secondary Sorts in Spark
Where to Go from Here

9． Estimating Financial Risk through Monte Carlo Simulation
Terminology
Methods for Calculating VaR
Variance-Covariance
Historical Simulation
Monte Carlo Simulation
Our Model
Getting the Data
Preprocessing
Determining the Factor Weights
Sampling
The Multivariate Normal Distribution
Running the Trials
Visualizing the Distribution of Returns
Evaluating Our Results
Where to Go from Here

10． Analyzing Genomics Data and the BDG Project
Decoupling Storage from Modeling
Ingesting Genomics Data with the ADAM CLI
Parquet Format and Columnar Storage
Predicting Transcription Factor Binding Sites from ENCODE Data
Querying Genotypes from the 1000 Genomes Project
Where to Go from Here

11． Analyzing Neuroimaging Data with PySpark and Thunder
Overview of PySpark
PySpark Internals
Overview and Installation of the Thunder Library
Loading Data with Thunder
Thunder Core Data Types
Categorizing Neuron Types with Thunder
Where to Go from Here

A．Deeper into Spark
B．Upcoming MLlib Pipelines API
Index

《Spark高级数据分析（影印版 英文版）》：
Every record in our RDD will have its own Array（Double）， which we can translate into an RDD where each record is an Array（NAStatCounter）.Let's go ahead and do that now against the data in the parsed RDD on the cluster：
val nasRDD = parsed.map（md => {
md.scores.map（d => NAStatCounter（d））
}）
We now need an easy way to aggregate multiple instances of Array（NAStatCounter） into a single Array（NAStatCounter）.We can combine two arrays of the same length using zip.This produces a new Array of the corresponding pairs of elements in the two arrays.Think of a zipper pairing up two corresponding strips of teeth into one fastened strip ofinterlocked teeth.This can be followed by a map method that uses the merge function on the NAStatCounter class to combine the statistics from both objects into a single instance：
val nas1 = Array（l.0， Double.NaN）.rnap（d => NAStatCounter（d））
val nas2 = Array（Double.NaN， 2.0）.map（d => NAStatCounter（d））
val merged = nas1.zip（nas2）.map（p => p._1.rnerge（p._2））
We can even use Scala's case syntax to break the pair of elements in the zipped array into nicely named variables， instead of using the _1 and _2 methods on the Tuple2 class：
val merged = nasl.zip（nas2）.map { case （a， b） => a.merge（b） }
To perform this same merge operation across all of the records in a Scala collection，we can use the reduce function， which takes an associative function that maps two arguments of type T into a single return value of type T and applies it over and over again to all of the elements in a collection to merge all of the values together.
……