Introduction to Apache Spark for Data Science | Analyzing Big Data with Spark

Course Objectives

This course is approximately 50% hands-on, combining expert lecture, real-world demonstrations and group discussions with machine-based practical labs and exercises.  Working in a hands-on learning environment led by our expert practitioner students will explore:

• Spark Essentials
• DataFrames
• Spark SQL
• Spark MLib
• Spark Streaming
• Streaming with Kafka
• Data Flow with NiFi
• Spark GraphX
• Performance and Tuning
• Cluster Mode
• Spark - the Big Picture

Need different skills or topics?  If your team requires different topics or tools, additional skills or custom approach, this course may be easily adjusted to accommodate.  We offer additional related Spark, Hadoop, data science, programming and development courses which may be blended with this course for a track that best suits your development objectives. Our team will collaborate with you to understand your needs and will target the course to focus on your specific learning objectives and goals.

Course Prerequisites

Course Agenda

Please note that this list of topics is based on our standard course offering, evolved from typical industry uses and trends. We’ll work with you to tune this course and level of coverage to target the skills you need most.

Getting Started

• Our Data and our problem set
• Accessing the cluster, the data, and the tools
• The Continuous Workshop approach
• "Let's build a model together"
• Focus on analysis, exploration, data munging, algorithms
• Tooling and fundamentals as necessary to get the job done

Spark Overview

• Data Science: The State of the Art
• Hadoop, Yarn, and Spark
• Architectural Overview
• MLib Overview
• HDFS data - Accessing
• Lab Focus
• Working with HDFS data
• Distributed vs. Local Run Modes
• Spark vs. Other tools (when is Spark the right tool for the job?)
• Spark vs. SAS
• Spark Languages (Java, R, Python, and Scala)
• Hello, Spark

Spark Essentials

• Spark Core
• Spark SQL
• Spark and Hive
• Lab
• MLib
• Spark Streaming
• Spark API

DataFrames

• DataFrames and Resilient Distributed Datasets (RDDs)
• Partitions
• Adding variables to a DataFrame
• DataFrame Types
• DataFrame Operations
• Dependent vs. Independent variables
• Map/Reduce with DataFrames

Spark SQL

• Spark SQL Overview
• Data stores: HDFS, Cassandra, HBase, Hive, and S3
• Table Definitions
• Queries

Spark MLib

• MLib overview
• MLib Algorithms Overview
• Classification Algorithms
• Regression Algorithms
• Lab Focus
• Brief Comparison to SAS
• Here's your split, how to tune regression
• Decision Trees and forests
• Lab Focus
• Brief Comparison to SAS
• Stepwise approach to Decision Trees
• Working with Exit Criteria
• Recommendation with ALS
• Clustering Algorithms
• Lab Focus
• Key Clustering Algorithms
• Choosing Clustering Algorithms
• Working with key algorithms
• Machine Learning Pipelines
• Linear Algebra (SVD, PCA)
• Statistics in MLib

Spark Streaming

• Streaming overview
• Real-time data ingestion
• State
• Window Operations

Streaming with Kafka

• Kafka overview
• Kafka and Spark Streaming

Data Flow with NiFi

• Apache NiFi overview
• NiFi data flows with Spark/R

Spark GraphX

• GraphX overview
• ETL with GraphX
• Graph computation

Performance and Tuning

• Broadcast variables
• Accumulators
• Memory Management

Cluster Mode

• Standalone Cluster
• Masters and Workers
• Configurations
• Working with large data sets

Spark - the Big Picture

• Spark in Real-Time and near-Real-Time Decision Support Systems
• Spark in the Enterprise
• Best Practices

Course Materials

Our course materials include more than a simple slideshow presentation handout. Each student will receive a comprehensive course Student Guide, complete with detailed course notes, code samples, software tutorials, diagrams and related reference materials and links. Our courses also include detailed our Student Workbook, with step by step hands-on lab instructions and project files (as necessary) and solutions, clearly illustrated for users to complete hands-on work in class, and to revisit to review or refresh skills at any time.  Students will also receive the course set up files, project files (or code, if applicable) and solutions required for the hands-on work.