Introduction to Apache Spark | Hands-on Spark for Big Data & Machine Learning

Course Objectives

Need different skills or topics?  If your team requires different topics or tools, additional skills or custom approach, this course may be further adjusted to accommodate.  We offer additional Spark, big data, data science, Hadoop, AI / machine learning / deep learning, programming, analytics,  and other related topics that may be blended with this course for a track that best suits your needs. Our team will collaborate with you to understand your needs and will target the course to focus on your specific learning objectives and goals.

Course Prerequisites

This foundation-level course is geared for intermediate skilled, experienced Developers and Architects (with basic Python experience) who seek to be proficient in advanced, modern development skills working with Apache Spark in an enterprise data environment.

Take Before: Students should have attended the course(s) below, or should have basic skills in these areas:

• TTPS4800      Introduction to Python Programming
• TTSQLB3        Introduction to SQL (Basic familiarity is needed, not in-depth SQL skills)

Please see the Related Courses tab for specific Pre-Requisite courses, Related Courses that offer similar skills or topics, and next-step Learning Path recommendations.

Course Agenda

Please note that this list of topics is based on our standard course offering, evolved from typical industry uses and trends. We’ll work with you to tune this course and level of coverage to target the skills you need most.

Spark Introduction

• Big data, Hadoop, Spark
• Spark concepts and architecture
• Spark components overview
• Labs: installing and running Spark

The first look at Spark

• Spark shell
• Spark web UIs
• Analyzing dataset – part 1
• Labs: Spark shell exploration

Spark Data structures

• Partitions
• Distributed execution
• Operations: transformations and actions
• Labs: Unstructured data analytics using RDDs

Caching

• Caching overview
• Various caching mechanisms available in Spark
• In memory file systems
• Caching use cases and best practices
• Labs: Benchmark of caching performance

DataFrames and Datasets

• DataFrames Intro
• Loading structured data (JSON, CSV) using DataFrames
• Using schema
• Specifying schema for DataFrames
• Labs: DataFrames, Datasets, Schema

Spark SQL

• Spark SQL concepts and overview
• Defining tables and importing datasets
• Querying data using SQL
• Handling various storage formats: JSON, Parquet, ORC
• Labs: querying structured data using SQL; evaluating data formats

Spark and Hadoop

• Hadoop Primer: HDFS, YARN
• Hadoop + Spark architecture
• Running Spark on Hadoop YARN
• Processing HDFS files using Spark
• Spark & Hive

Spark API

• Overview of Spark APIs in Scala / Python
• The lifecycle of a Spark application
• Spark APIs
• Deploying Spark applications on YARN
• Labs: Developing and deploying a Spark application

Spark ML Overview

• Machine Learning primer
• Machine Learning in Spark: MLib / ML
• Spark ML overview (newer Spark2 version)
• Algorithms overview: Clustering, Classifications, Recommendations
• Labs: Writing ML applications in Spark

GraphX

• GraphX library overview
• GraphX APIs
• Create a Graph and navigating it
• Shortest distance
• Pregel API
• Labs: Processing graph data using Spark

Time Permitting Topics

Spark Streaming

• Streaming concepts
• Evaluating Streaming platforms
• Spark streaming library overview
• Streaming operations
• Sliding window operations
• Structured Streaming
• Continuous streaming
• Spark & Kafka streaming
• Labs: Writing spark streaming applications

Workshop

• Attendees will work on solving real-world data analysis problems using Spark

Course Materials

Student Materials: Each participant will receive a Student Guide with course notes, code samples, software tutorials, step-by-step written lab instructions, diagrams and related reference materials and resource links. Students will also receive the project files (or code, if applicable) and solutions required for the hands-on work.

Hands-On Setup Made Simple! Our dedicated tech team will work with you to ensure our ‘easy-access’ cloud-based course environment is accessible, fully-tested and verified as ready to go well in advance of the course start date, ensuring a smooth start to class and effective learning experience for all participants. Please inquire for details and options.