Mastering Scala with Apache Spark for the Modern Data Enterprise (TTSK7520)

Getting Started with Scala and Spark

1. Introduction to Scala

• Brief history and motivation
• Differences between Scala and Java
• Basic Scala syntax and constructs
• Scala's functional programming features

2. Introduction to Apache Spark

• Overview and history
• Spark components and architecture
• Spark ecosystem
• Comparing Spark with other big data frameworks
• Lab: Practice basic Scala syntax and functional programming concepts using the REPL.
• Setting up the Development Environment

3. Basics of Spark Programming SparkContext and SparkSession

• Resilient Distributed Datasets (RDDs)
• Transformations and Actions
• Working with DataFrames

4. Spark SQL and Data Sources

• Spark SQL library and its advantages
• Structured and semi-structured data sources
• Reading and writing data in various formats (CSV, JSON, Parquet, Avro, etc.)
• Data manipulation using SQL queries
• Lab: Setting up the Environment and Running a Simple Spark Application.
• Lab: Load and query data from different data sources using Spark SQL.

Data Processing and Spark Programming

5. Basic RDD Operations

• Creating and manipulating RDDs
• Common transformations and actions on RDDs
• Working with key-value data

6. Basic DataFrame and Dataset Operations

• Creating and manipulating DataFrames and Datasets
• Column operations and functions
• Filtering, sorting, and aggregating data
• Lab: RDD and DataFrame Operations

7. Introduction to Spark Streaming

• Overview of Spark Streaming
• Discretized Stream (DStream) operations
• Windowed operations and stateful processing

8. Performance Optimization Basics

• Best practices for efficient Spark code
• Broadcast variables and accumulators
• Monitoring Spark applications

9. Integrating External Libraries and Tools, Spark Streaming

• Using popular external libraries, such as Hadoop and HBase
• Integrating with cloud platforms: AWS, Azure, GCP
• Connecting to data storage systems: HDFS, S3, Cassandra, etc.
• Lab: Building an End-to-End Spark Application: Create a Spark application to process a large dataset, perform aggregations, and save the results.
• Lab: Implement a simple Spark Streaming application to process real-time data.

Machine Learning Basics with Spark MLlib

10. Introduction to Machine Learning Basics

• Overview of machine learning
• Supervised and unsupervised learning
• Common algorithms and use cases

11. Introduction to Spark MLlib

• Overview of Spark MLlib
• MLlib's algorithms and utilities
• Data preparation and feature extraction
• Lab: Data Preparation with Spark MLlib

12. Linear Regression and Classification

• Linear regression algorithm
• Logistic regression for classification
• Model evaluation and performance metrics

13. Clustering Algorithms

• Overview of clustering algorithms
• K-means clustering
• Model evaluation and performance metrics

14. Collaborative Filtering and Recommendation Systems

• Overview of recommendation systems
• Collaborative filtering techniques
• Implementing recommendations with Spark MLlib
• Lab: Basic Machine Learning with Spark MLlib
• Lab: Implementing a Recommendation System

Graph Processing and Generative AI Technologies

15. Introduction to Graph Processing

• Overview of graph processing
• Use cases and applications of graph processing
• Graph representations and operations
• Introduction to Spark GraphX
• Overview of GraphX
• Creating and transforming graphs
• Graph algorithms in GraphX
• Lab: Graph Processing with GraphX

16. Big Data Innovation! Using GPT and Generative AI Technologies with Spark and Scala

• Overview of generative AI technologies
• Integrating GPT with Spark and Scala
• Practical applications and use cases

Bonus Topics / Time Permitting

17. Introduction to Spark NLP

• Overview of Spark NLP
• Preprocessing text data
• Text classification and sentiment analysis
• Lab: Generative AI Technologies and Spark NLP: Integrate GPT for text generation in a Spark application and explore basic NLP tasks using Spark NLP.
• Lab: Text Classification with Spark NLP

18. Putting It All Together

• Work on a capstone project that integrates multiple aspects of the course, including data processing, machine learning, graph processing, and generative AI technologies.