Introduction to Kafka for Developers (TTDS6760)

Course Objectives

Working in a hands-on learning environment you’ll learn to:

• Implement and configure Apache Kafka effectively, demonstrating a deep understanding of its unique architecture, core concepts, and the differences between Kafka and other queue systems (JMS/MQ).
• Utilize Kafka APIs proficiently, including the Producer and Consumer APIs, and apply advanced techniques such as message compression, offset management, and Producer send modes.
• Design and develop streaming applications using the Kafka Streams API, performing complex operations like transformations, filters, joins, and aggregations, while working with KStream, KTable, and KStore concepts.
• Monitor and troubleshoot Kafka deployments, identifying performance bottlenecks, addressing common issues, and employing best practices for hardware selection, cluster sizing, partition sizing, and Zookeeper settings.
• Apply the skills and knowledge acquired throughout the course to real-world scenarios, showcasing the ability to develop, deploy, and optimize Kafka-based streaming applications for a variety of use cases.

Course Prerequisites

This course is geared for experienced Java Developers and architects with Java development background who are new to Kafka.  This course is not for non-developers.

In order to be successful in this course, and to participate in the hands-on labs, you should possess:

• Basic Java programming skills; practical Java development background.
• Reasonable experience working with databases
• Basic Linux skills and the ability to work from the Linux command line
• Basic knowledge of Linux editors (such as VI / nano) for editing code.

Take Before: Students should have practical skills equivalent to or should have attended the following course(s) as a pre-requisite:

• TT2104: Fast Track to Core Java Programming for OO Developers (C+, C#, etc.)

Next Steps / Follow-on Courses: We offer a wide variety of follow-on courses for next-level Kafka, Apache tools, data science, machine learning, DevOps, Java development skills and more. Please see our Learning Journey & Career Experience Paths for options based on your specific role and goals. 

Course Agenda

Introduction to Streaming Systems

• Understanding Fast data
• Streaming terminologies
• Understanding at-least-once / at-most-once / exactly-once processing patterns
• Popular streaming architectures
• Lambda architecture
• Streaming platforms overview
• Lab: Hands-on first look at Kafka

Introducing Kafka

• Comparing Kafka with other queue systems (JMS / MQ)
• Kafka Architecture
• Kaka concepts: Messages, Topics, Partitions, Brokers, Producers, commit logs
• Kafka & Zookeeper
• Producing messages
• Consuming messages
• Consumers, Consumer Groups
• Message retention
• Scaling Kafka
• Kafka across multiple data centers and disaster recovery
• Lab: Getting Kafka up and running
• Lab: Using Kafka utilities

Using Kafka APIs

• Configuration parameters
• Producer API - sending messages to Kafka
• Consumer API - consuming messages from Kafka
• Producer send modes
• Message compression
• Commits , Offsets, Seeking
• Managing offsets - auto commit / manual commit
• Lab: Writing Producer / Consumer
• Lab: Benchmarking Producer send modes
• Lab: Comparing compression schemes
• Lab: Managing offsets
• Lab: Clickstream processing

Kafka Streams API

• Introduction to Kafka Streams library
• Features and design
• Streams concepts: KStream / KTable / KStore
• Streaming operations (transformations, filters, joins, aggregations)
• Using Streams API: foreach / filter / map / groupby
• Lab: Kafka Streaming APIs

Monitoring & Troubleshooting Kafka

• Monitoring tools overview
• Monitoring Kafka
• Cluster level and host level monitoring
• Identifying performance bottlenecks
• Troubleshooting common Kafka issues

Bonus Content / Time Permitting

Kafka Best Practices

• Avoiding common mistakes
• Hardware selection
• Cluster sizing
• Partition sizing
• Zookeeper settings
• Compression and batching
• Message sizing
• Monitoring and instrumenting
• Troubleshooting

Course Materials

Student Materials: Each participant will receive a Student Guide with course notes, code samples, software tutorials, step-by-step written lab instructions, diagrams and related reference materials and resource links. Students will also receive the project files (or code, if applicable) and solutions required for the hands-on work.

Hands-On Setup Made Simple! Our dedicated tech team will work with you to ensure our ‘easy-access’ cloud-based course environment is accessible, fully-tested and verified as ready to go well in advance of the course start date, ensuring a smooth start to class and effective learning experience for all participants. Please inquire for details and options.