Description:

This intermediate course builds on Spark fundamentals to equip data engineers with advanced techniques for optimizing Spark 3.X applications. Participants will deepen their understanding of RDD and DataFrame internals, master performance tuning, explore advanced SQL operations, and implement complex data processing patterns. Through practical labs and real-world scenarios, attendees will gain expertise in building production-grade Spark solutions that scale efficiently across distributed clusters.

Duration: 

2 Days

Course Code: BDT168

Learning Objectives:

After this course, you will be able to:

1. Understand Spark 3.X architecture, Catalyst optimizer, and Tungsten memory management for performance gains
2. Master advanced RDD operations, custom partitioning, and shuffle optimization techniques
3. Leverage DataFrame/Dataset APIs with complex transformations, window functions, and performance best practices
4. Implement advanced Spark SQL techniques including Columnar Storage, query execution plans, and cost-based optimization

1. Data Engineers looking to optimize Spark applications for production
2. Data Scientists and Analysts seeking advanced data processing and transformation skills
3. Developers transitioning from traditional batch processing to distributed analytics

1. Completion of BDT167 (Beginner Apache Spark) or equivalent hands-on Spark experience
2. Working knowledge of Python, Jupyter notebooks, and pandas
3. Familiarity with Hadoop/distributed computing concepts and SQL queries

Course Outline:

Day 1: Spark 3.X Architecture & Advanced RDDs

Module 1: Spark 3.X Architecture Deep Dive

1. Spark 3.X enhancements: Performance improvements and new features
2. Driver-Executor architecture and task scheduling mechanisms
3. Catalyst query optimizer: Rule-based and cost-based optimization
4. Tungsten project: Off-heap memory management and binary encoding

Module 2: RDD Internals & Advanced Operations

1. RDD DAG (Directed Acyclic Graph) and lineage tracking for fault tolerance
2. Partitioning strategies: Hash, range, and custom partitioners
3. Shuffle operations: Understanding reduce-side joins, group-by-key, and coalesce
4. RDD persistence, caching strategies, and storage levels
5. Hands-on: Optimizing RDDs with repartitioning and custom partitions

Day 2: DataFrames/Datasets & Advanced SQL

Module 3: DataFrame & Dataset APIs Mastery

1. DataFrame vs RDD: Trade-offs and when to use each
2. Dataset API: Type safety, encoders, and performance characteristics
3. Complex transformations: Window functions, lateral views, and pivot operations
4. Joins: Inner, outer, semi, anti, cross—optimization and broadcast patterns
5. Aggregations: GroupBy, cube, rollup, and custom aggregation functions (UDAF)
6. Hands-on: Building multi-step ETL pipelines with DataFrames

Module 4: Advanced Spark SQL & Query Optimization

1. Catalyst Query Optimizer: Physical plan generation and execution
2. EXPLAIN plans: Analyzing query execution and identifying bottlenecks
3. Advanced SQL: CTEs, recursive queries, subqueries, and analytical functions
4. Columnar storage: Parquet, ORC formats and compression trade-offs
5. Query hints and hint-based optimization techniques
6. Hands-on: Query tuning, index strategies, and performance benchmarking