Description:

This course provides an introduction to big data concepts and hands-on experience with Apache Spark, a powerful distributed data processing framework. Participants will learn how to process large-scale datasets efficiently using Spark’s core components, including Spark SQL, DataFrames, and basic data transformations.

Through practical labs and examples, learners will understand how Spark enables scalable data processing and analytics for real-world applications such as ETL pipelines, data analysis, and machine learning workflows.

Duration: 

2 Days

Course Code: BDT152

Learning Objectives:

After this course, you will be able to:

1. Understand the fundamentals of big data and distributed computing
2. Explain the architecture and components of Apache Spark
3. Work with Spark DataFrames and perform data transformations
4. Use Spark SQL for querying large datasets
5. Build simple data processing pipelines using Spark
6. Understand basic concepts of Spark for machine learning

Data engineers, data analysts, developers, and IT professionals interested in big data processing and analytics

Basic knowledge of Python or Scala, SQL, and fundamental data processing concepts; familiarity with distributed systems is a plus

Course Outline:

Module 1: Introduction to Big Data

1. What is big data (Volume, Velocity, Variety)
2. Traditional vs distributed data processing
3. Overview of big data ecosystem

Module 2: Introduction to Apache Spark

1. Spark architecture and components
2. Spark vs Hadoop MapReduce
3. Setting up Spark environment

Module 3: Working with Spark Core and RDDs

1. Resilient Distributed Datasets (RDDs)
2. Transformations and actions
3. Fault tolerance and parallel processing

Module 4: Spark DataFrames and Spark SQL

1. DataFrames and datasets
2. Querying data using Spark SQL
3. Data manipulation and aggregation

Module 5: Data Processing and ETL Pipelines

1. Data ingestion and transformation
2. Building ETL workflows
3. Handling large datasets efficiently

Module 6: Introduction to Spark MLlib

1. Overview of machine learning in Spark
2. Basic ML workflows
3. Example use cases

Module 7: Performance Optimization and Best Practices

1. Caching and partitioning
2. Performance tuning techniques
3. Monitoring and debugging Spark jobs