Data Engineering on Google Cloud Platform Training

Description:

This official Google Cloud training provides a comprehensive introduction to data engineering on GCP, equipping participants with the skills to design, build, and manage scalable data pipelines for both batch and streaming workloads.

The training covers the entire data lifecycle on GCP — from ingestion and storage to processing, orchestration, governance, and analytics. Learners explore core services, including Cloud Pub/Sub, Dataflow, BigQuery, Dataproc, Cloud Storage, and Data Fusion, while also gaining exposure to advanced capabilities such as Dataflow streaming, BigQuery ML, Cloud Composer, and AI/ML integration.

Through a blend of demos, labs, and case studies, participants will:

• Build and optimize ETL/ELT workflows.
• Design data lakes and warehouses on GCP.
• Implement streaming pipelines for real-time analytics.
• Use orchestration tools (Cloud Composer, Data Fusion) to manage production pipelines.
• Apply governance and security best practices.
• Explore analytics and machine learning options directly within GCP.

Duration: 5 Days

Course Code: BDT 524

Learning Objectives:

After this training, participants will be able to:

1. Ingest and process data with Pub/Sub and Dataflow
2. Build warehouses with BigQuery
3. Perform ML with BigQuery ML
4. Secure and monitor pipelines

1. Cloud architects
2. Systems engineers
3. Developers working with GCE

1. Knowledge of SQL
2. Experience with data processing tools
3. Basic understanding of cloud services

Course Outline:

Module 1: Introduction to Data Engineering on GCP

1. Role of a data engineer
2. Data engineering challenges
3. Data lakes vs data warehouses
4. GCP data ecosystem overview
5. Case study: Real-world GCP customer pipeline

Module 2: Data Ingestion

1. Introduction to Cloud Pub/Sub
2. Batch vs streaming ingestion patterns
3. Basics of Cloud Dataflow pipelines
4. Lab: Publishing and processing streaming data with Pub/Sub & Dataflow

Module 3: Data Storage

1. Cloud Storage (buckets, lifecycle, security)
2. Relational storage: Cloud SQL, Spanner
3. NoSQL storage: Datastore/Firestore, Bigtable
4. Lab: Loading structured/unstructured data into GCP storage services

Module 4: Data Warehousing with BigQuery

1. Introduction to BigQuery as a modern data warehouse
2. Loading and querying data
3. Partitioning, clustering, schema design
4. Lab: Running federated queries on external datasets

Module 5: Batch Data Processing

1. ETL/ELT concepts on GCP
2. Quality considerations and transformations
3. Using Dataproc for Spark and Hadoop workloads
4. Lab: Running Spark jobs on Dataproc

Module 6: Streaming Data Processing

1. Streaming concepts and challenges
2. Dataflow streaming pipelines
3. BigQuery streaming inserts
4. Cloud Bigtable for high-throughput streaming
5. Lab: Real-time analytics dashboard with Pub/Sub + Dataflow + BigQuery

Module 7: Data Orchestration and Automation

1. Cloud Data Fusion for visual ETL design
2. Cloud Composer (Apache Airflow) for orchestration
3. Scheduling workflows and monitoring pipelines
4. Infrastructure automation with Deployment Manager and Terraform (intro)
5. Lab: Building a pipeline with Data Fusion and Composer

Module 8: Monitoring, Security, and Governance

1. Cloud Monitoring, Logging, Error Reporting, Tracing
2. Data security and access control with IAM and DLP API
3. Governance best practices (projects, quotas, billing)
4. Lab: Detecting PII with Cloud DLP

Module 9: Analytics and Machine Learning with BigQuery

1. BigQuery ML: SQL-based ML models
2. Common supported models
3. Performance tuning for large queries
4. Lab: Building a predictive model with BigQuery ML

Module 10: AI/ML Services for Data Enrichment

1. Pre-built ML APIs (Vision, NLP, Translation) for unstructured data
2. Cloud AI Platform Notebooks for data exploration
3. AutoML for custom ML models
4. Kubeflow pipelines for production ML workflows
5. Lab: Classifying text data with Cloud NLP API

Module 11: Containers and Modern App Integration

1. Using GCP with containerized workloads
2. Google Kubernetes Engine (GKE) for scalable data apps
3. Cloud Run for serverless data services
4. Cloud Pub/Sub + Functions for event-driven data processing

Training material provided: Yes (Digital format)