About This Course
Data engineering has emerged as one of the most strategically critical disciplines in the modern enterprise. As organisations accumulate vast volumes of structured and unstructured data, the ability to move, transform, validate, and serve that data reliably — at speed and at scale — determines whether analytics and machine learning initiatives succeed or stall. Yet many organisations find themselves burdened by brittle pipelines, inconsistent data quality, and siloed architectures that cannot keep pace with business demand. This programme addresses those challenges head-on.
Applied Data Engineering: Building Scalable Pipelines and ML-Ready Data Systems is a comprehensive, hands-on 10-day course designed for professionals who need to build and maintain the data infrastructure that underpins intelligent business systems. Participants will progress from foundational data engineering principles through to the design of production-grade batch and streaming pipelines, data lakehouse architectures, orchestration frameworks, and ML feature engineering workflows. Every module is grounded in real-world scenarios drawn from finance, healthcare, retail, and public sector data environments.
Delivered by practitioners with deep industry experience, the programme combines conceptual frameworks with intensive tool-based labs. Participants will leave with a working portfolio of pipeline designs, transformation logic, and orchestration workflows built during the training — assets they can immediately apply within their organisations. Whether you are modernising a legacy data stack, migrating to the cloud, or preparing your data infrastructure for machine learning at scale, this programme provides the engineering rigour and architectural confidence to make it happen.
Who Should Attend
- Data Engineers seeking to deepen their pipeline design and orchestration skills
- Software Developers transitioning into data engineering roles
- Data Analysts who need to build and maintain their own data workflows
- ML Engineers who require a stronger foundation in feature engineering and data infrastructure
- Business Intelligence Developers working on scalable reporting data systems
- Cloud Architects involved in designing data platform solutions
- IT Managers overseeing data platform strategy and implementation
- Data Platform Leads responsible for enterprise data infrastructure
Learning Objectives
- Understand core data engineering principles including data modelling, storage paradigms, and pipeline architecture patterns
- Design and implement scalable batch and streaming data pipelines using Apache Spark and Apache Kafka
- Build transformation layers using dbt (data build tool) with robust testing and documentation practices
- Orchestrate complex multi-step workflows using Apache Airflow with dependency management and failure recovery
- Apply data lakehouse architecture principles using Delta Lake or Apache Iceberg on cloud-native platforms
- Implement data quality frameworks including schema validation, lineage tracking, and anomaly detection
- Engineer ML-ready feature stores and understand the data contracts between engineering and data science teams
- Apply CI/CD practices to data pipeline development using version control, automated testing, and deployment pipelines
- Optimise pipeline performance through partitioning strategies, caching, incremental loads, and cost-aware design
- Design and implement data governance controls including access management, cataloguing, and compliance tagging
- Monitor and observe data pipelines in production using logging, alerting, and data observability platforms
- Apply data security best practices including encryption, masking, and role-based access control in data systems
- Develop an end-to-end capstone data platform integrating ingestion, transformation, orchestration, and ML readiness
Course Modules
Module 1: Foundations of Modern Data Engineering
- The evolving role of the data engineer in analytics and AI-driven organisations
- Core data engineering concepts: ingestion, transformation, storage, and serving layers
- Batch vs. streaming vs. micro-batch processing paradigms and when to apply each
- Overview of the modern data stack: tools, platforms, and architectural patterns
- Data contracts and SLAs: defining reliability agreements between producers and consumers
- Setting up the development environment: Python, SQL, Docker, and cloud sandbox access
Module 2: Data Modelling and Storage Design
- Relational vs. columnar vs. document storage: choosing the right paradigm
- Dimensional modelling: star and snowflake schemas for analytical workloads
- Data Vault modelling for auditability and historical tracking
- Schema evolution strategies: handling breaking and non-breaking changes in production
- Partitioning and clustering strategies for query performance optimisation
- Storage format comparison: Parquet, ORC, Avro, and Delta for different pipeline use cases
Module 3: Data Ingestion: Batch and Real-Time Patterns
- Designing robust ingestion layers: full load, incremental load, and change data capture (CDC)
- Building batch ingestion pipelines using Python and Apache Spark
- Introduction to Apache Kafka: topics, producers, consumers, and consumer groups
- Real-time ingestion patterns: event streaming, message queues, and exactly-once delivery guarantees
- API-based ingestion: REST, GraphQL, and webhook integration patterns
- Handling ingestion failures: dead-letter queues, retry logic, and idempotency
Module 4: Data Transformation with Apache Spark
- Spark architecture: executors, drivers, DAGs, and the execution model
- DataFrame API operations: filtering, joining, aggregating, and windowing at scale
- Handling skewed data and optimising shuffle operations in Spark jobs
- User-defined functions (UDFs) and pandas UDFs for custom transformation logic
- Spark SQL for complex multi-table transformation workflows
- Writing and running Spark jobs in a distributed cluster environment using PySpark
Module 5: Scalable Transformations with dbt
- dbt architecture: models, sources, seeds, snapshots, and exposures
- Building a layered transformation model: staging, intermediate, and marts layers
- Implementing dbt tests for schema validation, uniqueness, and referential integrity
- Using dbt macros and Jinja templating for reusable transformation logic
- Documenting data lineage and generating a data catalogue from dbt metadata
- dbt deployment in production: running dbt Core in CI pipelines and scheduling on Airflow
Module 6: Workflow Orchestration with Apache Airflow
- Airflow architecture: DAGs, operators, sensors, hooks, and the scheduler
- Designing robust DAGs with dependency management, branching, and dynamic task generation
- Implementing failure recovery: retries, SLA alerts, and on-failure callbacks
- Parameterising DAGs with variables, connections, and runtime configuration
- Integrating Airflow with Spark, dbt, and cloud storage services
- Monitoring pipeline runs: the Airflow UI, logs, and integrating with external alerting tools
Module 7: Data Lakehouse Architecture and Cloud-Native Platforms
- Data lake vs. data warehouse vs. data lakehouse: architectural trade-offs and use cases
- Delta Lake and Apache Iceberg: ACID transactions, time travel, and schema enforcement on data lakes
- Designing a medallion architecture: Bronze, Silver, and Gold layer data flows
- Cloud-native data platform services: AWS Glue, Azure Data Factory, and Google Dataflow
- Managing storage costs: lifecycle policies, compaction, and vacuum operations
- Migration strategies: moving from legacy on-premise data warehouses to lakehouse architectures
Module 8: Data Quality, Observability, and Lineage
- Defining a data quality framework: completeness, accuracy, consistency, timeliness, and validity
- Implementing schema validation and data profiling with Great Expectations and dbt tests
- Data lineage tracking: column-level lineage from source to ML model feature
- Data observability platforms: monitoring freshness, volume, and distribution anomalies
- Building alerting workflows for pipeline failures and data quality breaches
- Root cause analysis techniques for diagnosing data quality incidents in production
Module 9: Feature Engineering and ML-Ready Data Systems
- Understanding the data engineering–data science interface: feature stores and training/serving skew
- Designing feature pipelines: point-in-time correctness and avoiding data leakage
- Building and registering features in an offline feature store using Feast or Hopsworks
- Online vs. offline feature serving: low-latency retrieval for real-time ML inference
- Versioning datasets and features for reproducible model training
- Data contracts between engineering and ML teams: schemas, expectations, and change management
Module 10: CI/CD for Data Pipelines and DataOps Practices
- Applying software engineering discipline to data: version control with Git for pipelines and models
- Automated testing strategies for data pipelines: unit tests, integration tests, and data validation gates
- Building CI/CD pipelines for dbt and Airflow using GitHub Actions or Azure DevOps
- Environment management: development, staging, and production data environments
- Infrastructure as Code for data platforms using Terraform and cloud-native IaC tools
- DataOps culture: cross-functional collaboration, agile data sprints, and documentation standards
Module 11: Pipeline Performance Optimisation and Cost Engineering
- Identifying and resolving pipeline bottlenecks through profiling and execution plan analysis
- Incremental processing strategies: watermarking, checkpointing, and late-arriving data handling
- Spark performance tuning: caching, broadcast joins, and adaptive query execution
- Cost-aware pipeline design: right-sizing compute, spot instances, and resource scheduling
- Materialisation strategies in dbt: table, view, incremental, and ephemeral trade-offs
- Capacity planning and SLA-driven scheduling for time-sensitive data workflows
Module 12: Data Security, Governance, and Compliance in Data Platforms
- Data classification frameworks: categorising data by sensitivity and regulatory scope
- Role-based access control (RBAC) and attribute-based access control (ABAC) in data platforms
- Data masking, tokenisation, and encryption at rest and in transit
- Implementing data cataloguing with Apache Atlas or cloud-native catalogues for discoverability
- GDPR, PDPA, and sector-specific compliance requirements for data pipeline design
- Audit logging and access monitoring for data governance and regulatory reporting
Module 13: Capstone Project: End-to-End Data Platform Build
- Capstone brief and requirements: designing a real-world data platform from business requirements
- Ingestion layer build: connecting sources, applying CDC patterns, and landing data in the lakehouse
- Transformation layer: implementing staging, intermediate, and mart models in dbt with full test coverage
- Orchestration: building and scheduling the full pipeline DAG in Apache Airflow with failure handling
- Data quality and observability integration: embedding quality checks and alerting into the pipeline
- Presentation and peer review: showcasing the completed data platform with architecture documentation
Requirements & Prerequisites
Participants should have working knowledge of Python programming and basic SQL query writing, as the course involves hands-on coding from the first day. Familiarity with relational databases and basic concepts of data warehousing or ETL processes is recommended. Prior exposure to cloud platforms such as AWS, Azure, or Google Cloud is helpful but not strictly required, as cloud concepts will be introduced progressively throughout the programme.
Training Methodology
- Instructor-led concept sessions with architectural diagrams and real-world case studies from finance, retail, and public sector data environments
- Hands-on pipeline labs using Apache Spark, Airflow, dbt, and Kafka on pre-configured cloud sandbox environments
- Data modelling workshops where participants design dimensional schemas and lakehouse table structures from raw business requirements
- Peer code review sessions simulating engineering team pull request and feedback workflows
- Pipeline debugging exercises using intentionally broken or degraded datasets to build fault-diagnosis skills
- Data quality challenge scenarios requiring participants to implement validation rules and lineage tracking from scratch
- ML feature engineering practicals connecting data pipelines to a sample model training workflow
- Group architecture design sprints where teams propose, critique, and refine end-to-end data platform blueprints
- Daily stand-up reflections modelled on agile data team ceremonies to consolidate learning and surface questions
- Capstone project development across the final two days, producing a fully documented, deployable data platform prototype
Organisational Benefits
- Accelerated delivery of production-grade data pipelines that support business analytics and ML initiatives
- Reduced data downtime and pipeline failures through adoption of observability and quality frameworks
- Stronger collaboration between data engineering, data science, and business intelligence teams
- Faster onboarding of new data engineers through standardised architectural patterns and reusable pipeline templates
- Improved data governance posture with traceable lineage, access controls, and compliance documentation
- Cloud infrastructure cost savings through optimised partitioning, incremental processing, and resource-aware pipeline design
- Greater organisational confidence in data reliability, enabling more ambitious analytics and AI investments
- A portfolio of internal engineering artefacts built during the training that can be directly adapted for live projects
Where Is Applied Data Engineering: Building Scalable Pipelines and ML-Ready Data Systems Training Offered?
This course runs in-person across: Nairobi, Riyadh, Dubai, Lagos, Abuja, Cape Town, Zanzibar, Kigali, Nakuru, Mombasa, Naivasha, Johannesburg, Dar es Salaam, Accra, Kisumu, Addis Ababa, Pretoria, Arusha, Kampala, plus Live Virtual (online) cohorts. Browse live dates for your city below.
Upcoming Schedules
| Location | Format | Dates | Price |
|---|
| Live Virtual |
Online |
24 Oct 2026 – 13 Dec 2026 |
KES 160,000 |
| Nairobi, Kenya |
In-Person |
26 Oct 2026 – 4 Nov 2026 |
KES 230,000 |
| Riyadh, Saudi Arabia |
In-Person |
26 Oct 2026 – 4 Nov 2026 |
USD 9,000 |
| Dubai, United Arab Emirates |
In-Person |
26 Oct 2026 – 4 Nov 2026 |
USD 8,200 |
| Lagos, Nigeria |
In-Person |
26 Oct 2026 – 4 Nov 2026 |
USD 5,000 |
| Abuja, Nigeria |
In-Person |
26 Oct 2026 – 4 Nov 2026 |
USD 5,600 |
| Cape Town, South Africa |
In-Person |
26 Oct 2026 – 4 Nov 2026 |
USD 7,600 |
| Zanzibar, Tanzania |
In-Person |
26 Oct 2026 – 4 Nov 2026 |
USD 5,000 |
| Kigali, Rwanda |
In-Person |
26 Oct 2026 – 4 Nov 2026 |
USD 3,800 |
| Nakuru, Kenya |
In-Person |
26 Oct 2026 – 4 Nov 2026 |
KES 240,000 |
| Mombasa, Kenya |
In-Person |
26 Oct 2026 – 4 Nov 2026 |
KES 240,000 |
| Riyadh, Saudi Arabia |
In-Person |
2 Nov 2026 – 11 Nov 2026 |
USD 9,000 |
| Abuja, Nigeria |
In-Person |
2 Nov 2026 – 13 Nov 2026 |
USD 5,600 |
| Live Virtual |
Online |
2 Nov 2026 – 13 Nov 2026 |
KES 160,000 |
| Kigali, Rwanda |
In-Person |
2 Nov 2026 – 13 Nov 2026 |
USD 3,800 |
+227 more dates — view live schedule →
Frequently Asked Questions
Do I need prior data engineering experience to attend this course?
Participants should have solid Python and SQL skills and a working understanding of databases or ETL concepts. You do not need prior data engineering job experience, but the programme moves quickly and is not suitable for complete beginners to programming.
What tools and platforms will we work with during the course?
The course covers Apache Spark, Apache Kafka, Apache Airflow, dbt, Delta Lake, Feast (feature store), Great Expectations, and cloud-native services on AWS, Azure, or GCP. All lab environments are pre-configured and provided — participants do not need to set up their own infrastructure before arriving.
Will I receive a certificate upon completion?
Yes. Participants who attend the full programme and complete the capstone project will receive a TrainingComplex Certificate of Completion in Applied Data Engineering, which can be shared on professional profiles such as LinkedIn.
Is this course more theory-based or hands-on?
The programme is heavily practical — approximately 60% of contact time is spent in hands-on labs, pipeline builds, and design workshops. Conceptual sessions are kept concise and are always followed immediately by applied exercises using real datasets.
Can organisations send a team of data engineers to this course?
Absolutely. TrainingComplex offers group enrolment discounts and can also deliver this programme as a private, customised cohort for organisational teams. Contact our corporate training desk for tailored scheduling and pricing.
