About This Course
In today's data-driven economy, the ability to move, transform, and deliver data reliably and at scale is one of the most sought-after technical competencies in any industry. Organisations that invest in strong data engineering foundations gain a decisive competitive advantage — their analytics are faster, their decisions are better informed, and their operations are more resilient. Yet many teams still rely on fragile manual processes, disconnected spreadsheets, and ad hoc scripts that create bottlenecks, errors, and compliance risks.
Data Engineering and Workflow Automation Training by TrainingComplex addresses this gap with a comprehensive, hands-on curriculum designed for professionals who work with data in operational, analytical, or technical capacities. The programme covers the full data engineering lifecycle — from sourcing and ingesting raw data to transforming, testing, scheduling, and delivering it to end consumers — with a strong emphasis on automation and repeatability.
Participants will work with widely adopted tools including Python, SQL, Apache Airflow, dbt, and cloud-native pipeline services, learning how to build pipelines that are not only functional but also maintainable, observable, and production-ready. Real-world scenarios drawn from sectors such as finance, healthcare, logistics, and government ensure that the learning is immediately applicable to participants' own working environments.
By the end of the programme, delegates will be equipped to lead data engineering initiatives within their organisations, automate repetitive data workflows, and contribute meaningfully to enterprise data strategy. Whether you are a data analyst looking to deepen your technical skills, a software developer transitioning into data roles, or an IT manager overseeing data infrastructure, this course provides the structured, practical grounding you need to succeed.
Who Should Attend
- Data analysts and business intelligence professionals seeking to expand into data engineering
- Software developers and backend engineers transitioning into data-focused roles
- IT managers and architects responsible for data infrastructure and integration
- Database administrators looking to modernise pipeline and automation skills
- Monitoring and evaluation officers managing large operational datasets
- Cloud and DevOps engineers involved in data platform deployments
- Technology consultants advising organisations on data strategy and tooling
- Government and NGO technical staff handling reporting and data aggregation workflows
Learning Objectives
- Understand the architecture and components of modern data engineering ecosystems
- Design and implement scalable batch and streaming data pipelines
- Apply Python and SQL to extract, transform, and load data across diverse sources
- Build and schedule automated workflows using Apache Airflow and similar orchestration tools
- Transform and model data using dbt for analytics-ready delivery
- Integrate APIs, databases, and cloud storage services into end-to-end data flows
- Implement data quality checks, error handling, and pipeline monitoring strategies
- Apply version control and CI/CD practices to data engineering projects
- Deploy data pipelines on cloud platforms including AWS, Azure, or Google Cloud
- Apply data governance, lineage tracking, and documentation best practices
- Troubleshoot and optimise underperforming or failing pipelines in production
- Collaborate effectively across data engineering, analytics, and business teams
- Design a complete automated data workflow solution aligned to a real-world use case
Course Modules
Module 1: Foundations of Data Engineering
- The modern data landscape: sources, consumers, and flows
- Roles and responsibilities in a data engineering team
- Data engineering vs. data science vs. data analytics
- Key architectural patterns: batch, streaming, and lambda
- Overview of the data engineering toolchain and ecosystem
- Setting up a local development environment for the course
Module 2: Data Ingestion and Source Systems
- Connecting to relational databases using SQL and Python
- Ingesting data from REST APIs and web services
- Working with flat files: CSV, JSON, XML, and Parquet
- Streaming ingestion concepts with Kafka and event queues
- Handling schema evolution and source system changes
- Building reusable ingestion connectors and configuration templates
Module 3: Data Storage and Warehouse Architecture
- Relational vs. columnar vs. NoSQL storage trade-offs
- Designing staging, raw, and curated data layers
- Introduction to cloud data warehouses: Snowflake, BigQuery, and Redshift
- Data lake architecture and object storage with S3 and Azure Blob
- Partitioning, indexing, and storage optimisation strategies
- Data retention policies and cost management in cloud storage
Module 4: SQL for Data Engineering
- Advanced SQL: window functions, CTEs, and subqueries
- Set operations and complex joins for data transformation
- Writing performant queries for large datasets
- Stored procedures, views, and materialised views
- SQL-based data quality checks and assertion queries
- Database profiling and exploratory data analysis with SQL
Module 5: Python for Data Pipelines
- Python data engineering libraries: Pandas, SQLAlchemy, and Requests
- Reading and writing data across multiple formats and storage systems
- Building modular, reusable Python pipeline functions
- Error handling, logging, and retry logic in Python scripts
- Environment management with virtual environments and dotenv
- Introduction to unit testing pipeline code with Pytest
Module 6: Data Transformation and Modelling with dbt
- dbt architecture: models, sources, tests, and documentation
- Building staging, intermediate, and mart layer models
- Applying dbt tests for data quality and schema validation
- Using Jinja templating and macros to reduce code repetition
- Generating and publishing dbt documentation and lineage graphs
- Managing dbt projects with version control and CI integration
Module 7: Workflow Orchestration with Apache Airflow
- Airflow architecture: DAGs, operators, sensors, and executors
- Building and scheduling DAGs for batch data workflows
- Using connections, variables, and XComs for pipeline configuration
- Handling dependencies, branching, and conditional task execution
- Monitoring DAG runs and diagnosing task failures in the Airflow UI
- Best practices for writing maintainable, idempotent Airflow DAGs
Module 8: Cloud Data Platforms and Integration
- Overview of cloud data services on AWS, Azure, and Google Cloud
- Building pipelines with cloud-native tools: Glue, Data Factory, and Dataflow
- Managing credentials, IAM roles, and secure cloud access
- Connecting on-premise databases to cloud pipelines via VPN and APIs
- Event-driven pipeline triggers using cloud messaging services
- Cost monitoring and resource optimisation on cloud data platforms
Module 9: Data Quality, Testing, and Observability
- Defining and measuring data quality dimensions
- Implementing row-count, null, uniqueness, and referential integrity checks
- Building alerting and notification systems for pipeline failures
- Introduction to data observability tools: Monte Carlo, Great Expectations
- Tracking data freshness and SLA compliance in production pipelines
- Creating runbooks and incident response procedures for data failures
Module 10: Version Control and CI/CD for Data Projects
- Git fundamentals for data engineering: branching, commits, and pull requests
- Organising data engineering repositories and project structure
- Automating dbt and pipeline tests in CI/CD pipelines with GitHub Actions
- Managing environment-specific configurations across dev, staging, and prod
- Deployment strategies for data pipeline releases
- Code review practices and collaboration workflows for data teams
Module 11: Data Governance, Lineage, and Documentation
- Core concepts of data governance: ownership, stewardship, and cataloguing
- Tracking data lineage from source to consumption layer
- Using metadata management tools: Apache Atlas and dbt docs
- Documenting pipelines, transformations, and business rules
- Applying data classification and access control policies
- Building a data dictionary and maintaining it as a living document
Module 12: Performance Tuning and Pipeline Optimisation
- Identifying and resolving bottlenecks in ETL and ELT pipelines
- Query optimisation techniques for warehouse and lake environments
- Parallel processing and partitioned workloads in Python and Spark
- Optimising Airflow DAG performance and reducing scheduler load
- Incremental loading strategies to minimise processing time and cost
- Load testing and benchmarking data pipelines before production deployment
Module 13: Capstone Project and Real-World Application
- Defining a data engineering problem statement from a real-world sector scenario
- Architecting an end-to-end automated pipeline solution
- Building ingestion, transformation, and delivery layers using course tools
- Implementing data quality checks, scheduling, and monitoring for the solution
- Presenting the pipeline design and live demo to the cohort and facilitators
- Reflecting on lessons learned and planning next steps for workplace application
Requirements & Prerequisites
Participants should have a working knowledge of SQL and at least basic familiarity with Python or another scripting language. Experience working with data in any professional capacity — such as analysis, reporting, or database administration — is expected. Prior exposure to cloud platforms or Linux command-line environments is helpful but not mandatory, as foundational concepts will be briefly reviewed at the start of the programme.
Training Methodology
- Guided live coding sessions where participants build real pipelines step by step using Python, SQL, and Airflow on pre-configured lab environments
- Case study analysis drawn from data engineering challenges in sectors such as finance, public health, logistics, and government reporting
- Hands-on dbt modelling exercises transforming raw datasets into analytics-ready data marts
- Pipeline debugging and troubleshooting labs where participants diagnose and fix intentionally broken workflows
- Cloud deployment walkthroughs on AWS or Azure with participant accounts provisioned for the duration of the programme
- Peer code review sessions where participants critique and improve each other's pipeline designs
- Group architecture design challenges requiring teams to plan a complete data platform for a simulated organisation
- Daily reflection check-ins linking the day's technical content to participants' own organisational data challenges
- A capstone project in which each participant designs and presents a fully automated end-to-end data workflow relevant to their sector
Organisational Benefits
- Reduced dependency on manual data processing, saving staff time and minimising human error
- Faster, more reliable delivery of data to analytics and reporting systems
- Improved data quality and consistency across organisational reporting
- Strengthened internal capacity to build and maintain data infrastructure without full dependence on external vendors
- Better alignment between IT and business teams through shared data engineering vocabulary and practices
- Accelerated time-to-insight for decision-makers across departments
- Reduced operational risk from brittle, undocumented data processes
- Ability to scale data operations as organisational data volumes and complexity grow
- Enhanced compliance and auditability through documented pipelines and data lineage
- A workforce equipped to adopt and leverage cloud-native data platforms effectively
Where Is Data Engineering and Workflow Automation Training Offered?
This course runs in-person across: Kampala, Arusha, Dar es Salaam, Addis Ababa, Accra, Johannesburg, Naivasha, Pretoria, Kisumu, Abuja, Nairobi, Kigali, Dubai, Cape Town, Zanzibar, Nakuru, Lagos, Mombasa, Riyadh, plus Live Virtual (online) cohorts. Browse live dates for your city below.
Upcoming Schedules
| Location | Format | Dates | Price |
|---|
| Live Virtual |
Online |
12 Oct 2026 – 23 Oct 2026 |
KES 160,000 |
| Kampala, Uganda |
In-Person |
26 Oct 2026 – 4 Nov 2026 |
USD 3,800 |
| Arusha, Tanzania |
In-Person |
26 Oct 2026 – 4 Nov 2026 |
USD 4,200 |
| Dar es Salaam, Tanzania |
In-Person |
26 Oct 2026 – 4 Nov 2026 |
USD 4,600 |
| Addis Ababa, Ethiopia |
In-Person |
26 Oct 2026 – 4 Nov 2026 |
USD 4,800 |
| Accra, Ghana |
In-Person |
26 Oct 2026 – 4 Nov 2026 |
USD 7,600 |
| Johannesburg, South Africa |
In-Person |
26 Oct 2026 – 4 Nov 2026 |
USD 7,000 |
| Naivasha, Kenya |
In-Person |
26 Oct 2026 – 4 Nov 2026 |
KES 240,000 |
| Pretoria, South Africa |
In-Person |
26 Oct 2026 – 4 Nov 2026 |
USD 6,600 |
| Kisumu, Kenya |
In-Person |
26 Oct 2026 – 4 Nov 2026 |
KES 240,000 |
| Arusha, Tanzania |
In-Person |
2 Nov 2026 – 11 Nov 2026 |
USD 4,200 |
| Abuja, Nigeria |
In-Person |
2 Nov 2026 – 13 Nov 2026 |
USD 5,600 |
| Nairobi, Kenya |
In-Person |
2 Nov 2026 – 13 Nov 2026 |
KES 230,000 |
| Live Virtual |
Online |
2 Nov 2026 – 11 Nov 2026 |
KES 160,000 |
| Kigali, Rwanda |
In-Person |
2 Nov 2026 – 13 Nov 2026 |
USD 3,800 |
+226 more dates — view live schedule →
Frequently Asked Questions
Do I need to be a software developer to attend this course?
No, but you should be comfortable working with data and have at least basic SQL skills and some Python familiarity. The course is designed for data-aware professionals who want to build or deepen engineering capabilities, not exclusively for software developers.
What tools and software will we use during the course?
Participants will work with Python, SQL, Apache Airflow, dbt, Git, and cloud platform services (AWS or Azure). All required software will be pre-configured in a lab environment provided by TrainingComplex, so you do not need to install anything in advance.
Will I receive a certificate upon completing the course?
Yes. Participants who attend the full programme and complete the capstone project will receive a Certificate of Completion in Data Engineering and Workflow Automation from TrainingComplex.
Is the course available online for remote participants?
Yes. A live virtual delivery option is available for participants who cannot attend in person. Virtual delegates receive the same lab environments, live instruction, and materials as in-person attendees, with sessions conducted over a video conferencing platform.
How much prior cloud experience do I need?
No prior cloud experience is required. The course introduces cloud data platforms progressively and provides guided walkthroughs with provisioned accounts. Familiarity with any cloud environment is a bonus but is not a prerequisite for enrolment.
