Airbyte – Open Source Data Integration

Module 1: Introduction to Airbyte and Open-Source ELT

• Airbyte overview: open-source data integration platform
• Airbyte architecture: server, scheduler, workers, and temporal workflow engine
• Airbyte Cloud vs self-hosted: deployment options and trade-offs
• Navigating the Airbyte UI: sources, destinations, and connections

Module 2: Sources and Destinations Configuration

• Configuring source connectors: databases, SaaS apps, and APIs
• Database sources: Postgres, MySQL, MongoDB, and SQL Server
• SaaS sources: Salesforce, Zendesk, Stripe, and GitHub
• Destinations: Snowflake, BigQuery, Redshift, S3, and Postgres

Module 3: Sync Modes and Replication Strategies

• Sync modes: full refresh overwrite, full refresh append, and incremental append
• Incremental deduped history: maintaining current state with CDC
• CDC with Airbyte: Debezium-based log reading for databases
• Custom cursor fields for incremental sync configuration

Module 4: Airbyte Custom Connectors

• Connector Development Kit (CDK) overview: Python and low-code CDK
• Building a custom source connector with the low-code CDK
• YAML-based connector definition and stream configuration
• Testing custom connectors with the Airbyte connector acceptance test suite

Module 5: Airbyte Transformations and dbt

• Basic normalisation: Airbyte's automatic schema normalisation
• dbt integration with Airbyte for custom transformations
• Running dbt models after Airbyte sync completion
• Combining Airbyte with dbt and Orchestration tools

Module 6: Airbyte Operations and Monitoring

• Connection scheduling: cron expressions and manual triggers
• Sync monitoring: job logs, error messages, and notification setup
• Resetting connections and clearing destination data
• Airbyte API for programmatic connection management

Module 7: Airbyte Deployment and Scaling

• Docker Compose deployment for small teams
• Kubernetes deployment with Helm charts for production
• Horizontal scaling: worker pods and temporal cluster configuration
• Secret management: Vault and Kubernetes secrets integration

Module 8: Capstone Project and Assessment

• Multi-source Airbyte pipeline: database and SaaS connectors setup
• Custom connector development using low-code CDK
• dbt transformation integration and pipeline monitoring setup
• Final assessment and course certification

Experience in the Industry Gain expertise from data engineers who have deployed and managed Airbyte open-source data integration pipelines for analytics teams building modern data stacks with full control over their data movement infrastructure.

Backgrounds at the Top Our Airbyte trainers have worked at data engineering teams and startups adopting open-source ELT tools for flexible, cost-effective data integration into cloud data warehouses and data lakes.

Clear & Effective Teaching Airbyte connector configuration, sync modes, CDC replication, custom connector development, dbt integration, monitoring, and Kubernetes deployment are explained with practical data integration examples.

Hands-On Learning Focus Students configure Airbyte sources and destinations, set up CDC pipelines, build custom connectors with the CDK, integrate dbt, and deploy Airbyte on Kubernetes through structured hands-on labs.

Up-to-Date Knowledge Trainers keep content current with the latest Airbyte platform releases, Airbyte Cloud enhancements, PyAirbyte for Python-based pipelines, and evolving open-source data integration community practices.