Data Engineering and Pipeline Development

Design, build, and optimize Apache Airflow DAGs for data pipeline orchestration with dynamic task generation, dependency management, and production-grade reliability patterns.

Tune Apache Spark jobs for performance, memory efficiency, and cost reduction with expert guidance on partitioning, shuffles, caching, and cluster configuration.

Provision and manage cloud data infrastructure on AWS, GCP, or Azure using Terraform or Pulumi — including data lakes, warehouses, compute clusters, and IAM for data platforms.

Build reliable data ingestion pipelines and change data capture systems using Debezium, Kafka Connect, Airbyte, or custom connectors for database and API sources.

Architect scalable data lakehouse solutions using Delta Lake, Apache Iceberg, or Apache Hudi with storage layer design, table format optimization, and governance patterns.

Implement data quality frameworks, anomaly detection, data contracts, and pipeline observability using Great Expectations, Monte Carlo, Soda, or custom validation logic.

Design dimensional models, star schemas, and data vault structures for Snowflake, BigQuery, Redshift, or Databricks with analytical performance and scalability in mind.

Build production-grade dbt projects with modular SQL models, testing frameworks, incremental strategies, and documentation for modern data warehouse transformation layers.

Design and implement robust ETL pipelines for extract, transform, and load workflows using modern data engineering tools, patterns, and best practices.

Build real-time streaming data pipelines with Apache Kafka, Flink, Spark Streaming, or Kinesis — from event ingestion to stateful stream processing and sink delivery.