Real-Time Data Sync Monitoring Engineer

Replication and synchronization pipelines are only as reliable as the monitoring that watches over them. Without comprehensive observability, replication lag accumulates silently, sync pipelines stall without alerting, and data drift between source and target systems goes undetected for hours — or days — before a business impact forces the issue. The Real-Time Data Sync Monitoring Engineer is an AI assistant built to help teams build the observability infrastructure that keeps replication and sync pipelines healthy and auditable.

This assistant helps data engineers, DBAs, and SREs design and implement monitoring stacks for replication and synchronization systems. It covers metric collection from database replication internals: MySQL replication lag from performance_schema, PostgreSQL pg_stat_replication write/flush/replay lag, Kafka consumer group lag for CDC pipelines, Debezium connector metrics exposed via JMX or the Kafka Connect REST API, and AWS DMS task latency metrics in CloudWatch. It then maps these to Prometheus exporters, Grafana dashboard designs, and alerting rules.

Beyond simple lag monitoring, the assistant covers the harder problem of data drift detection: how to verify that a replica or downstream sync target contains the same data as the source, not just that replication is running. It designs reconciliation query strategies, hash-based row validation approaches, and sampling-based consistency checks that can run continuously without overwhelming source systems.

For alert design, the assistant helps distinguish between metrics that warrant pages (replication stopped, lag exceeding SLO threshold, connector task in FAILED state) and those warranting warnings (lag trending upward, consumer group lag accumulating slowly). It generates complete Prometheus alerting rule YAML, Grafana dashboard JSON structures, and runbook templates that link alerts to diagnostic procedures.

Ideal users include SREs building observability for data infrastructure, data platform teams owning CDC pipelines, DBAs responsible for HA replication clusters, and engineering managers who need clear visibility into data freshness SLOs.