Observability Pipeline Architect

As organizations scale their observability programs, raw telemetry data — metrics, logs, and traces — can generate enormous volumes that overwhelm storage, inflate costs, and make signals harder to find. The Observability Pipeline Architect helps infrastructure engineers and platform teams design the routing, filtering, enrichment, and forwarding layers that sit between data sources and observability backends, turning raw telemetry into clean, cost-efficient, queryable signal.

This assistant specializes in the tools that make up the modern observability pipeline layer: OpenTelemetry Collector with its full processor and exporter ecosystem, Fluentd and Fluent Bit for log shipping, Vector for high-performance multi-signal pipelines, and Kafka or Kinesis as durable telemetry buses for large-scale deployments. It helps you design pipeline topologies — agent, gateway, and aggregator tiers — that are resilient, scalable, and maintainable.

You describe your telemetry sources (Kubernetes pods, cloud services, on-premises applications, third-party SaaS), your current or target backends (Prometheus, Loki, Elasticsearch, Datadog, Honeycomb, Splunk), and your constraints (data volume, cost budget, compliance requirements, latency sensitivity) — and the assistant designs a pipeline architecture that routes data to the right destination, applies enrichment or filtering at the right point, and avoids single points of failure.

The assistant generates real configuration files: OpenTelemetry Collector YAML pipelines with receivers, processors, and exporters; Vector TOML configs; Fluentd routing and filtering rules; and Kafka topic and consumer group designs for telemetry fan-out. It explains the trade-offs in every architectural decision, from where to apply sampling to whether to enrich data at the edge or at the aggregator.

Ideal users include platform engineers building centralized observability infrastructure, SREs fighting telemetry cost overruns, and teams migrating from legacy log shippers or proprietary agents to open-standard pipelines. The Observability Pipeline Architect helps you build the foundation that makes every other observability tool more effective.