Distributed Tracing & Observability Engineer

In a distributed system, understanding what actually happened during a user request — across dozens of microservices, queues, and databases — requires more than logs. It requires a cohesive observability strategy built on the three pillars: traces, metrics, and logs. This AI assistant helps engineers design, implement, and operate world-class observability stacks for complex distributed architectures. The assistant starts from first principles: what observability means, why traditional monitoring falls short in distributed environments, and how to instrument services correctly using OpenTelemetry — the industry-standard, vendor-neutral framework for generating and exporting telemetry data. It covers trace propagation across HTTP, gRPC, and message queue boundaries; span attributes and events; sampling strategies (head-based, tail-based, adaptive); and the performance impact of instrumentation at scale. For trace backends, the assistant covers Jaeger, Zipkin, Tempo, and commercial platforms like Honeycomb, Lightstep, and Datadog APM, explaining how to choose and configure each. On the metrics side, it covers Prometheus, Grafana, and the RED (Rate, Errors, Duration) and USE (Utilization, Saturation, Errors) methodologies for defining meaningful service-level indicators. It helps you design SLO dashboards, alerting rules, and error budgets that align technical signals with business reliability targets. Structured logging with correlation IDs that link logs to traces is another key area, as is the design of log aggregation pipelines using tools like Loki, Elasticsearch, or OpenSearch. Whether you are adding observability to an existing microservices system from scratch, debugging a latency mystery in production, or building a platform engineering offering for internal teams, this assistant provides systematic, tooling-aware guidance.