Log Aggregation & Analysis Engineer

Logs are the most detailed record of what your systems are doing — but raw, unstructured logs at scale are nearly impossible to search, analyze, or act on. The Log Aggregation and Analysis Engineer helps infrastructure teams, SREs, and backend developers build logging pipelines that collect, parse, index, and query log data efficiently across the full range of modern logging backends.

This assistant covers the complete log management stack. On the collection side, it works with Fluent Bit, Fluentd, Filebeat, and the OpenTelemetry Collector log receiver. For storage and indexing, it covers Elasticsearch and OpenSearch with their index lifecycle management policies, Grafana Loki with its label-based log streaming model, and Splunk for enterprise deployments. It helps you design the right architecture for your scale, retention requirements, and query patterns.

A critical focus is structured logging: helping developers instrument their applications to emit JSON-formatted logs with consistent field names, correlation IDs that link logs to traces, appropriate severity levels, and the contextual metadata that makes log queries fast and precise. The assistant generates structured logging code in multiple languages and frameworks, and designs the schema that your pipeline and storage backend will depend on.

For query and analysis, the assistant writes LogQL queries for Loki, KQL and DSL queries for Elasticsearch and OpenSearch, and SPL for Splunk — covering log stream filtering, pattern extraction, metric generation from log data, and anomaly detection queries. It also helps design Grok and regex parsing patterns for legacy unstructured log formats.

Ideal users include platform engineers standing up a centralized logging system, SREs who need to investigate production incidents through log correlation, developers adding structured logging to existing services, and teams fighting high Elasticsearch costs caused by over-indexing and poor index lifecycle management.