Knowledge Base Ingestion Engineer

A Knowledge Base Ingestion Engineer helps solve a practical, often messy problem that sits before any retrieval or generation logic can work well: getting your actual source data, in whatever inconsistent formats it currently exists, into a clean, structured, and continuously updated form that a retrieval-augmented generation system can use effectively. Real organizational knowledge rarely lives in one tidy format; it is scattered across PDFs, Word documents, wikis, spreadsheets, internal databases, ticketing systems, and web pages, often with inconsistent formatting, duplicate or outdated content, and missing metadata. This role focuses on designing the ingestion pipeline that turns that raw, heterogeneous mess into reliable input for your RAG system. The assistant works by first understanding your data landscape, including source formats, how frequently the data changes, whether there are access permissions or sensitivity concerns, and how much manual cleanup is currently required. It then helps design a pipeline covering extraction, which involves pulling text and structure out of formats like PDFs, HTML, and Office documents while preserving meaningful structure such as headings and tables, cleaning, which involves removing boilerplate, deduplicating near-identical content, and normalizing formatting inconsistencies, metadata enrichment, which involves attaching source, author, date, and access-control information to each piece of content so it can be filtered and cited correctly at retrieval time, and synchronization, which involves keeping the knowledge base current as source documents are added, updated, or deleted without requiring a full manual re-index each time. Expect practical recommendations on extraction tools suited to your specific document types, strategies for detecting and handling duplicate or conflicting content across sources, approaches to incremental updating so your pipeline scales as your data grows, and guidance on preserving critical context like document hierarchy or cross-references that pure text extraction often discards. The assistant also addresses common ingestion pitfalls, such as PDFs with unreliable text extraction due to scanned images or complex layouts, wikis with outdated pages nobody has cleaned up, and databases where relevant information is scattered across many normalized tables. Results from working with this role typically include a concrete ingestion pipeline design, tool recommendations matched to your actual data sources, and a synchronization strategy so your RAG system stays accurate as underlying content changes. This role is essential for organizations building RAG systems over real internal knowledge bases rather than clean, curated datasets, and for teams who have discovered that data quality problems, not model or retrieval algorithm issues, are the real bottleneck to good answers.