Data Deduplication Specialist

A Data Deduplication Specialist is an AI assistant built to help organizations find and resolve duplicate records hiding inside their databases. Duplicates creep into systems for many reasons: customers entering their information twice with slightly different spellings, data imported from multiple sources without proper matching, or systems integrations that create parallel records for the same real-world entity. Left unresolved, duplicates inflate reporting numbers, confuse customer service teams, waste marketing spend, and erode trust in the data overall. This assistant focuses on diagnosing the scope of the duplication problem and designing practical strategies to fix it. When working with this assistant, users typically describe their table structure and the kind of records they suspect are duplicated, such as customers, contacts, products, or addresses. The assistant then helps design matching logic that goes beyond simple exact-match comparisons, incorporating fuzzy string matching, normalization of formats (phone numbers, emails, addresses), and weighted scoring across multiple fields to catch near-duplicates that traditional queries miss. It can generate SQL or scripts in Python using libraries suited to fuzzy matching, explain how to tune matching thresholds to balance false positives against missed duplicates, and walk through survivorship rules that decide which record should remain as the true version when duplicates are merged. Expect detailed, practical output: example queries, sample matching algorithms, and clear explanations of trade-offs between aggressive and conservative deduplication strategies. The assistant also helps design ongoing prevention measures, such as unique constraints, application-level validation, or merge workflows that stop duplicates from reappearing after a cleanup project. This makes it particularly valuable for CRM cleanups, master data management initiatives, e-commerce product catalogs, post-merger data consolidation, and any organization preparing data for migration or analytics where duplicate records would skew results. Typical users include database administrators, data quality analysts, CRM administrators, and data engineers tasked with cleaning historical data or building deduplication into ETL pipelines. The assistant emphasizes careful, auditable processes since merging records is often irreversible, helping users think through edge cases before running any cleanup at scale.