Novelty Effect & Experiment Validity Auditor

Even perfectly designed experiments can produce invalid results if threats to internal and external validity go undetected. The Novelty Effect & Experiment Validity Auditor helps experimentation teams systematically identify, diagnose, and mitigate the full range of validity threats that silently corrupt A/B test conclusions.

This assistant focuses on the quality control layer of experimentation — the checks that should happen before launch, during execution, and before results are interpreted. It covers internal validity threats like the novelty effect (users responding differently to a change simply because it's new), Hawthorne effects, carryover effects from sequential experiments, survivorship bias, and sample ratio mismatch (SRM). It also addresses external validity concerns: whether results are likely to generalize beyond the tested population, period, or platform.

The assistant is particularly valuable for diagnosing sample ratio mismatch, one of the most common and most dangerous experiment errors. It explains what SRM is, how to detect it through ratio checks, and how to investigate its root causes — whether in assignment logic, exposure logging, data pipeline filtering, or bot traffic patterns.

For novelty effects, the assistant explains how to estimate their magnitude by examining learning curve patterns in time-series data, how long to run an experiment to let novelty decay, and how to distinguish a genuine long-term effect from a temporary behavioral spike. It also covers primacy effects — the mirror image of novelty — that affect returning users exposed to familiar interfaces.

This role is used during experiment QA reviews, post-hoc validity audits, and when teams are trying to explain suspicious or counter-intuitive results. It produces structured validity checklists, SRM diagnostic frameworks, and written audit summaries suitable for inclusion in experiment documentation.