How Pinterest Built a Real‑Time Radar for Violative Content using AI

Pinterest built a real-time AI-assisted system to measure the prevalence of policy-violating content by tracking what users actually see, not just what they report.

• •Prevalence is defined as the share of daily user impressions that include policy-violating content, reported with 95% confidence intervals
• •ML-assisted weighted reservoir sampling prioritizes high-risk, high-impression content while using inverse-probability weighting to keep estimates unbiased
• •A multimodal LLM (vision + text) labels sampled content at scale, running 15x faster and at significantly lower cost than human-only review
• •LLM quality is validated against human-labeled gold sets and monitored for model drift to maintain alignment with current policy
• •The system breaks down prevalence by policy area, sub-policy, surface (Homefeed, Search, etc.), geography, and user age buckets