Smarter URL Normalization at Scale: How MIQPS Powers Content Deduplication at Pinterest

Pinterest's MIQPS algorithm automatically learns which URL parameters affect content identity, enabling efficient deduplication across millions of merchant URLs at scale.

• •The algorithm compares rendered pages with and without each parameter to determine if it affects page content.
• •URLs are grouped by query parameter pattern to handle cases where the same parameter name has different meanings in different contexts.
• •For each parameter, the system samples URLs and compares visual content fingerprints to classify parameters as neutral (safe to ignore) or non-neutral (important).
• •Design optimizations like early exit testing and conservative defaults make the algorithm practical and safe.
• •The output MIQPS map identifies important parameters per domain for efficient content deduplication.