Scaling Recommendation Systems with Request-Level Deduplication

Pinterest shares their technique of request-level deduplication to manage infrastructure costs when scaling recommendation systems with 100x increased model parameters.

• •Request-level deduplication eliminates redundant processing of user data across multiple items scored in recommendation pipeline
• •Apache Iceberg with user/request-based sorting achieves 10-50x storage compression on user-heavy feature columns
• •SyncBatchNorm fixes 1-2% regression from disrupted IID assumption when using request-sorted data by aggregating batch statistics across devices
• •User-level masking in loss function addresses false negatives (up to 30%) that arise from multiple user engagements grouped together in batches
• •Techniques apply across full ML lifecycle including storage optimization, training correctness and speedups, and serving throughput gains