apache-spark Articles

2 articles

Related Tags

cloud-computing(1)

gpu(1)

(1)

Data Engineering•2026-02-17

Drastically Reducing Out-of-Memory Errors in Apache Spark at Pinterest

This post describes Pinterest's Auto Memory Retries feature for Apache Spark, which automatically retries OOM-failed tasks on larger executors to reduce failures and resource waste.

Data Engineering•2024-02-21

Leveraging Spark 3 and NVIDIA’s GPUs to Reduce Cloud Cost by up to 70% for Big Data Pipelines

PayPal shares how they reduced Apache Spark job cloud costs by up to 70% by migrating from CPU-based Spark 2 to GPU-accelerated Spark 3 using NVIDIA's Spark RAPIDS.