Slack migrated 700+ SSH-based data pipeline jobs to REST-based architecture using Quarry and YARN Distributed Shell.
- •Eliminated security risks and operational overhead from direct SSH access to EMR clusters
- •Implemented YARN Distributed Shell to execute arbitrary shell commands in containers with proper resource management
- •Quarry provides unified REST API gateway with authentication, job submission, state tracking, and observability
- •Migration across 8 data regions maintained zero downtime with phased approach
- •Replaced stateful SSH connections with stateless REST API calls enabling pod restarts and reliable job tracking
This summary was automatically generated by AI based on the original article and may not be fully accurate.
Slack
Databricks
Airbnb