What is the most common practical bottleneck when scaling a big data pipeline?
- A Network shuffle and data movement between nodes
- B CPU clock speed
- C Disk capacity
- D Number of columns
Answer
Network shuffle and data movement between nodes
Reducing shuffles through broadcast joins, better partitioning and pre-aggregation usually yields far more than adding hardware.





