While basic autoscaling from cloud service providers provides some elasticity, it can also lead to runaway costs. Autoscaling from cloud providers is based on the workload peak needs. This guarantees the maximums are met, but leads to wasted resources and inflated cloud costs. You need a way to better optimize your resources and keep costs in line.
Capacity Optimizer makes thousands of decisions per second, analyzing the resource usage of each node to optimize CPU, memory, and I/O resources. The net effect is that horizontal scaling is optimized and waste is eliminated. Pepperdata provides automated deployment options that can be seamlessly added to EMR, Dataproc, and Qubole deployments.