Updated November 2023 by Heidi


Pepperdata for Amazon EKS

While Kubernetes can reduce operating costs and make deployment more agile, it can also increase the complexity and cost of managing a dynamic and diverse combination of virtual machines, containers, and big data applications. 

Read the Solution Brief to discover how Pepperdata reduces the cost of running Spark applications at scale on Amazon EKS by up to 41.8% and optimizes resource utilization in real time for batch workloads like Spark as well as microservices.

Real-Time Optimization for Spark on EKS, Amazon EMR on EKS, and Microservices

Apache Spark on EKS

Pepperdata Capacity Optimizer Next Gen for Spark packs additional pending pods onto underutilized nodes, increasing node utilization and reducing the need for additional nodes—translating directly to reduced costs and increased performance.

Amazon EMR on EKS

Capacity Optimizer Next Gen delivers additional performance improvements and significant cost reductions for Amazon EMR workloads running on Amazon EKS by intelligently augmenting the native autoscaler to ensure all pods are fully utilized before additional pods are launched, eliminating waste and reducing costs by up to 42.5 percent.

Microservices Applications on EKS

Acting like a vertical pod autoscaler (VPA), Pepperdata Capacity Optimizer works with the Amazon EKS horizontal pod autoscaler (HPA) to align pod resource requests with actual usage so that pods, workloads, and nodes can be scaled more efficiently, leading to lower costs.

TPC-DS Benchmarking Results

Pepperdata found that for Spark Workloads running at scale on Amazon EKS, Capacity Optimizer Next Gen:

Pepperdata for KubeCon 2023 (1)
  • Group 2784

    Reduced Cost

    Decreased instance hours duration by 41.8%

  • Group 2764

    Increased Throughput

    Increased total workload run time by 45.5%

Pepperdata conducted its benchmarking using a 1 TB dataset on 500 nodes with 275 executors running on 10 parallel applications, and 99 queries TPC-DS jobs. TPC-DS is the Decision Support framework from the Transaction Processing Performance Council.  The benchmark was modeled on industry standard TPC-DS big data analytics benchmark. Pepperdata’s work is an unofficial benchmark as defined by TPC. 

Explore More

Looking for a safe, proven method to reduce waste and cost by up to 47% and maximize value for your cloud environment? Sign up now for a free savings assessment to see how Pepperdata Capacity Optimizer Next Gen can help you start saving immediately.