Managing multi-tenant big data clusters is complex. Pepperdata partners with you to deliver the following for your big data investments:
Pepperdata delivers proven products, operational experience, and deep expertise.
PHILIPS WELLCENTIVE
“Before Pepperdata, we experimented with various approaches to solve our performance issues, but we couldn’t see deep enough into the cluster. Pepperdata Platform Spotlight shined a bright light into our Hadoop environment and provided the detailed data that helped us isolate and resolve the problem.”
RUBICON PROJECT
“At Rubicon Project, having the appropriate visibility and insight into our Big Data applications is extremely important when delivering detailed reports to our clients and meeting our SLA. We challenged Pepperdata to find a solution to profile our applications before going to production that would help us maintain our SLA to our customers as we introduce new applications. Pepperdata listened to us and quickly understood the problem we were trying to address.”
CHARTBOOST
“Chartboost is the world’s largest mobile games-only advertising platform, reaching one billion active players around the world every month. Chartboost utilizes Apache Spark on large Amazon EC2 Hadoop clusters for machine learning and ETL workflows. Understanding Spark application performance in these complex environments is always a challenge. As a current user of Pepperdata Platform Spotlight, it has been great to work with Pepperdata on the development of the Application Spotlight self-service portal software. It will give us a comprehensive insight into Spark jobs.”
EXPEDIA
“The level of support and expertise that we receive from the Pepperdata team made a big difference to us. Pepperdata worked closely with us on our Cluster Analyzer implementation to ensure success. With Pepperdata Capacity Optimizer, the DevOps team runs more jobs, faster. We’ve seen a big performance boost across the cluster and have a much more efficient data footprint. Using the Pepperdata dashboard to see application-level metrics, unique custom views, resource utilization per workload drill-downs, and hardware utilization by various workgroups has significantly improved the way that we manage and troubleshoot.”
See who’s using Pepperdata Big Data performance solutions to achieve Big Data success.
Evaluating and purchasing a big data performance solutions can be complicated. We’ve made it easy to make sure you don’t miss anything with our Pepperdata Product and Solutions Guide.
See firsthand how Pepperdata solutions bring APM to every phase of the big data DevOps cycle with solutions for monitoring, tuning, troubleshooting, and improving collaboration between teams.
Schedule a demo today!
Extensive Enterprise Reporting Capabilities for Application Spotlight and Platform Spotlight and Expanded Professional Services Unveiled at Strata Data NYC
NEW YORK — September 11, 2018 (Strata Data New York 2018, Booth 741) — Pepperdata, the leader in Application Performance Management (APM) for big data success, announced enterprise-grade features to its APM suite that include auto-tuning, enhanced recommendations, and management and operational reporting, powered by an easy-to-use self-service interface. The company also announced professional services offerings that include best-practices, performance planning, capacity planning, and architecture design for big data success.
The company’s new professional services are directly enabled by the vast amount of metrics — 600 trillion data points every year — that Pepperdata collects from tens of thousands of nodes every few seconds. This data provides unique insight into all aspects of operationalizing big data applications. Pepperdata is unique in its ability to deliver not only enterprise-grade software, but also expertise, experience and knowledge that ensures big data success.
“Customers are demanding more than features and function from us — they’re asking us to become partners in making sure their big data investments yield business results,” said Ashfaq Munshi, Pepperdata CEO. “We are the only company offering expert services along with a solution delivering instantaneous time-series data that provides precise insight relevant to enterprise platforms and applications.”
Proven Products
The Pepperdata APM suite — comprised of Platform Spotlight and Application Spotlight — enables tight collaboration between developers and operators, improves overall efficiency and performance, and enables enterprises to do more with their existing big data investments.
Platform Spotlight provides infrastructure and capacity managers with:
Application Spotlight provides developers with:
Operational Experience and Deep Expertise
Pepperdata continuously monitors over 250 production clusters across its customer base — over 30,000 nodes across all Big Data distributions and hardware configurations — for a total 550 million jobs and 600 trillion data points every year. Coupled with its success serving Fortune 100 customers, this uniquely broad set of data empowers Pepperdata to help customers:
Helpful Links
About Pepperdata
Pepperdata is the leader in Application Performance Management solutions and services for big data success, solving application and platform issues throughout the stack for developers as well as capacity and infrastructure managers. The company partners with its customers to provide proven products, operational experience, and deep expertise to deliver predictable performance, empowered users, managed costs and managed growth for their big data investments, both on-premise and in the cloud. Leading companies like Comcast, Philips Wellcentive and NBC Universal depend on Pepperdata to deliver big data success. Founded in 2012 and headquartered in Cupertino, California, Pepperdata has attracted executive and engineering talent from Yahoo, Google, Microsoft and Netflix. Pepperdata investors include Costanoa Ventures, Signia Venture Partners, Silicon Valley Data Capital and Wing Venture Capital, along with leading high-profile individual investors. For more information, visit www.pepperdata.com.
###
Contact:
Samantha Leggat
samantha@pepperdata.com
Pepperdata and the Pepperdata logo are registered trademarks of Pepperdata, Inc. Other names may be trademarks of their respective owners.
Pepperdata Anticipates Doubling Team Within a Year to Support Increase in Seven-Figure Sales Deals
CUPERTINO, Calif. — September 4, 2018 — Pepperdata, the leader in Application Performance Management (APM) for big data success, announced the appointment of two executives reporting to CEO Ash Munshi. The appointments include Charles Marker as Vice President of Engineering and Dan Marx as Vice President of Sales. This announcement comes on the heels of the company’s latest funding, which Pepperdata will allocate to hiring and product development as it continues to deliver on feature requests from customers to support their mission-critical big data deployments.
Charles Marker joins Pepperdata as VP of Engineering from his previous position as Global Head of Engineering at Guidewire Software. Prior to Guidewire, Mr. Marker held Engineering VP positions at Kontagent, Yahoo, Qualcomm and Atheros. Dan Marx, who has been with Pepperdata since 2014, has been named VP of Sales. Mr. Marx brings deep expertise and experience in enterprise sales, including extensive success in big data technology sales at WANdisco and Zettaset.
“We are excited about the contagious enthusiasm and deep expertise Charles and Dan bring to Pepperdata,” said Mr. Munshi, Pepperdata CEO. “As we close more and more seven-figure deals, we are pleased to have the funding necessary to make appointments like these, and we will continue expanding to support the tremendous growth we’re experiencing. We anticipate more than doubling our team within a year.”
“Pepperdata is the leader in Application Performance Management for big data, delivering scalable solutions that enable Fortune 100 companies to achieve successful outcomes from their investments. We continue to be impressed with Pepperdata’s ability to facilitate adoption by these leading companies by identifying use cases that benefit from APM. We are pleased to work with them as they continue to execute their strategy,” said Jim McLean, Managing Director at Silicon Valley Data Capital.
“We were impressed to see the world’s biggest and best AI-driven companies already using Pepperdata so their Hadoop and Spark clusters perform at scale. Pepperdata helps ensure these global brands in e-commerce, voice applications and consumer banking optimize both the productivity and performance of their big data practices. Their continued team and company growth is exciting,” said Greg Sands, Managing Partner at Costanoa Ventures.
Since its founding in 2012, Pepperdata has established itself as a leader in APM for big data success, delivering proven products, operational experience, and deep expertise for its customers. Pepperdata is deployed at Fortune 100 companies in financial services, retail, healthcare, telecommunications and more, totaling more than 250 production clusters with 30,000 nodes spanning all big data distributions and hardware configurations. With the level of data the company collects — over 550 million jobs and 600 trillion data points annually — and its extensive global enterprise experience, Pepperdata is the wise choice for companies looking to get more value and optimal performance from their big data investments.
Pepperdata will be exhibiting at the Strata Data Conference at the Jacob Javits Center (booth 741) in New York City, September 12th and 13th.
Helpful Links
About Pepperdata
Pepperdata is the leader in Application Performance Management solutions and services for big data success, solving application and platform issues throughout the stack for developers as well as capacity and infrastructure managers. The company partners with its customers to provide proven products, operational experience, and deep expertise to deliver predictable performance, empowered users, managed costs and managed growth for their big data investments, both on-premise and in the cloud. Leading companies like Comcast, Philips Wellcentive and NBC Universal depend on Pepperdata to deliver big data success.
Founded in 2012 and headquartered in Cupertino, California, Pepperdata has attracted executive and engineering talent from Yahoo, Google, Microsoft and Netflix. Pepperdata investors include Costanoa Ventures, Signia Venture Partners, Silicon Valley Data Capital and Wing Venture Capital, along with leading high-profile individual investors. For more information, visit www.pepperdata.com.
###
Contact:
Samantha Leggat
Pepperdata and the Pepperdata logo are registered trademarks of Pepperdata, Inc. Other names may be trademarks of their respective owners.
CUPERTINO, Calif. — March 1, 2018 — Pepperdata, the Big Data Performance company that provides developer and operator solutions to optimize application and cluster performance in Big Data environments, announces Application Spotlight. This self-service portal enables Big Data application developers to generate application-specific recommendations to improve application performance, highlight applications that need attention, automatically identify bottlenecks, and alert on duration, failure conditions, and resource usage.
Application Spotlight helps developers to search for all applications running on the cluster or just the specific applications that they are interested in, compare current and previous runs, and visualize Spark applications and its stages for easy root cause failure analysis and performance tuning.
“Application Spotlight provides relevant application information, insights, and calls to action, all in one place, so that developers can easily and quickly perform these tasks,” said Ash Munshi, CEO of Pepperdata. “In addition to helping developers make jobs go faster, Application Spotlight enables them to be better tenants in multi-tenant clusters by showing them how to write optimal jobs and more efficiently use their queue and cluster resources with practical, innovative application performance management solutions. Application Spotlight enables developers to quickly understand performance impacts and get recommendations on how to better optimize their jobs.”
Big Data Performance Solutions for Both Developers and Operators
The Pepperdata product suite provides solutions for monitoring, tuning, troubleshooting, applications and clusters along with automated cluster optimization. Pepperdata allows enterprises to:
Pepperdata products improve collaboration between development and operations teams by providing both an application as well as cluster view of performance. Pepperdata products, are used to monitor and manage mixed workloads from frameworks such as Spark, MapReduce, Kafka, Tez, Solr, and Impala.
See a Demo of Application Spotlight at Strata San Jose
Pepperdata is sponsoring, speaking, and exhibiting at Strata Strata Data Conference on March 6–8 at the San Jose Convention Center. Please visit Pepperdata Booth to discuss your requirements and see a demonstration of Pepperdata Application Spotlight. For information about our Strata schedule, go to www.pepperdata/events.
Helpful Links
About Pepperdata
Pepperdata is the big data performance company. Leading companies such as Comcast, Philips Wellcentive, and NBC Universal depend on Pepperdata to manage and improve the performance of Hadoop and Spark. Enterprise customers use Pepperdata products and services to troubleshoot performance problems in production, increase cluster utilization, and enforce policies to support multi-tenancy. Pepperdata products and services work with customer Big Data systems both on-premise and in the cloud.
Founded in 2012, Pepperdata has raised $20M from investors including Citi Ventures, Signia Venture Partners and Wing Venture Capital, and attracted senior engineering talent from Yahoo, Google, Microsoft and Netflix. Pepperdata is headquartered in Cupertino, California. For more information, visit pepperdata.com.
###
Pepperdata and the Pepperdata logo are registered trademarks of Pepperdata, Inc. Other names may be trademarks of their respective owners.
Media Contact
Jim Dvorak
Offleash for Pepperdata
(415) 735-1622
pepperdata@offleashpr.com
Pepperdata makes Hadoop+YARN based systems better by providing total performance management (TPM) for big data. TPM is the combination of application performance management (APM) and operations performance management (OPM) in a single package so developers and operators can rely on the same underlying information to build and operate highly performant big data applications in multi-tenant clusters. For developers, the Application Spotlight self-service APM portal surfaces applications that require attention from a performance perspective. Application Spotlight provides precise recommendations to improve performance, automatically identifies bottlenecks and makes it easy to analyze root cause of errors and failures.
Application Spotlight provides a personalized experience via a dashboard that shows all of the developer’s applications, key performance indicators, and custom views in one place.
For operators, the OPM solution (Cluster Analyzer and the Capacity Optimizer add-on module) makes it easy to identify applications and users causing issues on the platform, proactively alert on those issues, and improve cluster performance. We include roll up reports for things like chargeback and capacity planning.
A summary view of the cluster giving the operator an ‘at-a-glance’ view of the cluster health, key performance indicators, and access to custom views.
The Pepperdata Capacity Optimizer add-on module can automatically add up to 50% more containers without any additional hardware by addressing some of the inefficiencies of how YARN does resource management today.
OPM Control (Capacity Optimizer) Adds up to 50% More Containers without any Additional Hardware
We are sometimes asked the question, doesn’t YARN already do this? Or, does Pepperdata replace YARN? The quick answers: YARN does not already do this and Pepperdata does not replace YARN or the ResourceManager, but it can significantly augment its capabilities.
YARN (“Yet Another Resource Negotiator”) was introduced as part of Hadoop 2.0 in 2012. YARN takes the resource management capabilities of MapReduce and packages them for use by new engines. YARN enables batch, interactive, and streaming jobs to run simultaneously on the same Hadoop cluster. This allows enterprises to deploy Hadoop for new and different applications and use cases. YARN coordinates consumption and usage reservations in an attempt to ensure resources are allocated fairly.
However, YARN does not track containers once they start running. This means that YARN must be conservative in its assumptions about memory usage and assume the worst case instead of monitoring and adjusting based on actual usage. The Pepperdata solution solves these problems by monitoring per-task hardware usage as jobs run and maximizing resource utilization.
The Capacity Optimizer is an optional add-on module for operators that uses active resource management to dynamically eliminate inefficiencies and bottlenecks without manual job or cluster tuning.
Working with the Pepperdata Cluster Analyzer OPM solution, Capacity Optimizer improves the capacity utilization of existing production clusters without manual tuning or intervention
At its core, YARN enables many different types of workloads to be run on Hadoop. However, YARN provides little to no resource management after jobs start running. Sometimes, YARN assumes that node memory utilization is high based on the static container reservations specified by developers’ run-time parameters rather than actual physical memory usage, thus leaving resources unused.
Capacity Optimizer picks up where YARN leaves off. Capacity Optimizer uses sophisticated, patented algorithms to track and predict the actual memory usage per container, allowing YARN to schedule more workload immediately. In effect, Pepperdata increases the amount of usable memory on the node made available to YARN. This proprietary advantage allows operators to achieve much higher hardware utilization. Typical enterprise deployments experience a 30-50% increase in throughput when Capacity Optimizer is enabled.
Operators who spend significant time tuning their Hadoop deployments may be skeptical of Capacity Optimizer’s ability to improve performance on a cluster that has already been tuned using industry-standard best practices. Capacity Optimizer identifies “holes” where a node can temporarily do more work and fills those holes with additional tasks, all while ensuring cluster reliability and safety. Capacity Optimizer automatically monitors and adjusts hardware resource usage at the process level in real time.
On a typical cluster, Capacity Optimizer makes hundreds or thousands of decisions per second. Even if Hadoop provided a mechanism to do so, the most talented dedicated operator or outside consultant could not make manual configuration changes with the precision and speed of Capacity Optimizer. Standard Hadoop configurations only affect up-front static resource reservations, so Hadoop must assume peak resource usage by every task, which typically wastes a significant amount of the cluster’s hardware resources. Additionally, YARN cannot engage in active resource management after container launch, except to kill jobs under certain conditions.
To hear me speak about this in a little more detail, please see this replay of a recent webinar on the same topic. Watch the replay here.
To see firsthand how Pepperdata can help you run more jobs on an existing cluster, run jobs faster on an existing cluster, and reduce or delay new hardware acquisition, sign up for the demo below.
Total Performance Management has two components:
The reason why those two components application and platform need to be thought about together is because your application is a distributed application it relies on what your platform is able to do as well. Those two components are interlinked. With classic APM, you didn’t need to worry about the fact that you have other things are running on this application or that the number of resources being used is massive. If you miscalculate the resources, you mess up the platform, and it costs a lot of money.
You have to look at the problem holistically. You can’t just look at your application. You must look at your application and also, you have to look at what it does to this shared resource, your cluster. The application and the platform have to be combined in order for you to understand performance. And that’s basically what we do – we look at both the application and platform and how they interact with each other to provide a holistic view of what is happening on the application and the platform. And you receive notifications, alerts, and useful recommendations or resource usage and tuning. And in this way, Pepperdata performance management solutions improve collaboration between operations and development teams because the combination of APM and OPM is presented cohesively in a single pane of glass so that both operators and developers receive the same information and it is interpreted in one way. The data exhaust informing all of the decisions being made is exactly the same from one unified system.
For developers to get the maximum performance out of their applications, this holistic view of everything that is going on is key. Without this holistic view, they cannot accomplish the business critically important things like meeting SLAs and meeting business requirements. The alternative, and the way that many are doing this today is they are receiving information from several different applications to get a piece of the puzzle, stitch all of the pieces together, and use their experience to make sense of how they all interact and how to manage it. And it’s a mess.
One of the common experiences that our customers share is that before using Pepperdata, they never got all of the information that would help them resolve an issue–they would only get small pieces of the information. While this information might help them understand the problem better, it wasn’t enough information to understand the issue and solve the problem.
This holistic view of application and platform for big data performance is how Pepperdata brings total performance management to big data.
For developers, the Application Spotlight self-service application performance management (APM) portal makes it easy to get recommendations and insights into how to optimize applications and the root cause of bottlenecks and failures.
For operators, Cluster Analyzer makes it easy to identify applications and users causing issues on the platform, proactively alert on those issues, and improve cluster performance. We also have roll up reports for things like chargeback and capacity planning. The Capacity Optimizer add-on module automatically increases cluster throughput 30-50% by addressing some of the inefficiencies of how YARN does resource management today.
This week, Pepperdata will be demonstrating, speaking, exhibiting (Booth #831) at Strata Big Data Conference in San Jose. Please come and ask us about our total performance management product solutions.
This blog is the first in a series that introduces big data developers to Pepperdata Application Summary. Application Summary is the first in a series of Pepperdata guided application performance management (APM) user experiences. In these experiences, we solve a particular user problem (or use case) by providing all the relevant information, insights, and calls to action in one place so that the user can perform these tasks easily and quickly.
What is Application Summary?
Before I start the tour, let me first introduce Application Summary, a self-service performance solution created for application developers of Spark, MapReduce, and other big data applications. When we talk about application performance, it could be in terms of running applications faster, using fewer resources, or, in the case of error resolution, mitigating these errors and quickly getting to the root cause. For developers who want to make their applications perform better, we target the following use cases:
Let’s start with finding applications by using the App Search function.
App Search
Based on user feedback, we simplified the search options so you can more easily search for all the applications running on your cluster or just specific ones that you are interested in. Either way, you can optionally specify a time range for your search, as well as an application’s full or partial name. If you want to narrow down your search to just one user or one queue, you can specify that as well. And, you can save your searches to use later so you don’t have to re-enter the same search criteria. Let’s see this in action. I’m going to specify “ScalaPageRank” as my specific app name, “prod” as the user name, and “root.prod” for the queue.
After I clicked the Find Matching Apps button, App Search returned eight results. The search criteria is displayed, and information is shown in a tabular form that can be sorted by columns. In this instance, we sorted by start time. You can compare the stats of any two runs of an app to see why one run took significantly longer than another or to see how the performance characteristics changed as the result of a small code or operational parameter change.
Alarms
Another APM feature of Application Summary is the ability to alert on duration or on peak memory usage. You can click the alarm icon in either column heading—I clicked on the duration alarm icon—and it opens a pane where I can set an alarm for future runs of this app, such as those that exceed a threshold of 25 minutes. This means that for any future runs of the app, if it takes more than 25 minutes to run, I receive an alert. If an application has an important SLA associated with it, you can use this feature to set associated alarms.
Application Summary
Returning to the tabular search results, let’s click the App ID for one of the ScalaTeraSort apps so we can take a look at its Application Summary. For this demonstration, I’m using ScalaTeraSort, a Spark application. There are three sections of Application Summary that I’ll discuss. After I do that, I’ll discuss Pepperdata recommendations as they relate to APM.
To start with, we have the header, which gives the app name, app type, the user who ran it. We answer questions like, “In which queue did the app run?” and “How long did it run?” The header also answers the question, “how much resources are you consuming?” In this case, the app took 87 percent of the cluster memory, and it held it for 18 minutes. It also took 53 percent of the CPU, and consumed that for 18 minutes.
The second section of Application Summary, Issues, gathers all the issues related to the app into one place and provides actionable recommendations for improving performance. By “issues”, we mean alarms, bottlenecks, and status and error information specific to the type of app, such as Spark or MapReduce. By working through the tabs from left-to-right, following the recommendations and addressing the root causes of the identified bottlenecks and app failures, you can address all aspects of APM.
Recommendations
So let’s start with the recommendations tab. Our aim is to provide very specific advice that is understandable for users. For example, in the screenshot above, we show you that your application experienced a Spark executor shuffle read bytes skew and that you need to increase the number of data partitions. We also tell you that you can achieve this by using the RDD repartition transformation or by decreasing the cluster’s dfs.block.size value.
The third section of the Application Summary shows stats—the underlying data for the second section’s recommendations and issues. In the Resource Usage tab, the first thing that we show is how much memory is being wasted by the application. Right now the severity levels correspond to hard-coded thresholds, but our goal is to configure the threshold for your particular environment so that you know whether this application falls within the norm of all of the other applications that you’re running on your cluster. Next, we provide charts that show resource usage over the lifetime of the app’s run. For example, in the Memory Used by Type, we break down the memory by total, heap, non-heap, and new I/O, which is of particular interest to Spark developers. So if your app asks YARN for a portion of cluster or queue memory, we’ll tell you how much was allocated in terms of memory and CPU, and how much was actually used.
Another chart in the Resource Usage tab is App Container Asks, which provides insight into the lifecycle of your application: “What’s the backlog of these asks, and what is running?” If you have a significant backlog, you know that that app is going to be constrained and therefore take longer to run. In this example, there was very little backlog, and the app got the majority of its containers as soon as it asked for them. The other tab in the Stats section is App History. It provides key metrics such as runtime duration and peak memory usage for the app’s current run and five previous runs.
Bottlenecks
Returning to the Issues section of Application Summary, I’d like to briefly walk through the Bottlenecks tab to talk about the types of bottlenecks that can occur for an application:
In this example, the app ran for just over ten minutes. But almost all of that time (99.12 %) was spent just waiting. So this bottleneck is highlighted in red in the Bottlenecks tab. Now that we’ve seen how to display bottlenecks that affect your app performance, let’s look at app status and, specifically, information about failures. Next, I’m going to show you a Spark application.
It’s important to note that as far as YARN is concerned, this app finished with the same successful status, COMPLETED, as the previous examples. However, when we look at the job history, we see that the app consisted of one job that failed. Which essentially means that the application failed. The Spark tab in the Issues section summarizes the failures.
In this case, there were 65 failures, which we break down by jobs, stages,executors, and tasks. It’s much easier to determine the root cause of the job failures by using this contextual breakdown instead of navigating through the Spark Web UI and analyzing the many log files. In addition, we’ve translated the complicated stack traces to simple English, which is much easier to act on. In this case, the job failed because a stage failed four consecutive times, and the stage failed because an executor it was relying on failed. This example showed how much easier it is to diagnose a Spark failure by using Application Summary than to work directly with the stack traces from Spark. And when you’ve used the Application Summary to trace a Spark application’s failure down to the root cause, you can use Pepperdata Code Analyzer for Apache Spark to further diagnose such failures and resolve them.
Thank you for taking this tour of Pepperdata Application Summary. To recap, we demonstrated:
Pepperdata works closely with customers and understand their unique requirements to provide the best user experience and improve our products. Please look for upcoming blogs in this series. I look forward to working with you and making our products better and more useful and valuable to you.
Things that you can do next:
