Cloudwick Collaborates with Pepperdata to Ensure SLAs and Performance are Maintained for AWS Migration Service

Pepperdata Provides Pre- and Post-Migration Workload Analysis, Application Performance Assessment and SLA Validation for Cloudwick AWS Migration Customers

San Francisco — Strata Data Conference (Booth 926)  — March 27, 2019 — Pepperdata, the leader in big data Application Performance Management (APM), and Cloudwick, leading provider of digital business services and solutions to the Global 1000, today announced a collaborative offering for enterprises migrating their big data to Amazon Web Services (AWS). Pepperdata provides Cloudwick with a baseline of on-premises performance, maps workloads to optimal static and on-demand instances, diagnoses any issues that arise during migration and assesses performance after the move to ensure the same or better performance and SLAs.

“The biggest challenge for enterprises migrating big data to the cloud is ensuring SLAs are maintained without having to devote resources to entirely re-engineer applications,” said Ash Munshi, Pepperdata CEO. “Cloudwick and Pepperdata ensure workloads are migrated successfully by analyzing and establishing a metrics-based performance baseline.”

“Migrating to the cloud without looking at the performance data first is risky for organizations and if a migration is not done right, the complaints from lines of business are unavoidable,” said Mark Schreiber, General Manager for Cloudwick. “Without Pepperdata’s metrics and analysis before and after the migration, there is no way to prove performance levels are maintained in the cloud.”

For Cloudwick’s AWS Migration Services, Pepperdata is installed on customers’ existing, on-premises clusters — it takes under 30 minutes — and automatically collects over 350 real-time operational metrics from applications and infrastructure resources, including CPU, RAM, disk I/O, and network usage metrics on every job, task, user, host, workflow, and queue. These metrics are used to analyze performance and SLAs, accurately map workloads to appropriate AWS instances, and provide cost projections. Once the AWS migration is complete, the same operational metrics from the cloud are collected and analyzed to assess performance results and validate migration success.

To learn more, stop by the Pepperdata booth (926) at Strata Data Conference March 25-28 at Moscone West in San Francisco.

More Info

• For more information on the offering, email sales@cloudwick.com or sales@pepperdata.com.
• Learn about Pepperdata’s free Big Data Cloud Migration Cost Assessment.
• Start a free Big Data Cloud Migration Cost Assessment.
• Pepperdata website: https://pepperdata.com

About Pepperdata
Pepperdata (https://pepperdata.com) is the leader in big data Application Performance Management (APM) solutions and services, solving application and infrastructure issues throughout the stack for developers and operations managers. The company partners with its customers to provide proven products, operational experience, and deep expertise to deliver predictable performance, empowered users, managed costs and managed growth for their big data investments, both on-premise and in the cloud. Leading companies like Comcast, Philips Wellcentive and NBC Universal depend on Pepperdata to deliver big data success.

 Founded in 2012 and headquartered in Cupertino, California, Pepperdata has attracted executive and engineering talent from Yahoo, Google, Microsoft and Netflix. Pepperdata investors include Citi Ventures, Costanoa Ventures, Signia Venture Partners, Silicon Valley Data Capital and Wing Venture Capital, along with leading high-profile individual investors. For more information, visit www.pepperdata.com.

About Cloudwick

Cloudwick is the leading provider of digital business services and solutions to the Global 1000. Its solutions include data migration, business intelligence modernization, data science, cybersecurity, IoT and mobile application development and more, enabling data-driven enterprises to gain competitive advantage from big data, cloud computing and advanced analytics. Learn more at www.cloudwick.com.

###

Contact:
Samantha Leggat
samantha@pepperdata.com

Pepperdata and the Pepperdata logo are registered trademarks of Pepperdata, Inc. Other names may be trademarks of their respective owners.

Pepperdata Announces Free Big Data Cloud Migration Cost Assessment to Automatically Select Optimal Instance Types and Provide Accurate Cost Projections

Pepperdata Eliminates Guesswork and Complexity Associated with Identifying Best Candidate Workloads Down to Queue, Job and User Level, for Moving to AWS, Azure, Google Cloud or IBM Cloud

CUPERTINO, Calif. — March 6, 2019 — Pepperdata, the leader in big data Application Performance Management (APM), today announced its new Big Data Cloud Migration Cost Assessment for enterprises looking to migrate their big data workloads to AWS, Azure, Google Cloud or IBM Cloud. By analyzing current workloads and service level agreements, the detailed, metrics-based Assessment enables enterprises to make informed decisions, helping minimize risk while ensuring SLAs are maintained after cloud migration.

The Pepperdata Big Data Cloud Migration Cost Assessment provides organizations with an accurate understanding of their network, compute and storage needs to run their big data applications in the hybrid cloud. Analyzing memory, CPU and IO every five seconds for every task, Pepperdata maps the on-premises workloads to optimal static and on-demand instances on AWS, Azure, Google Cloud, and IBM Cloud. Pepperdata also identifies how many of each instance type will be needed and calculates cloud CPU and memory costs to achieve the same performance and SLAs of the existing on-prem infrastructure.

“When enterprises consider a hybrid cloud strategy, they estimate the cost of moving entire clusters, but that’s not the best approach,” said Ash Munshi, Pepperdata CEO. “It’s far better to identify specific workloads that can be moved to take full advantage of the pricing and elasticity of the cloud. Pepperdata collects and analyzes detailed, granular resource metrics to accurately identify optimal workloads for cloud migration while maintaining SLAs.”

The Big Data Cloud Migration Cost Assessment enables enterprises to:

• Automatically analyze every workload in your cluster to accurately determine their projected cloud costs
• Get cost projections and instance recommendations for workloads, queues, jobs, and users
• Map big data workloads to various instance types including static and on-demand
• Compare AWS, Azure, Google Cloud, and IBM Cloud

Availability

Pepperdata Big Data Cloud Migration Cost Assessment is available free at pepperdata.com/free-big-data-cloud-migration-cost-assessment. Pepperdata customers should email support@pepperdata.com for their free assessment.

Learn more:

• About Pepperdata Big Data Cloud Migration Assessment: https://www.pepperdata.com/big-data-cloud-migration
• Download a sample: https://www.pepperdata.com/big-data-cloud-migration-cost-assessment-sample
• Start an Assessment: https://www.pepperdata.com/free-big-data-cloud-migration-cost-assessment

About Pepperdata
Pepperdata (https://www.pepperdata.com) is the leader in big data Application Performance Management (APM) solutions and services, solving application and infrastructure issues throughout the stack for developers and operations managers. The company partners with its customers to provide proven products, operational experience, and deep expertise to deliver predictable performance, empowered users, managed costs and managed growth for their big data investments, both on-premise and in the cloud. Leading companies like Comcast, Philips Wellcentive and NBC Universal depend on Pepperdata to deliver big data success.

 Founded in 2012 and headquartered in Cupertino, California, Pepperdata has attracted executive and engineering talent from Yahoo, Google, Microsoft and Netflix. Pepperdata investors include Citi Ventures, Costanoa Ventures, Signia Venture Partners, Silicon Valley Data Capital and Wing Venture Capital, along with leading high-profile individual investors. For more information, visit www.pepperdata.com.

###

Contact:
Samantha Leggat

925-447-5300
samantha@pepperdata.com

Pepperdata and the Pepperdata logo are registered trademarks of Pepperdata, Inc. Other names may be trademarks of their respective owners.

Pepperdata Unveils 360° Reports, Enabling Enterprises to Make More Informed Operational Decisions to Maximize Capacity and Improve Application Performance

360° Reports Empower Executives to Better Understand Financial Impacts of Operational Decisions

CUPERTINO, Calif. — February 19, 2019 — Pepperdata, the leader in big data Application Performance Management (APM), today announced the availability of 360° Reports for Platform Spotlight. Pepperdata 360° Reports leverage the vast amount of proprietary data collected and correlated by Pepperdata to give executives capacity utilization insights so they better understand the financial impacts of operational decisions.

“Pepperdata 360° Reports demonstrate the power of data and the valuable insights Pepperdata provides, enabling enterprises to make more informed and effective operational decisions,” said Ash Munshi, Pepperdata CEO. “Operators get a better understanding of what and where they’re spending, where waste can be reclaimed, and where policy and resource adjustments can be made to save money, maximize capacity and improve application performance.”

360° Reports for Pepperdata Platform Spotlight include:

• Capacity Optimizer Report: This gives operators insight into memory and money saved by leveraging Pepperdata Capacity Optimizer to dynamically recapture wasted capacity.
• Application Waste Report: This report compares memory requested with actual memory utilization so operators can optimize resources by changing resource reservation parameters.
• Application Type Report: This gives operators insight on the technologies used across the cluster and the percentage of each (percentage of Spark jobs, etc.). This provides executives with insights into technology trends to make more data-driven investment decisions.
• Default Container Size Report: This report identifies jobs using default container size and where any waste occurred so operators can make default container size adjustments to save money.
• Pepperdata Usage Report: This presents Pepperdata dashboard usage data, highlighting top users, days used, and more to give operators insights to maximize their investment. With this data, operators can identify activities to grow the user base, such as promoting features, scheduling onboarding sessions, and training on custom alarms.

Availability

Pepperdata 360° Reports are available immediately for Pepperdata Platform Spotlight customers. For a free trial of Pepperdata, visit https://www.pepperdata.com/trial.

About Pepperdata
Pepperdata (https://pepperdata.com) is the leader in big data Application Performance Management (APM) solutions and services, solving application and infrastructure issues throughout the stack for developers and operations managers. The company partners with its customers to provide proven products, operational experience, and deep expertise to deliver predictable performance, empowered users, managed costs and managed growth for their big data investments, both on-premise and in the cloud. Leading companies like Comcast, Philips Wellcentive and NBC Universal depend on Pepperdata to deliver big data success.

 Founded in 2012 and headquartered in Cupertino, California, Pepperdata has attracted executive and engineering talent from Yahoo, Google, Microsoft and Netflix. Pepperdata investors include Citi Ventures, Costanoa Ventures, Signia Venture Partners, Silicon Valley Data Capital and Wing Venture Capital, along with leading high-profile individual investors. For more information, visit www.pepperdata.com.

###

Contact:
Samantha Leggat
samantha@pepperdata.com

Pepperdata and the Pepperdata logo are registered trademarks of Pepperdata, Inc. Other names may be trademarks of their respective owners.

Sample report attached.

Sample Capacity Optimizer Report – memory and money saved with Capacity Optimizer

The 3 Main Challenges of Hadoop, and How Pepperdata Solves Them

Most companies today rely on Hadoop for their big data processing needs. However, such a sophisticated platform can still present pitfalls and challenges that DevOps teams grapple with almost every day.

These challenges can usually be summed up into three main issues. Continue reading to find out what these are, and see how Pepperdata takes care of these problems. 

Challenge #1: The Difficulty in Finding Root Cause of Problems

Picture this: a production job is running late, and users are complaining. And so, you check the monitoring system, asking yourself, “Is a node down? Is a cluster busy?” 

But you can’t tell what’s happening. So you ask around for anyone who might have done anything differently. No one has anything to offer because, most probably, everyone else is as clueless as you are. You can’t even replicate the problem, much less find what started it. And so you decide to let Hadoop operations run while you watch and wait, hoping and praying that the next time the problem reappears, you’re there to witness it.

Unsurprisingly, the problem resurfaces. Hurrah! Now you can jump in to do some node debugging. Only… you still don’t see where the problem is coming from. You’re not even sure which resources to tweak. Still, you make adjustments here and there, then you sit back and play the waiting game again. Eventually, you decide you’ll need to buy extra hardware just to buy yourself some peace. But really, in the back of your mind, there’s a nagging worry that this will happen again.

Now did that feel like a day in your life? It probably did.

But with Pepperdata solutions, you get visibility from all corners. Every five seconds, Pep Agents collect more than 350 real-time operational metrics from applications and infrastructure resources, including CPU, RAM, disk I/O, and network usage metrics on every job, task, user, host, workflow, and queue. 

The user doesn’t need to sift through this data; Pepperdata swiftly derives insights from the metrics to solve the problem at hand. This gives you full 360-degree visibility into application performance within the context of your big data infrastructure, whether on-premises or in cloud/hybrid environments.

Challenge #2: Inefficient Cluster Utilization

Let’s face it: production clusters struggle to fully utilize their full capacity given the constraints of the YARN scheduler. No matter how tuned your Hadoop is, these clusters are sized for peak SLA, and with Hadoop’s allocations being predefined and static, a lot of computing capacity gets wasted.

And what about ad-hoc jobs? They sit in their own production cluster queue, until someone runs a rogue job, using more resources than a given queue can handle, and causing other jobs to run late.

The more jobs and workloads you have, the more clusters you set up to isolate them. More clusters = more wasted capacity.

Not with Pepperdata Capacity Optimizer. Capacity Optimizer reclaims wasted capacity by leveraging patented, machine learning based automation.This allows you to use the true capacity of your hardware.

Challenge #3: The business impact of Hadoop inefficiencies

Problems that arise in Hadoop create major consequences for the business – especially on the financial side.

• A key customer-facing web feature not performing can lose the company up to $10,000/hour.
• Unavailable real-time ad impression data can lose you up to $5,000/minute.
• For every half-hour that problems remain unfixed, up to 150 customers leave your app.
• Hadoop operators can waste thousands of dollars worth of productivity per day backtracking and tracing the root cause of problems.
• Sluggish or offline external-facing reporting systems can bring your customer satisfaction down and drive churn.
• Consequently, late and/or unavailable reports can make you lose $5,000 worth of productivity per hour, and this cascades throughout the organization.

Pepperdata can help you with curbing these loses and more. Proactive application monitoring with Pepperdata Platform Spotlight helps raise the flag on symptoms before they become problems, from finding sluggish queries to identifying high volume requests that should be optimized. Companies can save annual costs of up to 70%, assuming at least a 70% reduction in mean-time-to-repair (MTTR).

This all comes to life in a recent case study of ours. A leading Fortune 100 multinational investment bank and financial services corporation, headquartered in the U.S., needed an intelligent solution to address the problem of runaway data center growth and constantly increasing spend on compute infrastructure. 

With the help of Capacity Optimizer, they reduced their infrastructure spend by nearly 30% by recapturing wasted resources, maximizing the capacity of their existing infrastructure, and ensuring that workloads performed optimally. For this customer, the approximate 30% improvement in server utilization and capacity is equivalent to 100s of nodes, significantly reducing the number of additional servers needed to support continued growth.

Whatever issues you may have with Hadoop, Pepperdata can easily integrate into your system and solve them. It only takes about 20 minutes to set up. You can even use your existing Hadoop, with its existing schedulers and with no change to job submission. Request demo here.

How to Overcome the Five Most Common Spark Challenges

Everyone in the Big Data world knows Spark. It’s an amazing, powerful piece of software that runs workloads 100x faster than other methods.

However, fantastic as Spark is, like all software, it has its challenges. In a recent webinar, we sat down with Alexander Pierce, a Pepperdata Field Engineer, to discuss them. Alex drew on his experiences across dozens of production deployments, and pointed out the best ways to overcome the five most common Spark challenges.

Serialization is Key

Serialization plays an important role in the performance of any distributed application. Formats that are slow to serialize objects into, or those that consume a large number of bytes, will greatly slow down the computation.

“Because you’ll have to distribute your codes for running and your data for execution, you need to make sure that your programs can both serialize, deserialize, and send objects across the wire quickly,” Alex explains. This will often be the first thing you should tune to optimize a Spark application. Furthermore, Alex recommends the use of the Kryo serializer, because Java’s default serializer has mediocre performance with respect to runtime, as well as the size of its results. 

Getting Partition Recommendations and Sizing to Work for You

Generally speaking, any performance management software that sees data skew will recommend more partitions, but not too many more. “The more partitions you have, the better your sterilizations could be,” says Alex. But that’s not always the case. 

The best way to decide on the number of partitions in an RDD is to equate the number of partitions to a multiple of the number of cores in the cluster. This is so that all the partitions will process in parallel and the resources receive optimum utilization. Alex further suggests that you’ll want to avoid a situation where you have four executors and five partitions.

Monitoring Both Executor Size, And Yarn Memory Overhead

Often, what you’re trying to do is subdivide your data set into the smallest pieces that can be easily consumed by your Spark executors, but you don’t want them to be too small. There are a few ways to find that happy middle ground, but you’ll have to find a way around data skew by ensuring a well-distributed key space.

“Make a guess at the size of your executor based on the amount of data you expect to be processed at any one time,” Alex says. “Know your data set, know your partition count.” However, that’s not everything there is to it. There are two values in Spark on YARN to keep an eye on: the size of your executor, and the YARN memory overhead. This is to prevent the YARN scheduler from killing an application that uses a large amount of NIO memory or other off-head memory areas.

Getting the Most out of DAG Management

It’s always a good idea to keep an eye on the complexity of the execution plan. Use the DAG (directed acyclic graph) Visualization tool that comes with SparkUI for one possible visual map. If something that you think should be straightforward (a basic join, for example) is taking 10 stages, you can look at your query or code and perhaps reduce it to two or three stages.

Alex offers one solid tip: “Look at each of the stages in the parallelization.” he says. “Keep an eye on your DAG, not just on the overall complexity. Make sure each stage in your code is actually running in parallel.” If you have a (non-parallel) stage using less than 60% of the available executors, the questions to keep in mind are: Should that compute be rolled into another stage? Is there a separate partitioning issue?

Managing Library Conflicts

When it comes to shading, one quick tip from Alex is to make sure that any external dependencies and classes you bring in are available in the environment you are using, and that they do not conflict with internal libraries used by your version of Spark. A specific example of such is the use of Google Protobuf, a popular binary format for storing and transporting data more compact than JSON.

Watch the webinar to learn more about these Spark challenges and how to overcome them, or start your Pepperdata free trial now and see how we can solve these challenges for you!

Data Correlation: The Key to Optimal Cluster Performance

Modern data analytics platforms such as Hadoop and Spark have become central to many Fortune 1000 businesses. As critical components for digital business success, they produce insights that can only be obtained by analyzing the massive amounts of data. The continued growth of these systems can mean that an enterprise may be running 100,000 applications a day on 1,000 nodes, and servicing over 2,000 users. Challenges at this massive scale include piecing together performance data from those applications and the infrastructure they are running on. 

There’s no foreseeable end to the relentless growth of users and applications. So how do you address performance management problems and end the headache of constant manual tuning? 

The answer: deploy a solution that automatically correlates both application and infrastructure performance data allowing you to be laser-focused in your efforts to improve performance. This solution must go beyond standard monitoring and provide real actionable insights. 

Auto-Correlate Infrastructure and Application Performance Events

It’s much easier to resolve bottlenecks and failures when you have rich contextual information that traverses infrastructure and application performance. 

Application performance management helps developers improve application and query performance within the context of cluster operations. This also supports better organizational alignment with IT Operations. Within today’s enterprise environment, it’s critical that the process is automated. Manual tuning is not an option.

With detailed application/workload metrics, IT Operations can quickly identify and troubleshoot infrastructure issues within such an environment, optimize related cluster resources, and quickly resolve performance problems. Streamlining this process is essential to successfully scaling analytics environments to meet the business’ needs.

Application and Infrastructure Correlation Requires a Holistic Approach

Gaining visibility across your distributed system means correlating and visualizing metrics to quickly pinpoint and resolve issues. This requires a holistic approach, one that looks at how your applications interact within the context of your big data infrastructure. 

Pepperdata solutions provide that holistic strategy, allowing a view of your cluster resources and delivering context-aware application tuning recommendations. You get a unified operational view, real-time granular data, and historical references to optimize application performance and resource utilization.

The solutions also make it easy to quickly see whether an application, the infrastructure, or a combination of both are contributing to the latency of your workloads. A 360-degree view of all your performance data in one dashboard lets you gauge performance, diagnose issues up to 90% faster, and improve the overall efficiency of your entire cluster. 

Furthermore, Pepperdata provides intelligent tuning recommendations for improving on application performance so you can better allocate and utilize resources by pinpointing exactly what specific resources each application requires. This application right-sizing also improves your ability to only deploy on-premise and cloud resources that are needed to support a given workload.

Further Your Efficiency with Pepperdata Application Spotlight and Pepperdata Platform Spotlight 

Application Spotlight is a self-service portal that provides 360-degree visibility and insights into your Hadoop and Spark application performance data, along with powerful APM tools. Developers get useful, actionable recommendations that eliminate the time-consuming “try-test-repeat” processes.  

Meanwhile, Platform Spotlight continuously monitors and collects unique data from all relevant hardware and execution framework sources, providing a 360-degree cluster view that enables IT Operations to quickly diagnose performance issues and make resource decisions based on user priorities and needs. It also enables alerting to identify the root cause of problems before your users and applications are impacted.  

Pepperdata solutions correlate metrics to provide you with rich telemetry data and actionable insights to monitor and manage the performance of your entire cluster. Tap into the unmatched experience and expertise of Pepperdata to:

• Get real-time visibility into resource utilization, along with the ability to recapture wasted resources and optimize capacity utilization.
• Easily troubleshoot difficult issues and automatically optimize your big data application and infrastructure performance, both in the cloud and on-premise.
• Get detailed tuning recommendations for each of your applications, derived from hundreds of  metrics that Pepperdata tracks in real time.