Cloudwick Collaborates with Pepperdata to Ensure SLAs and Performance are Maintained for AWS Migration Service

Pepperdata Provides Pre- and Post-Migration Workload Analysis, Application Performance Assessment and SLA Validation for Cloudwick AWS Migration Customers

San Francisco — Strata Data Conference (Booth 926)  — March 27, 2019 — Pepperdata, the leader in big data Application Performance Management (APM), and Cloudwick, leading provider of digital business services and solutions to the Global 1000, today announced a collaborative offering for enterprises migrating their big data to Amazon Web Services (AWS). Pepperdata provides Cloudwick with a baseline of on-premises performance, maps workloads to optimal static and on-demand instances, diagnoses any issues that arise during migration and assesses performance after the move to ensure the same or better performance and SLAs.

“The biggest challenge for enterprises migrating big data to the cloud is ensuring SLAs are maintained without having to devote resources to entirely re-engineer applications,” said Ash Munshi, Pepperdata CEO. “Cloudwick and Pepperdata ensure workloads are migrated successfully by analyzing and establishing a metrics-based performance baseline.”

“Migrating to the cloud without looking at the performance data first is risky for organizations and if a migration is not done right, the complaints from lines of business are unavoidable,” said Mark Schreiber, General Manager for Cloudwick. “Without Pepperdata’s metrics and analysis before and after the migration, there is no way to prove performance levels are maintained in the cloud.”

For Cloudwick’s AWS Migration Services, Pepperdata is installed on customers’ existing, on-premises clusters — it takes under 30 minutes — and automatically collects over 350 real-time operational metrics from applications and infrastructure resources, including CPU, RAM, disk I/O, and network usage metrics on every job, task, user, host, workflow, and queue. These metrics are used to analyze performance and SLAs, accurately map workloads to appropriate AWS instances, and provide cost projections. Once the AWS migration is complete, the same operational metrics from the cloud are collected and analyzed to assess performance results and validate migration success.

To learn more, stop by the Pepperdata booth (926) at Strata Data Conference March 25-28 at Moscone West in San Francisco.

More Info

• For more information on the offering, email sales@cloudwick.com or sales@pepperdata.com.
• Learn about Pepperdata’s free Big Data Cloud Migration Cost Assessment.
• Start a free Big Data Cloud Migration Cost Assessment.
• Pepperdata website: https://pepperdata.com

About Pepperdata
Pepperdata (https://pepperdata.com) is the leader in big data Application Performance Management (APM) solutions and services, solving application and infrastructure issues throughout the stack for developers and operations managers. The company partners with its customers to provide proven products, operational experience, and deep expertise to deliver predictable performance, empowered users, managed costs and managed growth for their big data investments, both on-premise and in the cloud. Leading companies like Comcast, Philips Wellcentive and NBC Universal depend on Pepperdata to deliver big data success.

 Founded in 2012 and headquartered in Cupertino, California, Pepperdata has attracted executive and engineering talent from Yahoo, Google, Microsoft and Netflix. Pepperdata investors include Citi Ventures, Costanoa Ventures, Signia Venture Partners, Silicon Valley Data Capital and Wing Venture Capital, along with leading high-profile individual investors. For more information, visit www.pepperdata.com.

About Cloudwick

Cloudwick is the leading provider of digital business services and solutions to the Global 1000. Its solutions include data migration, business intelligence modernization, data science, cybersecurity, IoT and mobile application development and more, enabling data-driven enterprises to gain competitive advantage from big data, cloud computing and advanced analytics. Learn more at www.cloudwick.com.

###

Contact:
Samantha Leggat
samantha@pepperdata.com

Pepperdata and the Pepperdata logo are registered trademarks of Pepperdata, Inc. Other names may be trademarks of their respective owners.

Pepperdata Announces Free Big Data Cloud Migration Cost Assessment to Automatically Select Optimal Instance Types and Provide Accurate Cost Projections

Pepperdata Eliminates Guesswork and Complexity Associated with Identifying Best Candidate Workloads Down to Queue, Job and User Level, for Moving to AWS, Azure, Google Cloud or IBM Cloud

CUPERTINO, Calif. — March 6, 2019 — Pepperdata, the leader in big data Application Performance Management (APM), today announced its new Big Data Cloud Migration Cost Assessment for enterprises looking to migrate their big data workloads to AWS, Azure, Google Cloud or IBM Cloud. By analyzing current workloads and service level agreements, the detailed, metrics-based Assessment enables enterprises to make informed decisions, helping minimize risk while ensuring SLAs are maintained after cloud migration.

The Pepperdata Big Data Cloud Migration Cost Assessment provides organizations with an accurate understanding of their network, compute and storage needs to run their big data applications in the hybrid cloud. Analyzing memory, CPU and IO every five seconds for every task, Pepperdata maps the on-premises workloads to optimal static and on-demand instances on AWS, Azure, Google Cloud, and IBM Cloud. Pepperdata also identifies how many of each instance type will be needed and calculates cloud CPU and memory costs to achieve the same performance and SLAs of the existing on-prem infrastructure.

“When enterprises consider a hybrid cloud strategy, they estimate the cost of moving entire clusters, but that’s not the best approach,” said Ash Munshi, Pepperdata CEO. “It’s far better to identify specific workloads that can be moved to take full advantage of the pricing and elasticity of the cloud. Pepperdata collects and analyzes detailed, granular resource metrics to accurately identify optimal workloads for cloud migration while maintaining SLAs.”

The Big Data Cloud Migration Cost Assessment enables enterprises to:

• Automatically analyze every workload in your cluster to accurately determine their projected cloud costs
• Get cost projections and instance recommendations for workloads, queues, jobs, and users
• Map big data workloads to various instance types including static and on-demand
• Compare AWS, Azure, Google Cloud, and IBM Cloud

Availability

Pepperdata Big Data Cloud Migration Cost Assessment is available free at pepperdata.com/free-big-data-cloud-migration-cost-assessment. Pepperdata customers should email support@pepperdata.com for their free assessment.

Learn more:

• About Pepperdata Big Data Cloud Migration Assessment: https://www.pepperdata.com/big-data-cloud-migration
• Download a sample: https://www.pepperdata.com/big-data-cloud-migration-cost-assessment-sample
• Start an Assessment: https://www.pepperdata.com/free-big-data-cloud-migration-cost-assessment

About Pepperdata
Pepperdata (https://www.pepperdata.com) is the leader in big data Application Performance Management (APM) solutions and services, solving application and infrastructure issues throughout the stack for developers and operations managers. The company partners with its customers to provide proven products, operational experience, and deep expertise to deliver predictable performance, empowered users, managed costs and managed growth for their big data investments, both on-premise and in the cloud. Leading companies like Comcast, Philips Wellcentive and NBC Universal depend on Pepperdata to deliver big data success.

 Founded in 2012 and headquartered in Cupertino, California, Pepperdata has attracted executive and engineering talent from Yahoo, Google, Microsoft and Netflix. Pepperdata investors include Citi Ventures, Costanoa Ventures, Signia Venture Partners, Silicon Valley Data Capital and Wing Venture Capital, along with leading high-profile individual investors. For more information, visit www.pepperdata.com.

###

Contact:
Samantha Leggat

925-447-5300
samantha@pepperdata.com

Pepperdata and the Pepperdata logo are registered trademarks of Pepperdata, Inc. Other names may be trademarks of their respective owners.

Pepperdata Unveils 360° Reports, Enabling Enterprises to Make More Informed Operational Decisions to Maximize Capacity and Improve Application Performance

360° Reports Empower Executives to Better Understand Financial Impacts of Operational Decisions

CUPERTINO, Calif. — February 19, 2019 — Pepperdata, the leader in big data Application Performance Management (APM), today announced the availability of 360° Reports for Platform Spotlight. Pepperdata 360° Reports leverage the vast amount of proprietary data collected and correlated by Pepperdata to give executives capacity utilization insights so they better understand the financial impacts of operational decisions.

“Pepperdata 360° Reports demonstrate the power of data and the valuable insights Pepperdata provides, enabling enterprises to make more informed and effective operational decisions,” said Ash Munshi, Pepperdata CEO. “Operators get a better understanding of what and where they’re spending, where waste can be reclaimed, and where policy and resource adjustments can be made to save money, maximize capacity and improve application performance.”

360° Reports for Pepperdata Platform Spotlight include:

• Capacity Optimizer Report: This gives operators insight into memory and money saved by leveraging Pepperdata Capacity Optimizer to dynamically recapture wasted capacity.
• Application Waste Report: This report compares memory requested with actual memory utilization so operators can optimize resources by changing resource reservation parameters.
• Application Type Report: This gives operators insight on the technologies used across the cluster and the percentage of each (percentage of Spark jobs, etc.). This provides executives with insights into technology trends to make more data-driven investment decisions.
• Default Container Size Report: This report identifies jobs using default container size and where any waste occurred so operators can make default container size adjustments to save money.
• Pepperdata Usage Report: This presents Pepperdata dashboard usage data, highlighting top users, days used, and more to give operators insights to maximize their investment. With this data, operators can identify activities to grow the user base, such as promoting features, scheduling onboarding sessions, and training on custom alarms.

Availability

Pepperdata 360° Reports are available immediately for Pepperdata Platform Spotlight customers. For a free trial of Pepperdata, visit https://www.pepperdata.com/trial.

About Pepperdata
Pepperdata (https://pepperdata.com) is the leader in big data Application Performance Management (APM) solutions and services, solving application and infrastructure issues throughout the stack for developers and operations managers. The company partners with its customers to provide proven products, operational experience, and deep expertise to deliver predictable performance, empowered users, managed costs and managed growth for their big data investments, both on-premise and in the cloud. Leading companies like Comcast, Philips Wellcentive and NBC Universal depend on Pepperdata to deliver big data success.

 Founded in 2012 and headquartered in Cupertino, California, Pepperdata has attracted executive and engineering talent from Yahoo, Google, Microsoft and Netflix. Pepperdata investors include Citi Ventures, Costanoa Ventures, Signia Venture Partners, Silicon Valley Data Capital and Wing Venture Capital, along with leading high-profile individual investors. For more information, visit www.pepperdata.com.

###

Contact:
Samantha Leggat
samantha@pepperdata.com

Pepperdata and the Pepperdata logo are registered trademarks of Pepperdata, Inc. Other names may be trademarks of their respective owners.

Sample report attached.

Sample Capacity Optimizer Report – memory and money saved with Capacity Optimizer

Five Mistakes to Avoid When Using Spark

Apache Spark is playing a critical role in the adoption and evolution of Big Data technologies because it provides sophisticated ways for enterprises to leverage Big Data compared to Hadoop. The increasing amounts of data being analyzed and processed through the framework is massive and continues to push the boundaries of the engine. Whether you’re programming in Java or working out with Python, these five items can impact your Spark applications.

1. Serialization – This plays an important role in the performance of any distributed application. Formats that are slow to serialize objects into, or consume a large number of bytes, will greatly slow down the computation. Often, this will be the first thing you should tune to optimize a Spark application. The Java default serializer has very mediocre performance with respect to runtime as well as the size of its results. Therefore, the Spark team instead recommends the use of the Kryo serializer.

   Tips: Avoid using anonymous classes, instead use static classes as an anonymous class will force you to have the outer class serialized. Avoid using static variables as a workaround for serialization issues, as multiple tasks can run inside the same JVM and the static instance might not be thread safe.

2. Partition Sizes – Generally speaking, any performance management software that sees data skew will recommend more partitions — but not too many more! The best way to decide on the number of partitions in an RDD is to make the number of partitions equal to a multiple of the number of cores in the cluster so that all the partitions will process in parallel and the resources will be utilized in an optimal way. You want to avoid a situation where you have 4 executors and 5 partitions (to reduce it to a simple case).

   Tip: As an upper bound, tasks should take longer than 100ms, or scheduling tasks will take more time than executing tasks. As a lower bound, in order for there to be parallelization, make the number of partitions at least 2x the number of cores being requested, or reasonably expected to be available. This may take a couple of tries to find the right balance while avoiding skew.

3. Executor Resource Sizing – What you’re trying to do is sub-divide your data set into the smallest pieces that can be easily consumed by your Spark executors, but you don’t want them too small. There are a few ways to find that happy middle ground, but first of all you’re trying to avoid data skew by making sure your key space is well distributed.Make a guess at the size of your executor based on the amount of data you expect to be processed at any one time. There are two values in Spark on YARN to keep an eye on: The size of your executor, and what is called the YARN memory overhead. This is for the YARN scheduler to not kill an application when it uses a large amount of NIO memory or other off-head memory areas.Make sure your driver is large enough to hold the execution plan, and the expected results to be delivered to the client. One way to simplify this is to use Dynamic Allocation, which requires Spark to run in a YARN (Hadoop) environment. This allows you to manage executor size/core count but lets the environment itself determine the number of executors that can/should be launched by the Application. Please talk to your administrators about this, as they may request you place a maxExecutor cap on the size of the ask, or Spark will ask for an executor for every task!

   Tip: Understand your resource usage! Being able to understand true heap utilization of your executors and total RAM used vs. RAM requested can assist you in right-sizing your requests to work better in multi-tenant environments. (This is one small piece of what Pepperdata provides and can assist with.)

4. DAG Management – It’s always a good idea to keep an eye on the complexity of the execution plan. Use the DAG (direct acyclic graph) Visualization tool that comes with SparkUI for one possible visual map. If something that you think should be straightforward (a basic join, for example) is taking 10 stages, look at your query or code to reduce it to 2 to 3 stages if possible.

   Tip: Look at each individual stage to understand the parallelization that is actually taking place. If you have a (non-parallel) stage that is using < 60% of the available executors, should that compute be rolled into another stage? Is there a separate partitioning issue?

5. Shading – Make sure that any external dependencies and classes you are bringing in do not conflict with internal libraries used by your version of Spark or are available in the environment you are using. For example, are you using Google Protobuf? It’s a popular format for storing and transporting data that is more compact that JSON. But let’s say you want to use the getUnmodifiableView() function. This is only available in Protobuf 2.6.0, and most Hadoop implementations are delivered with Protobuf 2.5.0. You would need to shade the jar while building your project to avoid conflicts in which Protobuf is being used by your application.

   Tip: Keep your eyes on the prize: Always be aware of potential conflicts.

Other Things You Can Do

Interested in learning more about APM and how it can be applied to optimize your application and infrastructure environment?

• Ask about FREE Pepperdata Application Spotlight APM for up to 20 nodes.
• Migrating big data applications and workloads to the cloud? Start a FREE Big Data Cloud Migration Cost Assessment.
• Start a FREE trial with Pepperdata

Key APM Functions and Metrics – A Primer

For those of us who are APM industry “insiders”, there’s not a lot of mystery when it comes to understanding the functionality and value of application performance management. We tend to take for granted that everyone else shares the same knowledge, but that’s not a wise assumption to make. So we thought it would be worthwhile to provide a brief APM tutorial for those who may not be fully familiar with the subject.

Application performance management (APM) is the monitoring and management of performance and availability of software applications. The primary goal of APM solutions is to maintain an expected level of service and availability by understanding why transactions in your application/workload are slow or failing. For example, a development or operations team can instantly tell from their APM solution if an application is causing some performance spikes. They can then leverage their APM solution to determine the root cause, identify which queries were affected and make appropriate adjustments to resolve the issue.

Here are some examples of common application problems that APM solutions can help to quickly identify and resolve.

• Track application usage to understand spikes in traffic
• Find bottlenecks or latency problems with application dependencies including SQL, queues, caching, etc.
• Identify slow SQL queries
• Find the highest volume and slowest web pages or transactions
• Find the root cause(s) for application problems
• Identify ways to optimize application performance

APM solutions perform a plethora of functions in the quest to optimize application performance, including:

• Anomaly detection, which enables a rapid response to application behaviors that begin to fall outside the normal operating range, and identify events that do not conform to expected behavioral patterns
• Application debugging, the process of finding the causes of undesirable effects on application behaviors. This requires identifying defects and errors in program code
• Distributed profiling of the many potential sources of application performance degradation. This involves identifying and localizing the sources of performance degradation across an ecosystem consisting of applications, services and machines.
• IT service monitoring shows whether response time and uptime commitments are being met, including adhering to service level agreements and performance thresholds
• Root cause analysis attempts to determine the probable cause of a problem, then constructing a causality chain correlating case and effect.
• End-user experience monitoring captures user-based performance data to gauge how well the application is performing and identify potential performance problems.
• Application topology discovery and visualization is the visual expression of the application in a flow-map to establish all the different components of the application and how they interact with each other.
• User-defined transaction profiling examines specific interactions to recreate conditions that lead to performance problems for testing purposes.
• IT operations analytics is the discovery of usage patterns, identification of performance problems, and anticipation of potential problems before they happen.

APM solutions also gather an incredible set of metrics as they continuously monitor the cluster and application environment. Here are just a few of the key metrics that are captured and analyzed.

Average Response Time is the amount of time an application takes to return a query or request to a user. To measure average response time, an application is tested under different circumstances (i.e. number of concurrent users, number of transactions requested) Typically, this metric is measured from the start of the request to the time the last byte is sent. Other factors, like geographic location of the user and the complexity of the information being requested, can affect the average response time for users. These should all be considered in the overall evaluation of application performance.

Error Rates – The last thing you want your users to see are errors. Monitoring error rates is a critical application performance metric.
There are potentially three different ways to track application errors:

• HTTP Error % – Number of web requests that ended in an error
• Logged Exceptions – Number of unhandled and logged errors from your application
• Thrown Exceptions – Number of all exceptions that have been thrown

It is common to see thousands of exceptions being thrown and ignored within an application. Hidden application exceptions can also cause a lot of performance problems.

Request Rate – Understanding how much traffic your application receives will impact the success of your application. Potentially all other application performance metrics are affected by increases or decreases in traffic. Request rates can be useful to correlate to other application performance metrics to understand the dynamics of how your application scales. Monitoring the request rate can also be good to watch for spikes or sudden inactivity. If you have a busy API that suddenly gets no traffic at all, that could signal something bad. A similar but slightly different metric to track is the number of concurrent users.

Application and Server CPU and Memory – If the CPU usage on your machines is extremely high, or you have limited memory resources, you can expect to eventually suffer from application performance problems. Monitoring the CPU and memory usage of your servers, cluster and applications is a basic and critical metric.

Application Availability – Monitoring and measuring if your application is online and available is a key metric you should be tracking. Most companies use this as a way to measure uptime for service level agreements (SLA).

All metrics should be evaluated over time, and one that is critical should be fed into a rules engine that raises alerts when a set threshold is exceeded. Ultimately all metrics can and should be used to understand what is normal/typical for your application so that abnormal/atypical behavior can be detected, analyzed and resolved.

Other Things You Can Do

Interested in learning more about APM and how it can be applied to optimize your application and infrastructure environment?

• Ask about FREE Pepperdata Application Spotlight APM for up to 20 nodes.
• Migrating big data applications and workloads to the cloud? Start a FREE Big Data Cloud Migration Cost Assessment.
• Start a FREE trial with Pepperdata

Three Keys to Understanding Application Performance in the Cloud

The key cloud advantages of self-service, automatic provisioning, and rapid elasticity come at the cost of increased complexity at the application level. Each newly provisioned instance can have a hidden impact on the performance of already-running applications, an impact that may be visible only when we look at the underlying shared infrastructure.

Cloud features such as automatic provisioning and workload management allow us to ignore the relationship between VMs and assigned underlying hardware, but our applications must still run on the actual hardware. And when things go wrong we need to be able to track down the problem quickly and accurately.

As technology evolves, new challenges will arise. One thing remains the same: We need to understand the impact of resource utilization and hardware latencies on the performance of our application and workload and maintain the correlation between application and hardware at any given point to resolve issues. Here are three ways you can achieve better insight into the performance of your applications in the cloud.

Make sure your APM/IPM solution offers fine-grained visibility

An effective application performance management (APM)/infrastructure performance management (IPM) solution for the cloud optimizes your time by automatically providing you with time-series performance data that spans your big data hardware, software, and applications, sampling every five seconds and allowing you to solve the hard problems. Real-time visibility into how an application is using resources beyond CPU and memory is required to find contention and resource hogs wherever they lurk. The ROI benefit is measured in hundreds of man-hours that can be spent on other tasks, as well as the money you don’t have to spend running inefficient apps that are unnecessarily fighting for resources. Pepperdata provides application tuning recommendations, auto-generated reports, and insight tables that give you actionable observations and recommendations, not just raw data.

Deploy an APM/IPM solution that provides AI-driven optimization

The primary driver for moving to the cloud is cost-savings. On-premises, over-provisioning for workloads is a given, and resource costs are a secondary concern. But in the cloud, you pay for every minute of compute and storage resources that you use, so over-provisioning is a major issue.  A standard YARN deployment allows for CPU and memory resources to be reserved and not used by applications as a normal practice. This inefficiency is wasting resources and driving up your cloud costs. By auto-tuning the platform, these inefficiencies are reduced by up to 50%. Pepperdata leverages AI to automatically optimize your big data platform in response to inefficient CPU and memory allocation. This programmatic approach to tuning is done at scale affecting thousands of applications simultaneously and eliminates the time-consuming hassle of trying to manually tune every job.  

Choose an APM/IPM solution that includes a world-class support team

Pepperdata leverages the richest set of performance data available to tackle unknown challenges that stem from managing a complex, ever-changing combination of big data hardware and software components, and delivers that information continuously to an industry-leading UI.  But new application and infrastructure demands are constantly emerging. Pepperdata Support and Data Science teams are ready to partner with you to solve the questions that can’t be easily answered in a static UI. With the combined expertise of your team and ours, you will be well-prepared to tackle any new performance problems that arise.

Getting a true picture of application performance in the cloud can be challenging if you don’t have the right tools. You can rely on Pepperdata to answer the currently unknown questions with performance metrics gathered every five seconds from each layer of the big data hardware, software, and application stack, and retained for long-term analysis.  We can help you solve the most challenging performance problems by leveraging our AI-based automation and insightful visualizations. No matter where you deploy Pepperdata, you can be assured that solutions and features are fully cross-compatible and support hybrid cloud architectures with zero loss in functionality.

With an eclectic combination of on-premises and cloud deployments represented across hundreds of clusters, Pepperdata optimizes application and infrastructure performance at scale for over thirty thousand production nodes.  The world’s largest enterprises rely on Pepperdata to help them meet the SLAs they need for their business-critical applications.

Other Things You Should Do While You’re Here

• Use Free Pepperdata Application Spotlight APM
• Start a Free Cloud Migration Cost Assessment
• Read the New Pepperdata White Paper: Cloud Migration – Opportunities and Risks for the Business Unit
• Subscribe to this blog (use the form below or to the right)