Last week, Pepperdata was a sponsor at Hadoop Summit, an event that we have attended the past 4 years. This year was by far and away our best experience yet, and it wasn’t because of the food (sorry, San Jose Expo Center). Nope, the reason that we keep coming back is because it confirms in our minds the fact Hadoop is growing every year, not just in the number of people who are interested and/or talking about it, but actual production users. With each conference, more attendees pack the halls, and more sessions are added to the agenda. The topics get more technical, and the number of customers speaking about their specific use cases grows. This year was the 10th anniversary of Hadoop (and the 8th Hadoop Summit), and the Pepperdata team came away exhausted but invigorated because of the encouraging discussions we had with current, and future, customers.

Observing the number of conference attendees (both sponsors and visitors) is evidence enough of this growing market. Last year’s conference drew over 4,000 attendees and though Hortonworks has not yet revealed the final number on this year’s event, I would bet it topped over 5,000. 165 sessions (business and technical) were topped with over 170 this year (plus additional “crash courses” and “birds of a feather” sessions). Spark was definitely a hot topic, as was security and cloud for Big Data deployments. Management and operational efficiency seemed to be prevalent themes (good news for us, since our Adaptive Hadoop Performance is all about automating performance improvements without the need for manual intervention).

Perhaps the greatest highlight from our standpoint (that standpoint being the far left corner of the expo hall) was the fact that so many people stopped by our booth and had heard of Pepperdata, and were actually interested in learning more about what we do for our customers. Several new prospects were driven to our booth when they attended Comcast’s session on Wednesday, and heard about how they are using Pepperdata to improve troubleshooting in their production environments (go to 21:05 for our glamour shot!). There was a steady stream during Wednesday’s evening reception, as our pepper pens took off on the conference app as one of the best giveaways of the conference! (marketing career highlight, right there!).

We spoke to hundreds of people over the course of two days, and some themes and topics seemed to come up again and again. Below are some of the common conversations we had, with users from organizations across the board (High Tech, Manufacturing, Media, Retail, etc).

  • Organizations grappling with multi-tenancy was a frequent topic — running Spark and MapReduce on the same cluster, for instance, and running into issues with resource contention and jobs missing their deadlines.
  • Adding more users to shared clusters was a recurring conversation — a few of the people who stopped by our booth currently had 2-3 business units accessing a shared cluster, but expected to be around 5 business units soon. The question they had was how to do that without letting rogue jobs and ad-hoc requests take down the higher-priority workloads.
  • Chargeback, a feature that Pepperdata added in Q4 2015, got a lot of wide eyes at the show. One healthcare company mentioned that during their demonstration — and already had a list of users that they could hit with usage bills if this unique feature were available to them!

A final observation from the show was the size of the deployments within the customer organizations. About 40% of the attendees we talked to had clusters between 50 and 100 nodes, with many of the expanding to grow to 200 nodes or more within the next 12 months. In 2014 and 2015, we heard a lot of organizations growing into the double digits range, but it seems now that Hadoop has grown in its breadth, it has also grown in its depth within individual user deployments. Yet another proof point that Hadoop has grown in popularity, in practice, and is starting to prove serious value to organizations of all sizes and industries.