If you’re just getting started with Apache Kafka, you know that there’s a lot to learn and watch out for. What is Kafka used for? How do I get the most out of it? These may be just a few of the questions running through your mind, and trying to search online for the answers can get overwhelming. We’ve done the research for you and put the answers here for easy access. Keep reading to find what it’s used for and what you should watch out for when using Kafka.
What is Kafka?
Apache Kafka is an open-source stream-processing software platform created by LinkedIn and currently developed by the Apache Software Foundation. Application developers, IT professionals, and data managers are just some of the people who use Kafka.
According to the Apache Software Foundation, over 80% of Fortune 100 companies use this technology. Here are some quick stats to put it into perspective just how many Kafka users are out there: 10/10 manufacturing companies, 7/10 banks, 10/10 insurance companies, and 8/10 telecom companies use the technology.
What Is Kafka Used For?
Kafka is used for ingesting, moving, and consuming large amounts of data rapidly. It allows for the creation of real-time, high-throughput, low latency data streams that are easily scalable. The platform is reliable, fast, and well known in the big data space by enterprise companies for these reasons.
When it comes to use cases, Kafka can be used for website activity tracking, providing operational tracking data, log aggregation, stream processing, event sourcing, as a replacement for a message broker, and as an external commit log for distributed systems.
To give a specific example, at one point the New York Times used Kafka to store every article they’ve ever published. In addition to that, they used Kafka with the Streams API to feed in real time published content to the various apps and systems their readers rely on to access their content.
What Should You Watch Out For? Unoptimized Kafka
In working with our customers, we find that success with Kafka starts by making sure that your platform is optimized. Since there’s so much potential within the platform, ensuring you’re getting all you can out of it is key. Here are 4 best practices—which we dive deeper into in another post—we suggest when it comes to optimizing Kafka:
1. Upgrade to the latest version.
Using an outdated version of Kafka can lead to long-running rebalances as well as rebalance algorithm failures. Making sure you’re using the most up-to-date version of Kafka can prevent those balance issues and ensure you’re getting the most out of the framework.
2. Understand how to improve data throughput rates.
Kafka has settings to control how data moves throughout the stack. Understanding and tuning these settings is the first step towards improving your data throughput rates and getting the most from your Kafka architecture.
3. Stick to random partitioning when writing to topics, unless architectural demands call for otherwise.
Kafka supports randomized writes. As you tune Kafka, you might be tempted to dictate where data is written. However, randomized writes will yield better performance in most cases.
4. Adjust consumer socket buffers to achieve high-speed ingest while maintaining data integrity.
To adjust consumer socket buffers, more capable networks can support higher buffer sizes. For example, 10Gbps networks may warrant socket buffers up to 16MB.
So now that you know what Kafka is, what it’s used for, and what you should watch out for, you can get started ensuring you get the most from the robust framework. You can start with the steps we covered here to ensure your Kafka is optimized, or you can let Pepperdata do the work. Pepperdata Streaming Spotlight allows IT operations teams to get detailed, near real-time visibility into Kafka cluster metrics, broker health, topics, and partitions in a single dashboard.
Pepperdata Streaming Spotlight topics view
If you’d like to learn more about working with Kafka, check out our webinar: Monitor and Improve Kafka Performance.