Evolve with Apache Kafka®

Apache Kafka is a versatile platform for building real-time, highly-available, linearly-scalable systems.

Released nearly a decade ago, Kafka has evolved from a simple message broker to the backbone of distributed compute for the modern enterprise.

In this article I describe our experience with Kafka so far, and explain what excites us for the future.

Let’s start with a familiar story – two engineers are in a room, struggling to diagnose the root cause of a production issue.

Laptops are open, they’ve scribbled all over the whiteboard, plenty of ideas have been thrown around, and not one has stuck. Then, as quickly as the issue arose the symptoms disappear. The problem is left unsolved, its cause unknown, time presses and our engineers move on.

This is one truth of software engineering, sometimes the hours we spend interrogating the machine don’t end in a neat conclusion.

Reasons are varied, often our systems are necessarily complex, but of all the reasons to come up short – a lack of good tools shouldn’t be one.

Kafka the Message Broker

My name is Derek Troy-West. I’m the Co-Founder and CEO of Operatr.IO. We make tools that empower teams to deliver with Apache Kafka.

I’m an engineer. I’ve spent 20 years building enterprise systems for clients in the UK, US, and Australasia, and I’ve been working with Kafka for nearly a decade.

kPow provides x-ray vision for Apache Kafka.
X-Ray Vision for Apache Kafka

In 2012 my clients began encountering a world where real-timescale, and availability were critical factors for their success and Kafka quickly became central to their system architectures.

Kafka became my go-to message broker for systems that could digest data as it happened, grow as the business needed, and run with little downtime.

As a message broker Kafka is easy to configure, simple to operate, and hard to break. I expect the small staging cluster that I installed all those years ago is still happily ferrying messages around today.

Kafka is so easy start using that often it is only after you have put data on topics that you wonder – where is my data? How is the cluster operating?

In the early days we built custom monitoring systems to observe Kafka clusters, using the excellent riemann.io to create dashboards that displayed the metrics that Kafka exposes via JMX.

As our clients leaned further into the real-time world, the solutions we built with Kafka grew in sophistication, but our tooling remained static.

Kafka the Streaming Compute Platform

What is the point of having a highly-available message broker if the systems that leverage it represent a single point of failure?

With the need to build highly-available systems in mind we used heavyweight distributed frameworks like Apache Storm to process data from Kafka.

Visualisation and Control of Consumers and Compute

Storm is a powerful tool with an operational complexity to match, and as far as observability is concerned – well good luck to you!

Then, in 2016, something interesting happened.

The Apache Foundation released Kafka Streams, a distributed compute DSL packaged as a simple library that they describe as ‘The easiest way to write mission-critical real-time applications and microservices’.

Kafka Streams allowed us to build the same real-time, highly-available, linearly-scalable compute that our clients needed and Storm facilitated, but with fewer options, fewer complications, and less operational overhead.

As we migrated from Storm to Kafka Streams it became apparent that truly there’s no free lunch. Our clients would continue to demand ever more sophisticated systems, and our cobbled-together solutions for observing and controlling Kafka were no longer fit for purpose. We needed to evolve.

The idea for Operatr.IO, and creating a suite of tools for engineers working with Apache Kafka, was born.

Kafka the Source of Truth

Kafka sits at the heart of Operatr.IO, each of our products are built for Kafka and from Kafka.

Deep Data Inspection with KJQ

We have seen Kafka evolve from a rock-solid message broker to an integral part of the solution for any enterprise competing for customers in the modern marketplace.

The challenge today is understanding how best to utilize Kafka in your technology stack, and how to remove roadblocks from the path of your engineering team. We have one very simple purpose at Operatr.IO – to build tools that shine a light on Apache Kafka and empower engineers to deliver.

So what is the most pressing need for our users today? Without a doubt, it is the seemingly simple requirement to find data on topics.

Without the right tool it can be like searching for a needle in a haystack.

We built kJQ to solve exactly that problem, giving your team the ability to use JQ – a language for querying JSON data – to search topics directly:

Easy to use and fast – 600,000 records filtered in a few seconds… We got to the bottom of something in a few hours that could have dragged on for _days_

– Alex  Hilton, FX Engineering Lead, Airwallex

So What’s Next?

Our tools are changing the way engineers work with Apache Kafka.

One great joy of building kPow is pushing delivery to meet the needs of engineering teams around the world. kJQ (included in kPow, standalone in Beta) helps our users get straight to the data they need. kSynth (Alpha) plays an important role in our own delivery, and shortly we hope in yours as well.

We’re excited to execute our product roadmap, and to see where Apache Kafka takes us next. The best is yet to come.

A Neat Conclusion

What happened to those two engineers stuck in a room?

Tom Crowley and I work on a suite of tools for Apache Kafka that come from our experience of delivering real-time, highly-available, linearly-scalable, streaming compute systems. We hope our experience helps your team deliver.

Enjoy this article?

Sign-up to the mailing list for operatr.io news, product updates and Kafka insights.

kPow by Operatr.IO is the ultimate engineering tool for managing and monitoring Apache Kafka®. 

A single Docker container or JAR file that installs in minutes, kPow gives you instant visibility of your Kafka clusters and immediate access to your data.

kPow is compatible with Apache Kafka+1.0, Amazon MSK, Instaclustr, Aiven, Red Hat AMQ Streams, Confluent Platform, and Confluent Cloud.

Start with a free 30-day trial and solve your Kafka issues within minutes.

Derek Troy-West is the Co-Founder and CEO of Operatr.IO and Troy-West.