A Guide to Apache Kafka® for the Tech Outsider

Operatr.IO / Articles  / A Guide to Apache Kafka® for the Tech Outsider
Apache Kafka,kafka,kafka tool,kafka blindness,kafka monitoring, kafka toolkit

A Guide to Apache Kafka® for the Tech Outsider

When I announced I was leaving my career in local government to co-found a tech company building tools for Apache Kafka®, I caught a lot of funny looks. 

It wasn’t common knowledge that I managed operations at a boutique Distributed Systems and Clojure Consultancy in my spare time, so I really can’t blame anyone for looking so shocked. Once everyone got over the shock, the first thing they asked me was, ‘What is Apache Kafka?’

For my local government colleagues, the technical answer I gave them didn’t really clear things up. I was focusing on the wrong things – I was answering the wrong question. I should have been talking about what Kafka does – that it powers some of the most recognizable companies in the world – that it powers systems that are highly available, scalable, and real-time.

So this is my take two.

Kafka in the Beginning

LinkedIn developed Kafka in-house for the real-time processing of the trillions of messages that move through that professional online network. It worked as a message broker, which is an intermediary program that transfers messages in real-time via a topic from one program to another. 

LinkedIn felt compelled to share it and it went open source in 2011. Today Kafka is one of the five most active projects of the Apache Software Foundation.

The Apache Foundation describes Kafka as “an open-source distributed event-streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.” 

Kafka is now a fully managed streaming platform – not just a message broker – it hands you the end-to-end-streaming capability to publish/write and subscribe to event streams, store them and process them in real-time or retrospectively. 

Kafka Right Now

A diverse range of industries and companies already use Kafka across all sectors, including:

  • Finance
  • Utilities
  • E-Commerce
  • Travel
  • Service provision
  • Social Media
  • Entertainment

Many high-profile international brands such as Netflix, Airbnb, Spotify, Twitter, Uber, PayPal, Mailchimp, LinkedIn and, of course, kPow the Kafka monitoring tool we’ve built, use Kafka as part of their technology stack. More than 80% of the companies on Fortune’s top 100 companies use Kafka today.

This year, use of Kafka in the enterprise is tipped to grow a third to 250,000 organizations. 

Kafka Powers Customer-Driven-Business

We built a business around Kafka because it encapsulates the qualities necessary to deliver modern, customer-focused systems. Systems that are real-time, highly available, and scalable. 

A straightforward example is Netflix; I’m in Melbourne, Australia. My state has been in lockdown for the best part of six months. Usually I watch Netflix a couple of times a week, at the moment it is a lifeline. Not just for me, but for the other 1.8 million households that tune in every day. Netflix uses the power of Kafka to ensure that their systems can scale to meet this increased demand without impacting their quality of service.

If Kafka is so great, why does it need a toolkit?

Kafka blindness is a thing. Where’s my data gone? Why are my systems working slowly? What’s happening under the hood? 

There’s no built in console or dashboard, and that can make work tricky. The line between development and operations has merged, as has the line between designing and monitoring systems. Engineers need insights into a system’s performance while it’s being built, not just once it’s in production. 

They need a Kafka toolkit like kPow.

kPow was designed for engineers who build, manage, and maintain systems using Kafka. You install it from a single Docker container and it provides simple, secure, self-contained support straight out of the box with an intuitive Kafka UI. Our toolkit allows engineers to unlock Apache Kafka’s full potential, resulting in a saving of development hours, increased team productivity, and the ability to rapidly resolve Kafka queries.

Apache Kafka is here to stay. And while it’s a technology my local government ex-colleagues won’t hear it talked about often, I guarantee it is impacting their everyday lives, for the better.

Enjoy this article?

Sign-up to the mailing list for operatr.io news, product updates and Kafka insights.

kPow by Operatr.IO is the ultimate engineering tool for managing and monitoring Apache Kafka®. 

A single Docker container or JAR file that installs in minutes, kPow gives you instant visibility of your Kafka clusters and immediate access to your data.

kPow is compatible with Apache Kafka+1.0, Amazon MSK, Instaclustr, Aiven, Red Hat AMQ Streams, Confluent Platform, and Confluent Cloud.

Start with a free 30-day trial and solve your Kafka issues within minutes.

Kylie Troy-West is Co-Founder and COO of Operatr.IO and the Director of Operations at Troy-West. Her focus is to empower the Operatr.IO team to deliver on their vision of building world-class tools for Apache Kafka®.