Using kPow with Instaclustr’s Managed Apache Kafka®

As industry adoption of Apache Kafka grows, many companies are opting to use managed service providers (MSP) to set up, scale, and manage their Kafka clusters in production. Instaclustr is one of these providers. 

Instaclustr’s Managed Apache Kafka service is designed for enterprise and offers simple provisioning, SOC 2 Certification, and 24×7 support. It is fully compatible with the kPow management and monitoring toolkit and when used together they give teams the ability to create great software with ease.

Instaclustr, in a Nutshell

Instaclustr are champions of open source; they were early adopters of Apache Cassandra®, Apache Spark®, and Apache Kafka and are focused on delivering reliability at scale. Their managed environments are delivered through cloud providers AWS, Azure, and GCP.

The big wins with Instaclustr are that you get Certified Security with SOC 2 and PCI-DSS. Their solutions were built to and independently audited against the Service Organisation Control – Type 2 (SOC 2) standards, assuring the confidentiality, integrity, and availability of your data. 

Instaclustr’s managed Apache Kafka service is Payment Card Industry – Data Security Standard (PCI-DSS) compliant when running in AWS. PCI-DSS is a mandated standard for many financial applications.

In this post, I’ll explain how to integrate kPow with Instaclustr’s Managed Apache Kafka and outline the benefits for Kafka engineers.

Why kPow is perfect for Instaclustr’s Managed Apache Kafka

Instaclustr was one of the earliest adopters of our toolkit (then called Operatr), and their global team of Kafka specialists has been providing feedback to help us improve and better target our features since day one. 

Because of this, kPow is fully compatible with Instaclustr’s managed Kafka service, and for Instaclustr Kafka Support Customers, you even get access to it as part of your support package. 

Instaclustr’s Top 4 Reason’s to use kPow

Reason #1 → It takes 30 seconds to get working! Simply download the jar file and configure the environment variables as instructed below, which is optimized for our managed cluster.

Reason #2 → Gain total visual awareness of all key Kafka metrics such as Consumer Lag and data usage on partition levels. 

Reason #3 → No need to Set up JMX access or Zookeeper access; just connect to Kafka on the default port.

Reason #4 → All your data stays in Kafka’s internal topics; you never send your data out of the organization.

Integrating kPow with Instaclustr

You can find detailed instructions on setting up Operatr with Instaclustr Managed Kafka in Ben’s Blog Post, Using Operatr with Instaclustr Managed Kafka.

If you are already an Instaclustr Managed Kafka customer and you’d like to give kPow a try, you can sign-up for a free trial and use our quick start guide below to launch in under 5 minutes.

These details are also available in our Support Hub.

Quick Start: Instaclustr Managed Kafka

> docker run -p 3000:3000 -v {absolute_path}:/ssl --env-file ./instaclustr.env operatr/kpow:latest

WHERE instaclustr.env:

SASL_MECHANISM=SCRAM-SHA-256 required username="ickafka" password="secret";
SCHEMA_REGISTRY_URL=https://ickafkaschema:[email protected]:8085

Benefits for Engineers using Instaclustr + kPow

Using Instaclustr to manage the provisioning, deployment, and running of your Kafka clusters enables you to focus on building applications and eliminates the need for costly operations teams and technology. kPow provides an engineering toolkit that empowers developers to build cutting-edge applications. 

Once kPow is installed and running, you’ll have the ability to:

  • Search tens of thousands of messages a second with kJQ
  • Gain x-ray vision of the brokers, topics, and groups that make up your cluster
  • Identify inactive topics, groups, and other areas of concern
  • Visualize data distribution through your cluster
  • Monitor lag in multiple dimensions

Explore Topics with kJQ

Search tens of thousands of messages a second with kPow’s unique, custom implementation of JQ-like queries for Kafka topics.

Easy to use and fast – 600,000 records filtered in a few seconds..

We got to the bottom of something in a few hours that could have dragged on for _days_

– Alex  Hilton, FX Engineering Lead, Airwallex

kJQ works with JSON or JSON-like data including Apache Avro®, Transit, EDN, and even Protobuf messages where you have configured a custom serdes.

Role Based Access Control for user management, Data Policies to mask sensitive data, an Audit Log for data governance, and the power to export query results and produce new messages make kPow the essential tool for your engineers.

Apache Kafka,kafka tool,kafka toolkit,kpow,,instacluster,managed Apache Kafka
kPow Data Screen

X-Ray Vision for Apache Kafka

Use kPow’s intuitive Kafka UI to gain instant visibility into your Kafka brokers, topics, partitions, offsets, consumer groups, members, assignments, streams, and more.

Get straight to the root-cause of production issues with live mode for real-time monitoring of your Kafka resources and kPow’s multidimensional view into your consumer group message lag and throughput.

Manage multiple clusters from a single installation with Kafka Connect and Schema Registry integration fully supported out of the box.

Apache Kafka,kafka tool,kafka toolkit,kpow,,instacluster,managed Apache Kafka
kPow Data Screen

Control your Compute

kPow provides in-depth multi-dimensional metrics that allow you to understand how your Apache Kafka applications are performing.

Understand the lag and message throughput of your consumer groups and streaming compute topologies by group, broker, topic, and consumer group host. Take control of your compute with kPow’s functions for resetting consumer offsets and more.

Apache Kafka,kafka tool,kafka toolkit,kpow,,instacluster,managed Apache Kafka

The Best of the Rest

  • OpenID, SAML, LDAP, and HashUser (Jetty) options for User Authentication.
  • Fine grained control of user actions with Role Based Access Control (RBAC).
  • Data Policies for masking and redacting sensitive data fields.
  • Schema Registry and Kafka Connect supported as standard.
  • Multi-cluster management supported as standard.
  • HTTPS connections supported out-of-the-box.

If you’d like any more information on kPow, please reach out to our team at [email protected].

For more information on Instaclustr’s Managed Kafka, you can email the Instaclustr team at [email protected].

Enjoy this article?

Sign-up to the mailing list for news, product updates and Kafka insights.

Manage, Monitor and Learn Apache Kafka with kPow by Operatr.IO

We know how easy Apache Kafka® can be with the right tools. We built kPow to make the developer experience with Kafka simple and enjoyable, and to save businesses time and money while growing their Kafka expertise. A single Docker container or JAR file that installs in minutes, kPow’s unique Kafka UI gives you instant visibility of your clusters and immediate access to your data.

kPow is compatible with Apache Kafka+1.0, Amazon MSK, Red Hat AMQ Streams, Instaclustr, Aiven, Vectorized, Azure Event Hubs, Confluent Platform, and Confluent Cloud.

Start with a free 30-day trial and solve your Kafka issues within minutes.

Kylie Troy-West is the Co-Founder and COO of Operatr.IO and the Director of Operations at Troy-West. Her focus is on empowering the Operatr.IO team to deliver on their vision of building world-class tools for Apache Kafka®.