INTRODUCTION


OPERATR is the toolkit for developers who build streaming compute with Apache Kafka®.

Say goodbye to bastion nodes and shell scripts by using our data-oriented UI to explore topics, monitor lag, manage system configuration, and much more.

Observe unique telemetry and insights from periodic snapshots of Kafka metadata computed into custom telemetry - all retained within your cluster.

Features that allow cluster mutation or deep topic inspection (including key and message data) are switched off by default.

OPERATION


Simple.A single docker container that connects to your cluster like any Consumer or Producer.
Secure.No access to Zookeeper, no requirement to open up JMX, no data leaves your network, and OKTA integration for authorization.
Self-Contained.A streaming compute application that retains data in internal topics, with no further dependencies.
Minimum Requirements. We recommend 2GB of heap, 2GB of disk, and ulimit NOFILE=100000.

AVAILABILITY


Each release is documented and published to Dockerhub, major releases are available on the AWS Marketplace, and the latest release is available as a JAR file.

The simplest way to provision for customers of AWS, OPERATR is 100% compatible with Amazon MSK.

Subcribe to our Marketplace listing then see our full guide to deploying OPERATR in ECS or EKS.

AWS Marketplace subscriptions are billed by the hour, they do not require you to purchase a license.

Dockerhub

Each release of OPERATR is published to Dockerhub.

Installations provisioned via Dockerhub require a license. Purchase one online or sign up for a free 30-day trial today.

Dockerhub Quick Start

Sign up for a free 30-day trial.

Create a config.env file containing your environment variable configuration.

> docker run -p 3000:3000 --env-file ./config.env operatr/operatr:latest

JAR File

The most release of OPERATR is available via our releases page.

If you prefer to build your own container see the example Dockerfile from our docker builds.

Installations provisioned via JAR file require a license. Purchase one online or sign up for a free 30-day trial today.

Local Evaluation

Sign up for a free 30-day trial and use operatr-io/local to evaluate OPERATR locally with Docker Compose.

TRIALS & LICENSES

Installations other than via AWS Marketplace require a license certificate.

Purchase a license or sign up for a free 30-day trial to receive a certificate via email:

--------BEGIN LICENSE CERTIFICATE-------
ID: 42481e7d5c4bf29538db8d1515491c9f
Type: Unlimited Instance
Licensee: Test User
Expiry: 2021-01-27
Installation Credits: Unlimited

43AD1EA9 17EBF3EE 18828A14 2ACAB181
9C537D28 0C5A58CD E47B5CEB EF573C1C
28C55722 9CC1AAA7 EA6484EA FE0803F5
772C9FC0 D24D19A4 4149BC48 34114BF0
AF0303EA 577FA9FB 218294F1 6479005A
428636B8 A6F81846 7BC69272 1ADEFB0E
7A0BBD27 46200164 CE3F29BA 0D166B70
4207A5AC C8016FFA 7961C6B4 429D8F51
--------END LICENSE CERTIFICATE---------

Then configure your license by one of two methods:

1. Provide License Specific Environment Variables

LICENSE_ID=b9facd75-d9a7-4555-bf90-0b1ab4877533
LICENSE_TYPE=Unlimited Instance
LICENSEE=Test User
LICENSE_EXPIRY=2021-01-27
LICENSE_CREDITS=Unlimited
LICENSE_SIGNATURE=05E39834B8C2C1...
or

2. Provide a License File

LICENSE_LOCATION=/path/to/license.txt

CONFIGURATION

OPERATR is configured entirely by environment variables. Many of them may be familiar to you as they are the same required to connect to any Kafka cluster.

BASIC SETTINGS
BOOTSTRAPkafka-1:9092,kafka-2:9092,kafka-3:9092The Kafka cluster bootstrap URL.
ENVIRONMENT_NAMEZ-Corp OPERATR (Staging)A unique system name to display in the navigation menu.
HTTP_PORTDefault: 3000The OPERATR UI port.
REPLICATION_FACTORDefault: 3The RF of internal OPERATR topics.
SNAPSHOT_PARALLELISMDefault: 3Increase OPEARTR internal parallelism for larger clusters.
FEATURE SWITCHES
SHOW_SPLASHDefault: trueTurn off the initial page splash screen.
ALLOW_TOPIC_INSPECTDefault: falseShow key/value fields when inspecting topics.
ALLOW_TOPIC_PRODUCEDefault: falseSupport sending messages to topics.
ALLOW_TOPIC_EDITDefault: falseSupport creating and configuring topics.
ALLOW_BROKER_EDITDefault: falseSupport configuring brokers.
ALLOW_GROUP_EDITDefault: falseSupport deleting groups and resetting group offsets.
ALLOW_SCHEMA_EDITDefault: falseShow Schema edit/delete/update UI.
ALLOW_ACL_EDITDefault: falseShow ACL edit/delete/create UI.
DATA INSPECTION
CUSTOM_SERDESeg. io.operatr.SerdeOne,io.operatr.SerdeTwoComma separated names of custom serdes found on the classpath.
DEFAULT_KEY_SERDESeg. JSONThe default key serde to use when inspecting data
DEFAULT_VALUE_SERDESeg. AVROThe default key value to use when inspecting data
AVAILABLE_KEY_SERDESeg. JSON,String,Transit / JSONThe list of key serdes to present when inspecting data
AVAILABLE_VALUE_SERDESeg. JSON,String,io.operatr.SerdeOneThe list of key serdes to present when inspecting data
SSO: OKTA INTEGRATION
OPENID_PROVIDER_TYPEoktaThe OPENID provider configured for SSO
OKTA_ORGANISATIONyour-organisationThe subdomain of your Okta browser session, eg z-corp.okta.com.
OPENID_CLIENT_IDeg. 0oa3zqe4plf29gk1234x5The 'Client ID' found in the "Client Credentials" section of your Okta integration.
OPENID_CLIENT_SECRETeg. Atxjoc-u_af1GAmabhdedv-fm3dojdMqn9zOSZyxThe 'Client Secret' found in the "Client Credentials" section of your Okta integration.
OPENID_LANDING_URIeg. https://staging.operatr.z-corp.com/The absolute URL to redirect to after successful OKTA togin
SSO: GITHUB INTEGRATION
OPENID_PROVIDER_TYPEgithubThe OPENID provider configured for SSO
OPENID_TOKEN_URIhttps://github.com/login/oauth/access_tokenGithub Enterprise: [Server URL]/login/oauth/access_token
OPENID_AUTH_URIhttps://github.com/login/oauth/authorizeGithub Enterprise: [Server URL]/login/oauth/authorize
OPENID_API_URIhttps://api.github.com/userGithub Enterprise: [Server URL]/api/v3/user
OPENID_CLIENT_IDeg. 0oa3zqe4plf29gk1234x5The 'Client ID' found in your configured Github Oath App.
OPENID_CLIENT_SECRETeg. Atxjoc-u_af1GAmabhdedv-fm3dojdMqn9zOSZyxThe 'Client Secret' found in your configured Github Oath App.
OPENID_LANDING_URIeg. https://staging.operatr.z-corp.com/The absolute URL to redirect to after successful Github togin
SECURITY
SECURITY_PROTOCOLSASL_SSLPLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL.
SASL_MECHANISMPLAINGSSAPI, AUTHBEARER, SCRAM, PLAIN, or Delegation Tokens.
SASL_JAAS_CONFIGorg.apache.kafka.common.security.plain.PlainLoginModule...Java Authentication and Authorization Service configuration.
SSL_KEYSTORE_LOCATION/ssl/kafka.keystore.jksThe path to a keystore when authenticating with certificates.
SSL_KEYSTORE_PASSWORDkeystore-pass-123The password to access a keystore.
SSL_KEY_PASSWORDkey-pass-123The password of the key within the keystore.
SSL_KEYSTORE_TYPEDefault: JKSThe file format of the keystore file.
SSL_KEYMANAGER_ALGORITHMDefault: SunX509The algorithm used by key manager factory for SSL connections.
SSL_TRUSTSTORE_LOCATION/ssl/kafka.truststore.jksThe path to a keystore when authenticating with certificates.
SSL_TRUSTSTORE_PASSWORDtruststore-pass-123The password to access a truststore.
SSL_TRUSTSTORE_TYPEDefault: JKSThe file format of the truststore file.
SSL_TRUSTMANAGER_ALGORITHMDefault: PKIXThe algorithm used by trust manager factory for SSL connections.
SSL_ENDPOINT_IDENTIFICATION_ALGORITHMhttpsOften required when authenticating via SSL.
SSL_PROVIDERDefault: The default security provider of the JVM.The name of the security provider used for SSL connections.
SSL_CIPHER_SUITESDefault: All the available cipher suites are supported.A list of cipher suites.
SSL_PROTOCOLDefault: TLSTLS, TLSv1.1 and TLSv1.2.
SSL_ENABLED_PROTOCOLSDefault: TLSv1.2,TLSv1.1,TLSv1The list of protocols enabled for SSL connections.
SCHEMA REGISTRY
SCHEMA_REGISTRY_URLhttps://a.schema.registryThe URL to your Schema Registry.
SCHEMA_REGISTRY_AUTHUSER_INFOURL, USER_INFO, or SASL_INHERIT
SCHEMA_REGISTRY_USERschema-userThe username when using URL / USER_INFO
SCHEMA_REGISTRY_PASSWORDschema-pass-123The password when using URL / USER_INFO
DATA EGRESS
RIEMANN_HOSTriemann.operatr.localThe Riemann host for metrics egress.
RIEMANN_PORT5555The Riemann port for metrics egress.

QUICK STARTS

Example OPERATR configuration for a variety of connection options, Schema Registry, and Riemann egress.


Apache Kafka

OPERATR was built for Open Source Apache Kafka v1.0+ and supports all connection options.


CLUSTER WITHOUT AUTHENTICATION
> docker run -p 3000:3000 --env-file ./cluster.env operatr/operatr:latest
cluster.envBOOTSTRAP=kafka-1:9092,kafka-2:9092,kafka-3:9092
ALLOW_TOPIC_INSPECT=true
CLUSTER WITH SASL_PLAINTEXT / PLAIN
> docker run -p 3000:3000 --env-file ./cluster.env operatr/operatr:latest
cluster.envBOOTSTRAP=kafka-1:9092,kafka-2:9092,kafka-3:9092
SECURITY_PROTOCOL=SASL_PLAINTEXT
SASL_MECHANISM=PLAIN
SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username="user" password="secret";
ALLOW_TOPIC_INSPECT=true
CLUSTER WITH SASL_SSL / PLAIN / HTTPS ENDPOINTS
> docker run -p 3000:3000 --env-file ./cluster.env operatr/operatr:latest
cluster.envBOOTSTRAP=kafka-1:9092,kafka-2:9092,kafka-3:9092
SECURITY_PROTOCOL=SASL_SSL
SASL_MECHANISM=PLAIN
SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username="key" password="secret";
SSL_ENDPOINT_IDENTIFICATION_ALGORITHM=https
ALLOW_TOPIC_INSPECT=true
CLUSTER WITH SASL_PLAINTEXT / SCRAM-SHA-256
> docker run -p 3000:3000 --env-file ./cluster.env operatr/operatr:latest
cluster.envBOOTSTRAP=kafka-1:9092,kafka-2:9092,kafka-3:9092
SECURITY_PROTOCOL=SASL_PLAINTEXT
SASL_MECHANISM=SCRAM-SHA-256
SASL_JAAS_CONFIG=org.apache.kafka.common.security.scram.ScramLoginModule required username="key" password="secret";
ALLOW_TOPIC_INSPECT=true

Amazon MSK

MSK doesn't expose JMX metrics and OPERATR doesn't use them. It's a match made in heaven.

See our full guide to deploying OPERATR in ECS or EKS.


AMAZON MSK / MUTUAL TLS / RIEMANN EGRESS
> docker run -p 3000:3000 -v {absolute_path}:/ssl --env-file ./msk.env operatr/operatr:latest
msk.envBOOTSTRAP=kafka-1:9092,kafka-2:9092,kafka-3:9092
SECURITY_PROTOCOL=SSL
SSL_ENDPOINT_IDENTIFICATION_ALGORITHM=
SSL_KEYSTORE_LOCATION=/ssl/keystore.jks
SSL_KEYSTORE_PASSWORD=password
RIEMANN_HOST=riemann.internal
RIEMANN_PORT=5555
ALLOW_TOPIC_INSPECT=true

Instaclustr

See Instaclustr's deep dive review of OPERATR and guide to integrating seamlessly with their managed Apache Kafka service.


INSTACLUSTR / SASL_SSL / SCHEMA REGISTRY
> docker run -p 3000:3000 -v {absolute_path}:/ssl --env-file ./instaclustr.env operatr/operatr:latest
instaclustr.envBOOTSTRAP=kafka-1:9092,kafka-2:9092,kafka-3:9092
SSL_ENABLED_PROTOCOLS=TLSv1.2,TLSv1.1,TLSv1
SSL_TRUSTSTORE_LOCATION=/ssl/truststore.jks
SSL_TRUSTSTORE_PASSWORD=instaclustr
SSL_PROTOCOL=TLS
SECURITY_PROTOCOL=SASL_SSL
SASL_MECHANISM=SCRAM-SHA-256
SASL_JAAS_CONFIG=org.apache.kafka.common.security.scram.ScramLoginModule required username="ickafka" password="secret";
SCHEMA_REGISTRY_URL=https://ickafkaschema:[email protected]:8085
SCHEMA_REGISTRY_AUTH=URL
ALLOW_TOPIC_INSPECT=true

Aiven

See Aiven's deep dive review of OPERATR and guide to integrating seamlessly with their managed Apache Kafka service.


AIVEN / SASL_SSL / TRUSTSTORE
> docker run -p 3000:3000 -v {absolute_path}:/ssl --env-file ./instaclustr.env operatr/operatr:latest
instaclustr.envBOOTSTRAP=brokers-f411.aivencloud.com:12312
SECURITY_PROTOCOL=SASL_SSL
SASL_MECHANISM=PLAIN
SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username="avnadmin" password="removed";
SSL_ENDPOINT_IDENTIFICATION_ALGORITHM=
SSL_TRUSTSTORE_LOCATION=/ssl/client.truststore.jks
SSL_TRUSTSTORE_PASSWORD=removed

Confluent

OPERATR is compatible with Confluent Platform, some disk telemetry is unavailable in Confluent Cloud.


CONFLUENT PLATFORM (SINGLE BROKER CLUSTER)
> docker run -p 3000:3000 --env-file ./cluster.env operatr/operatr:latest
cluster.envBOOTSTRAP=broker:29092
REPLICATION_FACTOR=1
CONFLUENT CLOUD / SASL_SSL / SCHEMA REGISTRY
> docker run -p 3000:3000 --env-file ./confluent.env operatr/operatr:latest
confluent.envBOOTSTRAP=kafka-1:9092,kafka-2:9092,kafka-3:9092
SECURITY_PROTOCOL=SASL_SSL
SASL_MECHANISM=PLAIN
SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username="key" password="secret";
SSL_ENDPOINT_IDENTIFICATION_ALGORITHM=https
SCHEMA_REGISTRY_URL=https://schema.us-east-2.aws.confluent.cloud
SCHEMA_REGISTRY_AUTH=USER_INFO
SCHEMA_REGISTRY_USER=schema-key
SCHEMA_REGISTRY_PASSWORD=schema-secret
ALLOW_TOPIC_INSPECT=true