Verrency – FinTech, Kafka, and kPow
Verrency is one of Melbourne’s most exciting FinTech start-ups, they use Apache Kafka® to power card and payments innovation.
Chief Technology and Chief Product Officer (and kPow early adopter) Euan Walker had a virtual chat with us covering FinTech, the highs and lows of working with Apache Kafka, and how Verrency use kPow in their organization.
Who are Verrency?
Verrency is an Australian FinTech, we allow banks to include value-added services to their cards and payments products without a big internal IT spend. We also provide solutions to personalise aspects of the banking experience so consumers can tailor products to their needs more effectively.
As a FinTech focused on global growth and working with some of the most conservative technology buyers (banks), we constantly have to balance the need to be operationally flexible. We do this while having a zero downtime goal, and hitting performance targets that don’t leave a lot of room for mistakes without gold plating the solution.
How does Verrency use Apache Kafka for its operations? How critical is it?
Verrency uses Kafka as a core component within our platform, and it’s mission-critical for us.
We effectively use it as a real-time transactional database where payments are processed and for different components of our solution to work synchronously. We also feed data into third-party systems which don’t necessarily have the same up-time or performance standards like us, so we rely on Kafka as part of our integration and re-try mechanism, letting our partners can operate at their own speed.
When did you first start working with Kafka? What are your impressions of it?
We started working with Kafka three years ago – initially as a pilot where we were also looking at Kinesis, and then as a full part of our infrastructure.
Kafka is a really powerful tool and when used well can bring great benefits to an organisation, but I think there are still a couple of challenges in implementing it. It’s almost a decade old however there still aren’t many people who’ve done big, complex Kafka projects here in Melbourne, particularly compared to something like MQ Series and that can pose a challenge. It takes time to really get the best out of it and we’ve been fortunate to have some engineers who’ve really got to know Kafka well, but it does have a learning curve. You have to allow for that time.
The other key things are that there aren’t particularly rich, cost-effective, supporting eco-system of tools to help with management, deployment, and monitoring. This has started to change in the past 12 months with things like Amazon’s MSK and tools like kPow. That paints a pretty good picture of things to come.
Tell me about your tech team, do they all work with Kafka?
We have a pretty modest engineering team – currently eight engineers – and everyone works with Kafka at some level as it’s a core component of our infrastructure.
Operatr.IO’s Co-Founder Derek Troy-West talks about engineers ‘flying blind’ due to Kafka not having a dashboard. Have your team experienced that?
That’s absolutely true in our experience. Your data is hidden away with very little out-of-the-box, user friendly, tools to engage with. Troubleshooting can be very time consuming – even for what you’d hope would be a simple task.
I think this is actually a key challenge for Kafka in smaller organisations. If you can’t find a cost-effective solution to address it, there would be times when as a CTO, you would need to really re-evaluate using Kafka.
When did Verrency start using kPow?
We already had a great relationship with Derek and his team through the Melbourne Distributed Forum that he runs with our former Head of Engineering.
We heard about Operatr (now kPow) while it was still in development. Naturally, we put up our hands to be one of the original beta users. Once it went to full GA release we had already made it a core part of our infrastructure, and I don’t see that changing any time soon.
What were your initial impressions of kPow?
We went in fairly open, as we started using it as beta users. Originally, we had expected to use it more for performance and monitoring, but quickly realised it was able to solve other problems we were looking to address. That included giving us an effective way to inspect topics and partitions for consistency and retrieve data more effectively. It was really useful for that from the start.
“kPow is clearly a product built by someone with a passion for Kafka, but who carries the scars of some tough implementations. It’s a tool by and for engineers, and it shows in its ability to interrogate Kafka for data that most people probably didn’t know you could get. I am really looking forward to seeing it continue to evolve over the next 12 months as the community using it grows. Certainly, it will be a key part of our infrastructure as long as we’re using Kafka”.
Was it difficult to start using kPow?
It was pretty easy to get started with and I think it is even easier if you’re starting now.
Since its initial release we have really seen kPow evolve – both on the their product roadmap as well as through feedback from the user community. Both the usability and functionality have got better and better since the first release, and the dashboard especially has really become a much easier-to-use tool.
How does your team use kPow today?
Our use case is the same: troubleshoot problems in Kafka – both looking for efficiencies and in debugging. But it’s used far more than we had originally anticipated. It’s certainly the easiest way for us to get access to data and gather information for troubleshooting – especially when looking to inspect data and monitor in real-time.
As a CTO, what the main value kPow brings your team?
It has certainly allowed my engineers to troubleshoot issues far more effectively, and for new staff to understand the inner workings of our platform more effectively. It’s hard to put a value on it, but I can say we found one particularly stubborn issue in testing several months ago that, without kPow, would have had me revisiting the decision to use Kafka. kPow allowed us to hone in on the root cause in ways I don’t think would have been practical to do manually and gave us the data to find the root cause and address the issue.
Would you recommend kPow to others?
I asked my development team this question, and this response sums up what they said:
“Yes. It is a must-have for operations. Without it, you really do fly blind. Yes, you can get to the data, but kPow makes it so easy. It’s your first port of call when there are production issues.”
Based on our experience, I think you’d be hard-pressed to really run Kafka in a mission-critical scenario without kPow.
Enjoy this article?
Sign-up to the mailing list for operatr.io news, product updates and Kafka insights.
A single Docker container or JAR file that installs in minutes, kPow gives you instant visibility of your Kafka clusters and immediate access to your data.
kPow is compatible with Apache Kafka+1.0, Amazon MSK, Instaclustr, Aiven, Red Hat AMQ Streams, Confluent Platform, and Confluent Cloud.
Start with a free 30-day trial and solve your Kafka issues within minutes.
Kylie Troy-West is the Co-Founder and COO of Operatr.IO and the Director of Operations at Troy-West. Her focus is on empowering the Operatr.IO team to deliver on their vision of building world-class tools for Apache Kafka®.