Kafka is an excellent distributed messaging platform, but operating it at scale often introduces challenges that go beyond message delivery itself. As more teams, applications, and external clients start depending on a Kafka cluster, concerns such as secure access, authorization, governance, auditing, and infrastructure evolution become increasingly important.
In this interactive session, we’ll start with a production-style Kafka deployment and gradually introduce the operational challenges we encountered while evolving it. Together, we’ll evaluate different approaches—from Kafka ACLs and networking improvements to protocol-aware proxies and other alternatives—and discuss the trade-offs behind each decision.
Through live demonstrations and audience-driven discussions, we’ll explore questions such as:
How should external applications securely access Kafka?
Are Kafka ACLs sufficient for growing operational needs?
How do we improve governance and observability without changing every client?
Should we solve these problems within Kafka, or introduce an additional layer?
If introducing a proxy is the answer, why Kroxylicious? What are the alternatives?
We’ll also discuss where Kafka itself remains the right choice, where operational tooling complements it, and the trade-offs involved in introducing components such as Kafka protocol proxies.
Rather than presenting a single “correct” solution, this session focuses on the engineering thought process behind evolving a production-style Kafka deployment and making informed architectural decisions.
What you’ll learn
Common operational challenges encountered in Kafka deployments as they scale
Where Kafka ACLs fit—and where they may not be sufficient
Different approaches for securing and governing Kafka deployments
How protocol-aware proxies such as Kroxylicious can help address operational concerns
Trade-offs, limitations, migration strategies, and alternative approaches
Format
We’re planning this to be an interactive session combining discussion, architecture exploration, and some live demonstrations. We discuss design decisions, evaluate alternative solutions together, challenge assumptions and propose different approaches.
Prerequisites
To get the most out of this session, participants should be familiar with:
Basic Kafka concepts (topics, partitions, producers, consumers, consumer groups)
Kubernetes fundamentals
Networking basics (TCP/IP, DNS, TLS/mTLS)
General understanding of distributed systems
No prior knowledge of Kafka security, ACLs, or Kroxylicious is required.
