Engineering Beyond the Broker

Where we started

Picture a Kafka deployment that has quietly become shared infrastructure. Producers and consumers belong to many different teams, each client application sits in its own network, and a growing number of services depend on the same cluster.

It runs continuously, there is no maintenance window, and if the stream stops, everything downstream stops with it.

Kafka did exactly what it promised. Messages flowed.
But as the estate grew, we noticed that running Kafka well was a different job from having chosen Kafka, and that nobody had done that second job.

Kafka in thirty seconds

A quick refresher, because the whole story hinges on one detail of how Kafka works.

⁍ Kafka is a log: producers append events to the end, and they stay put.
⁍ A topic is a named log split into partitions, so many consumers can work in parallel.
⁍ Partitions are replicated across brokers, and each partition is read by one consumer within a consumer group.
⁍ Producers and Consumers connect to Partition leader to produce and consume respectively.

The other half of the story

Our setup looked like this: external clients, each in its own network, each carrying a route table with one entry per Kafka broker. A router sat between the client networks and the cluster network, and SSL terminated on each broker, so every broker carried its own certificate.

This worked, until two problems surfaced.

1. Infrastructure coupling. A new broker means a new IP, and a new IP means a route table change in every client network, coordinated across teams that share no change window. Scaling the cluster became a cross-team negotiation.

2. Nobody can name a topic's owner. Traffic was anonymous. There was no identity to attribute a producer to, no centralised audit trail for access decisions, and no practical way to manage consumer group lifecycles or retire stale groups.

Kafka did its job. Nobody did the other half.
Kafka is a message platform plus an operational layer. It delivered the first.
The second was always ours to build.

The wheels are still on. What do you do with the time?

The cluster was healthy, which is the best moment to fix operational debt, before an incident forces your hand. We considered three options.

Option 1: Improve the routing (rejected)
Replace the three per-broker routes with one subnet route, for example 10.4.2.0/24. Route-table churn disappears because new brokers land inside an already-routed range. It's cheap, and it's the first week of any plan.

But a subnet route widens the blast radius: clients can now reach every host in that range, not just Kafka. And there's still no identity, no audit and no central place to authorise.

Option 2: Enable Kafka ACLs (necessary, not sufficient)
ACLs give you one decision at the broker: allow or deny, per principal, on topics, groups and cluster operations. That's valuable, and we'd still want it. But: 
Routing is still unchanged. Clients still dial broker IPs across networks.

The pattern
Each fix solved one thing. A subnet route reaches routing; ACLs reach authorisation. Two problems meant two separate fixes in two places, in every client network.
So we asked: what if one component in front of the cluster handled both?

Option 3: Put a proxy in front (selected)
A proxy gives you: 
⁍ One address: a single endpoint every client connects to, whatever the cluster does behind it.
⁍ One gate: it's in the path, so it can accept or reject a connection before the cluster ever sees it.
⁍ One router: it inspects traffic, works out which broker a request belongs to, and sends it there.

In practice that maps to routing (one endpoint per cluster, one route, one firewall rule), filtering (policy applied before the broker, with rules richer than allow/deny) and visibility (every connection passes through one place, so traffic can be attributed to a team before anything is restricted).

That last point matters.
We didn't want to start by locking things down blind. A proxy lets you observe first, then enforce.

The cost is real: a new component to deploy, scale and operate, plus one extra hop. We'll measure that hop below.

The proxy has to speak Kafka

Here's the catch.
Kafka isa binary protocol over TCP. A plain L4 proxy sees a TCP connection and forwards bytes to whichever backend it picked. It can't tell a metadata request from a produce request, and it can't know which broker leads a given partition. A protocol-aware proxy reads the Kafka frames, decodes each request, applies policy, routes it to the right broker, and can rewrite the response on the way back.

To sit in front of Kafka, you have to implement Kafka's protocol. That is the whole job.

How a client finds a broker
The reason a naive proxy fails comes down to how Kafka clients discover brokers:

1. Bootstrap. The client connects to any broker from  bootstrap.servers  (two or three addresses).
2. Metadata request. That broker describes the cluster: topics, partitions, the leader of each partition, and the host and port of every broker.
3. Produce / consume. The client dials the partition leader directly, at the address it was just given.

The addresses the client uses come from the cluster itself. So whoever controls the metadata response controls where clients connect.

Seeing it on the wire

Capturing a metadata response sent directly by Kafka, we see the cluster hand back its own internal IPs:

Those come from each broker's advertised.listeners, and every client network needs a route to all three. That's the coupling problem, visible in a packet capture.

Now the same request through  Kroxylicious, an open-source Kafka proxy: same node ids, but addresses the proxy owns.

node 1 → broker-1.…sahaj.ai:9083
node 2 → broker-2.…sahaj.ai:9083
node 3 → broker-3.…sahaj.ai:9083

The client obeys the metadata it's given, so it now dials the proxy for every broker and never learns an internal IP.
That's the trick: metadata rewriting.

Giving each broker its own address: ports vs. SNI

To rewrite metadata, the proxy needs one distinct address per broker. Kroxylicious offers two ways to produce them.

Option A: port per broker. One host, many ports ( proxy.kafka.com:9093  for node 1,  :9094  for node 2, and so on). It works without TLS, but the port range must be open end to end and widens every time the cluster grows.

Option B: SNI (our choice). Many hostnames, one port ( broker-1.kafka.com:9083 ,  broker-2.kafka.com:9083 , …). It needs TLS from the client, but you get one firewall rule, one port, and a cluster that can grow behind a wildcard name.

How SNI routing works

The client opens a TLS connection and states which host it wants in the handshake (Server Name Indication). The proxy reads that name, turns it into a node id, and connects to the matching upstream broker: 

.kafka-table-wrapper { width: 100%; overflow-x: auto; margin: 16px 0; } .kafka-table { width: 100%; border-collapse: collapse; font-family: inherit; font-size: 16px; color: #333333; border: 2px solid #cbd5e1; } .kafka-table th { background-color: #f8fafc; font-weight: 600; text-align: left; padding: 14px 18px; border: 1px solid #cbd5e1; } .kafka-table td { background-color: #ffffff; padding: 14px 18px; border: 1px solid #cbd5e1; }
SNI hostnameNode idUpstream broker
broker-1.kafka.com:9083node 110.90.0.16:9092
broker-2.kafka.com:9083node 210.90.0.18:9092
broker-3.kafka.com:9083node 310.90.0.21:9092

Because the node id lives in the hostname, the mapping is a pattern, not a table anyone maintains by hand.One gotcha worth printing on a sticky note: clients must connect by name. Point a client at the proxy's IP and there's no SNI, so the proxy can't tell which broker you meant.

Before and after

Before: client → three broker IPs, three addresses in every route table.
After: client → load balancer → Kafka proxy → Kafka. One name in every route table.

On a single connection, it works like this: the client resolves every broker name to the same load balancer IP, so the client network needs exactly one route. The load balancer passes traffic to Kroxylicious, where the client's certificate is verified. The proxy then routes the request to the brokers in the cluster network, and the response returns through the proxy with rewritten metadata. The client only ever sees proxy hostnames, never a broker IP.

What changes day to day:

.kafka-comparison-wrapper { width: 100%; overflow-x: auto; margin: 16px 0; } .kafka-comparison-table { width: 100%; border-collapse: collapse; font-family: inherit; font-size: 16px; color: #333333; border: 2px solid #cbd5e1; } .kafka-comparison-table th { background-color: #f8fafc; font-weight: 600; text-align: left; padding: 14px 18px; border: 1px solid #cbd5e1; } .kafka-comparison-table td { background-color: #ffffff; padding: 14px 18px; border: 1px solid #cbd5e1; vertical-align: top; line-height: 1.5; } .kafka-comparison-table th:nth-child(1), .kafka-comparison-table td:nth-child(1) { width: 22%; } .kafka-comparison-table th:nth-child(2), .kafka-comparison-table td:nth-child(2) { width: 45%; } .kafka-comparison-table th:nth-child(3), .kafka-comparison-table td:nth-child(3) { width: 33%; }
TaskTodayWith the proxy
Add a brokerA route and firewall change in every client networkNothing outside the cluster changes
Rotate a certificatePer broker, coordinated with every clientOnce, at the proxy
Ask who is producingNobody can answerVisible at the proxy

Two of the three land on the network team rather than on Kafka, and the third is a question nobody could answer before.

What else comes with it

Once traffic flows through a protocol-aware proxy, several things come almost for free: 

⁍ Auditing: who connected, to what, and when, recorded without touching a broker.
⁍ Independent scaling: clients stop knowing the backend, so the cluster can grow or move with no client-side networking change.
⁍ Central authorisation: allow or deny decisions keyed on the client's X.509 certificate principal, in one config, for the whole estate.
⁍ Filters: Kroxylicious has a middleware chain (encryption, validation, multi-tenancy and more) that we haven't even spent yet. That's a post of its own.

Measuring the hop

A proxy adds a hop, so we measured before arguing about it. Producer test: 100k records, 100-byte payload, 12 partitions, replication factor 3,  acks=1 , on three t3a.medium brokers.

.kafka-section-wrapper { width: 100%; font-family: inherit; font-size: 16px; color: #333333; } .kafka-table-scroll { width: 100%; overflow-x: auto; margin: 16px 0; } .kafka-table { border-collapse: collapse; border: 2px solid #cbd5e1; white-space: nowrap; } .kafka-table th { background-color: #f8fafc; font-weight: 600; text-align: left; padding: 14px 20px; border: 1px solid #cbd5e1; } .kafka-table td { background-color: #ffffff; padding: 14px 20px; border: 1px solid #cbd5e1; } .kafka-section-text { margin: 28px 0 24px 0; line-height: 1.5; }
PathThroughputBandwidthAvg latency
Plaintext, direct57.3k rec/s5.46 MB/s43 ms
SSL, direct36.2k rec/s3.46 MB/s393 ms
SSL, via proxy33.6k rec/s3.21 MB/s427 ms

Consumer test at 50,000 messages on the same topic and cluster:

PathMessages/secFetch rate
Plaintext, direct12.2k9.9 MB/s
SSL, direct9.1k3.9 MB/s
SSL, via proxy8.7k3.6 MB/s

The interesting finding is where the cost comes from. Moving from plaintext to SSL cost far more than adding the proxy did: producer throughput fell by roughly 37% going to TLS, and by only a further 7% or so going through the proxy. On the consumer side the proxy cost about 4%. Rebalances also take around 700 ms longer under TLS, with or without the proxy.

In short, TLS is the expensive part, and you were going to need it anyway. The proxy's own overhead is modest. Per-request latency for the extra hop is in the region of 1–2 ms, though the producer average above is dominated by batching and TLS effects, so benchmark your own workload before committing.

Trade-offs, honestly

What we gain: routing, policy and visibility, with Kafka staying exactly as it is.
What we pay:
⁍ An extra hop (roughly 1–2 ms of latency) 
⁍ A new component to operate 
⁍ A proxy that has to be scaled 
⁍ More deployment complexity 

If your estate is small, has one client network and no audit requirements, this is probably overkill. If you have many teams, many networks and a growing need to answer "who is using this?", the trade is easy.

Migrating without a big bang

The riskiest way to adopt this is a single call with every team on it, changing bootstrap addresses together. One failure there and the whole idea is dead.

We moved one client at a time, with the old path still open:
1. Whitelist the proxy load balancer, and only that. One firewall rule replaces the per-broker rules. The old routes stay in place until the client is done.

2. Point  bootstrap.servers  at the proxy domain. By name, never by IP, since SNI is how the proxy learns which broker the connection is for. Rollback is the same line, reversed.

3. Watch before enforcing. Traffic flows unrestricted while you learn what this client really produces and consumes, and which certificate principal it presents.

4. Turn on authorisation for that client. Scope it to the topics you observed, keyed on its X.509 principal. Identity finally has something to attach to.

One client, one week, one visible win, and the next team moves with a reference to point at.

Three things to keep :

The broker address was the contract. Everything else followed from that.

A proxy didn't make the estate secure. It made securing it survivable, one client at a time, with the old path still open.

The routing fix was the cheap part. Identity is the project that follows.

Further reading :

Kroxylicious documentation: https://kroxylicious.io/ 
Kafka security best practices: https://kafka.apache.org/44/security/