--- type: "article" title: "Kafka, AMQP, MQTT over HTTP/3? My thought process this week." summary: "I couldn't resist. This week I started exploring what would it take to have message brokers talk over HTTP/3. What happened next would shock you :P" newsletter: "The Weekly Shift" newsletter_handle: "franmendez" newsletter_url: "https://usecommune.com/n/franmendez" author: "Fran Mendez (@fmvilas)" published: "2025-05-16T06:00:00.000Z" canonical_url: "https://usecommune.com/n/franmendez/a/kafka-amqp-mqtt-over-http-3-my-thought-process-this-week" markdown_url: "https://usecommune.com/n/franmendez/a/kafka-amqp-mqtt-over-http-3-my-thought-process-this-week.md" chat_url: "https://usecommune.com/n/franmendez/a/kafka-amqp-mqtt-over-http-3-my-thought-process-this-week/chat" source_url: "https://newsletter.fmvilas.me/posts/kafka-amqp-mqtt-over-http-3-my-thought-process-this-week" body_source: "imported" likes: 1 replies: 0 body_words: 1380 --- # Kafka, AMQP, MQTT over HTTP/3? My thought process this week. ![](https://embed.filekitcdn.com/e/tppsnKSSV6RtcQVLeDZHT8/5nEuhXaXN7FANZ4f1qdrrM) # Kafka, AMQP, MQTT over HTTP/3? This week I posted a video on YouTube and [LinkedIn](https://www.linkedin.com/posts/fmvilas_eventdrivenarchitecture-http-http3-activity-7328074716030631955-WPts?utm_source=share&utm_medium=member_desktop&rcm=ACoAAAsbnDEBBLfl48I4dD1-JFdpDe8WDOcxQyg) throwing some thoughts that were coming to my head week after week: [![video preview](https://functions-js.convertkit.com/playbutton?play=%233197e0&accent=%23ffffff&thumbnailof=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D5DxZwma1y60&width=480&height=270&fit=contain)​](https://www.youtube.com/watch?v=5DxZwma1y60) We all work with messaging systems: Kafka for high-throughput streams, AMQP for complex enterprise routing, MQTT for IoT and lightweight pub/sub. They're powerful, but most still run on raw TCP connections, just like they have for years. Now, [HTTP/3](https://www.rfc-editor.org/rfc/rfc9114) (running on [QUIC](https://www.rfc-editor.org/rfc/rfc9000.html)) is here, offering a ton of under-the-hood improvements: faster handshakes ([0-RTT](https://blog.cloudflare.com/even-faster-connection-establishment-with-quic-0-rtt-resumption/)), built-in TLS 1.3+, no head-of-line blocking across streams, and connection migration (your client switches from Wi-Fi to mobile, the connection *might actually survive*). So, the question inevitably pops up: **can we get our messaging workhorses to leverage this new transport without a total overhaul? And critically, can they still evolve independently?** Let me walk you through my thought process this week, including a few dead ends. My first instinct was to think about HTTP/3's extensibility. "HTTP/3 has frames. Why not just create new ones? `KAFKA_PRODUCE_FRAME`, `MQTT_PUBLISH_FRAME`, `AMQP_TRANSFER_FRAME`. Direct. Clean." Sounds logical, right? On the surface, it’s a direct mapping. Then you dig a bit deeper. HTTP/3 frames (`DATA`, `HEADERS`, `SETTINGS`, `PRIORITY_UPDATE`, etc.) are the nuts and bolts for how *HTTP itself* functions – managing requests, responses, control information, and features *of the HTTP protocol*. If we started injecting `KAFKA_PRODUCE_FRAME`, we wouldn't really be speaking HTTP/3 anymore. We'd have a custom protocol that *looks a bit* like HTTP/3. Standard HTTP/3 clients, servers, proxies, and debug tools? They wouldn't have a clue what to do with it. We'd also lose all the semantic value of HTTP methods (`GET`, `POST`), status codes, and standard headers. Plus, trying to standardize such application-specific frames across the IETF would be… ambitious, to put it mildly πŸ˜… So, custom frames are out for this purpose. Then I thought: alright, if custom frames are a no-go, what about using *actual* HTTP requests? - Kafka `Produce` -> `POST /topics/my-topic/messages` - MQTT `Publish` -> `POST /mqtt/topics/device-updates` - AMQP `Transfer` -> `POST /amqp/exchanges/my-exchange/publish` This is much more HTTP-idiomatic. You get to use existing HTTP libraries, and conceptually, it's clear. For many request/response style interactions in these protocols, this could work. My brain started screaming "HACK! DON'T FOLLOW THIS PATH!" πŸ˜„ It's definitely not a perfect fit. Not even a good one, at least not for a protocol. The persistent, stateful, bidirectional nature of many messaging sessions (think AMQP channels or an MQTT client constantly subscribed and receiving messages) can feel clunky when forced into a pure HTTP request/response model for every interaction. And how do you ensure each protocol can still add its own unique, non-RESTful features down the line without being constrained by HTTP methods? Nah, discarded! Then I remembered my old days at the university, coding the network protocols. What about tunneling? This brings us closer. Remember how [WebSockets "upgrade" an HTTP/1.1 connection?](https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Protocol_upgrade_mechanism) HTTP/3 has a similar mechanism called ["Extended CONNECT"](https://datatracker.ietf.org/doc/html/rfc9220#name-websockets-upgrade-over-htt) (defined in RFC 9220 for bootstrapping WebSockets, but the concept is general). The idea: 1. Client makes an HTTP/3 `CONNECT` request. 2. Includes a header indicating the desired protocol, e.g., `protocol: kafka` or `Sec-Kafka-Protocol: v2.8`. 3. If the server agrees, the HTTP/3 stream (or potentially the whole connection) effectively becomes a raw binary tunnel for the specified protocol. This is pretty good. It leverages HTTP/3 for the initial secure, multiplexed handshake and then gets out of the way, letting the native protocol flow. This definitely allows for independent evolution. While Extended CONNECT is solid for tunneling, there's another IETF/W3C effort that feels even more tailor-made: [WebTransport](https://www.w3.org/TR/webtransport/). I found WebTransport completely by chance searching on Google. It is built *on top of* HTTP/3 and is specifically designed for client-server messaging that needs more than traditional HTTP request/response. Think of it as a framework for diverse application protocols to run over HTTP/3. Looks like I was not the only one thinking about this possibility! πŸŽ‰ Here’s the flow: 1. Client establishes an HTTP/3 connection to a server. 2. Client sends a specific HTTP/3 `CONNECT` request to a designated WebTransport endpoint (e.g., `https://messaging.example.com/.well-known/kafka-service-wt`). This request itself can signal the intent to speak Kafka, AMQP, etc., perhaps by different endpoint paths or a custom header. 3. Once the WebTransport session is established, the magic happens: - It provides an API to open multiple **bidirectional and unidirectional QUIC streams**. - It also supports sending **unreliable QUIC datagrams**. 4. Over these streams (or datagrams), the client and server exchange the **native binary messages** of Kafka, AMQP, MQTT, or whatever protocol was negotiated for that session. Why I think WebTransport Hits the Sweet Spot: - **Native Protocol Preservation:** Kafka still speaks Kafka binary. MQTT still speaks MQTT binary. No re-mapping of operations to HTTP methods if you don't want to. This is huge for "independent evolution." - **Full QUIC Benefits:** You get the multiplexing, security, and connection management of QUIC/HTTP/3. - **Flexibility:** Reliable streams map perfectly to most messaging operations. Unreliable datagrams could be a fit for things like MQTT QoS 0 or non-critical telemetry. - **Standardized Approach:** It's being developed as a standard, aiming to be a common way to run such applications over HTTP/3. So, a Kafka client could initiate a WebTransport session, tell the server "let's talk Kafka," and then just start firing off its usual binary `ProduceRequest`s and `WorkspaceRequest`s over WebTransport streams. Same for an MQTT client with its `CONNECT`, `PUBLISH`, `SUBSCRIBE` packets. Obviously, this would still require message brokers to accept connections via WebTransport. Or event gateways like Gravitee, Hoockdeck, and others to support it. It's not an easy task but it's definitely easier than standardizing new HTTP/3 frames for messaging protocols πŸ˜… If we want to bring our existing, powerful messaging protocols into the HTTP/3/QUIC era without rewriting their cores or boxing them into overly rigid HTTP semantics, WebTransport looks like the most promising path. It offers a standardized session layer that respects the integrity and independent evolution of the protocols running on top of it. It's still relatively early days for WebTransport adoption compared to raw TCP or even WebSockets, but the foundation is solid and designed for exactly this kind of challenge. Definitely something to keep an eye on (or even experiment with) as we look to modernize our infrastructure. And a good place for people looking to contribute to open source and standards! When I first published the video, I wasn't even aware of WebTransport existence. I'm happy they started this effort. It's now technically possible to communicate with a server using a binary protocol. In other words, it opens the doors for my dreamed capability: interacting with a broker from the browser. I'm quite excited this is now "possible". From time to time, I love to challenge the status quo β€”even if just in my head. This lets me visualize where each piece of the puzzle fits in. It's always (always!) a good idea to understand the reasons why things are the way they are today. But you gotta dream. Dreams are free and sometimes they ignite the flame of future interesting initiatives, if you know what I mean πŸ˜‰ For some reason, writing this email reminded me of this song. I know it makes no sense. But I figured I'd share it with you because it's been on repeat during all the time it took me to write this. [![artist](https://i.scdn.co/image/ab67616d0000b273eb0652df7fe1fddc6673d161) 1955 Hilltop Hoods, Montaigne, To... PREVIEW ![Spotify Logo](https://functions-js.convertkit.com/icons?icon=spotify&foreground=ffffff&background=000000&shape=icon-only&scale=1) ​](https://open.spotify.com/track/6oSmp7qnvAxUXeUKhl67a6) **P.S.** If you're stuck, need a second pair of eyes on your architecture, or want personalized guidance to accelerate your project, a 1:1 call can provide the clarity and direction you need. [Book Your 1:1 Consultation.](https://cal.com/fran_mendez)​ ### 1 Question For You In my last email, I asked a very simple question hoping to understand a bit more about you: "are you using AsyncAPI?". Well, it was a complete failure. I only got 3 people to click on the poll. Well, 4 because I also voted to test if it worked πŸ˜… So let me fail once again πŸ˜› Now seriously, understanding how many of you are using AsyncAPI would immensely help me craft future content pieces which, in turn, would be beneficial for you too. And it's free! 🎁 Are you using AsyncAPI in your organization? [https://polls.kit.com/22462?option_id=79612&token=preview](https://polls.kit.com/22462?option_id=79612&token=preview) [Yes](https://polls.kit.com/22462?option_id=79612&token=preview) [https://polls.kit.com/22462?option_id=79613&token=preview](https://polls.kit.com/22462?option_id=79613&token=preview) [No](https://polls.kit.com/22462?option_id=79613&token=preview) [![linkedin](https://functions-js.convertkit.com/icons?icon=linkedin&foreground=ffffff&background=153461&shape=square&v=3)​](https://linkedin.com/in/fmvilas)[![youtube](https://functions-js.convertkit.com/icons?icon=youtube&foreground=ffffff&background=153461&shape=square&v=3)​](https://youtube.com/@fmvilas)[![github](https://functions-js.convertkit.com/icons?icon=github&foreground=ffffff&background=153461&shape=square&v=3)​](https://github.com/fmvilas) Av. JoaquΓ­n Costa, 16, Badajoz, Badajoz 06001 ​[Unsubscribe](https://preview.convertkit-mail2.com/unsubscribe) Β· [Preferences](https://preview.convertkit-mail2.com/preferences)​ *** ## Discussion No replies yet.