A fingerprint names a bug. It does not decide that the bug is worth waking someone for. That decision is the clustering service: it counts each fingerprint inside a five-minute tumbling window keyed on event time, and the first time a window crosses ten occurrences it emits exactly one incident-events message, then never emits for that window again. A same-clock probe puts the whole path, first log to incident row, at p50 403ms. The mechanism is a ConcurrentHashMap and a synchronized boolean. This post is the counter, the emit-once guard, and the two honest failure modes that fall out of doing it in memory with no eviction and a hard window boundary. This is post 3 in a series on building log0. Post 2 covered how a log line collapses to a stable fingerprint. This one is about what happens to that fingerprint next: how a stream of identical errors becomes one counted, thresholded, deduplicated incident, and why the counting is the part that decides whether anyone gets paged. One occurrence is not an incident The fingerprint already solved deduplication of identity: ten thousand lines of the same bug share one hash. But identity is not severity. A bug that throws once at 3 AM and never again is noise. The same bug throwing ten times in a minute is an incident. Something has to count, and counting needs two boundaries: which occurrences belong together, and how many of them is enough. log0 answers both with three numbers, and they live in ClusteringConfig: private int occurrenceThreshold = 10; // page when a window reaches this private int windowDurationMinutes = 5; // tumbling window width private int maxTopMessages = 10; // sample messages carried on the incident Ten occurrences of one fingerprint inside one five-minute window emits an incident. Nine does not. The window is tumbling, not sliding: time is chopped into fixed five-minute blocks aligned to the clock, and every occurrence falls into exactly one block. That choice is what makes the counter O(1) per event and the failure modes legible, and it is also where the first sharp edge is. The key is the whole design Every decision about what counts together is compressed into one map key. ClusterKey.of builds it: key = tenantId : fingerprint : floor(eventTime / 5min) Three fields, and each one earns its place. tenantId keeps one customer's flood from ever counting toward another's incident; the count is per tenant by construction, not by a filter. fingerprint is the bug identity from post 2. And the third field is the window: the event's own timestamp, floor-aligned to a five-minute boundary. long bucketMinutes = timestamp.truncatedTo(ChronoUnit.MINUTES).getEpochSecond() / 60; long alignedMinutes = (bucketMinutes / windowDurationMinutes) * windowDurationMinutes; That floor-division is the tumbling window. 00:00 through 00:04:59 all map to bucket 00:00; 00:05:00 starts a fresh bucket with a fresh counter. Two occurrences of the same bug in the same tenant inside the same five minutes produce the same key and increment the same counter. Cross any one of the three boundaries, a different tenant, a different bug, or the next five-minute block, and it is a different counter that knows nothing about the first. The window is keyed on event time, the timestamp the log carries, not the wall-clock time the clustering service happened to read it. That is the right choice for correctness, because a backlog draining late should still bucket each event where it truly happened, not pile a delayed burst into whichever window the consumer caught up in. It is also the source of the boundary failure mode below. The counter and the emit-once guard The state is deliberately boring. InMemoryOccurrenceStore is a ConcurrentHashMap<ClusterKey, OccurrenceWindow>, and each OccurrenceWindow holds a count, a sample of recent messages, and a single boolean. FingerprintClusterer.cluster() is the whole hot path: OccurrenceWindow window = occurrenceStore.increment(key, event.getMessage()); if (window.getCount() >= config.getOccurrenceThreshold() && window.markIncidentEmitted()) { incidentEventProducer.publish(buildIncidentEvent(event, window)); } Two conditions, and the && order matters. The count check is the obvious one: has this window reached ten. The second is the one that makes the system usable. markIncidentEmitted() is synchronized and returns true exactly once: public synchronized boolean markIncidentEmitted() { if (incidentEmitted) { return false; } incidentEmitted = true; return true; } The tenth occurrence flips the flag and emits. The eleventh, the twelfth, the ten-thousandth all increment the count and re-check the threshold, but markIncidentEmitted() now returns false, so nothing is emitted. One window, one incident, regardless of how loud the bug gets after it trips. This is the same guarantee the fingerprint gives in the identity dimension, now in the time dimension: the fingerprint stops a bug fragmenting into thousands of incidents by message text, and the emit-once guard stops one window fragmenting into thousands of incidents by occurrence count. The count does not stop climbing after the emit, and that is intentional. Occurrences eleven and up keep accumulating, because the authoritative total for the incident is recomputed downstream from ClickHouse anyway; the in-window count exists only to make the page-or-not decision. The synchronized boolean is the cheapest possible thing that turns "this is a real incident" into an event exactly once. What it costs end to end: p50 403ms The point of all of this is to page fast when a real burst arrives. A same-clock probe measures exactly that: it fires ten identical-fingerprint logs at the gateway to trip one window, then polls Postgres until the incident row appears, repeated for n=60. Because the send and the poll share one clock, the number is honest end to end, ingest to incident. The distribution is tight: mean 405ms, p50 403ms, p99 477ms, almost everything between 380 and 440ms, one outlier near 477. The full histogram lives in post 10 in this series, where it belongs to incident creation, but the number is a clustering number first: most of that 403ms is the event crossing three Kafka topics and three services, and the clustering decision itself, the increment and the threshold check, is a hash-map lookup that costs nothing measurable next to the hops. Detection is eventually consistent and lands in well under half a second on this hardware. Failure mode one: a burst that straddles a boundary never pages A tumbling window has hard edges, and event time decides which side of an edge an occurrence lands on. That makes one specific burst invisible. I measured it directly: eight trials per split ratio, a distinct fingerprint each trial, exactly ten identical errors, split across two adjacent windows by their event timestamps, then poll Postgres for the incident. When all ten land in one window, 10:0, the window hits the threshold and pages, 8 of 8 trials. Every split, 9:1, 8:2, 7:3, 6:4, 5:5, pages 0 of 8. Nine occurrences in the first window and one in the next is nine and one, and neither bucket reaches ten, so the burst of ten genuinely identical errors creates zero incidents. The counters are correct; each holds the true count for its own window. The threshold is evaluated per window, and a burst unlucky enough to land on a boundary is under-counted into silence. This is the dishonest-by-omission failure, the opposite of the fingerprint's. The fingerprint fails toward noise: too many incidents, loud and visible. The window boundary fails toward silence: a real burst that should have paged does not, and nothing tells you. A sliding window or a small grace overlap would close most of this gap; the tumbling window was chosen for its O(1) keying and its legibility, and this is the bill for that choice. It is on the list at the end, named, not hidden. Failure mode two: the store has no eviction The second cost is memory, and it is the one that bites under sustained load. InMemoryOccurrenceStore never evicts. Its own javadoc says so: every distinct (tenant, fingerprint, window) key is a permanent entry in the map, and nothing ever removes a window after its five minutes pass. As long as new fingerprints keep arriving, the map keeps growing. I drove that on purpose. Pushing 710,439 unique fingerprints through the pipeline, one new fingerprint per request, each one a brand-new map entry, and sampled the clustering container's memory against the count of distinct windows it had consumed. The climb is dead linear, about 810 bytes per distinct window, exactly what an unbounded map predicts: 176 MiB at startup, 259 at 111k windows, 371 at 257k, 484 at 398k. Extrapolate the line and it meets the 512 MiB container cap at roughly 434,000 windows. What happens there is worse than a crash. The consumer stops advancing its offset while memory keeps climbing toward the cap, detection silently halts, and nothing in the map evicts itself. A process restart clears the state and can lose the emit-once guards for in-flight windows, but it does not fix the underlying problem: without an external store, the working set is unbounded. The gateway never noticed, it took all 710,439 requests at zero failures, roughly 3,382 a second, while the thing downstream that decides who gets paged quietly stopped deciding. That is the on-theme failure for this series: not a loud death but a green dashboard over a dead detector. The fix is designed and not built: the window state belongs in an external store, Redis with a TTL that matches the window width, so a window evicts itself five minutes after it closes and the working set stays bounded no matter how many distinct bugs pass through. In memory, the map is fast and simple and has exactly one ceiling, and I would rather show you where it is than pretend it is not there. What is not done To keep the scope honest, as everywhere in this series: The window state is in process memory with no eviction. A crash loses every in-flight window and its emit-once flags. Worse, because offsets are committed and delivery is at-least-once, a window that already paged before a restart can re-page after one, since the guard that prevented it did not survive. The real fix is an external windowed store (Redis, keyed the same way, TTL equal to the window) so the count and the guard both outlive the process and the footprint stays bounded. The tumbling boundary under-counts straddling bursts. A burst of exactly the threshold split across two adjacent windows pages zero times, as measured above. A sliding window, or a short overlap between adjacent windows, would catch the case the hard boundary drops. It is single-node. One clustering consumer, one partition's worth of state per key. The design is share-nothing and partition-keyed by tenantId, so it is meant to scale by adding consumers to the group, but I have not run it multi-node and will not draw a scaling claim I cannot reproduce. All numbers are single-laptop (Docker Desktop, 512 MiB per service, single Redpanda node, ClickHouse 24.3, driven by k6). The 403ms detection, the 810 bytes per window, and the 434k-window cliff characterize this configuration and its bottleneck, not a production ceiling. Next: post 4, accept fast, never block. Everything in this post happens after the gateway has already said 202 Accepted. That ordering, acknowledge first and count later, is the single most important decision in the pipeline, and the reason a frozen clustering consumer never once slowed the front door. Here is how the gateway gets out of the way in single-digit milliseconds. Try log0 log0 is the platform this series is built on, an open, multi-tenant incident pipeline you can run yourself or use hosted. Platform: log0.in Docs: log0.in/docs Console: console.log0.in charfield, the ASCII animation registry behind the log0 front ends: charfield.log0.in Written by Ashmit JaiSarita Gupta. Find me on LinkedIn, GitHub, and X, and read the rest of the series on Hashnode.

The core job of an incident platform is deciding that a flood of log lines is one bug, not ten thousand. log0 does it with no machine learning, deterministically, in O(1) per event. The whole trick is what gets stripped out of a message before hashing. Here is the exact code, the one ordering bug that bites if it is wrong, and the failure mode I chose on purpose. This is post 3 in a series on building log0. Post 2 was about getting a log line to reach the system. This one is about what happens the instant it arrives: how log0 decides which incident, if any, this line belongs to. The number this has to earn In a load test, log0 ingested 115,489 log lines all carrying one error pattern and created exactly one incident. A deduplication ratio of 115,489 to 1. Push ten distinct patterns through the same run and ten incidents come out; a hundred patterns, a hundred incidents. Incidents out always equals distinct bugs in, regardless of how loud each one is. That is the entire value of the product in one chart. A bug that fires 115,489 times pages once. The ratio falls as the number of distinct bugs rises, exactly as it should: more real problems, more incidents. What never happens is one bug fragmenting into thousands of incidents, or two different bugs silently merging into one. The mechanism that guarantees both is the fingerprint. The naive version, and why it does not work The obvious first idea is to group by the log message string. Identical message, same incident. It falls apart on the first real log line, because production error messages are almost never identical: Connection timeout to payment-gateway after 30000ms Connection timeout to payment-gateway after 28514ms Connection timeout to payment-gateway after 31003ms Same bug, three times. Group by the raw string and that is three incidents, then thirty thousand, one per slightly different timeout value, request ID, IP address, or user ID. String equality treats the variable data as if it were the signal. It is the opposite: the variable data is exactly the noise you want to throw away. The next idea is fuzzy matching, edit distance or an embedding model, to call two messages "similar enough." That works until something has to explain why two incidents merged, until a similarity threshold has to be tuned per log format, or a model inference has to be paid for on every single log line at ingestion rates. It is non-deterministic, expensive, and unexplainable, three properties no one wants in the component that decides who gets paged. log0 takes the deterministic route. Strip the variable data out to get a stable structural template, then hash the template along with a few other stable fields. Same bug, same hash, every time, with a one-line explanation of why. The fingerprint, end to end A fingerprint is a SHA-256 hash of four stable fields joined by a pipe: fingerprint = SHA-256( service | messageTemplate | exceptionType | firstStackFrame ) Each field adds one dimension of "is this the same bug": service scopes it. A NullPointerException in payment-service is a different incident from the same exception in auth-service. Same symptom, different system, different owner. messageTemplate is the message with all dynamic values replaced by placeholders. This is where the de-noising happens, and it is the part with the subtle bug, below. exceptionType separates two exception classes thrown from the same line. A TimeoutException and a NullPointerException out of PaymentProcessor.charge are two bugs, not one. firstStackFrame pins it to the exact throw site, with the line number deliberately removed. The hash itself is unremarkable, standard SHA-256, 64 hex chars. Everything interesting happens before it, in deciding what string goes in. Building the template: order is the whole game Stripping dynamic values is three regex replacements. The catch is that the order is not interchangeable; get it wrong and the fingerprint silently corrupts. public String buildMessageTemplate(String message) { if (message == null || message.isEmpty()) { return ""; } String template = UUID_PATTERN.matcher(message).replaceAll("<uuid>"); template = IP_PATTERN.matcher(template).replaceAll("<ip>"); template = NUMBER_PATTERN.matcher(template).replaceAll("<number>"); return template.trim(); } UUID first, then IP, then number. The reason is that all three are made of digits, and the most general pattern will eat the more specific ones if it runs first. NUMBER_PATTERN is \d+, any run of digits. Run it first on 192.168.1.1 and you get <number>.<number>.<number>.<number>, and IP_PATTERN never gets to match because there are no digits left. Run it first on a UUID and you shred the hex groups the same way. So the rule is most-specific-first: UUID (the most structured), then IP, then the catch-all number. Walk one message through it: Input: "Timeout from 192.168.1.1 after 30000ms, id=a3f4b2c1-1234-5678-abcd-ef0123456789" UUID: "Timeout from 192.168.1.1 after 30000ms, id=<uuid>" IP: "Timeout from <ip> after 30000ms, id=<uuid>" Num: "Timeout from <ip> after <number>ms, id=<uuid>" Every log line from that bug, whatever its IP, timeout, or request ID, lands on that same final template. That is the collapse the dedup ratio is measuring. The stack frame: stable on purpose The first stack frame is the call site that threw. Only the first frame is used, because deeper frames vary with the request path that led there, while the throw site is stable for the same bug. And the line number is stripped: String firstLine = stackTrace.split("\n")[0].trim(); firstLine = firstLine.replaceFirst("^at\\s+", ""); // remove "at " prefix firstLine = firstLine.replaceAll("\\(.*?\\)", ""); // remove (FileName.java:142) Input: "at com.log0.PaymentProcessor.charge(PaymentProcessor.java:142)" Output: "com.log0.PaymentProcessor.charge" The line number is removed on purpose. A refactor that shifts PaymentProcessor.charge from line 142 to 145 does not change the bug, so it must not change the fingerprint. If line numbers were part of the hash, every cosmetic edit above the throw site would fork a brand-new incident for a bug you already had open. Stripping the line number is the difference between a fingerprint that tracks a bug and one that tracks a source-file revision. Two small details that prevent silent collisions Two things in the final assembly look like boilerplate and are not. The pipe delimiter is load-bearing. The four fields are joined with |, not concatenated: String input = safe(service) + "|" + safe(template) + "|" + safe(exceptionType) + "|" + safe(firstFrame); Without a delimiter, ("ab", "c") and ("a", "bc") both produce "abc" and therefore the same fingerprint, despite being different error patterns. The pipe keeps the field boundaries injective, so two different field splits can never collide into one hash. Null becomes empty, never the text "null". Every field goes through a safe() guard: private String safe(String s) { return s == null ? "" : s; } A null service must contribute the empty string, giving "|template|...", not the literal four characters "null" giving "null|template|...". Those are different hashes, and letting a null stringify would mean a missing field changes a bug's identity. The guard makes a missing field contribute nothing, which is what "missing" should mean. The tradeoff I chose on purpose No fingerprint scheme is perfect, and the honest question is not "can it be fooled" but "which way does it fail." This one fails toward noise, never toward silence. If a log format uses dynamic values the regexes do not recognize, say a hex token that is not a UUID, those values survive into the template, and two lines from the same bug get two templates and two fingerprints. The bug under-deduplicates: it produces more incidents than it should. That is annoying. It is also loud and visible, you see two incidents for one bug and you go widen a regex. The failure I refused to accept is the opposite: two genuinely different bugs collapsing into one fingerprint and one incident. That hides a real problem behind an already-open incident, and the miss surfaces when the second bug takes down something the first incident's owner was never looking at. Over-dedup is a silent miss; under-dedup is a visible nuisance. Given a forced choice, a monitoring system should be biased toward too many incidents, never too few. The design is biased that way deliberately. What is not done exceptionType is in the formula but currently passed as null. The slot exists in generate() and contributes to the hash, but the normalizer does not yet parse the exception class out of the trace, so today it is always the empty string. Effect: two different exception types thrown from the same line currently share a fingerprint. It is a documented TODO; the wiring is half-built, the formula is ready for it. The template regexes cover UUID, IPv4, and integers. Not IPv6, not hex IDs, not ISO timestamps embedded mid-message, not floats. Those pass through and can cause the benign under-dedup above. The right fix is a small, ordered, tested library of patterns, extended as real formats demand, not a guess at every format up front. The dedup ratios above are single-node, one-laptop measurements (Docker Desktop, 512 MB per service, single Redpanda node, k6 at 50 VUs). They characterize the mechanism's behavior, not a throughput ceiling. Next: post 4, from fingerprints to an incident. A fingerprint names a bug, but one occurrence is not an incident; ten of them inside five minutes is. The next post is the clustering service that counts each fingerprint inside a tumbling event-time window and pages exactly once when a window crosses the threshold, the emit-once guard that keeps a loud bug to a single incident, and the in-memory window store that has no eviction and what that quietly costs. Try log0 log0 is the platform this series is built on, an open, multi-tenant incident pipeline you can run yourself or use hosted. Platform: log0.in Docs: log0.in/docs Console: console.log0.in charfield, the ASCII animation registry behind the log0 front ends: charfield.log0.in Written by Ashmit JaiSarita Gupta. Find me on LinkedIn, GitHub, and X, and read the rest of the series on Hashnode.

The core job of an incident platform is deciding that a flood of log lines is one bug, not ten thousand. log0 does it with no machine learning, deterministically, in O(1) per event. The whole trick is what gets stripped out of a message before hashing. Here is the exact code, the one ordering bug that bites if it is wrong, and the failure mode I chose on purpose. This is post 3 in a series on building log0. Post 2 was about getting a log line to reach the system. This one is about what happens the instant it arrives: how log0 decides which incident, if any, this line belongs to. The number this has to earn In a load test, log0 ingested 115,489 log lines all carrying one error pattern and created exactly one incident. A deduplication ratio of 115,489 to 1. Push ten distinct patterns through the same run and ten incidents come out; a hundred patterns, a hundred incidents. Incidents out always equals distinct bugs in, regardless of how loud each one is. That is the entire value of the product in one chart. A bug that fires 115,489 times pages once. The ratio falls as the number of distinct bugs rises, exactly as it should: more real problems, more incidents. What never happens is one bug fragmenting into thousands of incidents, or two different bugs silently merging into one. The mechanism that guarantees both is the fingerprint. The naive version, and why it does not work The obvious first idea is to group by the log message string. Identical message, same incident. It falls apart on the first real log line, because production error messages are almost never identical: Connection timeout to payment-gateway after 30000ms Connection timeout to payment-gateway after 28514ms Connection timeout to payment-gateway after 31003ms Same bug, three times. Group by the raw string and that is three incidents, then thirty thousand, one per slightly different timeout value, request ID, IP address, or user ID. String equality treats the variable data as if it were the signal. It is the opposite: the variable data is exactly the noise you want to throw away. The next idea is fuzzy matching, edit distance or an embedding model, to call two messages "similar enough." That works until something has to explain why two incidents merged, until a similarity threshold has to be tuned per log format, or a model inference has to be paid for on every single log line at ingestion rates. It is non-deterministic, expensive, and unexplainable, three properties no one wants in the component that decides who gets paged. log0 takes the deterministic route. Strip the variable data out to get a stable structural template, then hash the template along with a few other stable fields. Same bug, same hash, every time, with a one-line explanation of why. The fingerprint, end to end A fingerprint is a SHA-256 hash of four stable fields joined by a pipe: fingerprint = SHA-256( service | messageTemplate | exceptionType | firstStackFrame ) Each field adds one dimension of "is this the same bug": service scopes it. A NullPointerException in payment-service is a different incident from the same exception in auth-service. Same symptom, different system, different owner. messageTemplate is the message with all dynamic values replaced by placeholders. This is where the de-noising happens, and it is the part with the subtle bug, below. exceptionType separates two exception classes thrown from the same line. A TimeoutException and a NullPointerException out of PaymentProcessor.charge are two bugs, not one. firstStackFrame pins it to the exact throw site, with the line number deliberately removed. The hash itself is unremarkable, standard SHA-256, 64 hex chars. Everything interesting happens before it, in deciding what string goes in. Building the template: order is the whole game Stripping dynamic values is three regex replacements. The catch is that the order is not interchangeable; get it wrong and the fingerprint silently corrupts. public String buildMessageTemplate(String message) { if (message == null || message.isEmpty()) { return ""; } String template = UUID_PATTERN.matcher(message).replaceAll("<uuid>"); template = IP_PATTERN.matcher(template).replaceAll("<ip>"); template = NUMBER_PATTERN.matcher(template).replaceAll("<number>"); return template.trim(); } UUID first, then IP, then number. The reason is that all three are made of digits, and the most general pattern will eat the more specific ones if it runs first. NUMBER_PATTERN is \d+, any run of digits. Run it first on 192.168.1.1 and you get <number>.<number>.<number>.<number>, and IP_PATTERN never gets to match because there are no digits left. Run it first on a UUID and you shred the hex groups the same way. So the rule is most-specific-first: UUID (the most structured), then IP, then the catch-all number. Walk one message through it: Input: "Timeout from 192.168.1.1 after 30000ms, id=a3f4b2c1-1234-5678-abcd-ef0123456789" UUID: "Timeout from 192.168.1.1 after 30000ms, id=<uuid>" IP: "Timeout from <ip> after 30000ms, id=<uuid>" Num: "Timeout from <ip> after <number>ms, id=<uuid>" Every log line from that bug, whatever its IP, timeout, or request ID, lands on that same final template. That is the collapse the dedup ratio is measuring. The stack frame: stable on purpose The first stack frame is the call site that threw. Only the first frame is used, because deeper frames vary with the request path that led there, while the throw site is stable for the same bug. And the line number is stripped: String firstLine = stackTrace.split("\n")[0].trim(); firstLine = firstLine.replaceFirst("^at\\s+", ""); // remove "at " prefix firstLine = firstLine.replaceAll("\\(.*?\\)", ""); // remove (FileName.java:142) Input: "at com.log0.PaymentProcessor.charge(PaymentProcessor.java:142)" Output: "com.log0.PaymentProcessor.charge" The line number is removed on purpose. A refactor that shifts PaymentProcessor.charge from line 142 to 145 does not change the bug, so it must not change the fingerprint. If line numbers were part of the hash, every cosmetic edit above the throw site would fork a brand-new incident for a bug you already had open. Stripping the line number is the difference between a fingerprint that tracks a bug and one that tracks a source-file revision. Two small details that prevent silent collisions Two things in the final assembly look like boilerplate and are not. The pipe delimiter is load-bearing. The four fields are joined with |, not concatenated: String input = safe(service) + "|" + safe(template) + "|" + safe(exceptionType) + "|" + safe(firstFrame); Without a delimiter, ("ab", "c") and ("a", "bc") both produce "abc" and therefore the same fingerprint, despite being different error patterns. The pipe keeps the field boundaries injective, so two different field splits can never collide into one hash. Null becomes empty, never the text "null". Every field goes through a safe() guard: private String safe(String s) { return s == null ? "" : s; } A null service must contribute the empty string, giving "|template|...", not the literal four characters "null" giving "null|template|...". Those are different hashes, and letting a null stringify would mean a missing field changes a bug's identity. The guard makes a missing field contribute nothing, which is what "missing" should mean. The tradeoff I chose on purpose No fingerprint scheme is perfect, and the honest question is not "can it be fooled" but "which way does it fail." This one fails toward noise, never toward silence. If a log format uses dynamic values the regexes do not recognize, say a hex token that is not a UUID, those values survive into the template, and two lines from the same bug get two templates and two fingerprints. The bug under-deduplicates: it produces more incidents than it should. That is annoying. It is also loud and visible, you see two incidents for one bug and you go widen a regex. The failure I refused to accept is the opposite: two genuinely different bugs collapsing into one fingerprint and one incident. That hides a real problem behind an already-open incident, and the miss surfaces when the second bug takes down something the first incident's owner was never looking at. Over-dedup is a silent miss; under-dedup is a visible nuisance. Given a forced choice, a monitoring system should be biased toward too many incidents, never too few. The design is biased that way deliberately. What is not done exceptionType is in the formula but currently passed as null. The slot exists in generate() and contributes to the hash, but the normalizer does not yet parse the exception class out of the trace, so today it is always the empty string. Effect: two different exception types thrown from the same line currently share a fingerprint. It is a documented TODO; the wiring is half-built, the formula is ready for it. The template regexes cover UUID, IPv4, and integers. Not IPv6, not hex IDs, not ISO timestamps embedded mid-message, not floats. Those pass through and can cause the benign under-dedup above. The right fix is a small, ordered, tested library of patterns, extended as real formats demand, not a guess at every format up front. The dedup ratios above are single-node, one-laptop measurements (Docker Desktop, 512 MB per service, single Redpanda node, k6 at 50 VUs). They characterize the mechanism's behavior, not a throughput ceiling. Next: post 4, from fingerprints to an incident. A fingerprint names a bug, but one occurrence is not an incident; ten of them inside five minutes is. The next post is the clustering service that counts each fingerprint inside a tumbling event-time window and pages exactly once when a window crosses the threshold, the emit-once guard that keeps a loud bug to a single incident, and the in-memory window store that has no eviction and what that quietly costs. Try log0 log0 is the platform this series is built on, an open, multi-tenant incident pipeline you can run yourself or use hosted. Platform: log0.in Docs: log0.in/docs Console: console.log0.in charfield, the ASCII animation registry behind the log0 front ends: charfield.log0.in Written by Ashmit JaiSarita Gupta. Find me on LinkedIn, GitHub, and X, and read the rest of the series on Hashnode.

Running Qiskit circuits on a near-Clifford simulator: building the clifft-qiskit provider Most quantum simulators are exponential in the qubit count. clifft, a simulator from the Unitary Foundation, is not: it is a fast, exact simulator for near-Clifford circuits, and its cost scales with the non-Clifford content of a circuit rather than its width. A circuit that is mostly Clifford with a handful of T gates and rotations simulates in time that grows as 2^k, where k is the small "active" dimension the non-Clifford operations open up, not 2^n in the total qubit count. On low-magic workloads that is the difference between a circuit that is tractable and one that is not. There was one friction point. clifft's native input is Stim circuit text. If your circuit lives in Qiskit, as a large amount of quantum software does, you had to hand-write or hand-generate Stim before clifft could run it. During unitaryHACK 2026 I closed that gap by building clifft-qiskit, a Qiskit BackendV2 provider that lets you run a QuantumCircuit on clifft directly: from qiskit import QuantumCircuit from clifft_qiskit import ClifftProvider qc = QuantumCircuit(2, 2) qc.h(0); qc.cx(0, 1); qc.measure([0, 1], [0, 1]) backend = ClifftProvider().get_backend("clifft") counts = backend.run(qc, shots=1000).result().get_counts() print(counts) # {'11': ~500, '00': ~500} The work started in the main repo (issue #39, PR #120), then moved to a standalone package (clifft-qiskit) with its own release automation (issue #2, PR #3), now published on PyPI. This post is a walk through that: the shape of the adapter, the one part that is easy to get wrong, how I validated it, and what it took to turn a subpackage into a released library. Note: I am a contributor, not a maintainer, of clifft. This is my reading of the code and my own work on the provider; corrections from the clifft team are welcome. Why bridge to clifft at all Before the adapter, the case for it. clifft keeps the exact state in a factored form: a compile-time Clifford frame (resolved once, ahead of time, using Stim's tableau simulator), a lightweight runtime Pauli frame updated per shot, and only a small active subset of the state carried as a dense complex array of size 2^k. Clifford gates are absorbed into the frame at zero per-shot cost; each T gate or rotation is what grows k. The published benchmarks (arXiv 2604.27058) put it at roughly 10x slower than Stim on pure-Clifford QEC sampling, but far ahead of other near-Clifford tools on magic-heavy workloads (their numbers include ~370x over Tsim on distance-3 cultivation and ~13x over SOFT on distance-5), and competitive with full statevector simulators like Aer and qsim on dense circuits. The point for a Qiskit user is simple: for the large class of circuits that are mostly Clifford, clifft can be much cheaper than a general statevector simulator, and it is exact, not approximate. The only thing standing between a Qiskit QuantumCircuit and that speed was a format boundary. A provider removes it. The design, decided up front The issue that scoped this (#39) fixed the shape before any code: a single synchronous BackendV2, all-to-all connectivity, terminal measurement returning counts, a documented basis, clear errors on anything unsupported, and one hard constraint worth calling out - importing clifft must not require Qiskit. The simulator core stays dependency-light; the Qiskit dependency lives only in the provider. That single rule is what later justified pulling the provider into its own package. The basis was chosen as Clifford+T plus single-qubit rotations. That choice is what keeps the provider exact rather than approximate, for reasons covered in the section on rotations below. The provider is a translation layer over two clifft calls Here is the pipeline in one picture. A QuantumCircuit goes in, a counts dict comes out, and every quantum operation happens inside two clifft calls in the middle. The entire pipeline lives in ClifftBackend.run(). Stripped to its spine: def run(self, run_input, **options) -> ClifftJob: shots = options.get("shots", self.options.shots) seed = options.get("seed", self.options.seed) for circ in circuits: decomposed = transpile(circ, basis_gates=CLIFFT_BASIS, optimization_level=1) stim_text, measured_clbits = qiskit_to_stim(decomposed) if not measured_clbits: raise QiskitError("clifft backend requires at least one measurement.") program = clifft.compile(stim_text) sample = clifft.sample(program, shots=shots) # + seed, if given counts = counts_from_measurements(sample.measurements, measured_clbits, decomposed.num_clbits) ... Read that as three responsibilities: Qiskit lowers the circuit. transpile(..., basis_gates=CLIFFT_BASIS) rewrites arbitrary gates into clifft's basis. I do not write a decomposition pass; Qiskit's transpiler already knows how, and pointing it at the right basis is enough. The provider translates. qiskit_to_stim turns the lowered circuit into Stim text and returns a map from sample columns back to classical bits. This is the only code the provider truly owns, and it is deliberately small. clifft does the physics. clifft.compile(stim_text) returns a Program; clifft.sample(program, shots=...) returns a SampleResult whose .measurements is a (shots, num_measurements) array of bits. The exactness of Clifford+T and of the rotations is a property of the simulator, not of my adapter. The Qiskit surface around this is thin. ClifftProvider().get_backend("clifft") hands back a ClifftBackend(BackendV2); run() computes eagerly and wraps the finished Result in a ClifftJob(JobV1) whose submit() is a no-op and whose status() is already DONE. The backend's Target advertises the basis with num_qubits=None, which leaves the width unbounded so transpiling against the backend never caps the circuit. There is no async machinery because there is nothing to wait for: sampling has already happened by the time you hold the job. The whole translation, on one page The provider owns exactly one table. Eleven Clifford+T gates map to Stim by name, three rotations map and convert their angle, measurement records a classical bit, and anything else is refused. The non-parameterized gates are a plain lookup: _GATE_MAP = { "h": "H", "x": "X", "y": "Y", "z": "Z", "s": "S", "sdg": "S_DAG", "t": "T", "tdg": "T_DAG", "cx": "CX", "cy": "CY", "cz": "CZ", } The rotations are the interesting three, and they carry a unit conversion that is easy to miss and matters: _PARAM_GATE_MAP = {"rx": "R_X", "ry": "R_Y", "rz": "R_Z"} # inside qiskit_to_stim, for a parameterized gate: angle = float(instruction.operation.params[0]) / math.pi # radians -> half-turns lines.append(f"{_PARAM_GATE_MAP[name]}({angle}) {qubits[0]}") Qiskit measures rotation angles in radians. clifft's R_X/R_Y/R_Z extensions measure them in half-turns (multiples of pi). Divide by pi on the way across and the two agree; miss it, and every rotation is wrong by a factor of pi with no error raised. That is the kind of mistake a shape test misses and a distribution test catches, which is what the validation section is for. Everything that is not in one of those two maps, and is not a measurement or a trivial no-op like barrier, is rejected: raise QiskitError( f"clifft backend does not support operation '{name}'. " f"Supported basis: {supported}, measure." ) That is how reset and mid-circuit control end up rejected: the transpiler cannot lower a non-unitary into a unitary basis, so the unknown op reaches this branch and the error names both the offending operation and the full supported set. Refusing clearly is a feature; silently dropping an operation would be a correctness bug that surfaces as a wrong distribution three steps later. Why rotations belong in the basis Putting rx, ry, rz in the basis is not a convenience, it is what keeps the provider exact. With those three in the target, Qiskit's transpiler can lower any single-qubit unitary (u, p, and friends) and any controlled rotation into the basis by exact algebraic identities. There is no Solovay-Kitaev approximation, no synthesizing a rotation out of a long H/T sequence to some tolerance. The comment in the code states it plainly: # With rx/ry/rz in the basis, the transpiler lowers any single-qubit unitary # (u, p, ...) and controlled rotations exactly into this set -- no approximate # (Solovay-Kitaev) synthesis is needed. This only works because clifft treats rotations as first-class, native operations (they compile to exact complex-amplitude updates on the active array), rather than something to be approximated in a discrete gate set. The provider inherits that exactness directly. The general point is that a thin adapter can only expose what the engine underneath it supports, and the right choice is to surface that capability rather than hide it behind a weaker basis. Getting the counts right The gate mapping is mechanical. The part that took the most care is the boundary where clifft's sample array becomes a Qiskit counts dict, because two different ordering conventions meet there. clifft returns measurements as a (shots, num_measurements) array where column j is the j-th M instruction in program order. Qiskit reports counts as bitstrings indexed by classical bit, little-endian, and a circuit is free to measure qubits into classical bits in any order it likes. Those are not the same indexing, so qiskit_to_stim returns a measured_clbits list alongside the Stim text: measured_clbits[j] is the Qiskit clbit that the j-th measurement writes to. Packing then respects that map: for row in measurements: # one row per shot value = 0 for col, clbit in enumerate(measured_clbits): if int(row[col]): value |= 1 << clbit # clbit i contributes bit 1<<i key = hex(value) counts[key] = counts.get(key, 0) + 1 Two things make this correct. First, the bit for measurement col is placed at 1 << clbit, not 1 << col, so a circuit that measures qubit 0 into clbit 2 lands in the right place. Second, the counts are keyed as hex and the experiment result carries a memory_slots = num_clbits header, which is what lets Qiskit's own get_counts() render those hex keys into correctly-widthed, correctly-ordered binary strings, identical to what AerSimulator would produce. Decoupling measurement order from clbit order is the reason a permuted-measurement circuit comes out matching a reference simulator rather than reordered, and there is a dedicated test for that case. Validating against a full statevector simulator A simulator adapter is only useful if it is correct, so the tests do not check shapes, they check distributions, against qiskit-aer as the reference. The suite runs each circuit on both backends and asserts that every outcome's probability agrees to within a tolerance (SHOTS = 4000, TOL = 0.05), across thirteen cases: Bell and GHZ-3 entanglement, a non-Clifford H-T-H, a Toffoli that must decompose into Clifford+T, native rx/ry/rz, a general u gate and a controlled rotation lowered exactly, list execution, the permuted-clbit case, and the error paths. To sanity-check it fresh for this post, I ran an entangled rx/ry/rz circuit on both backends at 20,000 shots: The largest per-outcome gap between clifft and Aer was 0.6 percentage points, which is sampling noise at this shot count, not a modelling difference, and it is an order of magnitude inside the suite's own 5-point tolerance. On the circuits both tools can represent, the distributions match, because clifft is computing the same exact result by a cheaper route. The unit conversion, the bit packing, and the transpilation basis are all checked at once by a test that would fail if any of them were wrong. From a subpackage to a package The provider first landed inside the clifft repo as an optional clifft[qiskit] subpackage (PR #120). During review, the decision was to pull it out into a standalone repository and package, clifft-qiskit, rather than keep it in-tree. The reasoning is the same "keep clifft dependency-light" rule from the design, seen from the other side: clifft is a compiled C++ core with Python bindings, released on its own cadence. Qiskit's API moves faster and is pure-Python. Binding the two release cycles together helps neither. A separate package lets the provider declare qiskit>=2.0 and track it independently, while clifft keeps a minimal dependency surface and never imports Qiskit unless a user asks for the provider. This is the established pattern in the ecosystem: a lean simulator core with satellite integration packages (qiskit-aer, the pytket extensions) that each track one external SDK. So the code moved, the package directory was renamed from clifft/qiskit to a flat clifft_qiskit, and the dependency became explicit rather than an extra: [project] name = "clifft-qiskit" requires-python = ">=3.12" dependencies = ["clifft>=0.4", "qiskit>=2.0", "numpy>=1.26"] dynamic = ["version"] [tool.hatch.version] source = "vcs" # version comes from the git tag, not a hardcoded string Nothing about the physics changed. What changed is that the provider became a thing you can pip install and version on its own. One tag ships the release The last piece (PR #3) was release automation, so that cutting a version is a single action rather than a manual checklist. The pipeline is driven by a git tag. Push a vX.Y.Z tag and the workflow: Builds with uv build, runs twine check, and does a wheel smoke test - it installs the freshly built wheel into a clean environment and runs the test suite against it, so a broken wheel fails before it can be published, not after. Publishes to TestPyPI on every run, using OIDC trusted publishing, so there are no PyPI API tokens stored in the repo at all. Publishes to PyPI, but only when the trigger is a tag push. A manual workflow_dispatch run deliberately stops at TestPyPI, which makes "rehearse the release" a safe, first-class operation. Cuts a GitHub release, extracting the matching section from CHANGELOG.md as the notes. Two supporting pieces make that sustainable. Versioning is derived from the tag by hatch-vcs instead of a string someone has to remember to bump (an early review caught a hardcoded "0.1.0" in the backend; the fix was to read the version from package metadata so it can never drift from the tag). And the changelog is generated by git-cliff from conventional-commit messages, with a CI check that enforces conventional PR titles, so the release notes assemble themselves from the commit history rather than being written by hand each time. The CI matrix runs Python 3.12 and 3.13 against both a pinned and an upgraded dependency set, so a breaking change in a future Qiskit shows up as a scheduled-run failure rather than a surprise in production. The result of all of it is clifft-qiskit 0.1.0 on PyPI: pip install clifft-qiskit, and the four lines from the top of this post run a Qiskit circuit on a near-Clifford simulator. What I took away A good provider is a thin adapter, and thinness is the goal. The only code this package really owns is a gate-name table and a bit-packing loop. Everything hard - decomposition, the physics, the exactness - is delegated to Qiskit's transpiler and clifft's core. Recognizing what not to build was most of the work. Exactness lives in the engine, so expose it, do not approximate it. Putting rx/ry/rz in the basis, because clifft supports them natively, is what lets the transpiler avoid Solovay-Kitaev entirely. A weaker basis would have made every rotation an approximation for no reason. The boundary between two ordering conventions is where the bugs are. The gate map is trivial; the measurement-column-to-clbit mapping and the little-endian packing are where correctness is won or lost, and they are worth a dedicated test each. Validate distributions, not shapes. Checking every outcome's probability against Aer catches the radians-versus-half-turns class of bug that a "did it return counts" test never would. Extract a package when the cadences diverge. A compiled core and a fast-moving SDK wrapper want different release cycles; splitting them keeps the core's dependency surface honest. Make releasing routine. A tag-triggered build, a wheel smoke test, trusted publishing with no stored secrets, TestPyPI before PyPI, and an auto-generated changelog reduce "cut a release" to pushing one tag, which is about as much attention as the step deserves. If you want to read the code, start at clifft_qiskit/backend.py (the BackendV2 and the run pipeline), then clifft_qiskit/_translate.py (the gate table, the angle conversion, and the counts packing). The package is on PyPI, the source is on GitHub, and clifft itself is documented at unitaryfoundation.github.io/clifft. Corrections and improvements welcome, especially from the clifft maintainers, who know the core far better than I do. Built during unitaryHACK 2026.

A liveness probe and a real request take different paths through a service, and the gap between them hides a whole class of bug. The first real POST to log0's gateway hung with no error and no log line, while every health check stayed green and /actuator/health returned 200 OK with a {"status":"UP"} body. The cause was a Kafka producer doing exactly what it is designed to do against an address that pointed nowhere. This post is about that mechanism, and about the difference between "the process is up" and "the system works." This is post 2 in a series on building log0, a multi-tenant incident-management platform. It covers the first real bug, the producer behavior behind it, and why a green dashboard can confirm a service is running while it cannot do its job. A green dashboard and a request that never returned The stack was built. Seven services, a Redpanda broker, ClickHouse, PostgreSQL, all wired into one Docker Compose file. I brought it up, watched the health checks go green one by one, and it looked done. docker compose ps showed every service up. Each /actuator/health endpoint returned 200. Spring Boot Actuator was happy. The logs showed clean startup, no exceptions, every consumer subscribed to its topic. Then I sent the first real log line. A single POST /api/v1/logs to the ingestion-gateway, the front door of the whole system. It hung. No 202, no error, no timeout for a long time. The connection sat open. The gateway had accepted the TCP connection, read the request, and then stopped, holding the line, returning nothing. Nothing in the logs explained it. The gateway was not crashing and not throwing. CPU was near zero. By every signal I had been watching, the service was healthy. It was also doing nothing, and had been from the start. What the health check was checking Here is the thing I had not internalized until this moment: a liveness probe and a real request take two completely different paths through the service. The top half of that diagram is what ran in CI and what I had been watching: GET /actuator/health hits the gateway, the gateway answers 200 OK (UP), done. That probe never touches Kafka. It does not open a producer, it does not fetch metadata, it does not send anything. It confirms exactly one thing: the JVM is running and the HTTP server is accepting connections. That is all "healthy" ever meant. The bottom half is the first real request, the one that had never run before that moment. POST /api/v1/logs reaches the gateway, the gateway builds a KafkaProducer and calls send(), and the producer tries to connect to a broker and fetch cluster metadata before it can produce. That is where it died, and it died on a connection that the health check had no reason to ever attempt. The two paths share almost no code. The probe was green because the probe path worked. The probe path working told me nothing at all about whether the producer path worked, and I had quietly assumed it did. What triggered it: one hardcoded address The producer was configured like this, in the gateway's KafkaProducerConfig: @Bean public ProducerFactory<String, RawLogEvent> rawLogProducerFactory() { Map<String, Object> props = new HashMap<>(); props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"); // ... return new DefaultKafkaProducerFactory<>(props); } // the raw-logs-dlq factory carried the same hardcoded literal localhost:9092. Hardcoded. On my host machine, during early single-process testing, localhost:9092 is exactly where the broker was, so this had worked once, on the host, outside Docker. Inside the container, it is a lie. Each container has its own network namespace. From inside the ingestion-gateway container, localhost is the gateway container itself, where nothing is listening on 9092. The real broker is a different container, reachable on the Compose network as redpanda:9092. So when the producer called send(), it tried to open a connection to localhost:9092 inside its own container, found nothing listening, and began doing what a Kafka producer is designed to do when it cannot reach a broker: it blocked, waiting for cluster metadata, up to max.block.ms. Why it blocked instead of failing This is the part worth slowing down on, because the failure mode is counterintuitive. A Kafka producer does not fail fast when the broker is unreachable. By design, send() first needs cluster metadata: which partitions exist, which broker leads each one. Until it has that metadata, it cannot decide where the record goes, so it waits. The wait is bounded by max.block.ms, which defaults to 60 seconds. So the producer was not broken in the sense of throwing. It was patiently doing its job: trying to reach a broker, retrying the connection, waiting for metadata that was never going to arrive because nothing was listening at that address. The calling thread, the one handling my HTTP request, was parked inside send() for the full block duration. From the client's side, the request hung. There was no exception to find in the logs because, for those 60 seconds, nothing exceptional had happened yet. The producer was inside its normal "broker not available, keep trying" loop. The redpanda broker, the real one, sat idle the whole time, never contacted, because no code in the running system knew its address. The entire ingestion path, the front door of the product, was non-functional in Docker, and had only ever been health-checked, never exercised end to end. The green dashboard was accurate and misleading at the same time: accurate about the process, silent about the job. The fix The fix is small, which is the uncomfortable part. The address should never have been a literal. It should come from configuration, so it can differ between the host and the container without a code change: @Configuration public class KafkaProducerConfig { @Value("${spring.kafka.bootstrap-servers:localhost:9092}") private String bootstrapServers; @Bean public ProducerFactory<String, RawLogEvent> rawLogProducerFactory() { Map<String, Object> props = new HashMap<>(); props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers); // ... return new DefaultKafkaProducerFactory<>(props); } } And in the Compose environment for the gateway: environment: SPRING_APPLICATION_JSON: '{"spring.kafka.bootstrap-servers":"redpanda:9092"}' One value, read from the environment, resolving to redpanda:9092 inside the network. On the host, the same property defaults to localhost:9092. Same code, correct address in both places. After the fix, the first POST /api/v1/logs returned 202 Accepted right away. The producer connected to redpanda:9092, fetched metadata in milliseconds, wrote the event to raw-logs, and acknowledged. The path that had never worked now worked, and from there it kept working: with the address right, the gateway sustained 3,032 requests per second in a 60-second load test, 181,995 requests total, at p99 156 ms and a p50 near 26 ms. The endpoint was never slow. It had been blocked on an address that resolved to nowhere. The lesson The trigger was mundane: a hardcoded host that resolved correctly outside Docker and pointed nowhere inside it. The lesson under it generalizes well past this one bug, and it is the reason this is post 1. A health check that does not exercise the real path is a check that the process started, nothing more. My liveness probe answered "is the JVM up and serving HTTP," and I had been reading it as "does ingestion work." Those are different questions with different answers, and the gap between them is exactly where this bug lived for as long as it did. The probe was not wrong. It was answering the question it was designed to answer. I was the one asking it to mean more than it did. The honest version of "healthy" for an ingestion service would touch the thing that can break: a readiness probe that confirms the producer can reach a broker, or, more bluntly, a smoke test that sends one real event end to end and checks an incident comes out the far side. The cheapest possible version of that, a single real POST in CI, would have caught this in seconds. I had not written it, because everything was green and green felt like enough. This is also a quiet argument for load testing earlier than feels necessary. I did not find this bug by reasoning about my config. I found it the instant I sent real traffic. The load test was not measuring throughput yet; it surfaced a correctness bug first, because it was the first thing to ever exercise the path for real. Synthetic load is not only about performance numbers. Often the first thing it tells you is whether the system does anything at all. What is not done To keep the scope honest, in the spirit of the rest of this series: The gateway has a basic liveness probe and now reads its broker address from config, but it still does not ship a readiness probe that verifies broker reachability. It should. That is the real fix for the class of bug, not only this instance. There is no automated end-to-end smoke test wired into the build yet. The manual "send one event, watch an incident appear" check is documented, not automated. These numbers are from a single-node setup on one laptop, the same scope disclaimer as everywhere in this series: Docker Desktop, 512 MB per service, a single Redpanda node, driven by k6. They characterize the path's behavior, not production capacity. Next: post 3, the fingerprint. Once a log line reaches the system, how does log0 decide that 115,489 lines are the same bug and one line is a different one, deterministically, with no machine learning, in O(1) per event? It comes down to what you strip out before you hash. Try log0 log0 is the platform this series is built on, an open, multi-tenant incident pipeline you can run yourself or use hosted. Platform: log0.in Docs: log0.in/docs Console: console.log0.in charfield, the ASCII animation registry behind the log0 front ends: charfield.log0.in Written by Ashmit JaiSarita Gupta. Find me on LinkedIn, GitHub, and X, and read the rest of the series on Hashnode.

A bad deploy at 3 AM A deploy ships at 02:50. One null check is missing on a hot path. By 03:00, the payment service has thrown the same NullPointerException tens of thousands of times. If your alerting forwards each line as its own page, the on-call engineer's phone has now buzzed tens of thousands of times for a single bug. The bug is one thing. The error is one thing. The log stream does not know that, so it emits one line per occurrence, and a naive alerting layer turns each line into a page. The signal, one bug, drowns in the volume, ten thousand lines. In a load test that reproduced exactly this shape, log0 ingested 115,489 log lines carrying one error pattern and created exactly one incident. A deduplication ratio of 115,489 to 1. Same run, ten distinct error patterns going in: ten incidents out. One page per real problem, regardless of volume. That collapse, from a flood of lines to a single owned incident, is the entire product. This post is the map of how it happens: the services, the data flow, the handful of decisions that shaped it, and the real numbers from load-testing the whole thing on one laptop. Every later post in this series drills into one box on the diagram below. What log0 is log0 is a multi-tenant incident-management platform for backend teams. Your services POST their logs to it over HTTP. From there it: deduplicates errors by a structural fingerprint, so 10,000 occurrences of one bug are one thing; clusters those fingerprints in time windows and opens an incident when a pattern crosses an occurrence threshold; drives each incident through a lifecycle (new, assigned, acknowledged, resolved); notifies the on-call Slack channel, with an AI-written root-cause summary attached; keeps every tenant's data structurally isolated the whole way through. The functional requirements are the obvious ones: ingest logs, group similar errors, detect incidents at a threshold, manage their lifecycle, notify a channel, and summarise. The non-functional ones are where the design lives: accept a log in single-digit milliseconds, never let ingestion go down because something downstream is slow, tolerate eventual consistency on detection, and isolate tenants by construction rather than by a WHERE clause that is easy to forget. It is seven Java / Spring Boot services talking over a Kafka API, with ClickHouse for logs and PostgreSQL for incident state. It runs as one Docker stack. Nothing here is at the Meta scale, and I will be specific later about exactly what the numbers are and are not. The shape: one stream, one direction log0 is not a web of services calling each other. It is a single linear stream. A log event enters at one end and moves in one direction, and each hop between services is a Kafka topic, not a synchronous call. Read top to bottom, that is the entire data path: ingestion-gateway takes the HTTP POST, does the minimum validation, writes the event to the raw-logs topic, and returns 202 Accepted. It never waits for processing. That single decision is the subject of post 4 in this series, Accept fast, never block. normalization-service consumes raw-logs, parses each event into a stable shape, strips the dynamic values out of the message to build a template, computes the fingerprint, writes the full event to ClickHouse, and emits to normalized-logs. The fingerprint is discussed in post 2 in this series. clustering-service consumes normalized-logs and counts occurrences per fingerprint inside a 5-minute tumbling window. When a window crosses 10 occurrences, it emits one incident-events message. Streaming aggregation is discussed in post 3 in this series. incident-service consumes incident-events, runs the incident through its state machine, and upserts it into PostgreSQL. Because the same event can arrive more than once, every write is idempotent. That is covered in post 10 in this series. notification-service consumes notification-events and posts a Slack Block Kit alert to the owning team. ai-service attaches a root-cause summary out of band, via an async callback, so a slow model never blocks the alert. Two things never flow backwards, and that is the point. Nothing downstream can apply backpressure to the gateway by making it wait, and a single poison message cannot freeze the pipeline, because normalization routes anything it cannot parse to a dead-letter topic, raw-logs-dlq, instead of dying on it. auth-service sits outside the data path: it issues JSON Web Tokens and every other service validates them locally, with no network call back to auth on the hot path. That tradeoff is discussed in post 14 in this series. Five decisions that shaped everything log0 came down to a small number of decisions, each with a cost. This series is organized around them. Here they are, stated plainly, each with the tradeoff it carries. 1. Accept fast, never block. The ingestion endpoint returns 202 Accepted the moment the event is durably on the raw-logs topic. It does no clustering, no database write, and no enrichment inline. The cost: the caller gets an acknowledgement, not a result. Detection is eventually consistent, seconds behind. For a logging endpoint that must never be the thing that is down, that is the correct trade. 2. A log line is an event with an identity. Deduplication is not string matching. log0 strips the dynamic values out of a message (numbers, IP addresses, UUIDs become placeholders), joins four stable fields with a pipe and hashes them: fingerprint = SHA-256( service | messageTemplate | exceptionType | firstStackFrame ) Same bug, different timeout values or request ID: same fingerprint, same incident. The cost is that a log format the templating regexes do not recognize will under-duplicate. The failure mode is benign, it makes too many incidents (noisy but visible), never too few (a hidden, merged bug). We have discussed this in post 2 of this series. 3. Multi-tenancy is a partition key, not a column. Events are keyed by tenantId on the Kafka topics, which buys per-tenant ordering and noisy-neighbour isolation for free, and every stored row carries tenant_id. Isolation is structural. The cost is partition skew when one tenant is far louder than the rest. As discussed in post 7 of this series, I measured the skew rather than asserting it away. 4. At-least-once delivery, plus idempotency. Kafka redelivers on a consumer rebalance or a retry, so the same incident-events message can arrive twice. Rather than fight that, every incident write is an idempotent upsert keyed on (tenant, fingerprint): a duplicate increments the occurrence count, it never forks a second incident. At-least-once plus idempotent equals effectively-once. We have discussed this in post 10 of this series. 5. Two databases, on purpose. Logs go to ClickHouse, a columnar store built for GROUP BY fingerprint over millions of rows. Incident state goes to PostgreSQL, which gives transactional updates and a clean state machine. One database cannot be great at both 10k-insert-per-second analytics and a transactional lifecycle. The cost is two systems to operate. We have discussed this in post 10 of this series. The numbers, and exactly what they are The reason to build the thing and then load-test it, rather than describe it, is that the interesting parts of a design only show up under load. Here are the headline results the series is built on. Each gets its own post with the full chart. Finding Result Deduplication ratio 1 template in: 115,489 : 1. 10 templates: 8,760 : 1. 100 templates: 967 : 1. Incidents out always equalled templates in. Single-row vs batched inserts ClickHouse insert path went from 40 rows/sec at 171% CPU to 4,185 rows/sec at 15% CPU, one change. Backlog drain Single-row inserts built a 67,071-message backlog draining at ~68/sec; batched, the backlog peaked at ~283 and never sustained, clearing within ~15 seconds. End-to-end detection First error to incident row: p50 403 ms, p99 477 ms (n=60, same clock). Concurrency sweep 50 to 800 virtual users: throughput held in a ~1,600 to 2,200 req/sec band, p99 36 ms to 709 ms, under 0.4% errors, no collapse. Tenant skew A hot tenant at 9x the traffic of a quiet one: p99 80 ms vs 79 ms. The quiet tenant's tail did not move. Dead-letter routing 300 poison messages in: 300 to the DLQ, 0 dropped, clean path unaffected. Broker under stress Redpanda hit a real exit-137 OOM at 800 VUs while /actuator/health stayed 200. Topics survived; producers auto-reconnected. Scope, once and clearly. Every number above was measured on a single laptop: Docker Desktop, 512 MB per service, a single-node Redpanda broker, ClickHouse 24.3, driven by k6. These characterize the system's behavior and its bottlenecks, not its production capacity. A single-node broker that OOMs at 800 virtual users is not a scaling claim; it is a finding about where this configuration breaks, which is exactly the kind of thing the series is about. The most useful results are the unflattering ones. The broker OOM while the health check stayed green is a small, honest "health checks lie" story. The single-row insert wall is a textbook producer/consumer mismatch you can watch build and then drain. Those are the posts I would read first. What is not done Stating the limits is part of the design, not an apology for it. The clustering window lives in process memory. A crash loses the in-flight windows. Production wants an external store (Redis) so the count survives a restart. It is designed, not yet built. exceptionType is wired into the fingerprint formula but currently passed as null, a documented TODO. Until the normalizer parses the exception class out of the trace, two different exceptions thrown from the same line share a fingerprint. The slot exists; the wiring is half done. Everything runs single-node. The share-nothing, partition-keyed design is meant to scale horizontally by adding consumers to a group, but I have not run it multi-node, so I will not draw you a scaling graph I cannot reproduce. How to read this series Each post takes one box on that diagram, or one of the five decisions, and goes deep: the problem, the naive version and why it broke (usually with a number), the fix, the real code, and the tradeoff. They stand alone, but they chain. If you came for the algorithms, start with post 2, the fingerprint. If you came for the systems engineering, start with post 6, the day a one-line change took the insert path from 40 to 4,185 rows per second. Next: post 2, the bug that load testing found and health checks never could. The whole stack reported healthy, while the ingestion path had never once been exercised end-to-end. Here is how a green dashboard lied, and what it cost. Try log0 log0 is the platform this series is built on, an open, multi-tenant incident pipeline you can run yourself or use hosted. Platform: log0.in Docs: log0.in/docs Console: console.log0.in charfield, the ASCII animation registry behind the log0 front ends: charfield.log0.in Written by Ashmit JaiSarita Gupta. Find me on LinkedIn, GitHub, and X, and read the rest of the series on Hashnode.

Quantum software has a Babel problem. A circuit can live as a Qiskit QuantumCircuit, a Cirq Circuit, a pyQuil Program, a PyTKET Circuit, an OpenQASM 2 or 3 string, a Braket circuit, a PyQIR module, and so on. Tools, simulators, and hardware each speak a subset of these. If you want any format to reach any other, the naive answer is a converter for every ordered pair - and that count grows as N×(N−1). It does not scale, and most of those converters would duplicate each other's work. qBraid's transpiler takes a different approach: it turns the problem into a weighted shortest-path search over a directed graph - the same shape as network routing, GPS directions, or dependency resolution. Program types are nodes, converters are weighted edges, and "convert A to B" becomes "find the cheapest path from A to B and apply each edge along it." You add one edge; the graph gives you many new routes for free. I recently added a few of those edges during unitaryHACK 2026 - an openqasm3 → pyquil conversion (#1179) and PyTKET interop plus a pyquil → qasm3 conversion (#1200). This post is a walk-through of the subsystem that those edges plug into. All file references are under qbraid/transpiler/. Note: I'm a contributor, not a maintainer. This is my reading of the code as of writing; corrections are welcome. The mental model: nodes and edges Nodes are program types - "qiskit", "cirq", "pyquil", "pytket", "qasm2", "qasm3", "braket", "pyqir", "qrisp", … (the same aliases qBraid uses in its program registry, QPROGRAM_REGISTRY). The set keeps growing - Qrisp was added in #1211. Edges are conversion functions, each pointing one way: source → target. Transpiling from A to B becomes "find a path from node A to node B and apply each edge's function in order." If there is no direct A→B edge but there is A→qasm3→B, the graph finds and composes that route. A simplified slice (my unitaryHACK edges highlighted): How an edge is declared: naming is registration There is no central list you edit to register a converter. A conversion function is discovered by its name and location. You write a function named {source}_to_{target} in qbraid/transpiler/conversions/{source}/, and the package in conversions/__init__.py collects it into a module-level conversion_functions list by inspecting the submodules. The graph then turns those names back into edges. From graph.py: conversion_functions: list[str] = getattr(transpiler, "conversion_functions", []) edges = [conversion.split("_to_") for conversion in conversion_functions] So pyquil_to_qasm3 becomes the edge ("pyquil", "qasm3"). The naming convention is the API. My pyquil_to_qasm3 lives in conversions/pyquil/pyquil_to_qasm3.py and looks like: from qbraid.transpiler.annotations import weight @weight(1) def pyquil_to_qasm3(program: pyquil.quil.Program) -> Qasm3StringType: ... return "\n".join(lines) + "\n" That's the whole contract: take the source object, return the target object, name it correctly, drop it in the right folder. Two decorators that tune the graph conversions/annotations.py defines two small but important decorators. @weight(value) attaches a number in [0, 1] to the edge. Crucially, this value is not a hand-tuned "cost" - it's the gate-set coverage of the conversion: the fraction of a reference gate set that the converter handles correctly. A higher weight means a more complete conversion, and 1.0 means full coverage. (How that fraction is measured, and how it becomes a routing cost, is in the pathfinding section below.) def weight(value: float) -> Callable[[F], F]: if not 0 <= value <= 1: raise ValueError("Weight value must be between 0 and 1.") ... setattr(wrapper, "weight", value) @requires_extras(*packages) marks edges that need optional dependencies beyond the function's own package. For example, PyTKET's cross-framework converters rely on the pytket.extensions.* packages, so in conversions/pytket/pytket_extras.py: @requires_extras("pytket.extensions.qiskit") def pytket_to_qiskit(circuit: pytket.circuit.Circuit) -> qiskit.QuantumCircuit: ... The decorator records the requirement on the function (requires_extras attribute). The graph reads it when deciding whether an edge is actually usable in the current environment - which keeps optional, heavy SDKs from being hard requirements. From functions to a graph: the Conversion and ConversionGraph objects Each discovered function is wrapped in a Conversion (edge.py). The constructor records more than just the endpoints: class Conversion: def __init__(self, source, target, conversion_func, weight=None, bias=None): self._source = source self._target = target self._conversion_func = conversion_func self._bias = bias if bias is not None else 0 self._weight = self._get_adjusted_weight(weight) self._extras = getattr(conversion_func, "requires_extras", []) self._native = self._is_module_native(conversion_func) self._supported = self._is_conversion_supported() A few things worth calling out: weight is taken from the @weight attribute (the coverage fraction), then transformed into a routing cost (next section). bias fine-tunes the depth-vs-coverage tradeoff and is added once per hop. From the docstring: a bias of 0.25 slightly favours a single conversion at weight 0.8 over two conversions at weight 1.0; a smaller bias requires the single hop to have even higher coverage before it wins. This is how the router is steered between "fewest hops" and "highest-coverage hops." _native flags whether the converter is internal to qBraid. _supported checks (via _extras) whether the optional deps are importable, so unusable edges can be excluded. ConversionGraph (graph.py) builds the actual graph. load_default_conversions() constructs a Conversion per discovered function, and create_conversion_graph() adds them as edges, stashing the callable and weight on each edge's payload: self.add_edge( ..., {"native": edge.native, "func": edge.convert, "weight": edge.weight}, ) Pathfinding: Dijkstra over rustworkx The graph is backed by rustworkx (imported as rx), a Rust-accelerated graph library - chosen over the pure-Python networkx because the core operation is graph traversal, where a compiled implementation pays off. Routing is plain Dijkstra (O(E log V) with a binary heap) with the edge cost as the weight function: import rustworkx as rx path = rx.dijkstra_shortest_paths( self._graph, ..., weight_fn=lambda edge: edge["weight"], ) Coverage → cost: how the @weight value is used A higher-coverage edge should be cheaper to traverse, and Dijkstra minimises total cost, so the coverage fraction is inverted and log-transformed before it goes on the edge. Conversion._get_adjusted_weight (edge.py) does exactly that: # effective_weight is the @weight coverage fraction, in [0, 1] rx_adjusted_weight = float("inf") if effective_weight == 0 else np.log(1 / effective_weight) adjusted_weight = rx_adjusted_weight + self._bias So each edge's cost is log(1 / coverage) + bias: Full coverage (1.0) → log(1) = 0, cost is just the bias. Lower coverage → larger log(1/coverage) → more expensive. bias is paid once per hop, so a single direct edge beats a multi-hop chain unless the chain's edges are substantially higher-coverage. Why the log? What you actually want is the path that maximizes the product of the per-edge coverages (the chance the whole chain round-trips faithfully). Dijkstra minimizes a sum, not a product. Taking -log converts one into the other: max ∏ coverageᵢ becomes min Σ log(1/coverageᵢ), because log is monotonic and turns products into sums. It's the same trick used for maximum-probability paths (e.g. Viterbi). The bias is then a small per-hop penalty layered on top to break ties toward shorter paths. A worked example (default bias = 0.25): one direct edge at coverage 0.8 costs log(1/0.8) + 0.25 ≈ 0.47; two chained edges at coverage 1.0 cost (0 + 0.25) × 2 = 0.50. The direct edge wins - barely - which is the intended "prefer one good hop over two perfect ones" behaviour. This is also why adding pyquil → qasm3 matters: replacing the old pyquil → cirq → qasm2 → qasm3 chain (three biases, three lossy hops) with one edge is a big cost drop, and the router switches to it automatically. Where the coverage number actually comes from The part that surprised me: weights aren't guessed - they're measured. Each source type has a coverage benchmark in tests/transpiler/<source>/test_coverage_from_<source>.py. The test builds a circuit for every gate in a reference gate set, transpiles it to each target, and checks equivalence; the fraction that round-trips correctly is the weight. The Cirq benchmark even encodes the per-target baselines directly: ALL_TARGETS = [("braket", 0.85), ("pyquil", 0.74), ("pytket", 0.87), ("qiskit", 0.87)] When I added the openqasm3 source edges, there was no test_coverage_from_openqasm3.py yet, so the workflow was: write the benchmark, run it, and set @weight from the measured coverage - with the maintainer's guidance to "aim as close to 1.0 as we can." That's a nice property: the routing policy is grounded in tested behaviour, not vibes, and a regression in coverage shows up as a failing benchmark rather than a silently worse route. find_shortest_conversion_path returns the ordered conversion functions to apply; find_top_shortest_conversion_paths(..., top_n=...) returns several candidates (handy as fallbacks when a conversion fails at runtime). On top sit has_path, shortest_path / all_paths (human-readable routes like "pyquil -> qasm3 -> qiskit"), and closest_target (pick the cheapest reachable target from a set - useful when a backend accepts several input formats). That's the payoff: I only wrote pyquil → qasm3, and because qasm3 already reaches qiskit, cirq, braket, and more, pyQuil now converts to all of them - no pyquil → qiskit written by hand. Where my edges fit qBraid can render the live graph itself - ConversionGraph().plot(...) (matplotlib). Here is the current graph, with the nodes my unitaryHACK edges touch outlined: from qbraid.transpiler import ConversionGraph ConversionGraph().plot( legend=True, target_nodes=["pyquil", "qasm3", "pytket", "pyqir"], save_path="qbraid_conversion_graph.png", ) Blue nodes are qBraid-native program types, grey are external; solid edges are native conversions, lighter edges are requires_extras ones. Notice how qasm3 sits at the centre as the interchange hub - that's why adding a single edge to or from it unlocks so many routes. openqasm3 → pyquil (conversions/openqasm3/openqasm3_to_pyquil.py, #1179): rather than parse OpenQASM by hand, it reuses qBraid's pyqasm engine - loads → validate → unroll the program into a flat, basis-gate AST, then walks unrolled_ast.statements and maps each node onto pyQuil (QubitDeclaration → flat qubit indices, ClassicalDeclaration → declare("ro", "BIT", n), QuantumGate → pyquil.gates.*, measurement → MEASURE). It also converts duration literals to seconds (_duration_seconds over _TIME_UNIT_SECONDS) so timing-bearing programs survive, and raises ProgramConversionError on anything unsupported. Leaning on pyqasm for the heavy lifting is the same pattern as openqasm3_to_cudaq. pyquil → qasm3 (conversions/pyquil/pyquil_to_qasm3.py, #1200/#1208): the reverse direction, written from scratch over pyQuil's instruction model. Together with openqasm3 → pyquil it completes a PyQuil ↔ OpenQASM 3 round trip and replaces the old lossy pyquil → cirq → qasm2 → qasm3 chain with a single edge. PyTKET interop via extras (#1208): cirq_to_pytket and pytket_to_cirq (via pytket.extensions.cirq) and pytket_to_pyqir (via pytket.qir, wrapped in pyqir.Module.from_ir), each @requires_extras-gated and auto-discovered. These replace lossy multi-hop detours (e.g. cirq → qasm2 → pytket) with direct 1-hop edges and give PyTKET a direct outbound edge to PyQIR. A practical lesson from this PR: I originally proposed pyquil ↔ pytket via pytket-pyquil, but that extra pins pyquil<5 and conflicts with qBraid's pyquil>=5 in a single CI env, so it would have silently downgraded pyquil and broken existing conversions. Checking the dependency graph before committing to an edge saved a broken PR - the conflict-free set (cirq↔pytket, pytket→pyqir, plus the from-scratch pyquil→qasm3) closed the issue instead. Each is just one node-to-node edge, but each widens what the rest of the graph can reach. Testing conversions Conversions are easy to get subtly wrong, so the tests are deliberately strict. The rule I internalised from review: an A_to_B test should compare in the source and target types themselves and not lean on a third framework to "check the answer." If your pyquil → qasm3 test secretly parses the qasm3 with Qiskit and compares to a Qiskit-built circuit, you're now testing three libraries' agreement, not your one conversion - and a failure no longer tells you where the bug is. Keeping the test isolated to the source and target keeps it honest and the signal sharp. A second subtlety is the global phase. Adding a direct pytket -> cirq edge made it the default 1-hop route, replacing the old pytket -> qasm2 -> cirq path. The direct converter represents ZZPhase(t) as cirq.ZZ ** t, which is unitary-equivalent up to an unobservable global phase. So the default-route assertion uses circuits_allclose(..., strict_gphase=False), while the native pytket -> qasm2 -> cirq route is still checked exactly (including global phase) via a separate require_native=True graph in the same test. The lesson: when you change which route is the default, check what equivalence the new route actually guarantees, and assert at that level - not a stricter one it can't meet, nor a looser one that hides bugs. What I took away The naming convention is the plugin system. {source}_to_{target} plus folder placement is all the registration there is - a clean, discoverable contract. A graph turns N² into N. Adding one well-placed edge (especially to a hub like qasm3) multiplies reachable pairs without writing direct converters. It's a textbook algorithm under a domain skin. Strip away the quantum vocabulary and this is weighted shortest-path: model the domain as a graph, pick a cost that makes the metric you care about (coverage) additive via -log, run Dijkstra. The hard part isn't the algorithm - it's recognising the problem is that algorithm. Routing is a measured policy, not a hardcode. Edge cost is log(1/coverage) + bias, and the coverage number comes from a per-source benchmark test - so the graph's choices are grounded in tested behaviour, and a coverage regression fails a test instead of silently picking a worse route. requires_extras keeps the dependency surface honest - optional SDKs stay optional, and the graph simply omits edges it can't run. If you want to read the code, start at qbraid/transpiler/graph.py (the graph and pathfinding), then edge.py (the Conversion wrapper) and annotations.py (the decorators), and finally browse conversions/<source>/ for concrete edges. Corrections and improvements welcome - especially from the qBraid maintainers, who know this code far better than I do.

What actually happens in the 30 milliseconds between you hitting Enter and your browser knowing where to send a packet - and why "DNS propagation" exists.

The picture Many introductory diagrams show Open any "how the internet works" article. You'll see this picture: "Your browser asks a DNS server for the IP address. The DNS server responds. The browser connects." It's a useful starting point. The parts it leaves out are the parts that explain CDNs, "DNS propagation delays", and why your laptop's network settings have a "DNS server" field at all. In practice, resolution involves your browser's memory, your operating system's cache, your router, your ISP's recursive resolver, one of 13 root servers, a TLD server, and finally an authoritative server - typically four to six different machines, often on three different continents, all involved in resolving one question: "what's the IP for google.com?" Often, the full lookup never happens because the answer is already cached locally or at the resolver. This post walks through the entire chain - what each player does, why the design looks weird, and what tools like dig reveal about the conversation. 1. Why DNS Exists at All Computers route packets using IP addresses - 32-bit numbers like 142.250.183.78. Humans remember names like google.com. DNS is the translation layer between the two. That much is obvious. The interesting part is why the translation needs four servers instead of one. The answer is scale. There are around 350 million registered domain names. No single machine could answer queries for all of them, store all the records, and absorb the global query load. So DNS is structured as a distributed hierarchical database, where different organisations own different slices of the namespace and are responsible only for their own. Every DNS lookup is a tour through that hierarchy. 2. The Hierarchy Domain names read right to left in increasing specificity: The trailing dot (the root) is implicit in browsers but real in DNS. Every fully qualified domain name actually ends in .. Each segment is owned by someone: Root (.) - managed by ICANN, served by 13 root server clusters (named A through M). They know nothing except who runs each TLD. TLD (.com, .dev, .in) - managed by registries like Verisign (.com) or Public Interest Registry (.org). They know who owns each second-level domain in their TLD. Authoritative server for google.com - run by Google. Knows every record under google.com. The hierarchy is what allows DNS to scale globally without centralising the entire namespace. Nobody has to know everything, but everybody knows who to ask next. 3. Before Anything Hits the Network: Caches When you type google.com in your browser, the very first thing that happens is three cache lookups, in order, all on your own machine: Browser cache. Chrome, Firefox, and Safari each maintain their own. Try chrome://net-internals/#dns to see Chrome's. OS resolver cache. Linux uses systemd-resolved or nscd; macOS uses mDNSResponder; Windows uses the DNS Client service. /etc/hosts (or C:\Windows\System32\drivers\etc\hosts). Static overrides. This is how localhost resolves to 127.0.0.1 without any network call. If any of these have the answer and the TTL hasn't expired, the lookup is done in microseconds. No packets sent. This avoids repeated network lookups for frequently visited domains. Only when all three miss does the query leave your machine. 4. The Recursive Resolver Your machine doesn't talk to root servers directly. It sends one query to a recursive resolver - typically your ISP's, your router's, or a public one like: Resolver Operator 8.8.8.8, 8.8.4.4 Google Public DNS 1.1.1.1, 1.0.0.1 Cloudflare 9.9.9.9 Quad9 (security-focused) 208.67.222.222 OpenDNS The recursive resolver does all the heavy lifting on your behalf. You ask it once: "What's the IP for google.com?" and it goes off, talks to multiple servers, and returns a single answer. From the client’s perspective, the lookup appears as a single request-response cycle even though multiple servers may be involved internally. 5. The Full Chain - Resolver's Side Let's say the resolver itself has nothing cached. Here's what happens. Three referrals, one final answer, then back to you. The total round trip is typically 20-50 ms because each segment caches results from previous queries. Note: the root doesn't return Google's IP. It only knows where the .com servers live. The .com servers don't return the IP either - they only know where Google's authoritative servers live. Only the authoritative server has the actual record. 6. Record Types (More Than Just A) A DNS query specifies a type. Different types return different things: Type Returns Used for A IPv4 address the basic case AAAA IPv6 address IPv6 CNAME Another domain name aliases (see CDN section) MX Mail server hostname + priority email routing NS Authoritative name servers delegation TXT Arbitrary text SPF, DKIM, domain verification SOA Zone metadata replication, TTLs PTR Hostname for an IP reverse DNS Try yourself: dig A google.com dig MX gmail.com dig TXT google.com dig NS google.com Each query type is independent. Asking for an A record doesn't tell you about MX records. 7. Why CDNs Live and Die by CNAMEs A CNAME record says "this name is an alias for that name." When the resolver gets a CNAME response, it restarts the lookup using the new name. This mechanism is what makes CDNs and managed DNS routing possible. www.yoursite.com. IN CNAME yoursite.cdn.cloudflare.net. yoursite.cdn.cloudflare.net. IN A 104.21.42.7 yoursite.cdn.cloudflare.net. IN A 172.67.180.34 You set www.yoursite.com to a CNAME pointing at Cloudflare. Cloudflare resolves that to whichever edge node is closest to the requesting user. You never have to update yoursite.com's DNS again, even if Cloudflare adds 100 new edge locations next month. Without CNAMEs, every CDN would need direct write access to every customer's DNS. CNAMEs make CDNs possible. Catch: CNAMEs cannot coexist with other records on the same name, and they cannot live at the apex (yoursite.com without www.). This is why people use ALIAS or ANAME records (provider-specific extensions) at the apex. 8. TTL - The "Why DNS Changes Take 24 Hours" Mystery Every DNS record has a TTL (time to live), measured in seconds. It tells caches how long they're allowed to keep the answer before re-asking. $ dig A google.com +noall +answer google.com. 300 IN A 142.250.183.78 ^^^ TTL in seconds = 5 minutes When you change a DNS record, existing caches don't know. They keep serving the old value until their copy expires. With a 24-hour TTL, some users will see the old IP for up to 24 hours after the change. This is what people mean by "DNS propagation." DNS doesn't actually propagate - caches expire and reload. Tradeoff: Low TTL (60s) - fast changes, more queries, more load on authoritative servers, slightly slower page loads (more cache misses). High TTL (24 h) - fewer queries, cheaper, but painful when you need to migrate. Best practice: drop TTL hours before a planned change, then raise it back after. 9. UDP, TCP, and the 512-Byte Rule DNS queries fit in a single packet 99% of the time, so DNS uses UDP on port 53 by default. No handshake, no connection setup, just one packet out and one back. Original DNS spec capped responses at 512 bytes to fit in one UDP packet. Two things bust this limit: Large responses - many records, big TXT for SPF, etc. DNSSEC - cryptographic signatures bloat responses 5-10x. When the response would exceed 512 bytes, the server sets the TC (truncated) flag in its UDP reply. The resolver then re-sends the same query over TCP on port 53. Modern DNS uses EDNS(0) to negotiate larger UDP responses (up to 4096 bytes), but TCP is still the fallback. This is also why DNS over HTTPS (DoH) and DNS over TLS (DoT) are recent innovations - encrypted DNS requires TCP-style transport. 10. Watching It Live with dig dig is the Unix tool that shows the entire conversation. Two flags reveal the most: dig +trace google.com This makes dig start at the root and walk down the hierarchy itself, printing each step: ;; QUESTION SECTION: ;google.com. IN A ;; AUTHORITY SECTION (from root): . 518400 IN NS a.root-servers.net. . 518400 IN NS b.root-servers.net. ... 13 root servers ... ;; Received 1097 bytes from 198.41.0.4#53(a.root-servers.net) in 11 ms ;; AUTHORITY SECTION (from .com TLD): google.com. 172800 IN NS ns1.google.com. google.com. 172800 IN NS ns2.google.com. ;; Received 836 bytes from 192.5.6.30#53(a.gtld-servers.net) in 24 ms ;; ANSWER SECTION (from authoritative): google.com. 300 IN A 142.250.183.78 ;; Received 55 bytes from 216.239.32.10#53(ns1.google.com) in 8 ms Three round trips, each to a different server. The TTLs (518400, 172800, 300) tell you how long each layer caches its answers. dig +short google.com # just the IP dig @1.1.1.1 google.com # query a specific resolver dig CNAME www.github.com # see the CNAME chain 11. Security Model and Weaknesses DNS was designed in 1983, before anyone worried about adversaries. It shows. DNS Spoofing / Cache Poisoning. A resolver waits for a UDP response. UDP has no handshake - anyone who can guess the query ID and reply faster than the real server can inject a fake answer. The resolver caches it. Now everyone using that resolver gets the attacker's IP for paypal.com. Mitigations: Source port randomisation - adds 16 bits of entropy. DNSSEC - cryptographically signs records. Resolvers verify the signature before trusting. Adoption is patchy because deployment is operationally painful. DNS over HTTPS (DoH) - sends queries over HTTPS instead of plain UDP. Encrypts the channel and hides DNS from network observers. Cloudflare's 1.1.1.1 and Google's 8.8.8.8 both support it. DNS over TLS (DoT) - same idea, but on a dedicated port (853) rather than HTTPS. DoH is controversial because it bypasses network-level DNS filtering used by enterprises and (sometimes) governments. That's a feature for users and a problem for network admins. 12. Putting It Together The full sequence when you type google.com and hit Enter: Every step is cached at the layer above it. The full walk from root to authoritative is the worst case, not the typical case. In practice, your resolver usually has TLD answers cached, sometimes the second-level too, and the whole thing finishes in a single round trip. But when something is genuinely uncached - first lookup of the day, fresh resolver boot, brand-new domain - every layer fires. And it still finishes in 30-50 ms because the hierarchy makes each individual query small and stateless. Closing Thought DNS is one of the few internet systems that's barely changed in 40 years. The hierarchy from 1983 still runs the modern internet. Most "improvements" since then have been bolted on without breaking the original design - exactly like TCP window scaling, exactly like HTTP/2's relationship to HTTP/1. The key idea is the delegation model: nobody has to know everything, but everybody knows who to ask next. That structure is why DNS scales to 350 million domains, why CDNs work, why you can switch hosting providers without anyone noticing, and why a tiny edit in your registrar's panel propagates across the world in hours. Next time you watch a webpage load slowly, run dig +trace. You’ll be able to see each delegation step directly. Found this useful? Next post in the series: the TCP three-way handshake - why it's three packets and not two, and how an attacker once brought down a quarter of the internet by exploiting that exact design choice.