All articles
Accept fast, never block: the one decision that keeps ingestion alive
A logging endpoint has one job it can never fail at: being available to accept logs. The fastest way to break that is to make the accept path wait on anything downstream. log0's ingestion gateway returns 202 Accepted the instant the event is handed to a Kafka producer future, before Kafka has even acknowledged it, and never waits for clustering, a database write, or a Slack call. Here is the code, why the response is sent exactly where it is, and what the latency looks like from 50 to 800 concurrent users. This is post 5 in a series on building log0. Post 3 covered how a log line becomes a fingerprint and post 4 how those fingerprints cluster into an incident downstream. This one is about the decision that happens upstream of all of that, at the front door, and why it is the most important one in the pipeline. The trap: doing the work before answering Imagine the straightforward implementation of an ingestion endpoint. A log arrives, and the handler does the obvious thing: validate it, normalize it, compute the fingerprint, run it through clustering, upsert the incident in PostgreSQL, fire the Slack notification, and then return 200 OK. The handler does the whole job and then reports success. The left lane is that design. It is correct, and it is a trap, for two reasons. The first is latency. The response time the caller sees is the sum of every hop. Validation plus a fingerprint plus a clustering window lookup plus a PostgreSQL write plus an HTTP call to Slack. The caller, some service trying to emit a log line, is now blocked on a chain it does not care about and should never see. The second is worse: coupling availability to the slowest dependency. If PostgreSQL is under load, every log POST slows down. If Slack's API is having a bad day, the ingestion endpoint is having a bad day. If the clustering consumer is down entirely, ingestion returns errors, for logs, which is precisely when something is going wrong and they are most needed. The endpoint whose entire purpose is to never be down has been wired so that anything downstream can take it down. That is backwards. The fix: answer the moment the event is durable enough The right lane is log0. The gateway does the minimum, validate the request, build the event, hand it to the Kafka producer, and returns 202 Accepted right there. Not 200 OK, because nothing has been processed yet; 202 means "accepted for processing," which is the honest status. Everything else, normalize, cluster, incident, notify, happens afterward on Kafka consumers, with each hop being a topic, none of it on the request thread. The controller is deliberately thin: @PostMapping public ResponseEntity<Void> ingestLog( HttpServletRequest request, @Valid @RequestBody LogIngestionRequest logRequest) { // tenant is derived from the validated API key by ApiKeyAuthFilter, not a client header String tenantId = (String) request.getAttribute(ApiKeyAuthFilter.TENANT_ATTRIBUTE); String serviceName = RequestHeaderExtractor.getRequiredHeader(request, HeaderConstants.SERVICE_NAME); String environment = RequestHeaderExtractor.getRequiredHeader(request, HeaderConstants.ENVIRONMENT); String apiKey = RequestHeaderExtractor.getRequiredHeader(request, HeaderConstants.API_KEY); RequestContext context = new RequestContext(tenantId, serviceName, environment, apiKey); logIngestionService.ingest(logRequest, context); return ResponseEntity.accepted().build(); // 202, immediately } It reads the tenant that the auth filter already resolved from the API key, reads the service and environment headers, validates the body via bean validation, calls ingest, and returns 202. There is no clustering here, no database, no enrichment. That is the entire point: the handler cannot be slow, because it does not contain anything slow. The detail that makes it non-blocking: the producer future The interesting line is one level down, in the producer. It is easy to think "writes to Kafka, so it returns fast," but there is a subtlety here, because there is a way to write this that quietly reintroduces the blocking that was removed. public void publish(RawLogEvent event) { kafkaTemplate.send(KafkaTopics.RAW_LOGS, event.getTenantId(), event) .whenComplete((result, ex) -> { if (ex != null) { log.error("Failed to publish raw log event: {}", ex.getMessage(), ex); DlqEvent dlqEvent = DlqEvent.builder() .originalEvent(event) .errorMessage(ex.getMessage()) .failedAt("ingestion-gateway") .failedAtTs(Instant.now()) .build(); dlqProducer.publish(event.getEventId(), dlqEvent); } }); } kafkaTemplate.send(...) returns a CompletableFuture. The code attaches a whenComplete callback to it and returns immediately. It never calls .get(), never blocks the request thread waiting for the broker to acknowledge the write. The acknowledgement, success or failure, arrives later, on a Kafka producer-network thread, and the callback handles it there. This is the difference between accept-fast and a slower, sneakier version of process-inline. If this code called .get() on the future, the request thread would block until Kafka confirmed the write, and now ingestion latency is coupled to broker latency and broker availability. Attaching a callback instead means the request thread is free the instant the event is queued in the producer's buffer. The 202 goes back to the caller; the broker round-trip happens out of band. The callback is not only fire-and-forget, though. If the send ultimately fails, the whenComplete handler wraps the event in a DlqEvent and routes it to raw-logs-dlq, asynchronously. So "never block" does not mean "drop on failure." A failed write is captured and quarantined for inspection, off the request path. The caller already got its 202, and the durability concern is handled where it belongs, in the background. One more thing the producer does for free: it keys the record by tenantId. That co-locates a tenant's events in the same partition, which buys per-tenant ordering and is the foundation of the tenant-isolation story in a later post. It costs nothing here; it is only the partition key on the send. What "fast" measures: 50 to 800 concurrent users Accept-fast is a nice theory. The question is whether it holds up under real concurrency, so I drove POST /api/v1/logs with k6 at a constant-VU sweep, 50 users up to 800, median of three runs each. Three things to read off it. p50 holds near the floor until the top of the sweep. The median request stays single-digit to low-double-digit milliseconds through about 300 VUs, then climbs as the laptop runs out of cores, roughly 90 ms at 500 VUs and 170 ms at 800. The typical caller's experience holds flat across the range a single laptop is comfortable with, because the handler has no slow work in it to contend on, and only stretches once scheduling pressure dominates at the top end. p99 climbs, and that is expected. The tail goes from about 36 ms at 50 VUs to 709 ms at 800 VUs. That is contention, more virtual users than cores, requests queuing for threads, the usual cost of pushing a single laptop past its comfortable concurrency. The honest read is not "the endpoint got slow" but "the tail stretched under load while the median held," which is exactly the shape accept-fast predicts: no inline dependency to fall over, only scheduling pressure. Throughput holds and almost nothing drops. Across the sweep, throughput stays in a band, roughly 1,600 to 2,200 req/s, and the error rate stays under half a percent, peaking at 0.39% at 800 VUs. There is no collapse, no cliff where the endpoint stops accepting. It bends, it does not break. That is the property that matters for a logging front door: degrade gracefully, never go to zero. For context, the same gateway in a sustained single-run benchmark accepted 181,995 requests over 60 seconds, 3,032 req/s, p99 156 ms. Per request the gateway does almost nothing, validate and hand off to a producer future, no parse, no database, no wait. But it is the front door, so it pays for that cheapness in volume, not in per-request cost: under load it was the busiest application container in the stack, around 226% CPU (about 2.3 cores) at a steady ~509 MB. Cheap work, done a lot of times, with nothing in the handler that blocks. The tradeoff, stated plainly Accept-fast is not free, and pretending otherwise would be dishonest. The cost is that 202 is a promise, not a receipt. When the caller gets 202 Accepted, the log has not been processed. It has not been fingerprinted, not clustered, not turned into an incident. All of that is eventually consistent, seconds behind, happening on consumers the caller never sees. A synchronous "your log created incident X" answer is something this architecture cannot give, by construction. For a logging pipeline, that is obviously the right trade. Nobody emitting a log line wants to block on incident detection; they want the line to be accepted and to get on with their work. But it is a real constraint, and it shows up downstream: detection latency is a separate measurement (end-to-end first-error-to-incident is its own post), and any UI on top of this has to be built for eventual consistency, not request-response. log0 trades a synchronous result for an endpoint that cannot be taken down by the things behind it. For this job, that is the right trade. What is not done There is no backpressure signal to the client. Under genuine overload the gateway will keep accepting and let latency rise rather than shedding load. A production version wants a 429 path when the producer buffer saturates. Today it bends gracefully but does not push back. The DLQ on send-failure is wired, but there is no automated re-drive. Failed events land in raw-logs-dlq and are preserved, but replaying them back into the pipeline after the cause is fixed is a manual step, not yet tooling. The numbers are single-node, one laptop (Docker Desktop, 512 MB per service, single Redpanda node, k6). The 800-VU tail and the throughput band characterize this configuration's behavior, not a capacity ceiling. A broker that the next post will show OOMs under stress is part of that same honest picture. Next: post 5, where it broke. Accept-fast moves the work downstream; this is what happened when one of those downstream consumers wrote to the database the naive way, one row at a time, and the producer-consumer mismatch you could watch build up and then drain. Try log0 log0 is the platform this series is built on, an open, multi-tenant incident pipeline you can run yourself or use hosted. Platform: log0.in Docs: log0.in/docs Console: console.log0.in charfield, the ASCII animation registry behind the log0 front ends: charfield.log0.in Written by Ashmit JaiSarita Gupta. Find me on LinkedIn, GitHub, and X, and read the rest of the series on Hashnode.
From fingerprints to an incident: counting in windows, paging once
A fingerprint names a bug. It does not decide that the bug is worth waking someone for. That decision is the clustering service: it counts each fingerprint inside a five-minute tumbling window keyed on event time, and the first time a window crosses ten occurrences it emits exactly one incident-events message, then never emits for that window again. A same-clock probe puts the whole path, first log to incident row, at p50 403ms. The mechanism is a ConcurrentHashMap and a synchronized boolean. This post is the counter, the emit-once guard, and the two honest failure modes that fall out of doing it in memory with no eviction and a hard window boundary. This is post 3 in a series on building log0. Post 2 covered how a log line collapses to a stable fingerprint. This one is about what happens to that fingerprint next: how a stream of identical errors becomes one counted, thresholded, deduplicated incident, and why the counting is the part that decides whether anyone gets paged. One occurrence is not an incident The fingerprint already solved deduplication of identity: ten thousand lines of the same bug share one hash. But identity is not severity. A bug that throws once at 3 AM and never again is noise. The same bug throwing ten times in a minute is an incident. Something has to count, and counting needs two boundaries: which occurrences belong together, and how many of them is enough. log0 answers both with three numbers, and they live in ClusteringConfig: private int occurrenceThreshold = 10; // page when a window reaches this private int windowDurationMinutes = 5; // tumbling window width private int maxTopMessages = 10; // sample messages carried on the incident Ten occurrences of one fingerprint inside one five-minute window emits an incident. Nine does not. The window is tumbling, not sliding: time is chopped into fixed five-minute blocks aligned to the clock, and every occurrence falls into exactly one block. That choice is what makes the counter O(1) per event and the failure modes legible, and it is also where the first sharp edge is. The key is the whole design Every decision about what counts together is compressed into one map key. ClusterKey.of builds it: key = tenantId : fingerprint : floor(eventTime / 5min) Three fields, and each one earns its place. tenantId keeps one customer's flood from ever counting toward another's incident; the count is per tenant by construction, not by a filter. fingerprint is the bug identity from post 2. And the third field is the window: the event's own timestamp, floor-aligned to a five-minute boundary. long bucketMinutes = timestamp.truncatedTo(ChronoUnit.MINUTES).getEpochSecond() / 60; long alignedMinutes = (bucketMinutes / windowDurationMinutes) * windowDurationMinutes; That floor-division is the tumbling window. 00:00 through 00:04:59 all map to bucket 00:00; 00:05:00 starts a fresh bucket with a fresh counter. Two occurrences of the same bug in the same tenant inside the same five minutes produce the same key and increment the same counter. Cross any one of the three boundaries, a different tenant, a different bug, or the next five-minute block, and it is a different counter that knows nothing about the first. The window is keyed on event time, the timestamp the log carries, not the wall-clock time the clustering service happened to read it. That is the right choice for correctness, because a backlog draining late should still bucket each event where it truly happened, not pile a delayed burst into whichever window the consumer caught up in. It is also the source of the boundary failure mode below. The counter and the emit-once guard The state is deliberately boring. InMemoryOccurrenceStore is a ConcurrentHashMap<ClusterKey, OccurrenceWindow>, and each OccurrenceWindow holds a count, a sample of recent messages, and a single boolean. FingerprintClusterer.cluster() is the whole hot path: OccurrenceWindow window = occurrenceStore.increment(key, event.getMessage()); if (window.getCount() >= config.getOccurrenceThreshold() && window.markIncidentEmitted()) { incidentEventProducer.publish(buildIncidentEvent(event, window)); } Two conditions, and the && order matters. The count check is the obvious one: has this window reached ten. The second is the one that makes the system usable. markIncidentEmitted() is synchronized and returns true exactly once: public synchronized boolean markIncidentEmitted() { if (incidentEmitted) { return false; } incidentEmitted = true; return true; } The tenth occurrence flips the flag and emits. The eleventh, the twelfth, the ten-thousandth all increment the count and re-check the threshold, but markIncidentEmitted() now returns false, so nothing is emitted. One window, one incident, regardless of how loud the bug gets after it trips. This is the same guarantee the fingerprint gives in the identity dimension, now in the time dimension: the fingerprint stops a bug fragmenting into thousands of incidents by message text, and the emit-once guard stops one window fragmenting into thousands of incidents by occurrence count. The count does not stop climbing after the emit, and that is intentional. Occurrences eleven and up keep accumulating, because the authoritative total for the incident is recomputed downstream from ClickHouse anyway; the in-window count exists only to make the page-or-not decision. The synchronized boolean is the cheapest possible thing that turns "this is a real incident" into an event exactly once. What it costs end to end: p50 403ms The point of all of this is to page fast when a real burst arrives. A same-clock probe measures exactly that: it fires ten identical-fingerprint logs at the gateway to trip one window, then polls Postgres until the incident row appears, repeated for n=60. Because the send and the poll share one clock, the number is honest end to end, ingest to incident. The distribution is tight: mean 405ms, p50 403ms, p99 477ms, almost everything between 380 and 440ms, one outlier near 477. The full histogram lives in post 10 in this series, where it belongs to incident creation, but the number is a clustering number first: most of that 403ms is the event crossing three Kafka topics and three services, and the clustering decision itself, the increment and the threshold check, is a hash-map lookup that costs nothing measurable next to the hops. Detection is eventually consistent and lands in well under half a second on this hardware. Failure mode one: a burst that straddles a boundary never pages A tumbling window has hard edges, and event time decides which side of an edge an occurrence lands on. That makes one specific burst invisible. I measured it directly: eight trials per split ratio, a distinct fingerprint each trial, exactly ten identical errors, split across two adjacent windows by their event timestamps, then poll Postgres for the incident. When all ten land in one window, 10:0, the window hits the threshold and pages, 8 of 8 trials. Every split, 9:1, 8:2, 7:3, 6:4, 5:5, pages 0 of 8. Nine occurrences in the first window and one in the next is nine and one, and neither bucket reaches ten, so the burst of ten genuinely identical errors creates zero incidents. The counters are correct; each holds the true count for its own window. The threshold is evaluated per window, and a burst unlucky enough to land on a boundary is under-counted into silence. This is the dishonest-by-omission failure, the opposite of the fingerprint's. The fingerprint fails toward noise: too many incidents, loud and visible. The window boundary fails toward silence: a real burst that should have paged does not, and nothing tells you. A sliding window or a small grace overlap would close most of this gap; the tumbling window was chosen for its O(1) keying and its legibility, and this is the bill for that choice. It is on the list at the end, named, not hidden. Failure mode two: the store has no eviction The second cost is memory, and it is the one that bites under sustained load. InMemoryOccurrenceStore never evicts. Its own javadoc says so: every distinct (tenant, fingerprint, window) key is a permanent entry in the map, and nothing ever removes a window after its five minutes pass. As long as new fingerprints keep arriving, the map keeps growing. I drove that on purpose. Pushing 710,439 unique fingerprints through the pipeline, one new fingerprint per request, each one a brand-new map entry, and sampled the clustering container's memory against the count of distinct windows it had consumed. The climb is dead linear, about 810 bytes per distinct window, exactly what an unbounded map predicts: 176 MiB at startup, 259 at 111k windows, 371 at 257k, 484 at 398k. Extrapolate the line and it meets the 512 MiB container cap at roughly 434,000 windows. What happens there is worse than a crash. The consumer stops advancing its offset while memory keeps climbing toward the cap, detection silently halts, and nothing in the map evicts itself. A process restart clears the state and can lose the emit-once guards for in-flight windows, but it does not fix the underlying problem: without an external store, the working set is unbounded. The gateway never noticed, it took all 710,439 requests at zero failures, roughly 3,382 a second, while the thing downstream that decides who gets paged quietly stopped deciding. That is the on-theme failure for this series: not a loud death but a green dashboard over a dead detector. The fix is designed and not built: the window state belongs in an external store, Redis with a TTL that matches the window width, so a window evicts itself five minutes after it closes and the working set stays bounded no matter how many distinct bugs pass through. In memory, the map is fast and simple and has exactly one ceiling, and I would rather show you where it is than pretend it is not there. What is not done To keep the scope honest, as everywhere in this series: The window state is in process memory with no eviction. A crash loses every in-flight window and its emit-once flags. Worse, because offsets are committed and delivery is at-least-once, a window that already paged before a restart can re-page after one, since the guard that prevented it did not survive. The real fix is an external windowed store (Redis, keyed the same way, TTL equal to the window) so the count and the guard both outlive the process and the footprint stays bounded. The tumbling boundary under-counts straddling bursts. A burst of exactly the threshold split across two adjacent windows pages zero times, as measured above. A sliding window, or a short overlap between adjacent windows, would catch the case the hard boundary drops. It is single-node. One clustering consumer, one partition's worth of state per key. The design is share-nothing and partition-keyed by tenantId, so it is meant to scale by adding consumers to the group, but I have not run it multi-node and will not draw a scaling claim I cannot reproduce. All numbers are single-laptop (Docker Desktop, 512 MiB per service, single Redpanda node, ClickHouse 24.3, driven by k6). The 403ms detection, the 810 bytes per window, and the 434k-window cliff characterize this configuration and its bottleneck, not a production ceiling. Next: post 4, accept fast, never block. Everything in this post happens after the gateway has already said 202 Accepted. That ordering, acknowledge first and count later, is the single most important decision in the pipeline, and the reason a frozen clustering consumer never once slowed the front door. Here is how the gateway gets out of the way in single-digit milliseconds. Try log0 log0 is the platform this series is built on, an open, multi-tenant incident pipeline you can run yourself or use hosted. Platform: log0.in Docs: log0.in/docs Console: console.log0.in charfield, the ASCII animation registry behind the log0 front ends: charfield.log0.in Written by Ashmit JaiSarita Gupta. Find me on LinkedIn, GitHub, and X, and read the rest of the series on Hashnode.
Anatomy of a fingerprint: how log0 turns 115,489 log lines into one incident
The core job of an incident platform is deciding that a flood of log lines is one bug, not ten thousand. log0 does it with no machine learning, deterministically, in O(1) per event. The whole trick is what gets stripped out of a message before hashing. Here is the exact code, the one ordering bug that bites if it is wrong, and the failure mode I chose on purpose. This is post 3 in a series on building log0. Post 2 was about getting a log line to reach the system. This one is about what happens the instant it arrives: how log0 decides which incident, if any, this line belongs to. The number this has to earn In a load test, log0 ingested 115,489 log lines all carrying one error pattern and created exactly one incident. A deduplication ratio of 115,489 to 1. Push ten distinct patterns through the same run and ten incidents come out; a hundred patterns, a hundred incidents. Incidents out always equals distinct bugs in, regardless of how loud each one is. That is the entire value of the product in one chart. A bug that fires 115,489 times pages once. The ratio falls as the number of distinct bugs rises, exactly as it should: more real problems, more incidents. What never happens is one bug fragmenting into thousands of incidents, or two different bugs silently merging into one. The mechanism that guarantees both is the fingerprint. The naive version, and why it does not work The obvious first idea is to group by the log message string. Identical message, same incident. It falls apart on the first real log line, because production error messages are almost never identical: Connection timeout to payment-gateway after 30000ms Connection timeout to payment-gateway after 28514ms Connection timeout to payment-gateway after 31003ms Same bug, three times. Group by the raw string and that is three incidents, then thirty thousand, one per slightly different timeout value, request ID, IP address, or user ID. String equality treats the variable data as if it were the signal. It is the opposite: the variable data is exactly the noise you want to throw away. The next idea is fuzzy matching, edit distance or an embedding model, to call two messages "similar enough." That works until something has to explain why two incidents merged, until a similarity threshold has to be tuned per log format, or a model inference has to be paid for on every single log line at ingestion rates. It is non-deterministic, expensive, and unexplainable, three properties no one wants in the component that decides who gets paged. log0 takes the deterministic route. Strip the variable data out to get a stable structural template, then hash the template along with a few other stable fields. Same bug, same hash, every time, with a one-line explanation of why. The fingerprint, end to end A fingerprint is a SHA-256 hash of four stable fields joined by a pipe: fingerprint = SHA-256( service | messageTemplate | exceptionType | firstStackFrame ) Each field adds one dimension of "is this the same bug": service scopes it. A NullPointerException in payment-service is a different incident from the same exception in auth-service. Same symptom, different system, different owner. messageTemplate is the message with all dynamic values replaced by placeholders. This is where the de-noising happens, and it is the part with the subtle bug, below. exceptionType separates two exception classes thrown from the same line. A TimeoutException and a NullPointerException out of PaymentProcessor.charge are two bugs, not one. firstStackFrame pins it to the exact throw site, with the line number deliberately removed. The hash itself is unremarkable, standard SHA-256, 64 hex chars. Everything interesting happens before it, in deciding what string goes in. Building the template: order is the whole game Stripping dynamic values is three regex replacements. The catch is that the order is not interchangeable; get it wrong and the fingerprint silently corrupts. public String buildMessageTemplate(String message) { if (message == null || message.isEmpty()) { return ""; } String template = UUID_PATTERN.matcher(message).replaceAll("<uuid>"); template = IP_PATTERN.matcher(template).replaceAll("<ip>"); template = NUMBER_PATTERN.matcher(template).replaceAll("<number>"); return template.trim(); } UUID first, then IP, then number. The reason is that all three are made of digits, and the most general pattern will eat the more specific ones if it runs first. NUMBER_PATTERN is \d+, any run of digits. Run it first on 192.168.1.1 and you get <number>.<number>.<number>.<number>, and IP_PATTERN never gets to match because there are no digits left. Run it first on a UUID and you shred the hex groups the same way. So the rule is most-specific-first: UUID (the most structured), then IP, then the catch-all number. Walk one message through it: Input: "Timeout from 192.168.1.1 after 30000ms, id=a3f4b2c1-1234-5678-abcd-ef0123456789" UUID: "Timeout from 192.168.1.1 after 30000ms, id=<uuid>" IP: "Timeout from <ip> after 30000ms, id=<uuid>" Num: "Timeout from <ip> after <number>ms, id=<uuid>" Every log line from that bug, whatever its IP, timeout, or request ID, lands on that same final template. That is the collapse the dedup ratio is measuring. The stack frame: stable on purpose The first stack frame is the call site that threw. Only the first frame is used, because deeper frames vary with the request path that led there, while the throw site is stable for the same bug. And the line number is stripped: String firstLine = stackTrace.split("\n")[0].trim(); firstLine = firstLine.replaceFirst("^at\\s+", ""); // remove "at " prefix firstLine = firstLine.replaceAll("\\(.*?\\)", ""); // remove (FileName.java:142) Input: "at com.log0.PaymentProcessor.charge(PaymentProcessor.java:142)" Output: "com.log0.PaymentProcessor.charge" The line number is removed on purpose. A refactor that shifts PaymentProcessor.charge from line 142 to 145 does not change the bug, so it must not change the fingerprint. If line numbers were part of the hash, every cosmetic edit above the throw site would fork a brand-new incident for a bug you already had open. Stripping the line number is the difference between a fingerprint that tracks a bug and one that tracks a source-file revision. Two small details that prevent silent collisions Two things in the final assembly look like boilerplate and are not. The pipe delimiter is load-bearing. The four fields are joined with |, not concatenated: String input = safe(service) + "|" + safe(template) + "|" + safe(exceptionType) + "|" + safe(firstFrame); Without a delimiter, ("ab", "c") and ("a", "bc") both produce "abc" and therefore the same fingerprint, despite being different error patterns. The pipe keeps the field boundaries injective, so two different field splits can never collide into one hash. Null becomes empty, never the text "null". Every field goes through a safe() guard: private String safe(String s) { return s == null ? "" : s; } A null service must contribute the empty string, giving "|template|...", not the literal four characters "null" giving "null|template|...". Those are different hashes, and letting a null stringify would mean a missing field changes a bug's identity. The guard makes a missing field contribute nothing, which is what "missing" should mean. The tradeoff I chose on purpose No fingerprint scheme is perfect, and the honest question is not "can it be fooled" but "which way does it fail." This one fails toward noise, never toward silence. If a log format uses dynamic values the regexes do not recognize, say a hex token that is not a UUID, those values survive into the template, and two lines from the same bug get two templates and two fingerprints. The bug under-deduplicates: it produces more incidents than it should. That is annoying. It is also loud and visible, you see two incidents for one bug and you go widen a regex. The failure I refused to accept is the opposite: two genuinely different bugs collapsing into one fingerprint and one incident. That hides a real problem behind an already-open incident, and the miss surfaces when the second bug takes down something the first incident's owner was never looking at. Over-dedup is a silent miss; under-dedup is a visible nuisance. Given a forced choice, a monitoring system should be biased toward too many incidents, never too few. The design is biased that way deliberately. What is not done exceptionType is in the formula but currently passed as null. The slot exists in generate() and contributes to the hash, but the normalizer does not yet parse the exception class out of the trace, so today it is always the empty string. Effect: two different exception types thrown from the same line currently share a fingerprint. It is a documented TODO; the wiring is half-built, the formula is ready for it. The template regexes cover UUID, IPv4, and integers. Not IPv6, not hex IDs, not ISO timestamps embedded mid-message, not floats. Those pass through and can cause the benign under-dedup above. The right fix is a small, ordered, tested library of patterns, extended as real formats demand, not a guess at every format up front. The dedup ratios above are single-node, one-laptop measurements (Docker Desktop, 512 MB per service, single Redpanda node, k6 at 50 VUs). They characterize the mechanism's behavior, not a throughput ceiling. Next: post 4, from fingerprints to an incident. A fingerprint names a bug, but one occurrence is not an incident; ten of them inside five minutes is. The next post is the clustering service that counts each fingerprint inside a tumbling event-time window and pages exactly once when a window crosses the threshold, the emit-once guard that keeps a loud bug to a single incident, and the in-memory window store that has no eviction and what that quietly costs. Try log0 log0 is the platform this series is built on, an open, multi-tenant incident pipeline you can run yourself or use hosted. Platform: log0.in Docs: log0.in/docs Console: console.log0.in charfield, the ASCII animation registry behind the log0 front ends: charfield.log0.in Written by Ashmit JaiSarita Gupta. Find me on LinkedIn, GitHub, and X, and read the rest of the series on Hashnode.
Anatomy of a fingerprint: how log0 turns 115,489 log lines into one incident
The core job of an incident platform is deciding that a flood of log lines is one bug, not ten thousand. log0 does it with no machine learning, deterministically, in O(1) per event. The whole trick is what gets stripped out of a message before hashing. Here is the exact code, the one ordering bug that bites if it is wrong, and the failure mode I chose on purpose. This is post 3 in a series on building log0. Post 2 was about getting a log line to reach the system. This one is about what happens the instant it arrives: how log0 decides which incident, if any, this line belongs to. The number this has to earn In a load test, log0 ingested 115,489 log lines all carrying one error pattern and created exactly one incident. A deduplication ratio of 115,489 to 1. Push ten distinct patterns through the same run and ten incidents come out; a hundred patterns, a hundred incidents. Incidents out always equals distinct bugs in, regardless of how loud each one is. That is the entire value of the product in one chart. A bug that fires 115,489 times pages once. The ratio falls as the number of distinct bugs rises, exactly as it should: more real problems, more incidents. What never happens is one bug fragmenting into thousands of incidents, or two different bugs silently merging into one. The mechanism that guarantees both is the fingerprint. The naive version, and why it does not work The obvious first idea is to group by the log message string. Identical message, same incident. It falls apart on the first real log line, because production error messages are almost never identical: Connection timeout to payment-gateway after 30000ms Connection timeout to payment-gateway after 28514ms Connection timeout to payment-gateway after 31003ms Same bug, three times. Group by the raw string and that is three incidents, then thirty thousand, one per slightly different timeout value, request ID, IP address, or user ID. String equality treats the variable data as if it were the signal. It is the opposite: the variable data is exactly the noise you want to throw away. The next idea is fuzzy matching, edit distance or an embedding model, to call two messages "similar enough." That works until something has to explain why two incidents merged, until a similarity threshold has to be tuned per log format, or a model inference has to be paid for on every single log line at ingestion rates. It is non-deterministic, expensive, and unexplainable, three properties no one wants in the component that decides who gets paged. log0 takes the deterministic route. Strip the variable data out to get a stable structural template, then hash the template along with a few other stable fields. Same bug, same hash, every time, with a one-line explanation of why. The fingerprint, end to end A fingerprint is a SHA-256 hash of four stable fields joined by a pipe: fingerprint = SHA-256( service | messageTemplate | exceptionType | firstStackFrame ) Each field adds one dimension of "is this the same bug": service scopes it. A NullPointerException in payment-service is a different incident from the same exception in auth-service. Same symptom, different system, different owner. messageTemplate is the message with all dynamic values replaced by placeholders. This is where the de-noising happens, and it is the part with the subtle bug, below. exceptionType separates two exception classes thrown from the same line. A TimeoutException and a NullPointerException out of PaymentProcessor.charge are two bugs, not one. firstStackFrame pins it to the exact throw site, with the line number deliberately removed. The hash itself is unremarkable, standard SHA-256, 64 hex chars. Everything interesting happens before it, in deciding what string goes in. Building the template: order is the whole game Stripping dynamic values is three regex replacements. The catch is that the order is not interchangeable; get it wrong and the fingerprint silently corrupts. public String buildMessageTemplate(String message) { if (message == null || message.isEmpty()) { return ""; } String template = UUID_PATTERN.matcher(message).replaceAll("<uuid>"); template = IP_PATTERN.matcher(template).replaceAll("<ip>"); template = NUMBER_PATTERN.matcher(template).replaceAll("<number>"); return template.trim(); } UUID first, then IP, then number. The reason is that all three are made of digits, and the most general pattern will eat the more specific ones if it runs first. NUMBER_PATTERN is \d+, any run of digits. Run it first on 192.168.1.1 and you get <number>.<number>.<number>.<number>, and IP_PATTERN never gets to match because there are no digits left. Run it first on a UUID and you shred the hex groups the same way. So the rule is most-specific-first: UUID (the most structured), then IP, then the catch-all number. Walk one message through it: Input: "Timeout from 192.168.1.1 after 30000ms, id=a3f4b2c1-1234-5678-abcd-ef0123456789" UUID: "Timeout from 192.168.1.1 after 30000ms, id=<uuid>" IP: "Timeout from <ip> after 30000ms, id=<uuid>" Num: "Timeout from <ip> after <number>ms, id=<uuid>" Every log line from that bug, whatever its IP, timeout, or request ID, lands on that same final template. That is the collapse the dedup ratio is measuring. The stack frame: stable on purpose The first stack frame is the call site that threw. Only the first frame is used, because deeper frames vary with the request path that led there, while the throw site is stable for the same bug. And the line number is stripped: String firstLine = stackTrace.split("\n")[0].trim(); firstLine = firstLine.replaceFirst("^at\\s+", ""); // remove "at " prefix firstLine = firstLine.replaceAll("\\(.*?\\)", ""); // remove (FileName.java:142) Input: "at com.log0.PaymentProcessor.charge(PaymentProcessor.java:142)" Output: "com.log0.PaymentProcessor.charge" The line number is removed on purpose. A refactor that shifts PaymentProcessor.charge from line 142 to 145 does not change the bug, so it must not change the fingerprint. If line numbers were part of the hash, every cosmetic edit above the throw site would fork a brand-new incident for a bug you already had open. Stripping the line number is the difference between a fingerprint that tracks a bug and one that tracks a source-file revision. Two small details that prevent silent collisions Two things in the final assembly look like boilerplate and are not. The pipe delimiter is load-bearing. The four fields are joined with |, not concatenated: String input = safe(service) + "|" + safe(template) + "|" + safe(exceptionType) + "|" + safe(firstFrame); Without a delimiter, ("ab", "c") and ("a", "bc") both produce "abc" and therefore the same fingerprint, despite being different error patterns. The pipe keeps the field boundaries injective, so two different field splits can never collide into one hash. Null becomes empty, never the text "null". Every field goes through a safe() guard: private String safe(String s) { return s == null ? "" : s; } A null service must contribute the empty string, giving "|template|...", not the literal four characters "null" giving "null|template|...". Those are different hashes, and letting a null stringify would mean a missing field changes a bug's identity. The guard makes a missing field contribute nothing, which is what "missing" should mean. The tradeoff I chose on purpose No fingerprint scheme is perfect, and the honest question is not "can it be fooled" but "which way does it fail." This one fails toward noise, never toward silence. If a log format uses dynamic values the regexes do not recognize, say a hex token that is not a UUID, those values survive into the template, and two lines from the same bug get two templates and two fingerprints. The bug under-deduplicates: it produces more incidents than it should. That is annoying. It is also loud and visible, you see two incidents for one bug and you go widen a regex. The failure I refused to accept is the opposite: two genuinely different bugs collapsing into one fingerprint and one incident. That hides a real problem behind an already-open incident, and the miss surfaces when the second bug takes down something the first incident's owner was never looking at. Over-dedup is a silent miss; under-dedup is a visible nuisance. Given a forced choice, a monitoring system should be biased toward too many incidents, never too few. The design is biased that way deliberately. What is not done exceptionType is in the formula but currently passed as null. The slot exists in generate() and contributes to the hash, but the normalizer does not yet parse the exception class out of the trace, so today it is always the empty string. Effect: two different exception types thrown from the same line currently share a fingerprint. It is a documented TODO; the wiring is half-built, the formula is ready for it. The template regexes cover UUID, IPv4, and integers. Not IPv6, not hex IDs, not ISO timestamps embedded mid-message, not floats. Those pass through and can cause the benign under-dedup above. The right fix is a small, ordered, tested library of patterns, extended as real formats demand, not a guess at every format up front. The dedup ratios above are single-node, one-laptop measurements (Docker Desktop, 512 MB per service, single Redpanda node, k6 at 50 VUs). They characterize the mechanism's behavior, not a throughput ceiling. Next: post 4, from fingerprints to an incident. A fingerprint names a bug, but one occurrence is not an incident; ten of them inside five minutes is. The next post is the clustering service that counts each fingerprint inside a tumbling event-time window and pages exactly once when a window crosses the threshold, the emit-once guard that keeps a loud bug to a single incident, and the in-memory window store that has no eviction and what that quietly costs. Try log0 log0 is the platform this series is built on, an open, multi-tenant incident pipeline you can run yourself or use hosted. Platform: log0.in Docs: log0.in/docs Console: console.log0.in charfield, the ASCII animation registry behind the log0 front ends: charfield.log0.in Written by Ashmit JaiSarita Gupta. Find me on LinkedIn, GitHub, and X, and read the rest of the series on Hashnode.
Running Qiskit circuits on a near-Clifford simulator: building the clifft-qiskit provider
Running Qiskit circuits on a near-Clifford simulator: building the clifft-qiskit provider Most quantum simulators are exponential in the qubit count. clifft, a simulator from the Unitary Foundation, is not: it is a fast, exact simulator for near-Clifford circuits, and its cost scales with the non-Clifford content of a circuit rather than its width. A circuit that is mostly Clifford with a handful of T gates and rotations simulates in time that grows as 2^k, where k is the small "active" dimension the non-Clifford operations open up, not 2^n in the total qubit count. On low-magic workloads that is the difference between a circuit that is tractable and one that is not. There was one friction point. clifft's native input is Stim circuit text. If your circuit lives in Qiskit, as a large amount of quantum software does, you had to hand-write or hand-generate Stim before clifft could run it. During unitaryHACK 2026 I closed that gap by building clifft-qiskit, a Qiskit BackendV2 provider that lets you run a QuantumCircuit on clifft directly: from qiskit import QuantumCircuit from clifft_qiskit import ClifftProvider qc = QuantumCircuit(2, 2) qc.h(0); qc.cx(0, 1); qc.measure([0, 1], [0, 1]) backend = ClifftProvider().get_backend("clifft") counts = backend.run(qc, shots=1000).result().get_counts() print(counts) # {'11': ~500, '00': ~500} The work started in the main repo (issue #39, PR #120), then moved to a standalone package (clifft-qiskit) with its own release automation (issue #2, PR #3), now published on PyPI. This post is a walk through that: the shape of the adapter, the one part that is easy to get wrong, how I validated it, and what it took to turn a subpackage into a released library. Note: I am a contributor, not a maintainer, of clifft. This is my reading of the code and my own work on the provider; corrections from the clifft team are welcome. Why bridge to clifft at all Before the adapter, the case for it. clifft keeps the exact state in a factored form: a compile-time Clifford frame (resolved once, ahead of time, using Stim's tableau simulator), a lightweight runtime Pauli frame updated per shot, and only a small active subset of the state carried as a dense complex array of size 2^k. Clifford gates are absorbed into the frame at zero per-shot cost; each T gate or rotation is what grows k. The published benchmarks (arXiv 2604.27058) put it at roughly 10x slower than Stim on pure-Clifford QEC sampling, but far ahead of other near-Clifford tools on magic-heavy workloads (their numbers include ~370x over Tsim on distance-3 cultivation and ~13x over SOFT on distance-5), and competitive with full statevector simulators like Aer and qsim on dense circuits. The point for a Qiskit user is simple: for the large class of circuits that are mostly Clifford, clifft can be much cheaper than a general statevector simulator, and it is exact, not approximate. The only thing standing between a Qiskit QuantumCircuit and that speed was a format boundary. A provider removes it. The design, decided up front The issue that scoped this (#39) fixed the shape before any code: a single synchronous BackendV2, all-to-all connectivity, terminal measurement returning counts, a documented basis, clear errors on anything unsupported, and one hard constraint worth calling out - importing clifft must not require Qiskit. The simulator core stays dependency-light; the Qiskit dependency lives only in the provider. That single rule is what later justified pulling the provider into its own package. The basis was chosen as Clifford+T plus single-qubit rotations. That choice is what keeps the provider exact rather than approximate, for reasons covered in the section on rotations below. The provider is a translation layer over two clifft calls Here is the pipeline in one picture. A QuantumCircuit goes in, a counts dict comes out, and every quantum operation happens inside two clifft calls in the middle. The entire pipeline lives in ClifftBackend.run(). Stripped to its spine: def run(self, run_input, **options) -> ClifftJob: shots = options.get("shots", self.options.shots) seed = options.get("seed", self.options.seed) for circ in circuits: decomposed = transpile(circ, basis_gates=CLIFFT_BASIS, optimization_level=1) stim_text, measured_clbits = qiskit_to_stim(decomposed) if not measured_clbits: raise QiskitError("clifft backend requires at least one measurement.") program = clifft.compile(stim_text) sample = clifft.sample(program, shots=shots) # + seed, if given counts = counts_from_measurements(sample.measurements, measured_clbits, decomposed.num_clbits) ... Read that as three responsibilities: Qiskit lowers the circuit. transpile(..., basis_gates=CLIFFT_BASIS) rewrites arbitrary gates into clifft's basis. I do not write a decomposition pass; Qiskit's transpiler already knows how, and pointing it at the right basis is enough. The provider translates. qiskit_to_stim turns the lowered circuit into Stim text and returns a map from sample columns back to classical bits. This is the only code the provider truly owns, and it is deliberately small. clifft does the physics. clifft.compile(stim_text) returns a Program; clifft.sample(program, shots=...) returns a SampleResult whose .measurements is a (shots, num_measurements) array of bits. The exactness of Clifford+T and of the rotations is a property of the simulator, not of my adapter. The Qiskit surface around this is thin. ClifftProvider().get_backend("clifft") hands back a ClifftBackend(BackendV2); run() computes eagerly and wraps the finished Result in a ClifftJob(JobV1) whose submit() is a no-op and whose status() is already DONE. The backend's Target advertises the basis with num_qubits=None, which leaves the width unbounded so transpiling against the backend never caps the circuit. There is no async machinery because there is nothing to wait for: sampling has already happened by the time you hold the job. The whole translation, on one page The provider owns exactly one table. Eleven Clifford+T gates map to Stim by name, three rotations map and convert their angle, measurement records a classical bit, and anything else is refused. The non-parameterized gates are a plain lookup: _GATE_MAP = { "h": "H", "x": "X", "y": "Y", "z": "Z", "s": "S", "sdg": "S_DAG", "t": "T", "tdg": "T_DAG", "cx": "CX", "cy": "CY", "cz": "CZ", } The rotations are the interesting three, and they carry a unit conversion that is easy to miss and matters: _PARAM_GATE_MAP = {"rx": "R_X", "ry": "R_Y", "rz": "R_Z"} # inside qiskit_to_stim, for a parameterized gate: angle = float(instruction.operation.params[0]) / math.pi # radians -> half-turns lines.append(f"{_PARAM_GATE_MAP[name]}({angle}) {qubits[0]}") Qiskit measures rotation angles in radians. clifft's R_X/R_Y/R_Z extensions measure them in half-turns (multiples of pi). Divide by pi on the way across and the two agree; miss it, and every rotation is wrong by a factor of pi with no error raised. That is the kind of mistake a shape test misses and a distribution test catches, which is what the validation section is for. Everything that is not in one of those two maps, and is not a measurement or a trivial no-op like barrier, is rejected: raise QiskitError( f"clifft backend does not support operation '{name}'. " f"Supported basis: {supported}, measure." ) That is how reset and mid-circuit control end up rejected: the transpiler cannot lower a non-unitary into a unitary basis, so the unknown op reaches this branch and the error names both the offending operation and the full supported set. Refusing clearly is a feature; silently dropping an operation would be a correctness bug that surfaces as a wrong distribution three steps later. Why rotations belong in the basis Putting rx, ry, rz in the basis is not a convenience, it is what keeps the provider exact. With those three in the target, Qiskit's transpiler can lower any single-qubit unitary (u, p, and friends) and any controlled rotation into the basis by exact algebraic identities. There is no Solovay-Kitaev approximation, no synthesizing a rotation out of a long H/T sequence to some tolerance. The comment in the code states it plainly: # With rx/ry/rz in the basis, the transpiler lowers any single-qubit unitary # (u, p, ...) and controlled rotations exactly into this set -- no approximate # (Solovay-Kitaev) synthesis is needed. This only works because clifft treats rotations as first-class, native operations (they compile to exact complex-amplitude updates on the active array), rather than something to be approximated in a discrete gate set. The provider inherits that exactness directly. The general point is that a thin adapter can only expose what the engine underneath it supports, and the right choice is to surface that capability rather than hide it behind a weaker basis. Getting the counts right The gate mapping is mechanical. The part that took the most care is the boundary where clifft's sample array becomes a Qiskit counts dict, because two different ordering conventions meet there. clifft returns measurements as a (shots, num_measurements) array where column j is the j-th M instruction in program order. Qiskit reports counts as bitstrings indexed by classical bit, little-endian, and a circuit is free to measure qubits into classical bits in any order it likes. Those are not the same indexing, so qiskit_to_stim returns a measured_clbits list alongside the Stim text: measured_clbits[j] is the Qiskit clbit that the j-th measurement writes to. Packing then respects that map: for row in measurements: # one row per shot value = 0 for col, clbit in enumerate(measured_clbits): if int(row[col]): value |= 1 << clbit # clbit i contributes bit 1<<i key = hex(value) counts[key] = counts.get(key, 0) + 1 Two things make this correct. First, the bit for measurement col is placed at 1 << clbit, not 1 << col, so a circuit that measures qubit 0 into clbit 2 lands in the right place. Second, the counts are keyed as hex and the experiment result carries a memory_slots = num_clbits header, which is what lets Qiskit's own get_counts() render those hex keys into correctly-widthed, correctly-ordered binary strings, identical to what AerSimulator would produce. Decoupling measurement order from clbit order is the reason a permuted-measurement circuit comes out matching a reference simulator rather than reordered, and there is a dedicated test for that case. Validating against a full statevector simulator A simulator adapter is only useful if it is correct, so the tests do not check shapes, they check distributions, against qiskit-aer as the reference. The suite runs each circuit on both backends and asserts that every outcome's probability agrees to within a tolerance (SHOTS = 4000, TOL = 0.05), across thirteen cases: Bell and GHZ-3 entanglement, a non-Clifford H-T-H, a Toffoli that must decompose into Clifford+T, native rx/ry/rz, a general u gate and a controlled rotation lowered exactly, list execution, the permuted-clbit case, and the error paths. To sanity-check it fresh for this post, I ran an entangled rx/ry/rz circuit on both backends at 20,000 shots: The largest per-outcome gap between clifft and Aer was 0.6 percentage points, which is sampling noise at this shot count, not a modelling difference, and it is an order of magnitude inside the suite's own 5-point tolerance. On the circuits both tools can represent, the distributions match, because clifft is computing the same exact result by a cheaper route. The unit conversion, the bit packing, and the transpilation basis are all checked at once by a test that would fail if any of them were wrong. From a subpackage to a package The provider first landed inside the clifft repo as an optional clifft[qiskit] subpackage (PR #120). During review, the decision was to pull it out into a standalone repository and package, clifft-qiskit, rather than keep it in-tree. The reasoning is the same "keep clifft dependency-light" rule from the design, seen from the other side: clifft is a compiled C++ core with Python bindings, released on its own cadence. Qiskit's API moves faster and is pure-Python. Binding the two release cycles together helps neither. A separate package lets the provider declare qiskit>=2.0 and track it independently, while clifft keeps a minimal dependency surface and never imports Qiskit unless a user asks for the provider. This is the established pattern in the ecosystem: a lean simulator core with satellite integration packages (qiskit-aer, the pytket extensions) that each track one external SDK. So the code moved, the package directory was renamed from clifft/qiskit to a flat clifft_qiskit, and the dependency became explicit rather than an extra: [project] name = "clifft-qiskit" requires-python = ">=3.12" dependencies = ["clifft>=0.4", "qiskit>=2.0", "numpy>=1.26"] dynamic = ["version"] [tool.hatch.version] source = "vcs" # version comes from the git tag, not a hardcoded string Nothing about the physics changed. What changed is that the provider became a thing you can pip install and version on its own. One tag ships the release The last piece (PR #3) was release automation, so that cutting a version is a single action rather than a manual checklist. The pipeline is driven by a git tag. Push a vX.Y.Z tag and the workflow: Builds with uv build, runs twine check, and does a wheel smoke test - it installs the freshly built wheel into a clean environment and runs the test suite against it, so a broken wheel fails before it can be published, not after. Publishes to TestPyPI on every run, using OIDC trusted publishing, so there are no PyPI API tokens stored in the repo at all. Publishes to PyPI, but only when the trigger is a tag push. A manual workflow_dispatch run deliberately stops at TestPyPI, which makes "rehearse the release" a safe, first-class operation. Cuts a GitHub release, extracting the matching section from CHANGELOG.md as the notes. Two supporting pieces make that sustainable. Versioning is derived from the tag by hatch-vcs instead of a string someone has to remember to bump (an early review caught a hardcoded "0.1.0" in the backend; the fix was to read the version from package metadata so it can never drift from the tag). And the changelog is generated by git-cliff from conventional-commit messages, with a CI check that enforces conventional PR titles, so the release notes assemble themselves from the commit history rather than being written by hand each time. The CI matrix runs Python 3.12 and 3.13 against both a pinned and an upgraded dependency set, so a breaking change in a future Qiskit shows up as a scheduled-run failure rather than a surprise in production. The result of all of it is clifft-qiskit 0.1.0 on PyPI: pip install clifft-qiskit, and the four lines from the top of this post run a Qiskit circuit on a near-Clifford simulator. What I took away A good provider is a thin adapter, and thinness is the goal. The only code this package really owns is a gate-name table and a bit-packing loop. Everything hard - decomposition, the physics, the exactness - is delegated to Qiskit's transpiler and clifft's core. Recognizing what not to build was most of the work. Exactness lives in the engine, so expose it, do not approximate it. Putting rx/ry/rz in the basis, because clifft supports them natively, is what lets the transpiler avoid Solovay-Kitaev entirely. A weaker basis would have made every rotation an approximation for no reason. The boundary between two ordering conventions is where the bugs are. The gate map is trivial; the measurement-column-to-clbit mapping and the little-endian packing are where correctness is won or lost, and they are worth a dedicated test each. Validate distributions, not shapes. Checking every outcome's probability against Aer catches the radians-versus-half-turns class of bug that a "did it return counts" test never would. Extract a package when the cadences diverge. A compiled core and a fast-moving SDK wrapper want different release cycles; splitting them keeps the core's dependency surface honest. Make releasing routine. A tag-triggered build, a wheel smoke test, trusted publishing with no stored secrets, TestPyPI before PyPI, and an auto-generated changelog reduce "cut a release" to pushing one tag, which is about as much attention as the step deserves. If you want to read the code, start at clifft_qiskit/backend.py (the BackendV2 and the run pipeline), then clifft_qiskit/_translate.py (the gate table, the angle conversion, and the counts packing). The package is on PyPI, the source is on GitHub, and clifft itself is documented at unitaryfoundation.github.io/clifft. Corrections and improvements welcome, especially from the clifft maintainers, who know the core far better than I do. Built during unitaryHACK 2026.
Green does not mean working: what a health check confirms
A liveness probe and a real request take different paths through a service, and the gap between them hides a whole class of bug. The first real POST to log0's gateway hung with no error and no log line, while every health check stayed green and /actuator/health returned 200 OK with a {"status":"UP"} body. The cause was a Kafka producer doing exactly what it is designed to do against an address that pointed nowhere. This post is about that mechanism, and about the difference between "the process is up" and "the system works." This is post 2 in a series on building log0, a multi-tenant incident-management platform. It covers the first real bug, the producer behavior behind it, and why a green dashboard can confirm a service is running while it cannot do its job. A green dashboard and a request that never returned The stack was built. Seven services, a Redpanda broker, ClickHouse, PostgreSQL, all wired into one Docker Compose file. I brought it up, watched the health checks go green one by one, and it looked done. docker compose ps showed every service up. Each /actuator/health endpoint returned 200. Spring Boot Actuator was happy. The logs showed clean startup, no exceptions, every consumer subscribed to its topic. Then I sent the first real log line. A single POST /api/v1/logs to the ingestion-gateway, the front door of the whole system. It hung. No 202, no error, no timeout for a long time. The connection sat open. The gateway had accepted the TCP connection, read the request, and then stopped, holding the line, returning nothing. Nothing in the logs explained it. The gateway was not crashing and not throwing. CPU was near zero. By every signal I had been watching, the service was healthy. It was also doing nothing, and had been from the start. What the health check was checking Here is the thing I had not internalized until this moment: a liveness probe and a real request take two completely different paths through the service. The top half of that diagram is what ran in CI and what I had been watching: GET /actuator/health hits the gateway, the gateway answers 200 OK (UP), done. That probe never touches Kafka. It does not open a producer, it does not fetch metadata, it does not send anything. It confirms exactly one thing: the JVM is running and the HTTP server is accepting connections. That is all "healthy" ever meant. The bottom half is the first real request, the one that had never run before that moment. POST /api/v1/logs reaches the gateway, the gateway builds a KafkaProducer and calls send(), and the producer tries to connect to a broker and fetch cluster metadata before it can produce. That is where it died, and it died on a connection that the health check had no reason to ever attempt. The two paths share almost no code. The probe was green because the probe path worked. The probe path working told me nothing at all about whether the producer path worked, and I had quietly assumed it did. What triggered it: one hardcoded address The producer was configured like this, in the gateway's KafkaProducerConfig: @Bean public ProducerFactory<String, RawLogEvent> rawLogProducerFactory() { Map<String, Object> props = new HashMap<>(); props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"); // ... return new DefaultKafkaProducerFactory<>(props); } // the raw-logs-dlq factory carried the same hardcoded literal localhost:9092. Hardcoded. On my host machine, during early single-process testing, localhost:9092 is exactly where the broker was, so this had worked once, on the host, outside Docker. Inside the container, it is a lie. Each container has its own network namespace. From inside the ingestion-gateway container, localhost is the gateway container itself, where nothing is listening on 9092. The real broker is a different container, reachable on the Compose network as redpanda:9092. So when the producer called send(), it tried to open a connection to localhost:9092 inside its own container, found nothing listening, and began doing what a Kafka producer is designed to do when it cannot reach a broker: it blocked, waiting for cluster metadata, up to max.block.ms. Why it blocked instead of failing This is the part worth slowing down on, because the failure mode is counterintuitive. A Kafka producer does not fail fast when the broker is unreachable. By design, send() first needs cluster metadata: which partitions exist, which broker leads each one. Until it has that metadata, it cannot decide where the record goes, so it waits. The wait is bounded by max.block.ms, which defaults to 60 seconds. So the producer was not broken in the sense of throwing. It was patiently doing its job: trying to reach a broker, retrying the connection, waiting for metadata that was never going to arrive because nothing was listening at that address. The calling thread, the one handling my HTTP request, was parked inside send() for the full block duration. From the client's side, the request hung. There was no exception to find in the logs because, for those 60 seconds, nothing exceptional had happened yet. The producer was inside its normal "broker not available, keep trying" loop. The redpanda broker, the real one, sat idle the whole time, never contacted, because no code in the running system knew its address. The entire ingestion path, the front door of the product, was non-functional in Docker, and had only ever been health-checked, never exercised end to end. The green dashboard was accurate and misleading at the same time: accurate about the process, silent about the job. The fix The fix is small, which is the uncomfortable part. The address should never have been a literal. It should come from configuration, so it can differ between the host and the container without a code change: @Configuration public class KafkaProducerConfig { @Value("${spring.kafka.bootstrap-servers:localhost:9092}") private String bootstrapServers; @Bean public ProducerFactory<String, RawLogEvent> rawLogProducerFactory() { Map<String, Object> props = new HashMap<>(); props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers); // ... return new DefaultKafkaProducerFactory<>(props); } } And in the Compose environment for the gateway: environment: SPRING_APPLICATION_JSON: '{"spring.kafka.bootstrap-servers":"redpanda:9092"}' One value, read from the environment, resolving to redpanda:9092 inside the network. On the host, the same property defaults to localhost:9092. Same code, correct address in both places. After the fix, the first POST /api/v1/logs returned 202 Accepted right away. The producer connected to redpanda:9092, fetched metadata in milliseconds, wrote the event to raw-logs, and acknowledged. The path that had never worked now worked, and from there it kept working: with the address right, the gateway sustained 3,032 requests per second in a 60-second load test, 181,995 requests total, at p99 156 ms and a p50 near 26 ms. The endpoint was never slow. It had been blocked on an address that resolved to nowhere. The lesson The trigger was mundane: a hardcoded host that resolved correctly outside Docker and pointed nowhere inside it. The lesson under it generalizes well past this one bug, and it is the reason this is post 1. A health check that does not exercise the real path is a check that the process started, nothing more. My liveness probe answered "is the JVM up and serving HTTP," and I had been reading it as "does ingestion work." Those are different questions with different answers, and the gap between them is exactly where this bug lived for as long as it did. The probe was not wrong. It was answering the question it was designed to answer. I was the one asking it to mean more than it did. The honest version of "healthy" for an ingestion service would touch the thing that can break: a readiness probe that confirms the producer can reach a broker, or, more bluntly, a smoke test that sends one real event end to end and checks an incident comes out the far side. The cheapest possible version of that, a single real POST in CI, would have caught this in seconds. I had not written it, because everything was green and green felt like enough. This is also a quiet argument for load testing earlier than feels necessary. I did not find this bug by reasoning about my config. I found it the instant I sent real traffic. The load test was not measuring throughput yet; it surfaced a correctness bug first, because it was the first thing to ever exercise the path for real. Synthetic load is not only about performance numbers. Often the first thing it tells you is whether the system does anything at all. What is not done To keep the scope honest, in the spirit of the rest of this series: The gateway has a basic liveness probe and now reads its broker address from config, but it still does not ship a readiness probe that verifies broker reachability. It should. That is the real fix for the class of bug, not only this instance. There is no automated end-to-end smoke test wired into the build yet. The manual "send one event, watch an incident appear" check is documented, not automated. These numbers are from a single-node setup on one laptop, the same scope disclaimer as everywhere in this series: Docker Desktop, 512 MB per service, a single Redpanda node, driven by k6. They characterize the path's behavior, not production capacity. Next: post 3, the fingerprint. Once a log line reaches the system, how does log0 decide that 115,489 lines are the same bug and one line is a different one, deterministically, with no machine learning, in O(1) per event? It comes down to what you strip out before you hash. Try log0 log0 is the platform this series is built on, an open, multi-tenant incident pipeline you can run yourself or use hosted. Platform: log0.in Docs: log0.in/docs Console: console.log0.in charfield, the ASCII animation registry behind the log0 front ends: charfield.log0.in Written by Ashmit JaiSarita Gupta. Find me on LinkedIn, GitHub, and X, and read the rest of the series on Hashnode.
Building log0: a multi-tenant incident platform on Kafka
A bad deploy at 3 AM A deploy ships at 02:50. One null check is missing on a hot path. By 03:00, the payment service has thrown the same NullPointerException tens of thousands of times. If your alerting forwards each line as its own page, the on-call engineer's phone has now buzzed tens of thousands of times for a single bug. The bug is one thing. The error is one thing. The log stream does not know that, so it emits one line per occurrence, and a naive alerting layer turns each line into a page. The signal, one bug, drowns in the volume, ten thousand lines. In a load test that reproduced exactly this shape, log0 ingested 115,489 log lines carrying one error pattern and created exactly one incident. A deduplication ratio of 115,489 to 1. Same run, ten distinct error patterns going in: ten incidents out. One page per real problem, regardless of volume. That collapse, from a flood of lines to a single owned incident, is the entire product. This post is the map of how it happens: the services, the data flow, the handful of decisions that shaped it, and the real numbers from load-testing the whole thing on one laptop. Every later post in this series drills into one box on the diagram below. What log0 is log0 is a multi-tenant incident-management platform for backend teams. Your services POST their logs to it over HTTP. From there it: deduplicates errors by a structural fingerprint, so 10,000 occurrences of one bug are one thing; clusters those fingerprints in time windows and opens an incident when a pattern crosses an occurrence threshold; drives each incident through a lifecycle (new, assigned, acknowledged, resolved); notifies the on-call Slack channel, with an AI-written root-cause summary attached; keeps every tenant's data structurally isolated the whole way through. The functional requirements are the obvious ones: ingest logs, group similar errors, detect incidents at a threshold, manage their lifecycle, notify a channel, and summarise. The non-functional ones are where the design lives: accept a log in single-digit milliseconds, never let ingestion go down because something downstream is slow, tolerate eventual consistency on detection, and isolate tenants by construction rather than by a WHERE clause that is easy to forget. It is seven Java / Spring Boot services talking over a Kafka API, with ClickHouse for logs and PostgreSQL for incident state. It runs as one Docker stack. Nothing here is at the Meta scale, and I will be specific later about exactly what the numbers are and are not. The shape: one stream, one direction log0 is not a web of services calling each other. It is a single linear stream. A log event enters at one end and moves in one direction, and each hop between services is a Kafka topic, not a synchronous call. Read top to bottom, that is the entire data path: ingestion-gateway takes the HTTP POST, does the minimum validation, writes the event to the raw-logs topic, and returns 202 Accepted. It never waits for processing. That single decision is the subject of post 4 in this series, Accept fast, never block. normalization-service consumes raw-logs, parses each event into a stable shape, strips the dynamic values out of the message to build a template, computes the fingerprint, writes the full event to ClickHouse, and emits to normalized-logs. The fingerprint is discussed in post 2 in this series. clustering-service consumes normalized-logs and counts occurrences per fingerprint inside a 5-minute tumbling window. When a window crosses 10 occurrences, it emits one incident-events message. Streaming aggregation is discussed in post 3 in this series. incident-service consumes incident-events, runs the incident through its state machine, and upserts it into PostgreSQL. Because the same event can arrive more than once, every write is idempotent. That is covered in post 10 in this series. notification-service consumes notification-events and posts a Slack Block Kit alert to the owning team. ai-service attaches a root-cause summary out of band, via an async callback, so a slow model never blocks the alert. Two things never flow backwards, and that is the point. Nothing downstream can apply backpressure to the gateway by making it wait, and a single poison message cannot freeze the pipeline, because normalization routes anything it cannot parse to a dead-letter topic, raw-logs-dlq, instead of dying on it. auth-service sits outside the data path: it issues JSON Web Tokens and every other service validates them locally, with no network call back to auth on the hot path. That tradeoff is discussed in post 14 in this series. Five decisions that shaped everything log0 came down to a small number of decisions, each with a cost. This series is organized around them. Here they are, stated plainly, each with the tradeoff it carries. 1. Accept fast, never block. The ingestion endpoint returns 202 Accepted the moment the event is durably on the raw-logs topic. It does no clustering, no database write, and no enrichment inline. The cost: the caller gets an acknowledgement, not a result. Detection is eventually consistent, seconds behind. For a logging endpoint that must never be the thing that is down, that is the correct trade. 2. A log line is an event with an identity. Deduplication is not string matching. log0 strips the dynamic values out of a message (numbers, IP addresses, UUIDs become placeholders), joins four stable fields with a pipe and hashes them: fingerprint = SHA-256( service | messageTemplate | exceptionType | firstStackFrame ) Same bug, different timeout values or request ID: same fingerprint, same incident. The cost is that a log format the templating regexes do not recognize will under-duplicate. The failure mode is benign, it makes too many incidents (noisy but visible), never too few (a hidden, merged bug). We have discussed this in post 2 of this series. 3. Multi-tenancy is a partition key, not a column. Events are keyed by tenantId on the Kafka topics, which buys per-tenant ordering and noisy-neighbour isolation for free, and every stored row carries tenant_id. Isolation is structural. The cost is partition skew when one tenant is far louder than the rest. As discussed in post 7 of this series, I measured the skew rather than asserting it away. 4. At-least-once delivery, plus idempotency. Kafka redelivers on a consumer rebalance or a retry, so the same incident-events message can arrive twice. Rather than fight that, every incident write is an idempotent upsert keyed on (tenant, fingerprint): a duplicate increments the occurrence count, it never forks a second incident. At-least-once plus idempotent equals effectively-once. We have discussed this in post 10 of this series. 5. Two databases, on purpose. Logs go to ClickHouse, a columnar store built for GROUP BY fingerprint over millions of rows. Incident state goes to PostgreSQL, which gives transactional updates and a clean state machine. One database cannot be great at both 10k-insert-per-second analytics and a transactional lifecycle. The cost is two systems to operate. We have discussed this in post 10 of this series. The numbers, and exactly what they are The reason to build the thing and then load-test it, rather than describe it, is that the interesting parts of a design only show up under load. Here are the headline results the series is built on. Each gets its own post with the full chart. Finding Result Deduplication ratio 1 template in: 115,489 : 1. 10 templates: 8,760 : 1. 100 templates: 967 : 1. Incidents out always equalled templates in. Single-row vs batched inserts ClickHouse insert path went from 40 rows/sec at 171% CPU to 4,185 rows/sec at 15% CPU, one change. Backlog drain Single-row inserts built a 67,071-message backlog draining at ~68/sec; batched, the backlog peaked at ~283 and never sustained, clearing within ~15 seconds. End-to-end detection First error to incident row: p50 403 ms, p99 477 ms (n=60, same clock). Concurrency sweep 50 to 800 virtual users: throughput held in a ~1,600 to 2,200 req/sec band, p99 36 ms to 709 ms, under 0.4% errors, no collapse. Tenant skew A hot tenant at 9x the traffic of a quiet one: p99 80 ms vs 79 ms. The quiet tenant's tail did not move. Dead-letter routing 300 poison messages in: 300 to the DLQ, 0 dropped, clean path unaffected. Broker under stress Redpanda hit a real exit-137 OOM at 800 VUs while /actuator/health stayed 200. Topics survived; producers auto-reconnected. Scope, once and clearly. Every number above was measured on a single laptop: Docker Desktop, 512 MB per service, a single-node Redpanda broker, ClickHouse 24.3, driven by k6. These characterize the system's behavior and its bottlenecks, not its production capacity. A single-node broker that OOMs at 800 virtual users is not a scaling claim; it is a finding about where this configuration breaks, which is exactly the kind of thing the series is about. The most useful results are the unflattering ones. The broker OOM while the health check stayed green is a small, honest "health checks lie" story. The single-row insert wall is a textbook producer/consumer mismatch you can watch build and then drain. Those are the posts I would read first. What is not done Stating the limits is part of the design, not an apology for it. The clustering window lives in process memory. A crash loses the in-flight windows. Production wants an external store (Redis) so the count survives a restart. It is designed, not yet built. exceptionType is wired into the fingerprint formula but currently passed as null, a documented TODO. Until the normalizer parses the exception class out of the trace, two different exceptions thrown from the same line share a fingerprint. The slot exists; the wiring is half done. Everything runs single-node. The share-nothing, partition-keyed design is meant to scale horizontally by adding consumers to a group, but I have not run it multi-node, so I will not draw you a scaling graph I cannot reproduce. How to read this series Each post takes one box on that diagram, or one of the five decisions, and goes deep: the problem, the naive version and why it broke (usually with a number), the fix, the real code, and the tradeoff. They stand alone, but they chain. If you came for the algorithms, start with post 2, the fingerprint. If you came for the systems engineering, start with post 6, the day a one-line change took the insert path from 40 to 4,185 rows per second. Next: post 2, the bug that load testing found and health checks never could. The whole stack reported healthy, while the ingestion path had never once been exercised end-to-end. Here is how a green dashboard lied, and what it cost. Try log0 log0 is the platform this series is built on, an open, multi-tenant incident pipeline you can run yourself or use hosted. Platform: log0.in Docs: log0.in/docs Console: console.log0.in charfield, the ASCII animation registry behind the log0 front ends: charfield.log0.in Written by Ashmit JaiSarita Gupta. Find me on LinkedIn, GitHub, and X, and read the rest of the series on Hashnode.
How qBraid routes between quantum SDKs: a tour of the transpiler's conversion graph
Quantum software has a Babel problem. A circuit can live as a Qiskit QuantumCircuit, a Cirq Circuit, a pyQuil Program, a PyTKET Circuit, an OpenQASM 2 or 3 string, a Braket circuit, a PyQIR module, and so on. Tools, simulators, and hardware each speak a subset of these. If you want any format to reach any other, the naive answer is a converter for every ordered pair - and that count grows as N×(N−1). It does not scale, and most of those converters would duplicate each other's work. qBraid's transpiler takes a different approach: it turns the problem into a weighted shortest-path search over a directed graph - the same shape as network routing, GPS directions, or dependency resolution. Program types are nodes, converters are weighted edges, and "convert A to B" becomes "find the cheapest path from A to B and apply each edge along it." You add one edge; the graph gives you many new routes for free. I recently added a few of those edges during unitaryHACK 2026 - an openqasm3 → pyquil conversion (#1179) and PyTKET interop plus a pyquil → qasm3 conversion (#1200). This post is a walk-through of the subsystem that those edges plug into. All file references are under qbraid/transpiler/. Note: I'm a contributor, not a maintainer. This is my reading of the code as of writing; corrections are welcome. The mental model: nodes and edges Nodes are program types - "qiskit", "cirq", "pyquil", "pytket", "qasm2", "qasm3", "braket", "pyqir", "qrisp", … (the same aliases qBraid uses in its program registry, QPROGRAM_REGISTRY). The set keeps growing - Qrisp was added in #1211. Edges are conversion functions, each pointing one way: source → target. Transpiling from A to B becomes "find a path from node A to node B and apply each edge's function in order." If there is no direct A→B edge but there is A→qasm3→B, the graph finds and composes that route. A simplified slice (my unitaryHACK edges highlighted): How an edge is declared: naming is registration There is no central list you edit to register a converter. A conversion function is discovered by its name and location. You write a function named {source}_to_{target} in qbraid/transpiler/conversions/{source}/, and the package in conversions/__init__.py collects it into a module-level conversion_functions list by inspecting the submodules. The graph then turns those names back into edges. From graph.py: conversion_functions: list[str] = getattr(transpiler, "conversion_functions", []) edges = [conversion.split("_to_") for conversion in conversion_functions] So pyquil_to_qasm3 becomes the edge ("pyquil", "qasm3"). The naming convention is the API. My pyquil_to_qasm3 lives in conversions/pyquil/pyquil_to_qasm3.py and looks like: from qbraid.transpiler.annotations import weight @weight(1) def pyquil_to_qasm3(program: pyquil.quil.Program) -> Qasm3StringType: ... return "\n".join(lines) + "\n" That's the whole contract: take the source object, return the target object, name it correctly, drop it in the right folder. Two decorators that tune the graph conversions/annotations.py defines two small but important decorators. @weight(value) attaches a number in [0, 1] to the edge. Crucially, this value is not a hand-tuned "cost" - it's the gate-set coverage of the conversion: the fraction of a reference gate set that the converter handles correctly. A higher weight means a more complete conversion, and 1.0 means full coverage. (How that fraction is measured, and how it becomes a routing cost, is in the pathfinding section below.) def weight(value: float) -> Callable[[F], F]: if not 0 <= value <= 1: raise ValueError("Weight value must be between 0 and 1.") ... setattr(wrapper, "weight", value) @requires_extras(*packages) marks edges that need optional dependencies beyond the function's own package. For example, PyTKET's cross-framework converters rely on the pytket.extensions.* packages, so in conversions/pytket/pytket_extras.py: @requires_extras("pytket.extensions.qiskit") def pytket_to_qiskit(circuit: pytket.circuit.Circuit) -> qiskit.QuantumCircuit: ... The decorator records the requirement on the function (requires_extras attribute). The graph reads it when deciding whether an edge is actually usable in the current environment - which keeps optional, heavy SDKs from being hard requirements. From functions to a graph: the Conversion and ConversionGraph objects Each discovered function is wrapped in a Conversion (edge.py). The constructor records more than just the endpoints: class Conversion: def __init__(self, source, target, conversion_func, weight=None, bias=None): self._source = source self._target = target self._conversion_func = conversion_func self._bias = bias if bias is not None else 0 self._weight = self._get_adjusted_weight(weight) self._extras = getattr(conversion_func, "requires_extras", []) self._native = self._is_module_native(conversion_func) self._supported = self._is_conversion_supported() A few things worth calling out: weight is taken from the @weight attribute (the coverage fraction), then transformed into a routing cost (next section). bias fine-tunes the depth-vs-coverage tradeoff and is added once per hop. From the docstring: a bias of 0.25 slightly favours a single conversion at weight 0.8 over two conversions at weight 1.0; a smaller bias requires the single hop to have even higher coverage before it wins. This is how the router is steered between "fewest hops" and "highest-coverage hops." _native flags whether the converter is internal to qBraid. _supported checks (via _extras) whether the optional deps are importable, so unusable edges can be excluded. ConversionGraph (graph.py) builds the actual graph. load_default_conversions() constructs a Conversion per discovered function, and create_conversion_graph() adds them as edges, stashing the callable and weight on each edge's payload: self.add_edge( ..., {"native": edge.native, "func": edge.convert, "weight": edge.weight}, ) Pathfinding: Dijkstra over rustworkx The graph is backed by rustworkx (imported as rx), a Rust-accelerated graph library - chosen over the pure-Python networkx because the core operation is graph traversal, where a compiled implementation pays off. Routing is plain Dijkstra (O(E log V) with a binary heap) with the edge cost as the weight function: import rustworkx as rx path = rx.dijkstra_shortest_paths( self._graph, ..., weight_fn=lambda edge: edge["weight"], ) Coverage → cost: how the @weight value is used A higher-coverage edge should be cheaper to traverse, and Dijkstra minimises total cost, so the coverage fraction is inverted and log-transformed before it goes on the edge. Conversion._get_adjusted_weight (edge.py) does exactly that: # effective_weight is the @weight coverage fraction, in [0, 1] rx_adjusted_weight = float("inf") if effective_weight == 0 else np.log(1 / effective_weight) adjusted_weight = rx_adjusted_weight + self._bias So each edge's cost is log(1 / coverage) + bias: Full coverage (1.0) → log(1) = 0, cost is just the bias. Lower coverage → larger log(1/coverage) → more expensive. bias is paid once per hop, so a single direct edge beats a multi-hop chain unless the chain's edges are substantially higher-coverage. Why the log? What you actually want is the path that maximizes the product of the per-edge coverages (the chance the whole chain round-trips faithfully). Dijkstra minimizes a sum, not a product. Taking -log converts one into the other: max ∏ coverageᵢ becomes min Σ log(1/coverageᵢ), because log is monotonic and turns products into sums. It's the same trick used for maximum-probability paths (e.g. Viterbi). The bias is then a small per-hop penalty layered on top to break ties toward shorter paths. A worked example (default bias = 0.25): one direct edge at coverage 0.8 costs log(1/0.8) + 0.25 ≈ 0.47; two chained edges at coverage 1.0 cost (0 + 0.25) × 2 = 0.50. The direct edge wins - barely - which is the intended "prefer one good hop over two perfect ones" behaviour. This is also why adding pyquil → qasm3 matters: replacing the old pyquil → cirq → qasm2 → qasm3 chain (three biases, three lossy hops) with one edge is a big cost drop, and the router switches to it automatically. Where the coverage number actually comes from The part that surprised me: weights aren't guessed - they're measured. Each source type has a coverage benchmark in tests/transpiler/<source>/test_coverage_from_<source>.py. The test builds a circuit for every gate in a reference gate set, transpiles it to each target, and checks equivalence; the fraction that round-trips correctly is the weight. The Cirq benchmark even encodes the per-target baselines directly: ALL_TARGETS = [("braket", 0.85), ("pyquil", 0.74), ("pytket", 0.87), ("qiskit", 0.87)] When I added the openqasm3 source edges, there was no test_coverage_from_openqasm3.py yet, so the workflow was: write the benchmark, run it, and set @weight from the measured coverage - with the maintainer's guidance to "aim as close to 1.0 as we can." That's a nice property: the routing policy is grounded in tested behaviour, not vibes, and a regression in coverage shows up as a failing benchmark rather than a silently worse route. find_shortest_conversion_path returns the ordered conversion functions to apply; find_top_shortest_conversion_paths(..., top_n=...) returns several candidates (handy as fallbacks when a conversion fails at runtime). On top sit has_path, shortest_path / all_paths (human-readable routes like "pyquil -> qasm3 -> qiskit"), and closest_target (pick the cheapest reachable target from a set - useful when a backend accepts several input formats). That's the payoff: I only wrote pyquil → qasm3, and because qasm3 already reaches qiskit, cirq, braket, and more, pyQuil now converts to all of them - no pyquil → qiskit written by hand. Where my edges fit qBraid can render the live graph itself - ConversionGraph().plot(...) (matplotlib). Here is the current graph, with the nodes my unitaryHACK edges touch outlined: from qbraid.transpiler import ConversionGraph ConversionGraph().plot( legend=True, target_nodes=["pyquil", "qasm3", "pytket", "pyqir"], save_path="qbraid_conversion_graph.png", ) Blue nodes are qBraid-native program types, grey are external; solid edges are native conversions, lighter edges are requires_extras ones. Notice how qasm3 sits at the centre as the interchange hub - that's why adding a single edge to or from it unlocks so many routes. openqasm3 → pyquil (conversions/openqasm3/openqasm3_to_pyquil.py, #1179): rather than parse OpenQASM by hand, it reuses qBraid's pyqasm engine - loads → validate → unroll the program into a flat, basis-gate AST, then walks unrolled_ast.statements and maps each node onto pyQuil (QubitDeclaration → flat qubit indices, ClassicalDeclaration → declare("ro", "BIT", n), QuantumGate → pyquil.gates.*, measurement → MEASURE). It also converts duration literals to seconds (_duration_seconds over _TIME_UNIT_SECONDS) so timing-bearing programs survive, and raises ProgramConversionError on anything unsupported. Leaning on pyqasm for the heavy lifting is the same pattern as openqasm3_to_cudaq. pyquil → qasm3 (conversions/pyquil/pyquil_to_qasm3.py, #1200/#1208): the reverse direction, written from scratch over pyQuil's instruction model. Together with openqasm3 → pyquil it completes a PyQuil ↔ OpenQASM 3 round trip and replaces the old lossy pyquil → cirq → qasm2 → qasm3 chain with a single edge. PyTKET interop via extras (#1208): cirq_to_pytket and pytket_to_cirq (via pytket.extensions.cirq) and pytket_to_pyqir (via pytket.qir, wrapped in pyqir.Module.from_ir), each @requires_extras-gated and auto-discovered. These replace lossy multi-hop detours (e.g. cirq → qasm2 → pytket) with direct 1-hop edges and give PyTKET a direct outbound edge to PyQIR. A practical lesson from this PR: I originally proposed pyquil ↔ pytket via pytket-pyquil, but that extra pins pyquil<5 and conflicts with qBraid's pyquil>=5 in a single CI env, so it would have silently downgraded pyquil and broken existing conversions. Checking the dependency graph before committing to an edge saved a broken PR - the conflict-free set (cirq↔pytket, pytket→pyqir, plus the from-scratch pyquil→qasm3) closed the issue instead. Each is just one node-to-node edge, but each widens what the rest of the graph can reach. Testing conversions Conversions are easy to get subtly wrong, so the tests are deliberately strict. The rule I internalised from review: an A_to_B test should compare in the source and target types themselves and not lean on a third framework to "check the answer." If your pyquil → qasm3 test secretly parses the qasm3 with Qiskit and compares to a Qiskit-built circuit, you're now testing three libraries' agreement, not your one conversion - and a failure no longer tells you where the bug is. Keeping the test isolated to the source and target keeps it honest and the signal sharp. A second subtlety is the global phase. Adding a direct pytket -> cirq edge made it the default 1-hop route, replacing the old pytket -> qasm2 -> cirq path. The direct converter represents ZZPhase(t) as cirq.ZZ ** t, which is unitary-equivalent up to an unobservable global phase. So the default-route assertion uses circuits_allclose(..., strict_gphase=False), while the native pytket -> qasm2 -> cirq route is still checked exactly (including global phase) via a separate require_native=True graph in the same test. The lesson: when you change which route is the default, check what equivalence the new route actually guarantees, and assert at that level - not a stricter one it can't meet, nor a looser one that hides bugs. What I took away The naming convention is the plugin system. {source}_to_{target} plus folder placement is all the registration there is - a clean, discoverable contract. A graph turns N² into N. Adding one well-placed edge (especially to a hub like qasm3) multiplies reachable pairs without writing direct converters. It's a textbook algorithm under a domain skin. Strip away the quantum vocabulary and this is weighted shortest-path: model the domain as a graph, pick a cost that makes the metric you care about (coverage) additive via -log, run Dijkstra. The hard part isn't the algorithm - it's recognising the problem is that algorithm. Routing is a measured policy, not a hardcode. Edge cost is log(1/coverage) + bias, and the coverage number comes from a per-source benchmark test - so the graph's choices are grounded in tested behaviour, and a coverage regression fails a test instead of silently picking a worse route. requires_extras keeps the dependency surface honest - optional SDKs stay optional, and the graph simply omits edges it can't run. If you want to read the code, start at qbraid/transpiler/graph.py (the graph and pathfinding), then edge.py (the Conversion wrapper) and annotations.py (the decorators), and finally browse conversions/<source>/ for concrete edges. Corrections and improvements welcome - especially from the qBraid maintainers, who know this code far better than I do.
DNS Isn't a Server. It's a Conversation Between Four of Them.
What actually happens in the 30 milliseconds between you hitting Enter and your browser knowing where to send a packet - and why "DNS propagation" exists.
DNS Isn't a Server. It's a Conversation Between Four of Them.
The picture Many introductory diagrams show Open any "how the internet works" article. You'll see this picture: "Your browser asks a DNS server for the IP address. The DNS server responds. The browser connects." It's a useful starting point. The parts it leaves out are the parts that explain CDNs, "DNS propagation delays", and why your laptop's network settings have a "DNS server" field at all. In practice, resolution involves your browser's memory, your operating system's cache, your router, your ISP's recursive resolver, one of 13 root servers, a TLD server, and finally an authoritative server - typically four to six different machines, often on three different continents, all involved in resolving one question: "what's the IP for google.com?" Often, the full lookup never happens because the answer is already cached locally or at the resolver. This post walks through the entire chain - what each player does, why the design looks weird, and what tools like dig reveal about the conversation. 1. Why DNS Exists at All Computers route packets using IP addresses - 32-bit numbers like 142.250.183.78. Humans remember names like google.com. DNS is the translation layer between the two. That much is obvious. The interesting part is why the translation needs four servers instead of one. The answer is scale. There are around 350 million registered domain names. No single machine could answer queries for all of them, store all the records, and absorb the global query load. So DNS is structured as a distributed hierarchical database, where different organisations own different slices of the namespace and are responsible only for their own. Every DNS lookup is a tour through that hierarchy. 2. The Hierarchy Domain names read right to left in increasing specificity: The trailing dot (the root) is implicit in browsers but real in DNS. Every fully qualified domain name actually ends in .. Each segment is owned by someone: Root (.) - managed by ICANN, served by 13 root server clusters (named A through M). They know nothing except who runs each TLD. TLD (.com, .dev, .in) - managed by registries like Verisign (.com) or Public Interest Registry (.org). They know who owns each second-level domain in their TLD. Authoritative server for google.com - run by Google. Knows every record under google.com. The hierarchy is what allows DNS to scale globally without centralising the entire namespace. Nobody has to know everything, but everybody knows who to ask next. 3. Before Anything Hits the Network: Caches When you type google.com in your browser, the very first thing that happens is three cache lookups, in order, all on your own machine: Browser cache. Chrome, Firefox, and Safari each maintain their own. Try chrome://net-internals/#dns to see Chrome's. OS resolver cache. Linux uses systemd-resolved or nscd; macOS uses mDNSResponder; Windows uses the DNS Client service. /etc/hosts (or C:\Windows\System32\drivers\etc\hosts). Static overrides. This is how localhost resolves to 127.0.0.1 without any network call. If any of these have the answer and the TTL hasn't expired, the lookup is done in microseconds. No packets sent. This avoids repeated network lookups for frequently visited domains. Only when all three miss does the query leave your machine. 4. The Recursive Resolver Your machine doesn't talk to root servers directly. It sends one query to a recursive resolver - typically your ISP's, your router's, or a public one like: Resolver Operator 8.8.8.8, 8.8.4.4 Google Public DNS 1.1.1.1, 1.0.0.1 Cloudflare 9.9.9.9 Quad9 (security-focused) 208.67.222.222 OpenDNS The recursive resolver does all the heavy lifting on your behalf. You ask it once: "What's the IP for google.com?" and it goes off, talks to multiple servers, and returns a single answer. From the client’s perspective, the lookup appears as a single request-response cycle even though multiple servers may be involved internally. 5. The Full Chain - Resolver's Side Let's say the resolver itself has nothing cached. Here's what happens. Three referrals, one final answer, then back to you. The total round trip is typically 20-50 ms because each segment caches results from previous queries. Note: the root doesn't return Google's IP. It only knows where the .com servers live. The .com servers don't return the IP either - they only know where Google's authoritative servers live. Only the authoritative server has the actual record. 6. Record Types (More Than Just A) A DNS query specifies a type. Different types return different things: Type Returns Used for A IPv4 address the basic case AAAA IPv6 address IPv6 CNAME Another domain name aliases (see CDN section) MX Mail server hostname + priority email routing NS Authoritative name servers delegation TXT Arbitrary text SPF, DKIM, domain verification SOA Zone metadata replication, TTLs PTR Hostname for an IP reverse DNS Try yourself: dig A google.com dig MX gmail.com dig TXT google.com dig NS google.com Each query type is independent. Asking for an A record doesn't tell you about MX records. 7. Why CDNs Live and Die by CNAMEs A CNAME record says "this name is an alias for that name." When the resolver gets a CNAME response, it restarts the lookup using the new name. This mechanism is what makes CDNs and managed DNS routing possible. www.yoursite.com. IN CNAME yoursite.cdn.cloudflare.net. yoursite.cdn.cloudflare.net. IN A 104.21.42.7 yoursite.cdn.cloudflare.net. IN A 172.67.180.34 You set www.yoursite.com to a CNAME pointing at Cloudflare. Cloudflare resolves that to whichever edge node is closest to the requesting user. You never have to update yoursite.com's DNS again, even if Cloudflare adds 100 new edge locations next month. Without CNAMEs, every CDN would need direct write access to every customer's DNS. CNAMEs make CDNs possible. Catch: CNAMEs cannot coexist with other records on the same name, and they cannot live at the apex (yoursite.com without www.). This is why people use ALIAS or ANAME records (provider-specific extensions) at the apex. 8. TTL - The "Why DNS Changes Take 24 Hours" Mystery Every DNS record has a TTL (time to live), measured in seconds. It tells caches how long they're allowed to keep the answer before re-asking. $ dig A google.com +noall +answer google.com. 300 IN A 142.250.183.78 ^^^ TTL in seconds = 5 minutes When you change a DNS record, existing caches don't know. They keep serving the old value until their copy expires. With a 24-hour TTL, some users will see the old IP for up to 24 hours after the change. This is what people mean by "DNS propagation." DNS doesn't actually propagate - caches expire and reload. Tradeoff: Low TTL (60s) - fast changes, more queries, more load on authoritative servers, slightly slower page loads (more cache misses). High TTL (24 h) - fewer queries, cheaper, but painful when you need to migrate. Best practice: drop TTL hours before a planned change, then raise it back after. 9. UDP, TCP, and the 512-Byte Rule DNS queries fit in a single packet 99% of the time, so DNS uses UDP on port 53 by default. No handshake, no connection setup, just one packet out and one back. Original DNS spec capped responses at 512 bytes to fit in one UDP packet. Two things bust this limit: Large responses - many records, big TXT for SPF, etc. DNSSEC - cryptographic signatures bloat responses 5-10x. When the response would exceed 512 bytes, the server sets the TC (truncated) flag in its UDP reply. The resolver then re-sends the same query over TCP on port 53. Modern DNS uses EDNS(0) to negotiate larger UDP responses (up to 4096 bytes), but TCP is still the fallback. This is also why DNS over HTTPS (DoH) and DNS over TLS (DoT) are recent innovations - encrypted DNS requires TCP-style transport. 10. Watching It Live with dig dig is the Unix tool that shows the entire conversation. Two flags reveal the most: dig +trace google.com This makes dig start at the root and walk down the hierarchy itself, printing each step: ;; QUESTION SECTION: ;google.com. IN A ;; AUTHORITY SECTION (from root): . 518400 IN NS a.root-servers.net. . 518400 IN NS b.root-servers.net. ... 13 root servers ... ;; Received 1097 bytes from 198.41.0.4#53(a.root-servers.net) in 11 ms ;; AUTHORITY SECTION (from .com TLD): google.com. 172800 IN NS ns1.google.com. google.com. 172800 IN NS ns2.google.com. ;; Received 836 bytes from 192.5.6.30#53(a.gtld-servers.net) in 24 ms ;; ANSWER SECTION (from authoritative): google.com. 300 IN A 142.250.183.78 ;; Received 55 bytes from 216.239.32.10#53(ns1.google.com) in 8 ms Three round trips, each to a different server. The TTLs (518400, 172800, 300) tell you how long each layer caches its answers. dig +short google.com # just the IP dig @1.1.1.1 google.com # query a specific resolver dig CNAME www.github.com # see the CNAME chain 11. Security Model and Weaknesses DNS was designed in 1983, before anyone worried about adversaries. It shows. DNS Spoofing / Cache Poisoning. A resolver waits for a UDP response. UDP has no handshake - anyone who can guess the query ID and reply faster than the real server can inject a fake answer. The resolver caches it. Now everyone using that resolver gets the attacker's IP for paypal.com. Mitigations: Source port randomisation - adds 16 bits of entropy. DNSSEC - cryptographically signs records. Resolvers verify the signature before trusting. Adoption is patchy because deployment is operationally painful. DNS over HTTPS (DoH) - sends queries over HTTPS instead of plain UDP. Encrypts the channel and hides DNS from network observers. Cloudflare's 1.1.1.1 and Google's 8.8.8.8 both support it. DNS over TLS (DoT) - same idea, but on a dedicated port (853) rather than HTTPS. DoH is controversial because it bypasses network-level DNS filtering used by enterprises and (sometimes) governments. That's a feature for users and a problem for network admins. 12. Putting It Together The full sequence when you type google.com and hit Enter: Every step is cached at the layer above it. The full walk from root to authoritative is the worst case, not the typical case. In practice, your resolver usually has TLD answers cached, sometimes the second-level too, and the whole thing finishes in a single round trip. But when something is genuinely uncached - first lookup of the day, fresh resolver boot, brand-new domain - every layer fires. And it still finishes in 30-50 ms because the hierarchy makes each individual query small and stateless. Closing Thought DNS is one of the few internet systems that's barely changed in 40 years. The hierarchy from 1983 still runs the modern internet. Most "improvements" since then have been bolted on without breaking the original design - exactly like TCP window scaling, exactly like HTTP/2's relationship to HTTP/1. The key idea is the delegation model: nobody has to know everything, but everybody knows who to ask next. That structure is why DNS scales to 350 million domains, why CDNs work, why you can switch hosting providers without anyone noticing, and why a tiny edit in your registrar's panel propagates across the world in hours. Next time you watch a webpage load slowly, run dig +trace. You’ll be able to see each delegation step directly. Found this useful? Next post in the series: the TCP three-way handshake - why it's three packets and not two, and how an attacker once brought down a quarter of the internet by exploiting that exact design choice.
Introduction to System Design
System Design often feels intimidating at first - big diagrams, fancy terms, and interviews that expect you to “just know” how large systems are built. This blog series is my attempt to make Low-Level System Design (LLD) simple, practical, and approa...
Introduction to System Design
System Design often feels intimidating at first - big diagrams, fancy terms, and interviews that expect you to “just know” how large systems are built. This blog series is my attempt to make Low-Level System Design (LLD) simple, practical, and approachable, especially for freshers and undergrad students. Most of the concepts shared here are based on my learning from Rohit Negi and Aditya Tandon’s “Low-Level Design (LLD) Masterclass” on YouTube, combined with my own notes and understanding. This series is for you if: You want a quick and structured revision of LLD concepts Long video courses feel hard to follow (language, pace, or time issues) You’re starting system design from scratch and want clarity You’re preparing for SDE interviews and want strong fundamentals I’ll be publishing a new article every 1–2 weeks, keeping things crisp and focused. If you find this useful, consider liking the blog and subscribing to the newsletter so you don’t miss upcoming posts. Low‑Level Design (LLD) Low-Level Design focuses on the internal structure of an application. It answers questions like: What classes should exist? How do objects interact with each other? How does data flow inside the system? Where do algorithms and data structures fit into the code? In simple terms: LLD is about how your code is organised internally. It covers: class design object relationships interfaces and abstractions clean separation of responsibilities Core Principles of Low-Level Design LLD mainly focuses on three things: 1. Scalability The system should handle large numbers of users easily Code structure should allow easy expansion (new features, more servers, new requirements) 2. Maintainability Adding new features should not break existing functionality Code should be easy to debug, read, and modify 3. Reusability Write loosely coupled, plug-and-play modules Example: a notification system a matching or recommendation algorithm These can be reused across apps like Zomato, Swiggy, Amazon Delivery, etc. High-Level Design (HLD) While LLD focuses on code structure, High-Level Design (HLD) focuses on the big picture - the overall system architecture. HLD deals with: overall system architecture interaction between major components tech stack choices database selection (SQL / NoSQL / Hybrid) scaling strategies (load balancers, auto-scaling) deployment and cloud cost optimisation (AWS / GCP) In short: HLD explains how the entire system works at scale. DSA vs LLD vs HLD (Quick Summary) DSALLDHLD Brain of an applicationSkeleton of the applicationArchitecture of the application Algorithms that solve specific problems efficientlyClasses, object models, code structure, and where algorithms fitServers, databases, infrastructure, tech stack, and scaling All three are important—and they work best together. What’s Ahead Before diving into LLD design patterns, we need a standard and structured way to represent our designs. In the industry, designs are often explained using simple, well-defined diagrams that everyone understands. This is where Unified Modelling Language (UML) comes in. UML is a general-purpose modelling language used to visually represent system designs. Its goal is to provide a standard way to visualise how a system is built, similar to blueprints in civil or mechanical engineering. In the next article, I will deep-dive into UML diagrams, explain their different types and components, and show how they help us represent system design concepts clearly—so that we can discuss and apply design patterns efficiently. Final Note If you’re learning system design or preparing for interviews, you’re not alone—it does take time. The goal of this series is to build strong fundamentals step by step, without overcomplicating things. Feel free to drop questions or feedback in the comments—I’ll try to address them in future posts.
How to Get the Most Out of Open-Source Mentorship Programs like GSoC and (and how to get selected)
It’s again that time of year when my DMs are filled with people asking about how to crack the upcoming Google Summer of Code, how to contribute to open-source, how to make a place in the org community, and how to get their PR merged. So, let's cut th...
How to Get the Most Out of Open-Source Mentorship Programs like GSoC and (and how to get selected)
It’s again that time of year when my DMs are filled with people asking about how to crack the upcoming Google Summer of Code, how to contribute to open-source, how to make a place in the org community, and how to get their PR merged. So, let's cut the fluff and get down to the absolute minimum you need to know about navigating the GSoC organisation selection and getting your proposal accepted. Think of GSoC less as a competition to "crack" and more as a learning opportunity that pays off later. The most important thing you'll learn is not during the project, but during your preparation. Why You Can Trust This Methodology (My Experience with GSoC) I first tried for GSoC in 2023 during my second undergraduate year. I was confident I'd get selected, but I failed. I applied again and was selected as a GSoC mentee in 2024. Over those two years, I experimented with multiple strategies across various organizations, which is why I know what actually works. I worked on the "Development of a UI Kit for the AsyncAPI website using Storybook in Next.js and Typescript" as the GSoC’2024 project at the AsyncAPI Initiative. Crucially, I contributed to AsyncAPI for eight months before GSoC even started. In that time, I tackled major problems, including: Redesigning and building the Modelina Playground, an interactive tool for live experimentation. Migrating the Modelina website from class-based to functional React components. Migrating the AsyncAPI Website codebase to TypeScript and Next.js. I became a maintainer of the Modelina Website and a part of the Technical Steering Committee—all before being selected as a GSoC mentee. This year, I am currently a GSoC 2025 mentor for the project "Redesign and Redevelopment of the AsyncAPI Initiative website." In short: I've been on both sides of the fence, so you can follow this methodology without worrying. The Absolute Best Way to Shortlist GSoC Organisations Start Early (Now, not March): The surest way to win this game is to start contributing now (October-November). By the time the February-March crowd shows up, you’ll be an established contributor with an intimate knowledge of the codebase, upcoming potential projects and bigger issues to solve, leaving them to fight over the small, trivial bug fixes. Go to the Source: Visit the official GSoC organisation listing page (even last year's list works for initial research because most of the organisations repeat, their projects are follow-ups to last year's projects, and last year’s mentees will be the upcoming mentors). The A-Z Marathon: Forget third-party filters or sorting by tech stack—they're often misleading. A machine learning organisation might need a web developer for its project's dashboard, and you'd miss it if you filtered only for ML. The best way is the old-fashioned, methodical approach. Take one to two weeks and go through the list A to Z. Open each organisation's: Website: What is their core mission? What problem are they trying to solve? GitHub/Repository: Look at their active projects. Last Year's Projects: Can you understand what last year's GSoC students worked on? Do they feel exciting to you? Filter by Interest and Tech Stack: As you go through them, you'll filter out most of them. Keep an organisation if: The product/problem genuinely interests you. The tech stack they use aligns with your existing knowledge or something you want to learn. You can understand the goal of their projects. The First Shortlist: This method should narrow down the list to about 6–15 organisations. Most Important: Go through GSoC’s official guidelines for contributors/applicants/students: https://google.github.io/gsocguides/student/. It will be boring, but read all sections thoroughly—it’s a gem that will help you in every part of your GSoC journey. From Shortlist to Selection: The Contribution Phase This is where the rubber meets the road. Your final selection will come from doing the work. 1. Get Your Hands Dirty Join the Community: Find their communication servers (Slack, Discord, mailing list), go through recent chats and channels to get some context (and to avoid sending a message to the wrong channel in future). Don’t introduce yourself now. You should present yourself with purpose. Instead, first go through the issues of the repos/projects you shortlisted earlier for this organisation, and try to find any issue that you can solve and then ask in the issue discussion if you can work on it. Target the Easy Wins: Look for "good first issues," "beginner issues," or similar tags in their repositories. Don't be afraid to start small—the goal is to see how the codebase works and make your presence felt to the project maintainers (your potential mentors). Contribute Aggressively (and Smartly): Start solving these issues and submitting Pull Requests (PRs). You will quickly find that you cannot understand the codebases or relate to the work of a few organisations. You can drop those organisations. Go for the Jackpot: Now, try to tackle bigger issues or stale issues. Stale issues are often large and were left midway by previous contributors. If you can pick the most important one and fix it, you will earn a +10 aura point. The Grand Introduction: This is the time to introduce yourself to the community: No need to write a large paragraph about your background. Instead, discuss the complex issues you are trying to solve, or ask for a review of your Pull Request. This presents you with a proof of work, giving weight to your words and earning you +100 aura points. The Final Shortlist: By the end of this process (ideally by early December/late January at the latest), you will be left with 1-2 organisations where you feel comfortable, can understand the work, and can get PRs merged. If you feel that any org’s communication server is not that active or you can filter them out. The remaining 1-2 orgs are where you will apply for GSoC. 2. Communicate and Research Connect with Mentors: Pay attention to which project maintainers are friendly and active. Attend the organisation's community meetings and participate actively in discussions, asking doubts if you have them. Search First, Ask Later: Before asking anyone a technical question, search Google/YouTube first. Mentors highly prefer students who demonstrate a clear effort to solve problems independently before asking for help. Focus on Learning: Your goal right now shouldn't be "GSoC selection." Your primary goal should be to learn the tech and concepts the organization uses. The GSoC selection will follow naturally. The Proposal: Your Closing Argument Your success depends on two things: your pre-application contribution and your proposal. Most of the mentors look for two things while selecting their project mentee: 1) How much they have contributed before (in terms of time and PRs), and 2) How well researched and in-depth their proposal is. Proposal Research: Since you’ve been contributing for months, you’ll have an internal understanding of the organisation's needs and future plans. Use this knowledge to write a well-researched and detailed proposal. Make sure to add references from documentation or external sources wherever needed. And don’t use AI to make your proposal, we can smell that from miles away, by reading the first few paragraphs and reject the proposal at that point (You can use AI to research about your proposal or fix grammatical errors, but don’t use it to generate a whole proposal or sections of it). Get Reviewed: Treat your proposal like a professional document. Have the project mentors review it at least 2–3 times before the final submission. They can tell you exactly what they want to see. Most Important: Go through GSoC’s official guideline for contributors/applicants/students: https://google.github.io/gsocguides/student/ Bottom line: Start now, filter methodically, contribute aggressively, be active in the community, and write a well-researched, in-depth proposal (that you wrote yourself, not by AI).
Wrapping Things Up | GSoC'24@AsyncAPI
Google Summer of Code 2024 is coming to a close, and it’s time to wrap up my contributions to the AsyncAPI Website. This summer, I worked on developing a comprehensive UI Kit for the AsyncAPI Website, focusing on ensuring visual consistency, reducing...
Wrapping Things Up | GSoC'24@AsyncAPI
Google Summer of Code 2024 is coming to a close, and it’s time to wrap up my contributions to the AsyncAPI Website. This summer, I worked on developing a comprehensive UI Kit for the AsyncAPI Website, focusing on ensuring visual consistency, reducing code duplication, and improving the overall maintainability of the website. The UI Kit will serve as a foundation for the website's design, making it easier for new and existing contributors to reuse, develop, and maintain UI components. Given the importance of visual consistency and the growing number of components on the website, having a well-defined and well-structured UI Kit is essential. My task involved developing this UI Kit and integrating it into the AsyncAPI Website, creating a well-organized Storybook for the website. For more information, you can refer to the GitHub issue: https://github.com/asyncapi/website/issues/2090 Current State The UI Kit has been successfully developed and integrated into the AsyncAPI Website. The components have been documented and organized in Storybook, with code formatting and linting tools in place to maintain consistency. However, the Dropbox component is under development and the deployment of the project is not yet completed. Contributions Repository: AsyncAPI Initiative Website GitHub Repository GSoC Contributions: GSoC Contribution PRs for UI Kit Development Project PR LinkDescriptionStatus #3015Setting up storybook & chromatic and created typography storiesMerged #3078Added stories for icons, logos, and colorsMerged #3079Added stories for buttonsMerged #3081Created tags & avatar components and added stories for themMerged #3082Created toggle & checkbox components and added stories for themMerged #3093Created input box component and added stories for itMerged #3095Upgraded storybook from 8.1.11 to 8.2.4Merged #3121Created an accordion component, added stories for it, and used it in the FAQ section.Merged #3122Added stories for blog cardMerged #3123Created loader component and added stories for itMerged #3152Configured storybook themeMerged #3174Updated filters dropdown and created stories for itOpen What's Left to Do The development of the Dropbox component is under-process and the deployment of the UI Kit is pending. These tasks will be completed in the upcoming weeks. Conclusion Working on the AsyncAPI Website UI Kit was a challenging yet rewarding experience. The project provided valuable insights into building and maintaining design systems, automating documentation, and ensuring code quality. Special thanks to my mentors, Akshat Nema, Azeez Elegbede (Ace), and Aishat Muibudeen (Maya) for their continuous guidance and support. Their feedback helped me improve my coding skills and approach problems with a clearer perspective. I would also like to thank the entire AsyncAPI Community for their encouragement and collaboration throughout the project. I look forward to continuing my contributions to AsyncAPI and am excited to see how the UI Kit will benefit the community in the future. That's it for now—concluding with a sense of accomplishment and readiness for the next challenge! :)
Migration of AsyncAPI Website to Next.js v14 and TypeScript
We're thrilled to announce the successful migration of the AsyncAPI website from JavaScript and Next.js v12 to TypeScript and Next.js v14! This exciting upgrade unlocks a new chapter for the website, paving the way for improved scalability, streamlined feature implementation, and the powerful capabilities of Next.js. As a bonus, this migration also enabled a well-documented codebase and streamlined our testing process by reducing the reliance on Cypress tests. In this blog post, we'll delve into the exciting journey behind the migration and share what's new on the website. I'll share insights into our team's efforts, the research and planning involved, the challenges we tackled, the valuable lessons learned, and what exciting plans lie ahead for the AsyncAPI website. This migration journey spanned from February to May 2024, and it involved a dedicated team of contributors and maintainers passionate about the AsyncAPI Initiative and the ecosystem of tools it provides. We'd like to give a shout-out to our amazing team: Lukasz Gornicki, Rohit T, Akshat Nema, Ansh Goyal, Ashish Padhy, Sambhav Gupta, Vishvamsinh Vaghela, and myself, Ashmit JaiSarita Gupta. Their expertise and commitment were instrumental in achieving this exciting upgrade for the AsyncAPI website. Note: This blog was originally published on the AsyncAPI Initiative website. (Visit) What's new on the website? The migration to Next.js v14 and TypeScript brought several significant transformations to the AsyncAPI website. These changes paved the way for a more performant, scalable, and developer-friendly experience. Streamlined Development Workflow: TypeScript integration introduces static typing to the codebase, enhancing code maintainability, reducing errors, and providing better autocompletion. This simplifies the development process and promotes cleaner code. Component Refactoring: Much of the migration involved meticulously refactoring website components within the Next.js framework. This ensures optimal performance and lays the groundwork for future feature development. Improved Testability: Moving to TypeScript also enhances our testing capabilities. Static type checking helps identify potential issues early in the development cycle, leading to a more robust and reliable website. Improved Code Clarity with JSDoc: For developers diving deeper into the codebase, we've added comprehensive JSDoc documentation for all components and their parameters. This documentation clearly explains each component's purpose and function, along with detailed information about the parameters it accepts. This enhanced clarity simplifies understanding of the website's code structure and functionality, making it easier for developers to contribute. This will also help generate proper docs for UI components in the UI Kit, which I am currently developing as a part of my Google Summer of Code project. Enhanced Documentation and Blog Structure: We've introduced a new directory structure to streamline content management and leverage Next.js capabilities. Previously housed in the pages/docs and pages/blog directories, our documentation and blog now reside in the markdown/docs and markdown/blog directories, respectively. This change allows for better organization and integration with Next.js's built-in features for handling static content. Consistent Code Formatting: We've implemented well-defined Prettier and ESLint rules to enforce consistent code formatting and style across the entire codebase. This not only improves code readability and maintainability but also simplifies collaboration among developers. Improved Static Data Management: Static data that was previously hardcoded directly within components and pages is now housed in dedicated data folders. This separation of concerns promotes cleaner code, simplifies maintenance, and facilitates data reusability across the website. Refined Configuration Management: The config folder now strictly stores configuration-related data. Static data previously stored within the configuration folder has been relocated to the data folders. Choosing the Right Framework for the AsyncAPI Website A crucial step in our migration journey was selecting the most suitable framework. Given the current website setup and limited use of server-side rendering, we initially considered React with Vite.js. Vite offers advantages like faster development, improved performance, and a user-friendly development server. However, browser support and a less mature plugin ecosystem presented potential challenges. Next.js emerged as the preferred choice due to its focus on scalability, SEO, and developer experience. Features like server-side rendering (SSR), static site generation (SSG), and automatic routing contribute to these benefits. Additionally, Next.js offers built-in functionalities for image, font, and script optimization, streamlining the development process. While Next.js has a steeper learning curve, its comprehensive feature set and strong community support ultimately aligned best with our vision for the AsyncAPI website's future growth and feature expansion. Planning the Migration: A Stepwise Approach To manage the complexity of migrating the AsyncAPI website's codebase, which included various directories like components, pages, scripts, and configurations, we devised a structured plan. This plan involved dividing the codebase into manageable subdirectories, focusing heavily on dependencies between them. Key areas like contexts, utility functions (lib), and Netlify serverless functions required specific attention during the migration process. It's important to note that existing Node.js scripts used for build tasks fell outside the scope of the migration and would be covered in the upcoming GSoC 2024 project: Script Stability Enhancement for AsyncAPI Website. We also updated configurations related to tools like Git, Docker, and Prettier as needed within the TypeScript environment. Our migration strategy involved a meticulously planned, five-phase approach. Each phase tackled crucial aspects of the website's modernization: Basic Setup: This initial phase focused on establishing the new project's infrastructure. We configured a fresh package.json file, managing all dependencies. We set up the Next.js application, integrated Google Analytics and SEO functionalities, and made a production-ready build for deployment on Netlify. Setup Context: The second phase involved establishing contexts for elements like blogs, docs, and the tool filter system. Migrating Components: This was the longest migration phase in which we migrated over 250 components compromising icons, navigation, layout, buttons, dashboard, typography, tools, etc. Additionally, this phase involved the strategic elimination of any redundant components and optimizing the website's overall structure. It took us around the whole of March and April to complete this phase. Migrating Pages: In this phase, we migrated the pages of our website. Since all the components were migrated, this step was easy and quick for us. Final Touches and Launch: In the final step, we migrated Netlify functions, updated the readme, conducted a manual comparison between the old and new websites to address any UI discrepancies, resolved bugs, and took feedback from the community. Finally, we integrated the migrated website onto the main branch and deployed it to production. Challenges we faced No significant project is without its hurdles, and our migration journey was no exception. Here are some of the challenges that we faced: Challenge: How do you manage PRs that will be opened during the migration? Solution: We tried to merge all PRs to the migrated website instead of the production website. However, the urgent changes were merged into the master branch and then pulled into the migrate-ts branch. Challenge: Handling the redundancy of the same type of specifications in multiple PRs and files. Solution: We decided to have a common types folder and defined all the types inside it that were being used at multiple places. Challenge: Multiple structures of the TypeScirpt components by different team members. Solution: We decided and created a coding guideline to be followed by all team members. Challenge: Encountering usage of a lowlight package that wasn’t installed as a dependency. More interestingly, there's nothing like lowlight.registerLanguage as per their API docs, which were on the old website. Solution: This was the funniest part; the old codebase used lowlight.registerLanguage, which was not provided by the lowlight API. We noticed the codebase worked fine without lowlight, so we removed every usage. Challenge: Error in rendering custom JSX components inside .md files. Solution: This was the most annoying error that we faced, and it remained unresolved for several weeks. Finally, we all assembled in a huddle call to fight with this and got the solution. It required to be converted into mdx file with proper format. We used prettier and eslint rules to get the proper format after converting all .md files. What’s next? The migration of the AsyncAPI website to Next.js and TypeScript before the start of the Google Summer of Code 2024 coding phase marks a significant step forward. This upgrade unlocks a new era of scalability, streamlined feature implementation, and enhanced developer experience. We're incredibly proud of the collaborative effort that brought this project to fruition, and we extend a heartfelt thank you to our dedicated team and the invaluable community feedback. As we move forward, we're excited to witness the contributions of our Google Summer of Code mentees and the exciting new features they'll bring to life. Vishvamsinh Vaghela will be working on the script stability enhancement of the website and I will be developing a UI kit for the AsyncAPI website using the Storybook and Chromatic. These advancements will streamline future development and elevate the developer experience of the AsyncAPI website. We invite you to explore the revamped website and share your feedback! Your continued support inspires us to continuously improve the AsyncAPI ecosystem. Stay tuned for further updates on our progress!
Migration of AsyncAPI Website to Next.js v14 and TypeScript
We're thrilled to announce the successful migration of the AsyncAPI website from JavaScript and Next.js v12 to TypeScript and Next.js v14! This exciting upgrade unlocks a new chapter for the website, paving the way for improved scalability, streamlin...
Integrating Storybook and Chromatic to AsyncAPI Website
AsyncAPI Website UI Kit Development
Integrating Storybook and Chromatic to AsyncAPI Website
The AsyncAPI website is built using Next.js v14 and TypeScript. Next.js, a powerful framework that leverages React's capabilities, allows us to create feature-rich web applications. In this blog post, I will delve into setting up Storybook v8, a frontend workshop for building UI components and pages in isolation, and Chromatic, a visual testing & review tool that scans every possible UI state across browsers to catch visual and functional bugs, in the AsyncAPI website. I will also be discussing the challenges I encountered and sharing the valuable lessons I learned. Installing Storybook To initialize the storybook into an existing Next.js project, Storybook Documentation suggests to use the following command: # Add Storybook: $ npx storybook@latest init However, I encountered a snag while running this command inside my existing clone of the website repo in my WSL environment. The installation process seemed to stall indefinitely. I attempted various solutions, including switching to a Windows environment and verifying network connectivity, but none proved successful. The issue mysteriously resolved after I performed a fresh clone of the repository. While the cause remains unclear, a clean clone might be the solution if you encounter a similar installation freeze during Storybook setup. Exploring the boilerplate code The storybook installation adds the following folders to our project: .storybook and stories. The .storybook folder has the following files: main.ts: This is the storybook configuration file and has the following default code: import type { StorybookConfig } from "@storybook/nextjs"; const config: StorybookConfig = { stories: [ "../stories/**/*.mdx", "../stories/**/*.stories.@(js|jsx|mjs|ts|tsx)", ], addons: [ "@storybook/addon-onboarding", "@storybook/addon-links", "@storybook/addon-essentials", "@chromatic-com/storybook", "@storybook/addon-interactions", ], framework: { name: "@storybook/nextjs", options: {}, }, staticDirs: ["..\\public"], }; export default config; preview.ts: This configuration file allows users to control how the story renders in the UI and has the following default code: import type { Preview } from "@storybook/react"; const preview: Preview = { parameters: { controls: { matchers: { color: /(background|color)$/i, date: /Date$/i, }, }, }, }; export default preview; the stories folder contains sample boilerplate components and stories like Button. Here is the list of files it has: We won't be needing this boilerplate code so I removed it. Changing the default story location By default, Storybook stories are present in the stories folder. The main.ts file inside the .storybook folder has settings to find story files inside the stories folder: stories: [ "../stories/**/*.mdx", "../stories/**/*.stories.@(js|jsx|mjs|ts|tsx)", ], However, codes that change for the same reasons should be kept together. In that sense, the Storybook file for a given component will very likely change when that component changes — so we will be keeping the Storybook File with its component. Also, if the component folder is moved to another place in the project or even to another project, it will be easier to move the Storybook file along. So we update the above storybook configuration to look for stories in the components folder: stories: [ "../components/**/*.stories.mdx", "../components/**/*.stories.@(js|jsx|mjs|ts|tsx)", ], Setting up autodocs and custom documentation template Storybook Autodocs is a powerful tool that can quickly generate comprehensive documentation for UI components. By leveraging Autodocs, we can transform our stories into living documentation. To enable automatic documentation for all stories, I added it to tags in our .storybook/preview.ts file: const preview: Preview = { // ...rest of preview //👇 Enables auto-generated documentation for all stories tags: ['autodocs'], }; To enhance the documentation structure, I added a custom documentation template to our Storybook. To replace the default documentation template used by Storybook, I extended the UI configuration file (i.e., .storybook/preview.ts) and introduce a docs parameter. This parameter accepts a page function that returns a React component, which we can use to generate the required template. import { Title, Subtitle, Description, Primary, Controls, Stories } from '@storybook/blocks'; const preview: Preview = { parameters: { // ...rest of parameters docs: { toc: { title: 'Table of contents', }, page: () => ( <> <Title /> <Subtitle /> <Description /> <Primary /> <Controls /> <Stories /> </> ), }, }, }; This toc property of docs parameter adds a Table of Contents section on the right side of each documentation page. Changing the font used Storybook The AsyncAPI Initiative uses Work Sans font overall on its website. So, I decided to update the font used by Storybook. However, this was a tricky part. We can't change the font used by the storybook through its configuration file. After a little research, I found that we can use .storybook/preview-head.html file to add extra elements to the head of the preview iframe, for instance, to load static stylesheets, font files, or similar. When preview-head.html file is defined in our .storybook directory, it is then treated as the head element of our Storybook. This gives us a place to import our font from Google Fonts CDN and also define a style tag where we can apply the font to the body element. Here is how my preview-head.html file looks like this: <link rel="preconnect" href="https://fonts.googleapis.com"> <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin> <link href="https://fonts.googleapis.com/css2?family=Work+Sans:ital,wght@0,100..900;1,100..900&display=swap" rel="stylesheet"> <style> body { font-family: 'Work Sans', sans-serif; font-optical-sizing: auto; font-style: normal; } </style> Setting up Chromatic Chromatic is a visual testing & review tool that scans every possible UI state across browsers to catch visual and functional bugs in our project. It provides a unified workspace for designers, product managers, and other stakeholders to share feedback and sign off on UI. We will be publishing our Storybook using Chromatic as it also allows us to configure CI to run your visual tests whenever you push code and get pull request badges to get notified about the test and review results. Here is how I installed Chromatic to our project: $ npx storybook@latest add @chromatic-com/storybook We need to Sign in to Chromatic to create a new project for our repository. Then we need to grab the CHROMATIC_PROJECT_TOKEN from the project settings. Now whenever we need to publish our storybook we will need to run the following command: $ npx chromatic --project-token=<CHROMATIC_PROJECT_TOKEN> Integrating Chromatic into our CI pipeline We can configure CI to run visual tests whenever we push code and get pull request badges to get notified about the test and review results. We need to add a chromatic script to the package.json to run chromatic: "scripts": { "chromatic": "chromatic --exit-zero-on-changes" } To automate Chromatic with GitHub Actions, I created a new file called chromatic.yml in the .github/workflows directory and added the following: # .github/workflows/chromatic.yml name: "Chromatic" on: push jobs: chromatic: name: Run Chromatic runs-on: ubuntu-latest steps: - name: Checkout code uses: actions/checkout@v4 with: fetch-depth: 0 - uses: actions/setup-node@v4 with: node-version: 20 - name: Install dependencies # ⚠️ See your package manager's documentation for the correct command to install dependencies in a CI environment. run: npm ci - name: Run Chromatic uses: chromaui/action@latest with: # ⚠️ Make sure to configure a `CHROMATIC_PROJECT_TOKEN` repository secret projectToken: ${{ secrets.CHROMATIC_PROJECT_TOKEN }} We need to set the CHROMATIC_PROJECT_TOKEN environment variable (sometimes referred to as “secrets”) in our repository's GitHub actions. Conclusion The above setup and configurations set a base to add stories. That's it! Now we are ready to write, share, and review stories visually with Storybook and Chromatic for all components in our AsyncAPI Website. In the next blog, I will be delving into writing our first stories and will discuss the challenges I encountered, and share the valuable lessons I learned. Till then make sure to test your codebase with multiple edge cases before pushing it to the main branch.