--- type: "article" title: "Provisioning Infrastructure with Terraform from AsyncAPI" summary: "Provisioning Infrastructure with Terraform from AsyncAPI\nWhen your AsyncAPI says one thing and your Kafka platform does another, people stop trusting the API. In Arcadia the AsyncAPI file stays the source of truth for the platform too: we generate the Terraform for topics, schemas and ACLs from it, merging to develop provisions the development cluster, and each release packs that Terraform so the same files are promoted to pre and prod.\n\nWhen we talk about API First we usually think about the code. We generate models and clients from the API, we validate messages against it, and we test that producers and consumers do what the spec says.\nBut in an event driven architecture, part of the API lives in the broker: the topics, their partitions and retention, the schema compatibility, and the ACLs that say who can read and who can write.\n\nThis is not only for the platform. The producer and the consumer need this information too, to design and configure their applications:\nThree partitions or one, it’s not the same, because with three you can run three consumers in parallel and with one you can’t.\nOne week of retention or one year, it’s not the same, because it decides how far back a consumer can replay.\nA compacted topic or a normal one, it’s not the same either.\nAnd schema compatibility even decides who deploys first: with FORWARD the producer can go first, and with BACKWARD the consumers have to go first.\nSo everybody needs this information, and the natural place for it is the AsyncAPI.\nNow imagine the AsyncAPI says seven days, but someone changed it to one day in the Confluent console. Or the AsyncAPI says nothing at all, because the retention lives in a Terraform file in another repository. People stop reading the API and start asking someone, and from that moment nobody trusts the API.\nCreating Kafka topics is easy, there are scripts, the Confluent CLI, JulieOps, Jikkou or Terraform for that. What goes wrong is that we write the same information twice, once in AsyncAPI and once in the provisioning tool, and then we hope both stay in sync forever.\nIn Arcadia we write it only once, in the AsyncAPI file. Every service that talks Kafka already has one, with the topics it owns, their partitions, the Avro schemas of each message, and the topics it reads from other teams.\nWith the ZenWave AsyncAPI Ops plugin we generate the Terraform from that file, in the same way we generate code, docs or the Event Catalog. Nobody edits that Terraform by hand, so if you want to change something in the platform, you change the AsyncAPI first.\nWhat the generator creates\nThe plugin reads one or more AsyncAPI files and writes a folder with Terraform files:\ntopics.tf with the Kafka topics that the service owns, plus their retry and dead letter topics.\nschemas.tf with the Avro subjects for the Schema Registry.\nacls.tf with the read and write permissions, which come from the send and receive operations.\nThe service account for the service, when you use serviceAccountMode=managed.\nThe Avro schemas, bundled so Terraform can upload them.\nIt knows which topics belong to the service because they are declared inline with an address. Channels that come with a $ref from another team’s API are dependencies, so for those it only creates the ACLs to read them, and never the topic. This is why in Arcadia we pass both files together, asyncapi.yml with what the service publishes and asyncapi-client.yml with what it consumes.\nThis is how a channel looks in orders-checkout-api, and the topic size comes from a shared trait that every team can reuse:\nchannels:\n order-created-event-v1:\n address: orders-checkout.order-created.event.avro.v1\n x-visibility: global\n messages:\n OrderCreatedEvent:\n $ref: \"#/components/messages/OrderCreatedEvent\"\n bindings:\n kafka:\n partitions: 3\n replicas: 3\n topicConfiguration:\n cleanup.policy: [ \"delete\" ]\n retention.ms: 604800000\n retention.bytes: 1000000000\n delete.retention.ms: 86400000\n max.message.bytes: 1048588\n x-env-server-overrides:\n develop:\n partitions: 1\n replicas: 1\n topicConfiguration:\n cleanup.policy: [ \"delete\" ]\n retention.ms: 604800\nAnd this is the command that the pipeline runs, more or less:\njbang zw -p AsyncAPIOpsGeneratorPlugin \\\n apiFiles=asyncapi.yml,asyncapi-client.yml \\\n templates=TerraformConfluent \\\n serviceAccountMode=managed \\\n server=develop \\\n targetFolder=target/terraform\nThe server option chooses which entry of x-env-server-overrides the generator applies on top of the Kafka bindings. This is the kafkaTopicS trait from api-contract-commons:\nkafkaTopicS:\n bindings:\n kafka:\n partitions: 3\n replicas: 3\n x-env-server-overrides:\n dev:\n partitions: 1\n replicas: 1\n staging:\n partitions: 3\n replicas: 2\nNOTE: Channel traits are available on AsyncAPI v3.2.0 and later\nSo the same topic gets one partition with server=dev, and three partitions with server=staging or with any other server, like production, that has no override. For Arcadia we use Confluent Cloud, but there are also templates for plain Kafka with an open source Schema Registry.\nCluster IDs, endpoints and credentials never go inside the AsyncAPI file. The pipeline injects them from GitHub secrets and variables, and a small shared overlay in terraform/common adds the provider configuration and some safe defaults.\nTwo different rhythms\nOnce we can generate Terraform, we still need to decide when to apply it and where, and development and production don’t move at the same speed. In development you want to see your new topic as soon as you merge, because you are going to test against it this afternoon. But in production you want to apply only what was released, and only when somebody decides it is time.\nSo in api-product-workflows we have one path for develop and another one for releases:\n\nWhenWhat happensWorkspace\n\nYou push to any branch or open a PRGenerate and terraform plan, nothing is applied-develop\nYou merge to developGenerate, plan and apply-develop\nYou release asyncapi-allGenerate from the release tag, validate, zip and attach to the GitHub releasenone\nYou promote a releaseDownload the zip and apply it-pre, then -prod\n\nEvery service has one Terraform Cloud workspace per environment, and the name is always the repository name plus the environment.\nThe console and pipeline illustrations below use example values to explain the workflow; they are not captures of live infrastructure.\nMerge to develop, and develop is provisioned\nService repositories don’t need a special workflow for this. Their normal artifact-ci.yml calls the shared artifact-ci.yml, the same one that validates the APIs, and this one has two extra jobs for Kafka.\nFirst it looks at what changed in the commit, and if an asyncapi or asyncapi-client artifact changed, or any .avsc file in the repository, it selects all the Kafka contracts of that service as one unit.\nThen the kafka-plan job runs provision-kafka-plan.yml. It asks the architecture manifest which AsyncAPI files belong to this repository, generates the Terraform and makes a plan against the develop workspace. This happens in every branch and every pull request, so a reviewer can see what is going to change before the merge.\nWhen the push is on the develop branch, a second job, kafka-develop, runs provision-kafka-develop.yml. It does the same generation and plan, and then it applies, with no approval:\n# No approval gate: a successful plan against develop applies immediately.\n- name: Terraform apply\n run: terraform apply -auto-approve tfplan\nWe do this on purpose, because develop is where we want fast feedback: you merge your new event into develop, and some minutes later the topic, the schema and the ACLs exist in the development cluster.\n\n\nThe apply also records a Terraform output called provisioned_from with the value develop@, so you can always ask the workspace which commit it was built from.\n\nRelease, and the Terraform is packed\nFor production we don’t regenerate anything at deploy time, because then what you deploy could be different from what you tested.\nWhen a team releases its Kafka contracts with artifact-release.yml and artifact=asyncapi-all, the release job publishes the API as usual and creates the tag release/asyncapi-all/vX. After that, a job called release-kafka calls provision-kafka-release.yml, which:\nChecks out the service at that exact release tag.\nGenerates the Terraform with the prod server profile.\nRuns terraform init -backend=false and terraform validate, offline, without touching any cluster.\nWrites a RELEASE.json with the version and the AsyncAPI files that are inside.\nZips everything as -asyncapi-all-vX-terraform.zip and attaches it to the GitHub release.\nNothing is applied in this step. The Terraform Cloud backend stays as a template, cloud.tftpl, with the workspace name still empty, because the bundle doesn’t know yet where it will be applied, and that is why the same zip can go to pre today and to prod next week, once somebody approves it.\nPromote from environment to environment\nTo apply a release, the service calls provision-kafka-promote.yml by hand:\njobs:\n promote-kafka:\n uses: arcadia-editions/api-product-workflows/.github/workflows/provision-kafka-promote.yml@main\n with:\n service_repo: orders-checkout-api\n target_env: pre # or: prod\n version: \"1.4.0\" # only for pre\n secrets: inherit\nFor pre you choose the version, and the workflow downloads the zip from that release, renders cloud.tf for the -pre workspace, plans, applies, and saves the version in provisioned_from.\nFor prod you cannot choose a version, and this is my favourite part. The workflow reads provisioned_from from the pre workspace and applies exactly that same release in prod, so prod can only receive what was already running in pre and the two environments cannot drift apart by mistake.\nThe pipeline doesn’t approve anything by itself, because approving a release is part of your organization’s process, with its change requests, release boards or whatever your company uses. The pipeline is only the technical part: once the release is approved offline, somebody runs the promotion manually, first to pre and then to prod.\nThere is also a provision-kafka-destroy.yml to remove an environment, and to run it you need to type the exact workspace name, because there is no way back.\nThe AsyncAPI file is the source\nService teams only edit their AsyncAPI file and their Avro schemas, as they already do, and the pipelines take care of develop, of the release package and of the promotion to pre and prod. The platform team, on their side, owns the shared overlay and the credentials, and decides which defaults every topic gets.\nYou can read about all the options of the generator in the AsyncAPI Ops plugin docs, and all the workflows of this article are in arcadia-editions/api-product-workflows.\nOriginally published at ivangsa.com." newsletter: "Event-Driven Architecture Notes by Iván García" newsletter_handle: "ivangsa" newsletter_url: "https://usecommune.com/n/ivangsa" author: "Ivan García Sainz Aja (@ivangsa)" published: "2026-10-20T00:00:00.000Z" canonical_url: "https://usecommune.com/n/ivangsa/a/provisioning-infrastructure-with-terraform-from-asyncapi" markdown_url: "https://usecommune.com/n/ivangsa/a/provisioning-infrastructure-with-terraform-from-asyncapi.md" chat_url: "https://usecommune.com/n/ivangsa/a/provisioning-infrastructure-with-terraform-from-asyncapi/chat" source_url: "https://ivangsa.com/articles/arcadia/013-infrastructure-provisioning-asyncapi-terraform/" body_source: "imported" likes: 1 replies: 0 body_words: 1798 --- # Provisioning Infrastructure with Terraform from AsyncAPI When your AsyncAPI says one thing and your Kafka platform does another, people stop trusting the API. In Arcadia the AsyncAPI file stays the source of truth for the platform too: we generate the Terraform for topics, schemas and ACLs from it, merging to develop provisions the development cluster, and each release packs that Terraform so the same files are promoted to pre and prod. ![AsyncAPI as the source of truth, through GitHub Actions and Terraform plan and apply, to a Kafka platform with topics, schemas and ACLs](https://ivangsa.com/assets/articles/arcadia-editions/013-infrastructure-provisioning-asyncapi-terraform.webp) When we talk about API First we usually think about the code. We generate models and clients from the API, we validate messages against it, and we test that producers and consumers do what the spec says. But in an event driven architecture, part of the API lives in the broker: the topics, their partitions and retention, the schema compatibility, and the ACLs that say who can read and who can write. ![The declared AsyncAPI contract describes both application schemas, headers and operations, and platform topics, partitions, retention and schema evolution](https://ivangsa.com/assets/articles/arcadia-editions/013-declared-api-application-platform.png) This is not only for the platform. The producer and the consumer need this information too, to design and configure their applications: - Three partitions or one, it’s not the same, because with three you can run three consumers in parallel and with one you can’t. - One week of retention or one year, it’s not the same, because it decides how far back a consumer can replay. - A compacted topic or a normal one, it’s not the same either. - And schema compatibility even decides who deploys first: with `FORWARD` the producer can go first, and with `BACKWARD` the consumers have to go first. So everybody needs this information, and the natural place for it is the AsyncAPI. Now imagine the AsyncAPI says seven days, but someone changed it to one day in the Confluent console. Or the AsyncAPI says nothing at all, because the retention lives in a Terraform file in another repository. People stop reading the API and start asking someone, and from that moment nobody trusts the API. Creating Kafka topics is easy, there are scripts, the Confluent CLI, JulieOps, Jikkou or Terraform for that. What goes wrong is that we write the same information twice, once in AsyncAPI and once in the provisioning tool, and then we hope both stay in sync forever. In Arcadia we write it only once, in the AsyncAPI file. Every service that talks Kafka already has one, with the topics it owns, their partitions, the Avro schemas of each message, and the topics it reads from other teams. With the [ZenWave AsyncAPI Ops plugin](https://www.zenwave360.io/zenwave-sdk/plugins/asyncapi-ops/) we generate the Terraform from that file, in the same way we generate code, docs or the [Event Catalog](https://ivangsa.com/articles/arcadia/012-arcadia-architecture-manifest-meets-event-catalog). Nobody edits that Terraform by hand, so if you want to change something in the platform, you change the AsyncAPI first. ## What the generator creates The plugin reads one or more AsyncAPI files and writes a folder with Terraform files: - `topics.tf` with the Kafka topics that the service owns, plus their retry and dead letter topics. - `schemas.tf` with the Avro subjects for the Schema Registry. - `acls.tf` with the read and write permissions, which come from the `send` and `receive` operations. - The service account for the service, when you use `serviceAccountMode=managed`. - The Avro schemas, bundled so Terraform can upload them. It knows which topics belong to the service because they are declared inline with an `address`. Channels that come with a `$ref` from another team’s API are dependencies, so for those it only creates the ACLs to read them, and never the topic. This is why in Arcadia we pass both files together, `asyncapi.yml` with what the service publishes and `asyncapi-client.yml` with what it consumes. This is how a channel looks in [orders-checkout-api](https://github.com/arcadia-editions/orders-checkout-api), and the topic size comes from a shared trait that every team can reuse: ``` channels: order-created-event-v1: address: orders-checkout.order-created.event.avro.v1 x-visibility: global messages: OrderCreatedEvent: $ref: "#/components/messages/OrderCreatedEvent" bindings: kafka: partitions: 3 replicas: 3 topicConfiguration: cleanup.policy: [ "delete" ] retention.ms: 604800000 retention.bytes: 1000000000 delete.retention.ms: 86400000 max.message.bytes: 1048588 x-env-server-overrides: develop: partitions: 1 replicas: 1 topicConfiguration: cleanup.policy: [ "delete" ] retention.ms: 604800 ``` And this is the command that the pipeline runs, more or less: ``` jbang zw -p AsyncAPIOpsGeneratorPlugin \ apiFiles=asyncapi.yml,asyncapi-client.yml \ templates=TerraformConfluent \ serviceAccountMode=managed \ server=develop \ targetFolder=target/terraform ``` The `server` option chooses which entry of `x-env-server-overrides` the generator applies on top of the Kafka bindings. This is the `kafkaTopicS` trait from [api-contract-commons](https://github.com/arcadia-editions/api-contract-commons/blob/main/asyncapi-channel-traits.yml): ``` kafkaTopicS: bindings: kafka: partitions: 3 replicas: 3 x-env-server-overrides: dev: partitions: 1 replicas: 1 staging: partitions: 3 replicas: 2 ``` > NOTE: Channel traits are available on AsyncAPI v3.2.0 and later So the same topic gets one partition with `server=dev`, and three partitions with `server=staging` or with any other server, like production, that has no override. For Arcadia we use Confluent Cloud, but there are also templates for plain Kafka with an open source Schema Registry. Cluster IDs, endpoints and credentials never go inside the AsyncAPI file. The pipeline injects them from GitHub secrets and variables, and a small shared overlay in [`terraform/common`](https://github.com/arcadia-editions/api-product-workflows/tree/main/terraform/common) adds the provider configuration and some safe defaults. ## Two different rhythms Once we can generate Terraform, we still need to decide when to apply it and where, and development and production don’t move at the same speed. In development you want to see your new topic as soon as you merge, because you are going to test against it this afternoon. But in production you want to apply only what was released, and only when somebody decides it is time. So in [api-product-workflows](https://github.com/arcadia-editions/api-product-workflows) we have one path for `develop` and another one for releases: | When | What happens | Workspace | | --- | --- | --- | | You push to any branch or open a PR | Generate and `terraform plan`, nothing is applied | `-develop` | | You merge to `develop` | Generate, plan and apply | `-develop` | | You release `asyncapi-all` | Generate from the release tag, validate, zip and attach to the GitHub release | none | | You promote a release | Download the zip and apply it | `-pre`, then `-prod` | Every service has one Terraform Cloud workspace per environment, and the name is always the repository name plus the environment. The console and pipeline illustrations below use example values to explain the workflow; they are not captures of live infrastructure. ## Merge to develop, and develop is provisioned Service repositories don’t need a special workflow for this. Their normal [`artifact-ci.yml`](https://github.com/arcadia-editions/orders-checkout-api/blob/main/.github/workflows/artifact-ci.yml) calls the shared [`artifact-ci.yml`](https://github.com/arcadia-editions/api-product-workflows/blob/main/.github/workflows/artifact-ci.yml), the same one that validates the APIs, and this one has two extra jobs for Kafka. First it looks at what changed in the commit, and if an `asyncapi` or `asyncapi-client` artifact changed, or any `.avsc` file in the repository, it selects all the Kafka contracts of that service as one unit. Then the `kafka-plan` job runs [`provision-kafka-plan.yml`](https://github.com/arcadia-editions/api-product-workflows/blob/main/.github/workflows/provision-kafka-plan.yml). It asks the [architecture manifest](https://ivangsa.com/articles/arcadia/011-master-yaml-architecture-manifest) which AsyncAPI files belong to this repository, generates the Terraform and makes a plan against the develop workspace. This happens in every branch and every pull request, so a reviewer can see what is going to change before the merge. When the push is on the `develop` branch, a second job, `kafka-develop`, runs [`provision-kafka-develop.yml`](https://github.com/arcadia-editions/api-product-workflows/blob/main/.github/workflows/provision-kafka-develop.yml). It does the same generation and plan, and then it applies, with no approval: ``` # No approval gate: a successful plan against develop applies immediately. - name: Terraform apply run: terraform apply -auto-approve tfplan ``` We do this on purpose, because develop is where we want fast feedback: you merge your new event into `develop`, and some minutes later the topic, the schema and the ACLs exist in the development cluster. ![Topics of orders-checkout in Confluent Cloud](https://ivangsa.com/assets/articles/arcadia-editions/013-confluent-topics.png) ![The OrderCreatedEvent schema in Schema Registry](https://ivangsa.com/assets/articles/arcadia-editions/013-confluent-schema.png) The apply also records a Terraform output called `provisioned_from` with the value `develop@`, so you can always ask the workspace which commit it was built from. ![The latest Terraform state with generated Confluent resources and workspace outputs](https://ivangsa.com/assets/articles/arcadia-editions/013-tfc-develop-outputs.png) ## Release, and the Terraform is packed For production we don’t regenerate anything at deploy time, because then what you deploy could be different from what you tested. When a team releases its Kafka contracts with [`artifact-release.yml`](https://github.com/arcadia-editions/api-product-workflows/blob/main/.github/workflows/artifact-release.yml) and `artifact=asyncapi-all`, the release job publishes the API as usual and creates the tag `release/asyncapi-all/vX`. After that, a job called `release-kafka` calls [`provision-kafka-release.yml`](https://github.com/arcadia-editions/api-product-workflows/blob/main/.github/workflows/provision-kafka-release.yml), which: 1. Checks out the service at that exact release tag. 2. Generates the Terraform with the `prod` server profile. 3. Runs `terraform init -backend=false` and `terraform validate`, offline, without touching any cluster. 4. Writes a `RELEASE.json` with the version and the AsyncAPI files that are inside. 5. Zips everything as `-asyncapi-all-vX-terraform.zip` and attaches it to the GitHub release. Nothing is applied in this step. The Terraform Cloud backend stays as a template, `cloud.tftpl`, with the workspace name still empty, because the bundle doesn’t know yet where it will be applied, and that is why the same zip can go to `pre` today and to `prod` next week, once somebody approves it. ## Promote from environment to environment To apply a release, the service calls [`provision-kafka-promote.yml`](https://github.com/arcadia-editions/api-product-workflows/blob/main/.github/workflows/provision-kafka-promote.yml) by hand: ``` jobs: promote-kafka: uses: arcadia-editions/api-product-workflows/.github/workflows/provision-kafka-promote.yml@main with: service_repo: orders-checkout-api target_env: pre # or: prod version: "1.4.0" # only for pre secrets: inherit ``` For `pre` you choose the version, and the workflow downloads the zip from that release, renders `cloud.tf` for the `-pre` workspace, plans, applies, and saves the version in `provisioned_from`. For `prod` you cannot choose a version, and this is my favourite part. The workflow reads `provisioned_from` from the `pre` workspace and applies exactly that same release in `prod`, so prod can only receive what was already running in pre and the two environments cannot drift apart by mistake. The pipeline doesn’t approve anything by itself, because approving a release is part of your organization’s process, with its change requests, release boards or whatever your company uses. The pipeline is only the technical part: once the release is approved offline, somebody runs the promotion manually, first to `pre` and then to `prod`. There is also a [`provision-kafka-destroy.yml`](https://github.com/arcadia-editions/api-product-workflows/blob/main/.github/workflows/provision-kafka-destroy.yml) to remove an environment, and to run it you need to type the exact workspace name, because there is no way back. ## The AsyncAPI file is the source Service teams only edit their AsyncAPI file and their Avro schemas, as they already do, and the pipelines take care of develop, of the release package and of the promotion to `pre` and `prod`. The platform team, on their side, owns the shared overlay and the credentials, and decides which defaults every topic gets. You can read about all the options of the generator in the [AsyncAPI Ops plugin docs](https://www.zenwave360.io/zenwave-sdk/plugins/asyncapi-ops/), and all the workflows of this article are in [arcadia-editions/api-product-workflows](https://github.com/arcadia-editions/api-product-workflows). *Originally published at [ivangsa.com](https://ivangsa.com/articles/arcadia/013-infrastructure-provisioning-asyncapi-terraform/).* *** ## Discussion No replies yet.