Argo Events decoupled ingestion from dbt

Ingestion and dbt are two jobs with two owners; they must fail independently. We introduced Argo Events so a completion, not a shared Workflow DAG, is the seam. This post is that split: not the later informer-cache OOM, not the second Trino, and not Argo CD.

Two jobs, a completion between them

Three objects, and only one of them knows both sides.

• Producer. A CronWorkflow that runs Spark ingestion on a schedule. It does not name dbt.
• Watch. A Resource EventSource. A Kubernetes informer watches ingestion Workflow CRs reach Succeeded.
• Trigger. A Sensor. It is the only object that says "when that completes, submit this." It submits the dbt Workflow.

This sits after Argo Workflows and Spark as the writer. Writes already ran as Workflows. Nothing yet chained a second job on completion.

Figure 1. Resource EventSource watches ingestion Succeeded; Sensor submits dbt.

Figure 1 is the control plane in namespace argo, in two layers. Blue boxes are Workflows: the CronWorkflow that runs ingestion, and the dbt Workflow the Sensor submits. Sky dashed boxes are Events: the Resource EventSource and the Sensor. Every arrow is control. There is no data path on this map. Parquet and Nessie stay in the writer post. The ingestion template does not name dbt. The dbt template does not name the CronWorkflow.

Why Events, not a bigger Workflow DAG

Argo Workflows already expresses depends. We could have put Spark ingestion and dbt in one DAG: ingest, then dbt, one template, one CronWorkflow. That is the graph you draw on a whiteboard.

It couples the wrong things. The people who own the CronWorkflow are not the people who own the dbt project. Backfills become another branch in the same file.

Events makes the seam a completion. Argo Events wires "this workflow succeeded" to "start that workflow." A failed ingestion does not start dbt; that retry stays on the CronWorkflow. The two jobs page different owners. They backfill independently.

We did not add a broker when we introduced Events. Resource EventSources already watch Kubernetes objects. A queue, a bridge, and hooks on every template would have been a second product. We had two jobs and a completion. CRs were enough.

The cost is another controller in argo, and informers that cache the objects they watch. That cost was acceptable the day we split the graph. Bounding the watch set is a later story.

The chain is a watch, not a hook

Argo Workflows and Argo Events both live in argo, next to CronWorkflows and Sensors. Helmfile applies them from CI; we do not use Argo CD. Cluster-app GitOps and data-job scheduling are different failure domains.

The Resource EventSource is a Kubernetes watch. It lists Workflow CRs. When an ingestion Workflow reaches Succeeded, it emits. A Sensor's trigger submits the dbt Workflow. That is the whole chain in Figure 1.

dbt is not a Helm release. It runs as a container in the Workflow the Sensor submitted, compiles SQL against Trino, and leaves. It is a compiler, not cluster infrastructure. Putting that compiler on the same coordinator as humans is what dbt forced a second Trino: after this split, not instead of it.

Events is the stock argo-helm chart 2.4.15 (appVersion v1.9.6). Workflows is 0.45.15 (appVersion v3.6.7), with one template patch: the server readiness probe is gated and off in the overlay. Events templates are unmodified.

Keep the producer boring

The temptation, once Events exists, is to make every CronWorkflow Events-aware: annotations, sidecars, a producer client, a payload schema. We did not. Producers stay a CronWorkflow, a label, a Sensor. The CronWorkflow schedules. The label names the job for humans. The Sensor is the only object that says "when that completes, submit this."

Chaining lives in the Sensor. The ingestion template does not grow an onExit that knows about dbt. An onExit handler is still the same Workflow object, one owner, one retry policy. Events is a second Workflow. That is the point of the split.

TTL is not a watch filter

Workflows archive to MySQL so history survives CR deletion. Logs go to the cluster log stack, not Argo archiveLogs. CronWorkflows get count-based history limits; those limits belong on the CronWorkflow, not on the WorkflowTemplate. The controller ConfigMap sets a default TTL on every new Workflow:

controller:
  workflowDefaults:
    spec:
      ttlStrategy:
        secondsAfterSuccess: 604800
        secondsAfterFailure: 2592000

Seven days after success, thirty after failure. workflowDefaults apply at creation. Existing CRs do not inherit them. TTL and history limits bound how long a finished Workflow stays in etcd. They are not a substitute for telling the EventSource which CRs to watch.

A Resource EventSource watches Workflow CRs. Kubernetes informers cache the objects they watch. There is no keep-last-N. At introduction we asked for Workflows. We had not bounded that watch set with a label selector. The producer already had a label; the EventSource was not yet filtering on it. Memory still grew with CR count. The numbers, the OOMKills, and the filters that actually held are EventSources OOM on Workflow CR count.

The temptation the first time that cache hurts is to replace Resource EventSources with a message bus. We did not add a broker at introduction, and we would not start there. Fix the watch first. AMQP as an EventSource is a later rejection, not an origin story.

When this applies

Copy this if you already run Argo Workflows on Helmfile, ingestion and dbt should fail independently, and you can stand a Sensor instead of a mega-DAG. Skip it if the jobs are one graph and will stay one graph, if Argo CD is how you install charts, or if you think you need a broker before you have two templates.

If you copy it, copy the boring producer: CronWorkflow, label, Sensor. Archive so you can delete CRs. Put TTL on workflowDefaults and history limits on the CronWorkflow. Do not fork Events. Then bound the EventSource watch. Do not wait for the informer to teach you how large the namespace got. dbt as a second Trino client is the era after this one.