EventSources OOM on Workflow CR count

A Kubernetes informer lists objects, then watches them, and stores every object that matches. There is no LRU. There is no "keep last N." At ~1,400 Workflows, EventSources at 512 Mi were OOMKilled; at ~2,000, each Workflow CR still carried its node graph and the controller at 2 GiB joined them. Label filters and history limits bounded the cache. A message bus did not. This post is that later failure, not why Events was introduced.

The cache is the watch set

The failure is in the control plane, not in the warehouse.

• Resource EventSource. Watches Workflow CRs via a client-go informer.
• Informer cache. Every matching object stays in memory for the life of the watch.
• Count is the bound. Raising the EventSource limit buys time. The cache still grows with CR count.

This sits after Argo Events decoupled ingestion from dbt. Events had been the decoupling layer long enough for Workflow CR count to blow the caches.

Figure 1. EventSource informer cache of Workflow CRs. S3 is out of scope.

Figure 1 keeps S3 grey on purpose. The sky dashed box is the Workflow CR pile in argo. Kubernetes informers have no keep-last-N, so count is the memory bound. Vermillion is the EventSource informer, which caches every matching CR and died at ~1,400 / 512 Mi. The same pile, later, hit the workflow controller with nodeStatusOffLoad: false at ~2,000 / 2 GiB. The sky pills are the bounds that held: filter.labels, CronWorkflow history limits, offload, archive. There is no warehouse hop on this map.

What ~1,400 at 512 Mi actually was

That number was the watch set outrunning the pod, not a leak in the EventSource process.

The controller joined later for the same reason, with a worse per-object size. nodeStatusOffLoad was false, so each Workflow CR still carried its node graph. At ~2,000 CRs the controller at 2 GiB went CrashLoopBackOff. Same unbounded set, fatter objects.

Why not a broker

AMQP EventSources are O(1) in workflow count. They also mean a broker, a bridge, and hooks on every template. The temptation, the first time an EventSource OOMs, is to stop watching Kubernetes and start publishing from the workflow. We did not need that once the informer stopped caching the whole namespace.

Fix the watch first. A message bus is a second control plane, not a substitute for labelSelector.

What bounded it

Four changes, without replacing Resource EventSources.

Label filters on the EventSource. filter.labels with workflows.argoproj.io/cron-workflow is the CronWorkflow name whose completions that EventSource should see. In Argo Events 1.9.x that list becomes a server-side labelSelector on the informer's ListOptions. Field filters and prefix filters do not: they run after the object is already in the cache. Memory per EventSource dropped to tens of megabytes.

INGESTION_CRON is the CronWorkflow whose child Workflows this EventSource is allowed to watch. The rest of the resource spec is unchanged.

# EventSource spec.resource.<EVENT_NAME>.filter; remainder of the watch unchanged
filter:
  labels:
    - key: workflows.argoproj.io/cron-workflow
      operation: "=="
      value: INGESTION_CRON

CronWorkflow history limits so the controller's watch set is bounded. Hourly pipelines kept ten successes and three failures. Weekly jobs can keep a tighter window.

apiVersion: argoproj.io/v1alpha1
kind: CronWorkflow
metadata:
  name: INGESTION_CRON
  namespace: argo
spec:
  successfulJobsHistoryLimit: 10
  failedJobsHistoryLimit: 3
  # schedule and workflowSpec omitted; this snippet is the history bound only

nodeStatusOffLoad: true so node graphs live in the archive database, not in every CR.

Archive enabled, so deleting CRs does not delete history. Workflows already archived to MySQL; logs stay in the cluster log stack, not Argo archiveLogs.

controller:
  persistence:
    archive: true
    nodeStatusOffLoad: true
    # mysql connection omitted; already configured

Argo Events stayed stock argo-helm 2.4.15 (appVersion v1.9.6). The cache bound lives in EventSource filters and CronWorkflow fields, not in a forked Events chart.

History limits belong on the CronWorkflow

Count-based history belongs on the CronWorkflow, not on the WorkflowTemplate. The template is reused; the CronWorkflow is the producer whose children pile up.

The controller ConfigMap also carries a default TTL. Those seconds apply at creation. Existing CRs do not inherit them. Setting this after the pile-up does not shrink the 1,400 objects already in the informer.

controller:
  workflowDefaults:
    spec:
      ttlStrategy:
        secondsAfterSuccess: 604800    # 7 days
        secondsAfterFailure: 2592000   # 30 days

TTL is time-based GC for Workflows that received the default at creation. History limits are the count cap on the CronWorkflow that creates them. Both are required. A ConfigMap default is not a substitute for successfulJobsHistoryLimit on a busy hourly CronWorkflow, and a TTL added later does not reap CRs that already exist.

When this applies

Copy this if Resource EventSources watch a kind whose count grows (Workflows, or anything else an informer will cache in full) and the EventSource pods are sized for a handful of objects, not for the namespace. Skip it if the watch is already a server-side labelSelector and CronWorkflow history is already capped, or if you already run a broker for reasons that are not this OOM.

The seam to copy is a bounded CR count: filter.labels so the informer never sees the rest of the namespace, history limits on the CronWorkflow (not the template), nodeStatusOffLoad so node graphs leave the CR, archive so you can delete CRs without deleting history. Raising memory limits is how you buy the afternoon to do that. It is not the bound.