An Iceberg lakehouse on an existing EKS cluster

This is a production Iceberg lakehouse on an EKS cluster that already existed: S3 warehouse, Nessie catalog, Spark writer, two Trino clusters, Argo as the scheduler. AWS objects live in CDK; every pod lives in Helmfile. This post covers that seam and three memory outages (coordinator heap, catalog preemption, EventSource cache) — not a from-scratch EKS install, and not GitOps with Argo CD.

Figure 1. Two layers around existing EKS: CDK for AWS objects, Helmfile for pods.

Figure 1 is the two-layer seam. Grey dashed is the cluster we imported by name. The dashed sky box is CDK: the warehouse bucket, IRSA service accounts, extra node groups. The solid blue box is Helmfile: Nessie, trino-main, trino-etl, Spark, and Argo, in three namespaces. Orange is data: S3 iceberg/ and Aurora MySQL for Nessie JDBC2. Hop (1) is import-by-name. (2) is helmfile -e prod apply. (3) is REST /api/v2 to Nessie. (4) is s3:// on the warehouse prefix. (5) is JDBC2. DataHub, Superset, and the ingress NLB live in later sections; they are not on this map.

Two layers: CDK around EKS vs Helmfile on the cluster

CDK owns AWS: the warehouse bucket, IAM for in-cluster service accounts, extra node groups, Elastic IPs for the ingress NLB. It imports the EKS cluster by name. It does not install Nessie, Trino, or Argo.

Helmfile owns the cluster: one chart directory per component, dev.yaml / prod.yaml, applied with helmfile -e prod apply. Charts do not create namespaces, VPCs, or buckets. We vendor seven charts in-tree instead of pulling them from a Helm repo on apply. Four of those trees have template patches; three match their pinned upstream tag. That is how you keep a one-line template fix without waiting on upstream — and how you inherit the next upgrade as a merge, not a surprise.

aws eks update-kubeconfig --name prod-eks --region "$AWS_REGION"
 
cd charts/data-lakehouse/nessie
helmfile -e prod apply

If a change is an IAM policy or an S3 lifecycle rule, it is CDK. If a change is a JVM heap, a catalog URI, or a CronWorkflow, it is Helmfile. Mixing the two — installing Helm releases from CDK, or creating IAM roles by hand to match a values file — is how this architecture rots.

A Helmfile release looks like this:

environments:
  prod:
    values:
      - prod.yaml
  dev:
    values:
      - dev.yaml
---
releases:
  - name: nessie
    namespace: data-lakehouse
    chart: ./nessie_chart
    values:
      - "{{ .Environment.Name }}.yaml"
    createNamespace: false

createNamespace: false is deliberate. Namespaces are a cluster convention, not a side effect of the first chart you happen to apply.

Vendored charts vs stock templates

Vendoring is not the same as forking. We pinned each chart to an upstream tag and diffed the in-tree templates against that tag. Four charts have real template patches. Three are stock copies (plus, for DataHub, an in-chart version pin). Overlay files (prod.yaml) are not patches.

Spark — Ingress must name the TLS port

Bitnami 9.4.1 hardcodes the Ingress backend to named port http. With security.ssl.enabled, the master Service exposes https and not http. We changed both backend lines in templates/ingress.yaml to ternary "https" "http" .Values.security.ssl.enabled. Everything else in that chart, including values.yaml defaults, matches tag spark/9.4.1. Later Bitnami Spark on main still hardcodes http; this is not a backport.

Nessie — PDB, priority, affinity, shutdown

Stock 0.103.3 has no PDB template and no priorityClassName on the Deployment. We added:

• templates/pdb.yaml gated on podDisruptionBudget.enabled (minAvailable / maxUnavailable, mutually exclusive). Upstream added a PDB later, in 0.103.5, with a different schema. Ours is not that PR.
• priorityClassName on the pod spec. Upstream added this later, in 0.106.x, in a different place. Ours is earlier and separate.
• Affinity only renders when affinity.enabled is true; the enabled key is stripped before toYaml so Kubernetes does not see it.
• terminationGracePeriodSeconds plus, in the ConfigMap, quarkus.shutdown.timeout when gracefulShutdown.enabled is true.

Chart defaults leave those flags off (except an explicit 30s grace period, which is Kubernetes' default anyway). The env overlay turns them on: priorityClassName: high-priority, graceful shutdown 70s / PT60S, PDB minAvailable: 1, affinity enabled. Helpers (_helpers.tpl) are unmodified.

Trino — priorityClassName on both Deployments

Stock 1.39.1 cannot set a PriorityClass on coordinator or worker. We added templates/priorityclass.yaml (create: false by default) and one block after serviceAccountName on both Deployments. Main and ETL share one patched tree; they are not two forks. Upstream added coordinator/worker priorityClassName later, in 1.42.0, without an in-chart PriorityClass object. We do not create the class from Helm (priorityClass.create: false). The overlay sets coordinator.priorityClassName: high-priority on a class that already exists in the cluster.

Argo Workflows — the server probe is optional

Stock 0.45.15 always emits a hardcoded server readiness probe (/, port 2746, 10s delay). We wrapped it in server.readinessProbe.enabled. The extra probe fields in values.yaml (initialDelaySeconds: 60, scheme: HTTPS, …) are ignored by the template; only enabled is read. Turning the probe on still emits the stock HTTP probe. The overlay sets enabled: false. images.pullPolicy is IfNotPresent instead of Always. CRDs match upstream. Argo Events 2.4.15 is unmodified.

DataHub and Superset — vendor, do not fork templates

Superset 0.14.2 is byte-identical to the Apache release tarball. DataHub 0.6.1 templates and all six subcharts match acryldata/datahub-helm tag datahub-0.6.1. The in-chart values.yaml (not an env overlay) pins global.datahub.version to v1.4.0.2 and bumps Chart.yaml appVersion to match. Prerequisites 0.1.15 has no templates of its own; in-chart values turn the Elasticsearch PDB off (maxUnavailable: "") and default MySQL to disabled.

The rule is: patch the smallest template that stock cannot express, pin the rest, and keep the overlay for heap, catalogs, and hostnames.

The cluster already existed

The EKS cluster was created by hand. CDK never took over the control plane. It imported the cluster and attached the AWS objects the lakehouse needed:

import * as eks from "aws-cdk-lib/aws-eks";
import * as ec2 from "aws-cdk-lib/aws-ec2";
import * as iam from "aws-cdk-lib/aws-iam";
 
const kubectlRole = new iam.Role(this, "KubectlRole", {
  roleName: "kubectlRole",
  assumedBy: new iam.AccountRootPrincipal(),
});
 
const cluster = eks.Cluster.fromClusterAttributes(this, "ExistingCluster", {
  clusterName: "prod-eks",
  vpc,
  kubectlRoleArn: kubectlRole.roleArn,
});
 
new eks.Nodegroup(this, "SpotNodeGroup", {
  cluster,
  minSize: 0,
  desiredSize: 0,
  maxSize: 8,
  capacityType: eks.CapacityType.SPOT,
  instanceTypes: [
    ec2.InstanceType.of(ec2.InstanceClass.M5, ec2.InstanceSize.XLARGE),
  ],
  labels: { spot: "true" },
  taints: [
    {
      effect: eks.TaintEffect.NO_SCHEDULE,
      key: "spot",
      value: "true",
    },
  ],
});

The kubectlRole is mapped into aws-auth by hand. Project stacks use it to create service accounts. CDK cannot do that mapping for a cluster it did not create.

Node groups added later are opt-in. Spot and high-memory groups are tainted (spot=true:NoSchedule, high-memory=true:NoSchedule) and tagged for the cluster autoscaler, with minSize: 0. A group costs nothing until a pod that tolerates its taint is pending. Lakehouse coordinators stay on the original on-demand capacity: a Nessie replica on Spot dies mid-commit.

Namespaces: data-lakehouse, data-engineering, argo

Three namespaces, kept boring on purpose.

kubectl create namespace data-lakehouse
kubectl create namespace data-engineering
kubectl create namespace argo

Both Trino clusters live in data-lakehouse, including the ETL one. Clients (Spark jobs, dbt containers) may run in data-engineering or argo; the query engines stay next to the catalog. Cross-namespace DNS is the usual {service}.{namespace}.svc.cluster.local.

Spark RPC certs and warehouse credentials have to be projected into argo, where spark-submit drivers run. Treat that copy as part of the platform, not a one-off kubectl get secret.

S3 warehouse and the Iceberg prefix

One bucket per environment: {org}-data-lakehouse-{env}. Versioned. RemovalPolicy.RETAIN. Iceberg tables live under a single prefix, iceberg/. Everything else in the bucket — inventory reports, governance extracts — stays off that prefix so a careless remove_orphan_files cannot see it as garbage.

s3://{org}-data-lakehouse-{env}/
  iceberg/          # table data + metadata.json
  governance/       # S3 inventory reports, not Iceberg

Trino talks to the warehouse as s3://. The standing Spark cluster overlay does too. Argo spark-submit jobs use s3a://. Same bucket, same prefix — mixing prefixes is how you spend a day staring at empty schemas.

IAM is IRSA for in-cluster engines, not long-lived keys on the coordinator. CDK creates the service accounts in data-lakehouse (and Spark's in data-engineering) and attaches object/list permissions on that one bucket. Helm must use the same account names. A values-file typo looks like a random AccessDenied. Spark jobs submitted from Argo are the exception: they still use static keys. Both Spark paths are below. Daily S3 Inventory (Parquet, governance/ prefix) is cheap and is how you notice metadata files, not data files, are what grew.

Nessie as the Iceberg catalog

Iceberg needs a catalog. We used Nessie: git-like branches over the same warehouse, REST to every engine. Trino and Spark both speak it. Branching is real; we mostly stay on main and keep a staging ref for experiments.

Nessie itself needs a version store. The first deploy used JDBC2 against in-cluster MySQL with a PVC. It worked. It also had no TLS, no HA, and no backup story. After a few months the store was tens of gigabytes. RocksDB is single-node. DocumentDB looks like MongoDB until Nessie asks for operators the compatibility layer does not have.

We moved JDBC2 to Aurora MySQL. Same schema, TLS required, automated backups. Freeze Nessie, dump refs2 / objs2, import, flip the JDBC URL, keep the PVC until you trust the new store.

versionStoreType: JDBC2
jdbc:
  jdbcUrl: "jdbc:mysql://${DATABASE_HOST}:3306/nessie?useSSL=true&sslMode=REQUIRED"
  secret:
    name: aurora-mysql-creds
    username: username
    password: password
extraEnv:
  - name: DATABASE_HOST
    valueFrom:
      secretKeyRef:
        name: aurora-mysql-creds
        key: host

RDS requires TLS; in-cluster MySQL was running with SSL off. Do not keep the in-cluster database in production to dodge a certificate.

Nessie is small — hundreds of megabytes — and sits on every table load. Treat it as infrastructure, not as a replica you can evict.

Figure 2. Spark writes the warehouse; Trino plans only after Nessie returns a snapshot.

Figure 2 is the data path, not another topology. Spark (1) writes Parquet to s3://…/iceberg/ and (2) commits the snapshot to Nessie over REST. Trino (3–4) asks Nessie for the current snapshot, then (5) fetches metadata.json — that hop parses the full snapshot list into coordinator heap — and (6) workers read Parquet. Dashed arrows are catalog control; solid arrows are S3. The standing cluster and Trino use /api/v2. Argo spark-submit jobs in this post still use /api/v1.

Two Trino clusters: humans vs Spark and dbt

One Trino cluster is a bad idea the first time a dbt full-refresh and a dashboard land on the same coordinator.

We run two:

• trino-main — people and Superset. Interactive queries, the Web UI, BI.
• trino-etl — Spark-adjacent SQL and dbt. Fault-tolerant execution, sized for batch, allowed to be slow.

Both catalogs point at the same Nessie and the same warehouse:

catalog:
  lakehouse: |
    connector.name=iceberg
    iceberg.catalog.type=nessie
    iceberg.nessie-catalog.uri=http://nessie.data-lakehouse.svc.cluster.local:19120/api/v2
    iceberg.nessie-catalog.default-warehouse-dir=s3://{org}-data-lakehouse-{env}/iceberg
    iceberg.nessie-catalog.ref=main
    iceberg.allowed-extra-properties=*

iceberg.allowed-extra-properties lets you ALTER TABLE … SET PROPERTIES without a redeploy. Prefer an allowlist over * once you know which properties you actually set.

What shipped on trino-etl is fault-tolerant execution: retry-policy=TASK and a filesystem exchange manager on S3. That lets failed tasks retry; it is not spill-to-disk and it does not relieve CLUSTER_OUT_OF_MEMORY. Intermediate exchange data lives under a prefix next to the warehouse, not on the worker's disk.

server:
  config:
    query:
      maxMemory: "32768MB"
  exchangeManager:
    name: filesystem
    baseDir: "s3://{org}-data-lakehouse-{env}/trino-etl"
additionalConfigProperties:
  - "retry-policy=TASK"
  - "task-retry-attempts-per-task=4"

Spill-to-disk was considered. Trino spill writes a local filesystem path; there is no native S3 spill destination. An emptyDir mount looks convenient and then fills the kubelet root volume, which is why we rejected it. If you need spill later, give workers a dedicated disk — do not pretend the S3 exchange manager is that disk.

Session knobs from dbt cannot exceed cluster hard limits. SET SESSION query_max_memory_per_node = '2500MB' is a bandage, not a capacity plan. Coordinator query history is in-memory and dies on restart; Trino's MySQL event listener into Aurora (trino_queries) is how you still answer "it was slow yesterday."

Spark as the writer (standing cluster)

Spark is the writer. Trino can INSERT and CTAS, and dbt does, but the heavy maintenance and the large ETL jobs go to Spark.

We run a long-lived Spark cluster in data-engineering (Bitnami 9.4.1, vendored, RPC auth and SSL on) and submit from Argo. The Ingress backend patch is above; without it the UI talks HTTP to a TLS-only Service.

S3 credentials were two paths, both shipped.

Standing cluster — IRSA. Master and workers use a service account in data-engineering. CDK attaches the IAM role; Helm must use the same account name. Hadoop reads the projected web-identity token. This is the same IRSA pattern as Trino: no keys on the coordinator.

sparkConfiguration:
  spark.sql.catalog.nessie: "org.apache.iceberg.spark.SparkCatalog"
  spark.sql.catalog.nessie.catalog-impl: "org.apache.iceberg.nessie.NessieCatalog"
  spark.sql.catalog.nessie.uri: "http://nessie.data-lakehouse.svc.cluster.local:19120/api/v2"
  spark.sql.catalog.nessie.ref: "main"
  spark.sql.catalog.nessie.warehouse: "s3://{org}-data-lakehouse-{env}/iceberg"
  spark.hadoop.fs.s3a.impl: "org.apache.hadoop.fs.s3a.S3AFileSystem"
  spark.hadoop.fs.s3a.aws.credentials.provider: "com.amazonaws.auth.WebIdentityTokenCredentialsProvider"

Argo spark-submit — static keys. Drivers run in argo, not under the Spark service account. Those workflows export keys from a Secret mounted at /aws and force SimpleAWSCredentialsProvider. Trino catalogs and the standing cluster talk Nessie /api/v2; these jobs still use /api/v1. Static keys leak into the next namespace you copy them into — which is why they have to be projected into argo as platform, not as a one-off kubectl get secret.

export AWS_ACCESS_KEY_ID="$(cat /aws/access)"
export AWS_SECRET_ACCESS_KEY="$(cat /aws/secret)"
 
spark-submit \
  --master spark://spark-master-svc.data-engineering.svc.cluster.local:7077 \
  --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions,org.projectnessie.spark.extensions.NessieSparkSessionExtensions \
  --conf spark.sql.catalog.nessie=org.apache.iceberg.spark.SparkCatalog \
  --conf spark.sql.catalog.nessie.catalog-impl=org.apache.iceberg.nessie.NessieCatalog \
  --conf spark.sql.catalog.nessie.uri=http://nessie.data-lakehouse.svc.cluster.local:19120/api/v1 \
  --conf spark.sql.catalog.nessie.ref=main \
  --conf spark.sql.catalog.nessie.warehouse=s3a://{org}-data-lakehouse-{env}/iceberg \
  --conf spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider \
  job.py

IRSA on the submit pod (WebIdentityTokenCredentialsProvider on the driver) is the path we wanted for jobs too. It is not what those workflows ship today.

Spark-on-Kubernetes (--master k8s://…) or the Spark Operator would give you a driver per job and no idle cluster. We kept the standing cluster because job rate is high and the path is one spark-submit from a workflow. The cost is idle executors, cross-namespace SSL, and a driver that must bind a routable pod IP.

dbt is a client, not a platform chart

There is no dbt Helm release. dbt runs as a container in an Argo workflow, talks to trino-etl, and leaves. It is a compiler, not cluster infrastructure. It is also how Iceberg metadata explodes.

materialized='table' is CREATE TABLE AS SELECT plus an atomic swap. Every run is a new snapshot and a new metadata.json. An hourly rebuild of one fat report is hundreds of snapshots in a few weeks. The current metadata file holds the full snapshot list. Trino's coordinator parses that JSON into Java objects before it plans a query.

High-cardinality partitions make this worse: a user-id key on a frequently rebuilt table produces a fanout of tiny files. The first fix we adopted was dropping those keys and keeping time-only partitions (month(event_ts)). Revisit richer partitioning when the write path is CDC/MERGE with compact files. Scheduled Spark expire_snapshots was explicitly deferred at that point; the weekly CronWorkflow came later.

A dbt model that is too large for the default worker cap looks like this:

{{ config(
    materialized='table',
    schema='reports',
    pre_hook=[
        "SET SESSION query_max_memory_per_node = '2500MB'",
        "SET SESSION query_max_memory = '20GB'"
    ]
) }}

That pre-hook is a bandage. Incremental models are the real fix, and they are where SCD Type 2 gets subtle. We shipped session settings first. Skip dbt post_hook snapshot expiry: it couples every model to maintenance and still runs in the engine you are trying to protect.

Weekly Iceberg maintenance as an Argo CronWorkflow

Snapshot expiry has to happen, and it has to happen outside the Trino coordinator.

Trino's ALTER TABLE … EXECUTE expire_snapshots runs on the coordinator — the process that already OOM'd while reading the metadata. Nessie GC (nessie-gc.jar) is the wrong tool: commits are immutable, so GC does not rewrite metadata.json or shrink the snapshot list. It deletes files, including, in reported cases, files still referenced by live metadata.

Partitioning was the chosen remediation for the coordinator OOM. We added a weekly Spark maintenance job anyway — expiry still has to happen outside the Trino coordinator, even if it was not the ADR's "do this now." Sunday, off-peak, concurrencyPolicy: Replace. Per table: optional archive append, expire_snapshots, remove_orphan_files, rewrite data files and manifests. Skip .*__dbt_tmp.*. Keep seven days of snapshots and at least one. Weekly jobs can keep a tighter CronWorkflow history (3/1 below) than busy hourly pipelines (10/3).

apiVersion: argoproj.io/v1alpha1
kind: CronWorkflow
metadata:
  name: iceberg-weekly-maintenance
  namespace: argo
spec:
  schedule: "0 3 * * 0"
  timezone: "UTC"
  concurrencyPolicy: Replace
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 1
  workflowSpec:
    workflowTemplateRef:
      name: iceberg-weekly-maintenance
    arguments:
      parameters:
        - name: retention-days
          value: "7"
        - name: retain-last
          value: "1"
        - name: orphan-retention-days
          value: "7"

The procedure call is ordinary Iceberg Spark SQL:

from datetime import datetime, timedelta, timezone
 
older_than = datetime.now(timezone.utc) - timedelta(days=7)
 
spark.sql(
    f"""
    CALL nessie.system.expire_snapshots(
      table => 'reports.user_actions',
      older_than => TIMESTAMP '{older_than.strftime("%Y-%m-%d %H:%M:%S")}',
      retain_last => 1,
      stream_results => true
    )
    """
)

Two different growth problems, two different knobs:

1. Snapshot list inside the current metadata.json — this is what blows the coordinator heap. Only expire_snapshots removes it.
2. Old metadata.json files on S3 — each commit writes a new versioned file. write.metadata.delete-after-commit.enabled (and remove_orphan_files for data) is storage hygiene. It will not stop the OOM.

Do not delete metadata files by hand. You will desync Nessie's pointer from S3.

DataHub and Superset in the same namespace

Discovery and BI sit next to the catalog, in data-lakehouse.

DataHub is a real cluster: GMS, frontend, Kafka, ZooKeeper, Elasticsearch, Aurora in production. It is the heaviest chart in the namespace. Pin image digests. Superset is the BI surface on trino-main (Redis, Aurora metadata, TLS ingress). Generate admin credentials into a Secret; do not commit them.

One namespace means one network-policy story. It also means DataHub's Kafka brokers compete for nodes with Nessie — size the requests. If DataHub or Superset is down, Spark should still write and Trino should still query. That is the test that they are in the right layer.

Argo Workflows and Events as the control plane — not Argo CD

We do not use Argo CD. Charts are applied by Helmfile from CI. Argo Workflows is the data control plane: scheduled ETL, dbt, Iceberg maintenance, backfills. Argo Events wires "this workflow succeeded" to "start that workflow." Cluster-app GitOps and data-job scheduling are different failure domains.

Workflows archive to MySQL so history survives CR deletion. Logs go to the cluster log stack, not Argo archiveLogs. CronWorkflows get count-based history limits; the controller ConfigMap gets a default TTL:

controller:
  workflowDefaults:
    spec:
      ttlStrategy:
        secondsAfterSuccess: 604800
        secondsAfterFailure: 2592000

workflowDefaults apply at creation time. Existing CRs do not inherit them. History limits belong on the CronWorkflow, not on the WorkflowTemplate.

Sensors watch completions and submit the next template. Keep producers boring: a CronWorkflow, a label, a Sensor. The temptation is to replace Resource EventSources with a message bus the first time something OOMs. Fix the watch first.

The one Argo Workflows template change that is not overlay is the server readiness probe: stock always emits it; we made enabled a switch and turned it off. Events is stock 2.4.15. Label filters and history limits live in values and CRs, not in a forked Events chart.

What broke

Three failures, all memory, none of them the warehouse.

Figure 3. Three memory failures. Vermillion is the blast radius; the small pills are the fixes that held.

Figure 3 keeps S3 grey on purpose. Fail 1 is the coordinator parsing one fat metadata.json. Fail 2 is the scheduler killing Nessie to place a higher-priority Trino. Fail 3 is the Argo Events informer caching every Workflow CR. The rest of this section is the terrain under those three boxes.

Trino coordinator OOM

The coordinator died with java.lang.OutOfMemoryError: Java heap space while loading one table's metadata.json through Nessie. The file version was in the 700s. The JVM was 2 GiB. Raising it to 8 GiB stopped the crash and did not stop the growth.

The coordinator must fetch, parse, and hold the entire snapshot list to plan. One table is enough. iceberg.metadata.parallelism will not save you. The adopted fix was time-only partitions on the worst dbt tables. The weekly Spark expiry job came after, so metadata could not grow unbounded again. Do not run expiry on the coordinator you are trying to keep alive.

Nessie preemption

After the Aurora move, Nessie pods disappeared across three nodes at once. The events looked like taint evictions. They were preemption. Nessie was medium-priority (600000); Trino coordinators were high-priority (1000000). The scheduler killed Nessie to place them.

Anti-affinity is evaluated at schedule time, not at preemption time. A PodDisruptionBudget is best-effort: if the only victims violate the PDB, they still go. Stock Nessie 0.103.3 cannot set a PriorityClass or a PDB; those keys exist because we patched the chart. The mechanism that actually prevents this is a PriorityClass at the same level as the workloads that would otherwise eat you.

priorityClassName: high-priority

We still saw replacements during node turnover. Graceful shutdown is the rest of the fix, also a chart patch: quarkus.shutdown.timeout=PT60S (ISO-8601) and terminationGracePeriodSeconds: 70 so the grace period outlives the timeout and in-flight commits finish before SIGKILL. Align memory requests with actual usage — Nessie is small — so it is not a tempting victim for the wrong reason.

EventSource OOM

Argo Events Resource EventSources watch Workflow CRs. Kubernetes informers cache the objects they watch. There is no "keep last N." At ~1,400 Workflows, EventSources at 512 Mi were OOMKilled. At ~2,000, with nodeStatusOffLoad: false, the workflow controller at 2 GiB joined them in CrashLoopBackOff.

Raising limits buys time. The cache still grows with CR count.

What worked, without replacing Resource EventSources:

1. Label filters on the EventSource (filter.labels with workflows.argoproj.io/cron-workflow=…). In Argo Events 1.9.x that becomes a server-side labelSelector. Field and prefix filters do not. Memory per EventSource dropped to tens of megabytes.
2. CronWorkflow history limits (successfulJobsHistoryLimit: 10, failedJobsHistoryLimit: 3) so the controller's watch set is bounded.
3. nodeStatusOffLoad: true so node graphs live in the archive database, not in every CR.
4. Archive enabled, so deleting CRs does not delete history.

AMQP EventSources are O(1) in workflow count. They also mean a broker, a bridge, and hooks on every template. We did not need that once the informer stopped caching the whole namespace.

When this applies

Copy this if you already have an EKS cluster, you will not let CDK install Helm, and you can stand two Trino coordinators plus a standing Spark cluster. Skip it if you need Spark-on-Kubernetes / the Spark Operator, Argo CD as the chart installer, or a from-scratch control plane.

If you copy this, copy the seam first: CDK around the cluster, Helmfile on it, three namespaces, one warehouse prefix, Nessie on managed JDBC2, two Trinos, Spark as the writer, dbt as a job, expiry on Spark, Argo as the scheduler. Then budget heap for metadata, priority for the catalog, and a hard cap on Workflow CRs. Those three are not extras. They are the difference between a lakehouse that runs and one that pages you.