dbt CTAS OOMed the Trino coordinator

Iceberg does not keep "the latest snapshot" in metadata.json. It keeps the full snapshot list. Trino's coordinator parses that JSON into Java heap before it plans. dbt materialized='table' is CREATE TABLE AS SELECT (CTAS) plus an atomic swap, so each hourly rebuild appends another snapshot. The coordinator died with java.lang.OutOfMemoryError: Java heap space on a file version in the 700s. JVM 2 GiB, then 8 GiB, which stopped the crash, not the growth. This post is that snapshot list, time-only partitions, and weekly Spark expire_snapshots. Not how we split Trino, not preemption, not EventSource OOM.

The planner holds the whole list

Three facts, in order.

• Each CTAS is a new snapshot. materialized='table' does not update a table in place. It writes a new Iceberg snapshot and a new current metadata.json.
• That file names every snapshot, not the latest one. An hourly rebuild of one fat report is hundreds of snapshots in a few weeks. One table is enough.
• The coordinator must parse the list to plan. Raising heap buys time. It does not shrink the list.

This sits after dbt forced a second Trino and after Nessie was preempted. dbt was already on trino-etl. Nessie was already on Aurora. The blast radius here is coordinator heap, not the catalog store and not S3.

Figure 1. Failure domain: the Trino coordinator heap, not the S3 warehouse.

Figure 1 keeps S3 grey on purpose. The bytes of metadata.json sit under the iceberg/ prefix. That is not where the process died. Vermillion is the coordinator that fetches that file through Nessie, parses the full snapshot list into Java objects, and OOMs. The dashed arrow is not a call from that process: expire_snapshots runs on the standing Spark cluster (blue), on a weekly CronWorkflow. Grey dashed under Spark is the rejected pair: Nessie GC (nessie-gc.jar) and Trino ALTER TABLE … EXECUTE expire_snapshots. They do not shrink the list the planner holds.

Why those three tools fail

Raising heap buys the afternoon. It does not shrink the snapshot list. The coordinator died with java.lang.OutOfMemoryError: Java heap space while loading one table's metadata.json through Nessie. The file version was in the 700s. The JVM was 2 GiB. Raising it to 8 GiB stopped the crash. The list kept growing. iceberg.metadata.parallelism will not save you: it threads the fetch, it does not cut what the planner holds.

Trino can run ALTER TABLE … EXECUTE expire_snapshots. That procedure runs on the coordinator, the process that already OOM'd while reading the metadata. Do not run expiry there.

Nessie GC (nessie-gc.jar) is the wrong tool. Nessie commits are immutable, so GC does not rewrite metadata.json or shrink the snapshot list. It deletes files, including, in reported cases, files still referenced by live metadata.

Skip dbt post_hook snapshot expiry: it couples every model to maintenance and still runs in the engine you are trying to protect.

CTAS grew the file; partitions made it worse

dbt is a container in an Argo workflow. It talks to trino-etl and leaves. There is no dbt Helm release. It is also how Iceberg metadata exploded.

High-cardinality partitions made the same file worse. A user-id key on a frequently rebuilt table produces a fanout of tiny files; the metadata has to name them. Time-only keys (month(event_ts)) came first. The model that was too large for the default worker cap looked like this:

{{ config(
    materialized='table',
    schema='reports',
    pre_hook=[
        "SET SESSION query_max_memory_per_node = '2500MB'",
        "SET SESSION query_max_memory = '20GB'"
    ]
) }}

That pre_hook is worker query memory. It cannot exceed cluster hard limits, and it does not shrink coordinator heap. Incremental models are the real fix for the rewrite itself, and they are where SCD Type 2 gets subtle. We shipped session settings first.

Partitions, then weekly Spark

The adopted fix for the crash was dropping high-cardinality keys and keeping time-only partitions (month(event_ts)). Revisit richer partitioning when the write path is change-data-capture MERGE with compact files. Scheduled Spark expire_snapshots was deferred at that point; the weekly CronWorkflow came later so the list could not grow unbounded again.

Expiry still has to happen outside the Trino coordinator, even if it was not the ADR's "do this now."

Sunday, off-peak, UTC, concurrencyPolicy: Replace. Per table: optional archive append, expire_snapshots, remove_orphan_files, rewrite data files and manifests. Skip .*__dbt_tmp.*. Keep seven days of snapshots and at least one (retain_last => 1). Weekly jobs can keep a tighter CronWorkflow history (3 successful / 1 failed) than busy hourly pipelines (10/3). The weekly job is this CronWorkflow:

apiVersion: argoproj.io/v1alpha1
kind: CronWorkflow
metadata:
  name: iceberg-weekly-maintenance
  namespace: argo
spec:
  schedule: "0 3 * * 0"
  timezone: "UTC"
  concurrencyPolicy: Replace
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 1
  workflowSpec:
    workflowTemplateRef:
      name: iceberg-weekly-maintenance
    arguments:
      parameters:
        - name: retention-days
          value: "7"
        - name: retain-last
          value: "1"
        - name: orphan-retention-days
          value: "7"

0 3 * * 0 is Sunday 03:00 UTC. Replace drops an overrun instead of stacking a second expiry on the same tables. The procedure call is ordinary Iceberg Spark SQL; reports.user_actions is one table in the reports schema the dbt models write:

from datetime import datetime, timedelta, timezone
 
older_than = datetime.now(timezone.utc) - timedelta(days=7)
 
spark.sql(
    f"""
    CALL nessie.system.expire_snapshots(
      table => 'reports.user_actions',
      older_than => TIMESTAMP '{older_than.strftime("%Y-%m-%d %H:%M:%S")}',
      retain_last => 1,
      stream_results => true
    )
    """
)

older_than is seven days. retain_last => 1 keeps at least one snapshot even if every snapshot is older than that window. nessie.system is the Spark catalog's procedure namespace, not a Trino session.

Two knobs

Two different growth problems, two different knobs:

Only expire_snapshots removes the list the coordinator parses. Each commit also writes a new versioned metadata.json on S3; deleting those old files is storage hygiene. It will not stop the OOM.

Do not delete metadata files by hand. You will desync Nessie's pointer from S3.

When this applies

Use this if dbt materialized='table' (or any CTAS) rebuilds Iceberg tables that a Trino coordinator plans, and the current metadata.json is accumulating snapshots. Skip it if writes are already incremental MERGE with compact files and expiry already runs on Spark, or if the planner is not a Trino coordinator holding that list in heap.

The next memory failure in this cluster was not another metadata file: EventSources OOM on Workflow CR count.