dbt forced a second Trino

A Trino coordinator plans every query and hosts the Web UI. Query memory lives on the workers. One coordinator cannot be sized for a Superset dashboard and a dbt full-refresh at once. We split the cluster: trino-main for people, trino-etl for batch with task retry. This post is that split, and why dbt stayed a container in an Argo workflow instead of a Helm release. Not the later coordinator failure, and not the catalog store that was still sitting on a PVC.

Two coordinators, one catalog

The split is two planners. It is not two warehouses.

• trino-main. Humans and Superset. Interactive queries.
• trino-etl. dbt and batch SQL. Fault-tolerant execution (task retry).
• Shared underneath. The same Nessie catalog, the same S3 iceberg/ prefix.

dbt is not a third box. It is a client of trino-etl: a compiler that emits SQL and exits.

This sits after Argo Events decoupled ingestion from dbt. Semantic-layer jobs were already a separate Workflow. They still shared a coordinator with dashboards.

Figure 1. Two coordinators, one Nessie, one warehouse.

Figure 1 is the map, in layers. Both Trinos live in data-lakehouse. Dashed arrows are catalog control: both talk to the same Nessie over REST /api/v2. Solid arrows are warehouse data: both read and write the same s3:// prefix, iceberg/. The thicker stroke is the split this post is about: two coordinators, not two catalogs.

One coordinator was a bad idea

Planning and the Web UI live on the coordinator. One coordinator and its workers cannot be sized for interactive queries and a full-refresh at once. The first time a dbt full-refresh and a dashboard land together, that is the constraint.

Same catalog, two releases

We run two releases of the same chart tree: trinodb/charts 1.39.1, appVersion 475. They are not two forks. Both sit in data-lakehouse, next to Nessie. Clients (dbt containers, Spark jobs) may run in data-engineering or argo; the query engines stay next to the catalog. Cross-namespace DNS is the usual {service}.{namespace}.svc.cluster.local.

Both catalogs point at the same Nessie and the same warehouse. {org} and {env} are the bucket name parts: one warehouse bucket per environment, Iceberg tables under a single iceberg/ prefix.

catalog:
  lakehouse: |
    connector.name=iceberg
    iceberg.catalog.type=nessie
    iceberg.nessie-catalog.uri=http://nessie.data-lakehouse.svc.cluster.local:19120/api/v2
    iceberg.nessie-catalog.default-warehouse-dir=s3://{org}-data-lakehouse-{env}/iceberg
    iceberg.nessie-catalog.ref=main
    iceberg.allowed-extra-properties=*

iceberg.allowed-extra-properties=* lets you ALTER TABLE … SET PROPERTIES without a redeploy. Prefer an allowlist over * once you know which properties you actually set.

Task retry on etl, not spill-to-disk

What shipped on trino-etl is fault-tolerant execution: retry-policy=TASK and a filesystem exchange manager on S3. Failed tasks retry. Intermediate exchange data lives under a prefix next to the warehouse, not on the worker's disk. That is not spill-to-disk, and it does not relieve CLUSTER_OUT_OF_MEMORY.

server:
  config:
    query:
      maxMemory: "32768MB"
  exchangeManager:
    name: filesystem
    baseDir: "s3://{org}-data-lakehouse-{env}/trino-etl"
additionalConfigProperties:
  - "retry-policy=TASK"
  - "task-retry-attempts-per-task=4"

retry-policy=TASK turns query.max-memory and query.max-memory-per-node off. The matching SET SESSION knobs do not apply either. The overlay still sets query.maxMemory: 32768MB; that value does not cap etl queries under this retry policy.

Spill-to-disk was considered. Trino spill writes a local filesystem path; there is no native S3 spill destination. An emptyDir mount looks convenient and then fills the kubelet root volume, which is why we rejected it. If you need spill later, give workers a dedicated disk. Do not pretend the S3 exchange manager is that disk.

dbt is a client

There is no dbt Helm release. dbt runs as a container in an Argo workflow, talks to trino-etl, and leaves. It is a compiler, not cluster infrastructure.

That is the same layering as the rest of the job plane: Helmfile installs engines; Argo runs work. Promoting dbt to a chart would make it look like Nessie (a long-running service with a values file) for a process whose job is to emit SQL and exit. We kept it a workflow container.

dbt issues INSERT and CREATE TABLE AS SELECT against trino-etl. Heavy maintenance and the large ETL jobs still go to Spark. The split is which coordinator the SQL hits, not a second warehouse.

Session memory is a bandage

We shipped this pre-hook on dbt models we treated as too large for a worker cap:

{{ config(
    materialized='table',
    schema='reports',
    pre_hook=[
        "SET SESSION query_max_memory_per_node = '2500MB'",
        "SET SESSION query_max_memory = '20GB'"
    ]
) }}

On trino-etl those SET SESSION lines are a no-op. TASK retry has already turned the limits off; the hook does not raise a worker cap on this cluster. We shipped it anyway. Incremental models are the real fix, and they are where SCD Type 2 gets subtle.

Session pre-hooks also do not fix what happens when a table's metadata grows past the coordinator. That failure is dbt CTAS OOMed the Trino coordinator.

When this applies

Copy this if you already run dbt full-refresh and dashboards on one Trino coordinator, and you can stand a second coordinator against the same Nessie and the same iceberg/ prefix. Skip it if batch and BI never overlap on that coordinator, or if you are about to install dbt as a Helm release so it "looks like" the rest of the lakehouse.

The catalog store was still on a PVC. That is the next constraint: Nessie JDBC2 outgrew the PVC.