After Aurora, the scheduler preempted Nessie

The scheduler preempted Nessie to place Trino: Nessie was medium-priority (600000), coordinators were high-priority (1000000), and pods disappeared across three nodes at once. This post is that incident (PriorityClass, a best-effort PodDisruptionBudget, graceful shutdown), not the Aurora cutover.

A small catalog is a cheap victim

Nessie is hundreds of megabytes and sits on every table load. That combination makes it cheap to kill on a packed node, and expensive for the query path when it dies.

• PriorityClass. An integer. high-priority at 1000000 beats medium-priority at 600000.
• Preemption. The scheduler evicts the lower number so the higher number can place.
• Not a taint. Kubernetes writes an eviction either way. Three nodes going dark at once after a catalog move is an easy taint story. It was the wrong story.

This sits after Nessie JDBC2 outgrew the PVC. Aurora was already the store. The warehouse did not move.

Figure 1. Failure domain: the scheduler preempted Nessie to place Trino. The warehouse did not move.

Figure 1 is one failure, in layers. Grey dashed is S3 iceberg/, out of scope: Parquet and metadata.json were fine. Blue is the Trino coordinators, already high-priority (1000000), pending placement. Vermillion is the blast radius: Nessie at medium-priority (600000), gone on three nodes. The sky pill is the fix that held: the same PriorityClass as the coordinators, plus graceful shutdown. Anti-affinity and a PDB are on the fail box because they did not stop it.

PriorityClass is the mechanism

The integer is the whole story. When a Trino coordinator could not schedule, the scheduler looked for victims with a lower value and killed Nessie.

Pod anti-affinity does not run at that moment. It is evaluated when a pod is placed, not when another pod needs the node. Spreading replicas at schedule time does not protect them from a later high-priority placement.

A PodDisruptionBudget is best-effort under preemption. If the only victims that free enough capacity violate the PDB, they still go. minAvailable: 1 is not a fence.

The mechanism that actually prevents this is a PriorityClass at the same level as the workloads that would otherwise eat you:

priorityClassName: high-priority

That is the same class the Trino coordinators already used. Matching it is the prevention. The PDB stays in the overlay as hygiene for voluntary disruption, not as the preemption fix.

The catalog is infrastructure. Treat it as a replica you can evict and the query path goes with it.

Graceful shutdown

PriorityClass stops Nessie being the cheap victim. Nodes still drain. We still saw replacements during node turnover.

The rest of the fix is letting Quarkus finish in-flight commits before SIGKILL. The Nessie ConfigMap gets an ISO-8601 duration, not a Kubernetes seconds integer:

quarkus.shutdown.timeout=PT60S

The pod spec must outlive that timeout, or the kubelet sends SIGKILL while Quarkus still thinks it has a minute:

terminationGracePeriodSeconds: 70

Align memory requests with actual usage. Nessie is small; a request that does not match what it uses still makes it a tempting victim for the wrong reason. The class is what ranks it against Trino. The grace period is what keeps a drain from cutting a commit in half.

What we patched

Stock charts at our pins could not set this. Overlay values for heap and hostnames are not patches; a missing template key is. We patched the smallest templates and left _helpers.tpl alone.

Nessie 0.103.3 has no PDB template and no priorityClassName on the Deployment. We added:

• templates/pdb.yaml gated on podDisruptionBudget.enabled (minAvailable / maxUnavailable, mutually exclusive). Upstream's later PDB is not this patch.
• priorityClassName on the pod spec. Upstream's later field is not this patch.
• Affinity only when affinity.enabled is true; that enabled key is stripped before toYaml so the API server never sees it. It still does not run at preemption time.
• terminationGracePeriodSeconds plus, in the ConfigMap, quarkus.shutdown.timeout when gracefulShutdown.enabled is true.

Chart defaults leave those flags off, except an explicit 30s grace period, which is already Kubernetes' default. The env overlay turns them on:

priorityClassName: high-priority
podDisruptionBudget:
  enabled: true
  minAvailable: 1
gracefulShutdown:
  enabled: true
affinity:
  enabled: true

terminationGracePeriodSeconds: 70 and quarkus.shutdown.timeout=PT60S ship with that overlay. Helpers are unmodified.

Trino 1.39.1 cannot set a PriorityClass on coordinator or worker. We added templates/priorityclass.yaml (create: false by default) and one block after serviceAccountName on both Deployments. Main and ETL share that one patched tree; they are not two forks. Upstream later added coordinator/worker priorityClassName in 1.42.0, without an in-chart PriorityClass object.

We do not create the class from Helm. The overlay names a class that already exists in the cluster, on the coordinator:

priorityClass:
  create: false
coordinator:
  priorityClassName: high-priority

Do not wait on those upstream PRs if you are still on 0.103.3 / 1.39.1. The schemas and the insertion points differ; merging them is not a backport of what we run.

When this applies

Use this if an Iceberg catalog (or any small control-plane pod) shares nodes with higher-priority query engines, and your pinned chart cannot set priorityClassName. Skip it if the catalog already runs at the same PriorityClass as the coordinators. On Nessie 0.106.x or Trino 1.42.0, stock can set the field; the insertion point and PDB schema are still not our patches, and a values key the template ignores is the same hole.

A PDB is still worth enabling for drains. It will not save you from 1000000 versus 600000.

Aurora was already the JDBC2 store when this happened. The next failure in this lakehouse was not more preemption; it was coordinator heap on a fat metadata.json: dbt CTAS OOMed the Trino coordinator.