Nessie JDBC2 outgrew the PVC

The Iceberg catalog is not the warehouse. Parquet lives on S3. Nessie's refs and objects live in a database. After a few months that database, JDBC2 against in-cluster MySQL on a PVC, was tens of gigabytes, with no TLS, no HA, and no backup. We froze Nessie, dumped refs2 / objs2, and pointed the same JDBC2 schema at Aurora MySQL. This post is the store cutover only.

Catalog commits are not Parquet

Two stores, easy to confuse.

• Warehouse. s3://…/iceberg/. Table data and metadata.json.
• Version store. JDBC2. Named references (refs2: branches, tags) and serialized catalog objects (objs2). Nessie reads this on every table load.

The process is small (hundreds of megabytes). The store was tens of gigabytes. Treat the process as infrastructure: Spark and both Trinos resolve every table through it.

This sits after dbt forced a second Trino. Two engines already shared this catalog. The store was still MySQL on a PVC.

Figure 1. Freeze Nessie, dump JDBC2, cut over to Aurora; keep the PVC until trusted.

Figure 1 is the cutover, in layers. Blue is Nessie, the catalog process. Orange is the version store: MySQL on a PVC, then Aurora MySQL. Sky dashed is us (Helmfile): freeze, dump, flip. Grey dashed is the PVC we did not delete on the same change. The warehouse on S3 is not on this map. JDBC2 holds catalog commits, not Parquet.

What the PVC could not be

Iceberg needs a catalog: a service that names tables and points engines at the current snapshot. We used Nessie (Project Nessie 0.103.3), git-like branches over one warehouse, REST to every engine. Trino and Spark both speak it. Branching is real; we mostly stay on main and keep a staging ref for experiments.

The first deploy used JDBC2 against in-cluster MySQL with a PersistentVolumeClaim (PVC). It worked. It also had no TLS (SSL off), no high availability, and no backup story.

A PVC that large, on a single in-cluster MySQL, is a catalog you cannot restore. Volume snapshots are not a managed backup, not HA, and not encryption in transit. We were running production Iceberg commits through a database we would not have accepted for an application.

RDS requires TLS. The in-cluster server was running with SSL off so the JDBC URL could stay plain. That is the wrong direction: do not keep the in-cluster database in production to dodge a certificate.

RocksDB and DocumentDB

Two other Nessie stores looked like a way off the PVC without standing up Aurora.

RocksDB is a supported version store, production, single node only. One replica, one disk. We already knew Nessie sits on every table load. Embedding the store in that same replica makes the catalog and its disk one failure domain. HA is not a later flag you turn on.

DocumentDB looks like MongoDB until Nessie asks for operators the compatibility layer does not have. Nessie's Mongo stores (MONGODB / MONGODB2) assume a real MongoDB. We did not take that path.

JDBC2 to Aurora MySQL kept the schema we already had. Same tables, TLS required, automated backups. That is why we moved the store instead of changing the store type.

Cut over

Same JDBC2 schema means the cutover is a dump of two tables, not a catalog rebuild and not a Nessie store-type migration.

1. Freeze Nessie so writers cannot commit while the dump is in flight.
2. Dump refs2 (named references: branches, tags) and objs2 (serialized catalog objects) from in-cluster MySQL.
3. Import those tables into Aurora MySQL.
4. Flip the JDBC URL in the Helmfile overlay. TLS is required on the new URL.
5. Keep the PVC until the new store is trusted: commits land, engines still resolve main, a rollback still has somewhere to point.

The overlay that Nessie 0.103.3 ran after the flip:

versionStoreType: JDBC2
jdbc:
  jdbcUrl: "jdbc:mysql://${DATABASE_HOST}:3306/nessie?useSSL=true&sslMode=REQUIRED"
  secret:
    name: aurora-mysql-creds
    username: username
    password: password
extraEnv:
  - name: DATABASE_HOST
    valueFrom:
      secretKeyRef:
        name: aurora-mysql-creds
        key: host

aurora-mysql-creds is a Kubernetes Secret. username, password, and host are keys in that Secret, not the credentials. DATABASE_HOST is the Aurora endpoint; the JDBC URL interpolates it. sslMode=REQUIRED is the TLS we did not have on the PVC. useSSL=true is in the URL we shipped.

Do not flip the URL and delete the PVC in the same change. Trust means Nessie has been up against Aurora through enough commits that you would rather restore from RDS than from the old volume.

When this applies

Use this cutover if Iceberg already runs through Nessie JDBC2 on in-cluster MySQL, the PVC has grown into tens of gigabytes with no TLS / HA / backup, and you can dump refs2 / objs2 instead of rebuilding the catalog. Skip it if you are still on IN_MEMORY or a laptop RocksDB, or if JDBC2 already points at a managed store with TLS. Skip DocumentDB as a Mongo stand-in.

After Aurora, the scheduler preempted Nessie.