Argo Workflows and Spark as the writer

Trino can read Iceberg. It can also INSERT. We do not use it as the writer. Heavy ETL goes to Spark: write Parquet, commit a snapshot, leave. Argo Workflows was already the data scheduler; Spark came later as that writer. This post is the first job plane: a standing Spark cluster in data-engineering, submitted from Workflows in argo. Not Argo CD, not the Spark Operator, and not Events yet.

Writer, catalog, scheduler

Three jobs, three places.

• Writer. A long-running Spark cluster. It PUTs Parquet to S3, then tells Nessie the new snapshot exists.
• Catalog. Nessie, already serving one Trino. Spark commits over REST. Trino's later read is not this post.
• Scheduler. Argo Workflows. A Workflow CR is how a data job starts. Charts do not.

This sits after Nessie and Trino on in-cluster MySQL. The catalog and one query engine already existed. Nothing in that era scheduled a write.

Figure 1. Spark write path: PUT Parquet, then REST commit.

Figure 1 is the write path only, in two layers. Solid arrows are the S3 data plane: Spark PUTs Parquet under s3://…/iceberg/. Dashed arrows are catalog control: Spark commits the snapshot to Nessie over REST. The note on Nessie is the pointer on main. Trino's later GET of metadata.json is not on this map. That hop belongs with coordinator heap, not with the first writer.

Why Workflows, not charts and not Argo CD

Argo Workflows existed on this cluster before Spark did. A Workflow CR is how a data job starts.

We do not use Argo CD. Helmfile from CI installs the lakehouse charts. Cluster-app GitOps and data-job scheduling are different failure domains. Workflows is the data control plane. It is not how pods get onto the cluster.

Three namespaces, kept boring on purpose:

Clients run in data-engineering or argo; the catalog and the query engine stay in data-lakehouse. Cross-namespace DNS is the usual {service}.{namespace}.svc.cluster.local. Spark RPC certs and warehouse credentials have to be projected into argo, where the submit pods run. Treat that copy as part of the platform, not a one-off kubectl get secret.

Argo Events did not exist yet. A Workflow submits Spark. That is the whole control plane in this era.

A standing cluster, not a driver per job

Spark-on-Kubernetes (a driver per job) or the Spark Operator would give a driver per job and no idle cluster. We kept the standing cluster because job rate is high and the path is one spark-submit from a Workflow. The cost is idle executors, cross-namespace SSL, and a driver that must bind a routable pod IP.

We run that cluster in data-engineering: Bitnami spark 9.4.1 (appVersion 3.5.6), vendored, RPC auth and SSL on.

The warehouse is one bucket per environment, {org}-data-lakehouse-{env} ({org} is the account prefix; {env} is dev or prod). Iceberg tables live under a single prefix, iceberg/. The standing cluster overlay talks to that prefix as s3://. Argo spark-submit jobs use s3a://. Same bucket, same prefix. Mixing prefixes is how you spend a day staring at empty schemas.

Nessie URI versions split the same way. The standing cluster and Trino talk /api/v2. The Argo submit jobs in this post still use /api/v1. Figure 1 does not care which API version committed the pointer. Credentials follow the submit path, not the catalog.

IRSA on the cluster, keys on submit

S3 credentials on the standing cluster are IRSA (IAM Roles for Service Accounts), not long-lived keys on the coordinator. CDK creates the service account in data-engineering and attaches object/list permissions on the warehouse bucket. Helm must use the same account name. A values-file typo looks like a random AccessDenied. Hadoop reads the projected web-identity token. This is the same IRSA pattern as Trino.

The standing-cluster catalog block looks like this:

sparkConfiguration:
  spark.sql.catalog.nessie: "org.apache.iceberg.spark.SparkCatalog"
  spark.sql.catalog.nessie.catalog-impl: "org.apache.iceberg.nessie.NessieCatalog"
  spark.sql.catalog.nessie.uri: "http://nessie.data-lakehouse.svc.cluster.local:19120/api/v2"
  spark.sql.catalog.nessie.ref: "main"
  spark.sql.catalog.nessie.warehouse: "s3://{org}-data-lakehouse-{env}/iceberg"
  spark.hadoop.fs.s3a.impl: "org.apache.hadoop.fs.s3a.S3AFileSystem"
  spark.hadoop.fs.s3a.aws.credentials.provider: "com.amazonaws.auth.WebIdentityTokenCredentialsProvider"

Master and workers run under that service account. There are no static keys on the coordinator. The submit path in argo is a different story.

spark-submit drivers run in argo, not under the Spark service account. Those Workflows export keys from a Secret mounted at /aws (access and secret are the file names in the mount) and force SimpleAWSCredentialsProvider. Static keys leak into the next namespace you copy them into, which is why they have to be projected into argo as platform.

A submit from Argo looks like this:

export AWS_ACCESS_KEY_ID="$(cat /aws/access)"
export AWS_SECRET_ACCESS_KEY="$(cat /aws/secret)"
 
spark-submit \
  --master spark://spark-master-svc.data-engineering.svc.cluster.local:7077 \
  --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions,org.projectnessie.spark.extensions.NessieSparkSessionExtensions \
  --conf spark.sql.catalog.nessie=org.apache.iceberg.spark.SparkCatalog \
  --conf spark.sql.catalog.nessie.catalog-impl=org.apache.iceberg.nessie.NessieCatalog \
  --conf spark.sql.catalog.nessie.uri=http://nessie.data-lakehouse.svc.cluster.local:19120/api/v1 \
  --conf spark.sql.catalog.nessie.ref=main \
  --conf spark.sql.catalog.nessie.warehouse=s3a://{org}-data-lakehouse-{env}/iceberg \
  --conf spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider \
  job.py

job.py is the Spark job. The sample is the submit envelope, not the transform. /api/v1 and s3a:// are the Argo path; /api/v2 and s3:// are the standing cluster. Do not mix them in one job.

IRSA on the submit pod (WebIdentityTokenCredentialsProvider on the driver) is the path we wanted for jobs too. It is not what those Workflows ship.

Ingress TLS port

Bitnami 9.4.1 hardcodes the Ingress backend to named port http. With security.ssl.enabled, the master Service exposes https and not http. The UI talked HTTP to a TLS-only Service until we changed both backend lines in templates/ingress.yaml to pick the TLS port when SSL is on.

# both backend port lines in templates/ingress.yaml
port:
  name: {{ ternary "https" "http" .Values.security.ssl.enabled }}

The snippet is those two port names, not a full Ingress object. Everything else in that chart, including values.yaml defaults, matches tag spark/9.4.1. Later Bitnami Spark on main still hardcodes http; this is not a backport. The patch is the whole Ingress story. It is not a reason to fork the rest of the chart.

When this applies

Use this job plane if you already have Nessie and one Trino reading Iceberg, you will schedule data work as Workflow CRs rather than as chart installs, and you can live with a standing Spark cluster plus one spark-submit from argo. Skip it if you need Spark-on-Kubernetes or the Spark Operator, Argo CD as the chart installer, or an event bus between jobs.

The constraint that stays is the split credential: IRSA on the cluster, static keys on submit from argo. Project those keys as platform. Do not treat the Ingress http/https ternary as optional. With SSL on, stock Bitnami 9.4.1 will not reach the UI.

Events is the next control-plane change, not this one: Argo Events decoupled ingestion from dbt.