nest.codes blog
Day 5 · · 4 min read

Container Apps job cold start: 8 min to 1.8 with Blob

Our GPU job runs in East US, but the container image it needed lived in a registry in Seoul. This post answers a simple question: how do you speed up a Container Apps job that is slow because its registry sits in another region? Our answer wasn't ACR geo-replication. We moved to a small public base image plus same-region Blob Storage, and startup dropped from about 8 minutes to about 1.8.

Why did Advisor suggest geo-replication?

Today I went through all 70 Azure Advisor recommendations one by one. One of them was geo-replication for Azure Container Registry (ACR). Our speech transcription job is a Container Apps serverless GPU job that starts a few times a day in East US. Every start pulled a multi-GB image across from the Seoul registry.

We already ran into big-image cold starts in Day 2. Back then we only batched runs to cut how often it happened. We never questioned where the image lived. And the minutes spent pulling it are billed as GPU job time.

Following Advisor would have been easy. But geo-replication needs the Premium tier, and each replica costs about as much as another registry (roughly $50 a month at public prices).

Replicate, add a registry, or use Blob?

We had three options:

I leaned toward replication at first. Looking again, Advisor had pointed at a symptom, and the cause was our design. If the GPU job runs in East US, its inputs should live in East US too. So we picked option three.

What does the new setup look like?

I handed the restructuring to a Claude Code subagent and checked the result myself.

# Concept of the start command (names generalized)
token = get_managed_identity_token("https://storage.azure.com/")
boot = http_get(f"{BLOB_URL}/code/boot.py", bearer=token)
exec(boot)  # boot.py pulls the bundle, model and code, then starts the job

Three test runs finished and wrote their results correctly. We kept the old image and registry for rollback. ACR stays anyway, since our web container still uses it.

There is a trade-off. When everything was baked into one image, it was obvious what version ran. Now we rely on bundle names and boot logs to trace that. Honestly, this setup turned out cleaner than I expected. Shipping code now means uploading to Blob instead of rebuilding a large image, so deploys got faster too.

One setting caught before the first scheduled run

In the evening I had a review agent go over the automation once more. It found something urgent. The Functions app that launches the job still had an app setting pointing at the old image. Our code reads that setting before its own default.

Left alone, tomorrow's first scheduled run would have used the old image with the new bootstrap, a combination we never tested. We confirmed the deployed code was the new version, then deleted that single app setting.

That one rattled me a bit. The code was right and the tests passed, yet an app setting was quietly overriding the code. Now that the work runs on Azure instead of in front of me, this kind of thing is invisible. I won't fully relax until tomorrow morning's report email lands.

Correction: the $41 per day in Day 4 was the total bill for 11-16. We now treat about $33 as a normal day. The earlier number wasn't wrong; it used a different basis.

If you're dragging something heavy across regions, ask whether it really needs to be inside the image before you pay for replication.

Also today

#azure#azure-container-apps#azure-container-registry#azure-blob-storage#azure-advisor
All posts in this series
  1. Prologue · Azure Credits via NVIDIA Inception: Why We Moved
  2. Day 1 · Azure PostgreSQL Migration: Quota Isn't SKU Availability
  3. Day 2 · Azure Container Apps Serverless GPU: Real T4 ASR Cost
  4. Day 3 · Cosmos DB 429 at 1% Average RU: Retry, Don't Scale
  5. Day 4 · Cosmos DB for Transcript JSON: One Container, Not Two
  6. Day 5 · Container Apps job cold start: 8 min to 1.8 with Blob
  7. Days 6–7 · Azure service usage audit: proving each one does real work