Cosmos DB 429 at 1% Average RU: Retry, Don't Scale
Our average throughput use was under 1%, yet we hit as many as about 4,000 429s (Too Many Requests) in a single hour. This post answers a question many small teams run into: if Cosmos DB has capacity to spare, why is it throttling you, and should you raise RU/s? Our answer was no. The problem wasn't capacity but writes bunched into a few seconds, and we fixed it by cutting write concurrency and adding retries.
Why 429s at 1% average usage?
We use a single Cosmos DB for NoSQL container as both a work queue and a record store. It runs on manual (fixed) throughput of 600 RU/s. RU/s is the amount of request cost Cosmos will serve per second. Several Azure Functions and transcription jobs share that container.
The problem first showed up in the Functions error logs. The step that writes transcription results as per-line documents kept failing. Each finished transcription upserted hundreds of documents, eight at a time.
The hourly metrics told a split story. Most hours averaged 0.1–0.3% of 600 RU/s. But the moment a transcription finished, Normalized RU Consumption spiked to 100%. The limit is enforced per second, and we had only been looking at hourly averages.
My first read was "if we're getting 429s, we must be maxing out 600 RU/s." That was wrong. Across the day we used less than 1%; only a few seconds of each burst overflowed.
Where were the silent write failures hiding?
The worse part was that the failures were invisible. The code logged a failed write and moved on. The daily report email showed no warnings at all. The result files and the PostgreSQL records were intact, so only the Cosmos lines were missing.
Over the previous 24 hours, the lines for 18 transcriptions were missing entirely. When I used to drive this from the Claude window on my local PC, a red error jumped out immediately. Since moving it to Functions, the work runs on its own and I only see results by email. It's more comfortable, but this taught me that quiet failures are the scariest kind.
We made two changes:
- Cut concurrent upserts from 8 to 2.
- On a 429, wait 1s, 2s, 4s… up to 60s, and retry up to 12 times.
for attempt in range(12):
try:
container.upsert_item(doc)
break
except CosmosHttpResponseError as e:
if e.status_code != 429:
raise
time.sleep(min(60, 2 ** attempt))
With the fixed code we backfilled roughly 49,000 missing lines. 429s still appeared during the backfill, but the retries absorbed them and zero writes failed. Each line is written under the same ID, so there were no duplicates.
Raise RU/s, switch to autoscale, or leave it?
We had three options:
- Raise fixed RU/s.
- Switch to autoscale, which scales within a maximum hour by hour.
- Keep 600 RU/s and absorb bursts with retries.
Autoscale bills each hour at the highest level it reached, at 1.5x the fixed rate. It suits workloads that stay busy for hours, but our bursts last seconds, so the gain was small. We also looked at the free tier. It's a pure discount of 1,000 RU/s and 25 GB with no performance difference, which looked attractive. But you can only enable it when creating an account, and only one per subscription, so we'd have had to move the queue to a new account.
Before deciding, we asked Azure support exactly that: "Average usage is under 1% but we see bursts of 429s. Should we raise RU/s?" The gist of the answer: a 429 isn't an outage but a normal rate-limit signal, and if retries make the failures go away, there's no need to raise RU/s. So we went with option 3: keep it and retry. The trade-off is that writes can lag by tens of seconds during a burst, which is fine for our workload.
The support request taught us something too. On the Developer support plan, you can't create a case through the Support API; that requires Professional Direct or higher. Listing cases and reading replies does work via the API, so we create cases in the portal and read them via the API. The portal picking the problem type on its own, "Throughput and scaling / Request rate too large", was better than I expected.
Honestly, I first thought Cosmos was overkill for a team our size. Using it as a queue during the milestone period, the metrics surfaced a second-level problem fast enough that we found the cause quickly. Still, the default charts are hourly averages, which makes bursts easy to miss, and that remains annoying. Next we'll recheck 429 counts by hour and confirm real write failures stay at zero.
A short checklist for teams hitting the same thing:
- Watch the maximum of Normalized RU Consumption, not the hourly average.
- For bulk upserts, lower concurrency first.
- Add exponential-backoff retries for 429s.
- If any code logs failures and moves on, wire it into alerts.
- Know up front that the free tier can only be enabled at account creation.
Also today
- Correction: In Day 2 I estimated serverless GPU cost as GPU + vCPU + RAM. The actual bill showed only the T4 charge (about $0.263/hour); the monthly free grant absorbed vCPU and memory. Early in the month, while the grant lasts, real cost is lower than that estimate.
- Using
byas an alias in a Cosmos NoSQL query caused a syntax error because it's a reserved word, and one daily job never started. We renamed the alias and redeployed. - We set an Azure ML compute cluster's minimum nodes to 0. The Premium SSD OS disk attached to each node bills under Storage, and it disappears when the node scales down.