Every translation job used to go straight to Anthropic for processing. No throttle, no gate, no idea how many were in flight at once. This worked fine with a low number of users and a low number of simultaneous jobs. In fact, after posting Now I Get It! on Hacker News a couple of months ago, there was a spike in activity and I allowed 100 documents per day to be processed. This cap was hit quickly but it worked fine without a queue.
But I've now built this app to support thousands, if not tens of thousands, of users. At this scale, not having a job queue would be like customers going into a restaurant and shouting their orders directly to the cook. Eventually it would become so bottlenecked that nobody would get their meals.
Sizing with real numbers, not guesses
One of the best decisions I made was making the app free to use in the early days. That cost me a few hundred dollars but gave me a lot of data. And this data has allowed me to make intelligent decisions around architecture, pricing, and features. If I hadn't have done that, I would be releasing a product based on guesses.
The first draft of this plan assumed each job took around ninety seconds. That was a poor assumption so I ran an analysis against my database and computed actual distributions: median processing duration was 316 seconds -- about 5.3 minutes -- and input tokens had an extreme heavy tail, with p90 around 129K and p99 around 492K.
That shifted the binding constraint. At shorter durations, output-token-per-minute is the bottleneck. At five-minute durations, input-token-per-minute dominates, because each slot is consuming its entire input allocation for five straight minutes before freeing up. Running the math against Anthropic's Tier 4 limits, the safe ceiling is roughly ninety-four concurrent slots at p90 sizing.
What I built
Direct API calls were replaced with an SQS Standard queue between the upload-confirmation and PDF-processing halves of the pipeline. The queue's event source mapping has a MaximumConcurrency parameter that gates how many processor Lambda instances run at once. Default is 24 -- roughly a quarter of the ceiling -- with room to dial up to 70. Reserved concurrency on the Lambda is set two above the event source mapping's max to prevent transient throttling exceptions during scale-up bursts, which is a subtle AWS gotcha the plan's architecture review caught.
Retries without head-of-line blocking
The first draft used time.sleep() inside the processor for exponential backoff on Anthropic 429s. An architecture review flagged this as a head-of-line blocking risk: if all 24 slots are sleeping inside retry logic, the queue backs up and no API calls are being made even though we have capacity.
The fix was to re-enqueue the message back to the same queue with DelaySeconds set to the backoff interval. The original message returns success, SQS deletes it, the Lambda slot frees up for other work, and the retry message becomes visible after the delay. Three attempts total, with thirty-second, sixty-second, and two-minute backoffs, honoring any retry-after header when present. Retry exhaustion triggers a credit refund and surfaces the error to the user.
The re-enqueue pattern is cleaner than the sleep pattern in one more way: retries share capacity with new work. Under load, you want backoff delays and new uploads draining through the same 24-slot pipeline, not queued behind each other.
A single atomic write for two constraints
On the confirm side, the atomic transaction was trickier than I expected. The original plan called for two separate DynamoDB transaction operations on the user record: one to increment the in-flight counter, one to deduct credits. DynamoDB rejects transactions with two operations on the same item -- the whole point of a transaction is that one or the other happens first, and you can't linearize a single item against itself.
The fix was combining both into a single Update with a compound ConditionExpression: roughly, "the in-flight count must be below the per-user limit AND the balance must be sufficient." When the condition fails, a follow-up read tells the confirm Lambda which constraint was violated, so the user sees a 429 if they're over their concurrency limit and a 402 if they're out of credits. Two different error surfaces, one write.
The kill switch
I also added a kill switch - one SSM parameter, flipped from true to false, pauses all processing without dropping jobs. The processor Lambda reads the parameter (cached 60 seconds), calls ChangeMessageVisibility to push each in-flight message back five minutes into the future, and returns success. Messages stay in the queue. Users stay in the queued state. When the parameter gets flipped back to true, everything resumes automatically.
During an Anthropic outage, a cost investigation, or any "I'd rather stop the world for ten minutes than eat the next thousand failures" situation, this is the difference between draining the queue into errors and refunding everyone, and waiting it out.
The full deploy was eighteen files and about 1,900 lines of code -- new CloudFormation, a new retry module that's pure logic with no IO, refactored screening and generation paths, a substantially rewritten confirm and processor, frontend states for queued and retrying, and 342 passing tests with zero regressions.