Yesterday I added a job queue. Today I uploaded a 244-page document (20.7 MB, 487,000 input tokens) to exercise the new pipeline at a realistic upper bound and got "Processing timed out" -- except when I checked the result page, it had actually generated successfully and was sitting in S3 ready to view.
The diagnosis
The queue refactor included a new staleness detector in the status-check Lambda. On every poll from the frontend, the Lambda reads a progress timestamp on the job row. If it's been more than 300 seconds (5 minutes) since the last update, the detector declares the job dead and marks it timed out. The reasoning was sound: if the processing Lambda crashed mid-generation and left the job stuck on processing forever, someone needs to eventually mark it as failed so the user isn't staring at a spinner.
Good idea for zombie jobs. Bad idea with no heartbeat.
The code only updated the progress timestamp at stage transitions -- upload parsed, generation started, HTML post-processed. In between, the processing Lambda would sit in a single streaming call to Claude for minutes at a time with no DynamoDB activity. A normal 20-page paper generates in two or three minutes total and never bumps up against the 5-minute window. A 244-page model card with half a million input tokens takes six and a half minutes in one uninterrupted streaming call. No progress updates, no heartbeat, and at five minutes the detector fires and kills the job. Meanwhile the generation finishes successfully a minute later. The frontend has already shown the error.
The fix
A background heartbeat thread. While the stream is open, a new thread inside the generation module runs every 60 seconds and updates the progress timestamp regardless of whether text chunks are currently arriving. Streaming APIs are bursty -- long pauses between chunks don't mean the connection is dead, they mean the model is thinking. As long as the HTTP connection is open and the SDK hasn't raised an exception, the job is alive. The heartbeat is the observable proof of that, and at 60-second intervals the 5-minute threshold is never in danger.
The boto3 thread-safety gotcha
The first deploy of the heartbeat broke immediately. Five heartbeats fired, five heartbeats crashed with 'NoneType' object has no attribute 'update'. The heartbeat was calling into the DynamoDB Table resource from boto3 -- the same resource the main thread was using for other writes.
boto3 resources are not thread-safe. They maintain internal state that gets corrupted when two threads touch them at once. The low-level boto3 client is thread-safe, but the higher-level Table resource is not, and the docs say this if you look, which I hadn't.
The fix was a one-line scope change: switch the heartbeat's DDB call to go through the low-level client instead of the Table resource, plus a few extra lines to marshal the values into DynamoDB's wire format (the low-level client doesn't do the automatic type coercion that the resource does). The module already had both a client and a resource initialized side by side, so it was mostly a matter of picking the right one for the right thread.
Verified
Ran the same 244-page paper through again. Generation took 6 minutes 30 seconds, well past the old 5-minute threshold. Zero heartbeat errors in the logs. Zero stale-timeout log lines. Clean completion with no error fields. The status Lambda polled the job about forty times during generation and saw a fresh progress timestamp on every check.
Four files changed, +94/-36 lines, 120 tests passing.