Designing Failure-Tolerant Asynchronous Work

Building an asynchronous TTS endpoint sounds straightforward: accept some text, return the job ID, generate the audio in the background, and upload the result to a presigned URL.

The interesting part starts when things fail:

• What happens if Redis restarts while a worker is processing a job?
• What happens if the service restarts?
• Who should be responsible for retrying when the presigned URL expires?

This post explores one deliberately simple design for answering those questions without turning a small asynchronous service into a distributed-systems science project 😉.

tl;dr

• 202 HTTP status code means: "the request has been accepted for asynchronous processing", not "this operation will definitely complete".
• Redis is used as ephemeral job state, so losing Redis can legitimately make a previously accepted job unqueryable which is fine.
• The client owns the output destination. It supplies the presigned URL; the TTS service simply uploads to it.
• If the job disappears, the client can query the status endpoint, receive 404, generate a fresh presigned URL, and submit the request again.
• The job ID is provided by the client.
• The service should make a best effort to stop processing jobs whose authoritative Redis state has disappeared.
• There is an unavoidable race between checking Redis and uploading to an external storage service. The design should make that race acceptable rather than pretending it can be made atomic.
• The client is responsible for choosing a presigned URL with a suitable lifetime. The TTS service should measure how long TTS generation takes and whether uploads fail, so you can detect if the service is routinely taking longer than the URLs remain valid.

The Service

Let's start with a deliberately small API. The client sends:

POST /synthesize
Content-Type: application/json

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "uploadUrl": "https://storage.example.com/...",
  "text": "Hello from the TTS service!"
}

The service immediately responds:

HTTP/1.1 202 Accepted
Content-Type: application/json

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "ACCEPTED"
}

HTTP 202 Accepted is specifically intended for requests that have been accepted for processing but whose processing has not completed yet. The important part is that 202 is intentionally non-committal.

Note

"Non-committal" does not mean that 202 is a bad or unreliable response.

It means that the HTTP response does not commit the server to a particular eventual outcome.

The important thing here is that the ID comes from the client. The service does not create another internal identifier that the client has to keep track of. The same ID is used when asking about the request:

GET /synthesize/550e8400-e29b-41d4-a716-446655440000

For example:

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "IN_PROGRESS" // "SUCCEEDED" | "FAILED"
}

The output location is a completely separate concern. The client decides where the generated audio should go by providing the presigned URL.

Client handles the retry mechanism and it also gets to decide where it wanted to store the final result. So client can generate presigned URLs for the same ID to upload them to the same place, effectively overriding the old results.

But this is a decision that client has to make!

System Boundaries

The benefits of this system design:

• The TTS service doesn't need to know anything about the client's storage architecture.
• Simpler than having an outbox/inbox pattern. Instead of saying we should NOT reprocess the same job twice we say we do NOT care. We process it anyway. But of course this is possible since we are offloading the whole logic of retry and what if we override a newer version of text with an older one to the client which does make sense to me.

Tip

Overriding is fine even if you need a durable audit trail. I mean client can handle that on its own. The TTS service should NOT concern itself with that.

Why Redis?

It's not necessarily being used as durable storage here. It's being used as convenient shared job state. For example: job:{id}

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "text": "Hello from the TTS service!",
  "uploadUrl": "https://storage.example.com/...",
  "status": "IN_PROGRESS"
}

And since we said it is NOT a durable storage that means we are again offloading the responsibility of retrying to the client. In other words if Redis restarts itself and we loss the jobs. Then when client polls the state of a job:

GET /synthesize/550e8400-e29b-41d4-a716-446655440000/status

They will get a 404, so now they know something went wrong. They will just send a new request. And they can even invalidate the previous presigned URL.

So even if the TTS service has the audio files we just drop them since we lost their state in Redis. This way later you can fortify your Redis/infrastructure.

💡 Note

Like I said I am giving more control to the client, so in this case as I will explain in a bit we have a 2 days TTL for each job. That means client can ask about the same job up to two days before getting a 404.

What this entails is that the client will stop long-polling the TTS service as soon as it got a reply with state: "SUCCEEDED". Client can be even more defensive and check with its object storage to check if the file exists. But that is a different concern and not what we should think at the moment.

Solve one issue at a time.

If you're asking yourself what about distinguishing between:

1. The job never started.
2. The job was processing when Redis disappeared.
3. The TTS was generated.
4. The upload succeeded.
5. The upload failed.

I have you covered with OTel and nice logging we can have that info. Also we care about this info more as a developer and customer does not really care about it. The only thing which matters to them is the audio file.

And we already have a robust system which can support regenerating a TTS.

BTW we will be having a TTL of 2 days by default which can be configured for the TTS service via an env variable for each job we store in Redis to prevent cluttering it with data and running out of memory at some point.

Presigned URL Expiration

A presigned URL is intentionally temporary. The client creates the URL, so the client is responsible for choosing an appropriate lifetime.

This way we have two benefits:

1. If our TTS service if takes too long to generate and upload the audio file we can look at the logs (OTel is crucial) and see the issue. For example it might be our service whom is too slow or maybe our code is implemented poorly.
2. Client gets to choose different TTL for each presigned URL based on how many words we have, load, etc. In other words we have more room to maneuver.

Observability

For an asynchronous service like this, I'd want traces and metrics around at least these phases:

request accepted
    │
    ▼
job queued
    │
    ▼
job started
    │
    ▼
TTS generation started
    │
    ▼
TTS generation finished
    │
    ▼
upload started
    │
    ▼
upload finished

And when I say I want traces and metrics I am talking about:

• Queue wait time.
• TTS generation duration.
• Upload duration.
• Total job duration.
• Job failures.
• Redis failures.
• Upload failures.
• Expired/rejected presigned URLs.
• Number of jobs lost because job state disappeared (honestly this might be more traceable on the client side).

And this is where OTel really shines.

What About Duplicate TTS Generation?

Suppose the first attempt successfully generated the audio, but the client never learned about it for whatever reason. And client calls TTS service with the exact same text.

The system may do duplicate work. IMO TTS generation might not be so cheap if the text is long enough. But the thing is the failure is sufficiently rare IMO.

This is why I believe this is an acceptable trade-off. You don't necessarily need to solve every theoretically possibility.

┌────────────────────────────┐
| more guarantees            |      ┌───────────────────────────┐
|        │                   |      | simpler system            |
|        ▼                   |      |       │                   |
| more state                 |      |       ▼                   |
| more coordination          |  VS. | occasional duplicate work |
| more complexity            |      |       │                   |
| more failure modes         |      |       ▼                   |
|        │                   |      | client-driven retry       |
|        ▼                   |      └───────────────────────────┘
| fewer duplicate operations |
└────────────────────────────┘

When This Design Stops Being Good Enough

This approach isn't universal. For example if:

• Duplicate work is unacceptable:
  • For example the TTS generation is expensive. Expensive can mean scenarios such as:
    • Jobs can run for hours.
    • We make external API calls.
    • It is computationally intensive.
  • Or imagine your are charging customers for each TTS generation, and if we do that for no reason they will soon enough realize it.
• Every accepted request must eventually complete.
• Clients cannot safely retry.
• The output operation has irreversible side effects.
• You need exactly-once-like business semantics.

GitHub

So I am in the middle of adding this to my piper-tts-rest-api project. But it is still WIP. After I finished the work I will update this post.