(openai responses): add websocket connection pool

[tinalenguyen]

when there are two parallel streams, restart and send the full context on the next request
each request will use its own websocket connection (edit: and each WS connection is independent of response IDs)

[chenghao-mou]

But we can't reuse the same WS connection for different conversations, right? Even with store=True, the server has to rehydrate the chat history if the ws connection expects a different previous_response_id.

[tinalenguyen]

I could have worded my comment better, but each websocket connection is independent of previous_response_id

[chenghao-mou]

But the server-side in-memory cache for each connection seems very dependent:

On an active WebSocket connection, the service keeps one previous-response state in a connection-local in-memory cache (the most recent response). Continuing from that most recent response is fast because the service can reuse connection-local state. Because the previous-response state is retained only in memory and is not written to disk, you can use WebSocket mode in a way that is compatible with store=false and Zero Data Retention (ZDR).

If a previous_response_id is not in the in-memory cache, behavior depends on whether you store responses:

With store=true, the service may hydrate older response IDs from persisted state when available. Continuation can still work, but it usually loses the in-memory latency benefit.
With store=false (including ZDR), there is no persisted fallback. If the ID is uncached, the request returns previous_response_not_found.

Do we see previous_response_not_found when store=false if you reuse the same connection for two conversations?

[tinalenguyen]

Ah we don't use previous_response_id when store=False, we just send the entire context in that case (relevant line)

[chenghao-mou]

Gotcha, that makes sense.

[chenghao-mou]

mostly lgtm. Tested it locally and it worked well.

[chenghao-mou]

nit: the name seems misleading in a sense that it sends the request but also receives the responses. Maybe process_request?

[tinalenguyen]

how about generate_response?

[chenghao-mou]

lgtm. One nit about APIStatusError construction.

[chenghao-mou]

raw_msg has both .data and .extra we can leverage:

APIStatusError(
                        "AssemblyAI connection closed unexpectedly",
                        status_code=ws.close_code or -1,
                        body=f"{msg.data=} {msg.extra=}",
                    )