fix: recover from truncated JSON in tool call arguments

[giulio-leone]

Problem

LLMs (notably GPT-4.1 on Azure) sometimes return truncated JSON in streaming tool call arguments. When the streaming response is cut off before the JSON is complete, from_json() raises ValueError: EOF while parsing a string, causing tool calls to fail entirely.

From issue #4240, a real-world example of truncated output:

{"success":true,"reason":"The message explicitly asks the user to confirm if their name is John Doe"}

Was received as:

{"success":true,"reason":"The message explicitly asks the user

This affects ~10% of tool calls for some users, particularly with Azure GPT-4.1.

Solution

Add a JSON repair fallback in prepare_function_arguments():

1. Fast path unchanged: pydantic_core.from_json() is tried first
2. On ValueError: Attempt to repair the truncated JSON by closing open string literals, brackets, and braces
3. Log warning: When repair succeeds, log for observability
4. Re-raise: If repair also fails, raise with a descriptive error

The repair handles the most common truncation patterns:

• Unfinished string values (missing closing quote)
• Missing closing braces/brackets
• Combinations of the above

Tests

Added TestTruncatedJsonRepair with 4 test cases:

• Truncated string value → repaired and parsed
• Missing closing brace → repaired
• Valid JSON → unchanged (no regression)
• Completely invalid JSON → still raises ValueError

Fixes #4240

[giulio-leone]

Great catch on the trailing backslash edge case! Fixed in 1cc9d4a — now strips an unescaped trailing backslash before appending the closing quote, so {"key": "value\\ correctly repairs to {"key": "value"} instead of producing an escaped-quote.

[giulio-leone]

Fixed in 5a78724 — replaced endswith() check with proper trailing backslash counting using rstrip('\\') + len() to correctly handle 3+ consecutive backslashes. Added a test for this edge case.

[rajveerappan]

@giulio-leone wouldn't patching the JSON in this way cause incorrect arguments to be sent to the tools? in which case you would need to retry the call to the LLM so might as well just do that when you detect invalid JSON?

e.g. {"arr": [{"a": 1 might have been a truncation of {"arr": [{"a": 1, "b": 2}] but this would invoke the tool with {"arr": [{"a": 1}]

[giulio-leone]

Valid concern — you're right that repair can produce semantically different arguments (e.g. {"arr": [{"a": 1}]} instead of {"arr": [{"a": 1, "b": 2}]}).

The tradeoff is: partial-but-structurally-valid JSON lets the tool execute with whatever was received, while a hard failure loses the entire call. In practice, LLM truncation typically occurs at the end of long arguments (e.g., large arrays, long strings), so the repaired result is often semantically close enough for the tool to succeed or return a meaningful error.

That said, a retry-on-invalid-JSON strategy is also valid. The two approaches could be complementary: retry first, and only repair as a last resort if retries are exhausted or not configured. Happy to add a configurable behavior if that aligns better with the project's philosophy.

[giulio-leone]

Great point, @rajveerappan — you're right that silent repair can produce semantically incorrect arguments (e.g. a truncated array element missing fields).

The motivation was to avoid a hard crash for the agent when the LLM returns truncated JSON (which happens in streaming scenarios), since the alternative is ValueError with no recovery. But I agree the trade-off is non-trivial.

A few options:

1. Repair + warn: Keep the repair but emit a warning log so the caller knows the arguments may be incomplete. This at least makes the behavior observable.

2. Repair + retry signal: Return a flag/exception indicating the args were repaired, so the caller can choose to retry the LLM call instead.

3. Just retry: As you suggest, detect invalid JSON and retry the LLM call directly — simpler and more correct, though it adds latency.

I'm happy to pivot to option 3 (retry) if that better fits the project's philosophy, or to option 1 (repair + warning) as a middle ground. What's your preference?

[giulio-leone]

@rajveerappan You raise a valid point — the repaired JSON may semantically differ from the LLM's intent.

However, I think repair-then-proceed is the right default for this scenario:

1. The alternative is worse: Without repair, the tool call fails entirely with a ValueError. The user gets nothing — no partial result, no chance for the agent to continue. At least with a repaired call, the tool may succeed and the agent continues.

2. Retrying has the same problem: LLM retries aren't guaranteed to produce complete JSON either (especially under token limits or rate-limiting). The truncation in "ValueError: EOF while parsing a string" during tool calls #4240 affects ~10% of calls on Azure GPT-4.1 — retrying blindly could compound latency and cost without fixing the root cause.

3. Pydantic validation catches bad arguments: Since prepare_function_arguments validates the repaired JSON against the function's type signature via Pydantic, structurally wrong arguments (e.g., missing required fields) will still raise ValidationError before the tool executes. The repair only helps when the truncation is in a terminal value position (like a string being cut off).

4. Observability is built in: The logger.warning() call ensures repaired calls are visible in logs, so users can monitor the frequency and decide if they need upstream fixes (e.g., increasing max_tokens).

That said, if the livekit team prefers a retry-first approach, I'm happy to restructure this as: (1) try repair → (2) validate → (3) if validation fails, raise so the caller can retry. The current design already does this — if repair + validation fails, the original ValueError propagates.

[giulio-leone]

Good point. The JSON repair is a best-effort recovery for malformed tool arguments that would otherwise hard-fail. It only patches obvious syntax issues (missing quotes, trailing commas) without changing semantic content. If the model returned genuinely wrong arguments, those would still be sent, but without repair the call would fail with a parse error anyway. If you prefer a retry-first strategy, I can restructure.