Per-persona multi-language support (translate-for-retrieval + reply-in-user-language)

[rajivml]

Summary

Adds a per-persona toggle that makes Darwin work for non-English (Japanese / Chinese / Korean) users without changing the indexing pipeline, embedding model, or any infra. The flag is opt-in per assistant — English-only assistants pay nothing extra, and customers running primarily English traffic aren't impacted.

When the flag is on:

1. Retrieval — a non-English query is translated to English before hitting Vespa, so the existing English-corpus index returns the right docs (resolution: persona flag → MULTILINGUAL_QUERY_EXPANSION env var → off).
2. Answer language — after the answering LLM produces a (possibly English) reply, a small second-pass LLM call translates the answer back into the user's original language. Citation markers ([1], [2], [[1]](url)), URLs, and code blocks are preserved verbatim.
3. Chat session naming — chat titles follow the user's language too.

Both code paths are covered:

• Chat UI (/chat) — chat/process_message.py
• Slack one-shot (the path the slackbot listener uses) — one_shot_answer/answer_question.py

What was implemented

Backend

• Persona.multilingual_query_expansion: bool — new column with default false. Alembic migration a3f1d7c4e9b2_persona_multilingual_query_expansion.py.
• CreatePersonaRequest / PersonaSnapshot carry the field through the persona admin API.
• upsert_persona / create_update_persona accept and persist the flag.
• PromptConfig carries the flag forward so prompt builders can decide whether to append LANGUAGE_HINT.
• SearchPipeline reads the persona flag and (when on) sets multilingual_expansion_str=\"English\" so retrieval translates to English before search.
• citations_prompt.py and quotes_prompt.py source the language-hint from prompt_config, with the global env var as fallback.
• chat_session_naming.get_renamed_conversation_name accepts use_language_hint; the rename endpoint fetches the chat-session's persona and forwards it.
• chat/multilingual_translation.py (new module): detect_query_language (Unicode-script heuristic), language_name, translate_answer_to_language (LLM call that preserves citations, URLs, code blocks; falls back to English on failure).
• chat/process_message.py — when persona has the flag on AND the query is non-English: buffer DanswerAnswerPiece tokens during streaming, run the translate pass after stream-end, emit translated text as a single piece, persist translated text in the DB. Other packets (citations, tool responses) flow in real time.
• one_shot_answer/answer_question.py — same buffer-and-translate logic for the Slack path. CitationInfo packets keep flowing during the buffer, so the slackbot's existing 5-attempt citation-required retry loop works unchanged.

Frontend

• Persona TS interface, PersonaCreationRequest / PersonaUpdateRequest, and buildPersonaAPIBody carry the field.
• Assistant editor: a BooleanFormField checkbox in the Misc section labeled "Enable multi-language support" with subtext explicitly noting the cost (+1 LLM call per non-English query).

Drive-by fixes (rolled into this branch by request)

• web/src/app/admin/bot/SlackBotConfigCreationForm.tsx — fix silent-submit on the Slack-bot config create form. The Yup schema unconditionally required curated_response_config.response_message, but the matching input is only rendered when the integration toggle is on. Result: with the default toggle off, the field was empty, validation failed silently, and Create did nothing. Fixed by gating the validation with .when(...) to mirror the jira_config pattern.
• web/src/components/table/DragHandle.tsx — destructure isDragging before spreading onto the DOM <div>, silencing React's "unknown DOM attribute" warning that fired on every PersonasTable render.

What was tested

backend/scripts/test_multilanguage_e2e.py — end-to-end smoke test driving the real local stack (Postgres + Vespa + the configured GenAI provider). Five phases:

1. Setup — create test connector + cc-pair, seed 3 English docs about a unique fictional brand ("Zorblax") into Vespa via the real indexing pipeline, create two test personas (flag on / flag off).
2. English baseline — sanity check that retrieval and entity-in-answer work for English queries against the seeded corpus.
3. Non-English (chat-UI path) — for ja / zh / ko, hard-asserts (a) retrieval brought back the expected English doc, (b) the answer contains the factual entity (numerals / proper nouns survive translation), (c) the final answer text is in the user's language.
4. Control persona — same non-English queries against the flag-OFF persona; logged informationally to show the contrast.
5. Slack one-shot path — drives get_search_answer (the entry point the slackbot listener uses) with the same hard contract as Phase 3, proving the Slack flow honors the flag end-to-end.

The test is re-run safe: cascading SQL teardown handles chat_message__search_doc, tool_call, chat_feedback, and document_retrieval_feedback FK dependents before deleting chat_message / chat_session / persona.

Local results

Stability run before option C (post-translation pass) was wired:

• 6/6 retrieval contract: works every time
• 0/9 non-English answers came back in the user's language — gpt-4o-mini in the gateway routinely ignored LANGUAGE_HINT when context was English-heavy

After option C (post-translation pass):

• 27/27 hard assertions PASS (12 retrieval + 12 entity-in-answer + 3 English-baseline)
• 9/9 non-English answers in the user's language via the chat-UI path
• 3/3 non-English answers in the user's language via the Slack one-shot path (Phase 5)
• Control persona (flag off) still answers in English — confirms the post-pass only fires for opted-in personas

Re-run on this branch: please run cd backend && PYTHONPATH=$(pwd) python scripts/test_multilanguage_e2e.py --yes against your local stack to verify before merge.

Trade-offs / things to know

• Streaming UX for non-English replies: token-by-token streaming is suspended for opted-in personas because we buffer the answer and translate it before emitting. The user sees the whole translated answer at once after one extra LLM round-trip. English replies are unaffected.
• Cost: one extra LLM call per non-English query (translate). Only on personas with the flag on. English personas / queries pay zero overhead.
• Slack bot citation gate is preserved: the [1]/[2] citation retry loop still works because the translation prompt instructs the LLM to keep markers verbatim, and CitationInfo packets flow in real time even when answer pieces are buffered.
• LLM non-determinism: occasional borderline queries (e.g. JA office printer with multiple similar docs in the corpus) can have the LLM hedge on which doc to cite — observed in 1 of 6 stability runs. Not a wiring issue; would benefit from a stronger model.
• Resolution precedence: persona flag → MULTILINGUAL_QUERY_EXPANSION env var → off. Existing global-env-var deployments are untouched.

Process bounce required after merge + deploy

Per CLAUDE.md ("Modify a SQLAlchemy model → run alembic upgrade head, then bounce API server + background jobs"):

1. alembic upgrade head from backend/ (adds the new column with server_default false).
2. Bounce dapi (api-server) — new ORM mapping + new module imports.
3. Bounce dbe (background jobs).
4. Bounce dsl (Slack listener) — needed for the Slack post-translation path to take effect.

Test plan

• Pull, run alembic, bounce all three services
• Run python scripts/test_multilanguage_e2e.py --yes against the dev stack — expect exit 0 with all 5 phases green
• In the chat UI: pick a multilingual-flagged assistant, ask a question in Japanese; expect a Japanese answer with [1]/[2] citations preserved
• In a Slack channel bound to the same persona: post the same question; expect a Japanese reply
• Sanity-check English flow on a non-flagged persona — should still stream tokens normally
• Smoke-test the drive-by fixes: open /admin/bot/new, click Create with all fields default — should now show backend validation errors instead of doing nothing silently. PersonasTable should no longer log the isDragging warning in dev console.

🤖 Generated with Claude Code