Fix fallback model leak; add configurable fallback model per provider#588
Merged
Conversation
When a stage's primary provider was unavailable, the model id resolved against the primary (e.g. codex's 'codex-configured-default') was carried verbatim onto the fallback provider — so the run logged 'LM Studio/codex-configured-default' and 400'd because the fallback has no such model. stageRunner and the toolkit createRun now re-resolve the model against the fallback provider instead of forwarding the primary's. Adds a 'fallbackModel' field on providers (schema + createProvider + UI selector beside Fallback Provider) so a fallback can pin both provider and model; getFallbackProvider returns the configured model and both the pre-flight and runtime fallback paths run it.
…RunOnce; honor pin in agent lifecycle The createRun swap inside executeProviderRunOnce (the common path for callers that don't pre-create a runId) still re-resolved the model against the primary, leaking codex-configured-default onto the fallback. Re-resolve against the fallback using the surfaced fallbackModel, mirroring stageRunner. Also forward the task-level fallback model through the PortOS providerStatus wrapper and let a provider-/task-level fallbackModel pin override per-task model selection in agentLifecycle, so the feature applies to CoS agent runs too.
…don't pin onto user-override provider POST /api/runs executed API/TUI fallbacks with the original request model (resolved against the benched primary) and ignored the pin for CLI fallbacks. Derive runModel from createRun's usedFallback/fallbackModel so a fallback swap runs the fallback's model across all three provider types; non-fallback runs are unchanged. In agentLifecycle, a task-metadata provider override could replace the fallback provider while leaving fallbackModelPin set, applying the fallback's pinned model to the user-chosen provider. Clear the pin on that override.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Editorial Review with Codex CLI was failing with
400 Bad Request, logging🤖 AI run [pipeline-manuscript-completeness]: LM Studio/codex-configured-default. Root cause: when a stage's primary provider was unavailable (Codex had been benched after an OpenAI content-safety refusal), PortOS fell back to another provider but carried the model id it had resolved against the primary (codex-configured-default) onto the fallback. LM Studio has no such model → 400; Claude Code rejected it the same way ("issue with the selected model (codex-configured-default)"). So a perfectly healthy fallback (Claude Code) was being knocked out by a leaked model name.This PR fixes the leak and adds the ability to pin both a fallback provider and a fallback model.
Fix: no more model leak across fallback
stageRunner.runStagedLLMnow re-resolves the model against the fallback provider instead of forwarding the primary's already-resolved concrete model.createRundoes the same for its pre-flight provider swap, so the run record and the first🤖 AI run …log line show the correct model.runPromptThroughProvider's runtime-retry path honors the configured fallback model instead of always sendingmodel: undefined.Feature: choose a fallback model, not just a provider
fallbackModelfield on providers (createProvider+ ZodproviderSchema; validated on POST and PUT, returned viasanitizeProvider).providerStatus.getFallbackProvider()now returns{ provider, source, model }— the configuredfallbackModelfor a provider-level fallback, a task-level model when supplied, ornull(use the fallback's own default). It is never the primary's model.Fallback: <name> (<model>).Test plan
server: full suite green — 8813 passed, 7 skipped. New coverage:providerStatus.test.js— asserts the configuredfallbackModel(and task-level model) ride along on the returned object; system-priority picks returnnull.promptRunner.test.js— asserts a pinnedfallbackModelreaches the fallback run and is neither the primary's model (the leak) nor the fallback's own default (the pin must win).client: full suite green — 717 passed.providerSchema) and PUT (providerSchema.partial()) both validatefallbackModel, andsanitizeProviderreturns it to the client.