feat(ts-sdk): add modelOverride option to all evaluators by adnanrhussain · Pull Request #34 · learning-commons-org/evaluators

adnanrhussain · 2026-04-22T23:50:02Z

Summary

Adds modelOverride: { provider, model } to BaseEvaluatorConfig, letting callers swap the LLM provider and model for any evaluator without touching evaluation logic
Adds Anthropic as a supported provider (Provider.Anthropic, anthropicApiKey)
When modelOverride is set, only the API key for the override provider is required; default key validation is bypassed
Refactors provider label tracking: LLMProvider interface now has a readonly label property computed at construction time, eliminating the per-evaluator modelLabel field and getModelLabel helper
Telemetry: model_override: boolean flag when override is active; provider field reflects actual model used in all evaluators including TextComplexityEvaluator

Test plan

Instantiate any evaluator with modelOverride set — confirm only the override provider's key is required, default keys are not
Check result.metadata.model reflects the override provider and model (e.g. "anthropic:claude-sonnet-4-6")
Instantiate with modelOverride and a missing override key — confirm ConfigurationError is thrown
Run with telemetry enabled and override active — confirm model_override: true appears in the event and provider reflects the override model

codecov · 2026-04-22T23:54:16Z

Codecov Report

❌ Patch coverage is 82.85714% with 12 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
sdks/typescript/src/providers/ai-sdk-provider.ts	0.00%	9 Missing ⚠️
sdks/typescript/src/evaluators/base.ts	96.96%	1 Missing ⚠️
sdks/typescript/src/evaluators/text-complexity.ts	50.00%	1 Missing ⚠️
sdks/typescript/src/evaluators/vocabulary.ts	90.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

Copilot

Pull request overview

This PR extends the TypeScript SDK evaluators to support a modelOverride (provider + model) so callers can swap the underlying LLM without changing evaluator logic, while also adding Anthropic as a supported provider and improving provider/model labeling in metadata and telemetry.

Changes:

Add Provider enum + ModelOverride to BaseEvaluatorConfig, including key-validation rules that prioritize the override provider.
Refactor provider/model tracking to use LLMProvider.label (computed at provider construction) and remove per-evaluator label helpers.
Update telemetry schema/docs to include model_override, and update unit tests/docs to cover override behavior and new metadata fields.

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
sdks/typescript/src/evaluators/base.ts	Adds `Provider`, `ModelOverride`, override-aware key validation, telemetry flagging, and `createConfiguredProvider` helper
sdks/typescript/src/providers/base.ts	Extends `LLMProvider` interface with a canonical `label`
sdks/typescript/src/providers/ai-sdk-provider.ts	Implements `label` for `VercelAIProvider` at construction time
sdks/typescript/src/evaluators/vocabulary.ts	Uses `createConfiguredProvider`, switches metadata to `defaultProviders`, and derives model/provider labels from provider instances
sdks/typescript/src/evaluators/sentence-structure.ts	Uses a single shared provider instance + `label` for telemetry/metadata
sdks/typescript/src/evaluators/smk.ts	Uses `createConfiguredProvider` and `label` for telemetry/metadata
sdks/typescript/src/evaluators/conventionality.ts	Uses `createConfiguredProvider` and `label` for telemetry/metadata
sdks/typescript/src/evaluators/grade-level-appropriateness.ts	Uses `createConfiguredProvider` and `label` for telemetry/metadata
sdks/typescript/src/evaluators/text-complexity.ts	Updates metadata to `defaultProviders` and ensures telemetry `provider` reflects override when active
sdks/typescript/src/evaluators/index.ts	Re-exports `Provider` and `ModelOverride`
sdks/typescript/src/index.ts	Exposes `Provider` and `ModelOverride` in the public SDK entrypoint
sdks/typescript/src/telemetry/types.ts	Adds `model_override?: boolean` to telemetry event type
sdks/typescript/docs/telemetry.md	Documents the new `model_override` telemetry field
sdks/typescript/README.md	Documents `modelOverride`, adds example usage, and updates config/options docs
sdks/typescript/tests/unit/evaluators/validation.test.ts	Adds coverage for override key validation and provider factory params
sdks/typescript/tests/unit/evaluators/vocabulary.test.ts	Adds override metadata.model assertions; updates provider mocks to include `label`
sdks/typescript/tests/unit/evaluators/sentence-structure.test.ts	Updates tests for single shared provider and new mock shape
sdks/typescript/tests/unit/evaluators/smk.test.ts	Adds override metadata.model assertion; updates provider mocks
sdks/typescript/tests/unit/evaluators/text-complexity.test.ts	Updates metadata assertions to `defaultProviders` and provider mocks
sdks/typescript/tests/unit/evaluators/conventionality.test.ts	Updates metadata assertions and provider mocks
sdks/typescript/tests/unit/evaluators/grade-level-appropriateness.test.ts	Updates provider mocks to include `label`

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

czi-fsisenda

Looks good!
One non-blocking general comment.

czi-fsisenda · 2026-05-04T23:19:43Z

        'VercelAIProvider does not support custom type. Use config.customProvider directly.'
      );
    }
+    this.label = `${config.type}:${config.model ?? DEFAULT_MODELS[config.type]}`;


non-blocking: This change is only for the label, but I'm wondering, in general, if we should have a default model. Given that pricing could be very different for different models, I wonder if we should force users to pick a model themselves

This is a good callout. Ideally if the user is overriding the model, they need to specify both the provider & model (both valid). We should not silently fallback to a default model since that would hide that from the user.

Currently this was also a fallback in the getModel function
const modelId = requestModel || this.config.model || this.getDefaultModel();

I am removing this to ensure we dont silently fallback and the provider + model configuration (internal or override) are valid

Copilot

Pull request overview

Copilot reviewed 24 out of 24 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+    if (config.modelOverride) {
+      if (!keyFor[config.modelOverride.provider]) {
+        throw new ConfigurationError(
+          `${humanName[config.modelOverride.provider]} is required when using modelOverride with provider "${config.modelOverride.provider}". Pass ${configKey[config.modelOverride.provider]} in config.`
+        );


adnanrhussain requested review from czi-fsisenda and georgemelvin April 22, 2026 23:50

adnanrhussain requested a review from Copilot April 23, 2026 16:13

Copilot started reviewing on behalf of adnanrhussain April 23, 2026 16:13 View session

Copilot AI reviewed Apr 23, 2026

View reviewed changes

Comment thread sdks/typescript/src/evaluators/base.ts

Comment thread sdks/typescript/src/evaluators/vocabulary.ts Outdated

Comment thread sdks/typescript/README.md Outdated

czi-fsisenda approved these changes May 5, 2026

View reviewed changes

adnanrhussain requested a review from Copilot May 5, 2026 06:05

Copilot started reviewing on behalf of adnanrhussain May 5, 2026 06:06 View session

Copilot AI reviewed May 5, 2026

View reviewed changes

adnanrhussain force-pushed the ahussain/ts_sdk_override_model branch from 1c0b917 to c190ff4 Compare May 5, 2026 06:26

adnanrhussain added 7 commits May 6, 2026 13:21

feat(ts-sdk): add modelOverride option to all evaluators

a62c14c

re:copilot feedback

e4965bb

fix: remove default model impl

524b0ba

refactor: error handling

3ca13b9

re: copilot feedback

5e30490

Update and include new Purpose

14fb2b1

fix docs

857ea7c

adnanrhussain force-pushed the ahussain/ts_sdk_override_model branch from c190ff4 to 857ea7c Compare May 7, 2026 00:33

adnanrhussain merged commit c57c4fc into main May 7, 2026
15 checks passed

adnanrhussain deleted the ahussain/ts_sdk_override_model branch May 7, 2026 01:20

czi-github-helper Bot mentioned this pull request May 7, 2026

chore: release main #58

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(ts-sdk): add modelOverride option to all evaluators#34

feat(ts-sdk): add modelOverride option to all evaluators#34
adnanrhussain merged 7 commits intomainfrom
ahussain/ts_sdk_override_model

adnanrhussain commented Apr 22, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Apr 22, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

czi-fsisenda left a comment

Uh oh!

czi-fsisenda May 4, 2026

Uh oh!

adnanrhussain May 5, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

adnanrhussain commented Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

codecov Bot commented Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

czi-fsisenda left a comment

Choose a reason for hiding this comment

Uh oh!

czi-fsisenda May 4, 2026

Choose a reason for hiding this comment

Uh oh!

adnanrhussain May 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

adnanrhussain commented Apr 22, 2026 •

edited

Loading

codecov Bot commented Apr 22, 2026 •

edited

Loading