Skip to content

[AI-FSSDK] [FSSDK-12368] Local Holdouts - Cleanup flag base setup and add includedRules and rule-level lookup#501

Open
Mat001 wants to merge 5 commits intomasterfrom
ai/mat001/FSSDK-12368
Open

[AI-FSSDK] [FSSDK-12368] Local Holdouts - Cleanup flag base setup and add includedRules and rule-level lookup#501
Mat001 wants to merge 5 commits intomasterfrom
ai/mat001/FSSDK-12368

Conversation

@Mat001
Copy link
Copy Markdown
Contributor

@Mat001 Mat001 commented Apr 10, 2026

Summary

Implements Local Holdouts support for the Python SDK, replacing legacy flag-level holdout targeting (includedFlags/excludedFlags) with rule-level targeting (includedRules).

Related Ticket: FSSDK-12368

Changes

Data Model

  • Added included_rules field to Holdout model (optional array of rule IDs)
  • Added is_global property (True when included_rules is None)
  • Removed legacy fields: included_flags, excluded_flags

Holdout Configuration

  • Updated HoldoutConfig from flag-level to rule-level mapping
  • Implemented get_global_holdouts() - returns holdouts with included_rules == None
  • Implemented get_holdouts_for_rule(rule_id) - returns holdouts targeting specific rule
  • Added validation for rule holdouts map integrity

Decision Flow

  • Global holdouts evaluated at flag level (before forced decisions)
  • Local holdouts evaluated per-rule (after forced decisions, before audience/traffic)
  • If user hits local holdout, rule evaluation is skipped
  • Returns correct decision metadata: source="holdout", experiment_id, ruleKey

Edge Cases

  • Missing included_rules field defaults to None (global holdout)
  • Empty array [] vs None handled correctly (empty = local with no rules, None = global)
  • Invalid rule IDs skipped gracefully (no errors)
  • Cross-flag targeting supported (one holdout can target rules from multiple flags)

Testing

New Tests (11 comprehensive tests in tests/test_local_holdouts.py)

  • Global holdout evaluation
  • Local single-rule targeting
  • Local multi-rule targeting (same flag and cross-flag)
  • Precedence (global before local)
  • Edge cases (missing field, empty array, invalid rule IDs)

Updated Tests (60 existing tests)

  • Migrated from included_flags/excluded_flags to included_rules
  • Updated to use is_global property
  • Verified backward compatibility

Quality Metrics

  • ✅ Tests: 71/71 passing (100%)
  • ✅ Critical Issues: 0
  • ✅ Warnings: 0
  • ✅ Comprehensive edge case coverage

Files Modified

  • optimizely/entities.py - Holdout model updates
  • optimizely/project_config.py - Mapping and retrieval logic
  • optimizely/decision_service.py - Decision flow integration
  • optimizely/helpers/types.py - Type definitions
  • tests/test_local_holdouts.py - New comprehensive test suite
  • tests/test_decision_service_holdout.py - API updates
  • tests/test_config.py - Model updates

Test Plan

  • Unit tests pass (71/71)
  • Global holdouts work correctly
  • Local holdouts work correctly (single rule, multiple rules)
  • Edge cases handled (missing field, empty array, invalid IDs)
  • Backward compatibility verified (old datafiles work)
  • Cross-flag targeting works
  • Precedence correct (global → forced → local → rule)

🤖 Generated with Claude Code

Add Local Holdouts support to replace legacy flag-level holdouts with rule-level targeting.

Changes:
- Add includedRules field to Holdout model (replaces includedFlags/excludedFlags)
- Add isGlobal property for global vs local holdout detection
- Update HoldoutConfig mapping from flag-level to rule-level
- Implement get_global_holdouts() and get_holdouts_for_rule() methods
- Integrate local holdout evaluation in decision flow (per-rule, before audience/traffic)
- Handle edge cases (missing field, empty array, invalid rule IDs, cross-flag targeting)
- Add comprehensive unit tests for local holdouts
- Update existing tests to use new API

Quality Metrics:
- Tests: 71/71 passing (100%)
- Critical Issues: 0
- Warnings: 0

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Mat001 and others added 3 commits April 13, 2026 00:24
- Update copyright year to 2026
- Remove unnecessary if __name__ == '__main__' block from test file

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Update copyright headers in modified files to include 2026.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Remove unused imports flagged by ruff:
- unittest (using unittest.mock instead)
- decision_service (not used)
- OptimizelyDecideOption (not used)
- enums (not used)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@Mat001 Mat001 requested a review from farhan-opti April 14, 2026 21:34
Copy link
Copy Markdown
Contributor

@FarhanAnjum-opti FarhanAnjum-opti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes look good to me. A few things that needs addressing are commented.

Comment thread tests/test_local_holdouts.py Outdated
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests look good. But this file breaks existing convention of module level testing. The Python SDK organizes tests by source module (e.g., test_config.py for config/entities,
test_decision_service_holdout.py for decision service holdout logic), not by feature. The tests in LocalHoldoutsTest (entity properties, project config methods) should be added to
test_config.py, and LocalHoldoutsDecisionFlowTest should be added to test_decision_service_holdout.py. This file should then be removed.

Comment thread optimizely/helpers/types.py Outdated
Comment on lines +24 to +28
if version_info < (3, 11):
from typing_extensions import NotRequired
else:
from typing import NotRequired # type: ignore

Copy link
Copy Markdown
Contributor

@FarhanAnjum-opti FarhanAnjum-opti Apr 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NotRequired with the conditional import adds unnecessary complexity here. These TypedDicts are just intermediate types for type signatures (as noted in the file itself), and backward
compatibility is already handled in Holdout.__init__ where includedRules: Optional[list[str]] = None with **kwargs: Any — if an older datafile omits the
field, **kwargs absorbs any unknown keys and includedRules defaults to None (global holdout). The type system doesn't need to model the "key absent" case when the runtime already
handles it. Just using Optional[list[str]] would be simpler and avoids the new typing_extensions dependency for Python < 3.11.

Suggested change
if version_info < (3, 11):
from typing_extensions import NotRequired
else:
from typing import NotRequired # type: ignore

Comment thread optimizely/helpers/types.py Outdated
holdoutStatus: HoldoutStatus
includedFlags: list[str]
excludedFlags: list[str]
includedRules: NotRequired[Optional[list[str]]]
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
includedRules: NotRequired[Optional[list[str]]]
includedRules: Optional[list[str]]

…dule-level files

- Remove NotRequired import and use Optional[list[str]] for HoldoutDict.includedRules
- Migrate test fixtures from includedFlags/excludedFlags to includedRules
- Move LocalHoldoutsTest into test_config.py (HoldoutConfigTest + LocalHoldoutConfigTest)
- Move LocalHoldoutsDecisionFlowTest into test_decision_service_holdout.py
- Delete test_local_holdouts.py (violated module-level test convention)
- Tighten weakened assertions to exact counts and content checks

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants