CCM-16073 - Performance test changes and concurrency optimisation

[mjewildnhs]

Description

HTTPS Client Lambda

• Increased default concurrency limit from 5 to 10
• Added structured logging on each delivery record (correlationId, receiveCount, firstReceivedAt)
• Fixed object sink lint issue

Perf Runner Lambda

• Added skipPurge flag — allows multiple concurrent Lambda workers without purge conflicts
• Added cloudWatchSettlingMs passthrough to override the default 60s CloudWatch settling wait
• Added sqs-stats.ts — logs queue depth (ApproximateNumberOfMessages, ApproximateNumberOfMessagesNotVisible) per phase
• Extended event-factories.ts with forcedStatusCode and forcedStatusCodeUntilMs fields for time-bounded failure injection
• Extended runner.ts with queue stat logging and SQS GetQueueAttributes integration

Mock Webhook Lambda

• Refactored to support forcedStatusCode with time-bounded expiry (forcedStatusCodeUntilMs) — if the timestamp has passed, the webhook returns 200 instead of the forced code
• Structural refactor for reduced cognitive complexity (lint compliance)

Infrastructure

• New Terraform variables: cb_cooldown_period_ms, cb_recovery_period_ms — passed through from component to client-delivery module to https-client-lambda environment
• Changed pipe_sqs_input_batch_size default from 1 to 10 for throughput
• Added sqs:GetQueueAttributes permission for the perf-runner Lambda on inbound + delivery queues
• Fixed incorrect DLQ ARN reference in perf-runner IAM policy (-dlq suffix → module.sqs_inbound_event.sqs_dlq_arn)
• Fixed quoted-value handling in pre.sh for deploy_mock_clients / deploy_perf_runner tfvar parsing

Context

CCM-16073 — Performance testing and tuning of the rate-limiting and circuit-breaker mechanisms. Adds infrastructure and Lambda changes needed to run targeted perf scenarios (single-client throughput, circuit-breaker trip/recovery) with configurable CB timing, concurrent workers, and failure injection.

Type of changes

• Refactoring (non-breaking change)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would change existing functionality)
• Bug fix (non-breaking change which fixes an issue)

Checklist

• I am familiar with the contributing guidelines
• I have followed the code style of the project
• I have added tests to cover my changes
• I have updated the documentation accordingly
• This PR is a result of pair or mob programming

---

Sensitive Information Declaration

To ensure the utmost confidentiality and protect your and others privacy, we kindly ask you to NOT including PII (Personal Identifiable Information) / PID (Personal Identifiable Data) or any other sensitive data in this PR (Pull Request) and the codebase changes. We will remove any PR that do contain any sensitive information. We really appreciate your cooperation in this matter.

• I confirm that neither PII/PID nor sensitive data are included in this PR and the codebase changes.

[Copilot]

Pull request overview

This PR updates the performance-testing toolchain and related Lambdas/infra to support higher-throughput scenarios, concurrent perf runs, and failure-injection controls for circuit-breaker/rate-limiter tuning.

Changes:

• Perf runner: add skipPurge and configurable CloudWatch settling delay; add SQS queue depth sampling; extend scenario/phase event mix and failure injection fields.
• Mock webhook: refactor request handling and add time-bounded forced-status responses via messageId encoding.
• Delivery + infra: increase https-client Lambda default concurrency, add structured per-record logging, adjust SQS retry/visibility behavior, and pass circuit-breaker timing through Terraform.

Reviewed changes

Copilot reviewed 25 out of 25 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tests/performance/fixtures/subscriptions/perf-client-1.json	Adjust perf fixture invocation rate limit.
tests/performance/fixtures/subscriptions/perf-client-2.json	Adjust perf fixture invocation rate limit for high-throughput testing.
lambdas/perf-runner-lambda/src/types.ts	Extend scenario/event types with forced status fields, per-phase event mix override, and new payload flags.
lambdas/perf-runner-lambda/src/sqs-stats.ts	Add helper to sample queue depth via GetQueueAttributes.
lambdas/perf-runner-lambda/src/runner.ts	Add logging, queue depth sampling, per-phase eventMix override, skip purge option, and configurable CloudWatch settling wait.
lambdas/perf-runner-lambda/src/purge.ts	Update derived DLQ URL naming; simplify purge behavior.
lambdas/perf-runner-lambda/src/index.ts	Plumb cloudWatchSettlingMs and skipPurge from event into runner.
lambdas/perf-runner-lambda/src/event-factories.ts	Encode forced status settings into generated messageIds.
lambdas/perf-runner-lambda/src/tests/sqs-stats.test.ts	Add unit tests for queue depth sampling helper.
lambdas/perf-runner-lambda/src/tests/runner.test.ts	Update runner tests for new flags and queue depth sampling.
lambdas/perf-runner-lambda/src/tests/purge.test.ts	Update purge URL expectations and purge error behavior tests.
lambdas/perf-runner-lambda/src/tests/index.test.ts	Update handler tests for new runner args.
lambdas/perf-runner-lambda/src/tests/event-factories.test.ts	Add tests for forced-status messageId encoding and forwarding.
lambdas/mock-webhook-lambda/src/index.ts	Refactor handler logic and add timed forced-status behavior.
lambdas/mock-webhook-lambda/src/tests/index.test.ts	Add tests for timed forced-status behavior (pre/post expiry).
lambdas/https-client-lambda/src/handler.ts	Increase default concurrency; add per-record structured logs; adjust visibility delay logic and mapping of messages to records.
lambdas/https-client-lambda/src/tests/handler.test.ts	Update tests to match new deterministic visibility delay behavior.
infrastructure/terraform/modules/client-delivery/variables.tf	Increase default batch size and add batching window + CB timing variables.
infrastructure/terraform/modules/client-delivery/module_https_client_lambda.tf	Pass CB timing env vars; configure SQS event source batching window.
infrastructure/terraform/modules/client-delivery/README.md	Document new/updated module inputs.
infrastructure/terraform/components/callbacks/variables.tf	Increase pipe input batch size default; add CB timing variables.
infrastructure/terraform/components/callbacks/pre.sh	Fix tfvars quoted-value parsing for deploy flags.
infrastructure/terraform/components/callbacks/module_perf_runner_lambda.tf	Fix DLQ ARN reference and add sqs:GetQueueAttributes permissions.
infrastructure/terraform/components/callbacks/module_client_delivery.tf	Pass CB timing variables into client-delivery module.
infrastructure/terraform/components/callbacks/README.md	Document updated defaults and new CB timing inputs.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.