Fix Redis stream leak: MAXLEN on xadd + sliding TTL on stream keys

[x]

Summary

The RedisStreamRepository in this SDK has been leaking memory in production: every task:{id} stream grows unbounded for the lifetime of the task and is never deleted. On a long-lived cluster (BP) this manifested as 12k+ orphaned task streams consuming most of a 6GB Redis cache, requiring manual redis-cli DEL flushes.

Two complementary fixes, one commit each. Both mirror patterns the agentex server-side adapter (scaleapi/scale-agentex agentex/src/adapters/streams/adapter_redis.py) already uses.

Commit 1 — perf(streaming): add MAXLEN to Redis xadd

RedisStreamRepository.send_event was calling xadd with no MAXLEN, despite a comment claiming "Add to Redis stream with a reasonable max length". The cap was intended but never wired up.

Matches the server adapter's pattern from scale-agentex PR #111 (Jan 2). Default 10000 entries, overridable via REDIS_STREAM_MAXLEN env var.

Commit 2 — feat(streaming): add sliding TTL to Redis stream keys

Mirrors scale-agentex PR #215 (Stas, just merged) on the SDK side. Pipelines XADD with EXPIRE in one round-trip. Each delta write refreshes the TTL on the stream key, so:

• An actively-streaming agent (or actively-reading SSE consumer keeping the key warm via server-side reads) keeps the key alive.
• Once the agent stops writing, the key ages out and Redis auto-deletes it after REDIS_STREAM_TTL_SECONDS.
• An orphaned stream (no writes, no consumer) self-deletes within the TTL window — no explicit cleanup_stream call needed.

Default TTL: 3600 seconds (1h), matching the server. Configurable via REDIS_STREAM_TTL_SECONDS env var. Setting it to 0 short-circuits to plain XADD (no TTL refresh).

Implementation notes:

• transaction=False on the pipeline: connection-level batching, no MULTI/EXEC overhead for a hot path.
• raise_on_error=False: if EXPIRE fails after XADD succeeded, log and move on. We must not propagate the failure to the caller — the message has been published; a retry would duplicate it. The next successful XADD will reset the TTL anyway.

Why this approach instead of cleanup_stream?

An earlier draft of this PR added an explicit cleanup_stream call to all terminal task transitions in TasksService (complete, fail, cancel, terminate, timeout, delete). Stas pointed out two issues with that approach which the EXPIRE pattern sidesteps:

1. Race with task_updated events. When the server processes a terminal task transition (POST /tasks/{id}/complete), it publishes a task_updated event to the same task:{id} stream so connected SSE consumers see the state change. An SDK-side cleanup_stream immediately after the API call returns would race that event — a frontend mid-XREAD could miss the COMPLETED transition.
2. Multi-subscriber unfairness. The existing server-side cleanup_stream-on-disconnect path nukes the topic for all subscribers when one disconnects. Adding more cleanup_stream callsites compounds that latent bug.

EXPIRE avoids both: it only fires after inactivity, so any active reader keeps the key alive long enough to consume final events, and the topic-deletion-on-disconnect bug is rendered moot once the SDK no longer relies on explicit DEL.

Validation

End-to-end tested against BP's dev cluster on 2026-05-05 with this branch path-pinned into the agent images. Ran an eval (-k 2, 2 examples → 2 orchestrator workflows + 2 tardis subagent workflows = 4 task streams).

Redis state ~3 minutes after eval finished:

Task	Type	XLEN	TTL
task:58baa63e-...	orchestrator	105	3424s
task:251f2dac-...	orchestrator	112	3436s
task:7d87bbd3-...	tardis subagent	139	3402s
task:49bb2778-...	tardis subagent	83	3388s

All four streams have active TTLs of ~3400s (~57 min, which is 3600s - elapsed_since_last_write). They will self-delete from Redis ~57 min from the last delta write — no explicit cleanup call needed.

Side-by-side baseline (same Redis cluster, same time, but pods running the unmodified agentex-sdk==0.10.3 from PyPI):

Task	XLEN	TTL
task:e2963d73-...	120	NO_TTL
task:f7c9f250-...	171	NO_TTL
task:30526127-...	152	NO_TTL
task:c3dcb6ec-...	94	NO_TTL
(... ~80 more, all NO_TTL ...)

The only difference between the two cohorts is the SDK version pinned in the agent images. The TTL is the difference this PR makes.

This pattern also closes the orphan case our application layer can't fix: long-lived orchestrator workflows (which never call tasks.complete because they're designed to stay alive across multiple user messages) used to leak their stream forever. Now their stream auto-cleans within REDIS_STREAM_TTL_SECONDS of the last delta, regardless of workflow lifecycle.

Test plan

• SDK lints cleanly (uv run ruff check src/agentex/lib/core/adapters/streams/adapter_redis.py)
• Existing tests still pass (no behavior change to existing API surface; new kwargs are optional)
• Live cluster validation: pipelined writes succeed; TTL set on stream keys; side-by-side proof against unmodified SDK on same cluster
• Backwards compatibility: RedisStreamRepository() with no kwargs continues to work; setting REDIS_STREAM_TTL_SECONDS=0 disables the TTL behavior entirely

[smoreinis]

on the backend side, i just made a change to pipeline xadd with an expire call to ensure streams get cleaned up by redis if nothing else, maybe worth doing something similar here? ref: scaleapi/scale-agentex#215