fix(ci): silence sidecar flush-timeout log noise in client_side_stats test#3949
Draft
Leiyks wants to merge 2 commits into
Draft
fix(ci): silence sidecar flush-timeout log noise in client_side_stats test#3949Leiyks wants to merge 2 commits into
Leiyks wants to merge 2 commits into
Conversation
|
… test [test_extension_ci] Under concurrent run-tests load the best-effort dd_trace_synchronous_flush() can exceed its 5s window, emitting '[ddtrace] [error] Failed synchronously flushing traces: Kind(TimedOut)' on stderr. The stats still arrive via the async sidecar path that the test polls with waitForStats(), so the expected output is produced, but the stray error line breaks the output diff. Set DD_TRACE_LOG_LEVEL=off to match the sibling tests (client_side_stats_peer_tags, client_side_stats_trace_filters), which already suppress this benign log noise.
Leiyks
force-pushed
the
leiyks/fix-ci-crashtracker-threads
branch
from
d340f37 to
6da4162
Compare
Leiyks
changed the title
…flush timing [test_extension_ci] Under concurrent run-tests -j4 load two background-sender/sidecar tests flaked because best-effort flushing to the shared request-replayer can be delayed beyond their wait windows: - agent_headers_container_id_fargate.phpt threw 'wait for replay timeout': its DD_TRACE_AGENT_FLUSH_INTERVAL=333 gives a ~16s replay-poll window (maxIteration * flushInterval), half that of the reliably-passing sibling agent_headers_container_id.phpt (666). Bump to 666 to match the sibling and double the window the BGS delivery can use under load. - shm_data_internal_fns.phpt got 'Failed synchronously flushing traces: Kind(TimedOut)' stderr noise and a missing agent sampling config (bool(false)/float(-1)) because the best-effort synchronous_flush timed out, delaying the trace that makes the replayer return rate_by_service. Set DD_TRACE_LOG_LEVEL=off (same convention as client_side_stats) to silence the benign log line, and re-trigger the flush periodically while polling 100x100ms so the sampling config reliably propagates into SHM.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Under concurrent CI load,
tests/ext/request-replayer/client_side_stats.phptintermittently failed on a stray[ddtrace] [error] Failed synchronously flushing traces: Kind(TimedOut)line from the best-effort 5s synchronous flush — the stats themselves still arrive via the polled async sidecar path. AddDD_TRACE_LOG_LEVEL=off, matching its sibling testsclient_side_stats_peer_tags/client_side_stats_trace_filters. Verified 60/60 under load in docker.Note: the crashtracker thread-format fix originally here is now redundant — landed on master via #3953 — so this branch was rebased to contain only the client_side_stats fix.