feat: Phase 3c — ParquetMergeExecutor + full downloader#6358
Merged
g-talbot merged 4 commits intogtt/parquet-merge-pipeline-3bfrom May 1, 2026
Merged
feat: Phase 3c — ParquetMergeExecutor + full downloader#6358g-talbot merged 4 commits intogtt/parquet-merge-pipeline-3bfrom
g-talbot merged 4 commits intogtt/parquet-merge-pipeline-3bfrom
Conversation
4 tasks
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3b
branch
from
3b171a0 to
84c6dd3
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3c
branch
from
ceba410 to
5937440
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3b
branch
from
84c6dd3 to
a23011c
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3c
branch
from
5937440 to
0b1c9cc
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3b
branch
from
a23011c to
86fd55a
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3c
branch
2 times, most recently
from
66b97e0 to
0f051bc
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3b
branch
from
86fd55a to
ef391c9
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3c
branch
from
0f051bc to
16b46d7
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3b
branch
from
ef391c9 to
90c5589
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3c
branch
from
16b46d7 to
8e19b6b
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3b
branch
from
90c5589 to
49c6c19
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3c
branch
from
8e19b6b to
f32bd64
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3b
branch
from
49c6c19 to
93a0a20
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3c
branch
from
f32bd64 to
de17c0e
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3b
branch
from
93a0a20 to
3a91a31
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3c
branch
from
de17c0e to
1f6512e
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3b
branch
from
3a91a31 to
956a6e5
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3c
branch
from
1f6512e to
f2c4a8a
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3b
branch
from
956a6e5 to
4bde5b8
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3c
branch
from
f2c4a8a to
467f6fb
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3b
branch
from
4bde5b8 to
5556bb2
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3c
branch
from
467f6fb to
b6ca9bf
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3b
branch
from
5556bb2 to
f8f1528
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3c
branch
from
b6ca9bf to
be60cf6
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3b
branch
from
f8f1528 to
97295a6
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3c
branch
from
be60cf6 to
bb6cd8b
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3b
branch
from
97295a6 to
40e2b25
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3c
branch
from
bb6cd8b to
f34bd04
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3b
branch
from
40e2b25 to
dee01ee
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3c
branch
2 times, most recently
from
5c68071 to
daacf89
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3b
branch
from
fc4ce2e to
7c25b1e
Compare
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3c
branch
from
daacf89 to
2d82e7e
Compare
Contributor
Author
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2d82e7e1db
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
mattmkim
approved these changes
May 1, 2026
Contributor
mattmkim
left a comment
mattmkim
left a comment
There was a problem hiding this comment.
LGTM to unblock, but few comments
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3b
branch
from
7c25b1e to
1f6047f
Compare
…ase 3c) Phase 3 pipeline integration, third PR: - ParquetMergeSplitDownloader: downloads each input split's Parquet file from object storage to a local temp directory, forwards ParquetMergeScratch to the executor. Replaces the stub from PR 3b. - ParquetMergeExecutor: runs merge_sorted_parquet_files via run_cpu_intensive, builds output ParquetSplitMetadata via merge_parquet_split_metadata, renames output files to match generated split IDs, sends ParquetSplitBatch with replaced_split_ids to the uploader. - ParquetSplitBatch.checkpoint_delta -> checkpoint_delta_opt: now Option to support merge operations (no checkpoint delta for data reorganization). Ingest path passes Some(delta), merge path passes None. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When all input splits are empty, still send a batch with replaced_split_ids so the metastore marks them for deletion. Without this, empty splits drained by the planner stay Published forever and are never cleaned up. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ader Matches the Tantivy merge split downloader pattern: protect_zone() prevents the download I/O from triggering heartbeat timeouts, and the kill switch check allows early abort during pipeline shutdown. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two fixes: 1. The merge permit was dropped when the executor finished, releasing the semaphore before upload completed. Now it flows through ParquetSplitBatch → ParquetSplitsUpdate → publisher, matching the Tantivy pipeline. 2. Use the merge operation's pre-assigned split ID instead of generating a second one in merge_parquet_split_metadata(). Keeps the ID consistent across scheduling, tracing, and publishing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
g-talbot
force-pushed
the
gtt/parquet-merge-pipeline-3c
branch
from
ac2ae00 to
706e1dd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #6357 (Phase 3b).
storage.copy_to_file(), forwardsParquetMergeScratchto the executor.merge_sorted_parquet_filesviarun_cpu_intensive, builds outputParquetSplitMetadataviamerge_parquet_split_metadata, renames output files to match generated split IDs, sendsParquetSplitBatchwithreplaced_split_idsto the uploader.checkpoint_delta_opt:ParquetSplitBatch.checkpoint_deltachanged toOption<IndexCheckpointDelta>to support merge operations (no checkpoint delta for data reorganization). Ingest path passesSome(delta), merge path passesNone.Test plan
metricsfeaturecargo clippyclean🤖 Generated with Claude Code