feat(rest): support storage-credentials in PlanCompleted response #3235
feat(rest): support storage-credentials in PlanCompleted response #3235tanmayrauth wants to merge 1 commit intoapache:mainfrom
Conversation
…rtial fix for apache#3165) Adds storage-credentials support for CompletedPlanningResult (PlanCompleted). LoadCredentialsResponse support is pending.
ddd2380 to
cd844d3
Compare
|
@kevinjqliu @Fokko can you please review and approve this? |
geruh
left a comment
There was a problem hiding this comment.
I left this open intentionally because there are deeper design decisions that needed to get sorted out before we figure out a path forward.
The issue is that today there's tight coupling between FileIO and the objects that hold it like Table, the distinction between REST ops and base ops. FileIO gets set once at construction and everything downstream assumes it stays fixed. TOday that works fine for load_table and all current interactions.
But for scan planning, credentials arrive after table creation so what happens to the FileIO that was already wired up? Overwriting it will break other operations that still hold a reference to the original IO. I think this is a bigger change, and to give confidence in the path forward we should map out the relationship of FileIO and Table how it's used. So that when we eventually need something like a prefixed FileIO client, it's a simple change.
I think that would be a good start and take a look at the RESTTableScan and the sequence of operations there, and then we could figure out a more pythonic equivalent would look like. Happy to discuss more!
|
@geruh Thanks for the review! Happy to dig into the FileIO/Table relationship and map out how it's used before we finalize the approach. Will take a look at RESTTableScan and the sequence of operations and follow up with a more detailed design. |
Adds storage-credentials support for CompletedPlanningResult (PlanCompleted). LoadCredentialsResponse support is pending.
Fixes: #3165
Rationale for this change
Are these changes tested? Yes
Are there any user-facing changes? No