[FLINK-38336][state/forst] Avoid data copy during failover for ForSt statebackend #27042
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AlexYinHan
force-pushed
the
yh/dev_38336
branch
from
77b41af to
9c4c31e
Compare
AlexYinHan
force-pushed
the
yh/dev_38336
branch
from
9c4c31e to
a8460bc
Compare
Zakelly
reviewed
Sep 29, 2025
| .getSharedStateDirectory(); | ||
| FsCheckpointStorageAccess fsCheckpointStorageAccess = | ||
| (FsCheckpointStorageAccess) env.getCheckpointStorageAccess(); | ||
| remoteJobPath = fsCheckpointStorageAccess.getCheckpointsDirectory(); |
There was a problem hiding this comment.
We can only share state from the shared directory, so what exactly the remoteJobPath means?
Zakelly
reviewed
Sep 29, 2025
| : new CopyDataTransferStrategy(forStFlinkFileSystem); | ||
| LOG.info( | ||
| "Build DataTransferStrategy for Restore: {}, forStFlinkFileSystem: {}, cpSharedFs:{}, recoveryClaimMode:{}", | ||
| "Build DataTransferStrategy for Restore: {}, forStFlinkFileSystem: {}, cpSharedFs:{}, isDbUnderSameJobPathFromRestore:{}, recoveryClaimMode:{}", |
There was a problem hiding this comment.
I'd suggest we make this log as debug level one, as well the one in buildForSnapshot
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
This PR fixes FLINK-38336, by allowing ForSt statebackend to reuse the restored files in failover scenario.
Brief change log
ForStPathContainerReusableDataTransferStrategyif we are in a failover scenarioVerifying this change
This change added tests and can be verified as follows:
DataTransferStrategyTest#testBuildingStrategyAsExpectedDoes this pull request potentially affect one of the following parts:
@Public(Evolving): (no)Documentation