-
Notifications
You must be signed in to change notification settings - Fork 2.4k
Description
Describe the bug
A Store
will wrap the underlying Directory
with ByteSizeCachingDirectory
by default, and all the IndexOutput created from ByteSizeCachingDirectory
will be wrapped again with FilterIndexOutput
.
OpenSearch/server/src/main/java/org/opensearch/index/store/ByteSizeCachingDirectory.java
Lines 165 to 170 in 1c171b7
return new FilterIndexOutput(out.toString(), out) { | |
@Override | |
public void writeBytes(byte[] b, int length) throws IOException { | |
// Don't write to atomicXXX here since it might be called in | |
// tight loops and memory barriers are costly | |
super.writeBytes(b, length); |
However, this wrap does not delegate primitive write methods like writeLong
, writeInt
, etc. , which makes optimization like apache/lucene#321 cannot work as expected, since the FilterIndexOutput
does not delegate those methods too. So the those primitive write will eventually be written byte by byte (which is the default implementation of DataOutput
).
By delegating those methods like what RateLimitedIndexOutput
from Lucene does, the merge speed should be improved.
Please help me out if I got anything wrong : )
Related component
Indexing:Performance
To Reproduce
Check the code
Expected behavior
We should delegate those methods like what Lucene RateLimitedIndexOutput does, ref: https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/store/RateLimitedIndexOutput.java
Additional Details
Plugins
Please list all plugins currently enabled.
Screenshots
If applicable, add screenshots to help explain your problem.
Host/Environment (please complete the following information):
- OS: [e.g. iOS]
- Version [e.g. 22]
Additional context
Add any other context about the problem here.