Skip to content

[BUG] ByteSizeCachingDirectory wrapped IndexOutput should delegate primitive write methods #19420

@HUSTERGS

Description

@HUSTERGS

Describe the bug

A Store will wrap the underlying Directory with ByteSizeCachingDirectory by default, and all the IndexOutput created from ByteSizeCachingDirectory will be wrapped again with FilterIndexOutput.

return new FilterIndexOutput(out.toString(), out) {
@Override
public void writeBytes(byte[] b, int length) throws IOException {
// Don't write to atomicXXX here since it might be called in
// tight loops and memory barriers are costly
super.writeBytes(b, length);

However, this wrap does not delegate primitive write methods like writeLong, writeInt, etc. , which makes optimization like apache/lucene#321 cannot work as expected, since the FilterIndexOutput does not delegate those methods too. So the those primitive write will eventually be written byte by byte (which is the default implementation of DataOutput).
By delegating those methods like what RateLimitedIndexOutput from Lucene does, the merge speed should be improved.

Please help me out if I got anything wrong : )

Related component

Indexing:Performance

To Reproduce

Check the code

Expected behavior

We should delegate those methods like what Lucene RateLimitedIndexOutput does, ref: https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/store/RateLimitedIndexOutput.java

Additional Details

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

  • OS: [e.g. iOS]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions