apache / apache/lucene

Fix comment explaining reason for 8KB buffered/chunked writes [LUCENE-9503]

Open
#10,542 0 comments 0 reactions 0 assignees View on GitHub
affects-version:8.6.2 legacy-jira-priority:Minor module:core/store type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

FSDirectory uses an 8KB buffer and chunk size when writing to disk. The class has a comment explaining how file channels would do mallocs for larger chunks. However, that comment was originally introduced when RandomAccessFile and later FileOutputStream was used. Since the switch to file channels, there is no longer any difference (except the size of the buffer) for different buffer/chunk sizes.

Attached patch keeps the comment, clarifying that it is a historic reason.

Including my research here for background information in case this helps validate the change: 
1. The stack buffer for 8KB or below in the JDK is here:

This is used by both `FileOutputStream` and `RandomAccessFile`.
1. This was applicable when the comment to lucene was added and later changed to `FileOutputStream`:

1. Lucene cut over to nio here:

and the comment was later changed to say `file channels` here:

1. File channels use thread local direct buffers instead, see sun.nio.ch.IOUtil.write, which uses Util.getTemporaryDirectBuffer to get a thread local buffer.

---
Migrated from [LUCENE-9503](https://issues.apache.org/jira/browse/LUCENE-9503) by Henning Andersen (@henningandersen)
Attachments: [LUCENE-8kb-comment.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-9503/LUCENE-8kb-comment.patch)

Contributor guide

Open the contributing guide

Research direction

Start in FSDirectory at the comment describing the 8KB buffer and chunk size, then read the attached LUCENE-8kb-comment.patch and the cited history for context. Done means the comment clearly identifies the malloc explanation as historical while accurately reflecting the current file-channel implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Documentation
Difficulty
1/5
Estimated time
Under an hour
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.