apache / apache/lucene

Can we allow FSDirectory subclasses to customize whether the ctor does a mkdir? [LUCENE-7375]

Open
#8,428 4 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Today, just instantiating an `FSDirectory` always brings the directory into existence, even if you will do no writes (`createOutput`).

We have gone back and forth on this, over the ages. E.g. see #1094 (only 2 digits there!![), #1848 (only 3 digits there](), #1848 (only 3 digits there)!), #2538. At one point we created the directory lazily, on the first write (`createOutput`) attempt, but now we always create it when you instantiate `FSDirectory`.

This causes some hassle for consumers, e.g. in https://github.com/elastic/elasticsearch/pull/19338 ES is forking `SimpleFSDirectory` in order to have a read-only directory impl.

Maybe we can do the `Files.createDirectories` in protected method that a subclass could override?

---
Migrated from [LUCENE-7375](https://issues.apache.org/jira/browse/LUCENE-7375) by Michael McCandless (@mikemccand)

Contributor guide

Open the contributing guide

Research direction

Start by reading FSDirectory and its SimpleFSDirectory subclass, focusing on the constructor, Files.createDirectories, and createOutput. Trace when directory creation currently occurs and how a read-only subclass such as Elasticsearch's could avoid it. Done means FSDirectory subclasses can customize constructor-time directory creation without breaking the existing write path.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.