apache / apache/beam

Remove ResourceId.getCurrentDirectory()?

Open
#18,338 0 comments 0 reactions 0 assignees View on GitHub
backward-incompatible bug core java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

Beam ResourceId currently has a getCurrentDirectory method that returns the current resource id if it's a directory, or the parent directory if it's a directory.

To implement this you need to know whether or not a particular path is a directory or not.

I'm trying to implement the Hadoop ResourceId implementation, and it's not clear if it's possible. Hadoop's Paths do not end a / if they are a directory (they are stripped), nor do hadoop paths tell you if something is a directory, so it's not possible to determine if a given path is a file that does not have a suffix, or a directory.

It's not clear to me that all file systems can determine whether a path is a directory and thus I don't believe it can be implemented reliably.

The only usages of getCurrentDirectory that I could find are in tests so it's not clear we actually need this.

I propose that we remove this method.

cc [~davor]

Imported from Jira [BEAM-2069](https://issues.apache.org/jira/browse/BEAM-2069). Original Jira may contain additional context.
Reported by: sisk.

Contributor guide

Open the contributing guide

Research direction

Locate ResourceId.getCurrentDirectory and the test usages mentioned in the issue, then inspect the Hadoop ResourceId implementation and Hadoop Path behavior. Remove the method only if the usages can be retired; done means the API and affected tests no longer depend on directory inference.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java
Domain
api, backend
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.