linkedin / linkedin/openhouse

[Feature] HdfsStorage and LocalStorage's TableLocation should be full URI with scheme

Open
#121 0 comments 0 reactions 0 assignees View on GitHub
feat
Dominant language
Java
Stars
399
Forks
80
Avg merge
3d 23h
Merged PRs (30d)
15

Description

### Willingness to contribute

{"label"=>"Yes. I can contribute a fix for this bug independently.", "value"=>"independent"}

### Feature Request Proposal

Currently HDFSStorage and LocalStorage set TableLocation without scheme, ie: `/data/openhouse/db/tbl-UUID` instead of full URI `hdfs:///data/openhouse/db/tbl-UUID`.

This was done to ensure backward compatibility, replication and migration.

Iceberg spec mandates a full-URI (containing scheme) as part of its metadata. source: https://iceberg.apache.org/spec/?ref=blog.min.io#manifests

As part of this bugFix, HDFSStorage and LocalStorage should start using fully-qualified URI.

### Motivation

> #### What is the use case for this feature?
Making this change will generalize tableLocation behavior for various Storages such as s3, hdfs, local etc.

### What component(s) does this feature affect?

- [X] `Table Service`: This is the RESTful catalog service that stores table metadata. `:services:tables`
- [ ] `Jobs Service`: This is the job orchestrator that submits data services for table maintenance. `:services:jobs`
- [ ] `Data Services`: This is the jobs that performs table maintenance. `apps:spark`
- [X] `Iceberg internal catalog`: This is the internal Iceberg catalog for OpenHouse Catalog Service. `:iceberg:openhouse`
- [ ] `Spark Client Integration`: This is the Apache Spark integration for OpenHouse catalog. `:integration:spark`
- [ ] `Documentation`: This is the documentation for OpenHouse. `docs`
- [ ] `Local Docker`: This is the local Docker environment for OpenHouse. `infra/recipes/docker-compose`
- [ ] `Other`: Please specify the component.

### Details

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by tracing TableLocation handling in :services:tables and :iceberg:openhouse, focusing on HdfsStorage and LocalStorage. Verify how existing paths support backward compatibility, replication, and migration. Done means both storages produce fully qualified URIs with schemes, consistent with the Iceberg metadata requirement.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, backend, database
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.