apache / apache/hudi

Registry.getRegistryOfClass returns a registry of the wrong class instead of failing

Open
#19,893 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Found while reviewing #19892 (part of #19524).

`Registry.REGISTRY_MAP` (hudi-io, `org.apache.hudi.common.metrics.Registry`) is a JVM-wide static that is never cleared. `SparkHoodieBackedTableMetadataWriter.initRegistry()` interns the metadata registry under the table-name-less key `""::HoodieMetadata`. When metrics are on and executor metrics are off it stores a `LocalRegistry`; when executor metrics are on with a non-INMEMORY reporter it calls `Registry.getRegistryOfClass(..., DistributedRegistry.class)`. On a class mismatch `getRegistryOfClass` only logs "already exists with a different class" and returns the existing `LocalRegistry`, which the caller then hard-casts.

Reproduces at one surefire fork just by ordering two existing tests in one JVM:

```
mvn test -am -pl hudi-spark-datasource/hudi-spark -Dspark3.5 -Dscala-2.12 -DskipUTs=true \
-Dhudi.surefire.forkCount=1 -Dsurefire.runOrder=reversealphabetical \
-Dtest='TestMetricsReporter,TestMetadataTableWithSparkSQL' ...
# ClassCastException: LocalRegistry cannot be cast to DistributedRegistry
```

The alphabetical order passes, and each class passes alone. Any change to how classes are grouped into JVMs (fork count, new tests, renamed classes) can flip it.

Proposed fix: key the metadata registry per table (or per base path), and have `getRegistryOfClass` replace or throw on a class mismatch rather than return the wrong type. A test-side reset of `REGISTRY_MAP` between classes would hide the symptom but not the production bug.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with Registry.REGISTRY_MAP and Registry.getRegistryOfClass, then trace SparkHoodieBackedTableMetadataWriter.initRegistry(). Reproduce the collision with TestMetricsReporter and TestMetadataTableWithSparkSQL using one surefire fork and reverse order. Done means the metadata registry is isolated by table or base path and a class mismatch cannot return a registry that causes a ClassCastException, with a regression test covering the ordering.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.