apache / apache/beam

[Bug]: JdbcDriver causes metaspace leak

Open
#29,494 4 comments 0 reactions 0 assignees View on GitHub
bug java P2 pinned
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What happened?

As of now ,`org.apache.beam.sdk.extensions.sql.impl.JdbcDriver` doesn't de-register itself from `java.sql.DriverManager`. And this can causes metaspace leak, because it holds a ChildFirstClassLoader on a flink runner. If we submit jobs many times to a same flink cluster, corresponding cluster OOM eventually.

### Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

### Issue Components

- [ ] Component: Python SDK
- [X] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner

Contributor guide

Open the contributing guide

Research direction

Start at org.apache.beam.sdk.extensions.sql.impl.JdbcDriver and inspect how it registers with java.sql.DriverManager. Verify the driver is released after use so repeated jobs on the same Flink cluster do not retain ChildFirstClassLoader instances and eventually cause a metaspace leak.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
62/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.