google / google/ml-metadata

More resilient & long-lasting connections

Open
#221 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
C++
Stars
685
Forks
185
PR merge metrics
No merged PRs in 30d

Description

- ml-metadata==1.14.0
- mysql
- running using direct db connection & minio

I've seen this issue in my environment where I'm using a long lasting MetadataStore database connection. I'm using mysql database and connecting without the grpc server.

This connection disconnects after some idle timeout frequently. I've already checked configurations across the stack, likely something in my k8s runtime is killing the connection. I can see an issue with very simple code such as this. The second call to the client errors out with an Internal Error. Works fine if I re-instantiate it.

```
if __name__ == "__main__":
mlmd_client = get_mlmetadata_client() # returns instantiated MetadataStore object
context_types = mlmd_client.get_context_types_by_external_ids(["Test"])
print(context_types)
time.sleep(1800)
context_types = mlmd_client.get_context_types_by_external_ids(["Test"])
print(context_types)
```

For my use-case, I've implemented a simple retry in my code to workaround the issue by overriding the private `_call` method in the class. I felt this was most optimal for my use-case.

I think it would be a good feature to implement something that tries re-establishes the connection. If needed to be done in a later version I think that makes sense. Would be good to know if there is more stability with a workflow that's different from the way I'm doing it.

Contributor guide

Open the contributing guide

Research direction

Start with the MetadataStore client and its private _call method, which the report says is overridden for retries. Reproduce the two-call example with a 30-minute idle period against MySQL, then verify that the second call can re-establish the connection without reinstantiating the client.

Written by the indexing model from the issue text.

Assessment

Tech stack
mysql, python
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.