alibaba / alibaba/GraphScope

[BUG] Can not load vertices from parquet

Open
#4,647 3 comments 0 reactions 0 assignees View on GitHub
stale
Dominant language
C++
Stars
3.6k
Forks
468
Avg merge
29m
Merged PRs (30d)
1

Description

**Describe the bug**
When loading a graph from local Parquet files mounted via hostPath volume into a K8s session, GraphScope fails with Object not exists in vineyard followed by Empty table list cannot be used for normalizing schema. The same setup works correctly with CSV files.

**To Reproduce**

- Set up a K8s cluster (tested with K3s on single node Ubuntu)
- Mount a local directory containing Parquet files as a hostPath volume
- Create a GraphScope session with k8s_volumes pointing to the mount
- Call add_vertices with a file:/// path to a .parquet file
```
import graphscope as gs

with gs.session(k8s_volumes={
"data": {
"type": "hostPath",
"field": {
"path": "/path/to/host/dataset",
"type": "Directory"
},
"mounts": {
"mountPath": "/dataset"
}
}
}, k8s_image_tag='0.30.0', k8s_namespace='gs') as session:

graph = session.g()
graph = graph.add_vertices(
'file:///dataset/vertices.parquet',
label='person',
properties=['firstName', 'lastName', 'gender'],
vid_field='id'
)
graph = graph.add_edges(
'file:///dataset/edges.parquet',
src_label='person',
dst_label='person'
)
```

**Expected behavior**
Graph should load from Parquet files the same way it loads from CSV files.

**Environment**

- GraphScope version: v0.30.0
- vineyard version: v0.24.2
- OS: Ubuntu 24.04
- Kubernetes: K3s v1.24.9+k3s1(single node)
- Python 3.10.20
- Graphscope python library: v0.29.0

**Additional context**
I faced the the same issue if load dataset from S3:
```
import graphscope as gs
from graphscope.framework.loader import Loader
import argparse

parser = argparse.ArgumentParser(description="Process command-line arguments.")
parser.add_argument("--s3-key", type=str)
parser.add_argument("--s3-secret", type=str)

args = parser.parse_args()

# Doesn't work
with gs.session(k8s_namespace='gs', k8s_image_tag='0.31.0') as session:
print(session.info)
graph = session.g()
print("Graph loaded")
graph = graph.add_vertices(Loader('s3://bucket-name/csv/vertices.csv', delimiter='|', key=args.s3_key, secret=args.s3_secret, endpoint_url='https://custom-endpoint.ru'), label='person')
print("Vertices loaded")
graph = graph.add_edges(Loader('s3://bucket-name/csv/edges.csv', delimiter='|', key=args.s3_key, secret=args.s3_secret, endpoint_url='https://custom-endpoint.ru'), src_label='person', dst_label='person')
print("Edges loaded")
```

[k8s.logs.txt](https://github.com/user-attachments/files/26134890/k8s.logs.txt)
[python.logs.txt](https://github.com/user-attachments/files/26134889/python.logs.txt)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.