HDFGroup / HDFGroup/hdf5

Optimizing iterating over a group with 140000 entries

Open
#5,078 0 comments 0 reactions 1 assignee Claimed by @fortnern View on GitHub
Component - C Library
Dominant language
C
Stars
988
Forks
355
Avg merge
4d 2h
Merged PRs (30d)
12

Description

Part of user's messages from [HELP-2606](https://hdfgroup.atlassian.net/issues/HELP-2606):

> ...as I am investigating parameters on how to optimize the metadata cache, I discovered the following issue:
> When setting the initial MDC size to the maximum allowed value
>
> ```
>mdc_config.initial_size = 32*1024*1024;
> mdc_config.max_size = 32*1024*1024;
> mdc_config.min_size = 1024*1024;
>```

then I got HDF5 errors during file creation:
```
HDF5-DIAG: Error detected in HDF5 (1.14.4-3) thread 13728:
#000: H5Odeprec.c line 674 in H5Oget_info_by_name2(): can't get deprecated info for object
major: Object header
minor: Can't get value
#001: H5Odeprec.c line 243 in H5O__get_info_old(): can't get data model info for object
major: Object header
minor: Can't get value
#002: H5VLcallback.c line 5992 in H5VL_object_get(): get failed
major: Virtual Object Layer
minor: Can't get value
#003: H5VLcallback.c line 5960 in H5VL__object_get(): get failed
major: Virtual Object Layer
minor: Can't get value
#004: H5VLnative_object.c line 269 in H5VL__native_object_get(): object not found
major: Object header
minor: Object not found
#005: H5Gloc.c line 761 in H5G_loc_info(): can't find object
major: Symbol table
minor: Object not found
#006: H5Gtraverse.c line 824 in H5G_traverse(): internal path traversal failed
major: Symbol table
minor: Object not found
#007: H5Gtraverse.c line 596 in H5G__traverse_real(): traversal operator failed
major: Symbol table
minor: Callback failed
#008: H5Gloc.c line 722 in H5G__loc_info_cb(): can't get object info
major: Symbol table
minor: Can't get value
#009: H5Oint.c line 2071 in H5O_get_info(): unable to load object header
major: Object header
minor: Unable to protect metadata
#010: H5Oint.c line 1028 in H5O_protect(): unable to load object header chunk
major: Object header
minor: Unable to protect metadata
#011: H5AC.c line 1277 in H5AC_protect(): H5C_protect() failed
major: Object cache
minor: Unable to protect metadata
#012: H5Centry.c line 3106 in H5C_protect(): can't load entry
major: Object cache
minor: Unable to load metadata into cache
#013: H5Centry.c line 1179 in H5C__load_entry(): incorrect metadata checksum after all read attempts
major: Object cache
minor: Read failed
```
This error vanishes when setting the initial MDC size to a smaller value such as mdc_config.initial_size = 16*1024*1024;

Is this a known issue fixed in a more recent version? Would you like me to explore this issue further? At the moment I think I can reproduce it and investigate with other values or code modifications per your recommendations.
....

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.