google-deepmind / google-deepmind/torch-hdf5

Reading data of class STRING(50331965) is unsupported stack traceback

Open
#82 0 comments 2 reactions 0 assignees View on GitHub
Dominant language
Lua
Stars
241
Forks
125
PR merge metrics
No merged PRs in 30d

Description

I created my hdf5 file by python with
**embed_info.create_dataset('vocab',(count,),dtype='S10',data=vocab)**

When I open that file using torch, it goes wrong.
Here is the error log:
**torch/install/share/lua/5.1/hdf5/ffi.lua:335: Reading data of class STRING(50331965) is unsupported
stack traceback:
[C]: in function 'error'
/home/lyb/torch/install/share/lua/5.1/hdf5/ffi.lua:335: in function '_getTorchType'
/home/lyb/torch/install/share/lua/5.1/hdf5/dataset.lua:88: in function 'getTensorFactory'
/home/lyb/torch/install/share/lua/5.1/hdf5/dataset.lua:138: in function 'partial'
[string "a= file:read('/embed_info/vocab'):partial({1,2})"]:1: in main chunk
[C]: in function 'xpcall'
/home/lyb/torch/install/share/lua/5.1/trepl/init.lua:669: in function 'repl'
.../lyb/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:199: in main chunk
[C]: at 0x00406670**

Anyone has met such a question?

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure with the Python-created `embed_info/vocab` dataset, then inspect `torch/install/share/lua/5.1/hdf5/ffi.lua` at line 335 and `dataset.lua` at lines 88 and 138 to trace how the STRING type is handled. Done means the reported dataset can be read through the Torch HDF5 interface, or the unsupported case is clearly characterized.

Written by the indexing model from the issue text.

Assessment

Tech stack
lua, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.