google-deepmind / google-deepmind/torch-hdf5

How I ran torch-hdf5 on Windows (valid HDF5_INCLUDE_PATH and HDF5_LIBRARIES)

Open
#89 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Lua
Stars
241
Forks
125
PR merge metrics
No merged PRs in 30d

Description

Will tell you my little (but painful) experience run torch-hdf5 on Windows.
First I will mention that it worked on Win7 x64 SP1 and Win10 x64.
And that I tried to run torch-rnn ( https://github.com/jcjohnson/torch-rnn ) with distro-win ( https://github.com/BTNC/distro-win ).

After a lot of fixes for distro-win on Win7 (surprisingly, Win10, it booted almost immediately) I started to install torch-hdf5 approximately the same as written here: https://github.com/crisbal/docker-torch-rnn/blob/master/Base/Dockerfile#L42-L44 (given the fact that I am now in windows and I have enabled the environment in Anaconda + has already been launched `torch-activate.bat` + I already installed `hdf5` in conda in my Anaconda env).

> (torch-vc14) C:\distro-win\pkg> git clone https://github.com/deepmind/torch-hdf5
> (torch-vc14) C:\distro-win\pkg> cd torch-hdf5
> (torch-vc14) C:\distro-win\pkg\torch-hdf5> luarocks make hdf5-0-0.rockspec
... "bla-bla-bla" ... ERROR!

I do not remember what was the error but it was due to the following reasons:
In the build command in https://github.com/deepmind/torch-hdf5/blob/master/hdf5-0-0.rockspec#L21-L23 contains ";", after remove it and run same commands with hands:

> (torch-vc14) C:\distro-win\pkg\torch-hdf5> cmake -E make_directory build
> (torch-vc14) C:\distro-win\pkg\torch-hdf5> cd build
> (torch-vc14) C:\distro-win\pkg\torch-hdf5> cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_PREFIX_PATH="$(LUA_BINDIR)/.." -DCMAKE_INSTALL_PREFIX="$(PREFIX)"

^^^^^ replace $(LUA_BINDIR)/.. and $(PREFIX) those that appear above you in the time of the error

> (torch-vc14) C:\distro-win\pkg\torch-hdf5> nmake install

^^^^^ I remove `cd build` because I already in this directory(!). And `nmake` - this replaced by my command from bad output.

After this instructions torch-hdf5 installed, but sometimes in the wrong directory, you need (I don't know why, probably because some paths). If you run the command `th -e "require 'hdf5'"` you'll see an error that the package of torch-hdf5 is not found, then you need to copy all files from: `C:\distro-win\install\luarocks\systree\lib\luarocks\rocks\hdf5\0-0\lua` to `C:\distro-win\install\luarocks\systree\share\lua\5.1\hdf5`

Then running the command `th -e "require 'hdf5'"` will output something like this:

> Error: unable to find a valid HDF5 lib path in the config

To fix this you need to change `HDF5_LIBRARIES` in the file `C:\distro-win\install\luarocks\systree\share\lua\5.1\hdf5\config.lua` to the following value:

> C:/Anaconda2/envs/torch-vc14/Library/bin/hdf5.dll

**_(But it is worth considering that there is spelled out the path to an installed hdf5 in MY Anaconda environment)_**

And (very important!) to change the file `C:\distro-win\install\luarocks\systree\share\lua\5.1\hdf5\ffi.lua` ( https://github.com/deepmind/torch-hdf5/blob/41d87d827c431041cb679d7f39ad7b2868516bc9/luasrc/ffi.lua#L13 ) from
> if name == 'libhdf5' then

to:

> if name == 'hdf5' then

Because otherwise there are usually substituted libhdf5.lib, but when I try to run `local hdf5lib = ffi.load(hdf5LibPath)` with this file you will see: `%1 эх ты x∈ё Win32`
![error](http://i.imgur.com/IajMemi.jpg "error")

**This means that you need to pass in `ffi.load` the name of library with extension _*.dll_ on Win (!!!!!!!)**

After this hotfix, when you run `th -e "require 'hdf5'"` you most likely will see the error:

> Error: unable to locate HDF5 header file at

This is due to the fact that you have prescribed is not the right path in `HDF5_INCLUDE_PATH`. I corrected him on the path:

> C:/Anaconda2/envs/torch-vc14/Library/include

**_(But it is worth considering that there is spelled out the path to an installed hdf5 in MY Anaconda environment)_**

Then when you start `th -e "require 'hdf5'"` you can get around this error:

> gcc was not found.

To correct this you need to:

1. Install MinGW
2. To set the Path environment variable (in Windows) with the following: `...;C:\MinGW\bin` <- **_note that here the path to an installed MinGW in MY SYSTEM_**
3. Restart "VS* * Tools Command Prompt"
4. To re-activate the Anaconda environment and run torch-activate.bat

Then when you run the command `th -e "require 'hdf5'"` you will NOT see errors.

In conclusion, post below how looks the file `config.lua` I have:
> hdf5._config = {
> HDF5_INCLUDE_PATH = "C:/Anaconda2/envs/torch-vc14/Library/include",
> HDF5_LIBRARIES = "C:/Anaconda2/envs/torch-vc14/Library/bin/hdf5.dll"
> }

**_Note that you paths may be others._**

_I hope that the developers will fix this package under Win and no one will get the same pain and suffering as I am._

Contributor guide

Open the contributing guide

Research direction

Start with hdf5-0-0.rockspec and luasrc/ffi.lua, then reproduce the Windows setup with `th -e "require 'hdf5'"` using the paths and commands in the report. Trace the install location, DLL loading, HDF5 header and library lookup, and MinGW requirement; done means a Windows install can load hdf5 without manually copying files or editing generated configuration paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake, lua
Domain
build-system, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.