google / google/ml-compiler-opt

TFRecord generator not working

Open
#4 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
798
Forks
123
Avg merge
55m
Merged PRs (30d)
4

Description

Thanks for the opensource code!

I have encountered several problems while trying to preproduce the model following the README file:

1. I will really appreciate it if you can provide a explanation on how to use the `extract_ir.py` tool.
Currently I am compiling for the .bc code from source .c file using `clang –c –emit-llvm –O1 X.c`, and using an empty file for X.cmd.
While running generate_default_trace.py, the clang command in inclining_runner.py (i.e. clang +cmds -mllvm -enable-ml-inliner=development ... at line 119) does not generate any log file. I have tried to run the clang command outside of the script, but still no log file is generated. I wish to know if that is caused by having something empty for cmds? If yes, it might be useful to provide some instructions on how to use extract_ir.py to generate a correct .cmd file. If no, that might be worth figuring out why clang is not generating log files.
I get around the log file problem by breaking the clang into two commands:
Firstly use`opt -passes=scc-oz-module-inliner -ml-inliner-ir2native-model= -training-log= -enable-ml-inliner=development -o ` to generate the log file
Then run clang to compile for the native.
I am not sure if this is a good replacement for the original code, and will really appreciate it if you can enlighten me on what's wrong with it.

2. It seems like this script is requiring cuda10, while the others are using cuda11.

3. On running train_bc.py, it hangs forever on
```
experience = next(dataset_iter) (in trainer.py)
```
At a closer at `dataset_iter`, we find that when we read from the TFRecord file in `_file_iterator_fn `(defined in `data_reader.py`), the dataset is actually empty.
I am not sure if it is the problem with how we read the file, or the TFRecord file generated from previous step.

It will be really helpful if you can give some insights on these points!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.