tensorflow / tensorflow/models

Difficult to contribute/add features to the repository.

Open
#9,441 0 comments 0 reactions 3 assignees View on GitHub

Nobody has claimed this yet.

models:research:odapi type:support
Dominant language
Python
Stars
77.7k
Forks
44.8k
PR merge metrics
No merged PRs in 30d

Description

I've been working on adding an external_input_reader to bypass the need for TFRecords (which at least I find annoying that it is the only way to input data to the pipeline as of now). The issue is that there are so many places where "tf_record_input_reader" is hard-coded, and there seem to be no way to globally register new component to the repository that I do not know how to integrate it in a nice way.

Examples:
1: https://github.com/tensorflow/models/blob/master/research/object_detection/builders/dataset_builder.py, in build(...)

There is if input_reader_config.WhichOneof('input_reader') == 'tf_record_input_reader', which raises ValueError if not modified. (This is essentially where I would insert my custom reader)

2: https://github.com/tensorflow/models/blob/master/research/object_detection/utils/config_util.py, in _update_tf_record_input_path(...) we have the same:

if input_reader_type == "tf_record_input_reader", and sure, its called "updatetf_record ..." but nevertheless it does not seem like it is written to support easy contribution.

It seems from the comments that it should be possible to extend the repository, e.g.

input_reader.proto:
"
// An externally defined input reader. Users may define an extension to this
// proto to interface their own input readers.
message ExternalInputReader {
extensions 1 to 999;
}
"

I've managed to get the reader to work with train/eval but the solution modifies quite a lot of files, and now I keep hitting speed-bumps with things like hparams_overrides (config overrides), where it seems like many things are hard-coded to support just the tfrecord input reader and nothing else.

Am I missing something?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.