tensorflow / tensorflow/tensorboard

Global partial Dependence plots not working

Open
#1,797 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

plugin:what-if-tool stat:awaiting tensorflower type:bug
Dominant language
TypeScript
Stars
7.2k
Forks
1.7k
Avg merge
4d 22h
Merged PRs (30d)
1

Description

Summary
I'm haveing troubles getting the Global partial dependence plots to work, neither from Jupyter or Tensorboard. However, the partial dependence plots work when using "Selected datapoint". (screenshots below)

Info

  • TensorBoard version: 1.12.0
  • WitWidget version: couldn't find a version, but the latest you get from running: pip install --upgrade witwidget as of 2019-02-01.
  • OS Platform and version: from uname -a: Linux omitted.google.com 4.19.12-1rodete1-amd64 #1 SMP Debian 4.19.12-1rodete1 (2018-12-26) x86_64 GNU/Linux
  • Python version: 2.7.15

Description
As per the issue summary above, Global partial dependencies plot don't work, while the ones for "selected" datapoint do:

Global dependencies plots:
image

Selected datapoint dependencies plots:
image

On mouse hover in the broken plots you get this kind of tooltip:
image

So, the model I used is a canned estimator DNNLinearCombinedClassifier and I tried 2 different serving input functions but the result didn't change:


# first try
def what_if_serving_input_fn():
  feature_columns = featurizer.create_feature_columns()
  input_feature_columns = [
      feature_columns[feature_name] for feature_name in metadata.INPUT_FEATURE_NAMES]
  feat = tf.feature_column.make_parse_example_spec(input_feature_columns)
  return tf.estimator.export.build_parsing_serving_input_receiver_fn(feat)

# second try
def example_serving_input_fn():
    feature_columns = featurizer.create_feature_columns()
    input_feature_columns = [
      feature_columns[feature_name] for feature_name in metadata.INPUT_FEATURE_NAMES]

    example_bytestring = tf.placeholder(
        shape=[None],
        dtype=tf.string,
    )

    feature_scalars = tf.parse_example(
        example_bytestring,
        tf.feature_column.make_parse_example_spec(input_feature_columns)
    )

    features = {
        key: tensor
        for key, tensor in feature_scalars.iteritems()
    }

    return tf.estimator.export.ServingInputReceiver(
        features=process_features(features),
        receiver_tensors={'examples': example_bytestring}
    )

And the export is performed by the following code:


    estimator.export_saved_model(
      export_dir_base=os.path.join(extended_estimator.model_dir, 'what_if'),
      serving_input_receiver_fn=input.what_if_serving_input_fn()
    )
    # or if using the other input fn
    estimator.export_saved_model(
      export_dir_base=os.path.join(extended_estimator.model_dir, 'what_if'),
      serving_input_receiver_fn=input.example_serving_input_fn
    )

The code is deployed in a local docker container (tf serving) by running


    docker run -p 8500:8500 -p 8501:8501 --cpus=4 --memory=4g \
    --mount type=bind,source=$MODEL_DIR_LOCAL/what_if,target=/models/what_if \
    -e MODEL_NAME=what_if \
    -e TF_CPP_MIN_VLOG_LEVEL=0 \
    -t tensorflow/serving

And even by running the container with verbose logging no warnings/error were returned by tf serving.

Finally, when the jupyter notebooks code used to start the WitWidget is the following:

config_builder = WitConfigBuilder(examples) \
    .set_inference_address('localhost:8500') \
    .set_model_name('what_if') \
    .set_model_signature('classification') \
    .set_label_vocab(['low_affinity', 'high_affinity'])
WitWidget(config_builder, height=tool_height_in_px)

But as mentioned before, the other parts of the what_if tool work properly.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the issue in Jupyter and TensorBoard with the supplied DNNLinearCombinedClassifier, serving input functions, and WitConfigBuilder setup. Compare the working selected-datapoint plots with the broken global partial dependence plots, then inspect the WitWidget partial-dependence rendering and inference path. Done means global plots render correctly in both interfaces without breaking selected-datapoint plots.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, jupyter, python
Domain
data-visualization, frontend, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.