python / python/typeshed

Decoupling tensorflow from keras in stubs

Open
#11,714 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

stubs: improvement
Dominant language
Python
Stars
5.1k
Forks
2.1k
Avg merge
1d 19h
Merged PRs (30d)
82

Description

Whilst working on https://github.com/python/typeshed/pull/11696, I was having issues referencing the source code. After reading the doc and a bit of investigation, I realized why:

The runtime of https://github.com/python/typeshed/tree/main/stubs/tensorflow/tensorflow/keras actually are just references to the keras source code at https://github.com/keras-team/keras/tree/master/keras (import keras), and is generated / injected in import tensorflow.keras by https://github.com/keras-team/keras/blob/6454a4888a494c20ab0ea1dc6912cbcc13c5f940/pip_build.py#L62

Even tensorflow's own doc links back to karas source code https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/Callback

>>> from keras import callbacks
2024-04-03 23:28:35.387416: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-04-03 23:28:36.959410: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
>>> callbacks.__file__
'...\\typeshed\\.venv\\lib\\site-packages\\keras\\callbacks\\__init__.py'
>>> from tensorflow.keras import callbacks 
>>> callbacks.__file__
'...\\typeshed\\.venv\\lib\\site-packages\\keras\\callbacks\\__init__.py'

image

image

In other words, keras "pollutes" the tensorflow namespace! I don't know if the type stubs specifications allow two distributions to write to the same stub-only package, which would allow us to perfectly reflect what's truly happening. Second best thing would be to have stubs for keras and keep tensorflow.keras in the tensorflow stubs (since tensorflow is dependent on keras anyway)

CC @hoel-bagard & @hmc-cs-mdrissi

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read stubs/tensorflow/tensorflow/keras alongside Keras's pip_build.py and the linked runtime sources to understand how tensorflow.keras is injected. Confirm the supported stub-package arrangement with the maintainers; done means the chosen layout accurately represents both keras and tensorflow.keras without namespace ambiguity.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, tensorflow
Domain
devtools, machine-learning
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.