repro: stage cache hit although imported module has changed

Bug Report

Description

When defining a stage in dvc.yaml, it is important to declare dependencies via the deps field, so that DVC can leverage its cache to avoid rerunning the stage unnecessarily. Examples in the docs often show stages in which commands use local Python scripts and those scripts are also declared as a dependencies such that changes in those scripts will lead to reruns of the stages. See e.g.:

• https://dvc.org/doc/start/data-management/data-pipelines
• https://dvc.org/doc/user-guide/pipelines/defining-pipelines#stages

I think DVC's view on pipeline code organization and dependencies declaration is too narrow and may lead to incorrect stage caching. Let me elaborate.

A Python script is typically not self-contained but imports symbols from other local or third-party Python modules or packages. Especially local imports are subject to (breaking) change and must be declared as stage dependencies, too. In fact, the entire import tree of local imports must be considered for determining whether a stage needs to be rerun. But also third-party dependencies may change in ways that the output of a stage changes without any change in local code, e.g. when the version of a third-party dependency is bumped to a new major version. So ultimately, the entire import tree of a Python script must be considered. Of course, a change anywhere the import tree does not always mean the stage cache must be invalidated because the change may not affect the stage output, so there is a chance of rerunning a stage unnecessarily. But it's better to rerun too often than to miss a relevant change.

Reproduce

1. Run dvc init.
2. Add the following dvc.yaml file:

   stages:
     test:
       cmd: python a.py > out.txt
       deps:
         - a.py
       outs:
         - out.txt
   

3. Add a.py:

   from b import do
   
   do()
   

4. Add b.py:

   def do():
       print("hello")
   

5. Run dvc repro and observe that out.txt contains "hello".
6. Edit b.py to print "hello world" instead.
7. Run dvc repro and observe that out.txt hasn't changed and DVC prints:

   Stage 'test' didn't change, skipping                               
   Data and pipelines are up to date.
   

Expected

After editing b.py, dvc repro should have rerun the stage and out.txt should contain "hello world".

Environment information

Output of dvc doctor:

$ dvc doctor
DVC version: 2.47.1 (snap)
--------------------------
Platform: Python 3.8.10 on Linux-5.13.0-48-generic-x86_64-with-glibc2.29
Subprojects:
	dvc_data = 0.42.3
	dvc_objects = 0.21.1
	dvc_render = 0.2.0
	dvc_task = 0.2.0
	scmrepo = 0.1.15
Supports:
	azure (adlfs = 2023.1.0, knack = 0.10.1, azure-identity = 1.12.0),
	gdrive (pydrive2 = 1.15.1),
	gs (gcsfs = 2023.3.0),
	hdfs (fsspec = 2023.3.0, pyarrow = 11.0.0),
	http (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
	https (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
	oss (ossfs = 2021.8.0),
	s3 (s3fs = 2023.3.0, boto3 = 1.24.59),
	ssh (sshfs = 2023.1.0),
	webdav (webdav4 = 0.9.8),
	webdavs (webdav4 = 0.9.8),
	webhdfs (fsspec = 2023.3.0)
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/mapper/vgubuntu-root
Caches: local
Remotes: https
Workspace directory: ext4 on /dev/mapper/vgubuntu-root
Repo: dvc, git

Solution idea
To solve this problem, DVC could detect whether a stage dependency is written in Python, and if so, analyze the import tree using pydeps, based on which a better cache key could be computed that captures changes also in (transitively) imported files.

WDYT?

Start by running the reproduction with dvc.yaml, a.py, and b.py, then trace the stage dependency and cache-key handling used by dvc repro. The fix is complete when changing the imported b.py module causes the test stage to rerun and out.txt to contain the updated output, with regression coverage for the example.