apache / apache/beam

Cache clean up might fail on Windows

Open
#20,584 0 comments 0 reactions 0 assignees View on GitHub
improvement interactive P3 python runners windows
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

Details can be found in this PR:
https://github.com/apache/beam/pull/12779

```

apache_beam\runners\interactive\recording_manager_test.py:75:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
apache_beam\runners\interactive\interactive_environment.py:118:
in new_env
_interactive_beam_env.cleanup()
apache_beam\runners\interactive\interactive_environment.py:272:
in cleanup
cache_manager.cleanup()
apache_beam\runners\interactive\caching\streaming_cache.py:391:
in cleanup
shutil.rmtree(self._cache_dir)
c:\hostedtoolcache\windows\python\3.6.8\x64\lib\shutil.py:500:
in rmtree
return _rmtree_unsafe(path, onerror)
c:\hostedtoolcache\windows\python\3.6.8\x64\lib\shutil.py:390:
in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
c:\hostedtoolcache\windows\python\3.6.8\x64\lib\shutil.py:395:
in _rmtree_unsafe
onerror(os.unlink, fullname, sys.exc_info())
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

...

> os.unlink(fullname)
E
PermissionError: [WinError 32] The process cannot access the file because it is being
used by another process: 'D:\\a\\beam\\beam\\sdks\\python\\target\\.tox\\py36-win\\tmp\\it-4m1c1oje2145793178144\\full\\fb91a47796-2145832985040-2145832986608-2145793178144'

```

This does not happen on Linux-like systems:
https://user-images.githubusercontent.com/4423149/100034273-b343ef80-2db0-11eb-8988-dcfc2c79322c.png

A potential solution is to enable ignore_errors and log about manually cleaning up onerror.

Imported from Jira [BEAM-11339](https://issues.apache.org/jira/browse/BEAM-11339). Original Jira may contain additional context.
Reported by: ningk.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.