tar source plugin fails to download long urls
- Dominant language
- Python
- Stars
- 140
- Forks
- 45
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 6
Description
The following element file:
```yaml
kind: import
sources:
- kind: tar
# works
# url: curl:download/curl-8.14.1.tar.gz#000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
# fails
url: curl:download/curl-8.14.1.tar.gz#0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
ref: 6766ada7101d292b42b8b15681120acd68effa4a9660935853cf6d61f0d984d4
```
(with the curl alias defined in the project.conf as `curl: https://curl.se/`)
Fails to download:
```
$ bst source fetch harry-curl.bst
...
10:40:55 00:00:00 1525c594 fetch:harry-curl.bst BUG Fetch
An unhandled exception occured:
Traceback (most recent call last):
File "<...>/buildstream/src/buildstream/_scheduler/jobs/job.py", line 350, in child_action
result = self.child_process() # pylint: disable=assignment-from-no-return
^^^^^^^^^^^^^^^^^^^^
File "<...>/buildstream/src/buildstream/_scheduler/jobs/elementjob.py", line 81, in child_process
return self._action_cb(self._element)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<...>/buildstream/src/buildstream/_scheduler/queues/fetchqueue.py", line 75, in _fetch_not_original
element._fetch(fetch_original=False)
File "<...>/buildstream/src/buildstream/element.py", line 2252, in _fetch
self.__sources.fetch()
File "<...>/buildstream/src/buildstream/_elementsources.py", line 221, in fetch
self.fetch_sources()
File "<...>/buildstream/src/buildstream/_elementsources.py", line 250, in fetch_sources
self._fetch_source(source)
File "<...>/buildstream/src/buildstream/_elementsources.py", line 431, in _fetch_source
source._fetch()
File "<...>/buildstream/src/buildstream/source.py", line 1421, in _fetch
self.__do_fetch()
File "<...>/buildstream/src/buildstream/source.py", line 1927, in __do_fetch
new_source.fetch(**kwargs)
File "<...>/buildstream/src/buildstream/downloadablefilesource.py", line 341, in fetch
sha256 = self._ensure_mirror(
^^^^^^^^^^^^^^^^^^^^
File "<...>/buildstream/src/buildstream/downloadablefilesource.py", line 399, in _ensure_mirror
os.makedirs(self._mirror_dir, exist_ok=True)
File "", line 225, in makedirs
OSError: [Errno 36] File name too long: '<...>/.cache/buildstream/sources/tar/curl_download_curl_8_14_1_tar_gz_0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000'
Fetch failure on element: harry-curl.bst
````
I believe buildstream computes the mirror directory here https://github.com/apache/buildstream/blob/6558f9628aa812b8f31aac236e812d37f2be0b04/src/buildstream/downloadablefilesource.py#L277 using `utils.url_directory_name` which can generate filenames longer than 255 characters.
This URL is a ridiculous one but in an private project I have a file I need to download (with a very long url) where I cannot control the url I download from.
Contributor guide
Research direction
Start with src/buildstream/downloadablefilesource.py around mirror-directory computation and inspect utils.url_directory_name, then reproduce the failure with `bst source fetch harry-curl.bst` using the long URL from the issue. Done means the tar source can fetch that URL without triggering a file-name-too-long error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- build-system
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100