apache / apache/buildstream

tar source plugin fails to download long urls

Open
#2,034 2 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Python
Stars
140
Forks
45
Avg merge
1d 3h
Merged PRs (30d)
6

Description

The following element file:

```yaml
kind: import

sources:
- kind: tar
# works
# url: curl:download/curl-8.14.1.tar.gz#000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
# fails
url: curl:download/curl-8.14.1.tar.gz#0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
ref: 6766ada7101d292b42b8b15681120acd68effa4a9660935853cf6d61f0d984d4
```

(with the curl alias defined in the project.conf as `curl: https://curl.se/`)

Fails to download:

```
$ bst source fetch harry-curl.bst

...

10:40:55 00:00:00 1525c594 fetch:harry-curl.bst BUG Fetch

An unhandled exception occured:

Traceback (most recent call last):
File "<...>/buildstream/src/buildstream/_scheduler/jobs/job.py", line 350, in child_action
result = self.child_process() # pylint: disable=assignment-from-no-return
^^^^^^^^^^^^^^^^^^^^
File "<...>/buildstream/src/buildstream/_scheduler/jobs/elementjob.py", line 81, in child_process
return self._action_cb(self._element)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<...>/buildstream/src/buildstream/_scheduler/queues/fetchqueue.py", line 75, in _fetch_not_original
element._fetch(fetch_original=False)
File "<...>/buildstream/src/buildstream/element.py", line 2252, in _fetch
self.__sources.fetch()
File "<...>/buildstream/src/buildstream/_elementsources.py", line 221, in fetch
self.fetch_sources()
File "<...>/buildstream/src/buildstream/_elementsources.py", line 250, in fetch_sources
self._fetch_source(source)
File "<...>/buildstream/src/buildstream/_elementsources.py", line 431, in _fetch_source
source._fetch()
File "<...>/buildstream/src/buildstream/source.py", line 1421, in _fetch
self.__do_fetch()
File "<...>/buildstream/src/buildstream/source.py", line 1927, in __do_fetch
new_source.fetch(**kwargs)
File "<...>/buildstream/src/buildstream/downloadablefilesource.py", line 341, in fetch
sha256 = self._ensure_mirror(
^^^^^^^^^^^^^^^^^^^^
File "<...>/buildstream/src/buildstream/downloadablefilesource.py", line 399, in _ensure_mirror
os.makedirs(self._mirror_dir, exist_ok=True)
File "", line 225, in makedirs
OSError: [Errno 36] File name too long: '<...>/.cache/buildstream/sources/tar/curl_download_curl_8_14_1_tar_gz_0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000'

Fetch failure on element: harry-curl.bst
````

I believe buildstream computes the mirror directory here https://github.com/apache/buildstream/blob/6558f9628aa812b8f31aac236e812d37f2be0b04/src/buildstream/downloadablefilesource.py#L277 using `utils.url_directory_name` which can generate filenames longer than 255 characters.

This URL is a ridiculous one but in an private project I have a file I need to download (with a very long url) where I cannot control the url I download from.

Contributor guide

Open the contributing guide

Research direction

Start with src/buildstream/downloadablefilesource.py around mirror-directory computation and inspect utils.url_directory_name, then reproduce the failure with `bst source fetch harry-curl.bst` using the long URL from the issue. Done means the tar source can fetch that URL without triggering a file-name-too-long error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.