Filetype identification can fail for some URLs
- Dominant language
- Python
- Stars
- 5.3k
- Forks
- 2.2k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 81
Description
File this bug under "theoretical" -- but we'll probably want to address it at some point.
As an example, the `is_latex` identifier is:
```
def is_latex(origin, args, kwargs):
return isinstance(args[0], basestring) and args[0].endswith('.tex')
```
This will fail if the file comes from a remote URL with parameters, such as:
```
http://astropy.org/texserver.tex?foo=bar
```
Rather than using `endswith`, we should probably write a utility function to check the extension of a file that uses some combination of `os.path.splitext` and `urlparse.urlparse().path`.
Even then, we're still in trouble, because:
1) `download_file` doesn't read the mimetype in the HTTP header. If it did so, it could use the `mimetypes` module to make a reasonable guess about a file extension to use.
2) Once it did that, the cache system would have to be updated to store the mimetype either in the cache database or in the file extension of the file things are cached in.
So the real solution is to move all of the file identifiers to do content-based checks whenever humanly possible and avoid all of this...
Contributor guide
Research direction
Start with the is_latex identifier and the URL example in the issue, then inspect download_file and the cache system. Determine whether the intended scope is URL-aware extension checks, HTTP mimetype handling, cache metadata, or content-based identification; completion requires an agreed approach because the issue does not define a single finished change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100