astropy / astropy/astropy

Filetype identification can fail for some URLs

Open
#859 1 comment 0 reactions 0 assignees View on GitHub
unified-io
Dominant language
Python
Stars
5.3k
Forks
2.2k
Avg merge
1d 20h
Merged PRs (30d)
81

Description

File this bug under "theoretical" -- but we'll probably want to address it at some point.

As an example, the `is_latex` identifier is:

```
def is_latex(origin, args, kwargs):
return isinstance(args[0], basestring) and args[0].endswith('.tex')
```

This will fail if the file comes from a remote URL with parameters, such as:

```
http://astropy.org/texserver.tex?foo=bar
```

Rather than using `endswith`, we should probably write a utility function to check the extension of a file that uses some combination of `os.path.splitext` and `urlparse.urlparse().path`.

Even then, we're still in trouble, because:

1) `download_file` doesn't read the mimetype in the HTTP header. If it did so, it could use the `mimetypes` module to make a reasonable guess about a file extension to use.

2) Once it did that, the cache system would have to be updated to store the mimetype either in the cache database or in the file extension of the file things are cached in.

So the real solution is to move all of the file identifiers to do content-based checks whenever humanly possible and avoid all of this...

Contributor guide

Open the contributing guide

Research direction

Start with the is_latex identifier and the URL example in the issue, then inspect download_file and the cache system. Determine whether the intended scope is URL-aware extension checks, HTTP mimetype handling, cache metadata, or content-based identification; completion requires an agreed approach because the issue does not define a single finished change.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.