glob("*") does not support matching non-utf8 filenames

Issue by kballard
Wednesday Jan 29, 2014 at 23:12 GMT

For earlier discussion, see https://github.com/rust-lang/rust/issues/11916

This issue was labelled with: A-libs, A-unicode, E-easy, E-mentor in the Rust repository

glob::glob() does not have any support right now for matching non-utf8 filenames. Not only are its patterns restricted to strings, but it also explicitly skips any non-utf8 filenames it encounters (which should at least be able to match a * pattern).

Tasks that need to be done:

• glob() needs to accept both strings and byte-vectors. It can do this using std::path::BytesContainer
• glob() needs to process its pattern as a byte vector instead of a string, which will allow it to process filenames as byte vectors. This includes matching non-utf8 filenames against * and ? tokens (for the latter, matching a single byte is appropriate; ideally, it would match however many bytes are supposed to be consumed to create a U+FFFD REPLACEMENT CHARACTER as per the unicode standard)

This is a sub-task of #9639.

Start by reading the glob::glob implementation and std::path::BytesContainer, then review the earlier discussion in issue #11916 and the parent task #9639. Done means glob accepts string and byte-vector patterns and can match non-UTF-8 filenames with * and ? as described in the checklist.