Allow user to provide custom filter in ListingTableUrl

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Many object_stores are not as clean as expected. Eg: many have "hidden" files such as _SUCCESS or _temp/blahblah.
When using ListingTableUrl these files are not filtered and cause issues when inferring schema/options.

**Describe the solution you'd like**
ListingTableUrl has an optional glob but it would be more helpful if this was somewhat more generic, eg an optional Fn(&ObjectMeta) -> bool

**Describe alternatives you've considered**
We could implement the filtering at other layers in the application but they all seem to come with drawbacks/feeling awkward (eg: such filtering does not really belong in a wrapper around an object_store).

Start by locating ListingTableUrl and its existing optional glob filtering, then inspect how object metadata is used during schema and option inference. Done means an optional metadata predicate can exclude entries such as hidden files before inference, with tests covering the filtering behavior.

Written by the indexing model from the issue text.